Beyond RAGAS: A Compliance-Aware Evaluation Framework for Retrieval-Augmented Generation in Regulated Sectors

Abstract

Retrieval-Augmented Generation has become the dominant architecture for enterprise knowledge systems, with adoption spanning healthcare, education, and financial services. Existing evaluation frameworks — most notably RAGAS — measure faithfulness, answer relevance, context precision, and context recall. These metrics are necessary but insufficient for regulated sectors, where legal obligations imposed by the Family Educational Rights and Privacy Act (FERPA), the Health Insurance Portability and Accountability Act (HIPAA), and the General Data Protection Regulation (GDPR) introduce evaluation requirements that generic quality metrics cannot address. This paper introduces CAREF, the Compliance-Aware RAG Evaluation Framework, comprising five formally defined dimensions: Access-Control Precision (ACP), PHI Leakage Rate (PLR), Regulatory Citation Accuracy (RCA), Role-Aware Retrieval Precision (RARP), and Audit Trail Completeness (ATC). RAGAS faithfulness is shown to be statistically uncorrelated with FERPA Access-Control Precision (Pearson r = 0.12, p = 0.44), confirming that faithfulness optimisation cannot serve as a proxy for compliance. With CAREF applied, the composite Compliance Score improved from 0.51 to 0.93 on a production RAG system serving over 88,000 enrolled students.