Page 120

TEAMKY Demonstration

Approval Period: January 1, 2025 through December 31, 2029

Page 119 of 155

D. Methodological Limitations An important limitation of this evaluation is the absence of a comparison group. This is due to the statewide nature of the SUD demonstration and the lack of a comparable state not implementing similar SUD policies. The lack of a comparison group could generate bias in our estimate of the effect of the evaluation because we might be erroneously attributing changes in SUD-related outcomes to the demonstration. We will attempt to minimize this bias by including a rich set of covariates, but there remains a chance of bias due to factors we are unable to include in our model.

A second limitation, specific to the cost analysis, is the potential heterogeneity in the quality of the financial measures in the MCO encounter data. CMS's experience has been that Medicaid MCOs vary in the quality and completeness of their reporting; consequently, inference of expenditure effects could be confounded because of variation in financial data quality across plans and over time. If there is measurement error in the expenditure fields, standard errors will be inflated and analyses may understate the expenditure effects of the demonstration. Although we cannot rule out selection bias in the MCO encounter data, the Penn team's preliminary analyses of the financial data suggest that errors in these data fields appear to be small.

E. Attachments E.1. Independent Evaluator As experts in the implementation and evaluation of large randomized field experiments, the University of Pennsylvania was selected to be the independent evaluator of the full 1115 Medicaid waiver. Because the SUD demonstration was originally part of this broader 1115 waiver, the state contracted with University of Pennsylvania to evaluate the SUD demonstration as well.

In its role as evaluator of the larger waiver, the University of Pennsylvania team has developed significant experience conducting beneficiary surveys and collecting detailed qualitative interview data in Kentucky. The team also brings pre-existing deep expertise and experience working with administrative data, large datasets, survey data, and causal inference methods. The team will bring these skills and experience to bear on the SUD evaluation.

The University of Pennsylvania evaluation team commits to performing a fully independent evaluation of the Commonwealth of Kentucky’s Section 1115 Waiver demonstration. We attest to our independence in this evaluation, and agree to present our results to CMS and the general public through white papers and peer-reviewed journal articles without being influenced by any external partners, including the Commonwealth of Kentucky.

E.2. Evaluation Budget The budget for the SUD evaluation was initially encapsulated within the budget for the full 1115 waiver and was not developed as a separate budget. Below, we have estimated the total budget for the SUD evaluation as it would be if the evaluation of the SUD-specific part of the waiver were a completely separate evaluation. Since there are efficiencies in conducting both evaluations simultaneously, this SUD-only budget includes fixed costs that would have been spread out across the broader evaluation of the full demonstration.