Page 57

Pennsylvania Keystones of Health Section 1115 Demonstration Approval Period: December 26, 2024 through December 31, 2029 Page 44 of 90 providing data and analytic files to CMS, including entering into a data use agreement that explains how the data and data files will be exchanged, and providing a technical point of contact to support specification of the data and files to be disclosed, as well as relevant data dictionaries and record layouts. The state must include in its contracts with entities who collect, produce or maintain data and files for the demonstration, that they must make such data available for the federal evaluation as is required under 42 CFR 431.420(f) to support federal evaluation. This may also include the state’s participation—including representation from the state’s contractors, independent evaluators, and organizations associated with the demonstration operations, as applicable—in a federal learning collaborative aimed at cross state technical assistance, and identification of lessons learned and best practices for demonstration measurement, data development, implementation, monitoring, and evaluation. The state may claim administrative match for these activities. Failure to comply with this STC may result in a deferral being issued as outlined in Section 10.1. 11.2. Independent Evaluator. The state must use an independent party to conduct an evaluation of the demonstration to ensure that the necessary data is collected at the level of detail needed to research the approved hypotheses. The independent party must sign an agreement to conduct the demonstration evaluation in accord with the CMS-approved Evaluation Design. When conducting analyses and developing the evaluation reports, every effort should be made to follow the approved methodology. However, the state may request, and CMS may agree to, changes in the methodology in appropriate circumstances. 11.3. Evaluation Budget. A budget for the evaluation shall be provided with the Draft Evaluation Design. It will include the total estimated cost, as well as a breakdown of estimated staff, administrative and other costs for all aspects of the evaluation such as any survey and measurement development, quantitative and qualitative data collection and cleaning, analyses and report generation. A justification of the costs may be required by CMS if the estimates provided do not appear to sufficiently cover the costs of the design, if CMS finds that the design is not sufficiently developed, or if the estimates appear to be excessive. 11.4. Draft Evaluation Design. The state must submit, for CMS comment and approval, a draft Evaluation Design no later than 180 calendar days after the approval date of the demonstration. Any modifications to an existing approved Evaluation Design will not affect previously established requirements and timelines for report submission for the demonstration, if applicable. The draft Evaluation Design must be drafted in accordance with Attachment A (Developing the Evaluation Design) of these STCs, and any applicable evaluation guidance and technical assistance for the demonstration’s policy components. The Evaluation Design must also be developed in alignment with CMS guidance on applying robust evaluation approaches, such as quasi-experimental methods like difference-in-differences and interrupted time series, as well as establishing valid comparison groups and assuring causal inferences in demonstration evaluations. In addition to these requirements, if determined appropriate for the communities impacted by the demonstration, the state is encouraged to consider implementation approaches involving randomized control trials and staged rollout (for example, across geographic areas, by service setting, or by beneficiary characteristic)—as these implementation strategies help create strong comparison groups and facilitate robust evaluation.