Page 36

Attachment A Developing the Evaluation Design

Introduction

Both state and federal governments need rigorous quantitative and qualitative evidence to inform policy decisions. To that end, for states that are testing new approaches and flexibilities in their Medicaid programs through section 1115 demonstrations, evaluations are crucial to understand and disseminate information about these policies. The evaluations of new initiatives seek to produce new knowledge and direction for programs and inform Medicaid policy for the future. While a narrative about what happened during a demonstration provides important information, the principal focus of the evaluation of a section 1115 demonstration should be obtaining and analyzing data. Evaluations should include findings about the process (e.g., whether the demonstration is being implemented as intended), outcomes (e.g., whether the demonstration is having the intended effects on the target population), and impacts of the demonstration (e.g., whether the outcomes observed in the targeted population differ from outcomes in similar populations not affected by the demonstration).

Submission Timelines

There is a specified timeline for the state’s submission of its draft Evaluation Design and subsequent evaluation reports. The graphic below depicts an example of this timeline for a 5- year demonstration. In addition, the state should be aware that section 1115 evaluation documents are public records. The state is required to publish the Evaluation Design to the state’s website within thirty (30) calendar days of CMS approval, as per 42 CFR 431.424(e). CMS will also publish a copy to the Medicaid.gov website.

Interim Evaluation Summative Report (data from Evaluation Report Demo approved DY1-2.5) (data from DY1-5) Jan 1, 2017 Dec 31, 2020 June 30, 2023 @ @ @ @ @ Draft Evaluation Demo renewal Design Jan 1, 2022

April 30, 2017

Expectations for Evaluation Designs

CMS expects Evaluation Designs to be rigorous, incorporate baseline and comparison group assessments, as well as statistical significance testing. Technical assistance resources for constructing comparison groups and identifying causal inferences are available on Medicaid.gov: https://www.medicaid.gov/medicaid/section-1 1 15-demonstrations/1115-demonstration- monitoring-evaluation/1 1 15-demonstration-state-monitoring-evaluation-resources/index.html. If the state needs technical assistance using this outline or developing the Evaluation Design, the state should contact its demonstration team.

All states with section 1115 demonstrations are required to conduct Interim and Summative Evaluation Reports, and the Evaluation Design is the roadmap for conducting these evaluations.

Virginia FAMIS MOMS and FAMIS Select Section 1115 Demonstration Demonstration Period: October 25, 2019 through June 30, 2029; Amended November 18, 2021 Page 30 of 61