Page 42

Virginia FAMIS MOMS and FAMIS Select Section 1115 Demonstration Demonstration Period: October 25, 2019 through June 30, 2029; Amended November 18, 2021 Page 36 of 61 Attachment B: Preparing the Interim and Summative Evaluation Reports

Introduction Both state and federal governments need rigorous quantitative and qualitative evidence to inform policy decisions. To that end, for states that are testing new approaches and flexibilities in their Medicaid programs through section 1115 demonstrations, evaluations are crucial to understand and disseminate information about these policies. The evaluations of new initiatives seek to produce new knowledge and direction for programs and inform Medicaid policy for the future. While a narrative about what happened during a demonstration provides important information, the principal focus of the evaluation of a section 1115 demonstration should be obtaining and analyzing data. Evaluations should including findings about the process (e.g., whether the demonstration is being implemented as intended), outcomes (e.g., whether the demonstration is having the intended effects on the target population), and impacts of the demonstration (e.g., whether the outcomes observed in the targeted population differ from outcomes in similar populations not affected by the demonstration).

Submission Timelines There is a specified timeline for the state’s submission of Evaluation Designs and Evaluation Reports. These dates are specified in the demonstration Special Terms and Conditions (STCs). The graphic below depicts an example of a deliverables timeline for a 5-year demonstration. In addition, the state should be aware that section 1115 evaluation documents are public records. In order to assure the dissemination of the evaluation findings, lessons learned, and recommendations, the state is required to publish the Interim and Summative Evaluation Reports to the state’s website within thirty (30) calendar days of CMS approval, as per 42 CFR 431.424(d). CMS will also publish a copy to the Medicaid.gov website.

Expectations for Evaluation Reports All states with Medicaid section 1115 demonstrations are required to conduct evaluations that are valid (the extent to which the evaluation measures what it is intended to measure), and reliable (the extent to which the evaluation could produce the same results when used repeatedly). The already-approved Evaluation Design is a map that begins with the demonstration goals, then transitions to the evaluation questions, and to the specific hypotheses, which will be used to investigate whether the demonstration has achieved its goals. When conducting analyses and developing the evaluation reports, every effort should be made to follow