loading…
loading…
loading…
DEPARTMENT OF HEALTH & HUMAN SERVICES Centers for Medicare & Medicaid Services 7500 Security Boulevard, Mail Stop S2-25-26 Baltimore, Maryland 21244-1850 October 30, 2023 Brenda Tidball-Zeltinger Director, Division of Medical Services South Dakota Department of Social Services 700 Governors Drive, Kneip Building Pierre, SD 57501-2291 Dear Director Tidball-Zeltinger: The Centers for Medicare & Medicaid Services (CMS) is approving South Dakota’s (the “state”) request for a five-year extension of the demonstration titled, “South Dakota Former Foster Care Youth” (FFCY) (Project Number 11-W-00319/8)) (the “demonstration”), in accordance with section 1115(a) of the Social Security Act (“the Act”). This approval is effective from November 1, 2023, through October 31, 2028, upon which date, unless extended or otherwise amended, all authorities granted to operate this demonstration will expire. CMS’s approval is subject to the limitations specified in the attached expenditure authority, special terms and conditions (STC), and any supplemental attachment defining the nature, character, and extent of federal involvement in this project. The state may deviate from Medicaid state plan requirements only to the extent those requirements have been listed as not applicable to expenditures under the demonstration. Extent and Scope of Demonstration Extension South Dakota’s FFCY demonstration has been effective since May 2018. By CMS approving this extension request, the demonstration will continue to provide authority for FFCY under age 26, who currently reside in South Dakota, but were enrolled in Medicaid in a different state or tribe when they "aged out" of the foster care system. Section 1002(a) of the SUPPORT Act created a new Former Foster Care Children (FFCC) Medicaid state plan eligibility group, providing coverage for individuals who were receiving Medicaid while in foster care under the responsibility of any state; however, the new requirements apply exclusively to those who turn 18 on or after January 1, 2023.1 As a result, states still need section 1115 demonstration authority to continue coverage for individuals who turned 18 years old before January 1, 2023, until a beneficiary reaches age 26. Therefore, South Dakota’s FFCY demonstration eligibility will be limited to beneficiaries who turned 18 on or before December 31, 2022. 1 2022 State Health Official Letter. https://www.medicaid.gov/federal-policy-guidance/downloads/sho22003.pdf
Page 2 – Dir. Tidball-Zeltinger In the previous demonstration period, South Dakota’s eligibility requirements included an upper income limit of 182 percent of the federal poverty level. With this extension, South Dakota will no longer include an upper income limit, permitting out-of-state FFCY with any income to be demonstration eligible, provided they otherwise meet all demonstration eligibility requirements described in STCs. Budget Neutrality CMS has deemed this extension of the FFCY demonstration to be budget neutral because the demonstration authority to cover the FFCY population is needed for only a temporary period, through 2030, when all FFCY will be covered via the Medicaid state plan FFCC population. Further, through monitoring budget neutrality, CMS determined that the actual experience of states’ covering out-of-state FFCY resulted in limited total expenditures and low enrollment within the demonstrations. CMS generally believes that this FFCY demonstration coverage poses minimal financial risk to the federal government since FFCY demonstration spending is miniscule across states. This decision will increase the administrative ease of maintaining FFCY demonstration coverage in South Dakota. The state will be required to report total expenditures and member months in its demonstration monitoring reports. The state must still report quarterly claims and report expenditures on the CMS 64.9 WAIVER form. Failure to report FFCY expenditures and member months will result in reinstatement of the budget neutrality requirement. Monitoring and Evaluation Consistent with the demonstration STCs, the state submitted its Interim Evaluation Report for the prior demonstration approval period with the extension application. The report indicates that the small number of beneficiaries enrolled in the demonstration are generally maintaining coverage and receiving needed medical services, and there is no evidence the demonstration has negative impacts. Consistent with CMS’s requirements for section 1115 demonstrations, and as outlined in the demonstration’s STCs, the state is required to conduct systematic monitoring and comprehensive evaluation of the demonstration per applicable CMS guidance and technical assistance. Monitoring will support tracking the state’s progress toward its demonstration goals and offer timely mitigation opportunities if results indicate any areas of concern. The state and CMS will work collaboratively to finalize a list of metrics to be reported on its demonstration’s Annual Monitoring Reports, in addition to applicable narrative and qualitative information. Furthermore, the state must develop a sound Evaluation Design to support a meaningful evaluation of the demonstration to assess whether the demonstration is effective in producing the desired outcomes for its beneficiaries as well as the state’s Medicaid program overall. Public Notice To increase the transparency of demonstration projects, section 1115(d)(1) and (2) of the Act directs the Secretary of Health & Human Services to issue regulations providing for two periods
Page 3 – Dir. Tidball-Zeltinger of public comment on a state’s application for a section 1115 demonstration that would result in an impact on eligibility, enrollment, benefits, cost-sharing, or financing. The first comment period occurs at the state level before submission of the section 1115 application, and the second comment period occurs at the federal level after the application is received by the Secretary.
The Affordable Care Act (ACA) specified that comment periods should be “sufficient to ensure a meaningful level of public input, sections 1115(d)(2)(A) and (C) of the Act, but the statute imposed no additional requirement on the states or the Secretary to provide an individualized response to address those comments, as might otherwise be required under a general rulemaking. Accordingly, the implementing regulations issued in 2012 provide that CMS will review and consider all comments received by the deadline, but will not provide individualized written responses to public comments (42 CFR 431.416(d)(2)).
CMS held its federal comment period from November 2, 2022 through December 2, 2022, and received no comments. CMS has concluded that the demonstration is likely to assist in promoting the objectives of Medicaid.
Other Information
CMS’s approval of this demonstration project is contingent upon compliance with the enclosed expenditure authority and the STCs defining the nature, character, and extent of anticipated federal involvement in the demonstration. The award is subject to our receiving your written acknowledgement of the award and acceptance of these STCs within 30 days of the date of this letter.
The project officer for this demonstration is Valisha Andrus. Ms. Andrus is available to answer any questions concerning your section 1115 demonstration. Ms. Andrus’ contact information is as follows:
Centers for Medicare & Medicaid Services Center for Medicaid and CHIP Services Mail Stop: S2-25-26 7500 Security Boulevard Baltimore, MD 21244-1850 Email: Valisha.Andrus@cms.hhs.gov
We appreciate your state’s commitment to improving the health of people in South Dakota, and we look forward to partnering with you on the South Dakota Former Foster Care Youth 1115(a) demonstration. If you have questions with regards to this approval, please contact Ms. Mehreen Rashid, Acting Director, State Demonstrations Group, Center for Medicaid and CHIP Services, at (410) 786-9686.
Sincerely,
Page 4 – Dir. Tidball-Zeltinger Daniel Tsai Deputy Administrator and Director Enclosure Cc: Mandy Strom, State Monitoring Lead, Medicaid and CHIP Operations Group
1 Demonstration Approval: November 1, 2023, through October 31, 2028 CENTERS FOR MEDICARE & MEDICAID SERVICES EXPENDITURE AUTHORITY
NUMBER: 11-W-00319/8
TITLE:
South Dakota Former Foster Care Youth
AWARDEE: South Dakota Department of Social Services
Under the authority of section 1115(a)(2) of the Social Security Act (the Act), expenditures made by South Dakota for the items identified below, which are not otherwise included as expenditures under section 1903 of the Act shall, for the period of this demonstration, be regarded as expenditures under the state’s title XIX plan. All requirements of the Medicaid statute will be applicable to such expenditure authorities, except those specified below as not applicable to these expenditure authorities.
The following expenditure authority shall enable the state to operate its Former Foster Care Youth section 1115 Medicaid demonstration through October 31, 2028 and also assists the state with meeting the intended Medicaid program objectives of this demonstration, which are to increase and strengthen overall coverage of former foster care youth and improve health outcomes for this population.
Title XIX Expenditure Authority
Expenditures to extend eligibility for full Medicaid state plan benefits to former foster care youth who are under age 26, who turned 18 on or before December 31, 2022, who were in foster care under the responsibility of another state or tribe on the date of attaining 18 years of age (or such higher age as the state has elected for termination of Federal foster care assistance under title IV- E of the Act), were enrolled in Medicaid on the date of aging out of foster care, and are now applying for Medicaid in South Dakota.
2 Demonstration Approval: November 1, 2023, through October 31, 2028 CENTERS FOR MEDICARE AND MEDICAID SERVICES SPECIAL TERMS AND CONDITIONS
NUMBER: 11-W-00319/8
TITLE:
South Dakota Former Foster Care Youth
AWARDEE: South Dakota Department of Social Services
I. PREFACE
The following are the Special Terms and Conditions (STCs) for the South Dakota Former Foster Care Youth section 1115(a) Medicaid demonstration (hereinafter referred to as “demonstration”). The parties to this agreement are the South Dakota Department of Social Services (“South Dakota” or "the state") and the Centers for Medicare & Medicaid Services (CMS). The STCs set forth in detail the nature, character, and extent of federal involvement in the demonstration and South Dakota's obligations to CMS during the life of the demonstration. The STCs are effective November 1, 2023 through October 31, 2028.
The STCs have been arranged into the following subject areas:
I. Preface II. Program Description and Objectives III. General Program Requirements IV. Eligibility and Benefits V. General Monitoring and Reporting Requirements VI. Monitoring Budget Neutrality for the Demonstration VII. General Financial Requirements VIII. Evaluation of the Demonstration IX. Schedule of Deliverables
Attachment A: Developing the Evaluation Design
Attachment B: Preparing the Interim and Summative Evaluation Reports Attachment C: Approved Evaluation Design (reserved)
3 Demonstration Approval: November 1, 2023, through October 31, 2028 II. PROGRAM DESCRIPTION AND OBJECTIVES
South Dakota’s Former Foster Care Youth (FFCY) demonstration has been effective since May 2018. The demonstration enables South Dakota to provide Medicaid coverage to former foster care youth under age 26 who were in foster care under the responsibility of another state or tribe on the date of attaining 18 years of age (or such higher age as the state has elected for termination of federal foster care assistance under title IV-E of the Act), were enrolled in Medicaid on the date of aging out, and are now applying for Medicaid in South Dakota. The Medicaid program objectives of this demonstration are to increase and strengthen overall coverage of former foster care youth and improve health outcomes for this population.
Section 1002(a) of the SUPPORT Act created a Former Foster Care Children (FFCC) Medicaid state plan eligibility group, providing coverage for individuals who were receiving Medicaid while in foster care under the responsibility of any state, however, the new requirements apply exclusively to individuals who turn 18 on or after January 1, 2023. As a result, South Dakota maintains section 1115 demonstration authority to continue coverage for individuals who turned 18 years old on or before December 31, 2022, until a beneficiary reaches age 26.
III. GENERAL PROGRAM REQUIREMENTS
1. Compliance with Federal Non-Discrimination Statutes. The state must comply with all applicable federal statutes relating to non-discrimination. These include, but are not limited to, the Americans with Disabilities Act of 1990, Title VI of the Civil Rights Act of 1964, section 504 of the Rehabilitation Act of 1973, and the Age Discrimination Act of 1975, and section 1557 of the Patient Protection and Affordable Care Act (Section 1557).
2. Compliance with Medicaid Law, Regulation, and Policy. All requirements of the Medicaid programs expressed in law, regulation, and policy statement not expressly waived or identified as not applicable in the expenditure authority document (which is a part of these terms and conditions), must apply to the demonstration.
3. Changes in Medicaid Law, Regulation, and Policy. The state must, within the timeframes specified in federal law, regulation, or written policy, come into compliance any with changes in law, regulation, or policy affecting the Medicaid or CHIP programs that occur during this demonstration approval period, unless the provision being changed is expressly waived or identified as not applicable. In addition, CMS reserves the right to amend the STCs to reflect such changes and/or changes as needed without requiring the state to submit an amendment to the demonstration under STC 7. CMS will notify the state 30 days in advance of the expected approval date of the amended STCs to allow the state to provide comment. Changes will be considered in force upon issuance of the approval letter by CMS. The state must accept the changes in writing.
4. Impact on Demonstration of Changes in Federal Law, Regulation, and Policy.
a) To the extent that a change in federal law, regulation, or policy requires either a reduction or an increase in federal financial participation (FFP) for expenditures made
4 Demonstration Approval: November 1, 2023, through October 31, 2028 under this demonstration, the state must adopt, subject to CMS approval, a modified budget neutrality agreement for the demonstration as necessary to comply with such change, as well as a modified allotment neutrality worksheet as necessary to comply with such change. The trend rates for the budget neutrality agreement are not subject to change under this subparagraph. Further, the state may seek an amendment to the demonstration (as per STC 7 of this section) as a result of the change in FFP.
b) If mandated changes in the federal law requires state legislation, unless otherwise prescribed by the terms of the federal law, the changes must take effect on the earlier of the day such state legislation becomes effective, or on the last day such legislation was required to be in effect under the law, whichever is sooner.
5. State Plan Amendments. The state will not be required to submit title XIX or XXI state plan amendments (SPAs) for changes affecting any populations made eligible solely through the demonstration. If a population eligible through the Medicaid or CHIP state plan is affected by a change to the demonstration, a conforming amendment to the appropriate state plan is required, except as otherwise noted in these STCs. In all such cases, the Medicaid and CHIP state plans govern.
6. Changes Subject to the Amendment Process. Changes related to eligibility, enrollment, benefits, beneficiary rights, delivery systems, cost sharing, sources of non-federal share of funding, budget neutrality, and other comparable program elements must be submitted to CMS as amendments to the demonstration. All amendment requests are subject to approval at the discretion of the Secretary in accordance with section 1115 of the Act. The state must not implement changes to these elements without prior approval by CMS either through an approved amendment to the Medicaid or CHIP state plan or amendment to the demonstration. Amendments to the demonstration are not retroactive and no FFP of any kind, including for administrative or medical assistance expenditures, will be available under changes to the demonstration that have not been approved through the amendment process set forth in STC 7 below, except as provided in STC 3.
7. Amendment Process. Requests to amend the demonstration must be submitted to CMS in writing for approval no later than 120 days prior to the planned date of implementation of the change and may not be implemented until approved. CMS reserves the right to deny or delay approval of a demonstration amendment based on non-compliance with these STCs, including but not limited to the failure by the state to submit required elements of a complete amendment request as described in this STC, and failure by the state to submit required reports and other deliverables in a timely fashion according to the deadlines specified therein. Amendment requests must include, but are not limited to, the following:
a) An explanation of the public process used by the state, consistent with the requirements of STC 12. Such explanation must include a summary of any public feedback received and identification of how this feedback was addressed by the state in the final amendment request submitted to CMS;
5 Demonstration Approval: November 1, 2023, through October 31, 2028 b) A detailed description of the amendment, including impact on beneficiaries, with sufficient supporting documentation;
c) A data analysis which identifies the specific “with waiver” impact of the proposed amendment on the current budget neutrality agreement. Such analysis must include current total computable “with waiver” and “without waiver” status on both a summary and detailed level through the current approval period using the most recent actual expenditures, as well as summary and detailed projections of the change in the “with waiver” expenditure total as a result of the proposed amendment, which isolates (by Eligibility Group) the impact of the amendment;
d) An up-to-date CHIP allotment neutrality worksheet, if necessary; and,
e) The state must provide updates to existing demonstration reporting and quality and evaluation plans. This includes a description of how the evaluation design and annual progress reports will be modified to incorporate the amendment provisions, as well as the oversight, monitoring and measurement of the provisions.
8. Extension of the Demonstration. States that intend to request an extension of the demonstration must submit an application to CMS at least 12 months in advance from the Governor or Chief Executive Officer of the state in accordance with the requirements of 42 CFR 431.412(c). States that do not intend to request an extension of the demonstration beyond the period authorized in these STCs must submit phase-out plan consistent with the requirements of STC 9.
9. Demonstration Transition and Phase-Out. The state may only suspend or terminate this demonstration, in whole or in part, at any time prior to the date of expiration consistent with the following requirements:
a) Notification of Suspension or Termination: The state must promptly notify CMS in writing of the reason(s) for the suspension or termination, together with the effective date and a transition and phase-out plan. The state must submit a notification letter and a draft transition and phase-out plan to CMS no less than six months before the effective date of the demonstration’s suspension or termination. Prior to submitting the draft transition and phase-out plan to CMS, the state must publish on its website the draft transition and phase-out plan for a 30-day public comment period. In addition, the state must conduct tribal consultation in accordance with STC 12, if applicable. Once the 30-day public comment period has ended, the state must provide a summary of the issues raised by the public during the comment period and how the state considered the comments received when developing the revised transition and phase-out plan.
b) Transition and Phase-out Plan Requirements: The state must include, at a minimum, in its phase-out plan the process by which it will notify affected beneficiaries, the content of said notices (including information on the beneficiary’s appeal rights), the process by which the state will conduct administrative reviews of Medicaid eligibility
6 Demonstration Approval: November 1, 2023, through October 31, 2028 for the affected beneficiaries, and ensure ongoing coverage for those beneficiaries whether currently enrolled or determined to be eligible individuals, as well as any community outreach activities, including community resources that are available.
c) Transition and Phase-out Plan Approval: The state must obtain CMS approval of the transition and phase-out plan prior to the implementation of phase-out activities. Implementation of transition and phase-out activities must be no sooner than 14 calendar days after CMS approval of the transition and phase-out plan.
d) Transition and Phase-out Procedures: The state must comply with all notice requirements found in 42 CFR §431.206, §431.210 and §431.213. In addition, the state must assure all appeal and hearing rights are afforded to demonstration participants as outlined in 42 CFR §431.220 and §431.221. If a demonstration participant requests a hearing before the date of action, the state must maintain benefits as required in 42 CFR §431.230. In addition, the state must conduct administrative renewals for all affected beneficiaries in order to determine if they qualify for Medicaid eligibility under a different eligibility category as found in 42 CFR §435.916.
e) Exemption from Public Notice Procedures: CMS may expedite federal and state public notice requirements in accordance with the circumstances described in 42 CFR §431.416(g).
f) Enrollment Limitation during Demonstration Phase-Out: If the state elects to suspend, terminate, or not extend this demonstration, during the last six months of the demonstration, enrollment of new individuals into the demonstration must be suspended. The limitation of enrollment into the demonstration does not impact the state’s obligation to determine Medicaid eligibility in accordance with the approved Medicaid state plan.
g) FFP: If the project is terminated or any relevant waivers suspended by the state, FFP shall be limited to normal closeout costs associated with terminating the demonstration including services and administrative costs of disenrolling beneficiaries.
10. Withdrawal of Waiver or Expenditure Authority. CMS reserves the right to withdraw waivers and/or expenditure authorities at any time it determines that continuing the waiver or expenditure authorities would no longer be in the public interest or promote the objectives of title XIX and title XXI. CMS will promptly notify the state in writing of the determination and the reasons for the withdrawal, together with the effective date, and afford the state an opportunity to request a hearing to challenge CMS’s determination prior to the effective date. If a waiver or expenditure authority is withdrawn, FFP is limited to normal closeout costs associated with terminating the waiver or expenditure authority, including services, continued benefits as a result of beneficiary appeals, and administrative costs of disenrolling beneficiaries.
7 Demonstration Approval: November 1, 2023, through October 31, 2028 11. Adequacy of Infrastructure. The state must ensure the availability of adequate resources for implementation and monitoring of the demonstration, including education, outreach, and enrollment; maintaining eligibility systems applicable to the demonstration; compliance with cost sharing requirements; and reporting on financial and other demonstration components.
12. Public Notice, Tribal Consultation, and Consultation with Interested Parties. The state must comply with the state notice procedures as required in 42 CFR §431.408 prior to submitting an application to extend the demonstration. For applications to amend the demonstration, the state must comply with the state notice procedures set forth in 59 Fed. Reg. 49249 (September 27, 1994) prior to submitting such request. The state must also comply with the public notice procedures set forth in 42 CFR §447.205 for changes in statewide methods and standards for setting payment rates.
A state with Federally-recognized Indian Tribes, Indian Health Programs, and/or Urban Indian Health Organizations must comply with the tribal consultation requirements set forth in section 1902(a)(73) of the Act and implemented in regulation at 42 CFR §431.408(b) or the tribal consultation requirements contained in the state’s approved Medicaid State Plan, when any program changes to the demonstration, either through amendment as set out in STC 7 or extension, are proposed by the state.
13. Federal Financial Participation (FFP). No federal matching funds for administrative or medical assistance payments for services provided under this demonstration will take effect until the effective date identified in the CMS demonstration approval documents. Expenditures for this demonstration will take effect until the effective date identified in the demonstration approval letter associated with these STCs.
14. Administrative Authority. When there are multiple entities involved in the administration of the demonstration, the Single State Medicaid agency must maintain authority, accountability, and oversight of the program. The State Medicaid agency must exercise oversight of all delegated functions to operating agencies, managed care organizations (MCOs), prepaid inpatient health plans (PIHPs), prepaid ambulatory health plans (PAHPs), and any other contracted entities. The Single State Medicaid agency is responsible for the content and oversight of the quality strategies for the demonstration.
15. Common Rule Exemption. The state must ensure the only involvement of human subjects in research activities authorized and/or required by this demonstration is for projects conducted by or subject to the approval of CMS, and that are designed to study, evaluate, or otherwise examine the Medicaid or CHIP program—including public benefit or service programs, procedures for obtaining Medicaid or CHIP benefits or services, possible changes in or alternatives to Medicaid or CHIP programs and procedures, or possible changes in methods or levels of payment for Medicaid benefits or services. CMS has determined that this demonstration, as represented in these approved STCs, meets the requirements for exemption from the human subject research provisions of the Common Rule set forth in 45 CFR 46.104(d)(5).
IV. ELIGIBILITY AND BENEFITS
8 Demonstration Approval: November 1, 2023, through October 31, 2028
16. Eligibility for the Demonstration. Individuals eligible for this demonstration are limited to “out-of-state former foster care youth” who are defined as individuals under age 26 that meet the following criteria:
a) were in foster care under the responsibility of a state other than South Dakota or a tribe in such other state when they turned age 18 (or such higher age as the state has elected for termination of Federal foster care assistance under title IV-E of the Act); b) were enrolled in Medicaid at the time of aging out of foster care; c) turned 18 on or before December 31, 2022; d) are now applying for Medicaid in South Dakota; and, e) are not otherwise eligible for Medicaid.
17. Benefits and Cost-sharing provided under the Demonstration. Out-of-state former foster care youth will receive the same Medicaid State Plan benefits and be subject to the same cost-sharing requirements effectuated by the state for the mandatory title IV-E foster care youth eligibility category enacted by the Adoption Assistance and Child Welfare Act of 1980 (Pub. L. 96-272).
18. Delivery System. Enrollees in this demonstration will receive services through the state's managed care delivery system.
V. MONITORING AND REPORTING REQUIREMENTS
19. Deferral for Failure to Submit Timely Demonstration Deliverables. CMS may issue deferrals in accordance with 42 CFR part 430 subpart C, in the amount of $5,000,000 per deliverable (federal share) when items required by these STCs (e.g., required data elements, analyses, reports, design documents, presentations, and other items specified in these STCs (hereafter singularly or collectively referred to as “deliverable(s)”)) are not submitted timely to CMS or are found to be inconsistent with the requirements approved by CMS. A deferral shall not exceed the value of the federal amount for the demonstration. The state does not relinquish its rights provided under 42 CFR part 430 subpart C to challenge any CMS finding that the state materially failed to comply with the terms of this agreement. The following process will be used: 1) 30 days after the deliverable was due if the state has not submitted a written request to CMS for approval of an extension as described in subsection (b) below; or 2) 30 days after CMS has notified the state in writing that the deliverable was not accepted for being inconsistent with the requirements of this agreement and the information needed to bring the deliverable into alignment with CMS requirements:
a) CMS will issue a written notification to the state providing advance notification of a pending deferral for late or non-compliant submission of required deliverable(s). For each deliverable, the state may submit to CMS a written request for an extension to submit the required deliverable that includes a supporting rationale for the cause(s) of the delay and the state’s anticipated date of submission. Should CMS agree to the state’s request, a corresponding extension of the deferral process can be provided.
9 Demonstration Approval: November 1, 2023, through October 31, 2028 b) CMS may agree to a corrective action as an interim step before applying the deferral, if corrective action is proposed in the state’s written extension request.
c) If CMS agrees to an interim corrective process in accordance with subsection (b), and the state fails to comply with the corrective action steps or still fails to submit the overdue deliverable(s) that meets the terms of this agreement, CMS may proceed with the issuance of a deferral against the next Quarterly Statement of Expenditures reported in Medicaid Budget and Expenditure System/State Children's Health Insurance Program Budget and Expenditure System (MBES/CBES) following a written deferral notification to the state.
d) If the CMS deferral process has been initiated for the state’s non-compliance with the terms of this agreement for submitting deliverable(s), and the state submits the overdue deliverable(s), and such deliverable(s) are accepted by CMS as meeting the standards outlined in these STCs, the deferral(s) will be released.
As the purpose of a section 1115 demonstration is to test new methods of operation or service delivery, the state’s failure to submit all required reports, evaluations, and other deliverables will be considered by CMS in reviewing any application for an extension, amendment, or for a new demonstration.
20. Submission of Post-Approval Deliverables. The state must submit all deliverables as stipulated by CMS and within the timeframes outlined within these STCs. 21. Compliance with Federal Systems Updates. As federal systems continue to evolve and incorporate additional section 1115 demonstration reporting and analytics functions, the state will work with CMS to: a) Revise the reporting templates and submission processes to accommodate timely compliance with the requirements of the new systems; b) Ensure all section 1115, T-MSIS, and other data elements that have been agreed to for reporting and analytics are provided by the state; and c) Submit deliverables to the appropriate system as directed by CMS. 22. Annual Monitoring Reports. The state must submit one Annual Monitoring Report each demonstration year (DY) and is due no later than 90 calendar days following the end of the DY. The state must submit a revised Annual Monitoring Report within 60 calendar days after receipt of CMS’s comments, if any. The reports will include all required elements as per 42 CFR 431.428, and 431.428 and should not direct readers to links outside the report. Additional links not referenced in the document may be listed in a Reference/Bibliography section. The Annual Monitoring Reports must follow the framework to be provided by CMS, which is subject to change as monitoring systems are developed/evolve and must be provided in a structured manner that supports federal tracking and analysis.
a) Operational Updates. Per 42 CFR 431.428, the Monitoring Reports must document any policy or administrative difficulties in operating the demonstration. The reports must provide sufficient information to document key operational and other
10 Demonstration Approval: November 1, 2023, through October 31, 2028 challenges, underlying causes of challenges, how challenges are being addressed, as well as key achievements and to what conditions and efforts successes can be attributed. The discussion should also include any issues or complaints identified by beneficiaries; lawsuits or legal actions; unusual or unanticipated trends; legislative updates; and descriptions of any public forums held. In addition, Monitoring Reports should describe key achievements, as well as the conditions and efforts to which these successes can be attributed. Monitoring Reports should also include a summary of all public comments received through post-award public forums regarding the progress of the demonstration.
b ) Performance Metrics. Per 42 CFR 431.428, the Monitoring Reports must document the impact of the demonstration on beneficiaries’ outcomes of care, quality and cost of care, and access to care, as applicable. This should also include the results of beneficiary satisfaction or experience of care surveys, if conducted, as well as grievances and appeals.
The state and CMS will work collaboratively to finalize the list of metrics to be reported on in Annual Monitoring Reports. The required monitoring and performance metrics must be included in the Monitoring Reports, and will follow the CMS framework provided by CMS to support federal tracking and analysis.
c) Financial Reporting Requirements. Per 42 CFR 431.428, the Monitoring Reports must document the financial performance of the demonstration. The state must provide total enrollment and total expenditures with every Monitoring Report. In addition, the state must report annual expenditures associated with the populations affected by this demonstration on the Form CMS-64. Administrative costs for this demonstration should be reported separately on the CMS-64.
d) Evaluation Activities and Interim Findings. Per 42 CFR 431.428, the Monitoring Reports must document any results of the demonstration to date per the evaluation hypotheses. Additionally, the state shall include a summary of the progress of evaluation activities, including key milestones accomplished, as well as challenges encountered and how they were addressed. 23. Corrective Action Plan Related to Monitoring. If monitoring indicates that demonstration features are not likely to assist in promoting the objectives of Medicaid, CMS reserves the right to require the state to submit a corrective action plan to CMS for approval. A state corrective action plan could include a temporary suspension of implementation of demonstration programs in circumstances where monitoring data indicate substantial and sustained directional change inconsistent with demonstration goals. A corrective action plan may be an interim step to withdrawing waivers or expenditure authorities, as outlined in STC 10. CMS will withdraw an authority, as described in STC 10, when metrics indicate substantial and sustained directional change inconsistent with the state’s demonstration goals, and the state has not implemented corrective action. CMS further has the ability to suspend implementation of the demonstration should corrective actions not effectively resolve these concerns in a timely manner.
11 Demonstration Approval: November 1, 2023, through October 31, 2028 24. Close-Out Report. Within 120 calendar days after the expiration of the demonstration, the state must submit a draft Close-Out Report to CMS for comments. a. The Close-Out Report must comply with the most current guidance from CMS. b. In consultation with CMS, and per guidance from CMS, the state will include an evaluation of the demonstration (or demonstration components) that are to phase out or expire without extension along with the Close-Out Report. Depending on the timeline of the phase-out during the demonstration approval period, in agreement with CMS, the evaluation requirement may be satisfied through the Interim and/or Summative Evaluation Reports stipulated in STC 32 and 33, respectively. c. The state will present to and participate in a discussion with CMS on the Close- Out report. d. The state must take into consideration CMS’s comments for incorporation into the final Close-Out Report. e. A revised Close-Out Report is due to CMS no later than 30 days after receipt of CMS’s comments. f. A delay in submitting the draft or final version of the Close-Out Report may subject the state to penalties described in STC 19.
25. Monitoring Calls. CMS will convene periodic conference calls with the state. a. The purpose of these calls is to discuss ongoing demonstration operation, to include (but not limited to), any significant actual or anticipated developments affecting the demonstration. Examples include implementation activities, trends in reported data on metrics and associated mid-course adjustments, budget neutrality, and progress on evaluation activities. b. CMS will provide updates on any pending actions, as well as federal policies and issues that may affect any aspect of the demonstration. c. The state and CMS will jointly develop the agenda for the calls.
26. Post Award Forum. Pursuant to 42 CFR 431.420(c), within 6 months of the demonstration’s implementation, and annually thereafter, the state shall afford the public with an opportunity to provide meaningful comment on the progress of the demonstration. At least 30 calendar days prior to the date of the planned public forum, the state must publish the date, time, and location of the forum in a prominent location on its website. The state must also post the most recent Annual Monitoring Report on its website with the public forum announcement. Pursuant to 42 CFR 431.420(c), the state must include a summary of the public comments in the next Annual Monitoring Report.
VI. MONITORING BUDGET NUETRALITY FOR THE DEMONSTRATION
26. Budget Neutrality. CMS has determined that this demonstration is budget neutral based on CMS’s assessment that the expenditure authority granted for the demonstration has minimal federal Medicaid expenditures and these populations could have been covered through waiver only authority. The state will not be allowed to obtain budget neutrality “savings” from this demonstration. The demonstration will not include a budget neutrality expenditure limit; however, the state is required to report total expenditures and member months in their
12 Demonstration Approval: November 1, 2023, through October 31, 2028 demonstration monitoring reports, per STC 22. The state must report quarterly claims and report expenditures on the CMS 64.9 WAIVER form. Failure to report FFCY expenditures and member months will result in reinstatement of the budget neutrality requirement. CMS reserves the right to request budget neutrality worksheets, requirements, limits, and analyses from the state at any time, or whenever the state seeks a change to the demonstration, per STC 7.
VII. EVALUATION OF THE DEMONSTRATION
27. Cooperation with Federal Evaluators. As required under 42 CFR 431.420(f), the state shall cooperate fully and timely with CMS and its contractors in any federal evaluation of the demonstration or any component of the demonstration. This includes, but is not limited to: commenting on design and other federal evaluation documents; providing data and analytic files to CMS; entering into a data use agreement that explains how the data and data files will be exchanged; and providing a technical point of contact to support specification of the data and files to be disclosed, as well as relevant data dictionaries and record layouts. The state shall include in its contracts with entities that collect, produce, or maintain data and files for the demonstration, a requirement that they make data available for the federal evaluation as is required under 42 CFR 431.420(f) to support federal evaluation. The state may claim administrative match for these activities. Failure to comply with this STC may result in a deferral being issued as outlined in STC 19. 28. Draft Evaluation Design. The state must submit, for CMS comment and approval, a draft Evaluation Design no later than 180 calendar days after the approval of the demonstration. The Evaluation Design must be drafted in accordance with Attachment A (Developing the Evaluation Design) of these STCs, and any applicable CMS evaluation guidance and technical assistance for the demonstration’s policy components. The Evaluation Design must also be developed in alignment with CMS guidance on applying robust evaluation approaches, such as quasi-experimental methods like difference-in-differences and interrupted time series, as well as establishing valid comparison groups and assuring causal inferences in demonstration evaluations, to the extent feasible. The draft Evaluation Design also must include a timeline for key evaluation activities, including the deliverables outlined in STCs 32 and 33. For any amendment to the demonstration, the state will be required to update the approved Evaluation Design to accommodate the amendment component. The amended Evaluation Design must be submitted to CMS for review no later than 180 calendar days after CMS’s approval of the demonstration amendment. Depending on the scope and timing of the amendment, in consultation with CMS, the state may provide the details on necessary modifications to the approved Evaluation Design via the Monitoring Reports. The amendment components of the Evaluation Design must also be reflected in the state’s Interim and Summative Evaluation Reports, described below. 29. Evaluation Design Approval and Updates. The state must submit a revised Evaluation Design within 60 calendar days after receipt of CMS’s comments, if any. Upon CMS approval of the draft Evaluation Design, the document will be included as Attachment C to
13 Demonstration Approval: November 1, 2023, through October 31, 2028 these STCs. Per 42 CFR 431.424(c), the state will publish the approved Evaluation Design within 30 days of CMS approval. The state must implement the Evaluation Design and submit a description of its evaluation progress in each of the Annual Monitoring Reports. Once CMS approves the Evaluation Design, if the state wishes to make changes, the state must submit a revised Evaluation Design to CMS for approval if the changes are substantial in scope; otherwise, in consultation with CMS, the state may include updates to the Evaluation Design in Monitoring Reports. 30. Evaluation Questions and Hypotheses. Consistent with Attachments A and B (Developing the Evaluation Design and Preparing the Interim and Summative Evaluation Reports) of these STCs, the evaluation deliverables must include a discussion of the evaluation questions and hypotheses that the state intends to test. In alignment with applicable CMS evaluation guidance and technical assistance, the evaluation must outline and address well-crafted hypotheses and research questions for all key demonstration policy components that support understanding the demonstration’s impact and its effectiveness in achieving the goals. The hypothesis testing should include, where possible, assessment of both process and outcome measures. Proposed measures should be selected from nationally-recognized sources and national measures sets, where possible. Measures sets could include CMS’s Core Set of Health Care Quality Measures for Children in Medicaid and CHIP, Consumer Assessment of Health Care Providers and Systems (CAHPS), the Core Set of Health Care Quality Measures for Medicaid-Eligible Adults, the Behavioral Risk Factor Surveillance System (BRFSS) survey, and/or measures endorsed by National Quality Forum (NQF). 31. Evaluation Budget. A budget for the evaluation shall be provided with the draft Evaluation Design. It will include the total estimated cost, as well as a breakdown of estimated staff, administrative, and other costs for all aspects of the evaluation such as any survey and measurement development, quantitative and qualitative data collection and cleaning, analyses, and report generation. A justification of the costs may be required by CMS if the estimates provided do not appear to sufficiently cover the costs of the design or if CMS finds that the design is not sufficiently developed, or if the estimates appear to be excessive. 32. Interim Evaluation Report. If the state is seeking an extension of the demonstration, the state must submit an Interim Evaluation Report for the completed years of the demonstration, and for each subsequent extension of the demonstration, as outlined in 42 CFR 431.412(c)(2)(vi). When submitting an application for extension, the Interim Evaluation Report should be posted to the state's website with the application for public comment. a. The Interim Evaluation Report will discuss evaluation progress and present findings to date as per the approved Evaluation Design. b. For demonstration authority or any component within the demonstration that expires prior to the overall demonstration’s expiration date and depending on the timeline of the expiration/phase-out, the Interim Evaluation Report must include an evaluation of the authority, to be collaboratively determined by CMS and the state. c. The state must submit a revised Interim Evaluation Report within 60 calendar days after receiving CMS’s comments on the draft Interim Evaluation Report.
14 Demonstration Approval: November 1, 2023, through October 31, 2028 Once approved by CMS, the state must post the final Interim Evaluation Report to the state’s Medicaid website within 30 calendar days. d. The Interim Evaluation Report must comply with Attachment B (Preparing the Interim and Summative Evaluation Reports) of these STCs. e. If the state is not requesting an extension for the demonstration, a separate Interim Evaluation report is not required to be submitted. CMS and the state will coordinate to capture appropriate evaluation data and findings through the Annual Monitoring Report due to CMS closest to the date of the end of the fourth demonstration year during the demonstration approval period.
33. Summative Evaluation Report. The state must submit a draft Summative Evaluation Report for the demonstration’s current approval period within 18 months of the end of the approval period represented by these STCs. The draft Summative Evaluation Report must be developed in accordance with Attachment B of these STCs, and in alignment with the approved Evaluation Design. a. The state must submit a revised Summative Evaluation Report 60 calendar days after receiving CMS’s comments on the draft. b. Once approved by CMS, the state must post the final Summative Evaluation Report to the state's Medicaid website within 30 calendar days. 34. Corrective Action Plan Related to Evaluation. If evaluation findings indicate that demonstration features are not likely to assist in promoting the objectives of Medicaid, CMS reserves the right to require the state to submit a corrective action plan to CMS for approval. These discussions may also occur as part of a renewal process when associated with the state's Interim Evaluation Report, or as part of the review of the Summative Evaluation Report. A corrective action plan could include a temporary suspension of implementation of demonstration programs, in circumstances where evaluation findings indicate substantial and sustained directional change inconsistent with demonstration goals, such as substantial and sustained trends indicating increased difficulty accessing services. A corrective action plan may be an interim step to withdrawing waivers or expenditure authorities, as outlined in STC 10. CMS further has the ability to suspend implementation of the demonstration should corrective actions not effectively resolve these concerns in a timely manner. 35. State Presentations for CMS. CMS reserves the right to request that the state present and participate in a discussion with CMS on the Evaluation Design, the Interim Evaluation Report, and/or the Summative Evaluation Report. 36. Public Access. The state shall post the final documents (e.g., Monitoring Reports, Close-Out Report, Evaluation Design, Interim Evaluation Report, and Summative Evaluation Report) on the state’s Medicaid website within 30 calendar days of approval by CMS. 37. Additional Publications and Presentations. For a period of 12 months following CMS approval of the final reports, CMS will be notified prior to presentation of these reports or their findings, including in related publications (including, for example, journal articles), by the state, contractor, or any other third party directly connected to the demonstration over which the state has control. Prior to release of these reports, articles, or other publications,
15 Demonstration Approval: November 1, 2023, through October 31, 2028 CMS will be provided a copy including any associated press materials. CMS will be given 30 business days to review and comment on publications before they are released. CMS may choose to decline to comment or review some or all of these notifications and reviews. This requirement does not apply to the release or presentation of these materials to state or local government officials. VIII. SCHEDULE OF DELIVERABLES
Deliverable Timeline STC Reference Annual Monitoring Reports No later than 90 days following the end of each demonstration year. STC 22
Evaluation Design No later than 180 days after the approval of this demonstration project.
Revised no later than 60 days following receipt of CMS comments on Draft Evaluation Design. STC 28
Interim Evaluation Report With submission of a demonstration extension request, or on October 31, 2027, whichever is sooner.
Revised no later than 60 days following receipt of CMS comments on Draft Interim Evaluation Report. STC 32 Summative Evaluation Report No later than 18 months following the expiration of the demonstration.
Revised no later than 60 days following receipt of CMS comments on Draft Summative Evaluation Report. STC 33
16 Demonstration Approval: November 1, 2023, through October 31, 2028 Attachment A Developing the Evaluation Design
Introduction Both state and federal governments need rigorous quantitative and qualitative evidence to inform policy decisions. To that end, for states and territories that are testing new approaches and flexibilities in their Medicaid programs through section 1115 demonstrations, evaluations are crucial to understand and disseminate information about these policies. The evaluations of new initiatives seek to produce new knowledge and direction for programs and inform Medicaid policy for the future. While a narrative about what happened during a demonstration provides important information, the principal focus of the evaluation of a section 1115 demonstration should be obtaining and analyzing data. Evaluations should include findings about the process (e.g., whether the demonstration is being implemented as intended), outcomes (e.g., whether the demonstration is having the intended effects on the population of focus), and impacts of the demonstration (e.g., whether the outcomes observed in the population of focus differ from outcomes in similar populations not affected by the demonstration).
Submission Timelines There is a specified timeline for the state’s submission of its draft Evaluation Design and subsequent evaluation reports. The graphic below depicts an example of this timeline for a 5- year demonstration. In addition, the state should be aware that section 1115 evaluation documents are public records. The state is required to publish the Evaluation Design to the state’s website within 30 calendar days of CMS approval, as per 42 CFR 431.424(e). CMS will also publish a copy to the Medicaid.gov website.
Demonstrati on approved Jan 1, 2017
Interim Evaluation Report (data from DY1-2.5) Dec 31, 2020
Summative Evaluation Report (data from DY1-5) June 30, 2023
Draft Evaluation Design June 30, 2017
Demonstrati on extension Jan 1, 2022
Demonstration Approval: November 1, 2023, through October 31, 2028 17
Expectations for Evaluation Designs CMS expects Evaluation Designs to be rigorous, incorporate baseline and comparison group assessments, as well as statistical significance testing. Technical assistance resources for constructing comparison groups and identifying causal inferences are available on Medicaid.gov: https://www.medicaid.gov/medicaid/section-1115- demonstrations/1115-demonstration- monitoring-evaluation/1115-demonstration-state-monitoring-evaluation- resources/index.html. If the state needs technical assistance using this outline or developing the Evaluation Design, the state should contact its demonstration team.
The state should attempt to involve partners who understand the cultural context in developing an evaluation approach and interpreting findings. Such partners may include community groups, beneficiaries, health plans, health care providers, social service agencies and providers, and others impacted by the demonstration. For example, the state’s Request for Proposal for an independent evaluator could encourage research teams to partner with impacted groups.
All states and territories with section 1115 demonstrations are required to conduct Interim and Summative Evaluation Reports, and the Evaluation Design is the roadmap for conducting these evaluations. The roadmap begins with the stated goals for the demonstration, followed by the measurable evaluation questions and quantifiable hypotheses, all to support a determination of the extent to which the demonstration has achieved its goals. When conducting analyses and developing the evaluation reports, every effort should be made to follow the approved methodology. However, the state may request, and CMS may agree to, changes in the methodology in appropriate circumstances. The format for the Evaluation Design is as follows: A. General Background Information; B. Evaluation Questions and Hypotheses; C. Methodology; D. Methodological Limitations; E. Attachments.
A. General Background Information – In this section, the state should include basic information about the demonstration, such as: 1. The issue/s that the state is trying to address with its section 1115 demonstration and/or expenditure authorities, the potential magnitude of the issue/s, and why the state selected this course of action to address the issue/s (e.g., a narrative on why the state submitted an 1115 demonstration proposal). 2. The name of the demonstration, approval date of the demonstration, and period of time covered by the evaluation. 3. A description of the population groups impacted by the demonstration. 4. A brief description of the demonstration and history of its implementation, and whether the draft Evaluation Design applies to an amendment, extension, or expansion of, the demonstration. 5. For extensions, amendments, and major operational changes: a description of any changes to the demonstration during the approval period; the primary reason or reasons for the change; and how the Evaluation Design was altered or augmented to address these changes.
Demonstration Approval: November 1, 2023, through October 31, 2028 18
B. Evaluation Questions and Hypotheses – In this section, the state should: 1. Identify the state’s hypotheses about the outcomes of the demonstration, and discuss how the evaluation questions align with the hypotheses and the goals of the demonstration. 2. Address how the hypotheses and research questions promote the objectives of Titles XIX and/or XXI. 3. Describe how the state’s demonstration goals are translated into quantifiable targets for improvement, so that the performance of the demonstration in achieving these targets can be measured. 4. Include a Logic Model or Driver Diagram to visually aid readers in understanding the rationale behind the cause and effect of the variants behind the demonstration features and intended outcomes. A driver diagram, which includes information about the goals and features of the demonstration, is a particularly effective modeling tool when working to improve health and health care through specific interventions. A driver diagram depicts the relationship between the goal, the primary drivers that contribute directly to achieving the goal, and the secondary drivers that are necessary to achieve the primary drivers for the demonstration. For an example and more information on driver diagrams: https://innovation.cms.gov/files/x/hciatwoaimsdrvrs.pdf. 5. Include implementation evaluation questions to inform the state’s crafting and selection of testable hypotheses and research questions for the demonstration’s outcome and impact evaluations and provide context for interpreting the findings. Implementation evaluation research questions can focus on barriers, facilitators, beneficiary and provider experience with the demonstration, the extent to which demonstration components were implemented as planned, and the extent to which implementation of demonstration components varied by setting.
C. Methodology – In this section, the state is to describe in detail the proposed research methodology. The focus is on showing that the evaluation meets the prevailing standards of scientific and academic rigor, that the results are statistically valid and reliable, and that it builds upon other published research, using references where appropriate. The evaluation approach should also consider principles of equitable evaluations, and involve partners—such as community groups, beneficiaries, health plans, health care providers, social service agencies and providers, and others impacted by the demonstration who understand the cultural context—in developing an evaluation approach.
This section also provides evidence that the demonstration evaluation will use the best available data. The state should report on, control for, and make appropriate adjustments for the limitations of the data and their effects on results, and discuss the generalizability of results. This section should provide enough transparency to explain what will be measured and how, in sufficient detail so that another party could replicate the results. Table A below is an example of how the state might want to articulate the analytic methods for each research question and measure.
Specifically, this section establishes: 1. Methodological Design – Provide information on how the evaluation will be designed. For example, whether the evaluation will utilize pre/post data comparisons, pre-test or post-test only assessments.
Demonstration Approval: November 1, 2023, through October 31, 2028 19
If qualitative analysis methods will be used, they must be described in detail. 2. Focus and Comparison Populations – Describe the characteristics of the focus and comparison populations, incorporating the inclusion and exclusion criteria. Include information about the level of analysis (beneficiary, provider, or program level), and if populations will be stratified into subgroups. Additionally, discuss the sampling methodology for the populations, as well as support that a statistically reliable sample size is available. 3. Evaluation Period – Describe the time periods for which data will be included. 4. Evaluation Measures – List all measures that will be calculated to evaluate the demonstration. The state also should include information about how it will define the numerators and denominators. Furthermore, the state should ensure the measures contain assessments of both process and outcomes to evaluate the effects of the demonstration during the period of approval. When selecting metrics, the state shall identify opportunities for improving quality of care and health outcomes, and controlling cost of care. The state also should incorporate benchmarking and comparisons to national and state standards, where appropriate. Include the measure stewards (i.e., the organization(s) responsible for the evaluation data elements/sets by “owning”, defining, validating, securing, and submitting for endorsement, etc.) Proposed health measures could include CMS’s Core Set of Health Care Quality Measures for Children in Medicaid and CHIP, Consumer Assessment of Health Care Providers and Systems (CAHPS), the Core Set of Health Care Quality Measures for Medicaid-Eligible Adults, metrics drawn from the Behavioral Risk Factor Surveillance System (BRFSS) survey, and/or measures endorsed by National Quality Forum. Proposed performance metrics can be selected from nationally recognized metrics, for example from sets developed by the Center for Medicare and Medicaid Innovation or for meaningful use under Health Information Technology. 5. Data Sources – Explain from where the data will be obtained, describe any efforts to validate and clean the data, and discuss the quality and limitations of the data sources. If the state plans to collect primary data (i.e., data collected specifically for the evaluation), include the methods by which the data will be collected, the source of the proposed questions and responses, and the frequency and timing of data collection. Additionally, copies of any proposed surveys must be provided to CMS for approval before implementation. 6. Analytic Methods – This section includes the details of the selected quantitative and/or qualitative analysis measures that will adequately assess the effectiveness of the demonstration. This section should: a. Identify the specific statistical testing which will be undertaken for each measure (e.g., t-tests, chi-square, odds ratio, ANOVA, regression). b. Explain how the state will isolate the effects of the demonstration from other initiatives occurring in the state at the same time (e.g., through the use of comparison groups). c. Include a discussion of how propensity score matching and difference-in- differences designs may be used to adjust for differences in comparison populations over time, if applicable.
Demonstration Approval: November 1, 2023, through October 31, 2028 20
d. Consider the application of sensitivity analyses, as appropriate. 7. Other Additions – The state may provide any other information pertinent to the Evaluation Design for the demonstration.
Table A. Example Design Table for the Evaluation of the Demonstration Research Question Outcome measures used to address the research question Sample or population subgroups to be compared Data Sources Analytic Methods Hypothesis 1 Research -Measure 1 -Sample e.g. All -Medicaid fee- -Interrupted question 1a -Measure 2 attributed Medicaid for-service and time series -Measure 3 beneficiaries encounter claims -Beneficiaries with records diabetes diagnosis Research -Measure 1 -Sample, e.g., PPS -Patient survey Descriptive question 1b -Measure 2 patients who meet statistics -Measure 3 survey selection -Measure 4 requirements (used services within the last 6 months) Hypothesis 2 Research -Measure 1 -Sample, e.g., PPS -Key informants Qualitative question 2a -Measure 2 administrators analysis of interview material
D. Methodological Limitations – This section provides more detailed information about the limitations of the evaluation. This could include limitations about the design, the data sources or collection process, or analytic methods. The state should also identify any efforts to minimize these limitations. Additionally, this section should include any information about features of the demonstration that effectively present methodological constraints that the state would like CMS to take into consideration in its review.
CMS also recognizes that there may be certain instances where a state cannot meet the rigor of an evaluation as expected by CMS. In these instances, the state should document for CMS why it is not able to incorporate key components of a rigorous evaluation, including comparison groups and baseline data analyses. For example, if a demonstration is long- standing, it may be difficult for the state to include baseline data because any pre-test data points may not be relevant or comparable. Other examples of considerations include: 1. When the demonstration is: a. Non-complex, unchanged, or has previously been rigorously evaluated and found to be
Demonstration Approval: November 1, 2023, through October 31, 2028 21
successful; or b. Could now be considered standard Medicaid policy (CMS published regulations or guidance). 2. When the demonstration is also considered successful without issues or concerns that would require more regular reporting, such as: a. Operating smoothly without administrative changes; b. No or minimal appeals and grievances; c. No state issues with CMS-64 reporting or budget neutrality; and d. No Corrective Action Plans for the demonstration.
E. Attachments
1. Evaluation Budget. A budget for implementing the evaluation shall be provided with the draft Evaluation Design. It will include the total estimated costs, as well as a breakdown of estimated staff, administrative, and other costs for all aspects of the evaluation. Examples include, but are not limited to: the development of all survey and measurement instruments; quantitative and qualitative data collection; data cleaning and analyses; and reports generation. A justification of the costs may be required by CMS if the estimates provided do not appear to sufficiently cover the costs of the draft Evaluation Design, if CMS finds that the draft Evaluation Design is not sufficiently developed, or if the estimates appear to be excessive.
2. Timeline and Major Milestones. Describe the timeline for conducting the various evaluation activities, including dates for evaluation-related milestones, including those related to procurement of an outside contractor, if applicable, and deliverables. The final Evaluation Design shall incorporate milestones for the development and submission of the Interim and Summative Evaluation Reports. Pursuant to 42 CFR 431.424(c)(v), this timeline should also include the date by which the Final Summative Evaluation Report is due.
Demonstration Approval: November 1, 2023, through October 31, 2028 22
Attachment B Preparing the Interim and Summative Evaluation Reports
Introduction Both state and federal governments need rigorous quantitative and qualitative evidence to inform policy decisions. To that end, for states and territories that are testing new approaches and flexibilities in their Medicaid programs through section 1115 demonstrations, evaluations are crucial to understand and disseminate information about these policies. The evaluations of new initiatives seek to produce new knowledge and direction for programs and inform Medicaid policy for the future. While a narrative about what happened during a demonstration provides important information, the principal focus of the evaluation of a section 1115 demonstration should be obtaining and analyzing data. Evaluations should include findings about the process (e.g., whether the demonstration is being implemented as intended), outcomes (e.g., whether the demonstration is having the intended effects on the population of focus), and impacts of the demonstration (e.g., whether the outcomes observed in the population of focus differ from outcomes in similar populations not affected by the demonstration).
Submission Timelines There is a specified timeline for the state’s submission of Evaluation Designs and Evaluation Reports. These dates are specified in the demonstration Special Terms and Conditions (STCs). The graphic below depicts an example of a deliverables timeline for a 5-year demonstration. In addition, the state should be aware that section 1115 evaluation documents are public records. In order to assure the dissemination of the evaluation findings, lessons learned, and recommendations, the state is required to publish the Interim and Summative Evaluation Reports to the state’s website within thirty (30) calendar days of CMS approval, as per 42 CFR 431.424(d). CMS will also publish a copy to the Medicaid.gov website.
Demonstrati on approved Jan 1, 2017
Interim Evaluation Report (data from DY1-2.5) Dec 31, 2020
Summative Evaluation Report (data from DY1-5) June 30, 2023
Draft Evaluation Design June 30, 2017
Demonstrati on extension Jan 1, 2022 Expectations for Evaluation Reports All states and territories with Medicaid section 1115 demonstrations are required to conduct evaluations that are valid (the extent to which the evaluation measures what it is intended to measure), and reliable (the extent to which the evaluation could produce the same results when
Demonstration Approval: November 1, 2023, through October 31, 2028 23
used repeatedly). The already-approved Evaluation Design is a map that begins with the demonstration goals, then transitions to the evaluation questions, and to the specific hypotheses, which will be used to investigate whether the demonstration has achieved its goals. When conducting analyses and developing the evaluation reports, every effort should be made to follow the methodology outlined in the approved Evaluation Design. However, the state may request, and CMS may agree to, changes in the methodology in appropriate circumstances.
When submitting an application for extension, the Interim Evaluation Report should be posted on the state’s website with the application for public comment. Additionally, the Interim Evaluation Report must be included in its entirety with the application submitted to CMS.
CMS expects Interim and Summative Evaluation Reports to be rigorous, incorporate baseline and comparison group assessments, as well as statistical significance testing. Technical assistance resources for constructing comparison groups and identifying causal inferences are available on Medicaid.gov: https://www.medicaid.gov/medicaid/section-1115- demonstrations/1115-demonstration-monitoring-evaluation/1115-demonstration-state- monitoring- evaluation-resources/index.html. If the state needs technical assistance using this outline or developing the evaluation reports, the state should contact its demonstration team.
Intent of this Attachment Title XIX of the Social Security Act (the Act) requires an evaluation of every section 1115 demonstration. In order to fulfill this requirement, the state’s evaluation report submissions must provide comprehensive written presentations of all key components of the demonstration, and include all required elements specified in the approved Evaluation Design. This Attachment is intended to assist states and territories with organizing the required information in a standardized format and understanding the criteria that CMS will use in reviewing the submitted Interim and Summative Evaluation Reports.
Required Core Components of Interim and Summative Evaluation Reports The Interim and Summative Evaluation Reports present research and findings about the section 1115 demonstration. It is important that the reports incorporate a discussion about the structure of the Evaluation Design to explain the goals and objectives of the demonstration, the hypotheses related to the demonstration, and the methodology for the evaluation. The evaluation reports should present the relevant data and an interpretation of the findings; assess the outcomes (what worked and what did not work); explain the limitations of the design, data, and analyses; offer recommendations regarding what (in hindsight) the state would further advance, or do differently, and why; and discuss the implications on future Medicaid policy. The format for the Interim and Summative Evaluation reports is as follows: A. Executive Summary; B. General Background Information; C. Evaluation Questions and Hypotheses; D. Methodology;
Demonstration Approval: November 1, 2023, through October 31, 2028 24
E. Methodological Limitations; F. Results; G. Conclusions; H. Interpretations, and Policy Implications and Interactions with Other State Initiatives; I. Lessons Learned and Recommendations; and J. Attachment(s).
A. Executive Summary – A summary of the demonstration, the principal results, interpretations, and recommendations of the evaluation.
B. General Background Information about the Demonstration – In this section, the state should include basic information about the demonstration, such as: 1. The issue/s that the state is trying to address with its section 1115 demonstration and/or expenditure authorities, how the state became aware of the issue, the potential magnitude of the issue, and why the state selected this course of action to address the issues. 2. The name of the demonstration, approval date of the demonstration, and period of time covered by the evaluation. 3. A description of the population groups impacted by the demonstration. 4. A brief description of the demonstration and history of the implementation, and if the evaluation is for an amendment, extension, or expansion of, the demonstration. 5. For extensions, amendments, and major operational changes: A description of any changes to the demonstration during the approval period; whether the motivation for change was due to political, economic, and fiscal factors at the state and/or federal level; whether the programmatic changes were implemented to improve beneficiary health, provider/health plan performance, or administrative efficiency; and how the Evaluation Design was altered or augmented to address these changes. Additionally, the state should explain how this Evaluation Report builds upon and expands earlier demonstration evaluation findings (if applicable).
C. Evaluation Questions and Hypotheses – In this section, the state should: 1. Identify the state’s hypotheses about the outcomes of the demonstration, and discuss how the goals of the demonstration align with the evaluation questions and hypotheses. 2. Address how the research questions / hypotheses of this demonstration promote the objectives of Titles XIX and XXI. 3. Describe how the state’s demonstration goals were translated into quantifiable targets for improvement, so that the performance of the demonstration in achieving these targets could be measured. 4. The inclusion of a Logic Model or Driver Diagram in the Evaluation Report is highly encouraged, as the visual can aid readers in understanding the rationale behind the demonstration features and intended outcomes.
Demonstration Approval: November 1, 2023, through October 31, 2028 25
D. Methodology – In this section, the state is to provide an overview of the research that was conducted to evaluate the section 1115 demonstration, consistent with the approved Evaluation Design. The Evaluation Design should also be included as an attachment to the report. The focus is on showing that the evaluation builds upon other published research, (using references), meets the prevailing standards of scientific and academic rigor, and the results are statistically valid and reliable.
An Interim Evaluation Report should provide any available data to date, including both quantitative and qualitative assessments. The Evaluation Design should assure there is appropriate data development and collection in a timely manner to support developing an Interim Evaluation Report. This section provides the evidence that the demonstration evaluation used the best available data and describes why potential alternative data sources were not used. The state also should report on, control for, and make appropriate adjustments for the limitations of the data and their effects on results, and discuss the generalizability of results. This section should provide enough transparency to explain what was measured and how, in sufficient detail so that another party could replicate the results. Specifically, this section establishes that the approved Evaluation Design was followed by describing: 1. Methodological Design – Whether the evaluation included an assessment of pre/post or post-only data, with or without comparison groups, etc. 2. Focus and Comparison Populations – Describe the focus and comparison populations, describing inclusion and exclusion criteria. 3. Evaluation Period – Describe the time periods for which data will be collected. 4. Evaluation Measures – List the measures used to evaluate the demonstration and their respective measure stewards. 5. Data Sources – Explain from where the data were obtained, and efforts to validate and clean the data. 6. Analytic Methods – Identify specific statistical testing which was undertaken for each measure (t-tests, chi-square, odds ratio, ANOVA, regression, etc.). 7. Other Additions – The state may provide any other information pertinent to the evaluation of the demonstration.
E. Methodological Limitations – This section provides sufficient information for discerning the strengths and weaknesses of the study design, data sources/collection, and analyses.
F. Results – In this section, the state presents and uses the quantitative and qualitative data to demonstrate whether and to what degree the evaluation questions and hypotheses of the demonstration were addressed. The findings should visually depict the demonstration results, using tables, charts, and graphs, where appropriate. This section should include findings from the statistical tests conducted.
G. Conclusions – In this section, the state will present the conclusions about the evaluation results. Based on the findings, discuss the outcomes and impacts of the demonstration and
Demonstration Approval: November 1, 2023, through October 31, 2028 26
identify the opportunities for improvements. Specifically, the state should answer the following questions: 1. In general, did the results show that the demonstration was/was not effective in achieving the goals and objectives established at the beginning of the demonstration? a. If the state did not fully achieve its intended goals, why not? b. What could be done in the future that would better enable such an effort to more fully achieve those purposes, aims, objectives, and goals?
H. Interpretations, Policy Implications and Interactions with Other State Initiatives – In this section, the state will discuss the section 1115 demonstration within an overall Medicaid context and long-range planning. This should include interrelations of the demonstration with other aspects of the state’s Medicaid program, interactions with other Medicaid demonstrations, and other federal awards affecting service delivery, health outcomes and the cost of care under Medicaid. This section provides the state with an opportunity to provide interpretations of the data using evaluative reasoning to make judgments about the demonstration. This section should also include a discussion of the implications of the findings at both the state and national levels. Interpreting the implications of evaluation findings should include involving partners, such as community groups, beneficiaries, health plans, health care providers, social service agencies and providers, and others impacted by the demonstration who understand the cultural context in which the demonstration was implemented.
I. Lessons Learned and Recommendations – This section of the evaluation report involves the transfer of knowledge. Specifically, it should include potential “opportunities” for future or revised demonstrations to inform Medicaid policymakers, advocates, and stakeholders. Recommendations for improvement can be just as significant as identifying current successful strategies. Based on the evaluation results, the state should address the following questions: 1. What lessons were learned as a result of the demonstration? 2. What would you recommend to other states which may be interested in implementing a similar approach?
Demonstration Approval: November 1, 2023, through October 31, 2028 27
Attachment C Approved Evaluation Design (Reserved pending CMS approval)