Resolve Initiative Issues
Adding eligible comparison students
After selecting ‘Let’s fix this,’ you’ll be prompted to take one of the following actions:
- Yes, add for me: Initiative Analysis will automatically add students who met the eligibility criteria during other same-season terms. For example, if Fall 2014 triggered this warning, eligible comparison students will be added from other Fall terms when the initiative was offered. From our data science team’s prior ad-hoc impact analysis experience, we expect same-season terms to have fewer confounding factors that could influence the analysis (such as we expect a Fall 2014 term to look more similar to a Fall 2013 term than a Spring 2013 term across institutional operations, student populations, and student outcomes).
- No, I’ll choose: You will select other terms to include in the eligible comparison group. If you specified the eligible comparison group in your student list file, only the terms included in that file will appear as options. If you did not specify the eligible comparison group, all terms within the last four years that are available in your institution’s Civitas data set will appear as options. Note: The same caveat about additional confounding factors when matching and comparing participant and comparison students across different terms applies here.
- Ignore this term: This term’s data will be excluded from impact analysis.
If you try to ignore all available terms, you will not be able to continue with impact analysis. You will be prompted to edit your selections or upload a new list before continuing.
You can continue impact analysis without fixing the indicated terms. However, this is not recommended as the number of students analyzed for the indicated term(s) will be greatly reduced, which could cause inaccurate analysis.
No matches found
There might be rare scenarios where no similar participants and comparison students can be matched, that is, no matches are found during prediction-based propensity score matching (PPSM). This scenario is most likely to occur for an individual term included in analysis with little data. If no participants and eligible comparison students could be matched for a term, the lift in persistence cannot be calculated and the ‘Initiative Analysis by Term’ chart will not include a bar for that term.
Some common reasons why no matches could be found include:
- Very small number of submitted participants and/or eligible comparison students were available for matching
- Very small number of submitted participants and/or eligible comparison students from the same term or student group were available for matching
- There is a significant difference between the submitted participants and eligible comparison students, particularly in terms of persistence likelihood and/or propensity to participate in the initiative, for example:
- Participants were all new students, but eligible comparison students were all continuing students
- Participants were all online students, but eligible comparison students were all on-ground students
- Participants were all undergraduates, but eligible comparison students were all graduate students
To avoid this issue, verify that the submitted initiative data accurately represents the impact analysis question you are trying to answer. In particular, confirm that the participants and eligible comparison group were defined correctly in your student list file.
Interpreting potential suspect results
You’ll see a warning about potential suspect results when any of the following are true:
- Fewer than 200 analyzed participants were used for impact analysis
- A small number of analyzed participants could produce inaccurate results, or make it difficult to reach concrete evidence of impact.
- Fewer than 70% of submitted participants were matched
- The match rate is verified to confirm that impact analysis is not potentially skewed due to a significant proportion of participants being ignored. If the match rate is low, this could indicate that the submitted participant and eligible comparison groups are too dissimilar for accurate comparison (such as one group has very high persistence prediction scores and the other group has much lower persistence prediction scores). Check that the submitted initiative data accurately reflects the hypothesis you are testing, or review Designing an initiative for analysis using Initiative Analysis to ensure the best possible setup for analysis.
- Similarity of “After Matching” persistence prediction scores is less than 85% AND/OR similarity of “After Matching” propensity scores is less than 85%
- The similarity of the persistence and propensity scores between the matched participant and comparison groups is verified to confirm that the two groups are similar enough for accurate comparison of outcomes impact. Less than 85% similarity could indicate that the submitted participant and eligible comparison groups are too dissimilar for accurate comparison.
- If either of the “After Matching” graphs do not look like a smooth distribution, contact your Civitas Administrator to submit a support ticket or email impact-help@civitaslearning.com.
- Overall magnitude of impact (that is, lift in persistence) is greater than 5% AND there were over 500 analyzed participants
- The magnitude of impact and number of analyzed participants are verified because it is unlikely that a single initiative would show a lift in persistence greater than 5% with more than 500 analyzed participants. In these cases, note whether there were other concurrent initiatives impacting the same students, or verify that the submitted data file accurately reflects the students to be analyzed for only this specific initiative.
Calibration error warning
When reviewing impact analysis results, you should make sure that the comparison group calibration, or the difference between the predicted vs. actual persistence rates of the matched comparison group, is minimal — our recommended rule of thumb is < 3% for N > 500.
The comparison group calibration error can be calculated from the data provided by downloading the Raw Data File from the Initiative Initiative Analysis page:
Calibration Error = ABS( "Comparison Group Outcome (Predicted)" – "Comparison Group Outcome (Actual)"))In many cases, calibration issues can be ignored because impact results are based on the difference of differences between actual and predicted persistence rates for the matched groups, and calibration issues for the matched groups .should cancel out as long as the participant and comparison groups’ eligibility criteria are similar and appropriately defined in your uploaded initiative data file.
For example, if there is a highly effective program that is also available at the same time as the initiative you are analyzing in Initiative Analysis, and this program affects a large portion of the student population, it can cause a calibration issue when PPSM models do not incorporate a variable representing participation in this program. However, the difference-of-difference impact calculation takes care of this type of calibration error, and the calibration error warning message can be ignored.
In other cases, legitimate calibration issues can arise for the following reasons:
- Stringent eligibility criteria for participant and comparison groups are not accounted for in the PPSM models.
- Example: An initiative that targets a smaller, specific subgroup of students, such as on-ground students who skip multiple classes, and no PPSM model variable captures attendance. This problem can be less severe for students in mixed modality because LMS engagement features can serve as a surrogate variable for attendance,
- Eligibility criteria are directly related to the persistence outcome being measured.
- Example: The eligible comparison group was defined as students who could not register, where, for instance, registration hold was lifted only for those who met advisors with treatment defined as students seeing advisors. In that case, there is leakage of future information and the impact analysis results would be inaccurate.
- In pre-post matching, widely varying persistence trends or data non-stationarity over time can cause calibration errors.
- Example: Data can be non-stationary because of changes in registration policies, changes in academic standing criteria, changes in student population mix, etc.
- Small N can cause calibration issues since variance of group predictions increases as N gets smaller.
In general, the more different participant and comparison groups are, with no representation of these differences in the PPSM model variables, the more caution you should exercise with calibration errors.