Applying Machine Learning in Education
Early Warning System (EWS) Goals
- Identify students showing signs of risk to provide support and interventions early on to prevent escalation of issues or the occurrence of major events.
- Evaluate programs and interventions that work to increase effectiveness.
- Continue training the model as new variables emerge.
What is Risk?
“At-risk” is a concept that reflects a chance or a probability–it does not imply certainty. Risk factors raise the chance of poor outcomes, while protective factors raise the chance of good outcomes.
Areas of student risks are strongly interrelated and multidimensional.
EWS Literature Review – Highlights
Common Elements of Risk
- One size does not fit all: Risk differs tremendously across grade levels –the profile of what is risky for a 3rd grader is different, and more subtle, than that of a risky 10th grader.
- Multi-dimensionality of risk (multiple domains/multiple indicators): Risk manifests in subtle ways, typically within a single domain. If unaddressed risk factors spread in breadth, intensity, and frequency over time.
Observed Patterns of Drop Outs
- Attendance is highly important for all grade levels
- Dropping out is a process of disengagement and disassociation with school with roots in earlier grades
- Academic indicators become increasing more important as of the 4th and 5th grade onward as more information becomes available
- Variability in the relative importance of behavior due to a data quality issue. Implication: better behavioral data = better identification early on
Observed Patterns of Drop Outs
- Local settings are important: Dramatic increases in % accuracy when predictive models trained locally or according to “similar” types of districts
- As of 7th grade and continuing upward into the higher grade levels, suspensions/expulsions take on greater importance in predicting likelihood of dropping out
- 9th grade –GPA, pass rate are very important predictors
- If a student makes it to 12th grade, their likelihood of graduating increases substantially
Current State Models
- Most common model is the ABC model using a rule-based system
- Course Performance
- Wisconsin developed an Early Warning System in 2012 that uses a predictive model
Collaboration Process DoIT + ISBE
- Student level
- School level
- District level
- State level
- Discipline data
- Remove null values
- Convert categorical data to numerical
- De-identify entities
Created Data Matrix
3. Exploratory Data Analysis
- Feature engineering – Modified variables
- Ex. Feature for students with number of transfers > 0
- Joined datasets
- Conducted correlational analyses
4. Modeling and Evaluation
- Logistic Regression
- Multinomial Naïve Bayes
- Random Forest
Most Important Variables in EWS Model
Correlation Plot of Student Graduation
Correlation with Student Performance
Correlation with School level Data
5. Scoring students with the model
- Determined thresholds for 4 risk levels
- Low =.16
- Low Medium = 0.5
- High Medium = 0.8
- Ability to digest a significant amount of information at the same time
- Ability to draw insights from the study that educators can act upon
- Allowing the data and the inherent patterns to determine the importance of variables