Skip to main content
facebook  linkedin  twitter

Innovative and Effective Learning Solutions

Applying Machine Learning in Education

Early Warning System (EWS) Goals

  • Identify students showing signs of risk to provide support and interventions early on to prevent escalation of issues or the occurrence of major events.
  • Evaluate programs and interventions that work to increase effectiveness.
  • Continue training the model as new variables emerge.

What is Risk?

“At-risk” is a concept that reflects a chance or a probability–it does not imply certainty. Risk factors raise the chance of poor outcomes, while protective factors raise the chance of good outcomes.

Areas of student risks are strongly interrelated and multidimensional.

EWS Literature Review – Highlights

diagram

Common Elements of Risk

  • One size does not fit all: Risk differs tremendously across grade levels –the profile of what is risky for a 3rd grader is different, and more subtle, than that of a risky 10th grader.
  • Multi-dimensionality of risk (multiple domains/multiple indicators): Risk manifests in subtle ways, typically within a single domain. If unaddressed risk factors spread in breadth, intensity, and frequency over time.

Observed Patterns of Drop Outs

  • Attendance is highly important for all grade levels
  • Dropping out is a process of disengagement and disassociation with school with roots in earlier grades
  • Academic indicators become increasing more important as of the 4th and 5th grade onward as more information becomes available
  • Variability in the relative importance of behavior due to a data quality issue. Implication: better behavioral data = better identification early on

Observed Patterns of Drop Outs

  • Local settings are important: Dramatic increases in % accuracy when predictive models trained locally or according to “similar” types of districts
  • As of 7th grade and continuing upward into the higher grade levels, suspensions/expulsions take on greater importance in predicting likelihood of dropping out
  • 9th grade –GPA, pass rate are very important predictors
  • If a student makes it to 12th grade, their likelihood of graduating increases substantially

Current State Models

  • Most common model is the ABC model using a rule-based system
    • Attendance
    • Behavior
    • Course Performance
  • Wisconsin developed an Early Warning System in 2012 that uses a predictive model

diagram2

Collaboration Process DoIT + ISBE

diagram3

Prediction Methodology

diagram4

EWS Methodology

1.Import Data

  • Student level
  • School level
  • District level
  • State level
  • Discipline data

Data Cleaning

  • Remove null values
  • Convert categorical data to numerical
  • De-identify entities

    Created Data Matrixdiagram5

3. Exploratory Data Analysis

  • Feature engineering – Modified variables
  • Ex. Feature for students with number of transfers > 0
  • Joined datasets
  • Conducted correlational analyses

4. Modeling and Evaluation

  • XGBoost
  • Logistic Regression
  • Multinomial Naïve Bayes
  • Random Forest

Most Important Variables in EWS Model

diagram6

Correlation Plot of Student Graduation

diagram7

Correlation with Student Performance

diagram8

Correlation with School level Data

diagram9

Unbalanced Dataset

diagram10

Under Sampling

diagram11

Model Validation

diagram12 diagram13

5. Scoring students with the model

  • Determined thresholds for 4 risk levels
    • Low =.16
    • Low Medium = 0.5
    • High Medium = 0.8
      diagram14
  • Ability to digest a significant amount of information at the same time
  • Ability to draw insights from the study that educators can act upon
  • Allowing the data and the inherent patterns to determine the importance of variables