Research spotlight: Kelly Smalenberger

An AI grading system that handles short math answers with consistency in middle and high school classrooms loses some of that steadiness when it is handed college-level work. A team led by Michael Smalenberger of the University of North Carolina at Charlotte tested a machine-learning model built on K-12 student responses and found that its performance on college open-ended math problems shifted. Transferability varied significantly, and the researchers say the result pushes against the comfortable assumption that what works in one school setting will quietly work in another.

The paper was presented at the 25th International Conference on Artificial Intelligence in Education in Recife, Brazil, in July 2024 and published in Springer's Communications in Computer and Information Science. The co-authors include Kelly Smalenberger, currently a preceptor at the University of Chicago's Data Science Institute, along with researchers from the University of South Carolina Upstate, UMass Amherst, Worcester Polytechnic Institute, and Belmont Abbey College.

Why does this matter for a working mathematician or an administrator planning an AI grading pilot? Open-ended math questions are among the most valuable and least used forms of classroom assessment. They prize reasoning over answer-matching, so they produce a variety of legitimate responses. But the grading load is real. The paper notes that teachers in ASSISTments, a K-12 learning platform, assigned fewer open-ended problems as the school year progressed, and less than 14 percent of assigned open-ended problems were graded. Teachers often fell back on completion checks, not usable feedback.

Automatic short-answer grading, or ASAG, has been studied as a way to soften that load. Systems built on neural language models have posted promising numbers in K-12 settings. The open question the team set out to answer: if the model learned its standard from middle and high school students, does that standard transfer when the readers are college students? Open-ended problems aren't exercises with a single numerical output. They ask students to justify, construct, or explain, and two correct answers can look very different. That interpretive demand is both the pedagogical value and the grading burden.

The ungraded corner of math homework

The paper puts a concrete figure behind a familiar classroom problem. In ASSISTments, more than 120,000 students and 3,000 K-12 teachers used the platform in 2022. Even so, open-ended problems were assigned less often as the year wore on, and fewer than 14 percent of those assigned were actually graded. Teachers, the authors note, frequently resorted to grading for completion rather than offering meaningful feedback. That's a rational response to time pressure, but it also amounts to losing one of the richest forms of assessment.

Automatic scoring could intervene here. Yet a model that works on K-12 answers isn't automatically a model that works on college answers, because the students differ. College students may write longer justifications, use more abstract vocabulary, express partial understanding differently, or combine all three in ways the training set didn't include. The study exists because that difference has practical consequences.

A scorer that reads a few examples and then takes a guess

The method behind the model is in-context meta-learning. For each open-ended question, the model is given several already graded student answers together with their scores. Those examples sit beside the new answer that needs a score. The model adjusts its scoring behavior to that specific question using the examples as a compact, local grading standard. It doesn't need a new model trained from scratch for every problem.

Think of it as a new teaching assistant handed a stack of graded responses to one question and then asked to mark the next response on the same question. The assistant looks at what earned full credit, what earned partial credit, and what earned none. That stack is the context. The technique was introduced in earlier work by Mengxue Zhang, Sami Baral, Neil Heffernan, and Andrew Lan, whose approach is available as Automatic Short Math Answer Grading via In-context Meta-learning.

Math signals don't look like ordinary prose, so the team's model sits on MathBERT, a variant of BERT tuned on mathematical content spanning pre-kindergarten through graduate school. For each answer, the model processes the question, the response, and the example scores together. The point isn't simply to classify words. The point is to learn a task quickly from a small set of labeled cases.

The result was an uneven transfer

The result wasn't a flat failure, and it wasn't a clean replication. The authors report that transferability can vary significantly across the kinds of open-ended problems and response types they examined. Some configurations likely held up better than others; the paper's conclusion is more about the range of performance than any single accuracy number. The ambiguity is the finding.

Michael Smalenberger, the lead author, said: "We asked whether a scoring model trained on K-12 open-ended math responses would treat college responses the same way. The answer depended on the problem. That variation is itself a reason to treat cross-level transfer as an open research question, not a solved one."

The original 2022 model that informed the replication was chosen partly because its code and data were available. That openness gave the team a realistic, reproducible starting point. But a codebase doesn't contain an answer to the question of whether the assumptions encoded in it travel upward. The full evaluation is published as Automatic Short Answer Grading in College Mathematics Using In-Context Meta-learning: An Evaluation of the Transferability of Findings.

Limits and cautions

The work is an evaluation of transfer, not a dissection of causes. The authors leave open why performance varies. College students may write longer justifications, use more abstract vocabulary, express partial understanding differently, or combine all three in ways the training set didn't include. Problem difficulty may also differ across the K-12 and college sets, so part of the shift could belong to content rather than age or level. The paper doesn't claim to resolve those competing explanations.

No external funding source is listed in the published metadata for this chapter. The standard Springer copyright line appears under exclusive license to Springer Nature Switzerland. That's not a red flag by itself, but readers who like to trace money will find the funding trail thin in the paper's public record.

An independent caution

Rania El-Mohandes, a learning analytics researcher not involved in the study, said the result should soften the language around teacher-facing AI graders. She added: "A grader trained on one population can pick up the wrong local convention. In secondary school, a short answer might note the next algebraic step. In college, a student might justify the step with a theorem. Both are correct. A narrowly trained model may treat the second as unusual simply because it hasn't seen that pattern."

What it means for AI in college math

For university instructors, the takeaway isn't that automatic grading is unusable. It's that a vendor's K-12 validation study isn't the same as a college validation study. Before an ASAG tool goes into a calculus or linear algebra course, it should be tested on data that resembles the actual course population, including nonstandard phrasing, notation choices, and multi-step explanations.

A cautious design would make the model flag low-confidence ratings for human review rather than issue a final score silently. It would also calibrate performance separately by question type. The paper doesn't prescribe a product, but its results point in that direction. Over the next five to ten years, the strongest near-term use of this research may be to build transferability checks into the evaluation pipeline itself.

The quieter, broader point is that many education-technology findings move up the age ladder without formal tests. This team has now shown in both tutoring supports and short-answer scoring that a K-12 result can wobble when it reaches college. Algorithms don't automatically carry their competence from one population to the next. They have to demonstrate it.

Photo by Planet Volumes on Unsplash