How accurate is AI marking? What our first TeachEdge study found

We compared TeachEdge's marking of 67 real examination answers with published examiner marks. The results were encouraging — particularly when considered alongside Ofqual's research into human marking consistency. Read the full study, including the methodology and all 67 results.

How closely can AI mark an examination answer compared with an experienced examiner?

It's an important question for teachers considering AI-assisted marking. While the potential to reduce workload and give students more regular feedback is clear, teachers need to be confident that the marks and feedback are reliable.

At TeachEdge, we wanted to put that question to the test.

We took 67 real student answers from Pearson Edexcel A-Level Economics examiner reports, asked TeachEdge to mark them, and compared its results with the published examiner awards.

The answers covered 39 different questions, ranging from short four-mark responses to 25-mark essays requiring detailed economic analysis and evaluation. Most were handwritten, and some included diagrams.

We used TeachEdge's existing marking instructions, without adjusting them for the questions being tested. Each answer was marked once, and we retained every result.

The headline results

So, how did it perform?

The headline results were encouraging.

TeachEdge awarded exactly the same mark as the published examiner on 31 of the 67 answers.

On 55 answers, it came within two marks.

Agreement was particularly close on shorter questions. Every answer worth four, five or eight marks was within two marks of the examiner's award.

Extended responses proved more challenging, as we expected. But even on 25-mark essays, TeachEdge came within two marks on eight of the thirteen answers, with an average difference of 2.46 marks.

Of course, being two marks away on a four-mark question is very different from being two marks away on a 25-mark essay. That's why we examined the results by question length rather than relying solely on the headline figures.

And this led us to an interesting comparison.

How does this compare with human marking?

It's tempting to assume that experienced human examiners will always award the same marks when assessing an answer against an official mark scheme.

In practice, marking involves judgement. Two experienced examiners can reach different conclusions, particularly when assessing longer answers involving analysis and evaluation.

Ofqual, the qualifications regulator in England, has investigated this issue.

In its research into marking consistency, Ofqual asked experienced examiners to mark the same examination scripts and compared their marks with those awarded by a principal examiner.

The research included Pearson A-Level Economics.

We compared the variability in those human marking differences with the differences observed in our TeachEdge study.

The results were particularly interesting for longer questions.

For answers worth more than 20 marks, the standard deviation of human examiner differences in Ofqual's research was approximately 4.0 marks. In our study, TeachEdge's equivalent figure for 25-mark essays was 3.11 marks.

In other words, TeachEdge's marks on these essays varied less around the published examiner awards than the human examiner marks varied around the principal examiner's awards in Ofqual's separate research.

That is an encouraging result.

The role of examiner preparation

But there's another interesting detail about how Ofqual conducted its study.

The examiners participating in Ofqual's research were experienced markers who had previously marked the relevant examination papers.

Before undertaking the research marking exercise, they also went through a standardisation process.

They reviewed the original standardisation materials, marked practice responses and had their marking checked against the principal examiner's standards. Where necessary, the principal examiner discussed differences with them before they were allowed to begin the main exercise. Ofqual’s methodology

In other words, these weren't examiners encountering the assessment requirements for the first time. They had received question-specific preparation and an opportunity to align their marking with the principal examiner.

TeachEdge had no equivalent preparation for the 39 questions in our study.

We used its existing Economics marking instructions, which had been developed for the relevant question types but had not been calibrated using any of the questions selected for this test, according to the development records available.

TeachEdge received the questions, official mark schemes, supporting materials and student answers, but not the published marks or examiner comments for the answers being tested.

It then marked each response once, without further adjustment.

That makes the comparison particularly interesting. TeachEdge produced encouraging results using its existing marking approach, while the human examiners in Ofqual's research had benefited from specific standardisation.

There are important qualifications, however.

The two studies used different answers, with different sample sizes and marking arrangements. We therefore cannot conclude that TeachEdge is more accurate or consistent than human examiners.

TeachEdge also used marking instructions containing calibration examples from other sources. And because the examination materials are publicly available, we cannot completely rule out the possibility that the underlying AI model encountered some of them during its original training.

Nevertheless, we think the comparison provides a useful indication of how far AI-assisted marking has progressed.

It certainly suggests that AI marking deserves to be evaluated seriously alongside established human assessment practices.

What the disagreements revealed

One of the most useful aspects of the study was discovering where TeachEdge disagreed with the examiners.

We investigated 18 answers in greater detail, concentrating on larger marking differences and disagreements over assessment levels. This closer review was AI-assisted and could see both awards; it was not an independent re-marking exercise.

Two patterns emerged.

One particularly strong Economics essay received 24/25 from the examiner but only 17/25 from TeachEdge.

Another received 23/25 from the examiner and 18/25 from TeachEdge.

Looking more closely, we found that TeachEdge appeared to place considerable weight on the quality of the student's final judgement.

In both cases, the published examiner had been prepared to award high evaluation marks despite weaknesses in the overall conclusion.

This suggests that TeachEdge's marking instructions may sometimes be stricter than the approach taken by experienced examiners.

We also identified the opposite problem.

Some weaker answers received more evaluation credit than the examiner awarded.

For example, TeachEdge sometimes appeared to reward students for identifying a relevant limitation without sufficiently explaining or developing their argument.

This is an important distinction in Economics assessment. Recognising that an argument has limitations is not necessarily the same as providing convincing evaluation.

Across our sample, TeachEdge tended to mark weaker answers more generously and some stronger answers more strictly.

These are precisely the kinds of findings we wanted the study to uncover.

The purpose wasn't simply to produce a favourable accuracy statistic. It was to understand where TeachEdge performs well and where its marking needs further investigation.

Identifying these patterns gives us a clearer basis for testing potential improvements.

What this means for teachers and students

We think the results are encouraging for schools considering AI-assisted assessment.

TeachEdge demonstrated substantial agreement with published examiner marks across a range of question types, including extended essays.

But the educational value of TeachEdge goes beyond awarding a numerical mark.

Teachers need students to receive feedback that explains what they've done well, where their understanding or analysis needs developing, and what they should do next.

That's particularly important when students are completing regular practice questions.

Traditionally, the difficulty is finding enough time to mark all those responses and provide meaningful individual feedback.

AI-assisted marking offers a different possibility.

A teacher can set regular written work, use TeachEdge to generate initial marks and feedback, and then review the results where appropriate.

Students can receive individual guidance much more frequently, without every piece of work adding substantially to the teacher's marking workload.

For longer or more consequential assessments, teacher judgement remains especially important.

And it's worth stressing that our pilot measured agreement with published marks, rather than independently evaluating the educational quality of TeachEdge's feedback. That's something we would like to investigate separately.

Making the evidence available

There's no shortage of claims about what AI can do in education.

We think those claims should increasingly be supported by evidence.

That's why we've published this pilot study, including the individual marking results, methodology, limitations and examples where TeachEdge disagreed with the published examiner.

We haven't removed the disappointing results or rerun answers until we obtained a closer match.

Nor are we claiming that this relatively small study proves TeachEdge is better than human examiners.

Instead, we want to establish a transparent approach to evaluating AI marking that we can build on.

This first study focuses on Pearson Edexcel A-Level Economics. We hope to extend the approach to other subjects and qualifications, investigate the consistency of repeated marking, and explore how well AI-generated feedback reflects teachers' professional judgements.

Most importantly, we'll use what we've learned to investigate areas where TeachEdge's marking could be improved.

An encouraging starting point

The results of our first pilot give us considerable encouragement.

TeachEdge matched the published examiner's mark exactly on almost half the answers and came within two marks on more than 80%.

Its performance on extended essays also compared favourably, on one measure of marking variability, with findings from Ofqual's separate research into human marking consistency.

And it achieved these results without being specifically standardised on the questions used in the study.

That doesn't mean AI marking is perfect. Our investigation revealed some important areas for improvement, particularly in assessing evaluation.

But it does suggest that AI-assisted marking can be a credible part of the assessment process.

For teachers, the opportunity is significant: more regular written practice, more individual feedback, and more time to concentrate on helping students improve.

That's what we're working towards with TeachEdge.

Read the full pilot study: Evaluating AI Marking Accuracy in A-Level Economics