HMC / GSA Media Brief – unsafe examination grades

Today (15.04.16) HMC and GSA are publishing a report that reveals that the grades awarded to independent school candidates in the English Language IGCSE qualification set by Cambridge International Examinations (CIE) in 2015 cannot be trusted. This is part of a long-standing campaign to improve exam setting and marking for all pupils and to ensure greater transparency, confidence and fairness for future candidates.

GSA_RGB_logo_with_strapline_FINAL_LARGE

New research reveals ‘overwhelming evidence’ of a ‘major problem’ in the Cambridge 2015 IGCSE in English Language. Grades awarded show ‘huge disparities’ and are ‘not safe’, say school leaders.

An unprecedented number of independent schools have come forward to report problems, representing 30% of the national entries. Strong doubts remain about the accuracy of grades in a qualification also taken by 200,000 state school candidates.

Only results awarded in 2015 are likely to have been defective on a large scale. Candidates sitting the qualification in 2016 should assume that the board will have learned from its mistakes in 2015 and reset standards accordingly.

Summary

  • Exam results received by thousands of pupils last year are revealed as ‘unsafe’ say leading independent school heads.
  • New research uncovers overwhelming evidence that a ‘major problem’ with a Cambridge IGCSE sat in May 2015 resulted in the large-scale award of false grades.
  • The problem arose mainly because of the setting of grade boundaries – unusually, there were only four marks difference separating an A and C grade in the most problematic paper. Exam regulator Ofqual was aware of the risk to candidates and subsequently ratified the false grades awarded.
  • Despite questions over whether the exam board followed its Code of Practice, no appeals by independent schools were upheld by the board.
  • HMC and GSA call on the exam board to re-grade the papers for this qualification and ensure such mistakes do not affect this year’s candidates. They also call on Ofqual to formally review the qualification again in the light of new evidence. 
  • Over 200 candidates were awarded unclassified (U) grades in one paper even though ‘many of these students would never have achieved less than A grade in any examination they had ever taken’ says the report. Whilst they were not their final grades, they were shown on the candidates result sheets, causing anxiety and loss of confidence

A technical report, commissioned by HMC and GSA and published today (15 April), reveals that the grades awarded to independent school candidates in the English Language IGCSE qualification set by Cambridge International Examinations (CIE) in 2015 cannot be trusted.

This arose because the board decided to make the qualification more difficult in 2015 and then re-designed it so badly that the grade patterns set for at least a quarter of all candidates were almost certainly wrong, says the report.

The core standard of the 2015 qualification sat by independent school pupils was identical to that sat by 200,000 state school candidates. This has led representatives of state secondary schools to say that this new research strengthens the doubts they had about the accuracy of many of their 2015 results.

Background

Following a large number of results in 2015 which were exceptionally out of line with predictions and candidates’ overall  performance in other subjects, HMC and the Girls’ Schools Association (GSA) commissioned a detailed report on the reliability of grades awarded by Cambridge in IGCSE English Language. The study was based on the patterns of results recorded for 4,836 candidates who took the syllabus 0500 version of the qualification – more than one quarter of all of the entrants – compared with 43,293 grades that this group of students achieved across all other subjects at GCSE.

The report was written by Dr Peter Mason, former Chair of the HMC/GSA Academic Policy Committee. Dr Mason, now retired, also has extensive experience as a senior examiner and as a former member of the governing council of one of England's leading examination boards.

Results

A detailed breakdown of the results for each paper within the IGCSE qualification showed highly unlikely variations in grade patterns. Similar unexplained patterns were found when these students’ English Language results were compared with those they achieved at the same time in English Literature and History – subjects which test similar skills.

Within the English Language IGCSE, candidates could choose one of two pairs of papers contributing to their final grade: papers 2 and 3; or papers 2 and 4. The overall grade achieved varied hugely depending on the pairing of papers chosen, something that could not be attributed to differences in the ability of the candidates entered. Cumulative grade profiles were as follows:

  A* A B
Candidates taking papers 2 & 3 42.3% 69.1% 88.4%
Candidates taking papers 2 & 4 16.3% 50.6% 81.2%

Source: table 2 on p. 7; synopsis report p. 2

When the results of candidates entered for the Cambridge IGCSE were compared with the average results that they secured for all other subjects at GCSE in 2015, there was a similarly strong misalignment in the cumulative grade profiles.

  A* A B
4,836 candidates taking CIE IGCSE English 27.4% 61.0% 86.6%
43,293 grades of these same candidates in all other GCSE subjects 48.7% 76.2% 91.6%

Source: table 8 on p. 12; synopsis report, p.3

Similarly, the results profile for IGCSE English Language was strongly out of alignment compared to the results achieved by all GCSE candidates nationally, leading Dr Mason to conclude that ‘the problems identified in this study cannot be explained by attributing them to a national trend’ (p. 13).

Specific report findings

  • Overall, a major problem with accuracy occurred when the board set the A/B and C/D grade boundaries, leading to unsafe results on a large scale.
  • Over 200 of the independent school candidates sitting Paper 2 were awarded unclassified (U) grades even though ‘many of these students would never have achieved less than A grade in any examination they had ever taken’. (p.10)
  • This exam paper (Paper 2) ‘showed a grade profile completely out of line with the abilities of candidates’. (p. 4; synopsis p.2)
  • This paper was also taken by almost 200,000 state school candidates in 2015. Enquiries from schools to Cambridge about grades awarded to these candidates rose 225% compared to 2014. (p. 2)
  • The standard set across the whole IGCSE was the same for both independent school and state school entrants.
  • A ‘huge disparity’ occurred in grades awarded depending on the combination of papers taken by each candidate that cannot be otherwise explained (the ability profile of the different groups being very similar). (p. 7)
  • The exam regulator, Ofqual, reported in December 2015 that, in the key problem paper (Paper 2), the difference separating grade A and grade C was only 4 (raw) marks. Dr Mason comments: ‘It is astonishing that Ofqual seemingly fail[ed] to challenge this or analyse it further but [said], rather meekly: “we concluded that Cambridge International had carried out its grading appropriately. Cambridge International has carried out further analysis since the summer and is confident that their grading was appropriate.”’  (p. 18)
  • One inadequacy in this judgement is made clear by the board’s practice of regarding marker accuracy as sufficient if within a band of two raw marks. The extremely narrow grade bands in paper 2 meant that, when reviewed in response to a school enquiry:  ‘if a senior examiner judge[d] a candidate’s paper to be top grade B standard but the original examiner [thought] it was only a low grade C, the original examiner’s mark will [have] remain[ed]’.  (p. 21)
  • There is no clear evidence to show that the accuracy checks required of the board by its own Code of Practice were properly followed.
  • Failure to follow procedure is a strong ground for making a formal appeal to the board. No appeals by independent schools were upheld by the board ‘despite schools raising non-compliance with the Code of Practice as part of their case’.  (pp. 21-23; synopsis p. 4)

Quotes

Chris King, Chair of HMC and Headmaster of Leicester Grammar School, said:

“This is a very thorough technical report. It is clear that the 2015 grades awarded to candidates in Cambridge IGCSE English Language are not safe.”

“Yet again, schools have known that students have been graded unfairly but have been unable to gain justice for pupils under the current system. We are deeply frustrated by the denials and unconvincing explanations received from the examinations industry – in this case Cambridge International Examinations and the regulator Ofqual.”

“Following months of meetings with the exam board and fruitless formal appeals made by schools we have been compelled to publish this report in order to ensure greater transparency, confidence and fairness for future candidates”.

“The regulator must be more effective and not approve such badly designed qualifications, with tiny differences between grade boundaries. This is particularly important given Ofqual’s current proposals to disallow any change to an original result if it is thought to be accurate within a few marks. Under such rules candidates for this qualification would still have been awarded false and highly unfair grades damaging to their future careers.”

Caroline Jordan, President of GSA and Headmistress of Headington School, said:

“Schools have suspected for months that last summer’s Cambridge IGCSE English Language candidates received the wrong grade and now this rigorous statistical analysis has proven it.”

“What the CIE should now do is reverse the incorrect grading of 2015 which has had devastating consequences for thousands of pupils. At the very least it is imperative that CIE learns from its mistakes and ensures that all grades it awards and signs off this summer are accurate.”

Malcolm Trobe, acting General Secretary of the Association of School and College Leaders, said:

“Many ASCL members who lead state-maintained schools reported very similar concerns after last summer’s results to the grade awarding errors clearly identified in the report now published by HMC and GSA.”

“Thanks to this valuable new research we can see that school leaders in both independent and state schools were entirely right to doubt the accuracy of many of the grades awarded by Cambridge for its English-Language IGCSE in 2015.”

“Given the evidence in this report, the fact that this qualification has been given a clean bill of health by the regulator is especially concerning”.

Bill Watkin, Director at SSAT and Chief Executive designate of the Sixth Form Colleges Association, said:

“When the CIE results were released last August, students in a large number of schools experienced an unexpected drop in their IGCSE grades.  SSAT (the students, schools and teachers network) was contacted by a large number of school leaders who were surprised by the grades awarded and were extremely disappointed on behalf of the young people whose grades were, in too many cases, completely out of line with their prior performance.”

“Dr Mason’s report shows that they were right to be alarmed.  It would not surprise me if Ofqual were to formally review this qualification for a second time in the light of what this report has found.”

Heads who have experienced lengthy appeals to Cambridge English language 0500 results include: 

Click here to download the full CIE 2015 IGSE English (0500) Report

Click here to download the Synopsis of the CIE 2015 IGSE English (0500) Report

For further information please contact:

Sue Bishop, External Relations Director, HMC 07787 294808

Rachel Kerr, Communications Manager, GSA 07921 685012