How I learned to stop worrying and love the algorithm
United Kingdom, 2020 AD. Government decision to calculate A-level results with an algorithm ends up in a generalised revolt. As students around the country find out that their calculated grade is significantly lower than expected, they welcome the news with “a bit of disappointment”.
A fierce debate sparks in no time: it turns out that grades are favouring private schools over public ones, at the expenses of students from disadvantaged backgrounds. Ofqual shuts down appeals against test results, while political pressure starts mounting against the Secretary of Education.
Eventually, A-level grades are scrapped in favour of teachers’ predicted results.
All’s well that ends well?
Should we feel relieved at this point? And what does the GDPR has to do with all of this?
Indeed, the GDPR gives the right to individuals “not to be subject to a decision based solely on automated processing […] which produces legal effects […] or similarly significantly affects” an individual. This includes the right to contest an automated decision, as well as to obtain a human review of the latter.
Needless to say, this provision has likely played an extremely important role in our story. Ofqual attempt to suspend appeals against test results effectively opened the way for a surge of GDPR complaints, while non-profit organisations were already preparing for court cases. Ofqual must have realised that and, as reported by the ICO in a rather weak statement, they later claimed that no automated decision making was involved in their grading. It is reasonable to assume that such a weak line of defence, together with the threat of hefty fines, eventually tipped the balance in favour of UK students.
However, what if the GDPR had not been there? Would have Ofqual revived empathy for students’ future careers prevailed over the prospect of a humiliating U-turn?
This is hardly a rhetorical question: the Brexit implementation period will end this year, and Government is working on a trade deal with the US which covers e-commerce and personal data flows. As Number 10 has already stated that they intend to diverge from GDPR standards, a future where GDPR safeguards are scrapped for the sake of a trade agreement seems awfully close.
A lesson to be learned
The A-level grading debacle should be taken as a warning sign. Government plans to diverge from the GDPR means that we may lose an incredibly important protection against algorithmic decision making. This would leave us defenseless against the next A-level failure, while reliance on automated decision making is on the rise — from Visa applications to predictive policing, job recruitment, mortgages and rents.
In other words, if you are not convinced that we should “stop worrying and love the algorithm”, you are not alone.
Open Rights Group has highlighted before the risks involved in a trade deal with the US, and will keep fighting to defend and reinforce our digital rights. Want to know more? Follow our digital trade section, or join us and help us shape the future of UK digital rights.