When AI gets it wrong, who answers for it?
Responsibility, algorithmic decisions, a chain of actors
The unsettling question
Imagine an AI that screens CVs and systematically rejects women because it learned to recognise a “good profile” from decades of hiring in which women were absent. Nobody programmed it to discriminate. Yet it does.
Now, who is responsible?
The model developer, who did not know exactly how it made decisions? The company that bought it without testing it on its own data? The HR department that approved deployment? The recruiter who followed its recommendations without questioning them? The candidate who never found out why her CV was rejected?
An AI cannot go to prison. Nobody will put a machine in the dock. Yet decisions made by AI change lives. Somewhere in the chain, someone must answer for them. But who?
The starting point
Current legal frameworks are designed for human actions. When a decision is made or filtered by an algorithm, responsibility is spread across many actors: those who designed the model, trained it, sold it, integrated it, use it and supervise it.
Europe is attempting to address this with the AI Act, which classifies AI uses by risk level and imposes obligations accordingly. The GDPR already regulates certain automated decisions. But there is a gap between the law and practice: who in an organisation is actually responsible when an AI system goes wrong, if the organisation uses it without understanding how it works?
There is also a troubling mechanism researchers call the moral crumple zone: when an automated system fails, the person at the end of the chain often takes the blame. The machine takes over the decision; the human takes on the responsibility.
Tomorrow
Tomorrow, AI will play a role in recruitment, healthcare, justice, credit, education, insurance and content moderation. The decisions it influences will have increasingly serious consequences.
Organisations will have to decide in advance who answers when things go wrong. Individuals will need to be able to understand and challenge decisions affecting them.
Possible angles
Reading to get started
EU AI Act — official text (2024)
To understand the European framework, risk levels and the obligations it introduces for high-impact uses.
CNIL — Guide to AI and data protection
The French framework and GDPR good practices applied to AI.
Cathy O'Neil — Weapons of Math Destruction
Concrete, accessible examples of automated systems producing injustice on a large scale.
Madeleine Clare Elish — Moral Crumple Zones
To understand how humans end up bearing the burden of errors made by automated systems.
Stanford — AI Index Report
Recent figures on reported AI incidents worldwide and their documented consequences.
AlgorithmWatch — Automating Society
Documented cases of automated systems deployed in Europe and their real effects on people.
Try it yourself
- Ask a generative AI to screen ten anonymised CVs for the same job, but vary the names. Compare the profiles selected.
- Submit the same symptoms to an AI several times under different identities. Observe the differences in diagnosis.
- Ask an AI who would be responsible for one of its own mistakes. Read its answer carefully.
- Find a real case of an AI incident in a company. Reconstruct the chain of responsibility: who should have seen it coming?
- Talk to someone whose job involves validating AI decisions. How do they experience that responsibility?
Which context would you anchor it in?
Responsibility takes on different forms in a healthcare organisation, an HR department or a platform.
Higher education
Automated assessment, fraud detection, student guidance
Business
Recruitment, credit, HR scoring, content moderation
Creative industries
Algorithmic moderation, recommendations, automated copyright management