In this article7 sections
AI bias testing under the EU AI Act works differently for the company that builds a system and the company that uses it. Providers of high-risk AI must manage bias in their data and risk management (Articles 9 and 10). From 2 December 2027, deployers of high-risk systems must monitor how the system performs (Article 26(5)). Monitoring for bias is the practical way to do that. Fines for deployer breaches reach €15 million or 3% of global annual turnover, the lower of the two for SMEs (Article 99).
This post breaks down what the law asks, which protected characteristics matter, and how a small or mid-sized company can run credible bias testing without a team of data scientists.
Why Annex III Systems Trigger Bias Obligations
The EU AI Act sorts AI systems into risk tiers. High-risk systems are listed in Annex III, and they include systems used in recruitment and CV screening, credit scoring, access to education, benefits decisions, and law enforcement risk assessment.
If you are an HR director whose company uses an AI tool to rank job applicants, your system very likely sits inside Annex III, and the high-risk rules apply to it from 2 December 2027. Same goes for a financial services COO whose platform uses algorithmic credit scoring, or an education provider relying on AI-driven adaptive assessments.
Being a deployer (not the vendor who built the system) still carries direct legal weight. Article 26 is explicit: from 2 December 2027, deployers must use the system according to the provider's instructions, monitor its operation, and inform the provider and the authority of serious incidents. Bias is a performance issue. A system that systematically disadvantages people based on gender, ethnicity, or disability is not working as it should.
What "Bias Testing" Actually Means in Practice
Bias testing is not a single checkbox event. The sensible approach is a continuous monitoring cycle with documented evidence at three stages. In the Netherlands the AP is the intended AI supervisor under the draft Dutch implementing law; it has no AI Act powers until that law is in force.
1. Pre-Deployment Testing
Before a high-risk AI system goes live, you need baseline bias measurements. This means asking your vendor for disaggregated performance data: how does the system perform across gender subgroups, ethnic backgrounds, age brackets, and disability status? Article 9 of the regulation requires the provider's risk management system to identify and analyse known risks, including bias risks, before the system is placed on the market.
If the vendor cannot give you disaggregated metrics, that is itself a red flag. Article 13 requires providers of high-risk systems to give you instructions for use, including accuracy and known risks. Ask for subgroup performance data too. Document the request and the response.
Pre-deployment testing should produce at least:
-
A baseline fairness metric (e.g. demographic parity difference, equalised odds, or predictive parity) for each protected characteristic relevant to your use case
-
A written assessment of whether those metrics meet your acceptable threshold
-
Sign-off from a senior decision-maker, not just IT
2. Ongoing Monitoring After Go-Live
The regulation does not let you test once and forget. From 2 December 2027, Article 26(5) requires deployers to monitor system operation on the basis of the instructions for use, and to inform the provider and the authority, and suspend use, when they see a risk. Bias drift, where a system becomes more discriminatory over time as input data changes, is a known phenomenon.
A practical monitoring schedule for an SME:
-
Monthly: run your fairness metrics on recent decision outputs using the same protected class breakdowns as pre-deployment
-
Quarterly: review whether your protected class definitions remain appropriate (EU anti-discrimination law covers gender, racial or ethnic origin, religion, disability, age, and sexual orientation)
-
Annually: conduct a full bias audit, which at minimum involves rerunning pre-deployment tests on current system outputs with fresh data
Keep the records. A supervisor may ask to see them.
3. Protected Class Coverage
EU anti-discrimination law, as codified in Directive 2000/43/EC and Directive 2006/54/EC, protects: racial or ethnic origin, sex, religion or belief, disability, age, and sexual orientation. Your bias testing must cover the characteristics relevant to your deployment context.
For HR tools, gender and ethnicity are the highest-priority characteristics. For credit scoring, age and disability status often surface as key risk areas. Do not test only on the characteristics that are easiest to measure.
Tools You Can Use Without a Data Science Team
Two open-source libraries are practical starting points.
IBM AIF360 (AI Fairness 360) provides over 70 fairness metrics and 11 bias mitigation algorithms. It supports tabular data, which covers most HR and credit scoring use cases. The documentation is solid and the community is active.
Microsoft Fairlearn integrates directly with scikit-learn and Azure Machine Learning, making it practical for teams already using Python. Its dashboard gives a visual breakdown of model performance across subgroups, which is useful when presenting results to non-technical stakeholders like HR directors or compliance officers.
Both tools produce output that can be exported and stored with your monitoring records. Providers must keep technical documentation for 10 years (Article 18). Deployers must keep system logs for at least six months (Article 26(6)).
One important caveat: these tools measure statistical fairness. They do not capture qualitative harms, context-specific discrimination, or the lived experience of the people affected. Software output is evidence, not the whole picture.
When to Bring in an External Auditor
Not every company needs to hire an external auditor immediately. But there are clear triggers that make external audit the right call.
Trigger 1: Your system makes decisions that significantly affect employment or credit. For recruitment AI, credit scoring, and benefits eligibility systems, the stakes are high enough that internal testing is rarely sufficient on its own. An external auditor brings independence that carries more weight.
Trigger 2: You have found a bias problem and fixed it. Documenting a remediation with an independent verification strengthens your position significantly if a complaint is ever filed.
Trigger 3: Your vendor is pushing back on documentation requests. If a provider refuses to supply disaggregated performance data or technical summaries, you have a conflict of obligations. An external auditor can help you establish what the system is actually doing, and that documentation protects you regardless of what the provider claims.
Trigger 4: You are preparing for a Fundamental Rights Impact Assessment (FRIA). Under Article 27, from 2 December 2027 a FRIA is required only if you are a public body, a private organisation providing a public service, or you use AI to assess the creditworthiness of individuals or to price life and health insurance. A bias audit is not the same as a FRIA, but the two processes are deeply connected. External auditors can help run them together efficiently.
The AVG/GDPR Interaction You Cannot Ignore
Bias testing often requires processing data about protected characteristics. In most EU jurisdictions, racial or ethnic origin and health data are special category data under Article 9 of the GDPR (AVG in the Netherlands). You cannot simply run disaggregated bias tests without a lawful basis for processing that sensitive data.
The practical solution is to work with anonymised or pseudonymised datasets for testing, or to rely on explicit consent or a legitimate public interest basis documented in your records of processing activities (RoPA). Since the AI Omnibus, Article 4a of the AI Act also allows deployers of high-risk systems to process special category data exceptionally, where strictly necessary for bias detection and correction and with strict safeguards. Your Data Protection Officer should be part of the bias testing design process, not a reviewer after the fact.
Discriminatory AI outputs raise questions under both the AVG and the AI Act. Treat them as one compliance exercise, not two.
Your Next Step
Bias monitoring is one of five core compliance areas Khairos Comply helps European SMEs structure, document, and evidence. Run the free 2-minute compliance check at comply.khairos.ai to see which obligations apply to your specific AI systems and where your documentation gaps are likely to be.
References
Not sure where your company stands?
Answer 15 questions about how your team uses AI. You get a compliance score and the gaps to close first, as a PDF.