Model-Change Regression
A test set on your own cases that your team re-runs whenever the model may have changed, with each run compared with the last. We set it up; your team runs it.
industry
Validate vendor AI on your own cases whenever the model may have changed.
Test AI systems and organise evidence for review.
Rules that travel
Supervisors use different words, but they ask the same questions about AI. The Basel Committee's principles on operational resilience and third-party risk expect a bank to withstand technology failures, including at its providers, and to check each provider before signing and monitor it after. In the EU, DORA requires banks, insurers and other financial entities to keep a register of their ICT arrangements and to put set terms into those contracts. The EU AI Act treats credit scoring of individuals, and risk assessment and pricing for life and health insurance, as high-risk from 2 December 2027 (as amended in July 2026). We test vendor AI on your own cases, before you sign and whenever the model may have changed.
- ISO/IEC 42001: AI management system Standard · published Dec 2023 · as at Oct 2026
- NIST AI RMF and the GenAI Profile Guidance · published Jan 2023 · as at Oct 2026
- OECD AI Principles Guidance · published May 2024 · as at Oct 2026
- EU AI Act: reaches AI output used in the EU Law · in force · as at Oct 2026
Which rules apply to you is your compliance or legal team's call. We map our evidence to them; we don't certify against them.
The problem
In Australia, vendor AI is approved on the vendor's presentations and a point-in-time sample check, and the contract says little about model changes.
Your own test set is re-run whenever the model may have changed, and every failure is listed.
APRA, Letter to industry on artificial intelligence (AI), as at
How the work flows
Choose
Pre-contract Vendor AI Evaluation
Validate
Independent validation with conduct-risk recall
Run and change
Model-Change Regression
Agentic AI Control Evaluation
Hand over
Second-line harness training
Policy & Procedure Assistant
Only where your platform doesn't already do it.
- System
- Contact-centre call summaries
- Compared
- Current model against the vendor's upgrade
- blocked: Complaint missed by the upgrade
- review: Hardship-related complaint flagged by one model only
- ok: Summary cites the call segment
Sources
- Fictional call Segment 3
What we build for financial services
Lead solutions
Vendor AI Evaluation
1–3 vendor tools tested before you sign, side by side, on your own content and questions, in accounts your organisation controls.
Independent Evaluation Report
One system and one use case, the full method and the evidence bundle, with the harness handed over.
Use your platform's AI first
If your AI runs as a Copilot Studio agent, use its built-in agent evaluation to re-run the same test set after each change. Bring us in for an independent test on your own labelled cases.
What we don't do
- We don't red-team or penetration-test in-house. A CREST-accredited tester you engage does.
- We don't build a private AI platform or model gateway.
- We don't write AI policy paperwork. Your policy owners do.
- We don't take on work that supports a critical or important function of an EU bank or insurer under DORA.
Illustrative scenario: not a client engagement
Australian exampleComplaint flags after a summary-model change
Problem
A fictional customer-owned bank's contact-centre AI flags possible complaints. Its vendor announces a model upgrade without saying what changed.
Approach
The complaints team labels past, de-identified calls in the bank's own environment. The same calls run through both models.
How we'd measure
- Complaint recall, current model against the upgrade
- Recall on hardship and vulnerability cues, reported separately
What ships first
A labelled test set and a baseline run to compare the upgrade against.
- 10:00:00 set.label Past calls labelled by two complaints staff
- 10:12:31 run.compare Current and upgraded models run on the same calls
- 10:31:44 flag.miss Hardship-related complaint missed by the upgrade
- 10:40:15 report.misses Every miss listed with its call reference
A first engagement in financial services
What we need from you
- One AI system and one use case, with a named control owner
- Read-only access to past calls, chats or emails
- One or two people to label examples
What you get
- An Independent Evaluation Report listing every failure
- An evidence bundle and replay script for your own subscription
- A handover so your second line can re-run the test
The exit ramp
Once thresholds are agreed, you decide whether to continue. The test set is yours either way.
Australia: our detailed map
Australia is our home market and the one we know best.
Designed around
Works alongside
- Copilot Studio (conversation transcripts) Read-only access
- Microsoft Copilot (Purview audit log) Read-only access
- Genesys Cloud CX (S3 recording export) Read-only access
- SharePoint (procedure libraries) Read-only access
Regulatory map
| Obligation | What it means for an AI system | How we design for it | Evidence you get | Source |
|---|---|---|---|---|
| APRA letter to industry on AI (30 Apr 2026): continuous monitoring, second-line capability | APRA expects AI monitoring to be continuous and proportionate to how critical the use case is, and expects second line and internal audit to have the skills and tooling to assess AI systems independently. | Your second line gets the harness to re-run the test on its own schedule. | A reproducible evidence bundle with a replay script for your own subscription | Source, as at 30 April 2026 |
| CPS 230 ¶29: regularly test controls for design and operating effectiveness | Where an AI tool performs or supports a control, such as flagging complaints, that control must be tested at a frequency matched to the risk, with results reported to senior management. | Thresholds are agreed with your control owner before the first run. | Control-test scripts for your GRC, in a format agreed at scoping, with the failure list | Source, as at 1 July 2026 |
| CPS 230 ¶52, ¶59–60: due diligence, monitoring and APRA notice for material providers | If an AI vendor supports a critical operation or exposes you to material operational risk, you must do due diligence before signing or materially changing the arrangement, regularly assess how well your controls manage the risks of using it, and notify APRA of critical-operation arrangements and before any material offshoring, which can include inference run overseas. | Vendor AI is tested on your cases before you sign, and the data note records the vendor's published processing location, with its date. | A side-by-side vendor test report before contract | Source, as at 1 July 2026 |
| ASIC REP 798 (29 Oct 2024): licensee duties applied to third-party AI models | ASIC says existing licensee obligations apply to AI and licensees stay responsible for outsourced functions, and asks how they validate, monitor and review third-party AI models. | The report records the model details your vendor discloses, and those it won't. | A vendor evaluation report listing undisclosed model details | Source, as at 29 October 2024 |
| RG 271.27–271.32: recognise complaints even when not labelled or not in writing | An AI that summarises or triages customer contacts sits where your firm must recognise complaints: dissatisfaction where a response is expected, even if the customer never says 'complaint' or speaks rather than writes. | Recall is measured separately for complaints and hardship-related complaints, and every miss is listed. | A recall report by category with the list of missed complaints | Source, as at 2 September 2021 |
Last reviewed
General information, not legal advice.
Related insights
The EU AI Act's new dates for non-EU suppliers.
Regulation (EU) 2026/1744 moved Annex III high-risk duties to 2 December 2027 and Annex I to 2 August 2028. What applies now to AI suppliers outside the EU.
Australia Ask an AI vendor for evidence you can check.
Questions to ask before you buy or renew an AI product, drawn from APRA, the DTA, NSW buying guidance and the OAIC, and what a good vendor answer looks like.
Australia Find the automated decisions in your software.
From 10 December 2026, privacy policies must name decisions a computer program makes or substantially assists. How to find yours, licensed software included.
Questions
How can we check your findings when you have no clients yet?
Don't rely on our name; we have no clients yet. The method is published, and every finding comes with the test set, labels and a replay script, so your second line can re-perform it.
Can we give your report to the board or our regulator as a formal opinion?
No. It's findings, not an assurance opinion, and it isn't an ISAE 3000 or ISRE engagement (ASAE 3000 or ASRE in Australia). What you table, and how, is your decision.
Does customer data leave our environment?
We don't take a copy. The harness and graders run in your own Microsoft or AWS subscription. Our team reaches your environment remotely from Sydney, under the data-transfer terms in your contract. The vendor's system processes each test case where it runs in normal use, and the data note records the vendor's published processing location, with its date. If a grader would need a model outside your region, we tell you before it runs.
Do we need to onboard you as a new supplier?
Not necessarily. Through our Evaluation Partner route, the work can run under a firm you've already onboarded, such as your internal-audit co-source. How your outsourcing rules apply (CPS 230 in Australia) is your call.
Our vendor won't give model-change notice. Can testing make up for that?
It can't replace contract terms, but it gives you your own evidence. Some platforms don't record which model version answered, so a scheduled re-run is one practical way to notice a change.
Let’s discuss your workflow.
Talk to us about AI for financial services