Independent Evaluation Report
One system and one use case, the full method and the evidence bundle, with the harness handed over.
industry
Measure how your AI fails before it leaves pilot.
Evaluate AI and streamline policy-led workflows.
Rules that travel
Public bodies in many countries are asked the same questions about AI: what it's for, who it affects, how it was tested and who is accountable. The OECD AI Principles ask for traceability and ongoing risk management, and UNESCO's Recommendation on the Ethics of AI, adopted by 193 member states in 2021, sets out shared principles. In the EU, from 2 December 2027 (as amended in July 2026), public bodies must assess the impact on fundamental rights before they first use most high-risk AI systems listed in the AI Act's Annex III. We measure how your AI fails, on your own questions, before it leaves pilot. For government, we only claim to know Australia's rules. Outside Australia, we test unclassified systems on public or synthetic data.
- ISO/IEC 42001: AI management system Standard · published Dec 2023 · as at Oct 2026
- NIST AI RMF and the GenAI Profile Guidance · published Jan 2023 · as at Oct 2026
- OECD AI Principles Guidance · published May 2024 · as at Oct 2026
- EU AI Act: reaches AI output used in the EU Law · in force · as at Oct 2026
Which rules apply to you is your compliance or legal team's call. We map our evidence to them; we don't certify against them.
The problem
In Queensland, Australia, AI tools reach daily staff use before a dedicated risk assessment is done or monitoring controls are in place.
Each use case leaves pilot with a measured test on your own questions, its failures listed, and a re-test when the model changes.
Queensland Audit Office, Managing the ethical risks of artificial intelligence, as at
How the work flows
Choose
Vendor AI Evaluation before contract
Tender AI-Claims Verification
Leave pilot
Independent Evaluation Report
Council Policy & Procedure Assistant
For local government and small bodies.
Run agents
Agentic AI Control Evaluation
After a change
Model-Change Regression
- Use case
- Staff questions on leave and travel
- Ground truth
- Answers labelled by two policy officers
- Official accountable for AI use
- (redacted)
- blocked: A confident answer the policy doesn't support
- review: Declined, though the policy covers it
- ok: Cited clause says what the answer claims
Sources
- Fictional travel policy Section 4
What we build for government
Lead solutions
Use your platform's AI first
If your staff have Microsoft Copilot, try agents in SharePoint first: they answer from a site's pages within each person's permissions. If Copilot Studio's built-in evaluation answers your question, we'll say so.
What we don't do
- We don't build a general staff chat assistant to replace the one you already run.
- We don't do adversarial testing in-house. A CREST-accredited tester you engage does it.
- We don't sell AI policy paperwork, such as AI use-case registers or transparency statements.
- We don't independently evaluate a system we built, configured or advised on for the same agency.
Illustrative scenario: not a client engagement
Australian exampleA policy agent leaving pilot at a fictional state department
Problem
A fictional department has piloted a Copilot Studio agent that answers staff policy questions. Before rollout, the official accountable for AI use wants to know how often it answers wrongly, and whether it declines when the policy is silent.
Approach
Two policy officers label expected answers independently, and the test runs in the department's own subscription.
How we'd measure
- False answers the policy doesn't support
- False refusals when the answer is in the policy
- Whether each cited clause says what the answer claims
What ships first
A test plan and a first failure list from public policy questions, before any staff data is touched.
- 09:00:00 plan.thresholds Pass thresholds agreed before the run
- 09:05:02 run.replay Labelled questions replayed in the department's own subscription
- 09:38:41 grade.answer Confident answer the travel policy doesn't support
- 09:52:30 report.failures Failure list written, rated to the department's risk matrix
A first engagement in government
What we need from you
- A named owner, and the AI assessment framework you report to (in Australia, the DTA tool, the NSW AIAF or a FAIRA)
- 30–50 questions your staff can answer, or approval to start on public or synthetic data
- Read-only access to the system, and your risk matrix
What you get
- A test plan with thresholds agreed before testing
- An Independent Evaluation Report with the explicit failure list
- The evidence bundle, replay script and harness, handed over to your team
The exit ramp
Once the test set and thresholds are agreed, and before testing starts, you decide whether to continue. If you stop, the test set and any results so far are yours.
Australia: our detailed map
Australia is our home market and the one we know best.
Designed around
Commonwealth
Designed around
Works alongside
- Microsoft Copilot (Purview audit log) Read-only access
- Copilot Studio (conversation transcripts) Read-only access
- SharePoint and OneDrive (Sites.Selected) Read-only access
What we build for government
Model-Change Regression
A test set on your own cases that your team re-runs whenever the model may have changed, with each run compared with the last. We set it up; your team runs it.
Agentic AI Control Evaluation
A configuration review of your agents' identity, permissions, approval gates and logging, before they act in case or customer systems. It is not an attack. Adversarial testing is done by a CREST-accredited tester you engage.
Regulatory map
| Obligation | What it means for an AI system | How we design for it | Evidence you get | Source |
|---|---|---|---|---|
| DTA AI policy v2.0: finalise the use case impact assessment before deployment | Before an in-scope AI use case is deployed, the agency must finalise its impact assessment and apply the agreed risk treatments. Agencies must begin these assessments within 12 months of v2.0 taking effect (15 December 2025), and bring existing use cases into line by 30 April 2027. | Findings are mapped to the tool's sections, such as reliability and safety, or privacy. | A findings table keyed to the tool's sections, with the explicit failure list | Source, as at 15 December 2025 |
| DTA AI policy v2.0: monitor in-scope use cases; re-validate after a material change | A deployed use case must be monitored, and its impact assessment re-validated after a material change. Agencies should also watch for vendor changes. The DTA says the remaining v2.0 requirements take effect in December 2026. | The handed-over test set re-runs when the model or vendor changes. | A regression table comparing baseline and new runs on the same test set | Source, as at 15 December 2025 |
| DTA agentic AI addendum, AGT.3.3 and AGT.8.1: agent identity and monitoring | Each agent should have its own identity and only the access it needs, and be monitored for goal drift and tool misuse. It's best-practice guidance that agencies using agentic AI are expected to apply. | The Agentic AI Control Evaluation reviews each agent's identity, permissions and logging, as configured. | A permissions and approval-gate matrix per agent, each gap rated to your risk matrix | Source, as at 4 June 2026 |
| ASD, Careful adoption of agentic AI services: low-risk tasks, never broad access | The authoring agencies recommend agentic AI only for low-risk, non-sensitive tasks, never with broad or unrestricted access, and a person should approve high-impact actions. It's guidance, not a mandate. | The review checks that your designers, not the agent, decide when a person must approve. | A record of which actions need human approval, per agent, and where that gate is enforced | Source, as at 1 May 2026 |
Last reviewed
General information, not legal advice.
State (NSW, Vic, Qld)
Designed around
Works alongside
- Microsoft Copilot (Purview audit log) Read-only access
- Copilot Studio (conversation transcripts) Read-only access
- SharePoint and OneDrive (Sites.Selected) Read-only access
What we build for government
Independent Evaluation Report
One system and one use case, the full method and the evidence bundle, with the harness handed over.
Tender AI-Claims Verification
The government form of the Vendor AI Evaluation: the AI claimed in each tender response is tested on your own questions, with a probity note and sign-off from your procurement lead.
Vendor AI Evaluation
1–3 vendor tools tested before you sign, side by side, on your own content and questions, in accounts your organisation controls.
Regulatory map
| Obligation | What it means for an AI system | How we design for it | Evidence you get | Applies to | Source |
|---|---|---|---|---|---|
| NSW DCS-2026-02 and AI Operational Policy: register AI use cases, apply the AIAF | NSW agencies must register AI use cases in the AIAF Platform before use, and re-assess when risk, context or functionality changes. The policy says routine productivity use of approved tools, with meaningful human oversight, doesn't generally need registering. | Findings are rated to AIAF risk levels, and every test run is dated. | An evaluation report rated to AIAF risk levels, with a dated run history | NSW | Source, as at 1 September 2026 |
| Vic, Navigating AI in procurement: assess AI, get supplier disclosure, list permitted uses | Victorian buyers are guided to complete the Victorian Government AI Assurance Framework (being piloted), have suppliers disclose AI use, and list permitted AI uses in the contract. It's guidance, not a mandate. | Tender AI-Claims Verification tests tendered AI on your questions, with a probity note. | A vendor comparison with each tender claim marked supported or not supported | Vic | Source, as at 3 August 2026 |
| Vic Administrative Guideline on gen AI in the VPS (made 27 Nov 2024) | The guideline says gen-AI tools shouldn't be used to make decisions or undertake assessments, and inputs to an approved tool must not exceed the protective marking set for that tool. | Assistants retrieve and cite, not decide, and the tool's marking sets what they can read. | A data note listing the collection's protective markings and approved tools | Vic | Source, as at 12 February 2025 |
| Qld AI governance policy v2.0.0: a Foundational AI risk assessment (FAIRA) when evaluating AI | Queensland departments must prepare a FAIRA, or an agreed equivalent, when evaluating AI solutions. The policy says a FAIRA doesn't replace privacy impact, security or human rights assessments. | Findings are cross-referenced to your FAIRA, so it cites measured results. | Findings cross-referenced to the agency's FAIRA, with measured error rates attached | Qld | Source, as at 26 August 2026 |
Last reviewed
General information, not legal advice.
Local government
Designed around
Works alongside
- SharePoint and OneDrive (Sites.Selected) Read-only access
- Copilot Studio (conversation transcripts) Read-only access
- Microsoft Copilot (Purview audit log) Read-only access
What we build for government
Policy & Procedure Assistant
One policy and procedure library for one staff group, delivered inside Microsoft 365 (Copilot, SharePoint or Teams) where you license it, otherwise on the web. Answers cite the policy, section and version, and say "not in our documents" when the library is silent.
One system, one use case
One AI system and one use case, tested on your own questions and cases in your own environment, with thresholds agreed before testing and the same written method throughout.
Vendor AI Evaluation
1–3 vendor tools tested before you sign, side by side, on your own content and questions, in accounts your organisation controls.
Regulatory map
| Obligation | What it means for an AI system | How we design for it | Evidence you get | Source |
|---|---|---|---|---|
| NSW PPIP Act, IPP 5 and IPP 9: store securely, and check accuracy before use | NSW agencies and councils must take reasonable steps to make sure personal information is accurate and complete before using it, and must store it securely. In our reading, that covers an AI summary of a person's records. | Answers cite their source page, so staff can check them. | An acceptance test report counting false answers separately from false refusals | Source, as at 17 August 2026 |
| Qld Information Privacy Act 2009, QPP 10 and QPP 11: data quality and security | Queensland agencies, including local governments, must take reasonable steps to make sure the personal information they use is accurate, up to date, complete and relevant, and to protect it from misuse and unauthorised access. | Each user sees only what they can already open, checked for every staff group. | A permission-check log from the acceptance test, per user group | Source, as at 1 July 2026 |
| NSW State Records Act 1998 and FA450: council AI-generated records are State records | A record a council creates with generative AI during council business is a State record: it must be saved in an official recordkeeping system and disposed of only as authorised. Drafts that show how decisions were made must be kept. | Prompts, answers and citations are logged for capture, marked as AI-generated. | A sample log export with AI-generated, tool and prompt metadata fields | Source, as at 9 February 2024 |
| PROV AI Technologies and Recordkeeping Policy v1.0: document AI use; human checks | Victorian councils are public offices, and AI-generated content is a public record. Councils are to document, as fully as practical, what the AI can and cannot access, and a skilled person with authority must oversee accuracy checks. | A data note says what each assistant can read and who checks it. | A one-page data note naming readable and excluded collections and the checker | Source, as at 29 February 2024 |
| Qld Public Records Act 2023: council records are public records in any format | Queensland councils have the same recordkeeping duties as state public authorities. In our reading, an AI output that documents council business is a public record; QSA hasn't published AI-specific guidance for councils. | Logs export for your records team, with a proposed retention class they confirm. | A log export with a proposed retention class for your records team | Source, as at 22 September 2026 |
Last reviewed
General information, not legal advice.
Questions
We're a council. What do you design around?
Your state's privacy and records rules, listed with their sources in the Local government section: the PPIP Act and State Records Act in NSW, PROV's recordkeeping policy for Victorian records, and the Information Privacy Act and Public Records Act in Queensland.
Related insights
The EU AI Act's new dates for non-EU suppliers.
Regulation (EU) 2026/1744 moved Annex III high-risk duties to 2 December 2027 and Annex I to 2 August 2028. What applies now to AI suppliers outside the EU.
Australia Ask an AI vendor for evidence you can check.
Questions to ask before you buy or renew an AI product, drawn from APRA, the DTA, NSW buying guidance and the OAIC, and what a good vendor answer looks like.
Australia NSW agencies need AI evidence from suppliers.
What circular DCS-2026-02 and the NSW AI Operational Policy require of agencies, and the testing, change and record evidence they'll need from AI suppliers.
Questions
How can we check your evaluation when you have no clients yet?
Don't take it on trust. We have no clients yet, in government or anywhere else, so we publish the method instead: sample size, ground truth and thresholds. You get the test set and a replay script, so your team can check our work.
Will your report tell us the system is safe to deploy?
No. You get findings, not an assurance opinion, and it isn't an ISAE 3000 (ASAE 3000 in Australia) engagement. Each finding is rated to your risk matrix, and the official accountable for AI use decides whether to deploy.
Do you evaluate systems you built, or take money from vendors?
Not independently, and no. We never independently evaluate a system we built, configured or advised on for the same client: our tests of our own builds are labelled "acceptance test (not independent)". We take no resale margin or referral fees from vendors we evaluate.
Will you red-team or penetration-test our AI agents?
No. The Agentic AI Control Evaluation is a configuration review. Where prompt-injection or tool-misuse testing is needed, a CREST-accredited tester you engage does it, under written rules of engagement.
Does our test data leave our environment?
We don't take a copy. The test and its graders run in your own subscription. Our team reaches your environment remotely from Sydney, under the data-transfer terms in your contract. The system under test handles each question where its vendor runs it, as in normal use, and the report records the vendor's published processing location, with its date. While your privacy and security assessments run, we start on public or synthetic data in a Feasibility Sprint.
Let’s discuss your workflow.
Talk to us about AI for government