Techsider
See a demoTalk to us

solution ④

Independent AI Evaluation

Independent tests of AI you bought or had built elsewhere.

Talk to us about AI EvaluationTry the demo
Example operations workspace connecting request intake, extraction, business rules and human approval.
Illustrative application concept

Explore an example

Open the illustrative example
Australian example

Sample evaluation report

Sample report: Techsider testing its own demo system, so not independent.

Illustrative sample: not a real test run

System
Techsider’s Knowledge Assistant demo: cited answers from Victoria’s public-sector generative AI Guideline and Guidance, and from APRA’s CPS 230
Sample size
n = 120
Method
Each question asked once, in a fresh session, and its answer graded against the ground truth. A right answer that cites a passage that doesn’t hold it counts as a wrong citation, not a false answer. Each threshold is the highest rate accepted, agreed before testing, and it is met only when the whole Wilson score interval sits below it. The failures listed are model version A’s. Model version B’s rates are shown for comparison, without intervals, and under the same rule it meets only the wrong-citation threshold.
Ground truth
An expected answer, or an expected refusal, for each question, with the passage that supports it, from the Victorian Guideline and Guidance and from CPS 230
Inter-rater agreement
Cohen’s kappa, on the double-labelled questions: 0.81
Failures mapped to
NSW AI Assessment Framework risk bands, as the NSW AI Operational Policy, Final version 1.1 (approved 31 August 2026), defines them: each failure takes the band its impact matches

Thresholds

Each threshold, its target and the result
MetricTargetResultConfidence intervalOutcome
False-answer rate5%1.7%0.5–5.9% (95% CI)Fail
False-refusal rate5%0.8%0.1–4.6% (95% CI)Pass
Wrong-citation rate5%0.8%0.1–4.6% (95% CI)Pass

Failures found

  1. false-answer-1 Rating: High Framework level: High-Risk

    Asked “Is Microsoft 365 Copilot approved for use in my department?”, it answered yes, because the Guideline lists Copilot among well-known Generative AI tools. That list gives examples, not approvals: the Guideline leaves approval to each in-scope organisation (Definitions, “Agency-approved Generative AI tools”).

  2. false-answer-2 Rating: High Framework level: High-Risk

    Asked “How quickly must we tell APRA when an incident disrupts a critical operation?”, it gave only the 72-hour clock for an operational risk incident likely to have a material impact (¶32). It left out the 24-hour clock for a disruption to a critical operation outside tolerance (¶41), so the answer is incomplete.

  3. false-refusal-1 Rating: Low Framework level: Low-risk

    Asked “What if our agency wants to work differently from the guideline?”, it said the documents don’t cover it. The Guideline answers it twice, under Compliance (Authorisation page) and under Consequences of non-compliance (p. 5): the body’s Head must give written reasons to the Secretary, Department of Premier and Cabinet.

  4. wrong-citation-1 Rating: Medium Framework level: Medium-risk

    Asked “How does CPS 230 define critical operations?”, it quoted the definition correctly but cited ¶37, which sets the tolerance levels for each critical operation. The definition is ¶34.

Model-change regression

The same test set on Model version A and Model version B
MetricModel version AModel version B
False-answer rate1.7%2.5%
False-refusal rate0.8%3.3%
Wrong-citation rate0.8%0.8%

The job

Independent AI evaluation tests a system supplied or built by another party on your own questions and cases. Agree the use case, thresholds and test method before comparing results.

What you receive

A findings report with rated failures, test inputs and a replay script. Your accountable decision maker decides whether to proceed; a test by Techsider of a system it built is acceptance testing.

Who it's for

  • A pilot is about to scale.

    Your Copilot agent or vendor tool is leaving pilot, and the official accountable for AI use wants evidence before a wider rollout. The report measures its failures on your own cases.

  • The vendor may have changed the model.

    Where your contract gives no audit rights or notice of model changes, a test set on your own cases shows whether the results changed. Your team re-runs it whenever the model may have changed.

  • You're choosing between vendors.

    Shortlisted tools run on the same questions from your own work, side by side, before you sign. In a tender, the test comes with a probity note.

  • Agents are about to act in your systems.

    Before the agents you run touch case or customer systems, a configuration review checks their identity, permissions, approval gates and logging.

Engagements

One system, one use case

For
An agency, regulated financial institution, law firm or health service with one AI system it bought, licensed or had built elsewhere, and one use case to test before relying on it.
Scope
One AI system and one use case, tested on your own questions and cases in your own environment, with thresholds agreed before testing and the same written method throughout.
Included
  • A sample size chosen and explained, with confidence intervals on every result
  • Ground truth built with your own experts, and their agreement measured
  • LLM-judge graders calibrated against human labels
  • False answers and false refusals reported separately, with an explicit list of failures
  • Findings rated to your risk matrix and mapped to the instrument you report to
  • A reproducible evidence bundle, with a replay script that runs in your own subscription
  • The harness handed over, with a session for your data or risk team
Every package includes
  • A test on your own examples, with thresholds agreed before testing and every failure shown, or, for AI Switch-On, the hours saved, measured against a baseline taken before anything changes.
  • Where the work has a test set, a re-run of it when the AI model changes or may have changed: by us on a system we build or run, and by your team, with the kit we hand over, after an evaluation.
  • A one-page data note: where your data sits, where the model runs, retention and deletion.
  • A runbook and a handover.
  • The delivery choice that fits the package, named in its region note.
  • Your time commitment, stated in hours and weeks.
Your time
A named owner who agrees the thresholds, time from one or two people who know the cases to label examples, and read-only access to the system in your environment or a test environment. Hours are stated at scoping.
Timeline
3–5 weeks. For government, approvals run in parallel in Week 0 while testing starts on public or synthetic data.
Out of scope
  • Audit opinions: the report is findings, not an assurance opinion, and not an ISAE 3000 or ISRE engagement (ASAE 3000 or ASRE in Australia)
  • Any system Techsider built, configured or advised on for you
  • Adversarial testing in-house. Where it's needed, it's done by a CREST-accredited tester you engage.
  • The decision to deploy: the official accountable for AI use makes it
Go/no-go gate
Once the test set and thresholds are agreed, and before any testing, you decide whether to continue. The test set is yours either way.
Region note
The test runs in your own environment, and the graders run in your own subscription, so test data and results stay with you. The system under test processes each test question where its vendor runs it, as it does in normal use, and the report records that vendor's published processing location with its date.

We show you where your data is processed.

This page describes a packaged offer, not a delivered engagement.

Independent Evaluation Report

Independent Evaluation Report

For
Agencies, regulated financial institutions and law firms with one AI system to test before it scales, such as a Copilot Studio agent leaving pilot or a contact-centre summary tool.
Scope
One system and one use case, the full method and the evidence bundle, with the harness handed over.
Included
  • A test plan with thresholds agreed before testing starts
  • For financial services, an independent validation of the use case: recall on complaint recognition, hardship-related complaints and vulnerability cues, each reported on its own, with every miss listed
  • The Independent Evaluation Report, with the failure list and findings rated to your risk matrix
  • Control-test scripts for your GRC, where you use one, in a format agreed at scoping
Every package includes
  • A test on your own examples, with thresholds agreed before testing and every failure shown, or, for AI Switch-On, the hours saved, measured against a baseline taken before anything changes.
  • Where the work has a test set, a re-run of it when the AI model changes or may have changed: by us on a system we build or run, and by your team, with the kit we hand over, after an evaluation.
  • A one-page data note: where your data sits, where the model runs, retention and deletion.
  • A runbook and a handover.
  • The delivery choice that fits the package, named in its region note.
  • Your time commitment, stated in hours and weeks.
Your time
A use-case owner, one or two people who know the cases for labelling, and read-only access in your environment. Hours are stated at scoping.
Timeline
3–5 weeks.
Out of scope
  • The decision to deploy: the official accountable for AI use makes it
  • Any system Techsider built, configured or advised on for you
Go/no-go gate
Once the test plan and thresholds are agreed, and before testing starts, you decide whether to continue.
Region note
The test runs in your own environment, and the graders run in your own subscription. The system under test runs where its vendor runs it, and the report records the vendor's published processing location, with its date.

We show you where your data is processed.

This page describes a packaged offer, not a delivered engagement.

Scribe & Clinical AI Evaluation

Scribe & Clinical AI Evaluation

For
Health services and GP groups choosing, rolling out or already using a digital scribe or other clinical AI product.
Scope
The health form of the Independent Evaluation Report: selection against the regulator and safety guidance that applies to you (in Australia: TGA, Ahpra and ACSQHC), local validation on your own specialties, omission audits and a monitoring plan for after go-live. We evaluate these products; we never build them.
Included
  • Tests on scripted consultations from your own specialties, with omissions counted
  • Ground truth from clinicians at your service, with their agreement measured
  • A scope test that flags any diagnosis, test or treatment in a note that the clinician never said
  • A monitoring plan after go-live, with the test set to re-run whenever the product may have changed
Every package includes
  • A test on your own examples, with thresholds agreed before testing and every failure shown, or, for AI Switch-On, the hours saved, measured against a baseline taken before anything changes.
  • Where the work has a test set, a re-run of it when the AI model changes or may have changed: by us on a system we build or run, and by your team, with the kit we hand over, after an evaluation.
  • A one-page data note: where your data sits, where the model runs, retention and deletion.
  • A runbook and a handover.
  • The delivery choice that fits the package, named in its region note.
  • Your time commitment, stated in hours and weeks.
Your time
A clinical lead to agree thresholds, clinicians' time to script and grade test consultations, and access to the product in a test environment. Hours are stated at scoping.
Timeline
3–5 weeks.
Out of scope
  • Building scribes, or anything that interprets clinical conversations
  • Clinical coding, discharge summaries and imaging AI, which we evaluate rather than build
  • The decision to use a product: your clinical governance makes it
Go/no-go gate
Once the scripted consultations and thresholds are agreed, and before testing starts, you decide whether to continue.
Region note
Testing runs on scripted consultations, in the product's test environment or your own. The product sends audio or text wherever its vendor processes it, and the report records that vendor's published processing location with its date.

We show you where your data is processed.

This page describes a packaged offer, not a delivered engagement.

Vendor AI Evaluation

Vendor AI Evaluation

For
Buyers comparing 1–3 vendor AI tools before contract, such as law firms weighing legal AI tools or agencies buying an AI-enabled upgrade.
Scope
1–3 vendor tools tested before you sign, side by side, on your own content and questions, in accounts your organisation controls.
Included
  • The same question set run through each tool, side by side
  • Every raw answer, graded, with unsupported citations listed
  • Findings on each tool, rated to your risk matrix
  • The test set, ready to re-run after contract
Every package includes
  • A test on your own examples, with thresholds agreed before testing and every failure shown, or, for AI Switch-On, the hours saved, measured against a baseline taken before anything changes.
  • Where the work has a test set, a re-run of it when the AI model changes or may have changed: by us on a system we build or run, and by your team, with the kit we hand over, after an evaluation.
  • A one-page data note: where your data sits, where the model runs, retention and deletion.
  • A runbook and a handover.
  • The delivery choice that fits the package, named in its region note.
  • Your time commitment, stated in hours and weeks.
Your time
People who know the work to write and grade questions, and a procurement or practice lead to agree thresholds. Hours are stated at scoping.
Timeline
2–4 weeks.
Out of scope
  • Holding vendor accounts ourselves: each tool runs in an account you control
  • Negotiating, buying or reselling licences
Go/no-go gate
Once the question set and thresholds are agreed, and before any tool is tested, you decide whether to continue.
Precondition
Each vendor gives your organisation trial or pilot access.
Region note
Each tool runs in an account your organisation controls, so its vendor's processing location applies; the report records each vendor's published processing location, with its date. The graders run in your own subscription.

We show you where your data is processed.

This page describes a packaged offer, not a delivered engagement.

Model-Change Regression

Model-Change Regression

For
Organisations that rely on vendor AI with no audit rights or notice of model changes in the contract, such as contact-centre AI at a regulated financial institution.
Scope
A test set on your own cases that your team re-runs whenever the model may have changed, with each run compared with the last. We set it up; your team runs it.
Included
  • A labelled test set on your own cases, with a baseline run on the current model
  • A replay script that runs in your own subscription
  • A comparison of each re-run with the baseline, with every change listed
  • Training for your second-line or data team to run it
Every package includes
  • A test on your own examples, with thresholds agreed before testing and every failure shown, or, for AI Switch-On, the hours saved, measured against a baseline taken before anything changes.
  • Where the work has a test set, a re-run of it when the AI model changes or may have changed: by us on a system we build or run, and by your team, with the kit we hand over, after an evaluation.
  • A one-page data note: where your data sits, where the model runs, retention and deletion.
  • A runbook and a handover.
  • The delivery choice that fits the package, named in its region note.
  • Your time commitment, stated in hours and weeks.
Your time
People who know the cases to label a sample, and a control owner to agree thresholds. Hours are stated at scoping.
Timeline
2–4 weeks to set up. Your team runs it after that, with support if you want it.
Out of scope
  • Running the re-tests for you as a standing control: your team owns and runs them
  • Routine re-tests that a platform's built-in evaluation already covers
Go/no-go gate
Once the test set and thresholds are agreed, and before the baseline run, you decide whether to continue.
Region note
The test set and the replay script live and run in your own subscription, so test data and results stay with you. The system under test processes each test question where its vendor runs it, and the report records the vendor's published processing location, with its date.

We show you where your data is processed.

This page describes a packaged offer, not a delivered engagement.

Tender AI-Claims Verification

Tender AI-Claims Verification

For
Government buyers assessing the AI claimed in tender responses before contract.
Scope
The government form of the Vendor AI Evaluation: the AI claimed in each tender response is tested on your own questions, with a probity note and sign-off from your procurement lead.
Included
  • A probity note agreed with your procurement lead before testing
  • The same questions run through each tendered tool, side by side
  • Findings on each claim tested, with every failure listed
  • The test set, ready to re-run after contract
Every package includes
  • A test on your own examples, with thresholds agreed before testing and every failure shown, or, for AI Switch-On, the hours saved, measured against a baseline taken before anything changes.
  • Where the work has a test set, a re-run of it when the AI model changes or may have changed: by us on a system we build or run, and by your team, with the kit we hand over, after an evaluation.
  • A one-page data note: where your data sits, where the model runs, retention and deletion.
  • A runbook and a handover.
  • The delivery choice that fits the package, named in its region note.
  • Your time commitment, stated in hours and weeks.
Your time
A procurement lead for probity sign-off, and subject-matter staff to write and grade questions. Hours are stated at scoping.
Timeline
2–4 weeks, inside your procurement timetable.
Out of scope
  • The procurement decision: your evaluation panel makes it
  • Advice on procurement rules
Go/no-go gate
Once the probity note, question set and thresholds are agreed, and before any tool is tested, you decide whether to continue.
Precondition
Your procurement lead signs off the probity note, and each tenderer gives access to its tool for the test.
Region note
Each tendered tool runs where its vendor runs it, and the report records that vendor's published processing location with its date. The graders run in your own subscription.

We show you where your data is processed.

This page describes a packaged offer, not a delivered engagement.

Agentic AI Control Evaluation

Agentic AI Control Evaluation

For
Agencies and financial-services entities with Copilot Studio or other agents leaving pilot.
Scope
A configuration review of your agents' identity, permissions, approval gates and logging, before they act in case or customer systems. It is not an attack. Adversarial testing is done by a CREST-accredited tester you engage.
Included
  • Each agent's identity, and the access it holds
  • The tools and permissions each agent can use, against what its job needs
  • Approval gates, the kill switch and logging, checked in your configuration
  • Findings rated to your risk matrix, mapped to the framework you report to
Every package includes
  • Where the work has a test set, a re-run of it when the AI model changes or may have changed: by us on a system we build or run, and by your team, with the kit we hand over, after an evaluation.
  • A one-page data note: where your data sits, where the model runs, retention and deletion.
  • A runbook and a handover.
  • The delivery choice that fits the package, named in its region note.
  • Your time commitment, stated in hours and weeks.
Your time
A platform owner to walk through the configuration, read-only admin access to the agent platform, and your risk matrix. Hours are stated at scoping.
Timeline
2–3 weeks.
Out of scope
  • Prompt-injection, tool-misuse or other adversarial testing, which a CREST-accredited tester you engage does under written rules of engagement
  • Changing your configuration: you get findings, and your platform team makes the changes
Go/no-go gate
Once the scope and your risk matrix are agreed, and before the review starts, you decide whether to continue.
Region note
The review reads configuration and logs in your environment with read-only access. The agent platform runs where its vendor runs it, and the report records the vendor's published processing location, with its date.

We show you where your data is processed.

This page describes a packaged offer, not a delivered engagement.

  • Australia On request SOCI AI Risk Evidence: Maps each AI use a utility or energy business runs to the relevant CIRMP rule items, and gives its risk team the data-flow maps, access lists and test reports behind each finding.

The program

Independent validation and control testing

Repeat evaluations across several AI systems, built into the validation and control testing you already run. It can be delivered as a workstream under your internal-audit co-source firm, law firm or prime.

Per program, phased by system

  • Validation and revalidation of each AI system, tiered to your model-risk framework
  • Control-test scripts for your GRC, in a format agreed at scoping
  • A harness your internal audit can re-perform
  • Harness training for your second-line, risk or data teams

How we test it

On this page the test is the product. Every engagement follows the same written method, with thresholds agreed before testing starts.

  • The sample size, with its rationale and confidence intervals
  • Ground truth, how it's built and by whom, and how often the raters agree
  • LLM-judge graders calibrated against human labels
  • Thresholds agreed before testing
  • False answers and false refusals reported separately
  • An explicit list of failures, not only a score

Read the evaluation method

Where it runs

  1. (a) In your own Microsoft or AWS account

    Runs in your own Microsoft or AWS account, in the cloud region you choose. Our team reaches it remotely from Sydney, under the data-transfer terms in your contract.

  2. (c) Inside a platform you already license

    Runs inside software you already pay for, such as Microsoft Copilot, Xero or Dext, where the vendor sets the processing location. Where we build part of it, such as a retrieval index, that part runs in your own account in the region you choose, or in ours in an Australian region. Our team reaches it remotely from Sydney.

Testing runs in your own environment, and the graders run in your own subscription, so test data and results stay with you. The system under test processes test questions where its vendor runs it, and the report records the vendor's published processing location, with its date.

The region note

  • Where inference actually runs, which can differ from where the platform is hosted.
  • The provider's retention, abuse-monitoring and human-review settings.
  • What happens when the newest model isn't available in your chosen region.
  • For a platform you already license: the vendor's published processing location, or "not published", with its source and an as-at date, re-checked quarterly.

Independence policy

  • We never issue an independent evaluation of a system we built, configured or advised on for the same client.
  • We take no resale margin or referral fees from vendors we evaluate, and we disclose any vendor relationship.
  • Acceptance tests on systems we build are labelled "acceptance test (not independent)".
  • Our sample report on our own demo system is always captioned as not independent.
  • We stay vendor-neutral: any vendor technical certification is disclosed, and we don't resell under a vendor partner tier.

What we don't do

Use your platform's AI first

If your AI runs as a Copilot Studio agent, use its built-in agent evaluation for routine re-tests first: you can re-run the same test set after each change. If Copilot Studio's built-in evaluation answers your question, we'll say so. Bring us in when you need an independent test on your own labelled cases.

  • We never do adversarial testing in-house. Where it's needed, it's done by a CREST-accredited tester you engage.
  • We don't sign off systems as safe to deploy. The report is findings, not an assurance opinion.
  • We don't write AI policy paperwork or build a GRC or register product. Findings go into the tools you already run.

Questions

How can we check your evaluation when you have no clients?
You shouldn't take it on trust. We have no clients yet, so you judge the method instead: how the sample size is chosen, how ground truth is built, how often raters agree, and thresholds agreed before testing. Every engagement hands over the test set and a replay script that runs in your subscription, so your team can re-run it and check our work.
Do you test systems built on large language models?
Yes. Assistants, copilots, AI scribes and document tools built on large language models are tested on your own questions in your own environment. Thresholds are agreed before testing, results come with confidence intervals, and every failure is listed, as our published evaluation method sets out.
Will your report tell us the system is safe to deploy?
No. You get findings, not an assurance opinion, and it isn't an ISAE 3000 engagement (ASAE 3000 in Australia). Each finding is rated to your risk matrix and mapped to the instrument you use. The official accountable for AI use decides whether to deploy, on measured evidence rather than a demo.
Do you evaluate systems you built, or take fees from vendors you evaluate?
Not independently, and no. We never issue an independent evaluation of a system we built, configured or advised on for the same client: our tests of our own builds are labelled "acceptance test (not independent)". We take no resale margin or referral fees from vendors we evaluate, and we disclose any vendor relationship.
Can we engage you through a firm we already use?
Yes. The Evaluation Partner route delivers the same evaluation as a subcontract workstream under your existing prime, internal-audit co-source firm, law firm or systems integrator. The scope, method and deliverables stay the same, and the independence policy still applies.
Does test data leave our environment?
We don't take a copy. The test runs in your environment, and the graders run in your own subscription, so test data and results stay with you. Our team reaches your environment remotely from Sydney, under the data-transfer terms in your contract. The system under test processes each test question where its vendor runs it, as in normal use, and the report records the vendor's published processing location, with its date. While your approvals run, testing can start on public or synthetic data.

Practical guides

Let’s put AI to work.

Talk to us about AI Evaluation