Vendor AI Evaluation
1–3 vendor tools tested before you sign, side by side, on your own content and questions, in accounts your organisation controls.
industry
AI for legal and professional services.
Evaluate legal AI and prepare work for human review.
Rules that travel
Courts and professional bodies in many countries have reached the same few rules for generative AI. The Council of Bars and Law Societies of Europe published its guide on generative AI for lawyers in 2025. Its 2026 technical guide says lawyers must understand how the tools they use work, and explains hosting options so they can choose tools that protect client confidentiality. Courts and bar associations elsewhere have published guidance along the same lines. The EU AI Act's high-risk category for the courts covers AI that helps a court research and interpret facts and law, not a law firm's own research tools. We test tools on your own matters and keep a record of every check. Your own court's practice notes and conduct rules always come first.
- ISO/IEC 42001: AI management system Standard · published Dec 2023 · as at Oct 2026
- NIST AI RMF and the GenAI Profile Guidance · published Jan 2023 · as at Oct 2026
- OECD AI Principles Guidance · published May 2024 · as at Oct 2026
- EU AI Act: reaches AI output used in the EU Law · in force · as at Oct 2026
Which rules apply to you is your compliance or legal team's call. We map our evidence to them; we don't certify against them.
The problem
In Victoria, Australia, lawyers reach for general-purpose AI and check each answer by hand, often in workplaces with no AI guidelines.
Your firm chooses its AI on a test of your own matters, and keeps a record of every check on court work.
VLSB+C, Generative AI Use in the Legal Profession (2025 Victorian Lawyer Census), as at
How the work flows
Choose
Vendor AI Evaluation before you sign
Switch on
Admin Hours Audit + Switch-On
Private Workspace
Only for firms with no suitable licence.
Court work
Verification and AI-use record
Keep it working
Model-Change Regression when a vendor changes its model
- Question
- Which clause sets the rent review date?
- Tool A
- Clause 4.2, linked to the page
- Tool B
- Clause 7.1, the make-good clause
- ok: Tool A: correct, citation found
- blocked: Tool B: wrong clause cited
Sources
- Lease, de-identified Clause 4.2
What we build for legal & professional
Lead solutions
Independent Evaluation Report
One system and one use case, the full method and the evidence bundle, with the harness handed over.
Admin Hours Audit + Switch-On
The Admin Hours Audit takes 1 week and maps where staff hours go. Switch-On takes 2–3 weeks: it configures features you already own; for firms in Australia, it applies the relevant Safe-Use Kit as published, where one has been published; and it trains staff and measures the hours saved at day 30.
Ask the Firm Manual
Answers staff questions at larger accounting and law firms from the firm manual, precedents and procedures, not matter files, and shows the page it used.
Use your platform's AI first
If your firm runs LEAP, use Matter AI and LawY first; both come with every LEAP subscription. If you license Ask iManage, try its repository search before asking us to build.
What we don't do
- Rebuild matter Q&A, legal research or drafting inside Word. Your practice and research tools already do these.
- Build KYC, ID verification or AML/CTF program software.
- Draft or tailor legal wording. Your own adviser approves any policy, consent or engagement-letter text.
Illustrative scenario: not a client engagement
Two research tools, one renewal date.
Problem
A fictional commercial law firm must choose between two legal AI tools before its research subscription renews.
Approach
Senior lawyers pick questions from closed matters with known answers. Both tools answer them in the firm's own trial accounts, and two lawyers grade each answer separately.
How we'd measure
- Wrong answers by task type, for each tool
- Citations that don't exist or don't support the proposition
- Agreement between the two graders
What ships first
The question set, with agreed answers and pass thresholds, before either tool runs.
- 14:00:00 tools.ask Same question put to Tool A and Tool B
- 14:00:05 cite.check Tool B's cited clause checked against the lease
- 14:00:07 grade.queue Both answers sent to two graders
A first engagement in legal & professional
What we need from you
- One decision to test, and a partner who agrees the pass thresholds first.
- Questions from closed matters whose answers your lawyers know.
- Access to each tool, in accounts your firm controls.
What you get
- A side-by-side report listing every wrong answer and unsupported citation.
- A data note per tool, with its vendor's published storage and processing locations and their dates.
- The question set and a replay script for the next model change.
The exit ramp
Once the questions and thresholds are agreed, you decide whether to continue. The question set is yours either way.
Australia: our detailed map
Australia is our home market and the one we know best.
Designed around
Works alongside
- iManage Work (iManage Cloud) Read-only access
- Microsoft SharePoint Read-only access
- Smokeball Read-only access
- Harvey (history export) File/CSV import
Regulatory map
| Obligation | What it means for an AI system | How we design for it | Evidence you get | Source |
|---|---|---|---|---|
| Joint AI statement (6 Dec 2024), ASCR r 9.1: keep client confidences out of public AI | The NSW, Victorian and WA regulators say confidential, sensitive or privileged client information can't safely go into public chatbots or copilots. If a commercial AI tool is used with any client information, the firm needs to review its contract terms to check that information is kept secure. | Before any matter material is used, we record each vendor's published storage and processing locations, with their dates, and whether the tool keeps or trains on inputs. | A data note per tool on storage, processing, retention and training | Source, as at 6 December 2024 |
| NSW SC Gen 23 ¶9A: suppressed, Harman or subpoena material only in closed Gen AI | Material under a suppression or non-publication order, the Harman undertaking, a subpoena or a statutory publication ban can't go into any Gen AI program unless the responsible lawyer is satisfied that it stays in a controlled environment under supplier confidentiality terms, is used only for that proceeding, and isn't used to train any model. | For each tool, the data note records whether inputs stay in a controlled environment and out of training, or says we couldn't confirm it. | A data note per tool on its environment and training settings | Source, as at 28 January 2025 |
| NSW SC Gen 23 ¶16–17: verify every citation in submissions, not solely with Gen AI | If Gen AI helped prepare written submissions, the author must verify in the submissions that every citation and reference exists, is accurate and is relevant, and that check can't be done solely with a Gen AI tool. Gen AI must not generate the content of affidavits or witness statements (¶10–13). | The log records each authority, the source a person checked it against, who checked it and when. | A citation-check log per document | Source, as at 28 January 2025 |
| Vic Supreme Court SC Gen 25 ¶7.1–7.6: verify AI content; explain how it was verified | AI-produced content must be verified under meaningful human control, and asking one AI tool to confirm another's output doesn't count. Court users must be ready to identify which parts of a document were produced with AI and explain how each was verified. | The record marks which sections came from AI and how each was checked, so the lawyer can answer the Court from the file. | A per-document record of AI-produced sections and their checks | Source, as at 14 May 2026 |
Last reviewed
General information, not legal advice.
Related insights
Australia Ask an AI vendor for evidence you can check.
Questions to ask before you buy or renew an AI product, drawn from APRA, the DTA, NSW buying guidance and the OAIC, and what a good vendor answer looks like.
Australia Copilot is only as safe as your settings.
Where Microsoft 365 Copilot stores and processes Australian prompts, what goes to Bing, and what lawyers, tax practitioners and clinicians should check first.
Australia Measure Copilot before the licences go out.
Time 3–5 recurring tasks and their error rates before Copilot licences go out, then compare on a set date. What the government trial and Microsoft reports miss.
Questions
How can we check your evaluation when you have no clients yet?
Don't take it on trust: we have no clients yet. Your lawyers write and grade the questions, and you can re-run the test without us.
Does testing mean our client files leave the firm?
Only to the tools under test; we don't take a copy. Tests run in tool accounts your firm controls, on cleared matters or de-identified copies. Our team reaches those accounts remotely from Sydney, under the data-transfer terms in your contract. A data note first records each vendor's published storage and processing locations, with their dates. Whether a client must consent is your call.
Are you paid by legal AI vendors?
No. We take no resale margin or referral fees from vendors we evaluate, and disclose any vendor relationship. We never issue an independent evaluation of a system we built, configured or advised on for the same firm.
Does your verification record meet your court's practice notes?
No. The record is a log, not a safeguard. The responsible lawyer still verifies each authority; the record shows who checked it, against which source, and when.
Is this page only for law firms?
Mostly. Consultancies can commission an independent evaluation of AI they deliver to clients, or use the Evaluation Partner route under their own contract.
Let’s discuss your workflow.
Talk to us about AI for legal & professional