Internal AI Knowledge Assistant

Question

Can I paste a difficult stakeholder’s email chain into a public AI tool to help draft my reply?

Answer

No. The Guidance gives this case as an example of inappropriate use: “The information you provided to the Generative AI tool is sensitive and not publicly available. The tool may inappropriately share this information with other users.”[Source 2] For a tool your organisation hasn’t approved, the guideline’s rule is: “Only publicly-available information can be shared with, input or uploaded into Generative AI tools that have not been approved for use by the in-scope organisation.”[Source 1] An approved tool has a limit too: “Information shared with, input or uploaded into an agency-approved Generative AI tool must not exceed the protective marking determined by the in-scope organisation as appropriate for that tool.”[Source 1]

1. Retrieve hybrid (BM25 + dense), k=5, reciprocal-rank fusion 37ms
2. Rerank cross-encoder, keep top 2 21ms
3. Generate grounded prompt, quote and cite 1180ms

Question

If AI helps me write a briefing and gets a fact wrong, who is responsible?

Answer

You are. The guideline says: “In-scope organisations’ Personnel remain responsible and accountable for the accuracy and quality of their work, including advice, decisions and content created.”[Source 3] It also warns: “Relying on Generative AI content to support decision-making without verification of its accuracy could be a breach of the Codes of Conduct for Victorian Public Sector Employees and this guideline.”[Source 4] What the Guidance asks you to do: “Fact-check information against verifiable sources.”[Source 5]

1. Retrieve hybrid (BM25 + dense), k=5, reciprocal-rank fusion 39ms
2. Rerank cross-encoder, keep top 3 23ms
3. Generate grounded prompt, quote and cite 1260ms

Question

Is Microsoft 365 Copilot approved for use in my department?

False answer, caught by the acceptance test (not independent)

Yes, it’s approved. The guideline lists it: “Copilot (including Microsoft 365, Edge and GitHub)”[Source 6].

What the test caught: An acceptance test marks this answer false. The cited passage lists well-known tools as examples and approves none of them: each organisation approves its own tools. The expected answer says the documents don’t state it, and points to the agency’s own approved list.

1. Retrieve hybrid (BM25 + dense), k=5, reciprocal-rank fusion 36ms
2. Rerank cross-encoder, keep top 2 20ms
3. Generate grounded prompt, quote and cite 640ms

Question

I work at a local council. Does this guideline apply to me?

Refused: the answer isn't in these documents

These documents don’t say whether councils are covered, so I won’t guess. The scope clause reads: “It applies to all public service bodies and public entities as defined in the Public Administration Act 2004 (in-scope organisations).”[Source 8] Whether a council is one of those bodies is a question about that Act, which isn’t in these documents. Escalating to a person, with the scope clause attached.

1. Retrieve hybrid (BM25 + dense), k=5 35ms
2. Abstain no passage says whether councils are covered, so quote the scope clause and stop 2ms

Question

What are our incident-notification timeframes under CPS 230?

Answer

CPS 230 sets two notification clocks, plus a separate rule for service providers. An operational risk incident must be notified to APRA as soon as possible, and no later than 72 hours after you become aware of it, where you determine it is likely to have a material financial impact or a material impact on your ability to maintain critical operations[Source 1]. A disruption to a critical operation outside tolerance has a shorter clock: notify APRA as soon as possible, and within 24 hours, covering the nature of the disruption, the action taken, the likely impact and the timeframe for returning to normal operations[Source 2]. “Critical operations” means processes that, if disrupted beyond tolerance levels, would have a material adverse impact on your depositors, policyholders, beneficiaries or other customers, or on your role in the financial system[Source 3]. Separately, service-provider arrangements have their own notice: as soon as possible and within 20 business days of entering into or materially changing an agreement for a service a critical operation relies on[Source 4].

1. Retrieve hybrid (BM25 + dense), k=5, reciprocal-rank fusion 41ms
2. Rerank cross-encoder, keep top 4 24ms
3. Generate grounded prompt, cite-as-you-write 1420ms

Question

And what’s the financial penalty if we miss the 72-hour deadline?

Refused: the answer isn't in these documents

That isn’t stated in the provided documents. CPS 230 sets the 72-hour notification obligation, but the excerpts here don’t specify any financial penalty for late notification, so I won’t guess. Escalating to a person, with the relevant clause attached.

1. Retrieve hybrid (BM25 + dense), k=4 35ms
2. Abstain top score below the grounding threshold, so refuse 2ms