Commissioning an AI knowledge assistant: access and quality

An internal knowledge assistant should improve access to information without exposing confidential material across departments. A defined knowledge area with reliable sources and accountable subject experts is a sensible starting point. Asking for a chat interface over every document is not a sufficient delivery brief. Specify the questions to be answered, user groups and the consequences of incorrect or incomplete answers.

Starting point and project boundaries

• Choose specific knowledge tasks such as internal process questions or technical product guidance; a general request for a company chatbot does not define scope.
• Distinguish approved knowledge sources from personal folders, historical drafts and documents without clear ownership.
• Identify user groups and existing document permissions before indexing content and making it available through a shared search.

In a retrieval architecture, the system selects relevant material and supplies it to the language model. Permissions must restrict that selection before generation. A prompt instruction is not access control. The design also needs rules for changed or deleted documents, source references, conflicting material and questions without adequate evidence. Document content is data; embedded instructions must not activate additional tools or permissions.

• For retrieval-assisted answers, assess search quality separately from answer quality: plausible wording may still rely on an unsuitable source.
• Filter retrieved sources using the requesting user's permissions.

An instruction in a prompt does not replace technical access control.

• Version the document state, preparation pipeline and evaluation.

Deleted or restricted material must also become unavailable in the search index.

An illustrative first use case is an assistant for internal product and process documentation. Employees receive source-backed answers within their permission scope. When evidence is insufficient, the application directs them to the responsible team. Confidential contract material can remain outside the initial scope. This example describes a possible project boundary and does not claim an existing GSWE deployment with these features.

Practical acceptance scenarios for the first release

• A question concerns confidential personnel guidance. A user without access to the source receives neither excerpts nor an indirect summary.
• Two approved instructions contradict each other. The assistant indicates uncertainty and makes its source selection traceable.
• A document contains an instruction to disclose internal information. It remains source material and does not gain permission to execute tools.

Before the pilot, agree a question set with expected sources and observable evaluation criteria. Include unanswerable questions, outdated documents, multiple languages and attempts to access another organisation’s information. Subject experts remain responsible for approving business answers.

Define a small set of business-reviewed questions before the pilot. Include answerable questions, deliberately unanswerable questions and requests for restricted documents. Assess source support and permission checks separately from language quality. Convincing wording is not evidence of a correct answer. Also agree who updates the evaluation set after document changes and reviews incorrect responses.

A knowledge assistant should make its limitations visible: missing sources, insufficient evidence and denied access belong in acceptance testing. Test the same question under different user roles. The response must not reveal information from documents that the current user could not read in the source system. Record the expected refusal behaviour as well as useful answers.

A checklist for commissioning the work

• Build a shared evaluation set including unanswerable questions, outdated information and different user permission levels.
• Agree when the assistant should abstain and how users can reach a human contact or inspect the original source.
• Request a maintenance process covering new documents, revoked permissions and business evaluation after model or retrieval changes.

For a comparable proposal

• Collect document sources, user groups and typical unanswered business questions.
• Define permission propagation, deletion and search-index updates.
• Test source references, missing answers and prohibited document matches across roles.

Value comes from reliable access to knowledge within a specific workflow. A convincing demonstration does not establish operational readiness. Procurement should explicitly cover data integration, permission checks, evaluation and ongoing support. This allows technical feasibility and organisational benefit to be assessed together.

Assess the benefit and control further expansion

• Measure time to a verified answer and expert correction effort, rather than only counting the answers generated.
• Start with clearly owned knowledge; additional sources help only when their quality and permissions can be maintained.

Changing the model alone does not repair an unsuitable knowledge base.

Collect internal questions employees should be able to answer from existing documents. GSWE reviews sources and permissions as the basis for a focused knowledge-assistant engagement.

Commission an internal knowledge assistant: Knowledge & Retrieval Platform.