A company knowledge assistant can sound convincing while citing the wrong policy. Evaluation should therefore ask whether the answer is useful, grounded, current, and visible to the right person. Fluency is only one small part of that assessment.
Build a question set from actual work
Collect questions employees ask repeatedly, then add cases where the right response is uncertainty. Include an outdated policy, two documents that disagree, an ambiguous product name, and a question the library cannot answer.
Keep some questions out of the prompt-tuning process. Otherwise, repeated improvements may fit the demonstration set without improving performance on new work. Have the process owner record what a satisfactory answer needs to contain.
Inspect retrieval separately from writing
If the wrong document is retrieved, editing the tone of the final response will not solve the problem. Check whether the source collection is clean, the document sections retain context, and the search respects access permissions.
For an illustrative service policy question, the answer should point to the actual policy section and state any conditions. If the source only describes standard returns, the assistant should not invent an exception for a frustrated customer.
Make unsupported answers visible
A useful refusal explains what is missing and where the employee can go next. “I could not find an approved exception policy; ask the service manager” is actionable. A generic error or a confident guess is not.
Track source correctness, factual completeness, unauthorized disclosure, and employee correction effort separately. A high average score should not hide a serious access failure. Define those stop conditions before the pilot starts.
Agree on maintenance
Someone must own document retirement, new versions, and corrections reported by users. Include the update path in acceptance: replace a policy, ask the same question again, and verify that the old version no longer drives the answer.
A personal workflow with selected documents can be explored through Golub Bot. A shared knowledge service with role-based access and repository synchronization needs its own integration scope. Talk to Golub about the questions, sources, and boundaries your team actually needs.