When a team considers an assistant that answers questions about regulations, the first problem is choosing the right document. OpenAI's retrieval guide explains semantic search, which retrieves passages related by meaning, with the option of filtering results by file attributes. These tools provide a technical foundation, but whether a document can legitimately be relied on needs rules set by subject-matter specialists.
A record card for each document before indexing
My suggestion is that the project begin with a register entry for each document: its title, issuing body, version number, effective date, status, and who is entitled to see it. Its relationship to earlier documents should also be made clear, since one document may revoke another or amend only part of it. These relationships cannot be inferred from word similarity alone.
Scanned files also need a check on the quality of text extraction. An error in an article number or a unit of measurement can change the meaning of an answer. That is why I would keep the page image and the passage's location alongside the text, so a reviewer can go back to the original when the machine reading is unclear.
The answer I want to see
In a hypothetical example, an employee asks which documents are required for a specific type of application. The assistant gives a brief answer, then links each requirement to its source, location, and version. If it finds a conflict, or the category of the application has not been determined, it asks for clarification or refers the matter for review instead of assembling an answer that merely looks complete.
User permissions should be applied before passages reach the model. Hiding the file name after the model has already read the contents does not achieve this. Reference content is likewise kept separate from system instructions, because text inside an attachment does not give the assistant authority to carry out an action or disclose information.
A trial that measures both retrieval and the answer
- Was the right document retrieved, in the version appropriate to the question?
- Does the cited passage actually support the sentence that follows it?
- Did the answer stay within the user's permissions?
- Did the system state that information was missing when no suitable source was available?
These questions cannot be reduced to the accuracy of the wording. A system may reach the correct source and then summarise it in a way that loses an exception, or produce well-written text based on an outdated version. Keeping the two tests separate helps show which part needs improvement.
This approach starts to pay off when the tool cuts search time while keeping the evidence visible, so that the specialist can examine the answer, discuss it, and record their decision on a known basis.