Maintenance teams often need to locate the right instruction among manuals, service bulletins, internal procedures and past work records. A language-model assistant can be considered for this knowledge workflow, but a fluent answer is not enough. It must retrieve suitable material, respect access permissions and make uncertainty visible. Retrieval-augmented generation (RAG) and fine-tuning address different parts of that challenge.
1. Start with a narrow knowledge task
Define what the assistant should help someone find. An example is locating the approved inspection procedure for a specific equipment model and identifying the relevant document section. This is more testable than promising an assistant that “knows the entire plant”.
Describe the expected response format: a short explanation, linked supporting passages and a clear request for missing equipment information. Decide which questions should be escalated to an authorised specialist. When model or revision details are ambiguous, asking for clarification is preferable to combining instructions from different machines.
Keep the first workflow separate from work-order approval and physical operation. Finding information does not authorise its use on equipment.
2. Understand RAG and fine-tuning
RAG retrieves selected information when a question is processed and supplies it as context to the language model. Fine-tuning changes a model using training examples, for purposes such as a more consistent task behaviour or response style. Microsoft’s Foundry RAG guidance explains this distinction.
For a changing library of maintenance documents, retrieval is often the first approach worth evaluating. Updating the library should not require treating every revised procedure as a model-training project. Fine-tuning can be assessed separately when there is a demonstrated behaviour requirement and suitable examples.
The approaches can be combined, but neither guarantees correct answers. Fine-tuning is not a substitute for checking current source material, and retrieving a document does not ensure the generated response follows it accurately.
3. Build a controlled, equipment-aware library
Inventory the documents before indexing them. Record owner, approval status, revision, applicable equipment and access group. Identify superseded procedures and contradictory instructions, and decide how those documents should be excluded or clearly labelled.
Document preparation needs practical checking. Tables, diagrams and scanned pages can lose meaning during extraction. Ensure an instruction is not separated from its warning, prerequisites or applicability notes. Preserve links back to the actual source and the location supporting the answer.
Microsoft’s RAG search overview discusses retrieval approaches including keyword and vector search. Test retrieval using the terms your team actually uses: exact part identifiers, abbreviations, equipment names and natural-language questions. A plausible semantic match may still be the wrong equipment revision.
4. Evaluate evidence before polished wording
Build a small evaluation set with maintenance specialists. Include answerable questions, missing information, conflicting documents and questions outside the library. Review the retrieved passages separately from the final answer so that retrieval failures are not hidden by confident language.
- Does the system retrieve the approved procedure for the correct equipment and revision?
- Does each citation support the specific claim beside it?
- Does it preserve relevant warnings and state what remains uncertain?
- Does it decline unsupported requests rather than invent settings or instructions?
- Are documents inaccessible to a user also excluded from that user’s retrieval results?
- Do document changes and permission removals take effect as intended?
Record response time and reviewer effort alongside answer quality. Keep difficult examples for regression testing when the model, prompts, search settings or document library change.
5. Separate knowledge assistance from control
Retrieved text must be treated as information, not as authority to change the application’s rules. A malicious or compromised document can contain instructions aimed at the model. The OWASP prompt-injection guidance explains this risk and the need for layered safeguards.
Enforce permissions in the application and retrieval system, not only in a prompt asking the assistant to behave. Limit available tools and data to the defined task. Test how the workflow handles suspicious source content without exposing confidential information.
A maintenance knowledge assistant should not have an implicit route to write PLC or SCADA values, bypass interlocks or authorise hazardous work. Real-time control belongs within the separately engineered and approved control system. Keep qualified human review and site procedures responsible for consequential actions.
6. Pilot for a useful, bounded outcome
Common pitfalls are indexing every available file, trusting citations without checking them and fine-tuning before identifying the actual failure in the workflow. Begin with a curated document set and one user group, then expand only after evaluation.
A realistic pilot outcome is easier access to selected approved knowledge with traceable answers and visible limitations. Skymics provides custom LLM fine-tuning and RAG pipelines alongside AI consultancy and integration services. Discuss your maintenance knowledge workflow without sending confidential documents through the initial enquiry form.