Local AI · Design exploration
Thinking through local AI for sensitive documents
My local-model experiments made me think about businesses that want useful answers from their documents without sending everything to a cloud model.
Trying LM Studio and AnythingLLM led me to a practical business question: could a team ask useful questions of its own documents while keeping the model and the documents on its own systems? I’ve explored the idea; I haven’t delivered a sensitive-data system for a client.
One approach is retrieval-augmented generation, usually shortened to RAG. The system finds relevant passages in a chosen document collection and gives them to the model as context for its answer. I’d start with a small, well-understood set of material and questions whose answers could be checked.
Running locally is one design choice. It doesn’t settle who should be able to read which documents, what is kept in logs, or whether an answer is accurate. Retrieval can expose information to the wrong person if the access rules are wrong, and retrieved text can contain instructions that try to mislead the assistant.
A useful next experiment would use fictional business documents, a few known answers and clear access boundaries. I’d want to check the source passages, deliberately ask questions outside the material and see whether the system admits when it doesn’t have an answer. That would be evidence to learn from before considering any real sensitive records.