Start with a few plain-text documents. Split them into passages, generate an embedding for each passage with Ollama, and embed each search query with the same model. Rank passages by cosine similarity and show the matching text with its filename. That gives you a working retrieval test before you add a chatbot or a vector database.
Decide what the search should retrieve
A document search tool is most useful when it can point you to the paragraph that answers your question. Start with a small collection that contains some overlapping topics, such as several notes about the same software or a set of meeting summaries. Write a few queries that use different phrasing from the documents. For example, a note might say weekly snapshots while your query asks how often a backup is taken. Put a distinctive project identifier into another query so you can compare how semantic matching handles a case that exact text search should solve easily.
List the source document and passage you expect for each test question before indexing the collection. Include one query whose answer is absent and one where the latest and previous versions say different things. This set helps you detect a system that returns convincing but stale passages. If you are new to running models on your machine, begin with our guide to running AI locally. Document retrieval adds a separate step, and it is easier to diagnose when the local runtime already works.
Prepare passages and keep their source details
Begin with plain text or Markdown. For each document, store a stable identifier, its path, the section heading if available and the passage text. Preserve a short excerpt as it appeared in the source. You can normalize excessive whitespace for indexing while retaining a copy that shows the original formatting. For a PDF, first confirm the extracted text is in reading order. Two-column layouts, footers and scanned pages can produce misleading passages even when the similarity calculation works correctly.
Split along headings and paragraphs before choosing a fixed size. Each passage should retain enough context to identify the subject of any instructions it contains. A sentence about selecting a profile has little value if the name of the application appeared in another chunk. If you split a long section, repeat its heading or retain a small overlap when necessary. Excessive overlap can fill the results with nearly identical passages. Our long-document guide explains how to find meaningful boundaries in larger inputs.
Generate embeddings with the Ollama API
Choose a dedicated embedding model. Ollama’s embeddings documentation describes model choices, including embeddinggemma. Download your chosen model with ollama pull embeddinggemma if you use that example, and make sure Ollama is running locally. Use a request to http://localhost:11434/api/embed with a JSON body containing model and input. For example, send {"model":"embeddinggemma","input":["Backups run every Friday.","The display uses an HDMI cable."]}. The current embed endpoint reference documents accepting a string or array and returning vectors in the embeddings field.
Pair each returned vector with the corresponding passage and keep the model name alongside the index. Start with small batches so errors are easy to trace. If a long passage exceeds the input limit, split it at a useful boundary. The API documents a truncate option, so consider setting it to false when you want oversized input to produce an error you can investigate. Silent truncation can omit the exact sentence you later expect to retrieve. When testing a new model, rebuild the document vectors and use that model for queries as well.
Rank passages and show the evidence
Embed the user’s question using the same model and compare its vector with the stored vectors. Cosine similarity compares their direction. To calculate it, take the dot product and divide by the product of the two vector lengths. Skip invalid vectors and verify that both vectors have the same dimension. Sort the scores from highest to lowest and show a handful of passages, with source names and headings. For a modest prototype, an in-memory list is enough to test this approach before choosing a storage service.
If a query contains a model number, filename or error code, compare semantic retrieval with a simple text match. The exact string can be a stronger clue in those cases. You can show both sets of results or use exact matches to reorder plausible semantic candidates. A numerical similarity score by itself does not establish that a passage answers the question. Inspect several top results and test a query for which you know the collection has no answer before relying on any score cutoff.
Evaluate retrieval before adding generated answers
Use the query list you wrote before building the index. Check whether the expected answer appears in the first three or five results and record the first useful rank. When a good answer is absent, see whether the source passage was extracted correctly and indexed under the right heading. Try a smaller or larger passage size on the same collection. Change one variable at a time so you can identify what helped. Our local AI benchmarking guide covers setting up repeatable comparisons.
If you later ask a language model to write an answer, pass only the passages you selected, keep the source references visible and verify that the answer reflects them. A retrieval error can reach the user through polished text if nobody checks the underlying evidence. Keep the raw search view available for debugging. It allows you to tell whether the system missed the right source or the answer model misread a passage that had already been retrieved.
Keep the index current and local
Store a content hash or another reliable change marker for each indexed file. Rebuild passages for changed documents and remove their old entries, including entries for deleted files. Test by editing a harmless example, reindexing, and searching for both its old and new wording. If the old version still appears when it should have been removed, inspect the update logic before relying on the results for time-sensitive material.
Treat the local index and the passage text as part of the document collection. Set its file permissions accordingly and consider whether it belongs in your backups. Download models before trying offline use and test with the network disconnected if offline behavior is a requirement. Keep the retrieval endpoint local unless you have deliberately designed how other devices should connect. A local embedding workflow can still include external pieces if extraction, document storage or the final answer model uses a hosted service.
Frequently asked questions
Do I need a vector database?
For a small prototype, compare the query against a list of stored vectors. Revisit the storage design if indexing or search latency becomes a measurable issue.
Can I mix models?
Use the same embedding model for both the indexed passages and the queries. Rebuild the index when you switch models.
Can the embedding tell me what a document says?
It helps rank passages by similarity. Read the returned passage and verify its source when you need the actual answer.
