A local document search can look good on a few hand-picked queries and still miss the passages your readers need. To evaluate a search built with Ollama embeddings, label a small set of queries and expected passages, run the same set against two configurations, and inspect where each configuration fails. Measure retrieval separately from the answers generated from those results. A model can write a convincing answer from the wrong passages, and even an excellent answer generator cannot quote a passage that the search never retrieved.

Define what a useful result looks like

Choose a collection with a clear purpose, such as internal how-to guides or public product manuals saved locally. Give each document and passage a stable identifier. For every test query, mark one or more passages that contain enough information to answer it. A page that mentions the right product without explaining the requested step is related but may be insufficient. Record that distinction in your notes before you look at any results.

If you are still assembling the pipeline, start with our guide to local document search with Ollama embeddings. Ollama generates the embeddings. Your retrieval code or vector store determines how to compare and rank them. The Ollama API reference documents the embedding endpoint. Log the model, input preparation, chunk size, overlap and ranking settings. Knowing which configuration produced a result makes the test worth repeating.

Build a query set that reflects real searches

Write queries in the language someone would use before they know the answer. Include paraphrases, exact identifiers, questions whose answers appear in several documents and cases where your collection has no answer. For example, a maintenance manual might say “restore factory defaults” while a user asks “how do I revert my custom settings?” A query about error E117 is a different test. It rewards exact matching and may expose a weakness that paraphrases never reveal.

Start with a few dozen queries if the collection is small. Save the query text plus the expected document and passage identifiers in a CSV or another format your evaluation script can read. Include a short reason for each judgment so you can review it later. Pull examples from support requests or actual search logs if you can use those appropriately. Alternatively, ask someone familiar with the collection to write questions without looking at your index or chunk boundaries.

Reserve some queries for a final check after you choose your settings. An improvement on the same ten prompts you used to tune the index can reflect those particular prompts. A small set held back from tuning gives you another opportunity to see whether the change works on different examples. Keep difficult queries in the set even when neither configuration handles them well. They tell you what the headline score leaves out.

Compare hit rate and rank

For answerable queries, hit rate at k is the fraction with at least one relevant result among the first k results. If eight of ten answerable queries have a relevant passage in the top five, hit rate at five is 80 percent. Pick a cutoff that matches your product. A reading interface that shows ten results and a question-answering pipeline that sends three passages to a model care about different positions. State the cutoff with the score so readers can interpret it.

Mean reciprocal rank gives the first useful result more credit when it appears near the top. A first relevant result at rank one is worth 1, at rank two it is worth 0.5, and at rank four it is worth 0.25. Use zero when no relevant result is returned within your evaluated range, then average across the answerable queries. This measure can reveal an improvement when both systems find a result but one places it closer to where users start reading.

For questions with no supported answer, keep a separate count of cases your application incorrectly treats as answered. Similarity search can still return its nearest passages when none contains the answer. Inspect those passages and decide how your product should handle them. It might show source previews or decline to generate an answer when the available evidence is insufficient. Rerun those cases alongside the answerable set so you can see how both groups respond to a configuration change.

Compare one change at a time

Start with a baseline and change one variable, such as the embedding model, chunk size or search method. When switching embedding models, rebuild the document embeddings and use that model for queries as well. Keep the collection, judgments, cutoff and evaluation scripts fixed across both runs. If you change chunk boundaries, judge whether the returned text covers the same labeled source passage. The previous chunk IDs may no longer exist.

Inspect a handful of wins and losses for each configuration. A relevant heading returned without the following instructions suggests a chunk boundary issue. A failed error-code query may benefit from an exact or lexical search component. Duplicate material in the top results can crowd out other useful passages, making deduplication worth testing. Write down which class of query improved and which regressed. If your project connects multiple clients to Ollama, our OpenAI-compatible client guide covers integration settings you can record separately from retrieval settings.

Store ranked passage identifiers alongside scores. A week later, you can compare a rebuilt index with those runs and see if a failure appeared when a source changed or when rankings changed. For each run, record the collection version and how long indexing and retrieval took on the machine that runs them. If two configurations are similar on relevance, installation size, indexing time and query latency can help you choose based on how readers will use the system.

Review a failed query from start to finish

Suppose your test query asks how to recover a project after an interrupted export. The index returns several passages about exporting a project, all of which explain the menu and output options. The passage you labeled explains where to find a temporary copy after failure. First confirm that the passage actually entered the index and that the extracted text contains the recovery instructions. If the text was dropped when a PDF was parsed, changing the ranking model cannot recover it.

If the text is present, inspect its neighbors and the competing results. The winning passages may all repeat the same introductory paragraph. Compare a version that removes duplicates with the original and rerun the test set. If the recovery instructions move up, check whether deduplication also removed genuinely different passages for other queries. Record that result with the example so another person can understand why you kept or reverted the change.

Frequently asked questions

How many queries do I need?

Start with enough examples to cover the main kinds of searches your users make. A few dozen carefully labeled queries can reveal obvious weaknesses. Add examples as real failures appear and keep a set aside for evaluating larger changes.

Should I judge generated answers in the same test?

Record answer quality separately. First measure whether retrieval provided the evidence. Then check whether the answer used it accurately and pointed to the right source. That separation makes it easier to tell which part needs fixing.

Can I compare raw similarity scores from different embedding models?

Treat scores as specific to the model, inputs and similarity function. To compare models, use common relevance judgments and rank-based metrics. Calibrate any threshold again when you change the index or model.

What if many passages answer the same query?

Label each passage that independently supports a useful answer. If a question needs several facts from different passages, record which facts each result covers and inspect how much evidence the returned set contains in addition to the position of the first hit.