To compare two local AI models, give them the same small set of tasks and score the outputs against answers you prepared in advance. Measure speed separately from accuracy. A model that responds quickly and invents a deadline may be worse for meeting notes than a slower one that consistently extracts the correct dates. Pick the model that performs the work you actually need.
Build a small test set from real work
Start with three tasks you repeat often. A student might summarize notes, extract dates from a handout, and rewrite an email. A developer might explain a short function, identify an input edge case, and reformat a list into JSON. Choose tasks whose outputs you can check. Use sample material without personal or sensitive details while testing an unfamiliar interface.
Give each task at least two examples. For extraction, include a clean example and one with a missing field. For summarization, use a short source and a longer source with one tempting but irrelevant detail. For formatting, include punctuation or a value that needs escaping. A model that does well on tidy examples may behave differently on the exceptions you see in practice.
Write the expected answer before running either model. A summary can have several acceptable phrasings, so list facts that must appear and facts the model must avoid inventing. For a structured output, list the expected keys and what to return when a value is unknown. Save the examples in a simple file or sheet, with a short identifier you can use to match outputs to prompts later.
Keep model comparisons reproducible
Record the exact model tags, the application, and the runtime version. Two variants from the same model family can differ in size or quantization. If you have only one installed, follow our guide to running local AI with Ollama to start with a small model and a repeatable first run. Keep the same chat interface for both models unless you specifically want to compare interfaces.
Start a fresh conversation for every example. Paste the same instructions and source material and request the same output length. If the client exposes generation options, record them and keep comparable settings where possible. An earlier message can quietly supply an answer or contradict the current instructions. A fresh chat gives both models the same starting point.
Check that your material actually fits. Use our Ollama context guide to verify the configured window and avoid comparing a truncated input with a complete one. If long documents are essential, run a short baseline and then a separate long-document test. Record which examples are affected by the context setting and whether the app is pasting text or retrieving selected passages.
Score accuracy and usability before timing
For a simple extraction task, give one point for each required fact that appears correctly. Count an incorrect or invented fact separately. A model that returns four correct fields and one invented field needs a different review from a model that honestly leaves one field blank. If exact formatting matters, validate the structure separately. For JSON, try parsing the output and check whether the expected keys and value types are present.
For a summary, read the source beside the response and mark unsupported statements. Look for errors that matter to the decision, such as assigning an action to the wrong person or reporting an estimate as a confirmed number. A polished paragraph can hide those mistakes. If useful, remove the model names from the outputs while you score them so that your preference for one model has less influence.
Repeat difficult examples at least once. If a model gives a correct answer on one run and a confident wrong answer on the next, treat that variation as part of the result. Record whether you could catch the failure easily and how much time checking would add. A model can be suitable for drafting while still needing too much supervision for extracting data that goes into another tool.
Measure speed for the workflow you will keep
Use a stopwatch or runtime timing data if available. Record how long it takes to see the first output and how long the full response takes. Run both models with a similar requested answer length. A long answer can take more time simply because there is more text to generate. If you have access to the API, see the Ollama generate reference for the timing fields the endpoint returns.
Compare a first run after loading with a repeat run while the model is already active. The first run reflects what you might experience opening an occasional tool. The repeat run better reflects a session with several prompts. If one model uses more memory or changes the processor split, note that alongside the timing. Our RAM and VRAM guide shows how to check that behavior during a run.
Decide based on the cost of an error in each task. Use a fast model for a low-stakes rewrite if its output is easy to check. For extracting dates into a tracker, prefer the model that returns accurate values and handles missing information correctly. Keep the test examples and rerun them after you change the model, application, or context settings. Add a new test whenever you find a failure that matters to your work.
Keep a copy of raw outputs along with the scores. If you change the scoring rules later, you can review the original examples without rerunning the models. Note the cases that would have changed a real decision. Those examples make a more useful regression test than a larger pile of prompts with no expected answers. If a model update changes the results, rerun the affected example on both versions where possible before attributing the change to the runtime or the model.
For an example scorecard, imagine a task with five required fields and one field that should be unknown. One model returns four correct values, one blank required field, and an invented value for the unknown field. Record four correct, one missing, and one unsupported. Another model might return only three correct values but leave all uncertain fields blank. Which output is easier to use depends on whether you can tolerate a missing value or whether a wrong value could trigger work downstream. Write that decision rule before you compare the model names.
Frequently asked questions
How many examples do I need?
Start with several varied examples that represent your own tasks. Add difficult or unusual cases as you find them. This is a personal workflow check rather than a broad benchmark.
Should I use a leaderboard instead?
A leaderboard can help you find candidates. Your test set shows how those candidates handle the inputs, output format, and hardware you actually use.
What if the faster model makes more mistakes?
Compare how serious the mistakes are and how much review they require. You may choose different models for different tasks.
Can I use the same score for every task?
Use separate measures where useful. Factual extraction, valid formatting, and writing quality have different failure modes.
