Start with a quantized variant that fits your machine, then compare it with another variant of the same model using the same tasks and settings. Judge success by correct answers, reliable formatting, and speed under your normal context length. A model that fits only with a short prompt may struggle in your real workflow.
Understand what the label describes
A quantization label describes how a model’s weights are represented. Using fewer bits can reduce the size of the weights and the memory needed to load them. The trade-off is that the stored values are approximations. How much that affects a task depends on the model, the quantization method and what you ask it to do. A four-bit label is not a universal quality score. Two methods that use a similar number of bits can make different choices about how to represent parts of the model.
Separate quantization from parameter count and model family. An eight-billion-parameter model and a much smaller model can use similar quantization labels while behaving very differently. An instruction-tuned model and a base model may also respond differently with the same prompt. For a first comparison, pick two published variants of the same model release and tune. Record their exact tags. Comparing unrelated models can answer which one you prefer, but it cannot tell you how much of the result came from quantization.
Find suitable variants and check the hardware
Use the model publisher’s release notes and the Ollama library to identify the available variants. Review the tag details and download sizes before pulling both versions. If you work from your own compatible model files, the Ollama model import guide describes creating a quantized model from supported higher-precision inputs. Most readers can begin with ready-made variants. Check that you are comparing local variants and that the model licenses suit your use.
A download size gives you a useful starting estimate, but a running model also needs memory for context and the runtime’s own work. Other apps consume memory as well. Check available system RAM and dedicated GPU memory where applicable. For the distinction and why it matters when the runtime offloads work, see our RAM and VRAM guide. Keep room for your usual browser, editor or other tools if you intend to run them at the same time.
Set up a fair comparison
Choose five to ten short tasks you expect to repeat. If your goal is document extraction, include one example with a missing field and another with two dates that are easy to confuse. If your goal is code assistance, use small cases with tests that have clear expected results. Write down what counts as success before running either model. An answer that looks polished but silently omits a required value should fail the extraction test.
Keep the input, context setting and generation options the same when comparing variants. If a task samples responses, run it more than once and keep the results. Test both models with a short prompt and with a representative long prompt. Review our Ollama context length guide if the amount of retained text differs between runs. When you compare model variants, context should stay constant until you deliberately test it as a separate variable.
Measure speed and memory under real conditions
Record the delay before the first useful output and the time to finish. A model can be quick to begin but slow to complete a long response. Also distinguish a first run that loads weights from a later run while the model remains loaded. Decide which behavior matters for your workflow. An occasional desktop query may be dominated by load time, while batch processing may depend more on steady generation speed. Compare the same kind of run for each variant.
Check for signs that the machine is under memory pressure, such as sustained swapping or other apps becoming unresponsive. Use the system’s own monitoring tools and note which other programs are running. If the model is distributed across CPU and GPU memory, results can differ substantially from those on a machine with enough dedicated GPU memory. Recheck with a normal workload before choosing the largest variant that completes a test on an otherwise idle computer.
Score the outputs before picking a default
Review accuracy, completeness and format separately. A summary can be coherent while leaving out the one exception the reader needed. For a structured output task, validate the fields before comparing how fluent the answers sound. Put the raw outputs next to your expected answers and score the same criteria for both variants. Our local model benchmarking guide shows how to separate task quality from speed.
If a smaller variant meets the target and responds faster, it can be the better default for that specific workload. Keep the higher-precision variant when the measured improvement is worth the additional memory and time. If both fail, test a different model or improve the input before changing quantization again. A poor extraction prompt or an incomplete source document can affect every variant. Record the failure type so you know which change to try next.
Keep the decision reproducible
Write down the model tag, runtime version, context setting and hardware used in your comparison. Save a few representative input and output pairs, using examples that contain no private data when you need to share them. Recheck after a model upgrade or a significant change to your workflow. A setup selected for short summaries may need a new baseline when you move to larger documents or ask for strict JSON.
If you compare different computers, report that separately. A fast desktop and a laptop on battery have different power and cooling behavior, even when they can load the same model. Time the same tasks on each and note whether the machine was plugged in. Make the final choice for the device and workload you expect to use most often. That decision is easier to revisit when the test inputs and scoring criteria are already available.
For a final check, reserve one task that you did not use while choosing settings. Run it once with the selected variant and review the output using the same scoring rules. A holdout example helps reveal a configuration that worked only because you tuned it around the earlier examples.
Keep separate notes for speed and task failures. For example, a faster variant that drops a mandatory field is useful for brainstorming, but an extraction workflow needs the field to survive. Record that distinction so the next comparison starts with the right success criteria.
Frequently asked questions
Is a higher bit count always better?
It often preserves more information in the weights, but the best practical choice depends on the model, the task and whether the variant fits with room for your workload.
Can a smaller model beat a larger one?
Yes, on a specific task or under a tight memory limit. Compare actual outputs and responsiveness on the target machine.
Do I need to quantize a model myself?
Start with published variants when they meet your needs. Custom quantization is useful when you have compatible files and need a controlled configuration.
