A week with a 7B model on my desk

27 October 2026 · Bristol

I run four things through models on a normal week: summarising long threads, extracting fields from invoices, rewriting documentation, and answering questions over a folder of PDFs. Last month I moved all four onto a 7B model running locally on a workstation with a 24 GB card, mostly to see which ones would survive.

Two did. Two went back to the API, and not for the reason I assumed.

Generation is bandwidth, not clock speed

Token generation reads the weights every token, so the ceiling is roughly memory bandwidth divided by model size. A 4-bit 7B model is about 4 GB; on a card with 500 GB/s that caps you near 110 tokens a second before any inefficiency. Buying a faster GPU does not move that number much. Buying more bandwidth does.

Prefill is the opposite. It is compute bound, which is why long prompts hurt local setups disproportionately. The same machine that feels instant on a two-paragraph input takes several seconds to absorb a 10k-token context.

Quantisation is not the free lunch the charts suggest

For summarising and rewriting, 4-bit was indistinguishable from the original. For invoice extraction it was not. The model started transcribing a digit wrong about once in twenty invoices, always in reference numbers with unusual formatting. That is the kind of error that costs more than the API bill it was supposed to replace.

The two that stayed local

Summarising threads: local wins easily. High volume, short outputs, no precision requirement. Cost fell by about 80 percent and the p95 latency dropped because there is no network in the path.

Rewriting documentation: local wins, mostly because I can send the whole style guide in the prompt without thinking about the bill. That is a workflow change, not just a cost change, and it is the part I did not expect.

The two that went back

Invoice extraction, for the digit problem above. Not fixable by prompting.

Question answering over PDFs, because of context length. The folder has documents that need 30k tokens of context, and at that length the local setup both slowed down badly and started missing details that the larger model caught. It was not close.

What I would tell someone starting

Try it on the workload you consider boring. That is where local usually wins and where the quality bar is lowest. Then try it on the one you consider risky, and expect to keep paying for the API. The split is rarely where the benchmark table suggests.

One practical note: budget the KV cache explicitly. A setup that is fine at 8k will fall over at 32k, and the failure often looks like slowness rather than an error.