What changes when the weights are on your own machine

The usual framing is a choice between a hosted endpoint and locally run weights, as though one were simply the private version of the other. For the purpose of comparison the real difference is which variables you control and which ones move without telling you.

What you gain: a fixed target

Local weights do not change underneath a test. The same files, the same runtime version and the same settings produce a result you can reproduce next month, which makes regression testing meaningful and makes an old evaluation still worth something. Input length limits, batching and queueing become properties you set rather than properties you discover.

What you take on: the serving stack

In exchange, throughput, latency and memory ceiling become consequences of your own choices - hardware, quantisation, batch size, runtime, concurrency limits. Quantisation in particular is a quality variable disguised as a deployment setting, and comparing quantised local weights against a hosted endpoint compares two different artefacts. Record the exact configuration alongside the result, because without it the number is not reproducible even by you.

Comparing across the line

When the shortlist mixes both, keep the task, the prompt and the judgement rule identical and accept that the operational axes are not like-for-like. Report quality on the shared set, and report cost and latency separately with the configuration stated, rather than merging them into one ranking that quietly assumes the two are the same kind of thing.