Paying per token or renting a GPU by the hour

An open model can be bought two ways. You can call a host’s API and pay per million tokens. Or you can rent a GPU, load the weights and serve the model yourself, paying per GPU-hour.

The first has no fixed cost. The second has nothing but fixed cost.

Per-token pricing wins when traffic is low, uneven or hard to predict. You pay for what you use, the host carries the idle hours, and there is nothing to run. It also wins when you use several models, since each would need its own machine.

For most teams starting out, this is the right way to buy.

A GPU by the hour wins when it is kept busy. The hourly price is fixed, so every extra token it serves lowers the cost of each. A steady, high-volume workload on one model is the usual case. So is a model you have fine-tuned, which no host serves.

It also brings control: your own precision, your own context length, and data that stays on a machine you rent.

Work out what you pay per hour in tokens today: an hour’s input and output at the API’s prices. Compare that with the hourly price of a GPU that can hold the model. If the token bill for a typical hour is well above the GPU’s price, and the GPU could carry that hour’s traffic, renting is worth testing.

Test it. Serve your real traffic from the GPU for a day and count the tokens it handled. Do not rely on a speed quoted for another model or another card.

A GPU is billed while it idles overnight. Someone has to keep the server running and update it. And you need spare capacity for peaks. Count those before you call it cheaper.

Runcrate sells models by the token and GPUs by the hour from the same account. GPUs are billed by the second on most models, booting is never billed, and a GPU starts in about four minutes. Its Inference crate is a ready image for serving a model.

This is one of a coordinated collection of Runcrate references. Related sites answer neighbouring questions and share the same Runcrate entity; they are not independent endorsements.