inference-atlas

Should you self-host this LLM, or just pay the API?

GitHub Pencil drawing of a graphics card shaped like a water dispenser. Its clear tank holds a grey block for the model's weights and small coral cubes stacked above it, one cube per user, with empty space left at the top.

Use case

Your answer

First-order estimate, usually optimistic. Check with vllm bench serve. How we estimate

Pencil drawing of a graphics card shaped like a water dispenser. Its clear tank holds a grey block for the model's weights and small coral cubes stacked above it, one cube per user, with empty space left at the top.

Run it

Why these users fit

Tune assumptions

How the conversation memory is stored. FP8 fits about twice the users.

The longest request the server accepts (--max-model-len). Capped at the model's limit.

Chat is about 3, agents about 10: they read far more than they write.

Share of each prompt that repeats (system prompt, tool list) and is served from the cache.

Real traffic comes in waves. You pay for idle hours too.

Leave empty to use the most users that still meet the speed target (--max-num-seqs).

Marked on the chart so you can see which side of the break-even you are on.

Quality

Quality tier: Unrated. No quality tier yet.

What each precision costs in accuracy, quoted from the sources. Measured on the Llama 3.1 family; other models may lose more or less.

PrecisionWhat you loseSources

How we estimate
  • Memory. Weights take parameters × bytes per parameter (2 for FP16, 1 for FP8, 0.5 for INT4). vLLM gets 90% of each GPU's memory; the rest covers working space.
  • GPUs. We use the fewest GPUs (1, 2, 4 or 8) that hold the weights plus one conversation of the longest length, and that the model's attention heads can be split across.
  • Users. Each conversation keeps a KV cache: 2 × KV heads × head size × layers × bytes per token. Sliding-window layers keep only their window, and linear-attention layers keep a small fixed state. Users = free memory ÷ one average conversation.
  • Speed. Each new token reads the active weights and every user's cache from memory, or does 2 × active parameters × users of maths, whichever is slower (the roofline). We assume kernels reach 50% of peak maths and 85% of bandwidth when split across GPUs.
  • Throughput. A group of users first reads all their prompts (cached parts are free), then writes the answers together.
  • Cost. GPU rent per hour ÷ tokens served per hour at your busy time. The API price is OpenRouter's cheapest paid listing for the same model, mixed by your input:output ratio.
  • Limits. Mixture-of-experts speed is approximate. Real servers lose time to scheduling and uneven traffic, so treat every number as a best case and measure with vllm bench serve.