Tuning vLLM for a local coding agent and capping the power

A follow-up to Local LLM coding agent weekend. Last time the model picked one config and we moved on. This time I actually measured the tuning parameters, and then capped the GPUs to save power.

In short, raising max-num-seqs from 2 to 8 was the big win: four-request throughput jumped 55%. Speculative decoding (MTP) made short responses feel quicker but did not move throughput, and it shrank the KV cache. Doubling context to 256k fit, though it only allows just over one full-context request at a time. Capping the GPUs to 250 W cost a single user nothing.

Setup

• 2x RTX 3090 (24 GB) in one machine, tensor-parallel 2, no NVLink.
• vLLM 0.28.0, running as a vllm systemd service.
• Model: Avuja/Qwen3.8-27B-int4-AutoRound (INT4, Marlin kernels, BF16 activations).
• The model reports a 262144-token context and ships MTP tensors (explained below).

I benchmarked with a small Python harness that streams /v1/completions at temperature 0, always requesting exactly 256 output tokens with ignore_eos, plus a separate context probe that feeds a repeated token-id sequence up to the context limit. It's deliberately a synthetic comparison, not a broad workload study: enough to tell me which setting matters, not enough to publish a leaderboard.

The lever that mattered: max-num-seqs

My production config had been --max-num-seqs 2 since last weekend (conservative, "one agent, queue up the rest"). The obvious thing to try was letting more requests run concurrently:

--max-num-seqs 8 --max-num-batched-tokens 8192

max-num-seqs is the active-sequence cap; max-num-batched-tokens is how much prefill+decode work one iteration can do. The result:

Three things stand out. First, single-request decode speed is unchanged at about 58 tok/s; concurrency costs a single client nothing, which makes this a pure win. Second, aggregate throughput scales roughly in line with concurrency up to the cap, from 77 to 120 to 174 tok/s. Third, first-token actually improves as concurrency rises (3.90 s to 2.29 s at four requests), because chunked prefill spreads the long-prompt work across the extra sequences. That last one is counterintuitive; I'd expected the opposite.

So I kept it: the service now runs --max-num-seqs 8 --max-num-batched-tokens 8192.

Speculative decoding (MTP): nice latency, weaker throughput

MTP stands for Multi-Token Prediction. Rather than emitting one token per forward pass, the transformer gets a small draft head on top that predicts the next few tokens in advance. Those drafts are checked against the main model, and when they match they are accepted at no extra cost: the same forward pass that would have produced one token now turns out several. It is speculative decoding baked into the checkpoint itself, not a separate draft model that vLLM has to manage. Qwen3.8 ships the MTP tensors, and vLLM exposes it as speculative_config; the relevant field is num_speculative_tokens, how many draft tokens to attempt per step. I started with one:

--speculative-config '{"method":"mtp","num_speculative_tokens":1}'

At one speculative token the draft acceptance rate was around 86 to 92 percent, with a mean acceptance length of about 1.9. On its face that is a good number. Two things hold it back in practice. The first is that the speculative state eats into the KV cache: the pool shrank from about 280k to 242k tokens. The second is that it never beat the plain 8-seq configuration on throughput.

Going higher with num_speculative_tokens could in principle accept more tokens for free, but every extra draft token also carries its own chance of a rejection that wastes the step, and it costs more state. It is not automatically better, so I would benchmark it per model rather than assume.

In the end it does make short responses feel a touch snappier for a single user, but it steals cache from the thing I care about more, which is long agent context. I left it off by default and left it available as an opt-in profile.

Single GPU: it loads, don't rely on it

To free up a GPU for other work I tried running TP=1 on just GPU1 with a 32k context and a 0.95 memory budget. It loads and passes a 32k probe, and at 95% budget it fits ~31.7k tokens (a 90% budget only gets ~16.5k).

But two things keep it off the critical path: single-GPU decode is slower than TP2 (47 vs 58 tok/s), and the run threw an asynchronous engine-dead error under a heavier workload even though the benchmark files completed. Workable as a fallback to park GPU0, not something I would serve the main agent from.

256k context: it fits, but barely, for full-context requests

The model advertises 262144 tokens, so I raised it:

--max-model-len 262144

vLLM fit a KV pool of about 288k tokens, which is only 1.1x a full 256k request. So I can do one full-context request at a time, which is fine for my workload (a single big agent conversation), but not several. The original 128k setting gave me roughly 2.1x. The trade worked in my favor: I kept 8 active sequences for short turns while still holding a full 256k ceiling for the long ones.

The launcher (vllm-start) selects a profile through an environment variable, so switching is a one-liner, not a recompile:

Capping the power: 250 W

Once the throughput was sorted, the next lever was energy and thermals. The 3090s were sitting well under their default ceiling, and a benchmark with the limit set to 275 W came back clean: no HTTP 500 responses, no OOM, no engine faults, and decode at or above the earlier run. So I went further, to 250 W:

/usr/bin/nvidia-smi -i 0,1 -pl 250

It applied cleanly and the server stayed healthy. Single-request decode peaked at about 61.5 tok/s at the cap, no measurable loss for a single user.

The real question was whether the cap would bite under concurrent load, where a fixed power budget matters more than it does for a single stream. I reran the concurrency benchmark against the capped server over the network. Four concurrent requests came in at roughly 146 tok/s, statistically the same as the about 141 tok/s the 275 W run produced at that point. Pushing to eight concurrent requests gave about 214 tok/s, with no sign of throttling. In fact the capped numbers ran slightly higher than the earlier ones, which is run-to-run variance, not a real gain.

This is where a caution I had logged in an earlier draft paid off. I first tried putting that nvidia-smi line at the top of the service's startup script. It died on Insufficient Permissions, because it is a privileged operation; the set -e at the top then killed the whole startup. Wrapping it in sudo did not help either: a unit's main process has no TTY, so there is nothing to read a password from. The fix was to move the privilege into its own root unit, ordered to run before the model:

# gpu-power-limit.service
[Unit]
Description=Limit GPU power
Before=vllm-qwen.service
[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -i 0,1 -pl 250
[Install]
WantedBy=multi-user.target

with Wants=gpu-power-limit.service and After=gpu-power-limit.service in the vLLM unit. Wants (rather than Requires) is deliberate: if the cap ever fails to apply, the model still starts. Power is a nicety; inference is not. The vLLM script itself does no privileged work, so nothing in it can die on a permission error.

Wrapping up

A few things I would carry into the next model run. Start with the concurrency settings, the max-num-seqs and max-num-batched-tokens pair: that was the biggest single win and it cost a lone user nothing, so there is no reason not to raise it before touching anything else. Be more careful with speculative decoding. If a model ships MTP or EAGLE, an acceptance-rate number will always look impressive; the real cost is in the KV cache it steals, so profile it against your actual context budget and judge it on end-to-end throughput, not on how often the drafts hit. Watch the KV pool size rather than the max-model-len flag: a 256k context can fit in memory and still leave you just over a single full request from capacity, a detail nvidia-smi will not volunteer and only the startup report will show. And keep the two problems separate: a config meant to free a GPU and one meant to serve the agent are not the same job, and a profile that merely loads does not deserve to front a daemon until it has survived a sustained, heavier workload.

One last, small thing. While all of this was going on I also came across nvtop, a top-style live monitor for GPUs. It refreshes constantly and shows per-GPU utilization, power draw, and memory, with a process view underneath so you can see what is actually occupying the cards. It is not a substitute for nvidia-smi, which is still what I reach for when I need an exact field or am scripting something, but for the constant "what is the GPU doing right now" questions during a tuning session it is much easier on the eyes.