# Gemma 4 E2B on Four Accelerators: Which Serving Engine Wins Where

# Gemma 4 E2B on Four Accelerators: Which Serving Engine Wins Where

The same small model, Gemma 4 E2B, served by six engines on four chips: an 8 GB RTX 3050 in a desktop, a Tesla T4 on Compute Engine, an NVIDIA L4 in an EC2 G6, and one Cloud TPU v5e chip. Each platform got the same prompts, the same 1-to-64-request load sweep, and the same 1,319 GSM8K and 400 BFCL questions, with Google's E2B drafter wherever the engine supports one. The figures come from a four-part series by gde on dev.to. The question is practical: does the engine you should run change with the chip? The answers barely move. The speed does, and the drafter moves it more than the engine choice itself.

## The setup

Same model everywhere: Gemma 4 E2B on Google's QAT 4-bit weights, greedy decoding, thinking off for speed and accuracy, one engine at a time on the chip. On the three GPUs every engine version and checkpoint revision is the same: llama.cpp `7182ef71e`, vLLM 0.33.0, SGLang 0.5.23, mistral.rs v0.9.5, LiteRT-LM 0.19.0, and the exact Q4_0 GGUF or the W4A16 checkpoint carrying the same 4-bit values. The TPU runs vLLM from its pinned TPU image on the same W4A16 checkpoint, and the JAX engines and SGLang-JAX on bf16 `google/gemma-4-E2B-it`. llama.cpp and mistral.rs have no TPU backend.

| Platform | Chip | Memory | Host | Engines |
| --- | --- | --- | --- | --- |
| 3050 | RTX 3050 OEM, Ampere | 8,192 MiB | i7-5820K desktop | llama.cpp, vLLM, SGLang, mistral.rs, PyTorch, LiteRT-LM |
| T4 | Tesla T4, Turing | 15,360 MiB | GCE `n1-standard-2` | the same six |
| L4 | NVIDIA L4, Ada | 23,034 MiB | EC2 g6.4xlarge | the same six |
| v5e-1 | TPU v5e, one chip | 15.75 GiB | flex-start `v5litepod-1` | vLLM, SGLang-JAX, a JAX engine, a Rust model over PJRT, TorchTPU |

Each ladder ran the same three jobs per engine: one request (six chat prompts of 256 tokens, three repeats), load (a sweep at 1 to 64 requests, two shapes of 512 tokens in / 128 out and 128 in / 512 out, five repeats, a unique prompt per request so no prefix cache can hit), and accuracy (all 1,319 GSM8K test questions and the 400 BFCL simple function-calling questions).

## Answers do not depend on the chip

Five CUDA engines on three GPUs give 15 GSM8K scores between 0.8954 and 0.9030 and BFCL scores between 0.9150 and 0.9250, a spread of 4 questions. On the TPU the int4 builds score 0.8969 to 0.9007 and the bf16 engines 0.9113 to 0.9143.

| Engine | 3050 | T4 | L4 |
| --- | --- | --- | --- |
| vLLM | 0.9014 / 0.9250 | 0.8999 / 0.9225 | 0.8969 / 0.9225 |
| SGLang | 0.8992 / 0.9150 | 0.8954 / 0.9175 | 0.8954 / 0.9150 |
| llama.cpp | 0.9014 / 0.9225 | 0.9030 / 0.9250 | 0.8984 / 0.9250 |
| mistral.rs | 0.9030 / 0.9150 | 0.8999 / 0.9175 | 0.9014 / 0.9200 |
| PyTorch | 0.8992 / 0.9225 | 0.8961 / 0.9225 | 0.8961 / 0.9225 |
| LiteRT-LM, Google's file | 0.8795 | 0.8734 | 0.8908 |

(GSM8K / BFCL, one request at a time, no drafter.)

On the TPU, SGLang-JAX on bf16 leads at 0.9143 GSM8K, ahead of the JAX engine at 0.9128 and the Rust model over PJRT at 0.9113; vLLM on int4 tables scores 0.9007, and is the only TPU engine that returns tool calls (BFCL 0.9200).

One result deserves a closer look. LiteRT-LM is the lowest on every GPU. Google's `gemma-4-E2B-it.litertlm` file mixes int2, int4 and int8 weights, and it scores 0.0046 to 0.0220 below the lowest CUDA engine on the same card, while LiteRT-LM serving the exact Q4_0 build scores 0.8999 on the L4, level with the others. The engine is fine; the file it ships with costs answers. If you run LiteRT-LM, serve the Q4_0 build, not Google's mixed-precision file.

## One request, no drafter: the chip sets the speed

Tokens per second, end to end:

| Engine | 3050 | T4 | L4 | v5e-1 |
| --- | --- | --- | --- | --- |
| vLLM | 121.1 | 108.4 | 140.4 | 143.9 |
| SGLang | 119.9 | 97.8 | 136.5 | 159.9 (SGLang-JAX) |
| llama.cpp | 108.8 | 104.8 | 135.7 | n/a |
| mistral.rs | 118.9 | 113.5 | 144.6 | n/a |
| PyTorch | 118.9 | 109.8 | 134.7 | 6.6 (TorchTPU) |
| LiteRT-LM | 62.0 | 44.6 | 98.4 | n/a |
| JAX engine, bf16 | n/a | n/a | n/a | 134.3 |
| Rust model over PJRT | n/a | n/a | n/a | 145.2 |

On each GPU the five CUDA engines sit close together: fastest over slowest is 1.113 on the 3050, 1.161 on the T4 and 1.073 on the L4. The chip moves them all at once. Against the L4, the same engine runs at 0.80 to 0.88x on the 3050 and 0.72 to 0.82x on the T4, and vLLM on one v5e chip runs at 1.025x vLLM on the L4. LiteRT-LM runs at 0.426 to 0.725x llama.cpp on the same card, and TorchTPU manages 6.6 tokens per second.

## The drafter changes everything

Best drafter against no drafter, same engine and card:

| Engine | 3050 | T4 | L4 |
| --- | --- | --- | --- |
| vLLM | 1.90x (4 tokens) | 1.78x (3 tokens) | 1.96x (4 tokens) |
| llama.cpp | 1.35x (2 tokens) | 1.18x (1 token) | 1.53x (2 tokens) |
| SGLang | 1.33x (4 tokens) | 0.72x (4 tokens) | 1.39x (4 tokens) |
| LiteRT-LM | 1.10x (its own) | 0.98x (its own) | 1.15x (its own) |
| mistral.rs | 0.72x (1 token) | fails | 0.95x (1 token) |

The fastest engine for one request is vLLM with its drafter on every GPU: 230.5 tokens per second on the 3050, 193.0 on the T4 and 275.3 on the L4. On the TPU, vLLM with Google's drafter runs out of memory during warmup: the drafter does not fit beside vLLM on one v5e chip, and SGLang-JAX is fastest there at 159.9 tokens per second.

Two drafters actively hurt. SGLang's drafter loses on the T4 (0.72x), and mistral.rs's drafter is slower than no drafter on the 3050 and L4, and fails outright on the T4: the T4 runs the target model in float16 and the drafter loads as bf16, so every drafted request fails with a dtype mismatch in matmul. A drafter is not a free speedup. It is a second model that has to fit, match dtypes, and actually predict the target's tokens.

## At 64 requests, the picture splits

Aggregate output tokens per second at 64 requests, each engine at its best setting, 128 in / 512 out:

| Engine | 3050 | T4 | L4 | v5e-1 |
| --- | --- | --- | --- | --- |
| vLLM | 1,658 (1 drafted) | 1,165.5 (1 drafted) | 4,516.3 (4 drafted) | 1,689.1 (none) |
| SGLang | 1,711 (none) | 908.6 (none) | 2,967.1 (none) | 4,249.4 (SGLang-JAX) |
| llama.cpp | 633 (none) | 462.5 (4 drafted) | 1,170.3 (2 drafted) | n/a |
| mistral.rs | 308 (none) | 237.3 (none) | 1,825.2 (none) | n/a |

512 in / 128 out:

| Engine | 3050 | T4 | L4 | v5e-1 |
| --- | --- | --- | --- | --- |
| vLLM | 603 (none) | 328.9 (1 drafted) | 1,710.0 (1 drafted) | 1,288.4 (none) |
| SGLang | 544 (none) | 175.0 (none) | 1,373.0 (none) | 2,758.9 (SGLang-JAX) |
| llama.cpp | 368 (none) | 281.9 (none) | 630.8 (none) | n/a |
| mistral.rs | 259 (none) | 196.7 (none) | 1,132.0 (none) | n/a |

vLLM leads on the T4 and the L4 at both shapes, by 1.283x and 1.522x over SGLang on long replies. On the 3050, SGLang leads vLLM by 1.032x on long replies and vLLM leads on long prompts. On the TPU, SGLang-JAX leads vLLM by 2.516x on long replies and 2.141x on long prompts, with the bf16 model against vLLM's int4 build, so engine and weights differ together there.

The drafter still pays at 64 requests on the T4 and the L4: 1.337x and 1.215x on long replies, level on the 3050 from 32. But the best draft length falls as requests rise, from 3 or 4 tokens for one request to 1 token or none at 64, except the L4 on long replies, where 4 tokens stay best. SGLang's drafter loses under load: 0.30 to 0.49x of running without it at 64 on the 3050 and 0.69 to 0.73x on the L4.

## The same engine on another chip

vLLM without a drafter, against vLLM on the L4:

| Card | 1 request | 64 requests, 128/512 | 64 requests, 512/128 |
| --- | --- | --- | --- |
| 3050 | 0.863 | 0.444 | 0.378 |
| T4 | 0.772 | 0.235 | 0.185 |
| v5e-1 | 1.025 | 0.455 | 0.808 |

One request depends little on the card; many requests depend a lot. vLLM's output at 64 requests is 26.77x its one-request rate on the L4, 13.76x on the 3050, 11.69x on the v5e chip and 8.59x on the T4, for long replies.

## What each platform asks of each engine

| Engine | 3050 | T4 | L4 | v5e-1 |
| --- | --- | --- | --- | --- |
| vLLM | its own torch, no extra deps | one patch, Triton tile clamp | its own torch | pinned TPU image, int4 table patches |
| SGLang | two patches | four patches, Marlin from vLLM | two patches | SGLang-JAX E2B patch |
| llama.cpp | as installed | as installed | as installed | no TPU backend |
| mistral.rs | fixed cache | no flash-attn | cuRAND, fixed cache | no TPU backend |
| PyTorch | this repo's engine | this repo's engine | this repo's engine | TorchTPU at 6.6 tok/s |
| LiteRT-LM | Vulkan | Vulkan | Vulkan | n/a |

SGLang needs patches on every platform for this checkpoint. llama.cpp runs as installed everywhere, and is the right one for one user in under 3 GB.

## The short version

On any of the three GPUs, run vLLM with Google's W4A16 drafter: fastest for one request on all three, and at 64 requests it leads on the T4 and the L4 and is within 3% of SGLang on the 3050. Drop the draft length toward 1 token as load rises. llama.cpp with its drafter runs as installed everywhere, and is the right one for one user in under 3 GB. On one TPU v5e chip, vLLM from its pinned image serves E2B with tool calls as shipped; SGLang-JAX with the E2B patch is 2.14 to 2.52x faster at 64 requests.

## Sources

- gde, "Six Serving Engines on Four Accelerators: Gemma 4 E2B on an RTX 3050, a T4, an L4 and a TPU v5e", dev.to: https://dev.to/gde/six-serving-engines-on-four-accelerators-gemma-4-e2b-on-an-rtx-3050-a-t4-an-l4-and-a-tpu-v5e-1a0e
- Comparison repo and rollup (result files, per-ladder articles): https://github.com/xbill9/gemma4-dev

---

*P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam*

[![Buy me a coffee](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/87bycvv0drlpbwxjppyi.png)](https://www.patreon.com/c/AheadKit)