Qwen3.8 vs Gemma 4: Best Local LLM for a 24GB GPU

You want to run an open-weight LLM on your own GPU to process sensitive data without sending anything to a US cloud? The model choice decides feasibility, quality and budget. In 2026, a card with 24GB of VRAM (like the RTX 4090) remains the realistic entry point for local inference, and three models fit on it in Q4 quantization: Qwen3.8-27B, Qwen3.6-27B and Gemma 4 31B-it.

Direct answer: Kingy.ai's benchmark (one RTX 4090, llama.cpp, synthetic suites) picks Qwen3.8-27B as the best default on a 24GB GPU: the same decode speed as Qwen3.6 (~49 tokens/s), but 12/12 coding tasks on the first seed, 39/40 reasoning cases and 23/24 document questions. Gemma 4 31B-it remains the credible alternative for strict tool calls and vision. Qwen3.6-27B only makes sense for an existing integration that is already stabilized.

Why a local LLM matters for European businesses

A model running on your own servers keeps your data in-house. No prompt, no document, no log leaves your infrastructure. That is the structural answer to the two questions raised by cloud AI:

• Article 44 of the GDPR prohibits personal data transfers outside the EU without adequate safeguards.
• The CLOUD Act (2018) allows US authorities to demand access to data held by US companies, even when it is stored in Europe.

A local deployment on hardware you own eliminates both risks at the source. This is the core of sovereign AI: open-weight models under Apache 2.0 licenses running on your own infrastructure, with marginal inference costs independent of API pricing. The CNIL documents AI best practices on cnil.fr, and the post open weights: the debate shaping sovereign AI explains why this path is gaining ground over proprietary frontier models.

How the benchmark was run

Kingy.ai measured the three models on a single RTX 4090 (24,564 MiB), using llama.cpp (commit b10453), Ubuntu 22.04.3, NVIDIA driver 570.195.03 and CUDA 11.8, on a Ryzen 9 7950X with 128GB of RAM and no swap.

The settings were shared: full GPU offload, flash attention, F16 K/V cache, batch 2,048, speculative decoding disabled for quality tests. Two comparison lanes were defined:

• Equal Q4: the three exact Q4_K_M files, same settings.
• Best usable inside 24GB: only useful changes allowed (Gemma switches its K/V cache to Q8_0 to fit 64K).

The test suite contains 50 instruction checks, 40 reasoning cases over three seeds, 30 single-tool calls, 10 multi-step calls, 12 coding tasks, 24 document questions (8K and 32K packs) and 20 synthetic images. The runtime counted 27.32 billion parameters for Qwen3.8, 26.90 for Qwen3.6 and 30.70 for Gemma.

Which Q4 files to pick and how much they weigh

Model	GGUF file	Size
Qwen3.8-27B	Qwen3.8-27B-Q4_K_M.gguf	15.932 GiB
Qwen3.6-27B	Qwen3.6-27B-Q4_K_M.gguf	15.662 GiB
Gemma 4 31B-it	gemma-4-31B-it-Q4_K_M.gguf	17.065 GiB

All three model cards declare 262,144 maximum positions. Actual architecture: 64 text layers for the Qwen models (48 linear-attention, 16 full-attention, one full layer every 4 levels) and 60 layers for Gemma (50 sliding-attention, 10 full). The vision encoders have 27 layers on all three. That declared limit is an architectural bound, not a promise: the highest stable context measured was 64K.

Which model to pick per task?

Priority	Pick	Measured result
Coding agents, repository edits	Qwen3.8-27B	12/12 pass@1, 35/36 seeded runs
Reasoning under a token budget	Qwen3.8-27B	39/40 pass@3, 117/120
Private document QA	Qwen3.8-27B	23/24, ahead of Gemma (22/24)
Strict tool calls	Gemma 4 31B-it	90/90 single, 30/30 multi-step
Vision-heavy tasks	Gemma 4 31B-it	19/20, versus 18/20 for Qwen3.8
Writing and copyediting	Tie	94/96, 93/96, 93/96
64K context with headroom	Qwen3.8 or Qwen3.6	4.20 GiB free with F16 K/V
Minimum migration work	Qwen3.6-27B	Already validated stack

The verdict: for a new deployment, Qwen3.8-27B is hard to beat. Gemma wins on declared tool calls, vision and writing, but uses more VRAM and its iterative coding loop is less reliable. Qwen3.6 only stays relevant if your pipeline is already tuned to it. The full test and methodology are published on kingy.ai.

What actually fits in 24GB?

A 16 to 17 GiB weight file does not leave seven or eight GiB for context: the runtime, the K/V cache, vision projectors and speculative decoding also consume VRAM. "It loads", "it answers a short prompt" and "it is comfortable through a long session" are three different states.

Profile	8K	32K	64K
Qwen3.8-27B, F16 K/V	16,626 MiB	18,186 MiB	20,266 MiB, pass, 4,298 MiB free
Qwen3.6-27B, F16 K/V	16,626 MiB	18,186 MiB	20,266 MiB, pass
Gemma 4 31B-it, F16 K/V	19,962 MiB	21,906 MiB	CUDA OOM
Gemma 4 31B-it, K/V Q8_0	not measured	not required	21,956 MiB, pass

At 64K, both Qwen models pass with an F16 cache and 4.20 GiB free. Gemma's F16 cache fails during allocation; switching its K/V cache to Q8_0 lets it fit 64K (21,956 MiB) but with only 2.55 GiB free. A desktop compositor, a vision projector, a parallel slot or a long output can burn that margin quickly.

Speed: measured throughput and latency

Qwen3.8 and Qwen3.6 are neck and neck on every speed metric. Gemma matches them on short prompts, then falls back as context grows, with a decode roughly 8.3% slower.

Metric	Qwen3.8-27B	Qwen3.6-27B	Gemma 4 31B-it
512-token prompt	3,001 tok/s	2,980 tok/s	2,975 tok/s
8,192-token prompt	2,844 tok/s	2,840 tok/s	2,682 tok/s
32,768-token prompt	2,554 tok/s	2,555 tok/s	2,235 tok/s
256-token decode	49.09 tok/s	49.04 tok/s	45.00 tok/s
Warm TTFT (512)	0.107 s	0.108 s	0.421 s

Time to first token is the real gap: 0.107 s for Qwen3.8 versus 0.421 s for Gemma on a 512-token prompt. For a user interface, this difference is immediately noticeable. Before buying a card, estimate your VRAM needs with the VRAM simulator.

Is speculative decoding (MTP) worth it?

MTP (multi-token prediction) speeds up generation, at the cost of VRAM and a potentially different behavior. On 15 practical prompts:

• Qwen3.8: 1.64x faster end-to-end, 1.66x on decode, +1,032 MiB VRAM. Outputs with and without MTP were identical in none of the pairs (0/15).
• Gemma: 1.52x end-to-end, 1.67x on decode, +574 MiB, 0.479 GiB drafter. Outputs were identical in 1/15 pairs.

MTP is a deliberate speed/behavior tradeoff, not a transparent switch. Keep it off if output reproducibility is a requirement, on if throughput is the priority.

TL;DR

For a local 24GB GPU deployment in 2026: Qwen3.8-27B is the best default choice (coding, reasoning, document QA, same performance as Qwen3.6). Gemma 4 31B-it wins on strict tool calls and vision, at the cost of higher VRAM usage and a first-token latency roughly 4x slower. Qwen3.6-27B only makes sense for an existing integration.

The highest stable context measured was 64K (not the advertised 262K) and the VRAM headroom at 64K is thin: plan your sessions and your speculative decoding with the VRAM simulator.

An open-weight model on your GPU is sovereign AI that meets your GDPR constraints without depending on API pricing. As an AI integration specialist, I help SMBs and startups build these architectures: RAG integration, model selection and GPU sizing. To go to production, also read the guide Local LLMs and GDPR.

Still deciding between a local LLM and a cloud API for your sensitive data? Let's talk.