Logo
Overview
Running Qwen3.6-35B-A3B with llama.cpp on Fedora (WSL)

Running Qwen3.6-35B-A3B with llama.cpp on Fedora (WSL)

Nate Bracy Nate Bracy
July 26, 2026
3 min read
llm llama.cpp
index

I wanted a local model that could hold a 262k context window while providing a reasonable amount of capability. Here’s what got Qwen3.6-35B-A3B running well on a single RTX 4080.

Starting with a Google search, I found that there’s no shortage of writeups on doing this under Ubuntu on WSL. What I couldn’t find was a single one written for Fedora, so I decided to document this for posterity.

System prep: CUDA 13.3 on Fedora (WSL)

Starting from a fully updated Fedora box, the first step is adding NVIDIA’s CUDA repo and downloading the toolkit:

sudo dnf update -y
 
# Add NVIDIA's CUDA repo (match the fedora version number to your release)
sudo dnf config-manager addrepo --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora44/x86_64/cuda-fedora44.repo
 
# Build tools + CUDA 13 toolkit and cuBLAS dev headers
sudo dnf install -y \
  llvm clang cmake git \
  cuda-toolkit-13-3 \
  cuda-nvcc-13-3 \
  cuda-cudart-devel-13-3 \
  libcublas-devel-13-3
 
# Only needed if you want HTTPS support baked into llama-server
sudo dnf install -y openssl-devel

I used clang/clang++ as the host compiler instead of GCC, since CUDA toolkits usually certify against a narrower band of GCC versions than what’s actually shipping in a current Fedora release, and clang is usually more forgiving for me on newer distros.

Environment variables go in ~/.bashrc:

# CUDA 13 Compiler & Binaries
export PATH=/usr/local/cuda-13.3/bin:$PATH
 
# Library paths
export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:/usr/lib/wsl/lib:$LD_LIBRARY_PATH

Then source ~/.bashrc and check that the compiler and the driver can both see the GPU:

nvcc --version
nvidia-smi

If nvidia-smi doesn’t show your card here, stop. You’ll need to fix this before moving forward with building everything.

Finding your compute capability

llama.cpp’s CMake build wants an explicit CUDA architecture target that you can read off the card:

nvidia-smi --query-gpu=compute_cap --format=csv,noheader | tr -d '.'

An RTX 4080 reports 89 (Ada Lovelace). 30-series Ampere cards report 86. 20-series Turing cards report 75.

Building llama.cpp

A shallow clone is enough to get started, and git pull --depth 1 works fine for pulling in updates later:

git clone --depth 1 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

and these CMake configurations should get you a slightly above average baseline:

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_CUDA_HOST_COMPILER=clang++ \
  -DCMAKE_CUDA_FLAGS="-allow-unsupported-compiler" \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="89" \
  -DGGML_NATIVE=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON
 
cmake --build build --config Release -j$(nproc)

GGML_NATIVE=ON lets the CPU-side code build with -march=native, which matters for the parts of inference that never touch the GPU: tokenization, sampling, KV cache bookkeeping. GGML_CUDA_FA_ALL_QUANTS=ON compiles flash-attention kernels for every quant type instead of just the common ones, which will allow you to enable -fa on later.

Qwen3.6-35B-A3B

Qwen3.6-35B-A3B is a mixture-of-experts model, I’m running Unsloth’s UD-Q4_K_XL GGUF quant, which in my own testing holds up noticeably better than a flat Q4_K_M, especially on the MoE routing layers.

Model files come from Unsloth’s GGUF repo, which I saved to ~/.llama:

~/.llama/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/*/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf

by running

LLAMA_CACHE=~/.llama ./build/bin/llama-cli \
  -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL \
  -ngl 28 -fa

to download the weights.

Running it

I prefer a 262k context window, however you’re free to change that to whatever you like.

For interactive use:

./build/bin/llama-cli \
  -m ~/.llama/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/*/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  -fit on \
  -lm none \
  -t 12 \
  -c 262144 \
  -b 2048 \
  -ub 512 \
  -fa on \
  -cnv

And you can also start an OpenAI compatible endpoint:

./build/bin/llama-server \
  -m ~/.llama/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/*/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  -fit on \
  -t 12 \
  -c 262144 \
  -b 2048 \
  -ub 512 \
  -fa on \
  --port 8080

If you want an extra speedup you can also pass --spec-draft-n-max 2. Don’t forget to update your llama.cpp regularly.