# Project: Local LLM Inference Server (Qwen3.6-35B-A3B)

## Hardware Target
- **CPU:** Intel Core i5-12400F (6P-cores, 12 threads)
- **RAM:** 16GB Physical (15.8GB available)
- **Storage:** SSD Apacer AS350 (C: drive)

## Software Stack
- **Backend:** llama.cpp (compiled with AVX2, FMA, VNNI)
- **Model:** Qwen3.6-35B-A3B-UD-Q2_K_XL (Dynamic 2-bit MoE)
- **RAM:** 32GB Physical (Upgraded from 16GB)

## Optimization Parameters
To maintain stability:
- **Context Window:** 32768 tokens (expanded from 16k due to RAM upgrade)
- **Threads:** 5 for generation (`-t 5`), 6 for batch/prompt processing (`-tb 6`)
- **KV Cache:** 8-bit quantized (`-ctk q8_0 -ctv q8_0`)
- **Flash Attention:** Enabled (`-fa 1`)
- **Batch Size:** 4096 (`-b 4096 -ub 4096`)

## Operational Scripts
- `run_server.bat`: Starts the OpenAI-compatible API server.
- `benchmark_8k.bat`: Quick performance validation.

## Performance Metrics (Empirical)
- **Prompt Processing (PP):** ~29-30 t/s
- **Token Generation (TG):** ~9.8 t/s
- **Memory Usage:** ~13.5 GB (Model + 32k KV Cache) - plenty of headroom with 32GB RAM.


## Critical Constraints
- **Do NOT** use HDD for mmap/weights (extreme I/O latency).
- **Do NOT** exceed 16k context without upgrading RAM (OOM risk).
- **Do NOT** use IQ-quants (imatrix) due to high CPU overhead on dequantization.
