Llama.cpp CPU, RAM, and CUDA Configuration

_AugmentedIntelligence links against the external llama.cpp build under C:\_AugmentedIntelligence\_src\llama_build and uses the source headers under C:\_AugmentedIntelligence\_src\llama.cpp.

Build llama.cpp With CUDA

Run from the _AugmentedIntelligence project root:

powershell -ExecutionPolicy Bypass -File scripts\build_llama_cpp_cuda.ps1

This configures llama.cpp with:

GGML_CUDA=ON
GGML_NATIVE=OFF

The Visual Studio project already links the generated llama.lib, ggml.lib, ggml-cpu.lib, ggml-cuda.lib, CUDA runtime, cuBLAS, and cuBLASLt libraries.

Runtime Settings

Use the console settings menu:

13 LLM -> 9 Llama.cpp Local Inference Settings

Important values:

llama_cpp_enabled=true
llama_cpp_cuda_enabled=true
llama_cpp_gpu_layers=-1
llama_cpp_mmap_enabled=true
llama_cpp_mlock_enabled=false
llama_cpp_offload_kqv=true
llama_cpp_unified_memory=true
llama_cpp_threads=0
llama_cpp_batch_threads=0
llama_cpp_batch_size=512
llama_cpp_ubatch_size=512

llama_cpp_gpu_layers controls CPU/CUDA split:

0        CPU-only inference
1..N     hybrid CPU + CUDA layer offload
-1       offload all possible layers to CUDA

llama_cpp_mmap_enabled uses OS memory mapping and system RAM/file cache for the GGUF model. llama_cpp_mlock_enabled asks the OS to keep model memory resident; leave it off unless the machine has enough RAM for the full model.

llama_cpp_unified_memory sets GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 before llama.cpp initializes. On Linux this allows CUDA unified-memory fallback to system RAM. On Windows, use NVIDIA Control Panel's System Memory Fallback setting if VRAM fallback is needed.