Compute & Acceleration Fabric
Related: Remote CUDA, CUTLASS, TensorRT, TF32, llama.cpp CUDA, Build configurations.
_AI places compute across heterogeneous devices — CPU, local GPU, and a server — through one set of abstractions. This page is the map that ties the individual acceleration docs together and explains how work is routed to the right hardware.
The device abstraction
MathNN resolves an execution device per agent and per phase (training vs. inference):
_MathNNDevice::Kind = { Cpu, Gpu, Npu, Apu, Remote, Auto }
_MathNNDevice::ResolveForAgent(agent, /*train=*/true|false)An agent can therefore infer on the local GPU while its training resolves to a
Remote device on a compute server — the split is configured, not re-engineered.
Kernel routing
KernelHub dispatches linear-algebra kernels (e.g. GEMM) by device kind, routing to
CPU, GPU, or Remote/RemoteCuda back-ends behind one API. Heavy matrix work
can move to a server GPU without changing call sites.
LLM offload (llama.cpp)
llama_cpp_gpu_layers # 0 = CPU only, positive = CPU/CUDA hybrid, negative = all layers on GPU
llama_cpp_offload_kqv # offload KQV cache when CUDA is active
llm llama diagnose # validate model path, CUDA/offload, mmap/mlock, system infoA small model can run fully on a laptop GPU; a large model can be hosted on a server and reached through the LLM orchestration layer (Ollama / llama.cpp / commercial APIs).
Accelerator stacks
| Stack | Use | Doc |
|---|---|---|
| cuBLAS / CUTLASS / TF32 | dense GEMM, MathNN training | Cutlass, TF32 |
| TensorRT / LiteRT (TFLite) | vision + text detection/recognition inference | TensorRT |
| llama.cpp (CUDA) | local LLM inference | llama.cpp CUDA |
| Remote CUDA | server-side offload of GEMM/training | Remote CUDA |
Build matrix → hardware tiers
| Configuration | Intended node |
|---|---|
Release-CPU-AVX2, Release-CPU-NoAVX | laptop fallback / GPU-less hosts |
Release-GPU-AVX2, Release-GPU-AVX2-Tensor | CUDA laptop (Tensor-core variant for RTX-class GPUs) |
Release-GPU-AVX512, Release-GPU-AVX512-Tensor | compute server (AVX-512 CPUs + server GPU) |
Two-tier deployment
The intended operating point pairs a CUDA laptop (latency-critical, private, real-time inference) with a compute server (throughput-bound training, large-model hosting, shared MySQL state and checkpoints). The same binary, built for each tier, cooperates through the device abstraction, the kernel router, and MySQL. See the companion thesis Two-Tier Heterogeneous Inference.