Matrix multiply and TF32

Documentation for the matrix multiply / TF32 work from this chat: concepts, implementation in _CutlassAccel, commands, Lua, correctness, and roadmap.

Related: Cutlass.md · BrainstormIntegrations.md · RemoteCUDA.md · MathNN.md

What TF32 is

TensorFloat-32 is an Ampere+ Tensor Core math mode:

• Values stay stored as FP32 in memory.
• Multiplies on Tensor Cores use TF32 rounding; accumulators stay FP32.
• Higher throughput on large GEMMs; small numerical error vs true FP32.
• Not the same as FP16/BF16 storage, and not available as a CPU instruction (host stays AVX FP32).

Call graph in this codebase

Caller (MathNN, RL, benches, Lua, typed commands)
    │
    ▼
_MathNNDevice::MatMul(device, A,B,C,M,N,K)
    │
    ├─ CPU     → _MatrixAcceleration::MatMul   (AVX / AVX-512 / scalar)
    ├─ GPU     → _CutlassAccel::Gemm
    └─ Remote  → _RemoteCuda::MatMul (+ local fallback)

_CutlassAccel::Gemm
    │
    ├─ precision Tf32 + GPU → cuBLAS GemmEx  CUBLAS_COMPUTE_32F_FAST_TF32  (SM ≥ 8)
    ├─ precision Fp32 + GPU → cuBLAS GemmEx  CUBLAS_COMPUTE_32F
    ├─ cuBLAS / CUDA fail   → host MatrixAcceleration
    └─ no GPU               → host MatrixAcceleration

Layout: host tensors are row-major C[M,N] = A[M,K] @ B[K,N]. cuBLAS is column-major; the implementation uses the standard operand/leading-dimension trick so row-major buffers work without explicit transpose copies of the math result.

Precision enum

enum class Precision { Fp32, Tf32, Fp16, Bf16 };

Typed commands

cutlass status
cutlass set precision fp32|tf32|fp16|bf16
cutlass gemm [M N K]          # default 64³
cutlass bench [size] [n]      # default 256, 5
cutlass check [size]          # default 64; FP32 golden + TF32 probe

Alias: cute ….

Examples:

cutlass set precision tf32
cutlass gemm 1024 1024 1024
cutlass check 128
cutlass bench 1024 10

C++ API

_CutlassAccel::Init();
_CutlassAccel::SetPrecision(_CutlassAccel::Precision::Tf32);

string detail;
bool ok = _CutlassAccel::Gemm(A, B, C, M, N, K, detail);
// detail e.g. "cublas_tf32 sm>=8 ms=..."

// Fused linear (GEMM + bias + optional ReLU)
_CutlassAccel::LinearFused(W, x, bias, y, batch, in_features, out_features, relu, detail);

MathNNDevice GPU path already prefers _CutlassAccel::Gemm first.

Lua API

Registered via BrainstormIntegrations::RegisterLuaGlobals:

cutlass_status()
cutlass_set_precision("tf32")   -- or "fp32", "fp16", "bf16"
ok, detail = cutlass_gemm(128, 128, 128)
print(cutlass_check(64))
print(cutlass_bench(256, 3))

Correctness policy

cutlass check always runs a host golden and reports both FP32 and TF32 probe lines.

Never expect bit-exact TF32 == host FP32.

Performance notes

• TF32 wins when MMA-bound (large M, N, K).
• Tiny GEMMs may be faster on host (no PCIe).
• Optional future: size threshold to skip GPU for small problems.
• bench reports façade average ms vs pure host average ms.

Related “similar functions”

Build / link (optional TF32 via Configuration Manager)

# TF32 available (then: cutlass set precision tf32)
msbuild /p:Configuration=Release-GPU-AVX2

# GPU but no TF32 code path
msbuild /p:Configuration=Release-GPU-AVX2 /p:EnableTf32=false

# Opt-in on plain Release
msbuild /p:Configuration=Release /p:EnableTf32=true

• Links: cublas.lib / cublasLt.lib on GPU configs.
• Requires CUDA 11+ for CUBLAS_COMPUTE_32F_FAST_TF32.
• Runtime still needs Ampere+ (SM ≥ 8) for actual TF32; otherwise FP32 compute.
• Optional later: AI_USE_CUTLASS + device .cu.

cutlass status reports tf32_compiled= and tf32_available=.

Roadmap (from brainstorm)

Safety / product policy

Files

• CutlassAccel.hpp / CutlassAccel.cpp
• MatrixAcceleration.hpp / .cpp (+ AVX variants)
• MathNNDevice.cpp (dispatch)
• BrainstormIntegrations.cpp (Lua bindings)
• docs/Cutlass.md (short command card)