Docs  /  Layers Are What You Need

Layers Are What You Need

MathNN Building Blocks, Attention Variants, and Their Relevance to Gaming Reinforcement Learning

Technical research paper for _AugmentedIntelligence / MathNN
Version: 1.0 · Date: 2026
System: MathNN (MathNN*.{hpp,cpp}), Gaming RL (GamingRL.*), RLAgentNet, RLNeural

Abstract

The dominant narrative of modern machine learning is that a single elegant primitive—attention—suffices for sequence modeling at scale. Gaming reinforcement learning (RL), however, is not a single sequence problem. An agent must perceive patches of a frame, remember short trajectories, score discrete actions under partial observability, explore novel situations, and remain small enough to train inside a live desktop application.

We present MathNN, a compact C++ neural substrate inside _AugmentedIntelligence, and argue that layers are what you need: a small, composable inventory of dense, convolutional, recurrent, normalization, attention, and curiosity-attention blocks—wired into DQN, dueling DQN, actor–critic, world-model, and reward-model learners—covers the practical needs of local gaming RL without outsourcing every update to a remote GPU cluster.

This paper (i) surveys MathNN’s layer inventory and training capabilities, (ii) formalizes scaled-dot and curiosity attention as first-class RL operators, (iii) maps layers to gaming perception–action problems (FPS aim assist, Minecraft, navigation, strategy), and (iv) states what MathNN is—and is not—capable of. We position MathNN as an in-process learning layer: not a replacement for foundation models, but the correct tool when the loop must close in milliseconds on the machine that holds the game state.

1. Introduction

1.1 Motivation

Vaswani et al. titled their 2017 transformer paper “Attention Is All You Need”, crystallizing a bet that sequence transduction could discard recurrence and convolution in favor of multi-head self-attention. That bet paid off for language and large multimodal models. Gaming RL faces a different constraint set:

  1. Latency. Aim assist, movement, and combat decisions often require sub-frame or few-frame response.
  2. Locality. Game state, screenshots, and input injection live on a client machine; shipping every transition to the cloud is fragile and privacy-sensitive.
  3. Heterogeneous structure. Vision (frames), vectors (HUD: health, ammo), sequences (recent actions), and discrete multi-button actions coexist.
  4. Safety. Live control of mouse/keyboard or a game bridge needs shields, dry-run modes, and human override—not only higher Elo.

Under these constraints, no single layer family is “all you need.” Convolution organizes local visual structure; recurrence compresses short history; dense heads produce Q-values; attention routes over candidates (enemies, memory slots, skills); curiosity-biased attention steers exploration.

1.2 Thesis

Layers Are What You Need: for in-process gaming RL, a deliberately small algebra of neural layers plus a few standard RL learners is more useful than any single megamodel architecture—provided each layer is implemented, demable, checkpointable, and attachable to real agents.

MathNN is that algebra inside _AugmentedIntelligence.

1.3 Contributions

  1. A unified description of MathNN layers (dense, Conv2D, pool, LayerNorm, RNN, LSTM, attention, curiosity attention).
  2. A formal view of curiosity attention as attention plus novelty-shaped scores, with episodic memory and ICM-style prediction error.
  3. A mapping from layers → gaming RL roles (perception, policy, exploration, multi-agent routing).
  4. A capability statement: what MathNN can train, demo, and deploy today, and what it intentionally defers to external stacks.
  5. An integration sketch for GamingRL (gaming_fps) and related agents via RLAgentNet / MathNNAgentBridge.

1.4 Paper style note

We adopt the structure and rhetorical cadence of modern architecture papers (problem → primitive → composition → applications → limits), not the marketing style of product documentation. Equations use standard notation; implementation names use MathNN identifiers (_Conv2D, _Attention, …).

2. Background

2.1 Deep RL for games (brief)

Value-based methods (DQN and variants) learn (Q_\theta(s,a)) from replay. Policy-gradient and actor–critic methods learn (\pi_\phi(a\mid s)) with a critic. Dueling architectures factor (Q = V + A). Exploration bonuses (ICM, RND, episodic novelty) reshape reward when extrinsic signals are sparse—common in open-world and FPS settings.

2.2 Attention (brief)

Scaled dot-product attention:

[ \mathrm{Attention}(Q,K,V) = \mathrm{softmax}!\left(\frac{QK^\top}{\sqrt{d}}\right)V ]

Multi-head attention runs this in parallel subspaces and concatenates. Causal masks restrict attention to past positions for autoregressive decoding.

2.3 What gaming needs that language does not

Gaming need Typical structure Language-default
Local visual features Spatial neighborhoods Token order
Button combos Discrete multi-label / multi-head Single next-token
Sparse rewards Intrinsic curiosity Dense CE loss
Live control risk Shields / dry-run Offline eval only
Short-horizon physics Recurrence / world models Long context windows

3. MathNN system overview

MathNN is a native C++ neural-network subsystem compiled into _AugmentedIntelligence. It is designed for small-to-medium in-process models where the application needs local learning without sending every gradient update to TensorFlow, CUDA, or a remote policy server.

Module Role
MathNN (_NN) Dense stack: activations, losses, optimizers, dropout, save/load
MathNNLayers Conv2D, MaxPool2D, LayerNorm, RNN, LSTM, Attention, CuriosityAttention
MathNNArchitectures Catalog demos: transformer-lite, vision-CNN, dueling-DQN, world model, MoE, PINN, …
MathNNDevice CPU / GPU / NPU / APU / remote placement hooks
MathNNLearning Transfer, distill, federate, curriculum-scale helpers
RLNeural / RLAgentNet DQN family, actor–critic, bandit, hierarchical, reward/world models
MathNNAgentBridge Per-agent architecture recommendations (e.g. gaming_fps → actor–critic)

Design principle. Prefer correct, demable, attachable blocks over incomplete reimplementations of every SOTA paper. Where full training is not local (e.g. large transformers), MathNN exposes forward helpers and routes heavy work outward.

4. Layer inventory: the building blocks

We describe each family as a role in a gaming stack, then the MathNN realization.

4.1 Dense layers (_NN)

A dense layer implements

[ h = \sigma(Wx + b) ]

with analytic backpropagation. MathNN supports activations (\sigma \in {\mathrm{Linear}, \mathrm{ReLU}, \mathrm{LeakyReLU}, \mathrm{Sigmoid}, \mathrm{Tanh}, \mathrm{GELU}, \mathrm{Softmax}}), losses ({\mathrm{MSE}, \mathrm{CE}, \mathrm{BCE}}), and optimizers ({\mathrm{SGD}, \mathrm{Momentum}, \mathrm{Adam}}), plus inverted dropout and fused softmax/CE and sigmoid/BCE gradients.

Gaming role. Default Q-head and small policy networks:

state_dim → hidden ReLU → hidden ReLU → action_dim Linear

This is the backbone of _NNPolicy / _NNQLearner for discrete action spaces (e.g. GamingRL’s 24 discrete actions over a 32-D feature state).

4.2 Convolution and pooling (_Conv2D, _MaxPool2D)

Convolution is the natural inductive bias for image patches:

[ (Y_{c'}){u,v} = \sum ]} W_{c',c,i,j}\, X_{c,\,u\cdot s+i,\,v\cdot s+j} + b_{c'

MathNN uses channels-first tensors (C \times H \times W), with forward and backward passes and parameter steps. Max pooling caches argmax for backprop.

Gaming role. Screen crops, minimaps, HUD glyphs, aim-region features—before flattening into the FPS state vector or a vision-CNN tower.

4.3 Layer normalization (_LayerNorm)

[ \hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \varepsilon}},\quad y_i = \gamma_i \hat{x}_i + \beta_i ]

Stabilizes deep residual stacks and transformer-style blocks used in MathNN architecture demos.

Gaming role. Stable features under changing brightness, score scales, or multi-source concatenations (vision + HUD + audio threat).

4.4 Recurrent cells (_RNNCell, _LSTMCell)

Elman RNN and LSTM cells process temporal streams (x_t) with hidden (and cell) state. MathNN exposes sequence training with BPTT for the Elman cell and LSTM training paths for short sequences.

Gaming role. Short motion history (recoil, strafe patterns, enemy track stability), audio-threat sequences, and partial-observability memory when full transformers are unnecessary.

4.5 Attention (_Attention)

MathNN implements:

  • Scaled dot-product attention over flattened (Q,K,V \in \mathbb{R}^{T \times d})
  • Multi-head attention with optional causal masks

This is the same primitive popularized by Attention Is All You Need, exposed as a local forward/inference operator for small (T) and (d).

Gaming role. Soft selection among:

  • multiple enemy tracks,
  • skill options in hierarchical RL,
  • memory slots,
  • candidate waypoints,

without hard-coding “always track nearest.”

4.6 Curiosity attention (_CuriosityAttention) — the RL-native twist

Standard attention scores similarity. Agents also need novelty. MathNN’s curiosity attention uses:

[ s_{ij} = \frac{q_i \cdot k_j}{\sqrt{d}} + \beta \, n_j ]

where (n_j) is a novelty score for key (j) (e.g. (1 - \cos(q, k_j)) or distance to episodic prototypes). Softmax then shifts mass toward under-explored keys while remaining a valid attention distribution.

Supporting operators:

Operator Meaning
PredictionError ICM-style (| \hat{s}' - s' |^2) surprise
EpisodicNovelty (1 - \max) cosine similarity to memory
EpisodicMemory bounded store of embeddings; Observe / ScoreKeys
ShapeReward (r \leftarrow r + \lambda \cdot r_{\mathrm{int}})

Gaming role. Exploration when kills and objectives are rare: visit new map regions, re-attend to surprising tracks, avoid pure exploitation of the nearest enemy.

If classic attention is “what is similar,” curiosity attention is “what is similar *and still surprising.”

5. Composition: architectures and RL learners

Layers alone are inert; MathNN composes them into learners.

5.1 Architecture catalog (demos)

MathNNArchitectures registers a broad catalog including:

transformer, lstm, conv1d, vision-cnn, unet, crnn, clip, gnn, vae, gan, flow, diffusion, ebm, neural-ode, pinn, moe, dueling-dqn, actor-critic, bandit, worldmodel, reward-model, hierarchical-rl, federated, …

Many entries are smoke demos that prove wiring and shapes; production training for large generative models remains out of MathNN’s charter.

5.2 In-process RL modes (RLNeural / RLAgentNet)

Mode Idea Gaming use
dqn (Q(s,a)) + replay + target net Discrete combat / menu actions
dueling-dqn (V(s)+A(s,a)) Value sharing across buttons
actor-critic (\pi) + (V) Continuous-ish aim deltas + discrete fire
bandit Contextual arm selection Weapon / loadout choice
hierarchical Manager + worker DQN Skill macros (push, hold, rotate)
multihead-dueling Skill-group advantages Action groups (move / aim / utility)
worldmodel Predict (\Delta s), reward; curiosity Sparse open-world
reward-model Preference BCE + blend Human “good/bad” aim feedback

5.3 The closed loop

 World / Game Client
        │  encode
        ▼
  state ∈ R^{d_s}     (+ optional image tensor)
        │
        ├─ Conv / RNN / Attention (optional towers)
        ▼
  MathNN policy / Q-head
        │  ε-greedy or stochastic
        ▼
  discrete / structured action
        │  execute (or dry-run)
        ▼
  reward, next_state, terminal
        │  replay
        ▼
  TD / policy gradient update
        │
        └─ checkpoint (.nn + .meta [+ .rm/.wm])

6. Relevance to Gaming RL

6.1 GamingRL FPS stack

_GamingRL exposes:

  • State size (d_s = 32) engineered features (crosshair geometry, target tracks, health/ammo norms, cover, audio threat, NLU curiosity blend, …)
  • Discrete actions (n_a = 24) (move, aim buckets, fire, reload, utility, …)
  • Algorithms: heuristic → Q-table → linear Q → SARSA → REINFORCE → MLP → DQN → remote/auto
  • Opt-in MathNN path when RLAgentNet enables gaming_fps (default bridge recommendation: actor–critic)

Perception structures (FPSTarget, FPSPerception) feed features; MathNN scores actions; transitions call MathNNAgentBridge::ObserveTransition.

6.2 Layer → FPS problem map

FPS problem MathNN layer / mode
Crop / minimap pixels _Conv2D + _MaxPool2D
Recoil / track history _RNNCell / _LSTMCell
Multi-enemy soft select _Attention over track embeddings
Explore unseen angles _CuriosityAttention + episodic memory
Discrete combat policy Dense DQN / dueling
Continuous aim + fire Actor–critic
Human coaching Reward model (rl net reward gaming_fps good)
Sparse objective World model + curiosity scale

6.3 Beyond FPS: sibling game domains

Domain Typical MathNN profile
Minecraft DQN Dueling DQN + optional plan/executive steps; wall-guards
Space Engineers Same RL tech map as Minecraft bridge agents
Strategy / grid Actor–critic or hierarchical skills
Curriculum gaming_rl Synthetic navigation smoke before live clients
Driving RL Dueling DQN + strong safety shields

6.4 Why layers beat “one big net” here

  1. Debuggability. If aim fails, inspect vision tower vs Q-head vs reward separately.
  2. Partial training. Freeze conv features; train only the policy head online.
  3. Safety composition. Heuristic shields can veto neural actions regardless of Q-max.
  4. Device fit. Dense 32→64→24 nets train on CPU between frames; giant transformers do not.

6.5 Attention is not all you need for games—but it is not optional either

Self-attention alone does not solve:

  • spatial locality (use conv),
  • credit assignment over long sparse rewards (use curiosity / world models),
  • legal action masks (environment contract),
  • ethical/live control (shields).

Yet without attention-like soft selection, multi-target FPS policies collapse to brittle heuristics (“always nearest”). MathNN’s stance is compositional:

Attention is what you need when choices compete; convolutions when space matters; recurrence when time matters; dense heads when actions must be scored; curiosity when rewards hide.

That is the content of Layers Are What You Need.

7. What is MathNN capable of?

7.1 Capabilities (positive claims)

MathNN can:

  1. Train dense networks end-to-end with Adam/SGD, checkpoint them, and run inference in-process.
  2. Implement trainable Conv2D, MaxPool, LayerNorm with backprop (gradient-check recommended before critical use).
  3. Train short RNN sequences via BPTT; run LSTM forward (and sequence training paths where enabled).
  4. Run multi-head attention (and causal variants) for small sequences.
  5. Bias attention with novelty via curiosity attention + episodic memory + reward shaping.
  6. Serve as local RL brains: DQN, dueling DQN, actor–critic, bandits, hierarchical and multi-head variants.
  7. Attach per agent (gaming_fps, minecraft, driving, math tutoring fields, life domains, …) through RLAgentNet.
  8. Demo dozens of architectures for smoke tests (nn arch list, nn arch demo …).
  9. Support math tutoring RL across ~59 mathematical field domains (synthetic curricula).
  10. Blend human preference and world-model curiosity into the same agent registry.
  11. Place compute across CPU/GPU/NPU/APU hooks where configured.
  12. Integrate learning ops: transfer, distill, federate-scale helpers in MathNNLearning.

7.2 Non-capabilities (honest limits)

MathNN is not:

Not a… Use instead
Full LLM chat stack LLM orchestration / local model registry
Production large-CNN trainer CUDA / TF / PyTorch
ONNX-complete interoperability layer External export pipelines
Guarantee of superhuman game bots Careful state/reward design + evaluation
License to ignore safety Shields, dry-run, gated deployment
Replacement for game engine physics Simulators + domain adapters

LSTM/attention training coverage is intentionally lighter than dense DQN paths; heavy sequence models should use remote trainers when fidelity matters.

7.3 Capability matrix (condensed)

Area Status
Dense train/infer Production-ready for small nets
Conv / pool / LN train Implemented; verify grads
RNN BPTT Available for short sequences
Attention forward Available
Curiosity attention Available + episodic memory
DQN family Available with replay/target
Actor–critic Available
World / reward models Available as aux heads
GamingRL attach Available (gaming_fps)
Mega-transformer pretrain Out of scope

7.4 Operator surface (for reproducibility)

nn status | nn demo | nn conv-demo | nn rnn-demo | nn attn-demo
nn arch list | nn arch demo transformer | nn arch demo dueling-dqn
nn math domains | nn math train math_algebra 200
rl net enable gaming_fps
rl net mode gaming_fps actor-critic
rl net wm gaming_fps on
rl net curiosity gaming_fps 0.05
rl net reward gaming_fps good
rl net save gaming_fps checkpoints/gaming_fps_mathnn.txt
minecraft dqn net on | minecraft dqn train 1000

8. Experimental methodology (in-repo)

We recommend a ladder of evidence, not a single leaderboard claim:

  1. Unit demos — XOR dense, conv shapes, RNN loss drop, attention shapes (nn *demo).
  2. Synthetic curriculum — rl curriculum smoke gaming_rl / CartPole / math fields.
  3. Offline logs — GamingRL CSV transitions, dashboards, checkpoint reload.
  4. Dry-run / assist — neural suggestions without full bot authority.
  5. Short train — hundreds to low thousands of steps; inspect failure modes.
  6. Shielded live — only after observe/reward sanity checks.

Success metrics should be domain-native (time-to-target, damage taken, walls hit, objective progress), not only TD loss.

9. Discussion

9.1 Relation to “Attention Is All You Need”

Transformers showed that attention can replace recurrence for large-scale transduction. Gaming RL reintroduces physics, agency, and control risk. MathNN therefore imports the attention primitive without accepting the slogan that attention alone is sufficient. The correct slogan for this regime is compositional: layers are what you need.

9.2 Relation to pure end-to-end pixel RL

Classic Atari DQN learns from pixels with large CNNs. MathNN can host small conv stacks, but GamingRL currently emphasizes engineered 32-D features for latency and interpretability—optionally augmented by vision towers. This is a deliberate engineering choice for a desktop-integrated agent, not a claim that pixels are useless.

9.3 Safety and ethics

Learned gaming policies can:

  • violate game terms of service if used as cheats,
  • harm user experience if they seize input,
  • transfer poorly and act unsafely on desktop control agents.

MathNN deployments must remain behind assist modes, dry-run, and explicit enablement. Curiosity does not justify unrestricted exploration in real-world control domains.

9.4 Future work

  1. End-to-end trainable multi-head attention with residual FFN blocks at gaming scales.
  2. Shared visual backbone across Minecraft / FPS / desktop with domain-specific heads.
  3. Tighter coupling of curiosity attention to GamingRL multi-target lists (not only scalar curiosity features).
  4. Automated ablation: conv vs MLP vs attention for fixed FPS logs.
  5. Federated sharing of non-sensitive policy adapters across machines via existing mesh/federate hooks.

10. Conclusion

We described MathNN as an in-process neural layer system for _AugmentedIntelligence, with special emphasis on gaming reinforcement learning. The central claim is architectural and practical:

For local game agents, you need layers—dense scorers, spatial filters, temporal memory, attention over candidates, and curiosity over novelty—composed into standard RL learners and guarded by safety rails.

Attention remains indispensable when soft selection matters; it is not a substitute for perception structure, exploration bonuses, or control hygiene. MathNN’s capability is exactly that of a serious local learning kernel: train small models, attach them to agents like gaming_fps and Minecraft DQN, experiment with modern architectural ideas in-process, and leave megamodel pretraining to systems built for that scale.

Layers are what you need—and MathNN is how this codebase provides them.

Acknowledgments

MathNN and Gaming RL sit within the larger _AugmentedIntelligence architecture (curriculum orchestrator, bridges, device placement, life/math domains). Implementation identifiers referenced herein (_Attention, _CuriosityAttention, _GamingRL, RLAgentNet) are the system of record.

References (selected)

  1. Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.
  2. Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. Nature.
  3. Wang, Z., et al. (2016). Dueling Network Architectures for Deep Reinforcement Learning. ICML.
  4. Konda, V., & Tsitsiklis, J. (2000). Actor-Critic Algorithms. NeurIPS.
  5. Pathak, D., et al. (2017). Curiosity-driven Exploration by Self-supervised Prediction. ICML (ICM).
  6. Schaul, T., et al. (2015). Prioritized Experience Replay. ICLR.
  7. Ba, J., Kiros, J., & Hinton, G. (2016). Layer Normalization.
  8. Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation.
  9. LeCun, Y., et al. (1998). Gradient-Based Learning Applied to Document Recognition (CNNs).
  10. Sutton, R., & Barto, A. (2018). Reinforcement Learning: An Introduction (2nd ed.).

Appendix A — Layer cheat sheet

Class Train local? Primary gaming use
_NN dense Yes Q / policy heads
_Conv2D Yes (verify) Frame / minimap features
_MaxPool2D Yes Downsample spatial maps
_LayerNorm Yes Stabilize towers
_RNNCell Yes (BPTT) Short history
_LSTMCell Forward + seq paths Longer short-term memory
_Attention Forward Soft multi-target / memory
_CuriosityAttention Forward + novelty utils Exploration routing

Appendix B — GamingRL × MathNN integration points

Symbol / API Role
_GamingRL::kStateSize = 32 Feature dimension
_GamingRL::kDiscreteActionCount = 24 Action cardinality
_RLAgentNet::Enabled("gaming_fps") Neural path switch
_MathNNAgentBridge::SelectAction Policy query
_MathNNAgentBridge::ObserveTransition Learning update
NLU curiosity feature blend Intrinsic signal into state/reward
Modes Assist / Bot / Train / Eval Deployment ladder

Appendix C — One-sentence summaries

  • Dense layers score actions.
  • Convolutions see space.
  • Recurrence remembers the last second.
  • Attention chooses among many.
  • Curiosity attention chooses among the new.
  • DQN / actor–critic turn scores into play.
  • Shields keep play safe.

That composition—not any single slogan—is what MathNN offers gaming RL.

Document path: docs/MathNN_Layers_Are_What_You_Need.md
Related: docs/MathNN.md, docs/MathNNFields.md, docs/RLCurriculum.md, GamingRL.hpp

Generated from the project markdown docs on 2026-07-24. This is a static, self-contained site.