RL Domains And Systems

This document is the map of the reinforcement-learning surface in _AugmentedIntelligence. It explains the shared RL infrastructure, the domain modules, the training/deployment ladder, and the commands used to inspect or run the systems.

For MathNN details, see docs/MathNN.md. For the live curriculum catalog, see docs/RLCurriculum.md.

Big Picture

The RL stack has three layers:

1. Generic RL substrate: state/action spaces, environments, replay buffers, algorithms, exploration, planning, neural DQN, distributed RL, and deployment gates.
2. Domain modules: Minecraft, FPS gaming, strategy games, driving, knowledge, parenting, political-news triage, real-world hooks, and curriculum smoke environments.
3. Safety/deployment controls: simulation-first training, offline replay, shadow mode, gated live execution, safe-action shields, checkpoints, and promotion rules.

The preferred path for any domain is:

Sim -> Offline -> Shadow -> GatedLive -> Live

Do not skip directly to live actuation. High-stakes domains should remain in simulation, offline analysis, or human-reviewed advisory mode unless there is a separate validated safety layer.

Shared RL Infrastructure

System	Main files	Purpose
RL core	RLCore.hpp/.cpp	Defines RLEnv, spaces, transitions, replay buffers, prioritized replay, RNG, and checkpoint registry.
RL algorithms	RLAlgorithms.hpp/.cpp	Implements tabular and lightweight value-based learners; declares remote deep-RL trainers.
RL trainer	RLFeatures.hpp/.cpp	Provides the unified trainer loop and umbrella rl ... command dispatcher.
Exploration and safety	RLExplore.hpp/.cpp	Greedy, epsilon-greedy, Boltzmann, UCB, Thompson sampling, curiosity, reward shaping, and safe action shielding.
Neural DQN	RLNeural.hpp/.cpp	In-process DQN using MathNN online/target Q-networks and replay.
Agent neural registry	RLAgentNet.hpp/.cpp	Opt-in MathNN DQN registry for named agents such as Minecraft, driving, FPS gaming, and strategy.
Planning	RLPlanning.hpp/.cpp	Generic MCTS/UCT planner plus world-model interface.
Distributed RL	RLDistributed.hpp/.cpp	Actor/learner transition push, parameter publishing, PBT, RLAIF scoring, and offline dataset loading.
Differential RL	RLDifferential.hpp/.cpp	Physics/ODE environments, CartPole, OU exploration, stability tripwire, and reachability safety checks.
Real-world RL	RLReal.hpp/.cpp, RLRealEnvs.hpp/.cpp	Real environment registry, LLM-routing bandit, and hook-driven environment presets.
Deployment	RLDeployment.hpp/.cpp	Sim/offline/shadow/gated/live stage gate and action-execution choke point.
Curriculum	RLCurriculumOrchestrator.hpp/.cpp	Catalog of RL domains and safe smoke environments.

Algorithms

Local C++ Algorithms

These run locally without a tensor/autograd backend:

• Q-learning
• Double Q-learning
• SARSA(lambda)
• Expected SARSA
• Dyna-Q
• Linear-Q function approximation
• UCB bandits
• Thompson-sampling bandits
• Epsilon-greedy and Boltzmann exploration
• Count-based intrinsic curiosity
• MCTS/UCT planning
• MathNN-backed in-process DQN

Use these for smoke tests, tabular policies, smaller state vectors, action selection, and local DQN experiments.

Remote Or Heavy Deep-RL Algorithms

These are declared in the deep-RL interface and route to a remote policy server or tensor/autograd backend for training:

• PPO
• A2C
• SAC
• TD3
• DDPG
• IQL
• CQL
• Rainbow DQN
• C51
• QR-DQN
• Dreamer
• GAIL
• Behavior Cloning
• ICM
• RND
• DIAYN

Use these when the policy needs large neural networks, continuous-control training, offline deep RL, imitation learning, or heavy world-model training.

Core Commands

General RL:

rl status
rl seed <n>
rl registry
rl registry best <agent>
rl deep train <algo> <env> <state_dim> <action_dim> [continuous]
rl push <serialized-transition>

Neural per-agent policy registry:

rl net status
rl net enable <agent>
rl net disable <agent>
rl net save <agent> <path>
rl net load <agent> <path>

Real-world/deployment:

rl real status
rl real envs
rl real route
rl real stage <agent> <sim|offline|shadow|gatedlive|live>
rl real gate <agent> <action>
rl real promote <agent> <version>

Curriculum:

rl curriculum status
rl curriculum plan
rl curriculum next
rl curriculum domains [category]
rl curriculum domain <id>
rl curriculum envs
rl curriculum smoke all [episodes] [max_steps]
rl curriculum smoke <domain_id> [episodes] [max_steps]
rl curriculum export [path]

Short aliases:

rl domains [category]
rl domain <id>
rl sim game smoke [episodes] [max_steps]
rl sim physics smoke [episodes] [max_steps]
rl sim robot smoke [episodes] [max_steps]
rl sim energy smoke [episodes] [max_steps]
rl sim windows smoke [episodes] [max_steps]

Domain Modules

Minecraft DQN

Files:

• MinecraftDQN.hpp/.cpp
• MinecraftSpigotBridge.hpp/.cpp
• MinecraftClientController.hpp/.cpp
• minecraft_spigot_plugin/

Purpose:

Minecraft DQN observes player/server state through the Spigot bridge, selects actions, computes rewards, logs transitions, and can execute actions through the server bridge, local client input, or both.

State/action shape:

• State size: 48 features.
• Action count: 26 discrete actions.
• Features include position, health, food, XP, time, yaw/pitch, nearby entities, inventory/tool flags, blocked directions, target block information, world, biome, game mode, selected item, and bridge observation ID.

Main commands:

minecraft dqn status
minecraft dqn player <selector-or-name>
minecraft dqn observe
minecraft dqn train [interval_ms]
minecraft dqn eval
minecraft dqn off
minecraft dqn step
minecraft dqn dry-run
minecraft dqn qvalues
minecraft dqn explain [top_n]
minecraft dqn control server|client|both
minecraft dqn wall-guard on|off|status
minecraft dqn unstuck on|off|status
minecraft dqn interval <milliseconds>
minecraft dqn epsilon <0.0-1.0>
minecraft dqn reward <value> [terminal]
minecraft dqn save [path]
minecraft dqn load [path]
minecraft dqn net on|off|status|save <path>|load <path>

Client-control commands:

minecraft dqn client status
minecraft dqn client focus on|off
minecraft dqn client foreground on|off
minecraft dqn client hold <milliseconds>
minecraft dqn client mine-hold <milliseconds>
minecraft dqn client use-hold <milliseconds>
minecraft dqn client mine
minecraft dqn client place
minecraft dqn client slot <1-9>
minecraft dqn client look-up
minecraft dqn client look-down
minecraft dqn client action <id>
minecraft dqn client release
minecraft dqn client title <window-title-fragment>

Recommended use:

Start in observe and dry-run, enable wall-guard and unstuck, then train in short sessions. Use minecraft dqn net on when you want the shared MathNN DQN path instead of the compact built-in model.

Gaming RL FPS

Files:

• GamingRL.hpp/.cpp
• docs/GamingRL.md
• docs/GamingRLFPS.md
• docs/GameModBridges.md

Purpose:

FPS/game RL observes target, HUD, movement, resource, objective, audio, and threat signals; chooses discrete actions; records transitions; supports dry-run, bot, assist, train, eval, and imitation workflows.

State/action shape:

• State size: 32 features.
• Action count: 24 discrete actions.
• Algorithms: heuristic, random, Q-table, Linear-Q, SARSA, REINFORCE, MLP, DQN, remote, and auto.

Main commands:

gaming rl status
gaming rl features
gaming rl systems
gaming rl skills
gaming rl explain
gaming rl why
gaming rl mode off|assist|bot|train|eval
gaming rl algorithm heuristic|qtable|linear_q|sarsa|reinforce|mlp|dqn|remote|auto
gaming rl train on|off|episode start|episode end
gaming rl eval [episodes]
gaming rl selfplay on|off
gaming rl adapter set <name>
gaming rl adapter status
gaming rl perception <request>
gaming rl input hold <milliseconds>
gaming rl input mouse <max_pixels_per_tick>
gaming rl dry-run on|off
gaming rl curriculum status
gaming rl curriculum start [phase]
gaming rl curriculum next
gaming rl replay summary [path]
gaming rl replay train [path]
gaming rl bc train [epochs]
gaming rl reward good|bad|<number> [note]
gaming rl dashboard
gaming rl heatmap
gaming rl bridge spec
gaming rl checkpoint tag|list|rollback

Recommended use:

Only use on private/local games and servers. Start with dry-run and replay review. Do not use it to bypass anti-cheat or automate public multiplayer play.

Strategy Gaming RL

Files:

• GamingStrategyRL.hpp/.cpp
• docs/GameModBridges.md
• game_mods/civilization_mod/README.md

Purpose:

Strategy RL supports higher-level game decisions for League of Legends style observations and Civilization turn-strategy decisions.

State/action shape:

• Feature size: 32.
• Maximum actions: 16.
• Game modes: League of Legends and Civilization.

Capabilities:

• Build League observations from FPS/game perception plus champion detections.
• Build Civilization observations from turn, city, unit, economy, pressure, and victory-focus state.
• Select strategy actions.
• Observe transitions.
• Train batches.
• Save/load checkpoints.

Recommended use:

Use as a planning/policy layer above game-specific perception. For Civilization, it is better suited to turn-level advisory or private mod integration than direct real-time control.

Driving RL

Files:

• DrivingRL.hpp/.cpp
• docs/DrivingRL.md

Purpose:

Driving RL is a simulation-first subsystem for lane keeping, lead following, intersections, traffic controls, pedestrian yielding, weather/visibility, route progress, comfort, road-sign model response, and emergency braking.

State/action shape:

• State size: 32 features.
• Action count: 12 discrete actions.
• Algorithms: heuristic, Q-table, local MathNN DQN, auto.
• Modes: off, assist, train, eval.
• The active trained object-detection model can feed road-sign labels such as stop, yield, speed limit, traffic-light color, warning, crosswalk, and no-entry into the RL state vector.

Main commands:

driving rl status
driving rl features
driving rl actions
driving rl explain
driving rl dashboard
driving rl on|assist|train|eval|off
driving rl algorithm heuristic|qtable|dqn|auto
driving rl dqn on|off|status|save <path>|load <path>
driving rl scenario <name> [ticks]
driving rl curriculum status|start [phase]|next|phase <name>
driving rl adapter set <name>
driving rl adapter status
driving rl shield on|off
driving rl dry-run on|off
driving rl reward good|bad|<value>
driving rl checkpoint tag|list|rollback
driving rl save
driving rl load

Driving-law training commands:

driving rl law status
driving rl law mysql init
driving rl law seed <STATE>
driving rl law load [STATE]
driving rl law list [N]
driving rl law add <federal|state> <jurisdiction> <trigger> <required_action> <severity> <citation> <summary>
driving rl law on|off
driving rl law train [all|scenario] [ticks]
driving rl law explain

Recommended use:

Keep this in simulation/review. Do not wire it directly to a real vehicle actuator. Use dry-run on and shield on by default.

Knowledge RL

Files:

• KnowledgeAcquisitionRL.hpp/.cpp
• docs/KnowledgeRL.md

Purpose:

Knowledge RL learns which claims to present, review, label, train, or evaluate. It tracks claim text, epistemic labels, source trust, conflicts, confidence, replay, and policy state.

Modes:

• off
• acquire
• train
• eval

Main commands:

knowledge on
knowledge off
knowledge status
knowledge claim <statement>
knowledge next
knowledge true|false|depends|unknown
knowledge label true|false|depends|unknown <claim>
knowledge conflicts
knowledge trust <source_or_claim> <0.0-1.0>
knowledge save
knowledge load
knowledge save mysql
knowledge load mysql

knowledge rl status
knowledge rl observe
knowledge rl step
knowledge rl train replay [episodes]
knowledge rl eval
knowledge rl replay [limit]
knowledge rl benchmark [episodes]
knowledge rl explain
knowledge rl use
knowledge rl stats
knowledge rl graphs [directory]

Recommended use:

Use Knowledge RL to prioritize review and labeling, not as proof. It should surface conflicts, uncertainty, and source-trust issues before claims are used downstream.

Parenting / Child Psychology RL

Files:

• ChildPsychologyParentingRL.hpp/.cpp
• docs/CHILD_PSYCHOLOGY_PARENTING_RL.md

Purpose:

This subsystem is a safety-bounded parenting coaching simulator. RL is limited to offline simulation, shadow-mode evaluation, and parent-facing suggestions. It is not a diagnostic or clinical decision system and never autonomously messages, disciplines, or directs a child.

Main commands:

parenting rl status
parenting rl domains
parenting rl safety
parenting rl actions
parenting rl curriculum
parenting rl observe <de-identified situation text>
parenting rl evaluate <situation text>
parenting rl train [episodes] [max_steps]
parenting rl export <path>
parenting rl reset

Recommended use:

Keep observations de-identified. Treat output as coaching suggestions for an adult caregiver, not medical, psychological, or emergency guidance.

Political News RL

Files:

• PoliticalRL.hpp/.cpp
• PoliticalRLCore.hpp
• lua_scripts/Political-News-RL.lua

Purpose:

Political RL is a local political-news triage agent. It uses a UCB1 feed portfolio bandit to choose RSS feeds, rewards novelty and relevance, extracts legislative bill mentions, and flags volume spikes.

Main commands:

politics status
politics tick [n]
politics brief [n]
politics bills
politics save
politics load

Recommended use:

Use for local analysis and triage only. Rewards are novelty, coverage, and relevance, not engagement or persuasion. Keep source links/provenance attached.

Real-World RL Environments

Files:

• RLReal.hpp/.cpp
• RLRealEnvs.hpp/.cpp
• RLDeployment.hpp/.cpp
• RL_REALWORLD.md

Purpose:

Real-world RL registers RLEnv implementations that can observe real systems and optionally actuate through host hooks. The deployment gate decides whether any action may execute.

Built-in environment:

• llm_routing: contextual bandit over the NetLLM backend pool. This is the recommended first live RL environment because it has no physical actuation.

Hook-driven presets:

• Desktop automation
• Perception tuning
• Home IoT
• Fleet remediation
• ROS robot

Recommended use:

Start with rl real route or llm_routing. Keep physical or operational domains in sim, offline, or shadow until checkpoints are promoted and action gates pass.

Differential / Physics RL

Files:

• RLDifferential.hpp/.cpp
• MathControl.hpp/.cpp
• MathODE components

Purpose:

Differential RL connects RL to physical dynamics and control theory.

Capabilities:

• CartPole RLEnv with RK4-integrated ODE dynamics.
• Ornstein-Uhlenbeck exploration noise for continuous control.
• Return-stability tripwire.
• Reachability checks and control-barrier style safety gates.

Recommended use:

Use for physics simulator smoke tests, robotics-like control experiments, and safe-RL verification before connecting policies to real actuators.

Distributed RL / RLAIF / Offline RL

Files:

• RLDistributed.hpp/.cpp

Purpose:

Distributed RL lets multiple actors push transitions to a learner, publish parameters back to actors, run population-based training, load offline datasets, and use LLM-based reward/preference scoring.

Capabilities:

• Transition serialization and parsing.
• Actor-to-learner transition push.
• Incoming learner queue.
• Parameter publishing/pulling.
• Population-based training reports and exploit/explore config mutation.
• RLAIF trajectory scoring and pairwise preference scoring.
• Offline dataset loading into replay buffers.

Recommended use:

Use when training data is generated across machines or domains. Keep the reward rubric explicit when using RLAIF, and audit LLM-scored rewards before treating them as training signal.

Curriculum Domain Catalog

RLCurriculumOrchestrator exposes the broad catalog and safe smoke tests. Run:

rl curriculum domains
rl curriculum domain <id>

Current catalog IDs include:

Domain ID	Category	Smoke	Purpose
knowledge_rl	knowledge	yes	Claim triage, critique, and memory-consolidation checks.
gaming_rl	gaming	yes	Navigation, stuck recovery, resource timing, and reward shaping.
game_simulator	gaming	yes	Deterministic grid navigation and obstacle handling.
physics_simulator	simulation	yes	Force and trajectory control using local physics dynamics.
robot_simulator	robotics	yes	Differential-drive navigation with collision and boundary penalties.
robotics_gazebo	robotics	yes	ROS/Gazebo navigation and manipulation staging.
ros_robot	robotics	no	Real ROS/hardware bridge placeholder registered at Sim.
cybersecurity_soc	security	yes	Alert triage, evidence collection, ranking, and escalation.
windows_ops	operations	yes	Ticket triage, patching, diagnostics, rollback planning, and audits.
software_build	software	yes	Build, test, lint, bisect, retry, rollback, and documentation choices.
research_science	research	yes	Literature review, hypothesis selection, experiment planning, and replication.
healthcare_triage	healthcare	yes	Intake, guideline lookup, risk flags, and human escalation.
education_tutoring	education	yes	Hinting, examples, retrieval practice, review, and assessment timing.
math_algebra	mathematics	yes	Algebra step selection for MathNN DQN tutoring smoke/training.
math_geometry	mathematics	yes	Geometry theorem and measurement tutoring for MathNN.
math_trigonometry	mathematics	yes	Trigonometry identity and ratio tutoring for MathNN.
math_probability_statistics	mathematics	yes	Probability/stats workflow tutoring for MathNN.
math_linear_algebra	mathematics	yes	Linear algebra matrix/system tutoring for MathNN.
math_finite_math	mathematics	yes	Finite math (sets, combinatorics, LP, graphs) for MathNN.
child_parenting	parenting	yes	Routines, de-escalation, boundaries, choices, and safety escalation.
finance_portfolio	finance	yes	Allocation, risk balancing, rebalance timing, and drawdown-aware simulation.
stock_research	finance	yes	RSS reading, sentiment checks, fundamentals review, and watchlist updates.
product_recommendation	commerce	yes	Relevance, diversity, safety filtering, preference questions, and audits.
sales_pricing	commerce	yes	Offer timing, guarded discounts, follow-up cadence, and churn handling.
supply_chain	operations	yes	Reorder, supplier, routing, buffer stock, and disruption response.
energy_grid	infrastructure	yes	Load forecasting, storage dispatch, outage response, and demand shifting.
legal_research	legal	yes	Statute/case lookup, issue spotting, citation checks, and attorney escalation.
civic_policy	government	yes	Stakeholder tradeoffs, sequencing, equity checks, review, and audits.
manufacturing_quality	manufacturing	yes	Inspection, calibration, sampling, rework, and line-stop recommendations.
media_workflow	creative	yes	Ingest, tagging, edit queues, rendering, captioning, and publishing review.
personal_productivity	personal	yes	Prioritization, scheduling, focus blocks, breaks, reminders, and review.
smart_home	iot	yes	Comfort, energy, notifications, fallback-safe automation, and audit policy.
fleet_maintenance	operations	yes	Inspection, preventive service, parts staging, rerouting, and downtime reduction.
call_center	customer_ops	yes	Routing, response suggestions, clarifying questions, escalation, and QA.
llm_routing	llm	no	Backend selection by latency, quality, cost, and task fit.

The generic smoke domains use synthetic observations and safe action families. They do not touch external services.

Training And Evaluation Workflow

Use this sequence for any new RL domain:

1. Define the state vector and normalize every feature.
2. Define the action library and legal-action rules.
3. Define reward, terminal, and truncation conditions.
4. Run synthetic or simulator smoke tests first.
5. Log transitions and inspect replay before training long sessions.
6. Start with tabular/Q-table or local MathNN DQN.
7. Add neural or remote deep-RL training only after the domain loop is correct.
8. Evaluate with exploration disabled.
9. Save and tag checkpoints.
10. Promote only if evaluation clears the threshold and safety review passes.
11. Move to shadow mode before any live actuation.
12. Use gated live mode before live mode.

Useful first commands:

rl curriculum smoke all 8 96
rl curriculum next
rl net status
rl real envs

Safety Model

RL systems in this project should be treated as action-selection systems, not general-purpose truth engines. The safe pattern is:

• Simulation first.
• Offline replay second.
• Shadow mode before actuation.
• Gated live before live.
• Promoted checkpoints only.
• Safe-action shields always on for high-impact domains.
• Kill switch and manual override available.
• Human approval for healthcare, parenting, finance, legal, infrastructure, driving, robotics, cybersecurity, and desktop/Windows automation.

Documentation Map

Detailed docs:

• docs/MathNN.md
• docs/RLCurriculum.md
• docs/KnowledgeRL.md
• docs/DrivingRL.md
• docs/GamingRL.md
• docs/GamingRLFPS.md
• docs/MinecraftPluginTrainingGuide.md
• docs/MinecraftPluginOperations.md
• docs/MinecraftIntegration.md
• docs/GameModBridges.md
• docs/CHILD_PSYCHOLOGY_PARENTING_RL.md
• docs/SimpleAgents.md

Legacy/reference docs:

• RL_FEATURES.md
• RL_REALWORLD.md
• ML_APPLICATIONS.md

Short Summary

The RL stack has a reusable C++ substrate, multiple local algorithms, MathNN DQN support, distributed/offline hooks, a broad curriculum catalog, and several domain-specific systems. The highest-value operational habit is to separate learning from actuation: train and test in simulation or offline replay, inspect behavior, save checkpoints, evaluate, then advance through the deployment ladder only when the policy is measurably better and safety gates pass.