Gaming RL FPS Guide

This guide covers the first-person gaming RL subsystem in _AugmentedIntelligence. It is separate from the Minecraft Spigot plugin, but it uses the same idea: observe game state, choose actions, record transitions, train offline or online, and only actuate when the run mode allows it.

For the shorter command overview, see docs/GamingRL.md. For SourceMod, Quake, Garry's Mod, and Civilization bridge setup, see docs/GameModBridges.md.

Use this only on your own local/private games and servers. Do not use it to evade anti-cheat systems or automate play on public multiplayer servers.

What Was Added

The FPS subsystem now has:

• A 32-feature state vector for target tracking, HUD status, resources, threat, cover, objective pressure, audio cues, death/menu terminal states, and movement context.
• A 24-action discrete control library covering aiming, firing, movement, sprint, crouch, jump, reload, interact, melee, heal/armor, peeking, retreat reload, and objective push.
• Skill labels on top of actions: observe, aim_adjust, combat, peek_strafe, movement, weapon_resource, and objective_use.
• Behavior cloning hooks, offline replay training, self-play mode flag, human feedback, checkpoint tags, rollback, policy explanation, evaluation, dashboard export, heatmap export, replay summaries, and bridge protocol export.
• Smooth input controls for mouse step limits and timed key holds.
• Adapter hooks for game-specific perception such as HUD OCR, minimap threat, audio direction, hit markers, damage, death screen, menus, and objective state.
• MathNN/model integration points through the existing policy selection path.
• Aiming coach — tracks crosshair–target error, fire discipline, overshoot, acquire time, and head-height bias; grades the session and suggests drills (gaming rl aim status|tips|report|drill|llm).
• Native Source/Quake/Garry's Mod game-mod bridge receiver commands under gaming mod bridge ....
• Civilization turn-strategy bridge support through GamingStrategyRL.

Quick Start

Start in dry-run mode before allowing key and mouse output:

gaming rl init
gaming rl adapter set generic_fps
gaming rl input hold 120
gaming rl input mouse 32
gaming rl dry-run on
gaming rl curriculum start input_smoke
gaming rl status
gaming rl skills
gaming rl explain

When the actions and observations look reasonable:

gaming rl dry-run off
gaming rl mode bot
gaming rl train on
gaming rl eval 3
gaming rl dashboard
gaming rl heatmap
gaming rl checkpoint tag baseline

If behavior gets worse:

gaming rl checkpoint list
gaming rl checkpoint rollback baseline

Important Commands

gaming rl status
gaming rl features
gaming rl systems
gaming rl skills
gaming rl skill use <action_id>
gaming rl explain
gaming rl why

gaming rl aim status
gaming rl aim tips
gaming rl aim report
gaming rl aim drill
gaming rl aim llm
gaming rl aim reset
gaming rl aim on|off

gaming rl mode off|assist|bot|train|eval
gaming rl train on|off|episode start|episode end
gaming rl eval [episodes]
gaming rl selfplay on|off

gaming rl adapter set <name>
gaming rl adapter status
gaming rl perception <request>

gaming rl input hold <milliseconds>
gaming rl input mouse <max_pixels_per_tick>
gaming rl dry-run on|off

gaming rl curriculum status
gaming rl curriculum start [phase]
gaming rl curriculum next
gaming rl curriculum phase <name>

gaming rl replay summary [path]
gaming rl replay tail
gaming rl replay train [path]
gaming rl bc train [epochs]
gaming rl bc status

gaming rl reward good [note]
gaming rl reward bad [note]
gaming rl reward <number> [note]

gaming rl dashboard
gaming rl heatmap
gaming rl bridge spec
gaming mod bridge status
gaming mod bridge configure source_mod 0.0.0.0 8767 YOUR_TOKEN true
gaming mod bridge configure quake_mod 0.0.0.0 8768 YOUR_TOKEN true
gaming mod bridge configure garrys_mod 0.0.0.0 8771 YOUR_TOKEN true
gaming mod bridge configure civilization_mod 0.0.0.0 8772 YOUR_TOKEN true

gaming rl checkpoint tag <tag>
gaming rl checkpoint list
gaming rl checkpoint rollback <tag>

State Vector

The transition log writes state features as s0 through s31.

s0   target visible
s1   target dx
s2   target dy
s3   target size
s4   target velocity x
s5   target velocity y
s6   target confidence
s7   target centered/alignment
s8   aim error magnitude
s9   recent action bias
s10  recent reward signal
s11  fire readiness
s12  movement context x
s13  movement context y
s14  training mode flag
s15  bias feature
s16  health normalized
s17  armor normalized
s18  ammo normalized
s19  reserve ammo normalized
s20  objective distance normalized
s21  damage taken recently
s22  hit marker
s23  death screen
s24  menu open
s25  reloading
s26  scoped
s27  on ground
s28  cover score
s29  minimap threat
s30  audio threat yaw
s31  audio threat strength

Adapters should normalize values to 0.0..1.0 where possible. Directional values such as target dx, target dy, and audio threat yaw can use -1.0..1.0.

Action Library

0   noop
1   aim_left
2   aim_right
3   aim_up
4   aim_down
5   fire
6   strafe_left
7   peek_left_forward
8   peek_right_forward
9   move_forward
10  move_back
11  strafe_right
12  jump
13  crouch
14  sprint_forward
15  reload
16  use_interact
17  swap_weapon
18  melee
19  heal_or_armor
20  peek_left_crouch
21  peek_right_crouch
22  retreat_reload
23  objective_push

Use gaming rl skill use <action_id> to force one action once. This is useful for verifying that the focused window is receiving input correctly.

Files Written

The subsystem writes runtime data under:

C:/gaming_rl_logs/

Important files:

transitions.csv
fps_dashboard.json
fps_heatmap.csv
fps_bridge_protocol.json
fps_perception_requests.jsonl
fps_episode_events.jsonl
checkpoint.txt
checkpoint_<tag>.txt

The Python trainer in gaming_rl/train_fps_rl.py now detects the state columns from transitions.csv, so it can train on both old 16-feature logs and new 32-feature logs.

Adapter Boundary

The core RL code does not perform game-specific OCR or minimap parsing directly. It expects an adapter to populate FPSPerception. A complete adapter should provide:

target bbox/center/confidence
health, armor, ammo, reserve ammo
hit marker and recent damage
death screen and menu state
reload/scoped/on-ground state
cover score
objective distance
visible enemy count
minimap threat
audio threat yaw and strength

gaming rl perception <request> records adapter requests to fps_perception_requests.jsonl. gaming rl bridge spec exports the JSON bridge schema to fps_bridge_protocol.json.

The GameModBridge receiver handles server-side Source/Quake JSON telemetry and feeds simple health, damage, and death signals into FPSPerception. It does not replace richer client perception for aim targets, HUD OCR, audio threat, or minimap reasoning.

Training Curriculum

Recommended staged curriculum:

input_smoke
  Dry-run first. Then force every action with `gaming rl skill use <id>`.
  Verify mouse limits, key holds, sprint holding W, reload, crouch, jump, and
  no accidental input into the wrong window.

aim_static
  Use a stationary local target or aim trainer. Reward should come mostly from
  target centering and hit markers. Keep movement disabled or ignored.

aim_tracking
  Add moving targets. Watch `gaming rl explain` and `fps_heatmap.csv` to confirm
  it is correcting yaw/pitch rather than firing randomly.

movement_basic
  Train forward, back, strafe, jump, crouch, and sprint in an empty area. Use
  `gaming rl input hold 120` or higher if movement feels choppy.

cover_peek
  Add cover. Reward cover score and penalize recent damage. Check that peek
  actions appear in the action summary.

resource_loop
  Add limited ammo and reload timing. Penalize firing while empty/reloading and
  reward clean reloads from cover.

combat_close
  Add close-range combat. Enable melee and retreat_reload. Penalize death and
  menu-open terminal states heavily.

objective_route
  Add objective distance. Reward objective progress while preserving health and
  avoiding visible threats.

mixed_eval
  Run `gaming rl eval 10`, export dashboard and heatmap, then tag a checkpoint
  only if returns and death rate improve.

Human Feedback

Use feedback to bias the policy after a behavior happens:

gaming rl reward good held angle correctly
gaming rl reward bad fired while reloading
gaming rl reward -0.5 walked into wall
gaming rl reward 1.0 took cover before reload

This records a feedback event and feeds a shaped transition through the same learning path.

Offline Training

Collect transitions during dry-run, assist, bot, train, or eval modes. Then run:

gaming rl replay summary
gaming rl replay train
gaming rl bc train 5

For Python-side PPO or linear export:

python gaming_rl/train_fps_rl.py --algo linear
python gaming_rl/train_fps_rl.py --algo ppo

The Python script reads C:/gaming_rl_logs/transitions.csv by default.

Practical Operating Loop

gaming rl dry-run on
gaming rl curriculum start input_smoke
gaming rl skill use 9
gaming rl skill use 14
gaming rl skill use 5
gaming rl explain
gaming rl dashboard

gaming rl dry-run off
gaming rl curriculum phase aim_static
gaming rl mode train
gaming rl train on

gaming rl eval 5
gaming rl dashboard
gaming rl heatmap
gaming rl checkpoint tag aim_static_good
gaming rl curriculum next

Do not move to the next phase until the current phase is boringly reliable.