Gaming RL FPS Guide

This guide covers the first-person gaming RL subsystem in _AugmentedIntelligence.

It is separate from the Minecraft Spigot plugin, but it uses the same idea:

observe game state, choose actions, record transitions, train offline or online,

and only actuate when the run mode allows it.

For the shorter command overview, see docs/GamingRL.md. For SourceMod,

Quake, Garry’s Mod, and Civilization bridge setup, see

docs/GameModBridges.md.

Use this only on your own local/private games and servers. Do not use it to evade

anti-cheat systems or automate play on public multiplayer servers.

What Was Added

The FPS subsystem now has:

A 32-feature state vector for target tracking, HUD status, resources, threat,

cover, objective pressure, audio cues, death/menu terminal states, and movement

context.

A 24-action discrete control library covering aiming, firing, movement, sprint,

crouch, jump, reload, interact, melee, heal/armor, peeking, retreat reload, and

objective push.

Skill labels on top of actions: observe, aim_adjust, combat,

peek_strafe, movement, weapon_resource, and objective_use.

Behavior cloning hooks, offline replay training, self-play mode flag, human

feedback, checkpoint tags, rollback, policy explanation, evaluation, dashboard

export, heatmap export, replay summaries, and bridge protocol export.

Smooth input controls for mouse step limits and timed key holds.
Adapter hooks for game-specific perception such as HUD OCR, minimap threat,

audio direction, hit markers, damage, death screen, menus, and objective state.

MathNN/model integration points through the existing policy selection path.
Native Source/Quake/Garry’s Mod game-mod bridge receiver commands under

gaming mod bridge ….

Civilization turn-strategy bridge support through GamingStrategyRL.

Quick Start

Start in dry-run mode before allowing key and mouse output:

[text]
gaming rl init
gaming rl adapter set generic_fps
gaming rl input hold 120
gaming rl input mouse 32
gaming rl dry-run on
gaming rl curriculum start input_smoke
gaming rl status
gaming rl skills
gaming rl explain

When the actions and observations look reasonable:

[text]
gaming rl dry-run off
gaming rl mode bot
gaming rl train on
gaming rl eval 3
gaming rl dashboard
gaming rl heatmap
gaming rl checkpoint tag baseline

If behavior gets worse:

[text]
gaming rl checkpoint list
gaming rl checkpoint rollback baseline

Important Commands

[text]
gaming rl status
gaming rl features
gaming rl systems
gaming rl skills
gaming rl skill use <action_id>
gaming rl explain
gaming rl why

gaming rl mode off|assist|bot|train|eval
gaming rl train on|off|episode start|episode end
gaming rl eval [episodes]
gaming rl selfplay on|off

gaming rl adapter set <name>
gaming rl adapter status
gaming rl perception <request>

gaming rl input hold <milliseconds>
gaming rl input mouse <max_pixels_per_tick>
gaming rl dry-run on|off

gaming rl curriculum status
gaming rl curriculum start [phase]
gaming rl curriculum next
gaming rl curriculum phase <name>

gaming rl replay summary [path]
gaming rl replay tail
gaming rl replay train [path]
gaming rl bc train [epochs]
gaming rl bc status

gaming rl reward good [note]
gaming rl reward bad [note]
gaming rl reward <number> [note]

gaming rl dashboard
gaming rl heatmap
gaming rl bridge spec
gaming mod bridge status
gaming mod bridge configure source_mod 0.0.0.0 8767 YOUR_TOKEN true
gaming mod bridge configure quake_mod 0.0.0.0 8768 YOUR_TOKEN true
gaming mod bridge configure garrys_mod 0.0.0.0 8771 YOUR_TOKEN true
gaming mod bridge configure civilization_mod 0.0.0.0 8772 YOUR_TOKEN true

gaming rl checkpoint tag <tag>
gaming rl checkpoint list
gaming rl checkpoint rollback <tag>

State Vector

The transition log writes state features as s0 through s31.

[text]
s0   target visible
s1   target dx
s2   target dy
s3   target size
s4   target velocity x
s5   target velocity y
s6   target confidence
s7   target centered/alignment
s8   aim error magnitude
s9   recent action bias
s10 recent reward signal
s11 fire readiness
s12 movement context x
s13 movement context y
s14 training mode flag
s15 bias feature
s16 health normalized
s17 armor normalized
s18 ammo normalized
s19 reserve ammo normalized
s20 objective distance normalized
s21 damage taken recently
s22 hit marker
s23 death screen
s24 menu open
s25 reloading
s26 scoped
s27 on ground
s28 cover score
s29 minimap threat
s30 audio threat yaw
s31 audio threat strength

Adapters should normalize values to 0.0..1.0 where possible. Directional

values such as target dx, target dy, and audio threat yaw can use

-1.0..1.0.

Action Library

[text]
0   noop
1   aim_left
2   aim_right
3   aim_up
4   aim_down
5   fire
6   strafe_left
7   peek_left_forward
8   peek_right_forward
9   move_forward
10 move_back
11 strafe_right
12 jump
13 crouch
14 sprint_forward
15 reload
16 use_interact
17 swap_weapon
18 melee
19 heal_or_armor
20 peek_left_crouch
21 peek_right_crouch
22 retreat_reload
23 objective_push

Use gaming rl skill use <action_id> to force one action once. This is useful

for verifying that the focused window is receiving input correctly.

Files Written

The subsystem writes runtime data under:

[text]
C:/gaming_rl_logs/

Important files:

[text]
transitions.csv
fps_dashboard.json
fps_heatmap.csv
fps_bridge_protocol.json
fps_perception_requests.jsonl
fps_episode_events.jsonl
checkpoint.txt
checkpoint_<tag>.txt

The Python trainer in gaming_rl/train_fps_rl.py now detects the state columns

from transitions.csv, so it can train on both old 16-feature logs and new

32-feature logs.

Adapter Boundary

The core RL code does not perform game-specific OCR or minimap parsing directly.

It expects an adapter to populate FPSPerception. A complete adapter should

provide:

[text]
target bbox/center/confidence
health, armor, ammo, reserve ammo
hit marker and recent damage
death screen and menu state
reload/scoped/on-ground state
cover score
objective distance
visible enemy count
minimap threat
audio threat yaw and strength

gaming rl perception <request> records adapter requests to

fps_perception_requests.jsonl. gaming rl bridge spec exports the JSON bridge

schema to fps_bridge_protocol.json.

The GameModBridge receiver handles server-side Source/Quake JSON telemetry and

feeds simple health, damage, and death signals into FPSPerception. It does not

replace richer client perception for aim targets, HUD OCR, audio threat, or

minimap reasoning.

Training Curriculum

Recommended staged curriculum:

[text]
input_smoke
Dry-run first. Then force every action with `gaming rl skill use <id>`.
Verify mouse limits, key holds, sprint holding W, reload, crouch, jump, and
no accidental input into the wrong window.

aim_static
Use a stationary local target or aim trainer. Reward should come mostly from
target centering and hit markers. Keep movement disabled or ignored.

aim_tracking
Add moving targets. Watch `gaming rl explain` and `fps_heatmap.csv` to confirm
it is correcting yaw/pitch rather than firing randomly.

movement_basic
Train forward, back, strafe, jump, crouch, and sprint in an empty area. Use
`gaming rl input hold 120` or higher if movement feels choppy.

cover_peek
Add cover. Reward cover score and penalize recent damage. Check that peek
actions appear in the action summary.

resource_loop
Add limited ammo and reload timing. Penalize firing while empty/reloading and
reward clean reloads from cover.

combat_close
Add close-range combat. Enable melee and retreat_reload. Penalize death and
menu-open terminal states heavily.

objective_route
Add objective distance. Reward objective progress while preserving health and
avoiding visible threats.

mixed_eval
Run `gaming rl eval 10`, export dashboard and heatmap, then tag a checkpoint
only if returns and death rate improve.

Human Feedback

Use feedback to bias the policy after a behavior happens:

[text]
gaming rl reward good held angle correctly
gaming rl reward bad fired while reloading
gaming rl reward -0.5 walked into wall
gaming rl reward 1.0 took cover before reload

This records a feedback event and feeds a shaped transition through the same

learning path.

Offline Training

Collect transitions during dry-run, assist, bot, train, or eval modes. Then run:

[text]
gaming rl replay summary
gaming rl replay train
gaming rl bc train 5

For Python-side PPO or linear export:

[text]
python gaming_rl/train_fps_rl.py –algo linear
python gaming_rl/train_fps_rl.py –algo ppo

The Python script reads C:/gaming_rl_logs/transitions.csv by default.

Practical Operating Loop

[text]
gaming rl dry-run on
gaming rl curriculum start input_smoke
gaming rl skill use 9
gaming rl skill use 14
gaming rl skill use 5
gaming rl explain
gaming rl dashboard

gaming rl dry-run off
gaming rl curriculum phase aim_static
gaming rl mode train
gaming rl train on

gaming rl eval 5
gaming rl dashboard
gaming rl heatmap
gaming rl checkpoint tag aim_static_good
gaming rl curriculum next

Do not move to the next phase until the current phase is boringly reliable.

Macdaddy4sure's Blog