Computer RL

On-screen desktop reinforcement-learning operator — perception, policy, and safety for driving the computer.

Overview

Computer RL (_ComputerRL) is a live reinforcement-learning agent that watches the desktop and acts on it. Every frame of the desktop-capture loop calls ProcessComputerRlTick, feeding it the screenshot, the active window title, the cursor position, detected text regions, and object detections. From that it builds a 32-dimension state vector, selects one of 16 discrete actions (click a text region, type the next goal token, press a key, scroll, wait, confirm, abort…), and — unless in dry-run — drives the mouse and keyboard through the Win32 input API.

It is a Q-table agent by default, with an optional in-process neural DQN path (via _RLAgentNet / _MathNNAgentBridge, dueling-DQN, 32→16) and a heuristic fallback policy. Goals come from the _Computers goal subsystem; text/icon perception, the ethics gate, and MySQL checkpointing are shared with the rest of the app.

Quick start

computer rl status
computer rl features
computer rl actions
computer rl dry-run on
computer rl shield on
computer rl adapter set generic_desktop
computer rl scenario notepad_save 60
computer rl scenario browser_search 60
computer rl explain
computer rl dashboard

Training smoke run (synthetic, no live input needed):

computer rl train
computer rl curriculum start click_smoke
computer rl scenario notepad_save 200
computer rl scenario form_fill 200
computer rl scenario error_recovery 200
computer rl checkpoint tag baseline

Optional local DQN run:

computer rl dqn on
computer rl train replay 5
computer rl scenario form_fill 300
computer rl dqn status

Perception pipeline

Each desktop tick assembles perception rows before the RL agent runs. Text perception uses neural networks end to end: text detection proposes boxes and text recognition (a CRNN) reads the characters inside them.

From these rows Computer RL derives goal state (from _Computers), an application-family classification (browser / office / explorer / settings / creative / games), error-dialog and risky-text flags via keyword scan, window-change and cursor-stuck signals, and up to eight clickable text targets.

Modes

computer rl on        # enable + assist mode + auto algorithm
computer rl assist
computer rl train
computer rl eval
computer rl off

Algorithms

heuristic   # rule-based fallback policy
qtable      # tabular Q-learning over discretized state
dqn         # in-process MathNN dueling-DQN (the "computer" agent slot)
auto        # Q-table while training, heuristic otherwise

computer rl algorithm heuristic|qtable|dqn|auto
computer rl dqn on|off|status
computer rl epsilon <value>

auto uses the Q-table during train and the heuristic policy otherwise. Enabling dqn configures the neural computer policy slot (32 inputs, 16 actions, dueling-DQN) and trains it from Computer RL transitions while the mode is train.

State vector

ComputerState has 32 normalized features:

f0   cursor x (normalized)
f1   cursor y (normalized)
f2   text-region count / 40
f3   object count / 20
f4   goal active flag
f5   goal step progress
f6   application family / 6
f7   window changed recently flag
f8   cursor stuck flag
f9   repeat-action count / 5
f10  risky-text flag (from screen text)
f11  error-dialog flag (from screen text)
f12  best text target center x
f13  best text target center y
f14  best text target score
f15  goal keyword overlap with screen text
f16  goal status == completed flag
f17  adapter code (browser / office / other)
f18  dry-run flag
f19  safety-shield flag
f20  recent reward average (rescaled)
f21  episode return (rescaled)
f22  mode code (train / assist / off)
f23  scenario active flag
f24  tick dt / 2s
f25  total steps / 10000
f26  last discrete action (normalized)
f27  safety overrides / 100  (blended with NLU curiosity)
f28  policy-violation events / 50  (blended with NLU uncertainty)
f29  loop events / 50
f30  goal completions / 20
f31  text-target density

When the language bridge (_RLLanguageBridge) is enabled, f15 is blended with an NLU goal-match score and f10/f27/f28 are shaped by NLU safety, curiosity, and uncertainty signals over the recognized screen text.

Actions

0   click_region_0
1   click_region_1
2   click_region_2
3   click_region_3
4   click_best_object
5   type_next_token
6   press_enter
7   press_tab
8   press_escape
9   open_goal_app
10  scroll_down
11  scroll_up
12  wait
13  request_confirm
14  abort_goal
15  noop

computer rl actions

Click actions 0–3 map to the first four recognized text targets. Action 5 types the next token of the active goal; action 9 launches the goal application via _ComputerBot.

Rewards & feedback

The reward favors:

• goal step progress,
• increasing keyword overlap between the recognized screen text and the goal,
• reaching a completed goal state.

The reward penalizes:

• visible error dialogs,
• looping / repeated actions,
• risky clicks while risky text is on screen,
• clicking while the cursor is stuck,
• safety-shield overrides and policy-rule violations.

Manual (human) feedback nudges the Q-value of the last action:

computer rl reward good [note]
computer rl reward bad [note]
computer rl reward <number> [note]

Safety shield & policy rules

The safety shield wraps every selected action (exploration, Q-table, heuristic, or neural) and can override it based on the perceived state:

Named policy rules add reward penalties when a trigger is not answered by the required action. The default rules require confirmation on risky text, a wait on error dialogs, and confirmation before delete flows. Actual execution is additionally gated by the shared ethics check (_SafetyEthicsTrust::ApproveBotAction) and by the dry-run flag.

computer rl shield on|off
computer rl dry-run on|off
computer rl explain
computer rl policy status
computer rl policy seed
computer rl policy list [N]
computer rl policy add <trigger> <required_action> <severity> <summary>

Curriculum

Recommended phase order:

click_smoke  ->  form_fill  ->  multi_window  ->  error_recovery  ->  production

computer rl curriculum status
computer rl curriculum start [phase]
computer rl curriculum next

Synthetic scenarios

The scenario runner feeds synthetic perception ticks through the normal Computer RL pipeline — useful for fast smoke testing, reward inspection, and checkpoint creation without touching a live desktop.

computer rl scenario list
computer rl scenario notepad_save 60
computer rl scenario browser_search 60
computer rl scenario settings_wifi 60
computer rl scenario form_fill 60
computer rl scenario error_recovery 60

Replay, dashboard & checkpoints

Default log directory: C:/computer_rl_logs. Files:

computer_rl_checkpoint.txt
computer_rl_transitions.csv
computer_rl_dashboard.json
computer_rl_checkpoint_<tag>.txt

computer rl dashboard
computer rl observe
computer rl step
computer rl train replay [episodes]
computer rl replay [path]
computer rl checkpoint tag <tag>
computer rl checkpoint list
computer rl checkpoint rollback <tag>
computer rl save
computer rl load

When computer_rl_checkpoint_mysql_enabled is set, checkpoints are also stored to MySQL under the computer_rl_checkpoint_profile (table key kTableComputer).

Adapters

Adapters label the current environment so the state vector and behavior can specialize (for example browser vs office vs generic_desktop).

computer rl adapter status
computer rl adapter set <name>
computer rl features

Lua globals

Registered through _ComputerRL::RegisterLuaGlobals:

computer_rl_enabled          -- boolean global
computer_rl_status()         -- status string
computer_rl_set_mode(mode)   -- "off" | "assist" | "train" | "eval"
computer_rl_scenario(name, ticks)

Command reference

Safe workflow

1. Start with computer rl dry-run on and computer rl shield on.
2. Confirm perception with computer rl features and computer rl observe.
3. Run synthetic scenarios and inspect computer rl explain / dashboard.
4. Train with the Q-table (or DQN) and tag a baseline checkpoint.
5. Evaluate offline before letting any learned policy drive assist recommendations.
6. Only turn dry-run off on a live desktop deliberately, with the shield on and a goal set.