Driving RL

RL hub: ReinforcementLearning.md · neural policies: rl net enable driving — MathNN.md.

Driving RL is a simulation-first reinforcement-learning subsystem for lane keeping, lead-vehicle following, intersections, traffic controls, pedestrian yielding, bad-weather behavior, and emergency braking.

Treat this as simulation/review tooling unless a separate safety-certified control layer exists. Do not connect it directly to a real vehicle actuator.

Quick Start

driving rl status
driving rl features
driving rl actions
driving rl dry-run on
driving rl shield on
driving rl adapter set generic_driving
driving rl scenario lane_keep 120
driving rl scenario road_sign_stop 120
driving rl scenario speed_limit_sign 120
driving rl dashboard

Training smoke run:

driving rl curriculum start sensor_smoke
driving rl scenario lead_follow 200
driving rl scenario stop_sign 200
driving rl scenario road_sign_stop 200
driving rl scenario speed_limit_sign 200
driving rl scenario traffic_light 200
driving rl scenario pedestrian_yield 200
driving rl checkpoint tag baseline

Optional local DQN run:

driving rl dqn on
driving rl train
driving rl scenario road_sign_stop 300
driving rl scenario speed_limit_sign 300
driving rl dqn save checkpoints/driving_road_sign_dqn.txt

Modes

off
assist
train
eval

Commands:

driving rl on
driving rl assist
driving rl train
driving rl eval
driving rl off
driving rl status

driving rl on enables driving and starts in assist mode with auto algorithm.

Algorithms

heuristic
qtable
dqn
auto

Commands:

driving rl algorithm heuristic
driving rl algorithm qtable
driving rl algorithm dqn
driving rl algorithm auto
driving rl dqn on
driving rl dqn status
driving rl dqn save checkpoints/driving_mathnn_dqn.txt
driving rl dqn load checkpoints/driving_mathnn_dqn.txt
driving rl epsilon <value>

auto uses Q-table while training and heuristic behavior outside training. If the MathNN/RLAgentNet driving policy is enabled, the neural policy can select actions through the existing driving policy slot. dqn enables the local MathNN-backed DQN learner for the driving agent and trains it from Driving RL transitions while mode is train.

State Vector

DrivingState now has 32 normalized features:

s0   lane offset normalized around center
s1   time-to-collision risk
s2   collision probability
s3   vehicle velocity normalized
s4   lead vehicle distance normalized
s5   steering angle normalized
s6   left lane x normalized
s7   right lane x normalized
s8   lead relative velocity normalized
s9   track stable flag
s10  lead vehicle present flag
s11  lane width normalized
s12  lane confidence
s13  road curvature
s14  speed-limit ratio
s15  stop-sign flag
s16  traffic-light risk
s17  pedestrian risk
s18  intersection proximity
s19  route progress
s20  lane-departure/offroad risk
s21  visibility risk
s22  jerk/comfort cost
s23  safety context flag
s24  road-sign model confidence
s25  road-sign proximity
s26  road-sign stop/yield requirement
s27  road-sign speed-limit present flag
s28  posted-speed overage
s29  road-sign warning/no-entry/crosswalk flag
s30  road-sign x position
s31  road-sign control signal

The DrivingPerception input now supports lane confidence, road curvature, speed limits, route progress, intersection distance, traffic lights, stop signs, pedestrians, crosswalks, visibility, jerk, lane departure, offroad, and collision events in addition to the original lane/lead vehicle fields. It also accepts trained road-sign detector output: label, confidence, bounding box, estimated distance, stop/yield/speed-limit/warning/no-entry flags, and posted speed limit.

The driving camera loop passes the active object-detection model rows into Driving RL. Labels such as stop sign, stop_sign, yield_sign, speed_limit_35, traffic_light_red, traffic_light_yellow, wrong_way, school, crosswalk, and similar road-sign classes are normalized into the road-sign state features and safety shield.

Actions

0   maintain
1   accelerate
2   coast
3   brake_light
4   brake_moderate
5   brake_heavy
6   steer_left
7   steer_right
8   lane_center
9   yield
10  emergency_stop
11  crawl_forward

Use:

driving rl actions

Safety Shield

The safety shield wraps any selected policy action. It can override exploration, Q-table, heuristic, or neural choices when the perception state indicates:

collision event
critical lead vehicle risk
red light or stop sign near intersection
road-sign model stop/yield/no-entry signal
posted speed-limit overage
pedestrian near crosswalk
lane departure or offroad state

Commands:

driving rl shield on
driving rl shield off
driving rl dry-run on
driving rl dry-run off
driving rl explain

Keep dry-run on unless the recommendation is only being consumed by a simulation harness.

Curriculum

Recommended phase order:

sensor_smoke
lane_keep
lead_follow
stop_sign
road_sign_stop
speed_limit_sign
traffic_light
pedestrian_yield
intersection
merge
bad_weather
emergency_brake
mixed_eval

Commands:

driving rl curriculum status
driving rl curriculum start [phase]
driving rl curriculum next
driving rl curriculum phase <name>

Synthetic Scenarios

The scenario runner feeds synthetic DrivingPerception ticks through the normal DrivingRL pipeline. It is useful for fast smoke testing and checkpoint creation.

driving rl scenario list
driving rl scenario lane_keep 120
driving rl scenario lead_follow 200
driving rl scenario stop_sign 200
driving rl scenario road_sign_stop 200
driving rl scenario speed_limit_sign 200
driving rl scenario traffic_light 200
driving rl scenario pedestrian_yield 200
driving rl scenario bad_weather 200
driving rl scenario emergency_brake 200
driving rl scenario lane_departure 200
driving rl scenario merge 200

These scenarios are not a substitute for a real simulator such as CARLA, Gazebo, or a custom physics sim, but they are good for verifying command paths, rewards, shield overrides, logging, and checkpoint behavior.

Rewards And Feedback

The reward favors:

• lane centering,
• safe time-to-collision,
• reduced collision probability,
• obeying road-sign model stop/yield signals,
• slowing when above a detected posted speed limit,
• route progress,
• recovery from lane-departure risk.

The reward penalizes:

• collision risk,
• pedestrian/crosswalk risk,
• red-light/stop-sign acceleration,
• accelerating through detected road-sign control signals,
• accelerating while above a detected posted speed limit,
• offroad/lane departure,
• jerk/comfort cost,
• unnecessary braking and throttle,
• safety shield overrides.

Manual feedback:

driving rl reward good [note]
driving rl reward bad [note]
driving rl reward <number> [note]

Replay, Dashboard, And Checkpoints

Default directory:

D:/driving_rl_logs

Files:

driving_rl_checkpoint.txt
driving_rl_transitions.csv
driving_rl_events.jsonl
driving_rl_dashboard.json
driving_rl_checkpoint_<tag>.txt

Commands:

driving rl dashboard
driving rl replay summary [path]
driving rl checkpoint tag <tag>
driving rl checkpoint list
driving rl checkpoint rollback <tag>
driving rl save
driving rl load
driving rl save mysql
driving rl load mysql

MySQL checkpoints use the configured driving_rl_checkpoint_profile.

Adapter Boundary

Adapters should populate DrivingPerception from a simulator or perception stack:

lane lines and confidence
lead vehicle distance, velocity, TTC, collision probability
ego speed, steering angle, jerk
traffic light and stop sign state
pedestrian/crosswalk distance
visibility/weather/night condition
route progress and intersection distance
lane departure/offroad/collision events

Commands:

driving rl adapter status
driving rl adapter set <name>
driving rl features

Lua Globals

The subsystem exposes Driving RL helpers to Lua through _DrivingRL::RegisterLuaGlobals:

driving_rl_status()
driving_rl_set_mode(mode)
driving_rl_scenario(name, ticks)
driving_rl_shield(enabled)
driving_rl_dashboard()

Safe Workflow

1. Feed simulation perception only.
2. Start with driving rl dry-run on and driving rl shield on.
3. Run synthetic scenarios.
4. Inspect driving rl explain, driving rl dashboard, and replay summaries.
5. Train in a real simulator only after synthetic checks pass.
6. Save checkpoint tags before changing algorithms or epsilon.
7. Evaluate offline before using a learned policy as an assist recommendation.