Strategic Recursive Self-Improvement AgentTechnical Overview of RSI-Audio-Visual-Agent.lua
Abstract
RSI-Audio-Visual-Agent.lua is a goal-driven recursive self-improvement (RSI) orchestrator for the _AugmentedIntelligence v25 console application. The script does not implement machine-learning models directly; instead, it coordinates the host application’s speech commands, Lua globals, memory systems, and perception pipelines through a repeated evaluate–improve–apply cycle. Each cycle gathers multimodal feedback (vision, audio, memory, executive planning, ethics, and quality metrics), asks a strategist large language model (LLM) for structured improvement proposals, and applies safe configuration changes when proposals pass diffing, ethics, and optional human-style dry-run gates. The agent supports early convergence, rollback on quality regression, run persistence, fleet broadcast, learned command routing, and multiple operational profiles—including a StrategicOnly mode that disables all vision and audio pipelines.
1. Introduction
Modern augmented-intelligence systems must tune dozens of interdependent settings—LLM API selection, object-detection backends, transcription models, memory limits, and safety policies—while operating under GPU, API, and time budgets. Manual tuning is slow and does not scale across deployment targets. The RSI script automates this tuning loop by treating the entire _AugmentedIntelligence application as an environment the agent observes and incrementally improves.
The script is launched from within the application:
- simple script execute lua_scripts/RSI-Audio-Visual-Agent.lua
It binds to C++-exposed Lua globals (refresh_settings, db.*, ethics_action_classify, etc.) and routes work through execute_command(), which is the same pathway used by typed and spoken speech commands.
2. System Context
The RSI agent sits above the existing v25 command surface. It does not replace ObjectDetection.cpp, Whisper transcription, or LLM backends; it selects and sequences commands that invoke those subsystems. The script’s registry contains on the order of eighty registered command definitions spanning perception, vision, audio, memory, thoughts (MySQL), WikiHow, executive planning, awareness, ethics, hardware, and settings.
Installation root for model paths and runtime assets is typically C:\_AugmentedIntelligence. The script itself lives under lua_scripts/ in the project source tree and is copied to the application payload during deployment.
3. Core Architecture
3.1 The StrategicRSI Agent
The central object is StrategicRSI, instantiated as AudioVisionAgent. It maintains run state: goal, version, performance estimate, improvement log, metrics history, command-plan history, stagnation tracking, rollback snapshots, A/B test history, fleet broadcast log, and training-collect log.
3.2 Module Facades (AI.*)
| Module | Role |
| AI.Perception | Fused world model, activity/depth classifiers, cross-modal recall |
| AI.Executive | Goal planning via executive_plan / executive_next |
| AI.Awareness | Dashboard context via awareness refresh/status |
| AI.Vision / AI.Audio | Bridges to capture, OCR, detection, transcription |
| AI.Memory / AI.Thoughts | Working memory and MySQL llm_thoughts integration |
| AI.CommandRouter | Scores and selects commands per cycle |
| AI.Pipeline | Executes staged perception → analysis → memory → quality batches |
| AI.Metrics | Per-cycle quality scores and budget spend |
| AI.Improvement | Parses structured strategist output |
| AI.Ethics | ethics_action_classify gate before apply |
| AI.Settings | Snapshot/restore for rollback |
| AI.Persistence / AI.JSON | Save/resume run state |
| AI.Profiles / AI.Menu | Presets and interactive bootstrap menu |
| AI.Learning | Per-goal learned routing weights |
| AI.ABTest | Alternating command variants per cycle |
| AI.Report | Markdown run report export |
| AI.Fleet | remote commands broadcast to fleet hosts |
| AI.TrainingCollect | Auto dataset collection when quality is low |
3.3 The Improvement Loop
Each cycle performs four steps:
- Evaluate — refresh globals, build context, plan and execute command pipeline, compute metrics
- Improve — send all evaluation sections to simple text solve strategist with structured output schema
- Apply — parse proposal; pass diffing, ethics, dry-run, and snapshot gates; change settings and run commands
- Decide — check convergence criteria; persist state; optionally stop early
Structured strategist responses must include: HYPOTHESIS, SETTING_CHANGES, COMMANDS_TO_TRY, CONFIDENCE, and CONVERGED (yes/no). This replaces fragile regex-only parsing of free text.
4. Command Router and Pipeline
AI.CommandRouter scores every registry entry against the current goal, WikiHow tags, vision/audio flags, per-cycle budgets, dependency chains, learned weights, and A/B adjustments. Commands carry cost metadata (time_ms, gpu, api) and pipeline stage (Perception, Analysis, Memory, Quality). Dependencies are topologically sorted—for example, capture_images before image_analysis.
Per-cycle budgets (defaults via environment variables) cap GPU-heavy and API-heavy work:
- RSI_GPU_BUDGET — GPU command units per cycle (default 14)
- RSI_API_BUDGET — API command units per cycle (default 10)
- RSI_TIME_BUDGET_SEC — Wall-clock budget per cycle in seconds (default 180)
- Heavy command caps — OCR, object detection, and transcribe limited per cycle
5. Development Phases
| Phase | Capabilities |
| Phase 1 | Perception/Executive/Awareness modules; fused evaluation; structured improvements; convergence stop; v25 command registry |
| Phase 2 | Command cost model; dependency DAG; 4-stage pipeline; GPU/API/time budgets; success metrics |
| Phase 3 | Improvement diffing; settings rollback; ethics gate; JSON persistence; bootstrap menu; env profiles |
| Phase 4 | Learned routing weights; command A/B testing; markdown report export; fleet broadcast; training collect loops |
6. Safety and Reliability
Before any improvement is applied, the script applies multiple gates:
- Material diffing — skip apply when hypothesis, settings, and commands are unchanged
- Ethics gate — ethics_action_classify blocks deny/high-destructive proposals
- Dry-run mode (RSI_DRY_RUN=1) — log proposals without changing settings
- Rollback — snapshot settings before apply; restore if next-cycle quality drops by RSI_ROLLBACK_DELTA (default 0.08)
- Convergence — early stop on CONVERGED=yes, high confidence, stagnation, or quality plateau
7. Operational Profiles
Profiles bundle goal, cycle count, vision/audio flags, budgets, and feature toggles. Set RSI_PROFILE=<name> before launch.
| Profile | Vision | Audio | Description |
| DesktopMonitor | On | Off | Desktop OCR and object detection focus |
| MeetingTranscribe | Off | On | Meeting transcription focus |
| FullAVFusion | On | On | Balanced audio-visual fusion (default-style) |
| QuickEval | On | On | Fast 2-cycle health check, low budgets |
| StrategicOnly | Off | Off | Text/reasoning only—memory, thoughts, executive, ethics; no AV pipelines |
StrategicOnly disables A/B testing and training-collect loops, sets GPU budget to 1 and API budget to 8, and boosts executive, thoughts, and memory routing scores. It is the recommended profile when the goal is strategic planning, memory recall, or LLM routing without spending GPU time on capture, OCR, or Whisper.
8. Persistence and Outputs
| File | Purpose |
| lua_scripts/rsi-run-state.json | Resumable run state (RSI_RESUME=1) |
| lua_scripts/rsi-routing-weights.json | Learned per-goal command weights |
| lua_scripts/rsi-run-report.md | End-of-run markdown report (RSI_EXPORT_REPORT=1, default on) |
| RSI-Audio-Visual-Agent.pre-phase*.lua | Versioned backups before each phase |
9. Configuration Reference
| Variable | Purpose |
| RSI_GOAL | Optimization target for the run |
| RSI_CYCLES | Maximum cycles (default 5) |
| RSI_USE_VISION / RSI_USE_AUDIO | Enable or disable modality pipelines |
| RSI_MIN_CYCLES | Minimum cycles before convergence check |
| RSI_PROFILE | Named preset (e.g. StrategicOnly) |
| RSI_MENU | Interactive bootstrap menu |
| RSI_DRY_RUN | Propose improvements without applying |
| RSI_RESUME | Resume from rsi-run-state.json |
| RSI_FLEET | Broadcast lightweight commands to fleet |
| RSI_AB_TEST | Alternate command variants per cycle |
| RSI_SELFTEST | Run built-in unit tests (23 checks), no agent loop |
10. How to Run
Standard run:
- simple script execute lua_scripts/RSI-Audio-Visual-Agent.lua
Strategic-only example (PowerShell):
- $env:RSI_PROFILE = “StrategicOnly”
- $env:RSI_GOAL = “Improve memory recall for legal research”
- simple script execute lua_scripts/RSI-Audio-Visual-Agent.lua
Self-test (no GPU/API agent loop):
- $env:RSI_SELFTEST = “1”
- lua lua_scripts/RSI-Audio-Visual-Agent.lua
11. Runtime Expectations
On a workstation-class host (e.g. Intel i9-10980XE, 64 GB RAM, NVIDIA GTX 1080 Ti), a full 5-cycle run with vision and audio typically requires roughly 15–35 minutes; early convergence often finishes in 6–15 minutes. StrategicOnly runs are substantially faster because they skip GPU-heavy capture, OCR, detection, and transcription commands.
12. Limitations
- Coroutines in ThreadManager do not achieve true OS-level parallelism for heavy commands
- Improvement quality depends on strategist LLM availability and prompt adherence
- Fleet broadcast requires remote commands infrastructure configured in the host app
- The script name reflects its origin; StrategicOnly profile operates without audio-visual pipelines
13. Conclusion
RSI-Audio-Visual-Agent.lua transforms _AugmentedIntelligence from a static configuration into a self-tuning system. By combining structured LLM proposals, command-router intelligence, staged pipelines, safety gates, and persistent learning across runs, it provides a practical path toward autonomous optimization of complex audio-visual and strategic-reasoning workloads—while remaining fully inspectable through logs, JSON state, and exported markdown reports.
— End of document —
Filed under: Uncategorized - @ June 23, 2026 7:57 pm