{"id":1827,"date":"2026-06-23T19:57:16","date_gmt":"2026-06-24T02:57:16","guid":{"rendered":"http:\/\/macdaddy4sure.ai\/?p=1827"},"modified":"2026-06-23T19:58:22","modified_gmt":"2026-06-24T02:58:22","slug":"strategic-recursive-self-improvement-agenttechnical-overview-of-rsi-audio-visual-agent-lua","status":"publish","type":"post","link":"http:\/\/macdaddy4sure.ai\/index.php\/2026\/06\/23\/strategic-recursive-self-improvement-agenttechnical-overview-of-rsi-audio-visual-agent-lua\/","title":{"rendered":"Strategic Recursive Self-Improvement AgentTechnical Overview of RSI-Audio-Visual-Agent.lua"},"content":{"rendered":"\n

https:\/\/docs.google.com\/document\/d\/1KHNlXblKdrB9MfewDr7AzwlvZE-ZlBv4\/edit?usp=drive_link&ouid=107277234409707752355&rtpof=true&sd=true<\/a><\/p>\n\n\n\n

Abstract<\/h1>\n\n\n\n

RSI-Audio-Visual-Agent.lua is a goal-driven recursive self-improvement (RSI) orchestrator for the _AugmentedIntelligence v25 console application. The script does not implement machine-learning models directly; instead, it coordinates the host application\u2019s speech commands, Lua globals, memory systems, and perception pipelines through a repeated evaluate\u2013improve\u2013apply cycle. Each cycle gathers multimodal feedback (vision, audio, memory, executive planning, ethics, and quality metrics), asks a strategist large language model (LLM) for structured improvement proposals, and applies safe configuration changes when proposals pass diffing, ethics, and optional human-style dry-run gates. The agent supports early convergence, rollback on quality regression, run persistence, fleet broadcast, learned command routing, and multiple operational profiles\u2014including a StrategicOnly mode that disables all vision and audio pipelines.<\/p>\n\n\n\n

1. Introduction<\/h1>\n\n\n\n

Modern augmented-intelligence systems must tune dozens of interdependent settings\u2014LLM API selection, object-detection backends, transcription models, memory limits, and safety policies\u2014while operating under GPU, API, and time budgets. Manual tuning is slow and does not scale across deployment targets. The RSI script automates this tuning loop by treating the entire _AugmentedIntelligence application as an environment the agent observes and incrementally improves.<\/p>\n\n\n\n

The script is launched from within the application:<\/p>\n\n\n\n

    \n
  • simple script execute lua_scripts\/RSI-Audio-Visual-Agent.lua<\/li>\n<\/ul>\n\n\n\n

    It binds to C++-exposed Lua globals (refresh_settings, db.*, ethics_action_classify, etc.) and routes work through execute_command(), which is the same pathway used by typed and spoken speech commands.<\/p>\n\n\n\n

    2. System Context<\/h1>\n\n\n\n

    The RSI agent sits above the existing v25 command surface. It does not replace ObjectDetection.cpp, Whisper transcription, or LLM backends; it selects and sequences commands that invoke those subsystems. The script\u2019s registry contains on the order of eighty registered command definitions spanning perception, vision, audio, memory, thoughts (MySQL), WikiHow, executive planning, awareness, ethics, hardware, and settings.<\/p>\n\n\n\n

    Installation root for model paths and runtime assets is typically C:\\_AugmentedIntelligence. The script itself lives under lua_scripts\/ in the project source tree and is copied to the application payload during deployment.<\/p>\n\n\n\n

    3. Core Architecture<\/h1>\n\n\n\n

    3.1 The StrategicRSI Agent<\/h2>\n\n\n\n

    The central object is StrategicRSI, instantiated as AudioVisionAgent. It maintains run state: goal, version, performance estimate, improvement log, metrics history, command-plan history, stagnation tracking, rollback snapshots, A\/B test history, fleet broadcast log, and training-collect log.<\/p>\n\n\n\n

    3.2 Module Facades (AI.*)<\/h2>\n\n\n\n
    Module<\/strong><\/td>Role<\/strong><\/td><\/tr>
    AI.Perception<\/td>Fused world model, activity\/depth classifiers, cross-modal recall<\/td><\/tr>
    AI.Executive<\/td>Goal planning via executive_plan \/ executive_next<\/td><\/tr>
    AI.Awareness<\/td>Dashboard context via awareness refresh\/status<\/td><\/tr>
    AI.Vision \/ AI.Audio<\/td>Bridges to capture, OCR, detection, transcription<\/td><\/tr>
    AI.Memory \/ AI.Thoughts<\/td>Working memory and MySQL llm_thoughts integration<\/td><\/tr>
    AI.CommandRouter<\/td>Scores and selects commands per cycle<\/td><\/tr>
    AI.Pipeline<\/td>Executes staged perception \u2192 analysis \u2192 memory \u2192 quality batches<\/td><\/tr>
    AI.Metrics<\/td>Per-cycle quality scores and budget spend<\/td><\/tr>
    AI.Improvement<\/td>Parses structured strategist output<\/td><\/tr>
    AI.Ethics<\/td>ethics_action_classify gate before apply<\/td><\/tr>
    AI.Settings<\/td>Snapshot\/restore for rollback<\/td><\/tr>
    AI.Persistence \/ AI.JSON<\/td>Save\/resume run state<\/td><\/tr>
    AI.Profiles \/ AI.Menu<\/td>Presets and interactive bootstrap menu<\/td><\/tr>
    AI.Learning<\/td>Per-goal learned routing weights<\/td><\/tr>
    AI.ABTest<\/td>Alternating command variants per cycle<\/td><\/tr>
    AI.Report<\/td>Markdown run report export<\/td><\/tr>
    AI.Fleet<\/td>remote commands broadcast to fleet hosts<\/td><\/tr>
    AI.TrainingCollect<\/td>Auto dataset collection when quality is low<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    3.3 The Improvement Loop<\/h2>\n\n\n\n

    Each cycle performs four steps:<\/strong><\/p>\n\n\n\n

      \n
    • Evaluate \u2014 refresh globals, build context, plan and execute command pipeline, compute metrics<\/li>\n\n\n\n
    • Improve \u2014 send all evaluation sections to simple text solve strategist with structured output schema<\/li>\n\n\n\n
    • Apply \u2014 parse proposal; pass diffing, ethics, dry-run, and snapshot gates; change settings and run commands<\/li>\n\n\n\n
    • Decide \u2014 check convergence criteria; persist state; optionally stop early<\/li>\n<\/ul>\n\n\n\n

      Structured strategist responses must include: HYPOTHESIS, SETTING_CHANGES, COMMANDS_TO_TRY, CONFIDENCE, and CONVERGED (yes\/no). This replaces fragile regex-only parsing of free text.<\/p>\n\n\n\n

      4. Command Router and Pipeline<\/h1>\n\n\n\n

      AI.CommandRouter scores every registry entry against the current goal, WikiHow tags, vision\/audio flags, per-cycle budgets, dependency chains, learned weights, and A\/B adjustments. Commands carry cost metadata (time_ms, gpu, api) and pipeline stage (Perception, Analysis, Memory, Quality). Dependencies are topologically sorted\u2014for example, capture_images before image_analysis.<\/p>\n\n\n\n

      Per-cycle budgets (defaults via environment variables) cap GPU-heavy and API-heavy work:<\/p>\n\n\n\n

        \n
      • RSI_GPU_BUDGET \u2014 GPU command units per cycle (default 14)<\/li>\n\n\n\n
      • RSI_API_BUDGET \u2014 API command units per cycle (default 10)<\/li>\n\n\n\n
      • RSI_TIME_BUDGET_SEC \u2014 Wall-clock budget per cycle in seconds (default 180)<\/li>\n\n\n\n
      • Heavy command caps \u2014 OCR, object detection, and transcribe limited per cycle<\/li>\n<\/ul>\n\n\n\n

        5. Development Phases<\/h1>\n\n\n\n
        Phase<\/strong><\/td>Capabilities<\/strong><\/td><\/tr>
        Phase 1<\/td>Perception\/Executive\/Awareness modules; fused evaluation; structured improvements; convergence stop; v25 command registry<\/td><\/tr>
        Phase 2<\/td>Command cost model; dependency DAG; 4-stage pipeline; GPU\/API\/time budgets; success metrics<\/td><\/tr>
        Phase 3<\/td>Improvement diffing; settings rollback; ethics gate; JSON persistence; bootstrap menu; env profiles<\/td><\/tr>
        Phase 4<\/td>Learned routing weights; command A\/B testing; markdown report export; fleet broadcast; training collect loops<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

        6. Safety and Reliability<\/h1>\n\n\n\n

        Before any improvement is applied, the script applies multiple gates:<\/strong><\/p>\n\n\n\n

          \n
        • Material diffing \u2014 skip apply when hypothesis, settings, and commands are unchanged<\/li>\n\n\n\n
        • Ethics gate \u2014 ethics_action_classify blocks deny\/high-destructive proposals<\/li>\n\n\n\n
        • Dry-run mode (RSI_DRY_RUN=1) \u2014 log proposals without changing settings<\/li>\n\n\n\n
        • Rollback \u2014 snapshot settings before apply; restore if next-cycle quality drops by RSI_ROLLBACK_DELTA (default 0.08)<\/li>\n\n\n\n
        • Convergence \u2014 early stop on CONVERGED=yes, high confidence, stagnation, or quality plateau<\/li>\n<\/ul>\n\n\n\n

          7. Operational Profiles<\/h1>\n\n\n\n

          Profiles bundle goal, cycle count, vision\/audio flags, budgets, and feature toggles. Set RSI_PROFILE=<name> before launch.<\/p>\n\n\n\n

          Profile<\/strong><\/td>Vision<\/strong><\/td>Audio<\/strong><\/td>Description<\/strong><\/td><\/tr>
          DesktopMonitor<\/td>On<\/td>Off<\/td>Desktop OCR and object detection focus<\/td><\/tr>
          MeetingTranscribe<\/td>Off<\/td>On<\/td>Meeting transcription focus<\/td><\/tr>
          FullAVFusion<\/td>On<\/td>On<\/td>Balanced audio-visual fusion (default-style)<\/td><\/tr>
          QuickEval<\/td>On<\/td>On<\/td>Fast 2-cycle health check, low budgets<\/td><\/tr>
          StrategicOnly<\/td>Off<\/td>Off<\/td>Text\/reasoning only\u2014memory, thoughts, executive, ethics; no AV pipelines<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

          StrategicOnly disables A\/B testing and training-collect loops, sets GPU budget to 1 and API budget to 8, and boosts executive, thoughts, and memory routing scores. It is the recommended profile when the goal is strategic planning, memory recall, or LLM routing without spending GPU time on capture, OCR, or Whisper.<\/p>\n\n\n\n

          8. Persistence and Outputs<\/h1>\n\n\n\n
          File<\/strong><\/td>Purpose<\/strong><\/td><\/tr>
          lua_scripts\/rsi-run-state.json<\/td>Resumable run state (RSI_RESUME=1)<\/td><\/tr>
          lua_scripts\/rsi-routing-weights.json<\/td>Learned per-goal command weights<\/td><\/tr>
          lua_scripts\/rsi-run-report.md<\/td>End-of-run markdown report (RSI_EXPORT_REPORT=1, default on)<\/td><\/tr>
          RSI-Audio-Visual-Agent.pre-phase*.lua<\/td>Versioned backups before each phase<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

          9. Configuration Reference<\/h1>\n\n\n\n
          Variable<\/strong><\/td>Purpose<\/strong><\/td><\/tr>
          RSI_GOAL<\/td>Optimization target for the run<\/td><\/tr>
          RSI_CYCLES<\/td>Maximum cycles (default 5)<\/td><\/tr>
          RSI_USE_VISION \/ RSI_USE_AUDIO<\/td>Enable or disable modality pipelines<\/td><\/tr>
          RSI_MIN_CYCLES<\/td>Minimum cycles before convergence check<\/td><\/tr>
          RSI_PROFILE<\/td>Named preset (e.g. StrategicOnly)<\/td><\/tr>
          RSI_MENU<\/td>Interactive bootstrap menu<\/td><\/tr>
          RSI_DRY_RUN<\/td>Propose improvements without applying<\/td><\/tr>
          RSI_RESUME<\/td>Resume from rsi-run-state.json<\/td><\/tr>
          RSI_FLEET<\/td>Broadcast lightweight commands to fleet<\/td><\/tr>
          RSI_AB_TEST<\/td>Alternate command variants per cycle<\/td><\/tr>
          RSI_SELFTEST<\/td>Run built-in unit tests (23 checks), no agent loop<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

          10. How to Run<\/h1>\n\n\n\n

          Standard run:<\/strong><\/p>\n\n\n\n

            \n
          • simple script execute lua_scripts\/RSI-Audio-Visual-Agent.lua<\/li>\n<\/ul>\n\n\n\n

            Strategic-only example (PowerShell):<\/strong><\/p>\n\n\n\n

              \n
            • $env:RSI_PROFILE = “StrategicOnly”<\/li>\n\n\n\n
            • $env:RSI_GOAL = “Improve memory recall for legal research”<\/li>\n\n\n\n
            • simple script execute lua_scripts\/RSI-Audio-Visual-Agent.lua<\/li>\n<\/ul>\n\n\n\n

              Self-test (no GPU\/API agent loop):<\/strong><\/p>\n\n\n\n

                \n
              • $env:RSI_SELFTEST = “1”<\/li>\n\n\n\n
              • lua lua_scripts\/RSI-Audio-Visual-Agent.lua<\/li>\n<\/ul>\n\n\n\n

                11. Runtime Expectations<\/h1>\n\n\n\n

                On a workstation-class host (e.g. Intel i9-10980XE, 64 GB RAM, NVIDIA GTX 1080 Ti), a full 5-cycle run with vision and audio typically requires roughly 15\u201335 minutes; early convergence often finishes in 6\u201315 minutes. StrategicOnly runs are substantially faster because they skip GPU-heavy capture, OCR, detection, and transcription commands.<\/p>\n\n\n\n

                12. Limitations<\/h1>\n\n\n\n
                  \n
                • Coroutines in ThreadManager do not achieve true OS-level parallelism for heavy commands<\/li>\n\n\n\n
                • Improvement quality depends on strategist LLM availability and prompt adherence<\/li>\n\n\n\n
                • Fleet broadcast requires remote commands infrastructure configured in the host app<\/li>\n\n\n\n
                • The script name reflects its origin; StrategicOnly profile operates without audio-visual pipelines<\/li>\n<\/ul>\n\n\n\n

                  13. Conclusion<\/h1>\n\n\n\n

                  RSI-Audio-Visual-Agent.lua transforms _AugmentedIntelligence from a static configuration into a self-tuning system. By combining structured LLM proposals, command-router intelligence, staged pipelines, safety gates, and persistent learning across runs, it provides a practical path toward autonomous optimization of complex audio-visual and strategic-reasoning workloads\u2014while remaining fully inspectable through logs, JSON state, and exported markdown reports.<\/p>\n\n\n\n

                  \u2014 End of document \u2014<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

                  https:\/\/docs.google.com\/document\/d\/1KHNlXblKdrB9MfewDr7AzwlvZE-ZlBv4\/edit?usp=drive_link&ouid=107277234409707752355&rtpof=true&sd=true Abstract RSI-Audio-Visual-Agent.lua is a goal-driven recursive self-improvement (RSI) orchestrator for the _AugmentedIntelligence v25 console application. The script does not implement machine-learning models directly; instead, it coordinates the host application\u2019s speech commands, Lua globals, memory systems, and perception pipelines through a repeated evaluate\u2013improve\u2013apply cycle. Each cycle gathers multimodal feedback (vision, audio, memory, executive planning, ethics, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1827","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/posts\/1827","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/comments?post=1827"}],"version-history":[{"count":3,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/posts\/1827\/revisions"}],"predecessor-version":[{"id":1830,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/posts\/1827\/revisions\/1830"}],"wp:attachment":[{"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/media?parent=1827"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/categories?post=1827"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/tags?post=1827"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}