_AugmentedIntelligence Agent Orchestration

This document covers the unified agent ... command namespace added around the existing native and Lua-backed agents.

Related docs:

• docs/SimpleAgents.md
• docs/KnowledgeRL.md
• docs/GamingRL.md
• docs/DrivingRL.md
• docs/GameModBridges.md
• docs/AugmentedIntelligenceSystems.md

What Was Added

• Central agent registry for native agents and Lua agents from the workbook.
• Fleet profiles for common operating modes.
• Runtime commands for start, stop, pause, resume, restart, status, config, and emergency stop.
• In-memory approval queue for gated actions.
• JSONL event log at agent_runtime\events.jsonl.
• JSONL handoff log at agent_runtime\handoffs.jsonl.
• Per-agent local memory notes in agent_runtime\memory\.
• Windows integration catalog at windows_integration\commands\agents-actions.json.

Core Commands

agent help
agent registry
agent profiles
agent profile show <profile>
agent profile apply <profile> [dry-run]
agent list
agent status [agent]
agent start <agent>
agent stop <agent|all|type <type>>
agent pause <agent>
agent resume <agent>
agent restart <agent>
agent config
agent config max_concurrent <count>
agent rl status
agent rl context
agent emergency stop

Shared RL Context

Every agent runtime Lua state now receives the Knowledge RL and Gaming RL Lua globals. Blueprint-backed agents also append two review-safe steps to their plan and run sequence:

Knowledge RL status
Gaming RL status

Useful commands:

agent rl status
simple agent rl status
knowledge rl stats
gaming rl status

Per-agent opt-out is available with <AGENT_ENV_PREFIX>_RL_CONTEXT=0. For example, CFO_RL_CONTEXT=0 disables the automatic blueprint RL context steps for the CFO simple agent.

Simple Agent Commands

The simple agents section from C:\Users\Temp\Desktop\_AugmentedIntelligence Commands.xlsx is available through both the normal agent ... namespace and a convenience simple agent ... namespace.

See docs/SimpleAgents.md for the full simple-agent reference.

simple agent registry
simple agent <agent name>
simple agent start <agent name>
simple agent status <agent name>
simple agent permissions <agent name>
agent profile show spreadsheet_simple_agents
agent profile apply spreadsheet_simple_agents dry-run

Examples:

simple agent cfo
simple agent human resources agent
simple agent mechanical engineer
simple agent investment strategist agent
simple agent stocks

These agents use lua_scripts\Simple-Agent-Profiles.lua and are review-only by default. Medical, mental-health, legal, finance, security, and cannabis-related profiles are framed as information/research support, not professional advice.

Approval Commands

agent approvals list
agent approval request <agent> <action> [detail]
agent approval approve <id>
agent approval deny <id>
agent approval clear

Approval records are currently local runtime records. They are useful for operator workflow and audit trails. Agents that perform high-impact work should consult this queue before taking live actions.

Handoff, Task, And Memory Commands

agent handoff <from> <to> <summary>
agent task <agent> <task>
agent memory show <agent>
agent memory append <agent> <note>
agent memory clear <agent>
agent memory export <agent> <path>
agent events [count]
agent logs [count]

agent task records a task in the target agent's memory file and creates an approval request before action.

Built-In Agents

Native agents:

• human
• desktop
• cybernetic
• rsi

Lua-backed agents:

• sysadmin
• electrical
• cybersecurity
• red_blue
• android
• federal
• memory_curator
• safety_auditor
• training
• build_test
• remote_llm
• minecraft_rl
• parenting_rl

Spreadsheet simple agents:

• cfo, cto, cio
• doctor, counselor, psychologist, psychiatrist
• tyler, blogger, leonardo_da_vinci, steven_hawking
• mechanical_engineer, electrical_engineer, neuroscientist, software_engineer
• human_resources, debug_agent, tester, research_agent, security
• manager, assistant, it, marketing, lawyer, legal_secretary
• friend, comic, investment_strategist, technical_support
• guy_contractor, gal_support, brainstorming_person, event_summation
• researcher, marijuana_guru, stocks

Use agent registry for descriptions, script paths, aliases, and permission labels.

Built-In Profiles

• voice_first_assistant
• help_desk_operations
• multi_agent_orchestrator
• cybernetic_control_lab
• ai_red_team_lab
• neural_rl_lab
• minecraft_rl_lab
• safe_family_assistant
• engineering_lab
• spreadsheet_simple_agents

Examples:

agent profile apply minecraft_rl_lab dry-run
agent profile apply help_desk_operations
agent profile apply neural_rl_lab
agent fleet start
agent fleet stop
agent fleet debate compare minecraft training reward functions

Profiles that may touch sensitive workflows create approval records. The scripts still start as local runtime scripts; live/high-impact actions should be gated by approvals and existing safety systems.

Windows Integration

Search or run agent actions from PowerShell:

Get-AugmentedIntegrationActions -Catalog agents -Query "minecraft"
.\windows_integration\integration-action.ps1 -Prompt -Catalog agents -UseResident
.\windows_integration\integration-action.ps1 -Id agent_profile_minecraft_rl_dry_run -UseResident

The tray menu also includes Integration Actions > Agents.

Neural Network Ideas For Controlling Agents

These are roadmap ideas for a neural controller above the individual agents. The goal is not one giant model that blindly controls everything. The useful design is a layered controller that proposes routing, priority, and policy choices while safety gates and approvals remain outside the model.

1. Agent router network - Input: current command, user profile, active context, agent registry, safety labels, runtime status. - Output: best target agent, backup agent, confidence, and whether approval is required. - Training data: historical commands, accepted handoffs, failed routes, and operator corrections.

2. Multi-agent supervisor policy - Input: fleet state, open tasks, deadlines, agent health, recent errors, approval state. - Output: start/stop/pause/resume decisions and max concurrency recommendations. - Start in shadow mode by comparing recommendations against human choices.

3. Handoff quality scorer - Input: handoff summary, source agent memory, destination agent scope, task outcome. - Output: score for completeness, risk, ambiguity, and missing context. - Use it to improve agent handoff packets before they are accepted.

4. Approval risk model - Input: proposed action, target agent, command text, safety labels, environment, recent incidents. - Output: risk class, required reviewer type, and whether dry-run is mandatory. - This should be conservative and auditable, not an automatic approval model.

5. Agent memory retrieval ranker - Input: current task and each agent's memory notes. - Output: top memories to inject into the agent's context. - Useful for keeping long-running agents focused without flooding prompts.

6. Fleet reward model - Input: task outcome, number of agent turns, delays, safety events, human edits, approval denials. - Output: scalar reward for orchestration quality. - Feeds offline RL for routing and scheduling, not direct unrestricted action.

7. Mixture-of-experts control layer - Each agent is an expert. A gating network chooses one or more experts per task. - Works well when human, desktop, sysadmin, training, safety_auditor, and minecraft_rl have distinct scopes.

8. Graph neural network over agent topology - Nodes: agents, tasks, tools, memories, approvals, servers. - Edges: can-handoff-to, depends-on, blocks, supervises, shares-memory-with. - Output: routing plan and bottleneck predictions.

9. Offline imitation model from operator traces - Train on operator choices: which agent was started, what handoff was written, which approval was granted or denied. - Deploy as a recommendation system first.

10. Hierarchical RL controller

    • High-level policy chooses profile, agent, and task phase.
    • Low-level policies stay inside domain subsystems such as Minecraft DQN, desktop automation, or audio/vision RSI.
    • Use offline and shadow stages before any live control.

11. Debate summarizer and judge

    • Agents propose plans, a small judge model scores consistency, safety, and evidence.
    • The final output is a recommended plan plus dissent notes, not an automatic action.

12. Anomaly detector for agent behavior

    • Input: command rate, repeated failures, approval denials, unusual handoff loops, memory churn.
    • Output: pause recommendation or emergency-stop recommendation.

13. Curriculum controller

    • Chooses which profile or task class to train next based on weaknesses.
    • Useful for Minecraft RL, desktop support drills, and systems administration tabletop scenarios.

14. Capability embedding model

    • Embed every agent description, permission label, command, and script description.
    • Use nearest-neighbor search plus a reranker to select agents for new tasks.

15. Safety shield policy

    • Hard layer around all learned controllers.
    • Enforces no-go actions, approval requirements, dry-run-first rules, rate limits, and emergency stop.

Suggested Training Progression

1. Log-only mode: collect agent task, agent handoff, agent approvals, agent profile apply, and agent events.
2. Supervised router: train a classifier to predict the selected agent/profile from operator commands.
3. Shadow recommendations: show predicted route and approval risk, but do not act.
4. Offline fleet reward model: score historical sessions for efficiency, safety, and result quality.
5. Offline RL scheduler: learn start/stop/pause/handoff suggestions from logged sessions.
6. Gated dry-run: allow the controller to create plans, handoffs, and approval requests only.
7. Gated live actions: require explicit approval for live starts, shell-capable agents, desktop control, server changes, and game control.