_AugmentedIntelligence Agent Orchestration

This document covers the unified agent … command namespace added around the existing native and Lua-backed agents.

Related docs:

docs/SimpleAgents.md
docs/KnowledgeRL.md
docs/GamingRL.md
docs/DrivingRL.md
docs/GameModBridges.md
docs/AugmentedIntelligenceSystems.md

What Was Added

Central agent registry for native agents and Lua agents from the workbook.
Fleet profiles for common operating modes.
Runtime commands for start, stop, pause, resume, restart, status, config, and emergency stop.
In-memory approval queue for gated actions.
JSONL event log at agent_runtime\events.jsonl.
JSONL handoff log at agent_runtime\handoffs.jsonl.
Per-agent local memory notes in agent_runtime\memory\.
Windows integration catalog at windows_integration\commands\agents-actions.json.

Core Commands

[text]
agent help
agent registry
agent profiles
agent profile show <profile>
agent profile apply <profile> [dry-run]
agent list
agent status [agent]
agent start <agent>
agent stop <agent|all|type <type>>
agent pause <agent>
agent resume <agent>
agent restart <agent>
agent config
agent config max_concurrent <count>
agent rl status
agent rl context
agent emergency stop

Shared RL Context

Every agent runtime Lua state now receives the Knowledge RL and Gaming RL Lua

globals. Blueprint-backed agents also append two review-safe steps to their plan

and run sequence:

[text]
Knowledge RL status
Gaming RL status

Useful commands:

[text]
agent rl status
simple agent rl status
knowledge rl stats
gaming rl status

Per-agent opt-out is available with <AGENT_ENV_PREFIX>_RL_CONTEXT=0. For

example, CFO_RL_CONTEXT=0 disables the automatic blueprint RL context steps

for the CFO simple agent.

Simple Agent Commands

The simple agents section from C:\Users\Temp\Desktop\_AugmentedIntelligence Commands.xlsx

is available through both the normal agent … namespace and a convenience

simple agent … namespace.

See docs/SimpleAgents.md for the full simple-agent reference.

[text]
simple agent registry
simple agent <agent name>
simple agent start <agent name>
simple agent status <agent name>
simple agent permissions <agent name>
agent profile show spreadsheet_simple_agents
agent profile apply spreadsheet_simple_agents dry-run

Examples:

[text]
simple agent cfo
simple agent human resources agent
simple agent mechanical engineer
simple agent investment strategist agent
simple agent stocks

These agents use lua_scripts\Simple-Agent-Profiles.lua and are review-only by

default. Medical, mental-health, legal, finance, security, and cannabis-related

profiles are framed as information/research support, not professional advice.

Approval Commands

[text]
agent approvals list
agent approval request <agent> <action> [detail]
agent approval approve <id>
agent approval deny <id>
agent approval clear

Approval records are currently local runtime records. They are useful for operator workflow and audit trails. Agents that perform high-impact work should consult this queue before taking live actions.

Handoff, Task, And Memory Commands

[text]
agent handoff <from> <to> <summary>
agent task <agent> <task>
agent memory show <agent>
agent memory append <agent> <note>
agent memory clear <agent>
agent memory export <agent> <path>
agent events [count]
agent logs [count]

agent task records a task in the target agent’s memory file and creates an approval request before action.

Built-In Agents

Native agents:

human
desktop
cybernetic
rsi

Lua-backed agents:

sysadmin
electrical
cybersecurity
red_blue
android
federal
memory_curator
safety_auditor
training
build_test
remote_llm
minecraft_rl
parenting_rl

Spreadsheet simple agents:

cfo, cto, cio
doctor, counselor, psychologist, psychiatrist
tyler, blogger, leonardo_da_vinci, steven_hawking
mechanical_engineer, electrical_engineer, neuroscientist, software_engineer
human_resources, debug_agent, tester, research_agent, security
manager, assistant, it, marketing, lawyer, legal_secretary
friend, comic, investment_strategist, technical_support
guy_contractor, gal_support, brainstorming_person, event_summation
researcher, marijuana_guru, stocks

Use agent registry for descriptions, script paths, aliases, and permission labels.

Built-In Profiles

voice_first_assistant
help_desk_operations
multi_agent_orchestrator
cybernetic_control_lab
ai_red_team_lab
neural_rl_lab
minecraft_rl_lab
safe_family_assistant
engineering_lab
spreadsheet_simple_agents

Examples:

[text]
agent profile apply minecraft_rl_lab dry-run
agent profile apply help_desk_operations
agent profile apply neural_rl_lab
agent fleet start
agent fleet stop
agent fleet debate compare minecraft training reward functions

Profiles that may touch sensitive workflows create approval records. The scripts still start as local runtime scripts; live/high-impact actions should be gated by approvals and existing safety systems.

Windows Integration

Search or run agent actions from PowerShell:

[powershell]
Get-AugmentedIntegrationActions -Catalog agents -Query “minecraft”
.\windows_integration\integration-action.ps1 -Prompt -Catalog agents -UseResident
.\windows_integration\integration-action.ps1 -Id agent_profile_minecraft_rl_dry_run -UseResident

The tray menu also includes Integration Actions > Agents.

Neural Network Ideas For Controlling Agents

These are roadmap ideas for a neural controller above the individual agents. The goal is not one giant model that blindly controls everything. The useful design is a layered controller that proposes routing, priority, and policy choices while safety gates and approvals remain outside the model.

Agent router network
Input: current command, user profile, active context, agent registry, safety labels, runtime status.
Output: best target agent, backup agent, confidence, and whether approval is required.
Training data: historical commands, accepted handoffs, failed routes, and operator corrections.
Multi-agent supervisor policy
Input: fleet state, open tasks, deadlines, agent health, recent errors, approval state.
Output: start/stop/pause/resume decisions and max concurrency recommendations.
Start in shadow mode by comparing recommendations against human choices.
Handoff quality scorer
Input: handoff summary, source agent memory, destination agent scope, task outcome.
Output: score for completeness, risk, ambiguity, and missing context.
Use it to improve agent handoff packets before they are accepted.
Approval risk model
Input: proposed action, target agent, command text, safety labels, environment, recent incidents.
Output: risk class, required reviewer type, and whether dry-run is mandatory.
This should be conservative and auditable, not an automatic approval model.
Agent memory retrieval ranker
Input: current task and each agent’s memory notes.
Output: top memories to inject into the agent’s context.
Useful for keeping long-running agents focused without flooding prompts.
Fleet reward model
Input: task outcome, number of agent turns, delays, safety events, human edits, approval denials.
Output: scalar reward for orchestration quality.
Feeds offline RL for routing and scheduling, not direct unrestricted action.
Mixture-of-experts control layer
Each agent is an expert. A gating network chooses one or more experts per task.
Works well when human, desktop, sysadmin, training, safety_auditor, and minecraft_rl have distinct scopes.
Graph neural network over agent topology
Nodes: agents, tasks, tools, memories, approvals, servers.
Edges: can-handoff-to, depends-on, blocks, supervises, shares-memory-with.
Output: routing plan and bottleneck predictions.
Offline imitation model from operator traces
Train on operator choices: which agent was started, what handoff was written, which approval was granted or denied.
Deploy as a recommendation system first.
Hierarchical RL controller
High-level policy chooses profile, agent, and task phase.
Low-level policies stay inside domain subsystems such as Minecraft DQN, desktop automation, or audio/vision RSI.
Use offline and shadow stages before any live control.
Debate summarizer and judge
Agents propose plans, a small judge model scores consistency, safety, and evidence.
The final output is a recommended plan plus dissent notes, not an automatic action.
Anomaly detector for agent behavior
Input: command rate, repeated failures, approval denials, unusual handoff loops, memory churn.
Output: pause recommendation or emergency-stop recommendation.
Curriculum controller
Chooses which profile or task class to train next based on weaknesses.
Useful for Minecraft RL, desktop support drills, and systems administration tabletop scenarios.
Capability embedding model
Embed every agent description, permission label, command, and script description.
Use nearest-neighbor search plus a reranker to select agents for new tasks.
Safety shield policy
Hard layer around all learned controllers.
Enforces no-go actions, approval requirements, dry-run-first rules, rate limits, and emergency stop.

Suggested Training Progression

Log-only mode: collect agent task, agent handoff, agent approvals, agent profile apply, and agent events.
Supervised router: train a classifier to predict the selected agent/profile from operator commands.
Shadow recommendations: show predicted route and approval risk, but do not act.
Offline fleet reward model: score historical sessions for efficiency, safety, and result quality.
Offline RL scheduler: learn start/stop/pause/handoff suggestions from logged sessions.
Gated dry-run: allow the controller to create plans, handoffs, and approval requests only.
Gated live actions: require explicit approval for live starts, shell-capable agents, desktop control, server changes, and game control.

Macdaddy4sure's Blog