Tyler Crockett | Macdaddy4sure.ai

Abstract

This paper describes Augmented Intelligence v11.0 — Typed Commands, a large-scale Windows-native software system that integrates multimodal perception, tiered memory, large language model (LLM) inference, desktop automation, and domain-specific knowledge processing into a unified runtime. The system comprises approximately 299 C++ translation units and 309 header modules, orchestrated through a command-driven interaction model in which natural-language-style typed commands (and optionally speech) are parsed and dispatched to hundreds of specialized handlers. Core capabilities include dual-camera vision with TensorFlow and LiteRT-based object detection, dual-microphone audio capture and Whisper transcription, MySQL-backed persistent memory, local and remote LLM backends (Ollama, llama.cpp, and multiple cloud APIs), computer-use automation with goal-directed WikiHow integration, and a U.S. Code legal corpus with full-text search. The architecture is explicitly informed by cognitive psychology constructs—working memory, short-term and long-term recall, and multiple intelligences—translated into concurrent software threads and database schemas. This document provides a descriptive overview of the system’s design, components, and intended research trajectory rather than empirical evaluation results.

1. Introduction

Contemporary personal AI assistants are predominantly chat-centric: they accept text input, produce text output, and maintain limited session context. Augmented Intelligence pursues a broader thesis: that useful machine cognition for an individual operator requires continuous sensing, structured memory, action in the physical and digital environment, and specialized domain reasoning—not merely conversational fluency.

Version 11.0, subtitled “Typed Commands,” reflects a deliberate interface choice. While speech recognition is supported, the primary control surface is a console prompt (>>) accepting tokenized command streams routed through a central interpreter (SpeechCommands.cpp). This design treats the system as an operating environment for augmented cognition rather than a single application.

The software is authored by Tyler Crockett under the Apache License 2.0 and is built as a monolithic executable (_AugmentedIntelligence.exe) linking OpenCV, TensorFlow, LiteRT (TensorFlow Lite C API v2.18.0), llama.cpp/ggml (with optional CUDA), Whisper, MySQL, FFmpeg, Tesseract, Lua, Python, and numerous other native libraries on Windows x64.

2. Background and Motivation

2.1 Cognitive Foundations

The project’s AugmentedIntelligence.cpp header commentary frames design goals using established psychological models:

• Cattell–Horn–Carroll (CHC) factors: crystallized intelligence, fluid intelligence, quantitative reasoning, reading/writing, short-term and working memory, long-term storage and retrieval, visual and auditory processing, processing speed.
• Sternberg’s triarchic theory: analytical, creative, and practical intelligence.
• Gardner’s multiple intelligences: bodily-kinesthetic, musical, spatial, linguistic, logical-mathematical, interpersonal, and intrapersonal capacities.

These are not implemented as formal psychometric instruments; rather, they motivate parallel software subsystems—visual checking threads per academic domain, listening pipelines, reading ingestion, and executive-function-style goal management on the desktop.

2.2 Design Goals

1. Ambient awareness — persistently capture and index what the system sees and hears.
2. Recallable memory — promote salient events from working memory through short-term into long-term storage, queryable via ranked keyword search.
3. Grounded action — type into applications, navigate UIs, pursue procedural goals.
4. Verifiable reasoning — optional ethics, fallacy, bias, abuse, and legal checking across modalities.
5. Extensibility — hundreds of domain modules (“modes”) activatable by command.
7. System Architecture

3.1 Runtime Model

The application follows a three-phase lifecycle:

1. Initialization — legal terms acceptance; _AI::InitDatabases() initializes MySQL schemas (including U.S. Code, book databases, and program tables).
2. Configuration — extensive settings menus backed by MySQL persistence and named application profiles (save/load/update configuration snapshots).
3. Execution — “Run Program” invokes _AI::InitThreads(), spawning detached std::thread workers according to enabled settings; a parallel typing-command loop accepts user input.

┌──────────────────────────────────────────────────────────────┐
│ Main Thread: Terms → Menu → Settings / Run Program │
└────────────────────────────┬─────────────────────────────────┘
│ InitThreads()
▼
┌──────────────────────────────────────────────────────────────┐
│ Background Workers (conditional on settings) │
│ • TypingCommands • Vision (cameras) │
│ • Sound / Stream • Whisper (speech) │
│ • Visual checking • Listening checking │
│ • Reading • Computers (desktop agent) │
│ • Driving, Gaming, domain-specific analyzers │
└────────────────────────────┬─────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ SpeechCommands — central command router │
│ → LLM APIs, memory, parsers, modes, automation, databases │
└──────────────────────────────────────────────────────────────┘

3.2 Command-Centric Integration

Nearly all functionality converges on _SpeechCommands::SpeechCommands(vector command), a large dispatch function that:

• Handles emergency shutdown
• Routes legal/HTML parsing (parse federal law, parse html, search us code)
• Invokes LLM backends (simple_text, simple_image, multi-API comparison)
• Activates domain modes (mode engage )
• Delegates to Lua scripts with bindings for memory, settings, sockets, and reinforcement-learning globals
• Optionally executes each command in an isolated thread (ExecuteCommandThreaded) for concurrent operation and future GUI terminal visualization

This hub-and-spoke pattern trades modularity at the linker level for unified observability and consistent memory logging at the command layer.

4. Memory Architecture

The system implements a four-tier memory model synchronized between in-process buffers and MySQL databases.

Tier	Role	Representative globals / tables
Working memory	Live scratchpad for current cognition	wm_simple_text, wm_simple_image, vision paths, sound paths, LLM history
Short-term memory	Rolling buffer; promoted to working memory on schedule	stm_simple_text, stm_simple_image, etc.
Long-term memory	Cross-session persistence	MySQL long-term memory database
Specialized corpora	Domain-specific structured knowledge	US Code (us_code_sections), WikiHow goals, passwords, vision/sound metadata

_DatabaseFunctions provides:

• Ranked keyword search (MemorySearchRanked) across memory tables
• U.S. Code FULLTEXT search with SQL LIMIT (not full table scans)
• WikiHow article matching for computer goals
• Password storage with encryption integration
• Music metadata and lyrics search

Settings control enable flags and capacity limits (e.g., working_memory_limit, long_term_memory_limit defaulting to 1000 slots).

5. Perception Subsystems

5.1 Vision

The _Vision module manages up to two configurable cameras (e.g., Depstech webcam, GoPro Hero9) with independent FPS and resolution settings. Capabilities include:

• Object detection via TensorFlow SavedModel graphs and/or LiteRT/TFLite interpreters
• Specialized detectors for domain-specific classes (ants, birds, vessels, balls, trash, etc.)
• Vision memory — frames written to D:/vision_memory and indexed in MySQL
• Visual analysis — LLM vision models via Ollama (e.g., llama3.2-vision:11b)
• OCR and text pipelines — Tesseract plus TFLite text detection/recognition models
• Visual checking — when enabled, spawns parallel per-domain analysis threads (algebra, biology, ethics, law, engineering, and dozens more) that continuously interpret camera input through subject-specific lenses

5.2 Audio and Speech

• Microphone 1 — archival environmental recording (Sound.cpp → D:/sound_memory)
• Microphone 2 — low-latency streaming for speech pipeline (Sound_Streaming)
• Whisper — local (whisper.lib) or remote server transcription
• Speech commands — transcribed utterances fed into the same SpeechCommands router as typed input
• Listening checking — optional parallel domain analyzers on audio (mirroring visual checking architecture)

5.3 Reading

The _Reading and _Computers modules support screen and document text ingestion, integrating with working memory and LLM analysis commands (simple_image, reading reasoning flags).

6. Reasoning and Language Models

6.1 Backend Diversity

Backend	Integration	Notes
Ollama	_LLM3::OllamaAPI	Primary local/remote path; supports image models
llama.cpp	LlamaCppIntegration.cpp	Direct GGUF inference; llama_cpp_cuda_enabled gates GPU layers
Cloud APIs	Settings per provider	ChatGPT, Grok, Gemini, Llama API, Mistral, Claude

Responses can be logged to working memory, stored in MySQL, and printed comparatively across providers. Optional ethics_check_enabled gates generation.

6.2 Command Patterns

• simple_text — text-only LLM inference
• simple_image — vision-conditioned inference
• Multi-API mode — parallel requests with unified output formatting (PrintOutput)

Threading is configurable (typing_commands_threads) to avoid blocking the command prompt during long inference.

7. Action and Computer Use

7.1 Desktop Agent (_Computers)

The computer-use subsystem treats the Windows desktop as an observable, actionable environment:

• Desktop capture (CaptureDesktop), cursor/window tracking
• Active window identification and task logging to MySQL
• Goal management — SetGoal, WikiHow solution matching, step-indexed procedure execution
• On-screen text detection (Tesseract + TFLite) and object-detection assist during UI navigation
• AES-backed password retrieval for automated login flows
• Speech-command handlers for enabling/disabling “computer aids”

7.2 Keyboard Automation (_Typing)

A queued typing service uses Win32 SendInput for character-by-character injection, with optional vision-based virtual keyboard detection. This bridges LLM outputs and procedural plans into real application interaction.

7.3 Creative Application Modules

Dedicated handlers exist for Adobe Creative Cloud applications (Photoshop, Premiere, Illustrator, After Effects, Audition, Bridge, Lightroom, InDesign, Dreamweaver, Flash, Animate, Media Encoder), indicating intent to automate professional media workflows.

8. Domain Knowledge Modules

Beyond core infrastructure, the project contains hundreds of paired .hpp/.cpp modules, each encapsulating a subject domain. Representative categories:

• Formal sciences: algebra, calculus, geometry, logic, graph theory, cryptography
• Natural sciences: biology, chemistry, physics, ecology, geology, astronomy
• Engineering: aerospace, civil, electrical, mechanical, structural, computer hardware
• Social sciences: economics, political science, psychology, ethics, game theory
• Applied skills: cooking, camping, driving, farming, navigation
• Sports and games: baseball, basketball, football, golf, chess (with ChessBot), billiards; League of Legends vision pipeline (GamingLoL with TFLite champion/skin classifiers); RL policy server integration (GamingRL)

Domains are engaged via mode engage commands and may spawn physics, geometry, or simulation state tracked for a planned Dear ImGui visualization layer.

9. Data Infrastructure and Legal Corpus

9.1 MySQL as System of Record

Settings, memory tiers, vision/sound metadata, computer logs, passwords, and legal corpora reside in MySQL (ai_settings database and domain-specific databases). The SettingsMySQL layer provides:

• SaveAll / LoadAll across categorized settings tables (llm_settings, whisper_settings, driving_settings, etc.)
• Application profile CRUD (application_profiles, application_profile_data)
• Seeding of empty tables on first run

Recent work added llama_cpp_cuda_enabled to llm_settings and profile snapshots.

9.2 U.S. Code Integration

HtmlParser ingests federal law HTML (recursively through subdirectories). Sections are stored in us_code.us_code_sections with a FULLTEXT index (ft_us_code_search). Commands such as search us code invoke ranked SQL search (keywordSearchUsCodeRanked) rather than exhaustive scans.

9.3 Remote Storage

FTP settings support offloading vision and sound archives to remote hosts (e.g., ftp://macdaddy4sure.ai), with configurable storage limits.

10. Machine Learning and Acceleration Stack

Component	Version / Path	Function
LiteRT (TFLite C API)	v2.18.0 in third_party/litert/	Object detection, text OCR, LoL classifiers
TensorFlow C API	SavedModel graphs	COCO detection, legacy pipelines
llama.cpp / ggml	CUDA 12.5 build	Local GGUF LLM inference
Whisper	whisper.lib	Speech-to-text
OpenCV 4.9	opencv_world490	Capture, processing, DNN
CUDA	12.5 (aligned with ggml-cuda)	GPU matrix ops, LLM layers
Tesseract 5.5	OCR	Screen and document text

Post-build deployment copies runtime DLLs (tensorflowlite_c.dll, cudart64_12.dll, cublas64_12.dll, etc.) adjacent to the executable.

11. User Interface and Observability

11.1 Console Interface

The primary UI is a text console: main menu, nested settings wizards, and the >> command prompt. A help command prints the command catalog, organized into topics (emergency, knowledge, socket, gaming, computer, driving, reading, simple modes, chat, HTML, analysis, memory, settings, navigation, Lua, domains).

11.2 Planned GUI (Gui.hpp)

A Dear ImGui + DirectX 11 dashboard is scaffolded for:

• Live camera feeds with object-detection overlays
• Audio waveform and spectrogram panes
• Per-thread terminal windows (integrating with ExecuteCommandThreaded logging)
• Mode-specific physics/geometry visualizations

The GUI is designed for hybrid operation alongside the existing console.

12. Security, Ethics, and Legal Compliance

• Recording disclaimer at startup documents U.S. state wiretap consent regimes (one-party vs. all-party).
• Ethics module can gate LLM outputs.
• Multi-modal checking — fallacy, bias, abuse, and law analyzers across visual, listening, and reading pipelines.
• Encryption for stored credentials.
• Private computer and hostname-based access settings.

13. Implementation Status and Engineering Notes

During recent development sessions, several integration issues were addressed:

1. llama.cpp CUDA — ggml-cuda.lib built with shared CUDA runtime; project pinned to CUDA 12.5 (resolving cudaGetDeviceProperties_v2 link errors caused by CUDA 13.0 cudart.lib mismatch).
2. LiteRT upgrade — TensorFlow Lite upgraded to v2.18.0; binaries installed under third_party/litert/ with post-build DLL copy to x64/Release/.
3. Settings persistence — llama_cpp_cuda_enabled wired through Settings menu, MySQL llm_settings, and application profiles.
4. U.S. Code search — FULLTEXT indexing and bounded SQL queries; recursive HTML folder parsing.

The codebase exhibits organic growth characteristics: an extremely large SpeechCommands.cpp, hundreds of domain files, and settings menus numbering thousands of lines. This reflects iterative feature accretion over multiple versions (v4.5 through v11.0).

14. Discussion

14.1 Strengths

• Unusually broad integration surface for a personal system: perception, memory, action, and reasoning in one process.
• Explicit memory hierarchy with database backing, enabling longitudinal study of an operator’s digital life.
• Backend-agnostic LLM layer supporting local privacy-preserving inference and cloud augmentation.
• Legal and procedural knowledge (U.S. Code, WikiHow goals) moves beyond generic chat into structured corpora.

14.2 Limitations

• Monolithic architecture complicates testing, packaging, and modular deployment.
• Windows-only with heavy native dependencies.
• Empirical evaluation — no standardized benchmarks, user studies, or accuracy metrics are embedded in the repository.
• Command discoverability — the vast command surface requires help and experience; no graphical command builder is complete.
• Resource intensity — simultaneous vision, listening, checking, and LLM threads can stress CPU, GPU, and memory.

15. Future Work
16. Complete Dear ImGui dashboard and thread-terminal observability.
17. Formalize command grammar and publish machine-readable specification.
18. Empirical evaluation of memory recall, legal search precision, and computer-use goal completion.
19. Modularize SpeechCommands into plugin-style dispatch tables.
20. Cross-platform abstraction layer (currently Windows-specific: Win32, DXGI, SendInput).
21. Unified model registry for TensorFlow, LiteRT, and ONNX interchangeability.
22. Federated memory sync across machines via existing networking/socket infrastructure.

16. Conclusion

Augmented Intelligence v11.0 represents an ambitious attempt to build a personal cognitive architecture on commodity PC hardware. By coupling continuous multimodal sensing, tiered MySQL-backed memory, diverse LLM backends, desktop automation, and hundreds of domain-specific analyzers under a unified typed-command interface, the system pursues a vision of AI as an ambient, actionable, and memory-rich partner rather than a stateless chat endpoint. While formal evaluation remains future work, the codebase documents a comprehensive research prototype spanning computer science, cognitive science, law, robotics-adjacent perception, and human-computer interaction—a descriptive foundation for continued empirical and engineering study.

References (Conceptual)

• Cattell–Horn–Carroll theory of cognitive abilities
• Sternberg’s triarchic theory of intelligence
• Gardner’s theory of multiple intelligences
• Google LiteRT (formerly TensorFlow Lite) C API documentation
• llama.cpp / ggml GPU backend documentation
• OpenAI Whisper speech recognition
• U.S. Code corpus (House Office of the Law Revision Counsel)

---

Filed under: Uncategorized - @ June 21, 2026 1:59 pm