Docs  /  Semantic embeddings → full `_AugmentedIntelligence` brainstorm

Semantic embeddings → full _AugmentedIntelligence brainstorm

Status: training result is live in MySQL + offline cache (2026-07-18 run).
Purpose: brainstorm everything that result means for the AI — not only storage, but every consumer, safety gate, RL loop, and product surface.

Related

Doc Role
SemanticVectors.md Pipeline, methods, commands
BookBibleMediaVectors.md Original books/Bible/media *2vec design
MySQLDatabases.md Program DB catalog
MySQLAIFeatures.md Hybrid RAG / audit / dialogue
BrainstormStack.md Session stack hub
KnowledgeRL.md Epistemic / claim RL
LegalCorpora.md · LegalShield.md Law text + gates
EthicsGood.md “Be Good” shaping

Hub modules: _SemanticVectors · _CorpusFeatures · _MySQLAIFeatures · _BrainstormIntegrations

0. One-sentence vision

Treat the multi-2vec MySQL training result as the semantic nervous system of _AugmentedIntelligence: every typed command, agent, RAG query, ethics check, legal shield, game mod, and MathNN feature can ground language in a shared vector space trained on your* corpora (Bible, law, thoughts, literature, dictionary, wiki samples).

1. What the result actually is (facts)

1.1 Snapshot (post-train inventory)

Artifact Approx size / count Location
Document vectors (all methods) ~114k rows semantic_knowledge.semantic_vectors
Word2Vec tokens ~25k semantic_knowledge.glove_vectors model word2vec_corpus
Dual-write GloVe-mean terms ~19k semantic_vectors.embeddings
Offline C++ cache ~47 MB data/semantic_vectors_cache.tsv
Unit JSONL ~167 MB data/corpus/unit_vectors.jsonl
Teacher / anchor space GloVe 6B 50d (60k vocab load) data/glove/glove.6B.50d.txt

1.2 Methods stored (parallel views of the same world)

method Meaning in the AI Prefer when…
glove_mean Stable external space (Stanford GloVe mean) Cross-domain baselines, comparable to literature
w2v_mean Corpus-trained Word2Vec mean In-domain language (Bible/law/your books)
doc2vec_like Doc-id bias on top of mean Separate near-duplicate docs
sentence_mean First-sentence / first-40 tokens Short queries, verse-like units
fasttext_like Token mean + char-trigram hash OOV, archaic spelling, typos
top2vec_doc K-means topic centroid membership Theme browse, “what is this about?”

1.3 Source mix that trained the space

source_type ~rows (all methods) Program DBs / files
bible_verse ~60k bible.bible_verses_*
other ~41k thought, case_law, us_code, dictionary, discoveries…
book_unit ~12k literature_*, folded_literature*, books, wikisimple
wiki_page / wiktionary small offline samples + optional MySQL wiki

Trainer: python tools/semantic_vectors/train_2vec_to_mysql.py
Schema: sql/semantic_vectors_schema.sql · DB semantic_knowledge

2. Architecture: where vectors sit in the AI

  MySQL program corpora (bible, thought, case_law, us_code, literature, dict, …)
                │
                ▼
   train_2vec_to_mysql.py  ──►  semantic_knowledge.*  ──►  embeddings dual-write
                │                         │
                ▼                         ▼
   data/semantic_vectors_cache.tsv   SemanticVectors (RAM)
                │                         │
        ┌───────┴────────┬────────────────┼──────────────────┐
        ▼                ▼                ▼                  ▼
   CorpusFeatures   MySQLAIFeatures   Knowledge RL      Decision trees
   (search/topics)  (ai rag hybrid)   (claim similarity) (intent/domain)
        │                │                │                  │
        └────────────────┴────────────────┴──────────────────┘
                                │
                    EthicsGood · LegalShield · Domain LLM
                                │
                         Actions / Agents / Portal

Graph nodes (portal): semantic_multi2vec, semantic_mysql, corpus_rag — see graphs architecture.

3. Brainstorm by AI surface (everything the result enables)

3.1 Hybrid RAG (already partial)

Today: ai rag <query> → corpus glove_mean + token similar + doc similar.

Brainstorm next

Idea Detail
Multi-method vote Run w2v_mean + glove_mean + fasttext_like; fuse ranks (RRF)
Source filters ai rag law:… only case_law/us_code; ai rag bible:… only verses
Method flag ai rag method=w2v_mean …
Context pack for LLM Top-k labels + 400-char snippets into DomainLLM / CharacterChat
Audit trail Already logs rag_query_log — add method + source_type columns

3.2 Command intent & suggest

Idea Wire
Embed command utterance → nearest past successful commands command_usage + doc vectors of history
Disambiguate “train” (ML vs gym vs skills) via nearest domain docs _CommandSuggest + w2v_mean
Spell-check neighbors Similar tokens from GloVe/W2V for typos

3.3 Thought / decision trees

Idea Wire
Intent features from embed cosine to intent prototypes _ThoughtDecisionTrees::IntentFeatures
Domain route: closest cluster among top2vec_doc or hand-seeded centroids
Safety: cosine to known-harm phrase bank (LegalShield lexicon) Pre-gate before action trees

3.4 Ethics “Be Good” + LegalShield

Idea Wire
Shape reward when claim is near “truth / help / protect” clusters EthicsGood::EvaluateText + neighbor labels
Legal: if query near criminal-howto clusters in case_law space, refuse LegalShield + method=w2v_mean on law slice
Separate law-only subindex so fiction “murder mystery” ≠ real crime tutorial Filter corpus=case_law / us_code

3.5 Knowledge RL / Memory

Idea Wire
Epistemic label prior from nearest known claims KnowledgeAcquisitionRL
Memory promotion: embed episode summary; promote if novel vs long-term neighbors ai memory promote + doc NN
Working → short → long: store vector BLOB on memory_entries schema add embed_blob / method

3.6 Bible / books / literature (spiritual + literary AI)

Idea Wire
bible search method=w2v_mean love neighbor CorpusFeatures (exists)
Cross-ref: verse ↔ literature quote same theme dual source_type search
Tutoring: “explain this verse like Animal Farm theme” LLM + dual NN context
Full ASV 31k retrain (this run used 10k sample) raise --limit-bible

3.7 Law firm / US Code AI

Idea Wire
Section find: embed question → nearest us_code units already harvested
Case analogical search case_name + LEFT(case_text) vectors
Courtroom strategy agent: retrieve parallel cases lua Law-Firm scripts + ai rag
Scale: 829k USC sections — staged train / shard by title --limit-uscode growth plan

3.8 Thought graph & LLM thoughts

Idea Wire
“Have I thought this before?” NN over thought_graph_nodes content embeds
Session continuity: attach nearest past plan nodes AgentRuntime
Dedup LLM prompts by cosine threshold save tokens / cost

3.9 MathNN / KernelHub / NPU

Idea Wire
Embedding GEMM: batch cosine = (Q @ D^T) via KernelHub kernel gemm path
MathNN field math_embedding head: predict next-token sparse or rank docs MathNNMathFields
NPU later: ORT EP for encoder ONNX; today CPU AVX mean is fine NpuRuntime fallback
TF32 for large doc matrix once dim↑ (100d/300d GloVe) CutlassAccel

3.10 SimpleText / heuristics

Idea Wire
Verb extract → embed verb phrase → nearest action catalog SimpleTextBank + AugmentedActions
Offline heuristic when LLM down: retrieve only no network required

3.11 Agents (cybernetic, human, desktop, RSI)

Idea Wire
Each agent type gets a prototype vector (mean of its script docs) AgentRuntime register
Route user goal to agent by max cosine multi-agent dispatch
RSI audio-visual: caption → embed → memory CrossModal + semantic

3.12 Games / mods / RL

Idea Wire
Quest text / dialogue → nearest lore docs (if game wiki ingested) GameModBridge + corpus
Gaming RL reward shaping from ethics neighbors EthicsGood + gaming domain
Civ/Skyrim mods: natural language → nearest command template embed command bank

3.13 Portal / PHP graphs / Discovery

Idea Wire
Graph: method counts, source_type pie, train freshness graphs export + portal Charts
Discovery journal entry: “Semantic train complete 114k rows” auto discovery record after train
WordPress draft: “What the AI knows” digest from top topics Discovery → WP dry-run

3.14 Cross-modal & media (future of same spine)

Idea Wire
Whisper transcript → same doc table as media_segment BookBibleMediaVectors plan
Vision OCR text → embed → link to wiki/book vision DB + semantic
CrossModalEpisodicEmbedder align text space with image space via paired loss research track

3.15 Enterprise / AD / policy

Idea Wire
Policy text embed; user request NN to nearest policy EnterprisePolicy
Do not train on passwords/PII tables harvest denylist (settings, password columns)
Audit every train run in ai_features.ai_audit_log ai audit semantic_train …

4. Data plane brainstorm (MySQL as system of record)

4.1 Databases that feed training (read)

DB Tables (examples) Role in AI semantics
bible bible_verses_asv (+ KJV if present) Scripture unit space
thought llm_thoughts, thought_graph_nodes Self-memory of reasoning
case_law case_law_cases Precedent space
us_code us_code_sections Statute space
literature_* / folded_literature* / books book_text / rhetorical cols Literary culture
dictionary word/definition tables Lexicon anchors
discoveries journal_entries Program self-history
offline data/wiki|wiktionary|corpus JSONL Lab bootstrap

4.2 Databases that hold results (write)

DB.table Content
semantic_knowledge.semantic_vectors Multi-method doc BLOBs
semantic_knowledge.glove_vectors + glove_meta GloVe + W2V token BLOBs
semantic_vectors.embeddings Simple term→vector TEXT dual-write
ai_features.rag_query_log RAG usage
(planned) ai_settings profile Last train timestamp, limits, dim

4.3 Harvest policy (brainstorm rules)

  1. Clip bodies (--max-chars) so case_law does not explode RAM.
  2. Sample giant tables (us_code) with explicit limits; never silent full-table.
  3. Denylist credential / private tables.
  4. Idempotent upsert: delete prior multi-method bundle then insert.
  5. Always write offline cache so C++ works if MySQL is down.

5. Runtime command surface (operator cookbook)

# Status / load
semantic status
semantic load data/glove/glove.6B.50d.txt 60000
semantic load-docs
semantic methods
semantic method w2v_mean

# Query
semantic embed artificial intelligence
semantic similar intelligence 10
semantic docs method=w2v_mean neural network 10
semantic docs method=glove_mean love 8
semantic docs method=any faith hope 8

# Corpus / Bible / hybrid AI
corpus search method=w2v_mean neighbor 8
bible load-db both
ai rag method=w2v_mean due process equal protection
ai status

# Retrain (Python)
python tools/semantic_vectors/train_2vec_to_mysql.py --limit-bible 31102 --limit-uscode 10000

# Stack
integrate stack-status
graphs architecture

Lua (when bound):

-- via integrate / ai / corpus bindings as available
print(ai_mysql_rag_log("search for justice and mercy"))

6. Phased product roadmap (brainstorm → build)

Phase A — Done / live

  • [x] Multi-method train into semantic_knowledge
  • [x] Program MySQL harvest (bible, thought, law, literature, dict)
  • [x] Offline TSV + JSONL for C++
  • [x] Dual-write semantic_vectors.embeddings
  • [x] Typed commands: semantic *, corpus search, ai rag
  • [x] gensim Word2Vec path

Phase B — Wire deeper into AI ✅ implemented

  • [x] Auto semantic load-docs on InitBrainstormStack (size-aware via TryAutoLoadDocs)
  • [x] ai rag multi-method RRF + source filters (source=law|bible, law:…)
  • [x] Portal graph: method/source histograms (graphs semantic, graphs export → semantic.json)
  • [x] discovery record after successful train (discoveries.journal_entries + last_train_manifest.json)
  • [x] Intent prototypes for decision trees (emb_* features in IntentFeatures)
  • [x] LegalShield law-slice nearest-neighbor assist (howto+crime_lex + law NN)

Phase C — Scale & quality

  • [ ] Full ASV + KJV; full literature tables
  • [ ] Shard / incremental train (only new rows)
  • [ ] Real FastText / Doc2Vec (gensim) not only “like” projections
  • [ ] ANN index (HNSW / FAISS) once docs >> 100k
  • [ ] 100d/300d GloVe + KernelHub TF32 batch cosine
  • [ ] Media_segment + Whisper/OCR units

Phase D — Research / product

  • [ ] MathNN encoder distilled from multi-method targets
  • [ ] Cross-modal alignment with vision/sound
  • [ ] Multi-agent routing by prototype cosine
  • [ ] Federated embed store for remote CUDA box

7. Safety, ethics, and correctness

Risk Mitigation
Fiction / metaphor confused with real crime or self-harm Source filters; LegalShield + EthicsGood before action
PII leakage into embeddings Harvest denylist; no passwords, emails dumps
Stale vectors after corpus edit Train version in glove_meta.name + timestamp; retrain job
Dim mismatch GloVe vs W2V Trainer forces same dim; reject mixed loads
Overconfident NN (high cosine, wrong sense) Always show label + source_type; LLM must cite
Giant train OOM Limits + clip + offline fallback

8. Success metrics (how we know the AI improved)

Metric Target
semantic docs latency (k=10, ~20k docs) < 50 ms CPU after load
RAG hit rate (human spot-check) ≥ 70% relevant top-3 on lab queries
Cross-source theme recovery “mercy” hits bible + literature + law
Retrain reproducibility same seed → cosine drift < 0.05 on holdout
Offline resilience C++ works with TSV only, MySQL optional

Lab queries to keep

semantic docs method=w2v_mean love your neighbor 8
semantic docs method=w2v_mean due process 8
semantic docs method=glove_mean neural network 8
ai rag justice and mercy
corpus search method=fasttext_like algorithme 5

9. Concrete integration checklist for _AI code

Module Hook
_SemanticVectors load-docs, methods, SimilarDocuments, EmbedText
_CorpusFeatures method= search, topics, bible
_MySQLAIFeatures ai rag hybrid + audit
_BrainstormIntegrations architecture graph nodes; stack status line
_ThoughtDecisionTrees optional embed features (Phase B)
_EthicsGood / _LegalShield optional NN assist (Phase B)
_KernelHub future batch cosine GEMM
_DiscoveryJournal post-train record
program_web_portal graphs from method counts
Lua agents rag context injection

10. Summary for the product narrative

The multi-2vec result is not a side experiment. It is the shared semantic substrate* that lets _AugmentedIntelligence:

  1. Remember (thoughts, discoveries, memory promotion)
  2. Retrieve (Bible, books, law, dictionary)
  3. Reason with ground (RAG before LLM)
  4. Route (intent/domain trees, multi-agent)
  5. Stay safe (ethics/legal neighborhoods)
  6. Scale compute (KernelHub / NPU / TF32 later)

Operator one-liner after train:

semantic load-docs ; semantic method w2v_mean ; ai rag love justice neural network

Living brainstorm — update inventory numbers after each train_2vec_to_mysql.py run and tick Phase B/C items as they land in code.

Generated from the project markdown docs on 2026-07-24. This is a static, self-contained site.