Docs / Books, Bible, Media × MySQL × *2Vec training
Books, Bible, Media × MySQL × *2Vec training
Brainstorm / design note for _AugmentedIntelligence
Extends SemanticVectors.md, MySQLDatabases.md, and the earlier text-embedding brainstorm (Word2Vec / Doc2Vec / sentence / Top2Vec).
1. Vision in one sentence
Ingest books, scripture, and other media transcripts into MySQL as structured corpora, train or project multiple *2vec families, store weights + unit vectors as BLOBs, then power analysis, search, tutoring, ethics, and cross-media recall.
You already have:
| Piece | Today |
|---|---|
semantic_knowledge / semantic_vectors |
GloVe word vectors + glove_mean doc vectors |
dictionary.wiki_vectors |
Word-level vectors in MySQL |
data/glove/* |
Offline GloVe files |
CrossModalEpisodicEmbedder |
Multimodal episode embeddings |
| Vision / sound / thought DBs | Media streams as tables |
| MathNN | Local nets that can consume embeddings as features |
Missing (this brainstorm): book/Bible/media corpus schema, multi-method *2vec training, and analysis products over those weights.
2. What “all of the 2 vec” means
Treat “2vec” as a family of embedding trainers*, not one algorithm:
| Method | Unit of training | Output stored in MySQL | Best for |
|---|---|---|---|
| Word2Vec (SG/CBOW) | word co-occurrence window | token → R^d |
lexicon, analogy, verse words |
| GloVe (already) | global co-occurrence | same | stable word space (you have pipeline) |
| FastText | subword n-grams | token + OOV composition | morphology, archaic Bible English |
| Doc2Vec (PV-DM / PV-DBOW) | document/paragraph id | doc → R^d |
chapters, books, sermons |
| Sentence2Vec / Sent2Vec | sentence | sent → R^d |
verse, quote, dialogue line |
Paragraph mean (glove_mean) |
bag-of-tokens mean | cheap doc vector | baseline you already ship |
| Top2Vec | joint doc+word topic space | topics + doc membership | themes across Bible/books |
| LDA2Vec-ish / topic+embed hybrids | topic mixture + vectors | soft topics | thematic analysis |
| Char2Vec / Byte | characters | rare for literature | OCR noise, Greek/Hebrew translit |
| Graph2Vec (optional) | entity graph of people/places | node vectors | biblical genealogy / cast graphs |
| Scene2Vec / Clip-lite | image/video frame + text | multimodal | illustrated Bibles, film, games |
| Audio2Vec / Whisper-mean | utterance embedding | speech vectors | sermons, audiobooks, podcasts |
| CrossModal (existing) | episode fusion | episode vector | “when I heard X while reading Y” |
“Train weights” = either:
- Train from scratch on your MySQL corpus (Word2Vec/Doc2Vec/Top2Vec), or
- Project fixed GloVe/FastText into docs (mean / TF-IDF weighted / SIF), or
- Fine-tune / distill a small MathNN encoder that maps bag-of-ids → vector supervised by cosine to a teacher.
All three should land in the same MySQL shape: (source_type, source_id, method, model_version, dim, vector BLOB, meta JSON).
3. Corpora to support
3.1 Books (general)
| Asset | Granularity | Analysis products |
|---|---|---|
| Novel / nonfiction | book → chapter → paragraph → sentence | style, character graph, theme arcs, similar books |
| Textbooks | section / theorem / exercise | tutoring retrieval (pair with MathNN fields) |
| Papers / theses | abstract / section | discovery journal RAG |
| Fanfic / scripts | scene / dialogue turn | character voice similarity |
3.2 The Bible (and scripture generally)
Treat scripture as a first-class multi-translation corpus, not a single blob.
Installed program databases (authoritative):
| Database / table | Translation |
|---|---|
bible.bible_verses_kjv |
King James Version (KJV) |
bible.bible_verses_asv |
American Standard Version (ASV) |
bible_kjv (legacy) |
Per-book chapter tables used by older BibleVerseSearch |
Accessors: _DatabaseFunctions::BibleGetVerseText, BibleLoadAllVerses, BibleVerseSearchKJV / ASV.
Corpus load: bible load-db kjv|asv|both (embeds into in-memory multi-*2vec units; refs like kjv:John.3.16).
| Layer | Examples | Why |
|---|---|---|
| Canon structure | Testament → book → chapter → verse | stable IDs (John.3.16) |
| Translation | KJV + ASV in MySQL; optional other packs | parallel verse alignment |
| Pericope / pericope sets | lectionary readings | liturgical themes |
| Cross-refs | Treasury of Scripture Knowledge style | graph edges in MySQL |
| Names / places | Strong’s / entity tags | Graph2Vec + NER |
| Themes | salvation, covenant, wisdom, … | Top2Vec / supervised labels |
Sensitivity: faith content needs respectful UX (citation, not mockery; clear “educational/research” mode). KJV and ASV are public-domain defaults in this project.
3.3 Other media
| Media | Source tables / paths | Text channel | Vector channel |
|---|---|---|---|
| Film / TV | media library + subtitles |
dialogue lines | Scene2Vec, speaker turns |
| Podcasts / sermons | sound + Whisper |
transcript segments | Audio2Vec + Doc2Vec |
| Music lyrics | media library | verse/chorus | Word2Vec style + mood labels |
| Games | Minecraft/chat logs, strategy text | quest text | domain Word2Vec |
| Web / forum | MyBB lurk scripts | posts | Doc2Vec threads |
| Vision OCR | vision / reading tables |
OCR text | same *2vec stack |
| Legal | us_code, case_law |
sections | already adjacent |
| Recipes | ai_inventory / cooking |
steps | recipe2vec-style |
4. MySQL design (extend semantic_knowledge)
4.1 New logical database or schemas
Option A: expand semantic_knowledge.
Option B: dedicated media_corpus DB + FK into semantic_vectors (cleaner multi-machine mirrors).
Recommended tables:
corpora
id, name, kind ENUM('book','bible','media','wiki','custom'),
license, language, path_root, meta_json
works -- a book / film / album / translation
id, corpus_id, title, authors, year, kind, external_id
documents -- chapter, episode, track, sermon
id, work_id, ord, title, text MEDIUMTEXT, char_count
units -- verse, sentence, paragraph, subtitle cue
id, document_id, unit_type, ref_key, -- e.g. 'John.3.16' or 't=00:12.3'
text, start_ms, end_ms, speaker
entities
id, name, entity_type, -- person, place, concept
aliases_json
unit_entities -- many-to-many mentions
unit_id, entity_id, role
cross_refs -- especially Bible
from_unit_id, to_unit_id, rel_type, weight
embedding_models
id, family ENUM('glove','word2vec','fasttext','doc2vec','sentence',
'top2vec','mean','mathnn','crossmodal','other'),
name, dimensions, train_corpus_id, hyperparams_json,
weights_path, -- file for large matrices
created_at
token_vectors -- word / subword
model_id, token, vector BLOB, count
unit_vectors -- sentence / verse / paragraph
model_id, unit_id, method, vector BLOB, norm
doc_vectors -- chapter / work
model_id, document_id OR work_id, method, vector BLOB
topic_models -- Top2Vec / LDA-style
model_id, topic_id, label, vector BLOB, top_tokens_json
topic_membership
model_id, topic_id, document_id, score
analysis_runs
id, kind, corpus_id, model_id, status, metrics_json, started_at
analysis_artifacts
run_id, artifact_type, -- 'concordance','theme_map','character_arc','summary'
ref_key, payload_json / path
Reuse existing semantic_vectors rows by widening:
source_type ENUM(..., 'book_unit','bible_verse','media_segment','work','topic')
method VARCHAR(64) -- glove_mean | w2v_mean | doc2vec | sbert_distill | top2vec_doc | ...
4.2 Weight storage strategy
| Size | Store |
|---|---|
| Word vocab ≤ 400k × 300d | MySQL BLOB or memory-map file + embedding_models.weights_path |
| Doc2Vec millions of docs | vectors in MySQL; training matrices on disk |
| Top2Vec topics | small → MySQL |
| MathNN encoder | mathnn_*.nn path + meta in embedding_models |
Rule: query vectors in MySQL; training checkpoints on disk; MySQL holds pointers + hashes.
5. Training pipelines (“train weights with all of the 2 vec”)
5.1 Offline Python lab (extend tools/semantic_vectors/)
tools/corpus/
ingest_books.py # epub/txt/pdf→text → documents/units
ingest_bible.py # USFX/OSIS/JSON verse packs → units + ref_key
ingest_subtitles.py # srt/vtt → media units
ingest_audio_transcripts.py
train_word2vec.py # gensim Word2Vec / FastText
train_doc2vec.py
train_top2vec.py
project_glove_mean.py # existing method, multi-source
export_mysql.py
export_cpp_cache.py # TSV/JSONL for SemanticVectors C++
5.2 Training recipe per method
| Method | Input from MySQL | Train | Write back |
|---|---|---|---|
| Word2Vec | tokenized units.text stream |
gensim / custom | token_vectors |
| FastText | same + subwords | gensim | token_vectors + OOV rules |
| GloVe | cooc matrix job | external or preloaded | glove_vectors (exists) |
| Doc2Vec | docs as tagged documents | gensim Doc2Vec | doc_vectors |
| Sentence mean | GloVe/W2V + SIF optional | projection only | unit_vectors method=mean |
| Top2Vec | docs | top2vec lib | topic_models + membership |
| MathNN encoder | (unit_id, bag-of-hash) → teacher vec | _NN MSE/cosine |
embedding_models.weights_path |
| CrossModal | transcript + image path | existing embedder | episode tables + unit_vectors |
5.3 Multi-method matrix (the point of “all”)
For each verse / paragraph / chapter, store parallel vectors:
method |
Use case |
|---|---|
glove_mean |
Fast baseline, works offline now |
w2v_mean |
Domain-adapted lexicon (Bible English) |
fasttext_mean |
Rare/archaic tokens |
doc2vec |
“Whole chapter of Romans similar to…” |
sent2vec |
Verse-level paraphrase / parallel translations |
top2vec_doc |
Theme membership |
mathnn_v1 |
Tiny local encoder for agents |
crossmodal_v1 |
When audio/video exists |
Similarity search picks method explicitly:
corpus similar method=doc2vec "hope in suffering" --k 10
bible similar method=sent2vec John.3.16 --k 5
5.4 Alignment across translations (Bible-specific gold feature)
For each ref_key, store vectors per translation; learn or enforce:
[ \cos(v_{\mathrm{KJV}}(r), v_{\mathrm{WEB}}(r)) \uparrow ]
Enables translation-robust theme search and “show me this verse in another rendering.”
6. Analyses to run once vectors exist
6.1 Book analyses
| Analysis | Method stack | Output artifact |
|---|---|---|
| Thematic map | Top2Vec + doc vectors | topic labels per chapter |
| Style fingerprint | Word2Vec + function-word stats | author/work centroid |
| Character / entity arc | NER → entity timeline + Graph2Vec | analysis_artifacts |
| Sentiment / valence trajectory | lexicon + unit vectors | chapter curves |
| Similar books | Doc2Vec / mean chapter pool | nearest works |
| Quote finder | sentence vectors + FULLTEXT hybrid | cite unit ref |
| Reading level / complexity | classic stats + embed density | tutoring routing |
| Contradiction / claim pairs | NLI core + sentence vecs | knowledge RL claims |
| Curriculum extractor | section vectors vs MathNN domains | nn math links |
6.2 Bible analyses (respectful research mode)
| Analysis | Description |
|---|---|
| Concordance 2.0 | not only string match — semantic neighbors of “grace”, “law”, “kingdom” |
| Cross-ref expansion | seed edges + vector-suggested related verses |
| Theme topography | Top2Vec over chapters; “wisdom literature” cluster |
| Narrative chronology helpers | entity co-occurrence graphs (kings, prophets) |
| Parallel passage detection | high cosine across Synoptic units |
| Liturgical packs | vector centroids of lectionary sets |
| Intertextuality with other books | Bible units vs literature corpus (influence studies) |
| Multilingual bridge | aligned verse keys across languages |
Guardrails: no automated “doctrine adjudication”; present citations + similarity scores; human theology remains authoritative.
6.3 Other media analyses
| Media | Analysis |
|---|---|
| Sermon audio | transcript Doc2Vec vs scripture Top2Vec (“which themes?”) |
| Film | subtitle Scene2Vec; character dialogue style |
| Podcast | segment vectors; guest topic drift |
| Game lore | quest text Word2Vec; similar quests |
| Music | lyric mood clusters; chorus vs verse |
| Desktop OCR books | same as books after ai_reading ingest |
6.4 Agent / product surfaces
| Command (proposed) | Behavior |
|---|---|
corpus ingest bible <path> |
Load OSIS/JSON into MySQL |
corpus ingest book <path> |
EPUB/TXT pipeline |
corpus train word2vec\|doc2vec\|top2vec |
Kick Python/C++ job; write weights |
corpus embed all methods |
Project every unit under all registered methods |
bible search <text> |
Hybrid FULLTEXT + vector |
bible similar John.3.16 |
Nearest verses |
bible theme "forgiveness" |
Top2Vec + labeled topics |
book analyze <title> |
Run analysis suite → artifacts |
media similar <clip_id> |
Cross-modal + transcript vectors |
semantic docs … |
Existing entry point, expanded sources |
Lua: ai_corpus_similar, ai_bible_verse_embed, …
7. How this hooks existing AI features
| Existing subsystem | Hook |
|---|---|
_SemanticVectors |
Load multi-method caches; SimilarDocuments filters source_type |
| MathNN | Encode bag-of-tokens; reward models on “relevant verse” ranking |
| KnowledgeAcquisitionRL | Claims mined from books/Bible with citations |
| Command intent | Domain phrases from scripture/literature corpora |
| Character chat | Persona grounded in book character vectors |
| Ethics classifier | Training phrases from policy + optional sacred-text respect rules |
| Memory tiers | Promote high-importance analysis artifacts to LTM |
| CrossModal | Sermon audio + on-screen scripture OCR as one episode |
| Legal DBs | Same *2vec stack, different corpus kind |
| Discovery journal | “found parallel between X and Y” as discoveries |
8. Implementation packs (priority)
Pack A — Schema + ingest (foundation)
- MySQL tables for
works/documents/units/embedding_models - Bible public-domain loader + book TXT/EPUB loader
- Expand
semantic_vectors.source_type+method - Export cache for C++
Pack B — Multi-*2vec train
- Word2Vec + FastText on combined corpus
- Doc2Vec on chapters/books
- Keep GloVe mean projection
- Top2Vec theme model
- Store all in MySQL; version models
Pack C — Analyses
- Verse/chapter similar search
- Theme map UI /
bible theme - Book fingerprint + similar books
- Hybrid FULLTEXT + cosine
Pack D — Media + cross-modal
- Subtitle + Whisper segment ingest
- Scene/episode vectors
- Scripture–sermon alignment search
Pack E — MathNN / agent
- Tiny dual-encoder ranker (query vs unit)
- Tutoring: retrieve explanations from textbooks
- Gaming lore / quest retrieval (reuse stack)
9. Licensing & ethics checklist
| Topic | Practice |
|---|---|
| Copyrighted books | User-provided paths only; store license field; no redistribution |
| Bible translations | Prefer public domain defaults; licensed texts opt-in |
| Sacred texts | Respectful mode; citation required in answers |
| Training data provenance | corpora.license + analysis_runs audit |
| PII in personal media | Same redaction path as receipts/memory |
10. Minimal viable demo (story)
1. ingest WEB Bible → units with ref_key
2. project glove_mean for every verse → semantic_vectors
3. train word2vec on Bible+wiki tokens → token_vectors
4. train doc2vec on chapters → doc_vectors
5. train top2vec → topic_models
6. query: bible similar method=glove_mean "love your neighbor"
7. query: bible theme "covenant"
8. ingest a public-domain novel → same methods
9. book similar between novel chapters and biblical wisdom literature
That single story exercises MySQL storage, multi-*2vec weights, and cross-corpus analysis.
11. What not to do
- One giant unversioned BLOB of “the Bible embedding” with no verse IDs
- Train only Word2Vec and call it “all 2vec”
- Store 7B LLM weights in MySQL
- Silent doctrinal automation
- Mix copyrighted publisher text into shared fleet DBs without license fields
12. Relation to prior brainstorms
| Prior topic | Relation |
|---|---|
| MySQL AI features | This is a domain pack: corpus RAG + training warehouse |
| Word2Vec / Doc2Vec / Top2Vec brainstorm | Concrete corpora + schema + train jobs |
| MathNN layers paper | Embeddings as inputs to small nets / reward models |
| Cross-modal memory | Media sermons/films as episodes linking to verse units |
13. Implementation status (v11)
| Pack | Status | Location |
|---|---|---|
| A Schema + ingest | Done | sql/corpus_schema.sql, tools/corpus/ingest_*.py |
| B Multi-*2vec train | Done | tools/corpus/build_all_2vec.py (glove_mean, w2v_mean, doc2vec_like, top2vec_doc) |
| C Analyses / search | Done | CorpusFeatures — corpus / bible / book commands |
| D Media cross-modal | Partial | subtitle ingest can reuse unit schema; Whisper path TBD |
| E MathNN ranker | Partial | embeddings usable as features via caches |
Quick start
powershell -File tools/corpus/run_corpus_pipeline.ps1
# then in the app:
corpus load
bible search love hope 5
bible similar John.3.16
corpus theme list
ai rag forgiveness
Sample data: data/corpus/bible_sample.jsonl, data/corpus/book_sample.txt.
C++: CorpusFeatures.cpp · Python: tools/corpus/* · MySQL AI packs: MySQLAIFeatures + sql/ai_features_schema.sql.
Related: docs/SemanticVectors.md, sql/semantic_vectors_schema.sql, sql/corpus_schema.sql, tools/semantic_vectors/*, SemanticVectors.cpp, CrossModalEpisodicEmbedder.*
Generated from the project markdown docs on 2026-07-24. This is a static, self-contained site.