GloVe + Wikipedia / Wiktionary semantic vectors

Pipeline to load Stanford GloVe word embeddings, sample Wikipedia abstracts and Wiktionary glosses into MySQL, and build document semantic vectors (mean of in-vocab GloVe tokens, L2-normalized). Runtime lookup lives in SemanticVectors.*.

Full AI brainstorm (how the train result powers RAG, ethics, law, agents, MathNN, portal):
→ SemanticEmbeddingsAIBrainstorm.md

Quick start

# 1) Download GloVe + wiki/wt samples + import MySQL + export C++ cache
.\tools\semantic_vectors\run_pipeline.ps1 `
  -MysqlHost 127.0.0.1 -MysqlUser root -MysqlPassword "" `
  -WikiPages 200 -Query "artificial intelligence"

# Or step by step:
python tools/semantic_vectors/download_glove.py
python tools/semantic_vectors/download_wiki_sample.py --wiki-pages 200
python tools/semantic_vectors/import_to_mysql.py
python tools/semantic_vectors/export_cache.py
python tools/semantic_vectors/query_similar.py "neural network" --k 10

Requires: Python 3. MySQL optional (pip install mysql-connector-python).

If MySQL is unavailable, run_pipeline.ps1 falls back to:

python tools/semantic_vectors/build_offline_semantic.py --max-vocab 100000

which writes data/semantic_vectors_cache.tsv for C++ without a database.

MySQL schema (semantic_knowledge)

Schema file: sql/semantic_vectors_schema.sql.

How semantic vectors are built

Classic GloVe mean (original pipeline)

For each wiki abstract / wiktionary gloss:

1. Tokenize [A-Za-z][A-Za-z0-9_'-]*
2. Lookup each token in GloVe
3. Mean of hits → L2 normalize
4. Store as BLOB in semantic_vectors with method=glove_mean

Similarity = cosine (dot product of unit vectors).

Multi-*2vec MySQL train (recommended)

One document (wiki page, verse, book paragraph) can have several vectors in MySQL:

Word2Vec token weights also land in glove_vectors under model name word2vec_corpus (BLOB layout matches GloVe).

# Multi-method train into MySQL (reads program DBs + offline samples)
# Credentials: MYSQL_HOST / MYSQL_USER / MYSQL_PASSWORD or --host/--user/--password
# (settings.txt mysql_* also used by the app)
python tools/semantic_vectors/train_2vec_to_mysql.py `
  --host 127.0.0.1 --user root --password "..." `
  --limit-bible 10000 --limit-caselaw 1200 --limit-uscode 2000 `
  --limit-lit-tables 60 --limit-thoughts 2000

# Offline always writes:
#   data/semantic_vectors_cache.tsv
#   data/corpus/unit_vectors.jsonl

Sources unioned for training

Writes to semantic_knowledge.semantic_vectors (multi-method BLOB rows) + semantic_vectors.embeddings (glove_mean dual-write) + offline cache for C++.

C++ commands

semantic status
semantic load data/glove/glove.6B.50d.txt [max_vocab]
semantic embed artificial intelligence
semantic similar intelligence 10
semantic load-docs                  # multi-method cache TSV
semantic auto-load                  # size-aware stack load (Phase B)
semantic methods                    # list glove_mean / w2v_mean / ...
semantic method w2v_mean            # default filter for docs search
semantic stats                      # method/source histograms JSON
semantic docs neural network 10
semantic docs method=w2v_mean love 8
semantic docs source=law due process 8
semantic docs law:equal protection 8
semantic rrf justice mercy 8        # multi-method RRF fusion
semantic train                      # print train_2vec_to_mysql recipe

ai rag method=rrf source=bible love neighbor
ai rag law:due process
graphs semantic | graphs export

Aliases: sem …, glove ….

Corpus/Bible surface (same vectors): corpus search method=w2v_mean …, bible load-db both.

Data locations

data/glove/glove.6B.50d.txt          # after download
data/wiki/enwiki_sample.jsonl
data/wiktionary/enwiktionary_sample.jsonl
data/semantic_vectors_cache.tsv      # for C++ offline NN

Full Wikipedia / Wiktionary dumps

API samples are for labs. For production dumps:

1. Download from https://dumps.wikimedia.org/
   - enwiki-latest-abstract.xml.gz or pages-articles
   - enwiktionary-latest-pages-articles.xml.bz2
2. Convert to JSONL with your extractor (title, abstract/text, lemma, gloss).
3. Point import_to_mysql.py --wiki-jsonl … --wt-jsonl ….
4. Optionally raise --max-vocab / omit it for full GloVe 400k.

Files

sql/semantic_vectors_schema.sql
tools/semantic_vectors/download_glove.py
tools/semantic_vectors/download_wiki_sample.py
tools/semantic_vectors/import_to_mysql.py
tools/semantic_vectors/export_cache.py
tools/semantic_vectors/query_similar.py
tools/semantic_vectors/run_pipeline.ps1
SemanticVectors.hpp / .cpp
tools/semantic_vectors/train_2vec_to_mysql.py   # multi-method train → MySQL + TSV
docs/SemanticVectors.md
docs/BookBibleMediaVectors.md   # books, Bible, media × multi-*2vec × MySQL