Pipeline to load Stanford GloVe word embeddings, sample Wikipedia abstracts and

Wiktionary glosses into MySQL, and build document semantic vectors (mean of

in-vocab GloVe tokens, L2-normalized). Runtime lookup lives in SemanticVectors.*.

Quick start

[powershell]
# 1) Download GloVe + wiki/wt samples + import MySQL + export C++ cache
.\tools\semantic_vectors\run_pipeline.ps1 `
  -MysqlHost 127.0.0.1 -MysqlUser root -MysqlPassword “” `
  -WikiPages 200 -Query “artificial intelligence”

# Or step by step:
python tools/semantic_vectors/download_glove.py
python tools/semantic_vectors/download_wiki_sample.py –wiki-pages 200
python tools/semantic_vectors/import_to_mysql.py
python tools/semantic_vectors/export_cache.py
python tools/semantic_vectors/query_similar.py “neural network” –k 10

Requires: Python 3. MySQL optional (pip install mysql-connector-python).

If MySQL is unavailable, run_pipeline.ps1 falls back to:

[powershell]
python tools/semantic_vectors/build_offline_semantic.py –max-vocab 100000

which writes data/semantic_vectors_cache.tsv for C++ without a database.

MySQL schema (semantic_knowledge)

Table	Content
glove_meta / glove_vectors	Word → float32 blob
wiki_pages	Title + abstract (+ optional full path)
wiktionary_entries	Lemma, POS, gloss
semantic_vectors	Doc/phrase vectors (glove_mean)
semantic_nn_cache	Optional NN cache

Schema file: sql/semantic_vectors_schema.sql.

How semantic vectors are built

For each wiki abstract / wiktionary gloss:

1. Tokenize [A-Za-z][A-Za-z0-9_’-]* 
2. Lookup each token in GloVe 
3. Mean of hits → L2 normalize 
4. Store as BLOB in semantic_vectors with method=glove_mean

Similarity = cosine (dot product of unit vectors).

C++ commands

[text]
semantic status
semantic load data/glove/glove.6B.50d.txt [max_vocab]
semantic embed artificial intelligence
semantic similar intelligence 10
semantic load-docs                  # reads data/semantic_vectors_cache.tsv
semantic docs neural network 10

Aliases: sem …, glove ….

Data locations

[text]
data/glove/glove.6B.50d.txt          # after download
data/wiki/enwiki_sample.jsonl
data/wiktionary/enwiktionary_sample.jsonl
data/semantic_vectors_cache.tsv      # for C++ offline NN

Full Wikipedia / Wiktionary dumps

API samples are for labs. For production dumps:

• Download from https://dumps.wikimedia.org/ 
• enwiki-latest-abstract.xml.gz or pages-articles 
• enwiktionary-latest-pages-articles.xml.bz2 
• Convert to JSONL with your extractor (title, abstract/text, lemma, gloss). 
• Point import_to_mysql.py –wiki-jsonl … –wt-jsonl …. 
• Optionally raise –max-vocab / omit it for full GloVe 400k.

Files

[text]
sql/semantic_vectors_schema.sql
tools/semantic_vectors/download_glove.py
tools/semantic_vectors/download_wiki_sample.py
tools/semantic_vectors/import_to_mysql.py
tools/semantic_vectors/export_cache.py
tools/semantic_vectors/query_similar.py
tools/semantic_vectors/run_pipeline.ps1
SemanticVectors.hpp / .cpp
docs/SemanticVectors.md

---

Filed under: Uncategorized - @ July 18, 2026 6:52 am