{"id":2069,"date":"2026-07-18T06:52:23","date_gmt":"2026-07-18T13:52:23","guid":{"rendered":"http:\/\/macdaddy4sure.ai\/?p=2069"},"modified":"2026-07-18T06:52:23","modified_gmt":"2026-07-18T13:52:23","slug":"glove-wikipedia-wiktionary-semantic-vectors","status":"publish","type":"post","link":"http:\/\/macdaddy4sure.ai\/index.php\/2026\/07\/18\/glove-wikipedia-wiktionary-semantic-vectors\/","title":{"rendered":"GloVe + Wikipedia \/ Wiktionary semantic vectors"},"content":{"rendered":"\n

Pipeline to load Stanford GloVe<\/strong> word embeddings, sample Wikipedia<\/strong> abstracts and<\/p>\n\n\n\n

Wiktionary<\/strong> glosses into MySQL, and build document semantic vectors<\/strong> (mean of<\/p>\n\n\n\n

in-vocab GloVe tokens, L2-normalized). Runtime lookup lives in SemanticVectors.*.<\/p>\n\n\n\n

Quick start<\/h2>\n\n\n\n

[powershell]
<\/em># 1) Download GloVe + wiki\/wt samples + import MySQL + export C++ cache
.\\tools\\semantic_vectors\\run_pipeline.ps1 `
  -MysqlHost 127.0.0.1 -MysqlUser root -MysqlPassword “” `
  -WikiPages 200 -Query “artificial intelligence”

# Or step by step:
python tools\/semantic_vectors\/download_glove.py
python tools\/semantic_vectors\/download_wiki_sample.py –wiki-pages 200
python tools\/semantic_vectors\/import_to_mysql.py
python tools\/semantic_vectors\/export_cache.py
python tools\/semantic_vectors\/query_similar.py “neural network” –k 10<\/p>\n\n\n\n

Requires: Python 3. MySQL optional (pip install mysql-connector-python).<\/p>\n\n\n\n

If MySQL is unavailable, run_pipeline.ps1 falls back to:<\/p>\n\n\n\n

[powershell]
<\/em>python tools\/semantic_vectors\/build_offline_semantic.py –max-vocab 100000<\/p>\n\n\n\n

which writes data\/semantic_vectors_cache.tsv for C++ without a database.<\/p>\n\n\n\n

MySQL schema (semantic_knowledge)<\/h2>\n\n\n\n
Table<\/strong><\/td>Content<\/strong><\/td><\/tr>
glove_meta \/ glove_vectors<\/td>Word \u2192 float32 blob<\/td><\/tr>
wiki_pages<\/td>Title + abstract (+ optional full path)<\/td><\/tr>
wiktionary_entries<\/td>Lemma, POS, gloss<\/td><\/tr>
semantic_vectors<\/td>Doc\/phrase vectors (glove_mean)<\/td><\/tr>
semantic_nn_cache<\/td>Optional NN cache<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Schema file: sql\/semantic_vectors_schema.sql.<\/p>\n\n\n\n

How semantic vectors are built<\/h2>\n\n\n\n

For each wiki abstract \/ wiktionary gloss:<\/p>\n\n\n\n

    \n
  1. Tokenize [A-Za-z][A-Za-z0-9_’-]*\u00a0<\/li>\n\n\n\n
  2. Lookup each token in GloVe\u00a0<\/li>\n\n\n\n
  3. Mean<\/strong> of hits \u2192 L2 normalize<\/strong>\u00a0<\/li>\n\n\n\n
  4. Store as BLOB in semantic_vectors with method=glove_mean<\/li>\n<\/ol>\n\n\n\n

    Similarity = cosine<\/strong> (dot product of unit vectors).<\/p>\n\n\n\n

    C++ commands<\/h2>\n\n\n\n

    [text]
    <\/em>semantic status
    semantic load data\/glove\/glove.6B.50d.txt [max_vocab]
    semantic embed artificial intelligence
    semantic similar intelligence 10
    semantic load-docs                  # reads data\/semantic_vectors_cache.tsv
    semantic docs neural network 10<\/p>\n\n\n\n

    Aliases: sem \u2026, glove \u2026.<\/p>\n\n\n\n

    Data locations<\/h2>\n\n\n\n

    [text]
    <\/em>data\/glove\/glove.6B.50d.txt          # after download
    data\/wiki\/enwiki_sample.jsonl
    data\/wiktionary\/enwiktionary_sample.jsonl
    data\/semantic_vectors_cache.tsv      # for C++ offline NN<\/p>\n\n\n\n

    Full Wikipedia \/ Wiktionary dumps<\/h2>\n\n\n\n

    API samples are for labs. For production dumps:<\/p>\n\n\n\n

      \n
    • Download from https:\/\/dumps.wikimedia.org\/\u00a0<\/li>\n\n\n\n
    • enwiki-latest-abstract.xml.gz or pages-articles\u00a0<\/li>\n\n\n\n
    • enwiktionary-latest-pages-articles.xml.bz2\u00a0<\/li>\n\n\n\n
    • Convert to JSONL with your extractor (title, abstract\/text, lemma, gloss).\u00a0<\/li>\n\n\n\n
    • Point import_to_mysql.py –wiki-jsonl \u2026 –wt-jsonl \u2026.\u00a0<\/li>\n\n\n\n
    • Optionally raise –max-vocab \/ omit it for full GloVe 400k.<\/li>\n<\/ul>\n\n\n\n

      Files<\/h2>\n\n\n\n

      [text]
      <\/em>sql\/semantic_vectors_schema.sql
      tools\/semantic_vectors\/download_glove.py
      tools\/semantic_vectors\/download_wiki_sample.py
      tools\/semantic_vectors\/import_to_mysql.py
      tools\/semantic_vectors\/export_cache.py
      tools\/semantic_vectors\/query_similar.py
      tools\/semantic_vectors\/run_pipeline.ps1
      SemanticVectors.hpp \/ .cpp
      docs\/SemanticVectors.md<\/p>\n","protected":false},"excerpt":{"rendered":"

      Pipeline to load Stanford GloVe word embeddings, sample Wikipedia abstracts and Wiktionary glosses into MySQL, and build document semantic vectors (mean of in-vocab GloVe tokens, L2-normalized). Runtime lookup lives in SemanticVectors.*. Quick start [powershell]# 1) Download GloVe + wiki\/wt samples + import MySQL + export C++ cache.\\tools\\semantic_vectors\\run_pipeline.ps1 `  -MysqlHost 127.0.0.1 -MysqlUser root -MysqlPassword “” `  […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2069","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/posts\/2069","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/comments?post=2069"}],"version-history":[{"count":1,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/posts\/2069\/revisions"}],"predecessor-version":[{"id":2070,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/posts\/2069\/revisions\/2070"}],"wp:attachment":[{"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/media?parent=2069"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/categories?post=2069"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/macdaddy4sure.ai\/index.php\/wp-json\/wp\/v2\/tags?post=2069"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}