Desktop Icon Vision (Computer / Desktop RL)

Train a program-icon classifier from .exe files for Computer RL and desktop automation: extract Windows icons → augment → TensorFlow SavedModel + LiteRT (TFLite).

Why

Computer RL already tracks windows, OCR text, and generic object detections. Recognizing which app icon is under the cursor or on the taskbar/desktop gives a stronger app_family / target signal (Chrome vs VS Code vs Explorer).

Pipeline

scan .exe dirs
    → extract associated icons (Shell32)
    → dedupe identical artwork
    → augment (bg, scale, color, blur)
    → train CNN (default) or MobileNetV2
    → export SavedModel + model.tflite + desktop_icons.names

Script:

scripts/train_desktop_icon_classifier.py
scripts/train_desktop_icon_classifier.ps1

Train

Default (Local Programs + Program Files, cap 300 EXEs):

cd "E:\_AugmentedIntelligence\_src\_AugmentedIntelligence_v11.0 - Typed Commands"
.\scripts\train_desktop_icon_classifier.ps1

Custom directory:

.\scripts\train_desktop_icon_classifier.ps1 `
  -ScanDir "C:\Program Files","C:\Users\Temp\AppData\Local\Programs" `
  -MaxExes 500 `
  -Epochs 20 `
  -AugPerClass 64

Python:

python scripts/train_desktop_icon_classifier.py --scan-dir "C:\Program Files" --max-exes 400 --epochs 20

Reuse extracted icons only:

python scripts/train_desktop_icon_classifier.py --skip-extract --epochs 25

Outputs

Default root:

C:\_AugmentedIntelligence\tensorflow_models\desktop_icons\
  icons\                  per-class PNGs (icon_00 + aug_*)
  desktop_icons.names     class labels (line index = class id)
  labels.json             label + source EXE paths
  manifest.json           metrics + export paths
  saved_model\            TF SavedModel (frozen export)
  saved_model_tf\         copy for TF C API path
  model.tflite            LiteRT / TFLite
  model_float32.tflite    LiteRT float32
  keras_model.keras
  history.json
  README.md

Re-export only (if Keras exists but TFLite/SavedModel missing):

python scripts/export_desktop_icon_models.py

Runtime (C++ LiteRT)

DesktopIconClassifier loads model.tflite + desktop_icons.names.

desktop icon status
desktop icon enable
desktop icon file C:\_AugmentedIntelligence\tensorflow_models\desktop_icons\icons\chrome\icon_00.png
computer rl icon status
computer icon labels 20

Settings:

desktop_icon_classifier_enabled=true
tflite_desktop_icon_model=C:/_AugmentedIntelligence/tensorflow_models/desktop_icons/model.tflite
tflite_desktop_icon_labels=C:/_AugmentedIntelligence/tensorflow_models/desktop_icons/desktop_icons.names
tensorflow_desktop_icon_model=C:/_AugmentedIntelligence/tensorflow_models/desktop_icons/saved_model_tf/

Inference contract

Settings paths (see AugmentedIntelligence.cpp):

tflite_desktop_icon_model   = C:/_AugmentedIntelligence/tensorflow_models/desktop_icons/model.tflite
tflite_desktop_icon_labels  = C:/_AugmentedIntelligence/tensorflow_models/desktop_icons/desktop_icons.names
tensorflow_desktop_icon_model = C:/_AugmentedIntelligence/tensorflow_models/desktop_icons/saved_model_tf/

Computer RL integration notes

1. Crop candidate icon regions from the desktop/taskbar capture (OpenCV).
2. Resize crop → 64×64, convert BGR→RGB, /255.0.
3. Run LiteRT interpreter (same pattern as LoL skin classifier in GamingLoL.cpp).
4. Map argmax → desktop_icons.names[i] → feature or action target label.
5. Prefer dry-run in Computer RL before actuating clicks on recognized icons.

Tips

• Installer/updater EXEs (setup, unins, vcredist, …) are skipped by name.
• Identical icons collapse to one class; extra EXE paths stay in labels.json.
• For more classes, raise --max-exes and re-run (or --skip-extract after a wider extract).
• Use --backbone cnn for 64px icons (default). Use mobilenet only with --image-size 96+.

Safety

Local personal models only. Do not ship third-party proprietary icon artwork as a public dataset; keep icons/ on the workstation that owns the software.