Screen Text: Neural Detection & Recognition

Related: ComputerRL (primary consumer), Desktop Icon Vision, TensorRT.

Every desktop tick, _AI turns a raw screenshot into a list of recognized text regions that agents and goal logic read from. Text perception is neural end to end: a detection stage proposes boxes, and a recognition (CRNN) stage reads the characters inside them. This page documents the stages, the row format they share, and the flags/models that drive them.

Pipeline stages

Each frame the desktop loop assembles perception rows (see Computers.cpp):

The shared row format

Every text region is a vector of strings:

[ image_path, LABEL, x, width, y, height, score, source ]
    row[0]      row[1] row[2] row[3] row[4] row[5]  row[6]  row[7]

row[1] is the important field. Detectors emit a placeholder there (the string "text" from the MSER path, or a class label from the neural detector). The recognition stage overwrites it with the actual characters read from the box.

Recognition writes back into the row

_TextTflite::RecognizeRegionsInPlace(session, frame, text_regions) crops each detected box, runs the CRNN, and sets row[1] to the recognized string (leaving the placeholder only when the network returns empty). Because it mutates the same text_regions vector that flows to ProcessGoalTick, ComputerBot, and ComputerRL, downstream consumers see real on-screen words rather than a placeholder.

Before this wiring, the recognizer concatenated all text into one logged blob and the region rows kept the literal label "text", which silently broke keyword-overlap rewards, the risky-text scanner, and label-based clicking.

Flags and models

vision_text_detection_tflite   = true   # neural detector
vision_text_recognition_tflite = true   # neural CRNN recognizer
computer_fast_text_labeling    = true   # MSER heuristic detector (runs alongside)
vision_text_detection          = false  # legacy TF-session detector
tflite_text_detection_model    = .../text_detection/model.tflite
tflite_text_recognition_model  = .../text_recognition/model.tflite  (+ charset)

If the recognition model or charset is missing, the recognizer is skipped and rows keep their detector label — no crash, same behavior as before the model was installed.

Who consumes the recognized text

• ComputerRL builds perception.screen_text and the goal keyword-overlap feature from row[1]; the risky-text / error-dialog scanners and the new dialog classifier read it too.
• ComputerBot uses it for label-based clicking.
• The NLU / RLLanguageBridge derives goal-match, safety, and curiosity signals from the recognized screen language.

Tuning

Keep both detectors on for recall (the MSER path catches boxes the neural model misses); the recognizer labels all of them regardless of source. To reduce duplicate boxes, demote computer_fast_text_labeling to a fallback used only when the neural detector returns nothing.