Searching Text Inside Screenshots on macOS with Vision and FTS5

To make text inside screenshots searchable on a Mac, run Apple's Vision text recognition on each image in the background, store the recognised text next to the clip, and index it with SQLite's FTS5 full-text engine. With FTS5's trigram tokenizer, any three or more characters match as a substring, and bm25() with column weights puts the most relevant result first. That's how search works in Clipivo, the native macOS clipboard manager I build. Its source is on GitHub.

Here is the pipeline, the reasoning behind each choice, and the edge cases that took longest to get right.

The pipeline

Clipivo watches the clipboard, stores every copy, and does the slow work later:

clipboard change
  -> read every useful representation (text, image, file, PDF...)
  -> classify, apply privacy rules, dedupe by content hash
  -> store: SQLite row + FTS row + payload on disk
  -> background: thumbnail (ImageIO) -> OCR (Vision) -> FTS reindex

The important part is the last line. Saving a screenshot is instant. The thumbnail and the text recognition run afterwards on background executors, and when OCR finishes the clip's search entry is rebuilt with the new text. You never wait for OCR to copy something, and a screenshot becomes searchable a moment later.

On-device OCR with Vision

The recognition code is short. In ImageServices.swift it creates a VNRecognizeTextRequest with these settings:

let request = VNRecognizeTextRequest()
request.recognitionLevel = .accurate
request.usesLanguageCorrection = true
request.automaticallyDetectsLanguage = true

Why these:

• Accurate, not fast. Screenshots are full of small interface text, and the job runs in the background anyway. Spending a little more time per image is a good trade for better words in the index.
• Language correction on. Most screenshot text is sentences, labels and menus, where a language model cleans up misread words. Text you copy directly never goes through OCR, so codes and IDs you copy as text are indexed exactly as they are.
• Automatic language detection. People screenshot in more than one language, and I didn't want a setting for it.

Everything stays on the Mac: Clipivo has no account, no server and no analytics, and the recognised text is stored in the same local database as the clip.

Indexing with FTS5 and the trigram tokenizer

The search index is one FTS5 virtual table, clip_fts, with seven columns: title, body (the first 256 KB of text), OCR text, file names, URL, source app and tags. It uses the trigram tokenizer.

A normal word tokenizer only matches whole words or prefixes. That's wrong for a clipboard: you remember fragments. SQLite's FTS5 documentation explains that the trigram tokenizer treats every three-character sequence as a token, so a query can match any substring of a row, not just complete words. In practice, typing pabas finds a clip containing supabase, and a half-remembered part of an error message finds the screenshot it came from. It's case-insensitive by default.

There's one hard limit worth knowing early. The same documentation says substrings shorter than 3 characters don't match any rows in a trigram full-text query. So Clipivo's query planner splits terms:

Limiting the short-term fallback to short columns keeps it fast, because it never scans whole documents or OCR text.

Filters stay plain SQL. Typing app:, type:, tag:, date: or is:pinned in the search box adds an ordinary WHERE condition next to the full-text match.

Ranking: bm25 with column weights

Matching is only half of search. The other half is putting the right clip first. FTS5's bm25() function returns smaller numbers for better matches and accepts one weight per column. Clipivo's weights, in column order:

title 8.0, body 1.0, ocr 0.8, files 3.0, url 2.0, app 0.6, tags 4.0

The reasoning: a word you typed as a title or tag is a strong signal, so those weigh most. OCR text weighs slightly less than body text, because recognition can produce noise and long screenshots contain a lot of incidental words. On top of bm25 there's a small recency penalty, 0.35 for every 30 days since the clip was last used, so an old clip with a weak match doesn't outrank something you copied this morning.

Keeping it fast as history grows

Clipivo keeps history with no item limit and no automatic expiry, so the design has to hold up after months of copying:

• Big text lives in its own table. Full text and OCR text sit in clip_text, not in the main clips table, so list queries never read large rows.
• Large payloads are files. Anything over 32 KB is stored as a content-addressed file named by its SHA-256 hash, which also deduplicates repeated copies.
• All database work runs on one actor, never on the main thread, so typing in the search field stays smooth.
• Search is debounced by 45 ms, superseded queries are cancelled, and results come in pages of 80, so the interface never holds the whole history.

What I'd do

If you're adding search to a Mac app, start with SQLite FTS5 before reaching for anything heavier. It's part of SQLite, runs inside your app with no server, works offline, and with the trigram tokenizer it handles the "I remember part of it" searches people actually make. Decide early how you'll handle terms under three characters, and give each column a weight that matches how much you trust it.

This is keyword search: it finds the words you type. When you need search by meaning instead, for example "invoice from last spring" finding a clip that never uses the word invoice, that's where embeddings come in. I wrote about that approach in semantic search with pgvector in Supabase.

If you need a native Mac tool or an internal app built the same careful way, my custom software page explains how I work.