← notes

The parts an indexer grows: a rate limiter, a verify queue, and a WAL-safe backup

Aug 16, 2025

Ingest Zora markets into a queryable local database, and keep it correct while it runs.

The file list is the record, because every one of them exists because something went wrong first. zora_rate_limiter.py shapes calls to a provider that will otherwise cut you off. verification_queue.py, zora_verifier.py and zora_reverifier.py split classification into stages, with the reverifier on its own systemd timer, because first-pass classification is sometimes wrong and the answer is a retryable second look rather than a better first guess. fix_misclassified_pools.py is the repair tool, and its existence is the honest admission that you need a way to correct history, not just to stop making the mistake.

backup_with_wal.sh is the one to tell people about. SQLite in WAL mode does not keep all its data in the main file, so copying that file while the database is live gives a torn backup that looks fine until you restore it. Checkpoint first or use the backup API. It is a quiet and completely silent way to lose data.

.monitor_state.json holds restart position, the difference between a crash costing thirty seconds and costing a full rescan.