The parts an indexer grows: a rate limiter, a verify queue, and a WAL-safe backup
Ingest Zora markets into a queryable local database, and keep it correct while it runs.
The file list is the record, because every one of them exists because something
went wrong first. zora_rate_limiter.py shapes calls to a provider that will
otherwise cut you off. verification_queue.py, zora_verifier.py and
zora_reverifier.py split classification into stages, with the reverifier on
its own systemd timer, because first-pass classification is sometimes wrong and
the answer is a retryable second look rather than a better first guess.
fix_misclassified_pools.py is the repair tool, and its existence is the honest
admission that you need a way to correct history, not just to stop making the
mistake.
backup_with_wal.sh is the one to tell people about. SQLite in WAL mode does
not keep all its data in the main file, so copying that file while the database
is live gives a torn backup that looks fine until you restore it. Checkpoint
first or use the backup API. It is a quiet and completely silent way to lose
data.
.monitor_state.json holds restart position, the difference between a crash
costing thirty seconds and costing a full rescan.