← notes

What an indexer has to solve that a node does not

Nov 19, 2023

Build a Base indexer from scratch, to understand what an indexer has to solve that a node does not.

The repo is scaffolding and never became a working system, so what it records is the design. That is fine, because the hard parts here are decisions rather than code.

Reorgs are the centre. A block you indexed can be replaced, so every derived row needs the block hash it came from and a rollback path that removes everything descended from an orphaned block. Systems skipping this look correct for months and then serve data from a chain that no longer exists.

Then batching: eth_getLogs has provider caps on both range and result count, and they are not the same cap, so a window that works for a quiet contract fails for a busy one. The loop has to narrow adaptively.

Checkpointing decides what a restart costs. Without it a crash means rescanning from genesis, and the system can never recover faster than it can be rebuilt.

Finally, backfill and the real-time tail are different programs with opposite failure modes, one throughput-bound and resumable, one latency-bound and reorg-exposed. Running them as one process gets the worst of both.