← notes

Thread-local SQLite connections, and the write lock underneath them

Dec 31, 2024

Index Base contract deployers and count activity per contract.

indexer_script.py fans RPC calls across a ThreadPoolExecutor and writes to SQLite, which has one obvious trap and one that costs more. The obvious one is that a SQLite connection is bound to its creating thread, so the script keeps a connection per worker in threading.local, created lazily. check_same_thread=False exists and is the wrong answer, because it disables the check without making the use safe.

The one that costs more is that N connections to one file do not give N-way write throughput. Writes serialize on a single lock, so past a small number of workers the extra threads queue rather than work. The useful pool size is set by the RPC rate limit and that write lock, both far below what the core count suggests.

counter.py uses eth_getTransactionCount, which despite the name returns the account nonce and counts only outbound transactions, for a deployed contract, contracts created rather than calls received. The numbers probably do not mean what the filename implies.

The QuickNode URL with its token is committed in both scripts.