← notes

Reading a directory from its own JSON instead of its HTML

Aug 6, 2024

Get every project URL out of base.org/ecosystem in one click.

It is called a scraper and the interesting thing is that it is not one. The ecosystem page renders from a JSON file, so the tool fetches that file directly instead of parsing the DOM. Finding the data source behind a rendered directory is almost always worth the ten minutes, because a JSON endpoint survives redesigns that break every selector.

The fragility moved rather than disappeared. The upstream JSON has no version field and no schema, so a shape change arrives with no warning and no way to detect it except that the output is suddenly empty. Rendering nothing looks like “no results” and nobody investigates it, which is why validating the expected shape and failing loudly is the right defence.

The copy-all button needed the most care for the least interesting reason: writing several hundred URLs to the clipboard hits per-browser limits and permission prompts, and the failure is silent truncation.

awesome-base is the same information maintained by hand, and this is strictly better.