Crawler.sh

Mehmet Kose and his team looked at the SEO industry and said "you know what, everybody's crawling websites through cloud services that hoard your data—let's build something that runs locally on your machine." And thus Crawler.sh was born.

Crawler.sh is a local-first SEO spider and content extractor that runs entirely on your own machine—no cloud, no data harvesting, just you and your crawl. It extracts clean Markdown from any webpage with word counts, author bylines, and excerpts automatically. It runs 23 automated SEO checks on every page it hits—missing titles, duplicate meta descriptions, noindex directives, thin content, broken links, long URLs. Export your issues as CSV or TXT, or grab your data in NDJSON, JSON arrays, or W3C-compliant Sitemap XML. Configurable concurrency and depth limits let you rip through thousands of pages while staying polite to servers.

The free tier gets you 600 pages per session with basic export. The Pro version at $99/year pushes that to 10,000 pages, adds full Markdown content extraction, and unlocks that 16-category SEO analysis. It's privacy-friendly, it's fast, and it's built for everyone from freelance SEO consultants to in-house marketing teams who don't want their crawl data sitting on some third-party server.

This is what happens when developers actually use the tools they build—zero bloat, maximum utility.