Sriram-PR / doc-scraper

Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)

Why this job: says: web crawler, scraper; topics: web-crawler

What it does

Offline search for agents. BM25 full-text search (search_docs) and page reads (read_page) over the stored corpus, with no network access
Framework detection. 30+ docs generators recognized, each selector validated on the page, Readability fallback, JavaScript-only shells reported instead of crawled empty
Clean output. GitHub-Flavored Markdown per page with frontmatter, plus pages.jsonl, llms.txt, and llms-full.txt
Polite and resumable. robots.txt and sitemaps, per-host rate limits, retries with backoff, resumable crawls, and staged output that never clobbers the last good copy
Stays fresh. incremental re-crawls, watch mode, and crawl history (get_freshness, diff_crawl)

go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest

Repos similar to this one
Repo	Stars	Last commit	Licence	Downloads for
firecrawl / firecrawl	199,902	today	AGPL-3.0	No downloads
ScrapeGraphAI / Scrapegraph-ai	32,645	34 days ago	MIT	No downloads
fastcrw / crw	1,237	today	AGPL-3.0	Mac, Linux

More: download files, languages, topics and project record