paulpierre / markdown-crawler

What it does

This is a multithreaded web crawler that crawls a website and creates markdown files for each page. It was primarily created for large language model document parsing to simplify chunking and processing of large documents for RAG use cases. Markdown by nature is human readable and maintains document structure while keeping a small footprint.