adbar / trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Why this job: topics: web-scraping, crawler
What it does
Trafilatura is a comprehensive Python package and command-line tool designed to gather text from the Web and turn raw HTML into structured, meaningful data. It includes all necessary discovery and text processing components to perform web crawling, downloads, scraping, and extraction of main texts, metadata and comments. It aims at staying handy and modular: no database is required, the output can be converted to commonly used formats.
Get started
No single install command. Read the steps in the README (opens in new tab)
- Maintenance
- Active
- Last commit 3 days ago
- Stars
- 7,103
- +43 this week
- Licence
- Apache-2.0
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- v2.3.1
- Released 34 days ago
- Built with
- Python
- 100% of the code
Is it looked after?
- Contributors
- 45
- Last 90 days
- Issues
- 90% closed
- 51 still open
- Pull requests
- 430 merged
- 26 waiting
- Releases
- 43
- 4 in the last year
- Contributing guide (opens in new tab)
- No security policy
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| firecrawl / firecrawl | 199,902 | today | AGPL-3.0 | No downloads |
| alirezamika / autoscraper | 8,080 | 3 months ago | MIT | No downloads |
More: languages, topics and project record
Languages
- Python 100%
Topics
- web-scraping
- text-extraction
- nlp
- html2text
- text-mining
- crawler
- text-cleaning
- text-preprocessing
- article-extractor
- readability
- scraping
- news-crawler
Project record
- JobAutomation and scraping
- StatusListed
- Commits1,742
- Forks497
- Watching33
- Stars in 28 days+148
- CreatedApr 8, 2019
- First seen by usNov 7, 2026
- CheckedNov 9, 2026