Common Crawl
 | |
| Company type | 501(c)(3) non-profit |
|---|---|
| Founded | 2007 |
| Headquarters | San Francisco, California; Los Angeles, California, United States |
| Founder | Gil Elbaz |
Managing director | Rich Skrenta |
| Website | commoncrawl |
Content licence | Apache 2.0 (software) |
Preview warning: Page using Template:Infobox website with deprecated parameter
url. Replace with website.Preview warning: Page using Template:Infobox website with deprecated parameter
MD. Replace with managing_director.Preview warning: Page using Template:Infobox website with deprecated parameter
license. Replace with content_license.Preview warning: Page using Template:Infobox website with deprecated parameter
launch_date. Replace with launched.Preview warning: Page using Template:Infobox website with deprecated parameter
chairman. Replace with chairperson.Preview warning: Page using Template:Infobox website with deprecated parameter
foundation. Replace with founded.Preview warning: Page using Template:Infobox website with deprecated parameter
location. Replace with headquarters.Preview warning: Page using Template:Infobox website with deprecated parameter
subsid. Replace with subsidiaries.The Common Crawl Foundation (Common Crawl) is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Access to the data is free on Amazon Web Services, but users may incur storage and compute costs.
Common Crawl was founded by Gil Elbaz.
The data had mostly been primarily used by researchers and some startups until the 2020s, when AI companies started training large language models using the data. In November 2025, an investigation by The Atlantic revealed that Common Crawl misled publishers when it claimed it respected paywalls in its scraping and it was not honoring requests from publishers to have their content removed from its databases.