Snorkel AI
Paid
Programmatic data labeling and expert data development for specializing AI models.
About Common Crawl
Common Crawl is a nonprofit that has been crawling the web for many years and sharing the results for free. Its archive holds billions of pages and is one of the biggest sources of text used to train language models.
Anyone can download the data or query it in the cloud, and it also provides derived datasets such as extracted text and link graphs. Researchers use it for studies of the web, and companies filter it to build training sets.
Common Crawl is free to use. It is a foundation of open AI research, though the raw data needs a lot of cleaning.
Standout features
- Billions of web pages
- Free open access
- Regular new crawls
- Extracted text and metadata
- Link graph data
- Cloud-hosted archive
Tags
More in Datasets
View all →Label Studio
Freemium
A flexible open-source tool for labeling text, images, audio, video and time series.
Roboflow
Freemium
A platform for building computer vision models, with datasets, labeling and deployment.
Labelbox
Freemium
A data engine for labeling, curating and evaluating training data for AI.
Scale AI
Paid
Data labeling, evaluation and training data services for AI labs and enterprises.
Kaggle
Free
A home for data science competitions, public datasets and free notebooks with GPUs.
