OSCAR

OSCAR

Open Source Project on Multilingual Resources for Machine Learning

OSCAR

The OSCAR project (Open Super-large Crawled Aggregated coRpus) is an Open Source project aiming to provide web-based multilingual resources and datasets for Machine Learning (ML) and Artificial Intelligence (AI) applications. The project focuses specifically in providing large quantities of unannotated raw data that is commonly used in the pre-training of large deep learning models. The OSCAR project has developed high-performance data pipelines specifically conceived to classify and filter large amounts of web data. The project has also put special attention in improving the data quality of web-based corpora as well as providing data for low-resource languages, so that these new ML/AI technologies are accessible to as many communities as possible.

Grab the latest OSCAR release here! ­čÜÇ
Join our Discord community here! ­čĺČ

Data is distributed by language in both original and deduplicated form. There are currently 166 different languages available. If you use OSCAR please consider giving us some feedback by writing to our mail address down below. Also consider citing our papers.

If you want to contribute to OSCAR, please open a pull request here.

Since 2019, The OSCAR Project has been funded by Inria (project-team ALMAnaCH) and the PRAIRIE institute. Starting in 2023, DFKI and the German Federal Ministry for Economic Affairs and Climate Action (BMWK) through the project OpenGPT-X, have joined Inria, ALMAnaCH and the PRAIRIE institute in providing funding for the OSCAR Project. During 2022 and at the beginning of 2023, OSCAR was also shortly funded by The University of Mannheim.

If you are interested in OSCAR and would like to access the corpus, send us a mail using the mail address down below, with “OSCAR Access Request” as mail title. Please include your name, last name, affiliation, contact details, which languages do you need and a brief description of how you intend to use OSCAR.

Funding provided by

Avatar

Inria

Funding Organization

Avatar

DFKI

Funding Organization

Avatar

ALMAnaCH

Funding Lab

Avatar

Prairie Institute

Funding Institute

Avatar

OpenGPT-X

Funding Project

The OSCAR Team

Core

Avatar

Pedro Ortiz Suarez

Postdoctoral Researcher

Avatar

Julien Abadji

Research Engineer

Avatar

Rua Ismail

Research Engineer

Avatar

Laurent Romary

Senior Researcher

Avatar

Benoît Sagot

Senior Researcher

Collaborators

Avatar

Sebastian Nagel

Crawl Engineer & Data Scientist

Avatar

Ayyoob Imani

Ph.D. student at LMU Munich

Contributors

Avatar

Sotaro Takeshita

Ph.D. Student

Avatar

Patrick Teufert

Data Scientist

Partners

Avatar

Common Crawl

Partner Organization