Name: CCAligned
Description: CCAligned consists of parallel or comparable web-document pairs in 137 languages aligned with English. These web-document pairs were constructed by performing language identification on raw web-documents, and ensuring corresponding language codes were corresponding in the URLs of web documents. 
Provider: Multiple Institutions
Paper Title: CCAligned: A Massive Collection of Cross-lingual Web-Document Pairs
Tasks: machine translation
Venue Name: Empirical Methods in Natural Language Processing
Authors: Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzmá, Philipp Koehn
Abstract: Cross-lingual document alignment aims to
  identify pairs of documents in two distinct languages that are of comparable content or translations of each other. In this paper, we exploit the signals embedded in URLs to label
  web documents at scale with an average precision of 94.5% across different language pairs.
  We mine sixty-eight snapshots of the Common Crawl corpus and identify web document
  pairs that are translations of each other. We
  release a new web dataset consisting of over
  392 million URL pairs from Common Crawl
  covering documents in 8144 language pairs
  of which 137 pairs include English. In addition to curating this massive dataset, we introduce baseline methods that leverage crosslingual representations to identify aligned documents based on their textual content. Finally,
  we demonstrate the value of this parallel documents dataset through a downstream task of
  mining parallel sentences and measuring the
  quality of machine translations from models
  trained on this mined data. Our objective in
  releasing this dataset is to foster new research
  in cross-lingual NLP across a variety of low,
  medium, and high-resource languages