Name: X-CSR
Description: automatically translate the original CSQA and CODAH datasets, which only have English versions, to 15 other languages, forming development and test sets for studying X-CSR
Provider: University of Southern California
Paper Title: Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning
Tasks: commonsense reasoning
Venue Name: nan
Authors: Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, Xiang Ren
  
Abstract: Commonsense reasoning research has so far
  been mainly limited to English. We aim
  to evaluate and improve popular multilingual
  language models (ML-LMs) to help advance
  commonsense reasoning (CSR) beyond English. We collect the Mickey corpus, consisting of 561k sentences in 11 different languages, which can be used for analyzing and
  improving ML-LMs. We propose Mickey
  Probe, a language-agnostic probing task for
  fairly evaluating the common sense of popular ML-LMs across different languages. Also,
  we create two new datasets, X-CSQA and XCODAH, by translating their English versions
  to 15 other languages, so that we can evaluate
  popular ML-LMs for cross-lingual commonsense reasoning. To improve the performance
  beyond English, we propose a simple yet effective method — multilingual contrastive pretraining (MCP). It significantly enhances sentence representations, yielding a large performance gain on both benchmarks (e.g., +2.7%
  accuracy for X-CSQA over XLM-RL).