Name: AyaTEC
Description: QA on the Holy Qur’an Dataset
Provider: Qatar University
Paper Title: AyaTEC: Building a Reusable Verse-Based Test Collection for
  Arabic Question Answering on the Holy Qur’an
Tasks: question answering
Venue Name: ACM Transactions on Asian and Low-Resource Language Information Processing
Authors: R. Malhas,Tamer Elsayed
Abstract: The absence of publicly available reusable test collections for Arabic question answering on the Holy Qur’an has impeded the possibility of fairly comparing the performance of systems in that domain. In this article, we introduce AyaTEC, a reusable test collection for verse-based question answering on the Holy Qur’an, which serves as a common experimental testbed for this task.AyaTEC includes 207 questions (with their corresponding 1,762 answers) covering 11 topic categories of the Holy Qur’an that target the information needs of both curious and skeptical users. To the best of our effort, the answers to the questions (each represented as a sequence of verses) in AyaTECwere exhaustive—that is, all qur’anic verses that directly answered the questions were exhaustively extracted and annotated. To facilitate the use ofAyaTEC in evaluating the systems designed for that task, we propose several evaluation measures to support the different types of questions and the nature of verse-based answers while integrating the concept of partial matching of answers in the evaluation.