cannot load results

#811
by ysharma1126 - opened

datasets.load_dataset("open-llm-leaderboard/details_tiiuae__falcon-7b", name="results", split="latest",)
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
"because column names don't match"

regarding the new leaderboard, it seems even llama3 isn't loading (also tried removing the "-private" suffix)

datasets.load_dataset("HuggingFaceEvalInternal/meta-llama__Meta-Llama-3-70B-Instruct-details-private", name="results", split="latest",)
FileNotFoundError: Couldn't find a dataset script at ../HuggingFaceEvalInternal/meta-llama__Meta-Llama-3-70B-Instruct-details/meta-llama__Meta-Llama-3-70B-Instruct-details.py or any data file in the same directory. Couldn't find 'HuggingFaceEvalInternal/meta-llama__Meta-Llama-3-70B-Instruct-details' on the Hugging Face Hub either: FileNotFoundError: Dataset 'HuggingFaceEvalInternal/meta-llama__Meta-Llama-3-70B-Instruct-details' doesn't exist on the Hub. If the repo is private or gated, make sure to log in with huggingface-cli login.

Open LLM Leaderboard org
edited Jul 2

Hi @ysharma1126 ,

You can download the open-llm-leaderboard/tiiuae__falcon-7b-details dataset like this:

from datasets import load_dataset

falcon_ds = load_dataset("open-llm-leaderboard/tiiuae__falcon-7b-details", 
                         name="tiiuae__falcon-7b__leaderboard_arc_challenge", 
                         split="latest",
                         use_auth_token = "your_token",
                         streaming=True,
                         )

And check the samples inside:

for sample in falcon_ds.take(5):
    print(sample)

Note, that this dataset is gated, so you need to request the access. Plus, here I check one of the tasks – "arc_challenge". You can check other tasks by passing them into name parameter.

Considering Meta-Llama-3-70B-Instruct, we use HuggingFaceEvalInternal organisation as our private one, so you won't have access to it. You can check the details for open-llm-leaderboard/meta-llama__Meta-Llama-3-70B-Instruct-details the same way as for falcon-7b

Open LLM Leaderboard org

I close this discussion, please, write here if you have any other questions

alozowski changed discussion status to closed

Sign up or log in to comment