Models disappearing from eval queue?

#805
by ArkaAbacus - opened

Hello,

We added Smaug-Llama-3-70B-Instruct and Smaug-Qwen2-72B-Instruct to the new LLM leaderboard eval queue yesterday, but it seems they have disappeared today and also not yet turned up on the leaderboard.

Any idea what might have happened? Should we resubmit?

Open LLM Leaderboard org

Hi!
I think you could check our FAQ :)
TLDR: Either we have a problem with the display atm, or they ran but failed. You'll get this info by looking for their request and result files.

@clefourrier From what I've seen the leaderboard does not update until restarted. Whether that is intentional or not, restarts from time to time would be nice until there is a better solution.

Open LLM Leaderboard org

Interesting!
We actually have a new system with webhooks, where the leaderboard should be updated max 10 min after a change on our datasets (redownloaded with every change) - I'll take a look again at this

ArkaAbacus changed discussion status to closed
ArkaAbacus changed discussion status to open

Thanks for the pointer. I found the status of Smaug-Llama:

{
"model": "abacusai/Smaug-Llama-3-70B-Instruct",
"base_model": "",
"revision": "8f558d6211b9d8f1712b80df40c5b65bea0b56ea",
"precision": "bfloat16",
"params": 70.554,
"architectures": "LlamaForCausalLM",
"weight_type": "Original",
"status": "FAILED",
"submitted_time": "2024-06-26T16:29:36Z",
"model_type": "\ud83d\udd36 : \ud83d\udd36 fine-tuned on domain-specific datasets",
"job_id": "7215733",
"job_start_time": "2024-06-27T00:10:42.571625",
"use_chat_template": true
}

It's not clear what caused the FAILURE - we know the model files are non-corrupt as it worked fine on the old leaderboard. In any case, I've resubmitted for now.

Open LLM Leaderboard org

Hi! Please do not try to resubmit models which failed!
Instead, give us the link to the request file so we can investigate and relaunch if necessary!

Ah - my apologies, I've already resubmitted. The requests file was originally here: https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/abacusai/Smaug-Llama-3-70B-Instruct_eval_request_False_bfloat16_Original.json

although it has now been updated since I have resubmitted.

Open LLM Leaderboard org
edited Jun 28

It should no longer be possible to resubmit a model which was already submitted, so thanks for raising the issue, at least this has been fixed.

Re-Smaug, it got preempted - normally the job should have been rescheduled but apparently was not, tagging @SaylorTwift - note that since it's PENDING again, it will be relaunched soon, when there is enough space on the cluster

clefourrier changed discussion status to closed

Sign up or log in to comment