Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 11.3k

Running on CPU Upgrade

Hi @Stefan171 ,

Thank you for providing all the log files! That's strange but I also don't see the result files, so I sent your model back for evaluation (you can check the request file), and yes, since the cluster is quite busy currently, the evaluation will take some more time than usual but I hope it will be fine

I'll close this issue. Please feel free to ping me here if you need any help with this model

Hi @alozowski ,

I wanted to provide an update on the evaluation of my model, Stefan171/TinyLlama-QuantumQuill-chat-12-05-24. The test has finished running, but unfortunately, the result is marked as "Failed" in the results file. I appreciate your previous assistance in sending the model back for evaluation.

However, I am unable to locate the result logs for the test. I checked the usual location for processed results here: https://huggingface.co/datasets/open-llm-leaderboard/results/tree/main/Stefan171, but I cannot find any logs related to my model. Could you guide me on where to find the logs for failed tests? I apologise if I have overlooked something, but I couldn't find any documentation regarding if the failed test logs are uploaded elsewhere.

Thank you again for your support throughout this process. I look forward to your guidance on this matter.

Hi @Stefan171 !

I see the problem, I need time to investigate it, and I'll be back once I find a solution👍

Hi @Stefan171 !

UPD: I resubmitted your model, but since our research cluster is quite busy right now, it may take some time to start the evaluation process. I'll be checking that your model has started the evaluation, but if the model fails again, feel free to ping me here, I'll reopen the discussion

Hi @alozowski ,

I wanted to provide an update regarding the evaluation of my LLM model, Stefan171/TinyLlama-QuantumQuill-chat-12-05-24. Unfortunately, the status has updated to 'Failed' again. I've checked the results folder, but I cannot locate any new logs related to this model.

Additionally, I wanted to mention that when I initially created the repo, it only contained the PyTorch version of the model. However, I subsequently uploaded a SafeTensor version of the same model. I'm unsure if this could be a factor, but I wanted to bring it to your attention. I was under the impression that the repo could accommodate both versions without issue.

Thank you for your continued support and guidance throughout this process.

Hi @Stefan171 ,

According to the log, there is the following issue:

model = self.AUTO_MODEL_CLASS.from_pretrained(
  File "/fsx/lighteval/miniconda3/envs/lighteval_prod/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
    return model_class.from_pretrained(
  File "/fsx/lighteval/miniconda3/envs/lighteval_prod/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3534, in from_pretrained
    if metadata.get("format") == "pt":
AttributeError: 'NoneType' object has no attribute 'get'

The error indicates that the metadata variable, which should contain information about the model, is None. Could you please ensure the necessary files like config.json, tokenizer_config.json, and model weights are correctly formatted in the repository? Also, can you verify access permissions and the correct structure of your model repo?

Hi @alozowski ,

Thank you for your help. Yes, I believe I may not have completed all the necessary steps for the SafeTensor upload, and I have more to learn in that regard. For now, I think it's best to start afresh by uploading another version of the PyTorch model from Unsloth to a new repository. Should any errors or issues arise, I will let you know. Your assistance and support throughout this process have been invaluable.