meta-llama

/

Llama-3.2-1B

like 206

Hello, I finetuned the model and then when I want to evaluate it, I need to get the scores as well. I set output_scores=True, return_dict_in_generate=True but I got an error :
predictions_with_scores = pipe(input_texts, max_new_tokens=30, num_beams=k, num_return_sequences=1, output_scores=True, return_dict_in_generate=True)
File "/opt/anaconda3/envs/openllama/lib/python3.10/site-packages/transformers/pipelines/text_generation.py", line 272, in call
return super().call(text_inputs, **kwargs)
File "/opt/anaconda3/envs/openllama/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1249, in call
outputs = list(final_iterator)
File "/opt/anaconda3/envs/openllama/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in next
item = next(self.iterator)
File "/opt/anaconda3/envs/openllama/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 125, in next
processed = self.infer(item, **self.params)
File "/opt/anaconda3/envs/openllama/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1175, in forward
model_outputs = self._forward(model_inputs, **forward_params)
File "/opt/anaconda3/envs/openllama/lib/python3.10/site-packages/transformers/pipelines/text_generation.py", line 371, in _forward
out_b = generated_sequence.shape[0]
AttributeError: 'GenerateBeamDecoderOnlyOutput' object has no attribute 'shape'

My codes are:
for batch in tqdm(dataloader):
# Decode input IDs into text for the pipeline
input_texts = pipe.tokenizer.batch_decode(batch["input_ids"], skip_special_tokens=True)
predictions_with_scores = pipe(input_texts, max_new_tokens=30, num_beams=k, num_return_sequences=1, output_scores=True, return_dict_in_generate=True)

    # The generated sentences are the predictions
    generated_sents = [pred for pred in predictions_with_scores['sequences']]

    # Get the scores for each prediction
    prediction_scores = predictions_with_scores['sequences_scores']  # These are the scores returned from pipeline

    # Decode the gold (ground truth) sentences
    gold_sents = pipe.tokenizer.batch_decode(batch['label'], skip_special_tokens=True)

my transformers version : 4.45.1

Can anyone tell me how to get the score? Or why it fails when using 'output_scores=True, return_dict_in_generate=True'?

Thanks.