Handle Batch Sizes
I hope I'm not overstepping boundaries. I tried the fix on #16 to handle batch sizes larger than 1 but stumbled upon a few issues. For example, issues processing single strings as the text input or not handling padding when the input texts are of different size.
I tried to address all the issues I had. Hope this can be of use.
@gugarosa feel free to have a look and let me know if I'm overlooking something.
Best,
William
Yea my solution wasn't complete--I wish you could edit PRs in Huggingface. Setting labels[labels == pad_token] = -100 is what finally got the training to start running if you wanted to do supervised fine tuning. On that note, I'm not entirely clear on whether its preferable to use trl.SFTTrainer for that case.
hi
@WilliamSotoM
this solution will throw below error when Flash attention is enabled:
modeling_phi3_v.py", line 1135, in forward
raise ValueError(
ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Phi3. Make sure to call tokenizer.padding_side = 'left' before tokenizing the input.
Hello @haipingwu
That's weird, as you can see in line 198 I'm placing the padding tokens on the left of the input_ids.
Can you check the input _ids before passing them to the model and confirm the padding is falling on the wrong side?
Do you think you can provide a snippet of the code you are using to generate? I'm not having this issue using model.generate().
@haipingwu quick update... I found the bug. It happens when you don't pass any images. Line 149 defaults to the standard tokenizer and processes all the text in one go. Seems like the default padding size o the tokenizer is not set properly. I suppose it was not caught in the original code because, since no batches were being used, no padding was needed.
The solution is to add self.tokenizer.padding_side = 'left' on line 55right after initializing the tokenizer.
I'm going to close/delete this PR and create a new one with the fixed code.
@sebbyjp ,you were right, I wish you could edit PRs in Huggingface too!