nonsense response when bsz>1
#16
by
OliverNova
- opened
It seems that both 9b-it and 27b-it will generate nonsense responses when using a bsz > 1 for inference.
Would you kindly look into that?
Thank you!
Hey @OliverNova , thanks for your report! Can you please make sure you're using the latest transformers version (v4.42.3)?
If it still happens, do you mind sharing a reproducible code snippet for us to take a look at?