inquiry for gemma-7b : d_model

#61
by seongwoon - opened

gemma report: https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf

Table 1 in the report shows that the dimensionality of gemma-7b model is 3076.

but it also tells num_head is 16 and head size is 256.

so I guess the dimensionality of gemma-7b model should be 4096(16*256).

Can anyone tell me why the inconsistency occurs?

Google org

Hello there! Sorry for the very late reply, thanks for spotting a slight error; it should say "Key Size" instead of "Head Size".

Sign up or log in to comment