google

/

gemma-2-9b-it

like 231

The following error occurs and generation fails when input exceeding the context length of 8192 is given during inference.Does anyone know what the solution is?

• error and output:
  (llm) C:\code\llm>python gemma2-9b_test.py
  Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:12<00:00, 3.23s/it]
  c:\code\virtualens\llm\Lib\site-packages\transformers\models\gemma2\modeling_gemma2.py:577: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
  C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:92: block: [16485,0,0], thread: [64,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
  ...
  C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:92: block: [16485,0,0], thread: [92,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
  C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:92: block: [16485,0,0], thread: [93,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
  C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:92: block: [16485,0,0], thread: [94,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
  C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:92: block: [16485,0,0], thread: [95,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
  Traceback (most recent call last):
  File "C:\code\llm\gemma2-9b_test.py", line 73, in
  output = pipe(messages, **generation_args)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "c:\code\virtualens\llm\Lib\site-packages\transformers\pipelines\text_generation.py", line 257, in call
return super().call(Chat(text_inputs), **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\pipelines\base.py", line 1254, in call
return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\pipelines\base.py", line 1261, in run_single
model_outputs = self.forward(model_inputs, **forward_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\pipelines\base.py", line 1161, in forward
model_outputs = self._forward(model_inputs, **forward_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\pipelines\text_generation.py", line 349, in _forward
generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\generation\utils.py", line 1914, in generate
result = self._sample(
^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\generation\utils.py", line 2651, in _sample
outputs = self(
^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\models\gemma2\modeling_gemma2.py", line 1063, in forward
outputs = self.model(
^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\models\gemma2\modeling_gemma2.py", line 903, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\models\gemma2\modeling_gemma2.py", line 645, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\models\gemma2\modeling_gemma2.py", line 557, in forward
key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\cache_utils.py", line 1071, in update
return update_fn(
^^^^^^^^^^
File "c:\code\virtualens\llm\Lib\site-packages\transformers\cache_utils.py", line 1046, in _static_update
k_out[:, :, cache_position] = key_states
~~~~~^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

• pip package:

accelerate 0.31.0
certifi 2024.6.2
charset-normalizer 3.3.2
colorama 0.4.6
einops 0.8.0
filelock 3.13.1
flash-attn 2.5.9.post1
fsspec 2024.2.0
huggingface-hub 0.23.4
idna 3.7
intel-openmp 2021.4.0
Jinja2 3.1.3
MarkupSafe 2.1.5
mkl 2021.4.0
mpmath 1.3.0
networkx 3.2.1
numpy 1.26.3
packaging 24.1
pillow 10.2.0
pip 24.1.1
psutil 5.9.8
PyYAML 6.0.1
regex 2024.5.15
requests 2.32.3
safetensors 0.4.3
setuptools 65.5.0
sympy 1.12
tbb 2021.11.0
tokenizers 0.19.1
torch 2.3.1+cu121
torchaudio 2.3.1+cu121
torchvision 0.18.1+cu121
tqdm 4.66.4
transformers 4.42.2
typing_extensions 4.9.0
urllib3 2.2.1
wheel 0.43.0

• generation code:

import torch
import os
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
from datetime import datetime

os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

HF_TOKEN = '*****'
model = AutoModelForCausalLM.from_pretrained(
"google/gemma-2-9b-it",
device_map="cuda",
torch_dtype="auto",
trust_remote_code=True,
token=HF_TOKEN,
)
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b-it", token=HF_TOKEN)

messages = [
{"role": "user", "content": f"""\messages......"""},

]

pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
)

generation_args = {
"max_new_tokens": 500,
"return_full_text": False,
"temperature": 0.2,
"do_sample": True,
}

output = pipe(messages, **generation_args)
print(output[0]['generated_text'])