Support tool_use chat_template
The v3 version already supports tools ( function calling ) and is already supported in the mistralai/Mixtral-8x22B-Instruct-v0.1 project, this part should be reusable.
Once added, it can be used when using the Huggingface API:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "mistralai/Mistral-7B-Instruct-v0.3"
tokenizer = AutoTokenizer.from_pretrained(model_id)
conversation=[
{"role": "user", "content": "What's the weather like in Paris?"},
{
"role": "tool_calls",
"content": [
{
"name": "get_current_weather",
"arguments": {"location": "Paris, France", "format": "celsius"},
}
]
},
{
"role": "tool_results",
"content": {"content": 22}
},
{"role": "assistant", "content": "The current temperature in Paris, France is 22 degrees Celsius."},
{"role": "user", "content": "What about San Francisco?"}
]
tools = [{"type": "function", "function": {"name":"get_current_weather", "description": "Get▁the▁current▁weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location."}},"required":["location","format"]}}}]
# render the tool use prompt as a string:
tool_use_prompt = tokenizer.apply_chat_template(
conversation,
chat_template="tool_use",
tools=tools,
tokenize=False,
add_generation_prompt=True,
)
model = AutoModelForCausalLM.from_pretrained(model_id)
inputs = tokenizer(tool_use_prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Thanks for the PR, we are currently working on all available repos and reviewing the templates, there might be some changes at some point as some need to be fixed.
I was testing the chat template and one provided in the mistral_common library and I notice the tokens are not exactly matching. Here is the example on the same input mentioned in the repo.
Mistral Inferece:
Template-text: '<s>[AVAILABLE_TOOLS]▁[{"type":▁"function",▁"function":▁{"name":▁"get_current_weather",▁"description":▁"Get▁the▁current▁weather",▁"parameters":▁{"type":▁"object",▁"properties":▁{"location":▁{"type":▁"string",▁"description":▁"The▁city▁and▁state,▁e.g.▁San▁Francisco,▁CA"},▁"format":▁{"type":▁"string",▁"enum":▁["celsius",▁"fahrenheit"],▁"description":▁"The▁temperature▁unit▁to▁use.▁Infer▁this▁from▁the▁users▁location."}},▁"required":▁["location",▁"format"]}}}][/AVAILABLE_TOOLS][INST]▁What\'s▁the▁weather▁like▁today▁in▁Paris?[/INST]'
Tokens: array([ 1, 6, 1501, 7567, 1891, 2032, 1113, 3396, 1316,
1113, 3396, 2032, 10598, 1629, 2032, 1113, 1295, 29498,
3790, 29498, 1537, 1991, 1316, 1113, 7286, 2032, 1113,
2226, 1040, 2636, 8854, 1316, 1113, 12206, 2032, 10598,
1891, 2032, 1113, 3582, 1316, 1113, 11491, 2032, 10598,
3501, 2032, 10598, 1891, 2032, 1113, 2195, 1316, 1113,
7286, 2032, 1113, 1782, 3758, 1072, 2433, 29493, 1085,
29491, 29489, 29491, 4420, 10454, 29493, 10229, 8474, 1113,
4530, 2032, 10598, 1891, 2032, 1113, 2195, 1316, 1113,
10825, 2032, 8135, 29485, 1958, 3938, 1316, 1113, 29490,
19425, 13075, 9651, 1113, 7286, 2032, 1113, 1782, 8409,
5796, 1066, 1706, 29491, 1328, 1410, 1224, 1245, 1040,
6211, 5491, 1379, 11549, 1113, 11661, 2032, 8135, 3501,
1316, 1113, 4530, 3010, 1743, 10925, 7, 3, 2592,
29510, 29481, 1040, 8854, 1505, 3922, 1065, 6233, 29572,
4])
The template above:
Template-text: "<s>[AVAILABLE_TOOLS][{'type': 'function', 'function': {'name': 'get_current_weather', 'description': 'Get the current weather', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'format': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.'}}, 'required': ['location', 'format']}}}][/AVAILABLE_TOOLS][INST]What's the weather like in Paris?[/INST]"
Tokens: array([ 1, 6, 29560, 19205, 1891, 2637, 1232, 3396, 1415,
1232, 3396, 2637, 12780, 1629, 2637, 1232, 1295, 29498,
3790, 29498, 1537, 1991, 1415, 1232, 7286, 2637, 1232,
2226, 1040, 2636, 8854, 1415, 1232, 12206, 2637, 12780,
1891, 2637, 1232, 3582, 1415, 1232, 11491, 2637, 12780,
3501, 2637, 12780, 1891, 2637, 1232, 2195, 1415, 1232,
7286, 2637, 1232, 1782, 3758, 1072, 2433, 29493, 1085,
29491, 29489, 29491, 4420, 10454, 29493, 10229, 16809, 1232,
4530, 2637, 12780, 1891, 2637, 1232, 2195, 1415, 1232,
10825, 2637, 6704, 29485, 1958, 3938, 1415, 1232, 29490,
19425, 13075, 6575, 1232, 7286, 2637, 1232, 1782, 8409,
5796, 1066, 1706, 29491, 1328, 1410, 1224, 1245, 1040,
6211, 5491, 2583, 11549, 1232, 11661, 2637, 6704, 3501,
1415, 1232, 4530, 2189, 1743, 10925, 7, 3, 3963,
29510, 29481, 1040, 8854, 1505, 1065, 6233, 29572, 4])
If you compare the tokens it not exactly matching. Let me know what you think
This is exactly the kind of issue we have been seeing, also why we highly recommend using common. While the other repos all insist a lot on this fact, the normal chat template matches perfectly for this model but function calling not exactly.
The main issue is the tokenizer configuration on hugging face that is a bit tricky as we have a custom implementation (mistral_common)
Basically at the moment only the chat template for chat only on this repo works perfectly, the other templates and models all have similar issues.
Since the other repos it's worse than this one, we provided an example implementation on how to use it with mistral common for at least the tokenization (encode/decode) https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
But let me know if you believe it would be better to have that disclaimer with the implementation with this one too.
To sum up: We recommend make use of mistral_common if possible.
We are however open to PRs to make both match perfectly!
Also, what chat template did you use? Since this repos tokenizer config does not have one for tool usage.
I compared the chat template from mistral-common and the one in this PR
I compared the chat template from mistral-common and the one in this PR
The issue is that mistral_commonis a custom implementation that doesn't use exactly a chat template, we are open to PRs with chat templates that match 1-1 mistral-common, but if the chat template doesn't match I strongly recommend to use directly mistral_common if possible
I was analysing what is reason why this is happening and the difference is caused due to " and ' because by default python strings are used with ' and chat template is different.
mistral_inference = '<s>[AVAILABLE_TOOLS]▁[{"type":▁"function",▁"function":▁{"name":▁"get_current_weather",▁"description":▁"Get▁the▁current▁weather",▁"parameters":▁{"type":▁"object",▁"properties":▁{"location":▁{"type":▁"string",▁"description":▁"The▁city▁and▁state,▁e.g.▁San▁Francisco,▁CA"},▁"format":▁{"type":▁"string",▁"enum":▁["celsius",▁"fahrenheit"],▁"description":▁"The▁temperature▁unit▁to▁use.▁Infer▁this▁from▁the▁users▁location."}},▁"required":▁["location",▁"format"]}}}][/AVAILABLE_TOOLS][INST]▁What\'s▁the▁weather▁like▁today▁in▁Paris?[/INST]'
hf_inference = Template-text: "<s>[AVAILABLE_TOOLS][{'type': 'function', 'function': {'name': 'get_current_weather', 'description': 'Get the current weather', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'format': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit to use. Infer this from the users location.'}}, 'required': ['location', 'format']}}}][/AVAILABLE_TOOLS][INST]What's the weather like in Paris?[/INST]"
How this is fixed is by passing the tools as a json string then the tokens are the same although I made one small change that was adding a single space after [AVAILABLE_TOOLS] in the current PR, Would you like me to fix the chat-template in this PR?
.
import json
tools = [{"type": "function", "function": {"name":"get_current_weather", "description": "Get the current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location."}},"required":["location","format"]}}}]
tools_json = json.dumps(tools)
tools_json
So I think it would be better to have that disclaimer.
Feel free to make a new PR with the chat template you believe matches!