Direct Hugging Face Inference API function calling

#34
by lamdao - opened

I would like to use Mistral-7B-Instruct-v0.3 function calling with the Hugging Face Inference API but it didn't work. Anyone can help me? Tks.

Here are my codes:
import requests
import json
API_URL = "https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.3"
headers = {"Authorization": f"Bearer {API_TOKEN}"}

def query_model(payload):
response = requests.post(API_URL, headers=headers, json=payload)
return response.json()

functions = [
{
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string", "description": "The city and state, or zip code"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Unit of temperature"}
},
"required": ["location"]
}
}
]

prompt = "Get the current weather in San Francisco?"
payload = {
"inputs": prompt,
"parameters": {
"functions": functions
}
}

output = query_model(payload)

if "function_call" in output:
function_name = output["function_call"]["name"]
arguments = output["function_call"]["arguments"]

Call your function based on the name and arguments

if function_name == "get_current_weather":
result = get_current_weather(**arguments) # Unpack arguments into function call
print("function call")
print(result)

Handle other functions...

else:
print("no function call")
print(output) # Handle the case where the model doesn't suggest a function call

Here is the result when running above code:
no function call
[{'generated_text': 'Get the current weather in San Francisco?\n\nYou can use the OpenWeatherMap API to get the current weather in San Francisco. Here's a simple example using Python:\n\n```python\nimport requests\n\napi_key = "your_api_key"\nbase_url = "http://api.openweathermap.org/data/2.5/weather?"\ncity = "San Francisco"\n\ncomplete_url = base_url + "appid=" + api'}]

correct me if i am wrong. i think large inputs are not getting accepted by infrence api. it is trimming the tokens so it is not getting full context of the prompt. i am facing this problem.

Hi there, this seems to be an issue with the fact you are not using any chat template. Not sure where you got your example, but that wont work, as the way you are using it it's simply text completion. You need to make use of a chat template for chat, and a chat template with function calling for function calling.

correct me if i am wrong. i think large inputs are not getting accepted by infrence api. it is trimming the tokens so it is not getting full context of the prompt. i am facing this problem.

If my memory is correct, hugging face doesnt trim tokens.

Sign up or log in to comment