Skip to content

Models and API keys

Setting API keys

In order to access the LM of your choice (and to access private GitHub repositories), you need to supply the corresponding keys.

There are two options to do this:

  1. Set the corresponding environment variables.
  2. Create a .env file at the root of this repository. All of the variables defined there will take the place of environment variables.
  3. Use --agent.model.api_key to set the key

Here's an example

# Remove the comment '#' in front of the line for all keys that you have set
# GITHUB_TOKEN='GitHub Token for access to private repos'
# OPENAI_API_KEY='OpenAI API Key Here if using OpenAI Model'
# ANTHROPIC_API_KEY='Anthropic API Key Here if using Anthropic Model'
# TOGETHER_API_KEY='Together API Key Here if using Together Model'

See the following links for tutorials on obtaining Anthropic, OpenAI, and Github tokens.

Advanced settings

See model config for more details on advanced settings.

Supported API models

We support all models supported by litellm, see their list here.

Here are a few options for --agent.model.name:

Model API key Comment
claude-3-5-sonnet-20241022 ANTHROPIC_API_KEY Our recommended model
gpt-4o OPENAI_API_KEY
o1-preview OPENAI_API_KEY You might need to set temperature and sampling to the supported values.

Function calling and more: Setting the correct parser

The default config uses function calling to retrieve actions from the model response, i.e., the model directly provides the action as a JSON object. If your model doesn't support function calling, you can use the thought_action parser by setting agent.tools.parse_function.type to thought_action. Then, we extract the last triple-backticks block from the model response as the action. See our API docs for more details on parsers. Remember to document the tools in your prompt as the model will not be able to see the function signature like with function calling.

Specific models

See model config for more details on specific models.

Using local models

We currently support all models that serve to an endpoint with an OpenAI-compatible API.

For example, to use llama, you can follow the litellm instructions and set

config/your_config.yaml
agent:
  model:
    name: ollama/llama2  # (1)!
    api_base: http://localhost:11434
    per_instance_cost_limit: 0   # (2)!
    total_cost_limit: 0
    per_instance_call_limit: 100
    max_input_tokens: 0  # (3)!
  tools:
    # The default for obtaining actions from model outputs is function calling.
    # If your local model does not support that, you can use the thought_action parser
    # instead (see below)
    parse_function:
      type: "thought_action"
  1. Make sure that your model includes a "provider", i.e., follows the form provider/model_name. The model name and provider might be arbitrarily chosen.
  2. We cannot track costs, so you must disable this (see below)
  3. Disable max input tokens check

in your config file. If you're using a litellm proxy, make sure to set your agent.model.name to openai/... and set agent.model.api_key to the key you've configured for your proxy (or a random value; it cannot be empty).

Model providers

Make sure that your model name includes a "provider", i.e., follows the form provider/model_name. The model name and provider might be arbitrarily chosen for local models.

Cost/token limits

If you do not disable the default cost limits, you will see an error because the cost calculator will not be able to find the model in the litellm model cost dictionary. Please make sure to the set the per_instance_cost_limit to 0 and use the per_instance_call_limit instead to limit the runtime per issue. Please also make sure to set max_input_tokens to a non-None value to avoid a lot of warnings.

Parsing functions

The default config uses function calling to retrieve actions from the model response, i.e., the model directly provides the action as a JSON object. If your model doesn't support function calling, you can use the thought_action parser by setting agent.tools.parse_function.type to thought_action. Then, we extract the last triple-backticks block from the model response as the action. See our API docs for more details on parsers. Remember to document the tools in your prompt as the model will not be able to see the function signature like with function calling.

Message types

The cache_control history processor requires a different message format (e.g., {'role': 'user', 'content': [{'type': 'text', 'text': 'some text', 'cache_control': {'type': 'ephemeral'}}]}]). This might not be understood by all language models. Therefore, please remove this history processor if you do not need it (it's currently mostly used for anthropic cache control). See #957 for more information.

Something went wrong?

  • If you get Error code: 404, please check your configured keys, in particular whether you set OPENAI_API_BASE_URL correctly (if you're not using it, the line should be deleted or commented out). Also see this issue for reference.

Further reads & debugging

Further reads

See our API docs for all available options. Our model config page has more details on specific models and tips and tricks.