Models

This page documents the configuration objects used to specify the behavior of a language model (LM).

Normal LMs

sweagent.agent.models.GenericAPIModelConfig pydantic-model

Bases: BaseModel

This configuration object specifies a LM like GPT4 or similar. The model will be served with the help of the litellm library.

Config:

• extra: 'forbid'

Fields:

• name (str)
• per_instance_cost_limit (float)
• total_cost_limit (float)
• temperature (float)
• top_p (float | None)
• api_base (str | None)
• api_version (str | None)
• api_key (SecretStr | None)
• stop (list[str])
• completion_kwargs (dict[str, Any])
• convert_system_to_user (bool)
• retry (RetryConfig)
• delay (float)
• fallbacks (list[dict[str, Any]])
• id (str)

api_base pydantic-field

api_base: str | None = None

api_key pydantic-field

api_key: SecretStr | None = None

API key to the model. We recommend using environment variables to set this instead or putting your environment variables in a .env file. You can concatenate more than one key by separating them with :::, e.g., key1:::key2.

api_version pydantic-field

api_version: str | None = None

completion_kwargs pydantic-field

completion_kwargs: dict[str, Any] = {}

Additional kwargs to pass to litellm.completion

convert_system_to_user pydantic-field

convert_system_to_user: bool = False

Whether to convert system messages to user messages. This is useful for models that do not support system messages like o1.

delay pydantic-field

delay: float = 0.0

Minimum delay before querying (this can help to avoid overusing the API if sharing it with other people).

fallbacks pydantic-field

fallbacks: list[dict[str, Any]] = []

List of fallbacks to try if the main model fails See https://docs.litellm.ai/docs/completion/reliable_completions#fallbacks-sdk for more information.

id pydantic-field

id: str

name pydantic-field

name: str

Name of the model.

per_instance_cost_limit pydantic-field

per_instance_cost_limit: float = 3.0

Cost limit for every instance (task).

retry pydantic-field

retry: RetryConfig

Retry configuration: How often to retry after a failure (e.g., from a rate limit) etc.

stop pydantic-field

stop: list[str] = []

Custom stop sequences

temperature pydantic-field

temperature: float = 0.0

Sampling temperature

top_p pydantic-field

top_p: float | None = 1.0

Sampling top-p

total_cost_limit pydantic-field

total_cost_limit: float = 0.0

Total cost limit.

get_api_keys

get_api_keys() -> list[str]

Source code in sweagent/agent/models.py

def get_api_keys(self) -> list[str]:
    if self.api_key is None:
        return []
    return self.api_key.get_secret_value().split(":::")

Manual models for testing

The following two models allow you to test your environment by prompting you for actions. This can also be very useful to create your first demonstrations.

sweagent.agent.models.HumanModel

HumanModel(config: HumanModelConfig, tools: ToolConfig)

Bases: AbstractModel

Model that allows for human-in-the-loop

Source code in sweagent/agent/models.py

def __init__(self, config: HumanModelConfig, tools: ToolConfig):
    """Model that allows for human-in-the-loop"""
    self.logger = get_logger("swea-lm", emoji="🤖")
    self.config = config
    self.stats = InstanceStats()

    # Determine which commands require multi-line input
    self.multi_line_command_endings = {
        command.name: command.end_name for command in tools.commands if command.end_name is not None
    }
    self._readline_histfile = REPO_ROOT / ".swe-agent-human-history"
    self._load_readline_history()

config instance-attribute

config = config

logger instance-attribute

logger = get_logger('swea-lm', emoji='🤖')

multi_line_command_endings instance-attribute

multi_line_command_endings = {name: _hiIpQmGfor command in commands if end_name is not None}

stats instance-attribute

stats = InstanceStats()

query

query(history: History, action_prompt: str = '> ') -> dict

Wrapper to separate action prompt from formatting

Source code in sweagent/agent/models.py

def query(self, history: History, action_prompt: str = "> ") -> dict:
    """Wrapper to separate action prompt from formatting"""
    try:
        return {"message": self._query(history, action_prompt)}
    except KeyboardInterrupt:
        print("^C (exit with ^D)")
        return self.query(history, action_prompt)
    except EOFError:
        print("\nGoodbye!")
        return {"message": "exit"}

sweagent.agent.models.HumanThoughtModel

HumanThoughtModel(config: HumanModelConfig, tools: ToolConfig)

Bases: HumanModel

Model that allows for human-in-the-loop

Source code in sweagent/agent/models.py

def __init__(self, config: HumanModelConfig, tools: ToolConfig):
    """Model that allows for human-in-the-loop"""
    self.logger = get_logger("swea-lm", emoji="🤖")
    self.config = config
    self.stats = InstanceStats()

    # Determine which commands require multi-line input
    self.multi_line_command_endings = {
        command.name: command.end_name for command in tools.commands if command.end_name is not None
    }
    self._readline_histfile = REPO_ROOT / ".swe-agent-human-history"
    self._load_readline_history()

query

query(history: History) -> dict

Logic for handling user input (both thought + action) to pass to SWEEnv

Source code in sweagent/agent/models.py

def query(self, history: History) -> dict:
    """Logic for handling user input (both thought + action) to pass to SWEEnv"""
    thought_all = ""
    thought = input("Thought (end w/ END_THOUGHT): ")
    while True:
        if "END_THOUGHT" in thought:
            thought = thought.split("END_THOUGHT")[0]
            thought_all += thought
            break
        thought_all += thought
        thought = input("... ")

    action = super()._query(history, action_prompt="Action: ")

    return {"message": f"{thought_all}\n```\n{action}\n```"}

Replay model for testing and demonstrations

sweagent.agent.models.ReplayModel

ReplayModel(config: ReplayModelConfig, tools: ToolConfig)

Bases: AbstractModel

Model used for replaying a trajectory (i.e., taking all the actions for the .traj file and re-issuing them.

Source code in sweagent/agent/models.py

def __init__(self, config: ReplayModelConfig, tools: ToolConfig):
    """Model used for replaying a trajectory (i.e., taking all the actions for the `.traj` file
    and re-issuing them.
    """
    self.config = config
    self.stats = InstanceStats()

    if not self.config.replay_path.exists():
        msg = f"Replay file {self.config.replay_path} not found"
        raise FileNotFoundError(msg)

    self._replays = [
        list(json.loads(x).values())[0] for x in Path(self.config.replay_path).read_text().splitlines(keepends=True)
    ]
    self._replay_idx = 0
    self._action_idx = 0
    self.use_function_calling = tools.use_function_calling
    self.submit_command = tools.submit_command
    self.logger = get_logger("swea-lm", emoji="🤖")

config instance-attribute

config = config

logger instance-attribute

logger = get_logger('swea-lm', emoji='🤖')

stats instance-attribute

stats = InstanceStats()

submit_command instance-attribute

submit_command = submit_command

use_function_calling instance-attribute

use_function_calling = use_function_calling

query

query(history: History) -> dict

Logic for tracking which replay action to pass to SWEEnv

Source code in sweagent/agent/models.py

def query(self, history: History) -> dict:
    """Logic for tracking which replay action to pass to SWEEnv"""
    self.stats.api_calls += 1
    actions = self._replays[self._replay_idx]
    try:
        action = actions[self._action_idx]
    except IndexError:
        # log error
        self.logger.error("Reached end of replay trajectory without submitting. Submitting now.")
        if self.use_function_calling:
            action = {
                "message": f"Calling `{self.submit_command}` to submit.",
                "tool_calls": [
                    {
                        "type": "function",
                        "id": "call_submit",
                        "function": {
                            "name": self.submit_command,
                            "arguments": "{}",
                        },
                    }
                ],
            }
        else:
            action = f"```\n{self.submit_command}\n```"

    self._action_idx += 1

    # Assuming `submit` is always last action of replay trajectory
    if isinstance(action, str) and action == "submit":
        self._next_replay()
        return {"message": action}

    # Handle both dict and string actions
    if isinstance(action, dict):
        return action
    return {"message": action}