Configuration
This page contains details describing how to write your own configurations to control how agents can interact with the SWEEnv
environment.
A configuration is represented as a single .yaml
file, specified by the --config
flag in the command line interface, allowing you to...
- Define the commands that agents may use to traverse + modify a codebase (see here for more details)
- Write prompts that are deterministically/conditionally shown to the agent over the course of a single trajectory.
- Control the input/output interface that sits between the agent and
SWEEnv
.
Default config files
Our default config files are in the config/
directory.
Configuration File Fields
The configuration is a .yaml
file that consists of several fields. They are fully represented in this following outline:
# Prompt Templates: Control how observations of environment are shown to agent
system_template: | # .yaml syntax for multi-line string value
First `system` message shown to agent
instance_template: |- # .yaml syntax for multi-line string value w/ no new line
Instance prompt, contains task instance-specific content
next_step_template: |-
Format template of per-turn observation (Contains standard output from agent's action)
next_step_no_output_template: |-
Format template of observation when there is no standard output from the agent's action
format_error_template: |-
Format template of error message (Used when agent's action causes an error)
demonstration_template: |
Format template for showing a demonstration to the agent
demonstrations:
- `trajectories/<username>/<experiment folder>/*.traj`
- File is a demonstration of how to solve a task. This could an agent generated trajectory.
- You can include 1+ demonstrations
# Environment States: Define features of the SWEEnv environment
env_variables:
# Default variables for SWEEnv at the beginning of each instance
CURRENT_FILE: 0
CURRENT_LINE:
OVERLAP:
SEARCH_FILES:
SEARCH_INDEX:
SEARCH_RESULTS:
WINDOW_SIZE:
START_INDEX:
END_INDEX:
START_CURSOR:
END_CUROSR:
START_CURSORS_MARK:
END_CURSOR_MARK:
state_command: |
# `state_command` allows you to update state variables to reflect any aspect of the environment (e.g. current working directory)
name: state
code: |
state() { echo '{"pwd": "'$PWD'"}';
# Action Interface: Define how an agent interacts with the SWEEnv environment
command_files:
- path/to/bash_file.sh
- Each file contains a list of commands implemented in bash
- You can include 1+ command files
parse_command: Reference to functionality for defining command documentation
history_processor: Reference to functionality for controlling agent's message history
parse_function: Parser run on agent output
In the config/
directory, we recommend looking at...
configs/
for examples of properly formatted configuration files. Each configuration differs in its set of commands, input/output format, demonstrations, etc.commands/
for the bash implementations of the custom commands that SWE-agent uses to navigate + edit the codebase. More information here.
Relative paths
Relative paths in config files are resolved to the SWE_AGENT_CONFIG_ROOT
environment variable (if set)
or the SWE-agent repository root.
default_from_url.yaml
system_template: |-
SETTING: You are an autonomous programmer, and you're working directly in the command line with a special interface.
The special interface consists of a file editor that shows you {WINDOW} lines of a file at a time.
In addition to typical bash commands, you can also use the following commands to help you navigate and edit files.
COMMANDS:
{command_docs}
Please note that THE EDIT COMMAND REQUIRES PROPER INDENTATION.
If you'd like to add the line ' print(x)' you must fully write that out, with all those spaces before the code! Indentation is important and code that is not indented correctly will fail and require fixing before it can be run.
RESPONSE FORMAT:
Your shell prompt is formatted as follows:
(Open file: <path>) <cwd> $
You need to format your output using two fields; discussion and command.
Your output should always include _one_ discussion and _one_ command field EXACTLY as in the following example:
DISCUSSION
First I'll start by using ls to see what files are in the current directory. Then maybe we can look at some relevant files to see what they look like.
```
ls -a
```
You should only include a *SINGLE* command in the command section and then wait for a response from the shell before continuing with more discussion and commands. Everything you include in the DISCUSSION section will be saved for future reference.
If you'd like to issue two commands at once, PLEASE DO NOT DO THAT! Please instead first submit just the first command, and then after receiving a response you'll be able to issue the second command.
You're free to use any other bash commands you want (e.g. find, grep, cat, ls, cd) in addition to the special commands listed above.
However, the environment does NOT support interactive session commands (e.g. python, vim), so please do not invoke them.
instance_template: |-
We're currently solving the following issue within our repository. Here's the issue text:
ISSUE:
{issue}
INSTRUCTIONS:
Now, you're going to solve this issue on your own. Your terminal session has started and you're in the repository's root directory. You can use any bash commands or the special interface to help you. Edit all the files you need to and run any checks or tests that you want.
Remember, YOU CAN ONLY ENTER ONE COMMAND AT A TIME. You should always wait for feedback after every command.
When you're satisfied with all of the changes you've made, you can submit your changes to the code base by simply running the submit command.
Note however that you cannot use any interactive session commands (e.g. python, vim) in this environment, but you can write scripts and run them. E.g. you can write a python script and then run it with `python <script_name>.py`.
NOTE ABOUT THE EDIT COMMAND: Indentation really matters! When editing a file, make sure to insert appropriate indentation before each line!
IMPORTANT TIPS:
1. Always start by trying to replicate the bug that the issues discusses.
If the issue includes code for reproducing the bug, we recommend that you re-implement that in your environment, and run it to make sure you can reproduce the bug.
Then start trying to fix it.
When you think you've fixed the bug, re-run the bug reproduction script to make sure that the bug has indeed been fixed.
If the bug reproduction script does not print anything when it successfully runs, we recommend adding a print("Script completed successfully, no errors.") command at the end of the file,
so that you can be sure that the script indeed ran fine all the way through.
2. If you run a command and it doesn't work, try running a different command. A command that did not work once will not work the second time unless you modify it!
3. If you open a file and need to get to an area around a specific line that is not in the first 100 lines, say line 583, don't just use the scroll_down command multiple times. Instead, use the goto 583 command. It's much quicker.
4. If the bug reproduction script requires inputting/reading a specific file, such as buggy-input.png, and you'd like to understand how to input that file, conduct a search in the existing repo code, to see whether someone else has already done that. Do this by running the command: find_file "buggy-input.png" If that doesn't work, use the linux 'find' command.
5. Always make sure to look at the currently open file and the current working directory (which appears right after the currently open file). The currently open file might be in a different directory than the working directory! Note that some commands, such as 'create', open files, so they might change the current open file.
6. When editing files, it is easy to accidentally specify a wrong line number or to write code with incorrect indentation. Always check the code after you issue an edit to make sure that it reflects what you wanted to accomplish. If it didn't, issue another command to fix it.
7. It may be necessary to install the repository from source before you can run code. Please think about how to install the environment from the repository directory if you need to do so.
(Open file: {open_file})
(Current directory: {working_dir})
bash-$
next_step_template: |-
{observation}
(Open file: {open_file})
(Current directory: {working_dir})
bash-$
next_step_no_output_template: |-
Your command ran successfully and did not produce any output.
(Open file: {open_file})
(Current directory: {working_dir})
bash-$
demonstration_template: |
Here is a demonstration of how to correctly accomplish this task.
It is included to show you how to correctly use the interface.
You do not need to follow exactly what is done in the demonstration.
--- DEMONSTRATION ---
{demonstration}
--- END OF DEMONSTRATION ---
state_command:
name: state
code: |
state() {
local working_dir="$PWD";
if [ -z "$CURRENT_FILE" ]; then
echo '{"open_file": "n/a", "working_dir": "'$working_dir'"}';
else
echo '{"open_file": "'$(realpath "$CURRENT_FILE")'", "working_dir": "'$working_dir'"}';
fi
};
parse_function: ThoughtActionParser
env_variables:
WINDOW: 100
OVERLAP: 2
CURRENT_LINE: 0
CURRENT_FILE: ''
SEARCH_RESULTS: ()
SEARCH_FILES: ()
SEARCH_INDEX: 0
command_files:
- config/commands/defaults.sh
- config/commands/search.sh
- config/commands/edit_linting.sh
- config/commands/_split_string.py
- config/commands/submit.sh
parse_command: ParseCommandDetailed
history_processor: Last5Observations
demonstrations:
- trajectories/demonstrations/replay__marshmallow-code__marshmallow-1867__default__t-0.20__p-0.95__c-2.00__install-1___install_from_source/marshmallow-code__marshmallow-1867.traj
How a Configuration File is Processed
Some notes on processing that occurs on config fields when SWE-agent is run:
- Commands specified in
command_files
will be parsed into a single block of documentation text that can be referenced as{command_docs}
. env_variables
are the default variables for the bash environment at the beginning of each instance.state_command
is used to extract state information from the bash environment (formatted as json) to be used in the templates given to the agent.
Possible variables that can be used in templates are:
- {command_docs}
(an automatically compiled collection of available commands + their docstrings)
- any variable given in env_variables
(same spelling), e.g., {WINDOW_SIZE}
- any variable extracted as json as part of the state_command
function
- the last observation {observation}
- ... this list will grow as we implement more features!
Template Workflow
The following diagram illustrates where each template is shown within a single episode of solving one task instance.
One of three templates can be shown per turn:
- "Next Step" (
next_step_template
): Displayed if the model's action successfully runs. The output and a prompt for the next action is shown - "Next Step (No Output)" (
next_step_no_output_template
): Displayed if the model's action successfully runs, but does not produce any standard output (e.g.rm
,cd
) - "Format Error" (
format_error_template
): Displayed if the model's response is malformed. Over the next two turns... - If one of the model's next response is correct, the message history is updated such that the "Format Error" turn is not kept. The episode continues.
- If the model's next two responses are both malformed, the episode terminates.
-
Something broken? Report bug
-
Something unclear? Ask question