Skip to content

Batch instances

logger module-attribute 

logger = get_logger('swea-config', emoji='🔧')

BatchInstanceSourceConfig module-attribute 

BatchInstanceSourceConfig = InstancesFromHuggingFace | InstancesFromFile | SWEBenchInstances | ExpertInstancesFromFile

AbstractInstanceSource

Bases: ABC

Anything that adheres to this standard can be used to load instances.

get_instance_configs abstractmethod 

get_instance_configs() -> list[EnvironmentConfig]

BatchInstance pydantic-model

Bases: BaseModel

A single instance in a batch of instances. This specifies both the environment configuration and the problem statement.

Fields:

env pydantic-field 

env: EnvironmentConfig

problem_statement pydantic-field 

problem_statement: ProblemStatementConfig

SimpleBatchInstance pydantic-model

Bases: BaseModel

A simple way to configure a single instance in a batch of instances that all use similar deployment configurations.

Predominantly used for benchmarking purposes. Assumes that the repository is already present in the docker container.

Fields:

image_name pydantic-field 

image_name: str

problem_statement pydantic-field 

problem_statement: str

id pydantic-field 

id: str

repo_name pydantic-field 

repo_name: str = ''

Specifies the repository to use. If empty, no repository is used. If the string does not contain a slash, it is interpreted as an already existing repository at the root of the docker container. If it contains the word "github", it is interpreted as a github repository. Else, it is interpreted as a local repository.

base_commit pydantic-field 

base_commit: str = 'HEAD'

Used to reset repo.

extra_fields pydantic-field 

extra_fields: dict[str, Any]

Any additional data to be added to the instance. This data will be available when formatting prompt templates.

to_full_batch_instance 

to_full_batch_instance(deployment: DeploymentConfig) -> BatchInstance

Merge the deployment options into the SimpleBatchInstance object to get a full BatchInstance.

from_swe_bench classmethod 

from_swe_bench(instance: dict[str, Any]) -> Self

Convert instances from the classical SWE-bench dataset to the SimpleBatchInstance format.

InstancesFromFile pydantic-model

Bases: BaseModel, AbstractInstanceSource

Load instances from a file.

Fields:

path pydantic-field 

path: Path

filter pydantic-field 

filter: str = '.*'

Regular expression to filter the instances by instance id.

slice pydantic-field 

slice: str = ''

Select only a slice of the instances (after filtering by filter). Possible values are stop or start:stop or start:stop:step (i.e., it behaves exactly like python's list slicing list[slice]).

shuffle pydantic-field 

shuffle: bool = False

Shuffle the instances (before filtering and slicing).

deployment pydantic-field 

deployment: DeploymentConfig

Note that the image_name option is overwritten by the images specified in the task instances.

simple pydantic-field 

simple: Literal[True] = True

Convenience discriminator for (de)serialization/CLI. Do not change.

type pydantic-field 

type: Literal['file'] = 'file'

Discriminator for (de)serialization/CLI. Do not change.

id pydantic-field 

id: str

get_instance_configs 

get_instance_configs() -> list[BatchInstance]

InstancesFromHuggingFace pydantic-model

Bases: BaseModel, AbstractInstanceSource

Load instances from HuggingFace.

Fields:

dataset_name pydantic-field 

dataset_name: str

Name of the HuggingFace dataset. Same as when using datasets.load_dataset.

split pydantic-field 

split: str = 'dev'

filter pydantic-field 

filter: str = '.*'

Regular expression to filter the instances by instance id.

slice pydantic-field 

slice: str = ''

Select only a slice of the instances (after filtering by filter). Possible values are stop or start:stop or start:stop:step. (i.e., it behaves exactly like python's list slicing list[slice]).

shuffle pydantic-field 

shuffle: bool = False

Shuffle the instances (before filtering and slicing).

deployment pydantic-field 

deployment: DeploymentConfig

Deployment configuration. Note that the image_name option is overwritten by the images specified in the task instances.

type pydantic-field 

type: Literal['huggingface'] = 'huggingface'

Discriminator for (de)serialization/CLI. Do not change.

id pydantic-field 

id: str

get_instance_configs 

get_instance_configs() -> list[BatchInstance]

SWEBenchInstances pydantic-model

Bases: BaseModel, AbstractInstanceSource

Load instances from SWE-bench.

Fields:

subset pydantic-field 

subset: Literal['lite', 'verified', 'full'] = 'lite'

split pydantic-field 

split: Literal['dev', 'test'] = 'dev'

deployment pydantic-field 

deployment: DeploymentConfig

Deployment configuration. Note that the image_name option is overwritten by the images specified in the task instances.

type pydantic-field 

type: Literal['swe_bench'] = 'swe_bench'

Discriminator for (de)serialization/CLI. Do not change.

filter pydantic-field 

filter: str = '.*'

Regular expression to filter the instances by instance id.

slice pydantic-field 

slice: str = ''

Select only a slice of the instances (after filtering by filter). Possible values are stop or start:stop or start:stop:step. (i.e., it behaves exactly like python's list slicing list[slice]).

shuffle pydantic-field 

shuffle: bool = False

Shuffle the instances (before filtering and slicing).

evaluate pydantic-field 

evaluate: bool = False

Run sb-cli to evaluate

id pydantic-field 

id: str

get_instance_configs 

get_instance_configs() -> list[BatchInstance]

ExpertInstancesFromFile pydantic-model

Bases: BaseModel, AbstractInstanceSource

Load instances from a file. The difference to InstancesFromFile is that the instances are configured as full EnvironmentInstanceConfig objects, i.e., we could specify separate deployment configurations etc.

Fields:

path pydantic-field 

path: Path

filter pydantic-field 

filter: str = '.*'

Regular expression to filter the instances by instance id.

slice pydantic-field 

slice: str = ''

Select only a slice of the instances (after filtering by filter). Possible values are stop or start:stop or start:stop:step. (i.e., it behaves exactly like python's list slicing list[slice]).

shuffle pydantic-field 

shuffle: bool = False

Shuffle the instances (before filtering and slicing).

type pydantic-field 

type: Literal['expert_file'] = 'expert_file'

Discriminator for (de)serialization/CLI. Do not change.

id pydantic-field 

id: str

get_instance_configs 

get_instance_configs() -> list[BatchInstance]