Introduction to Taskpool Module¶

A taskpool is a collection of tasks that can be used to evaluate the performance of merged models. Each task in the taskpool is defined by a dataset and a metric.

A taskpool is specified by a yaml configuration file, which often contains the following fields:

type: The type of the taskpool.
dataset_type: The type of the dataset used in the tasks.
tasks: A list of tasks, each task is dict with the following fields:
- name: The name of the task.
- dataset: The dataset used for the task.
- metric: The metric used to evaluate the performance of the model on the task.

References¶

`load_taskpool_from_config(taskpool_config)` ¶

Loads a task pool based on the provided configuration.

The function checks the 'type' attribute of the configuration and returns an instance of the corresponding task pool. If the 'type' attribute is not found or does not match any known task pool types, a ValueError is raised.

Parameters:

taskpool_config (DictConfig) –

The configuration for the task pool. Must contain a 'type' attribute that specifies the type of the task pool.

Returns:

–

An instance of the specified task pool.

Raises:

ValueError –

If 'type' attribute is not found in the configuration or does not match any known task pool types.

Source code in fusion_bench/taskpool/__init__.py

def load_taskpool_from_config(taskpool_config: DictConfig):
    """
    Loads a task pool based on the provided configuration.

    The function checks the 'type' attribute of the configuration and returns an instance of the corresponding task pool.
    If the 'type' attribute is not found or does not match any known task pool types, a ValueError is raised.

    Args:
        taskpool_config (DictConfig): The configuration for the task pool. Must contain a 'type' attribute that specifies the type of the task pool.

    Returns:
        An instance of the specified task pool.

    Raises:
        ValueError: If 'type' attribute is not found in the configuration or does not match any known task pool types.
    """
    return TaskPoolFactory.create_taskpool(taskpool_config)

`TaskPool` ¶

Bases: ABC

Source code in fusion_bench/taskpool/base_pool.py

class TaskPool(ABC):
    _program = None

    def __init__(self, taskpool_config: DictConfig):
        super().__init__()
        self.config = taskpool_config

        if self.config.get("tasks", None) is not None:
            task_names = [task["name"] for task in self.config["tasks"]]
            assert len(task_names) == len(
                set(task_names)
            ), "Duplicate task names found in the task pool"
            self._all_task_names = task_names

    def evaluate(self, model):
        """
        Evaluate the model on all tasks in the task pool, and return a report.

        Take image classification as an example, the report will look like:

        ```python
        {
            "mnist": {
                "accuracy": 0.8,
                "loss": 0.2,
            },
            <task_name>: {
                <metric_name>: <metric_value>,
                ...
            },
        }
        ```

        Args:
            model: The model to evaluate.

        Returns:
            report (dict): A dictionary containing the results of the evaluation for each task.
        """
        report = {}
        for task_name in tqdm(self.task_names, desc="Evaluating tasks"):
            task = self.load_task(task_name)
            result = task.evaluate(model)
            report[task_name] = result
        return report

    @property
    def task_names(self):
        return self._all_task_names

    def get_task_config(self, task_name: str):
        for task in self.config["tasks"]:
            if task["name"] == task_name:
                return task
        raise ValueError(f"Task {task_name} not found in the task pool")

    def load_task(self, task_name_or_config: Union[str, DictConfig]):
        raise NotImplementedError

`evaluate(model)` ¶

Evaluate the model on all tasks in the task pool, and return a report.

Take image classification as an example, the report will look like:

{
    "mnist": {
        "accuracy": 0.8,
        "loss": 0.2,
    },
    <task_name>: {
        <metric_name>: <metric_value>,
        ...
    },
}