Skip to content

Introduction to Taskpool Module

A taskpool is a collection of tasks that can be used to evaluate the performance of merged models. Each task in the taskpool is defined by a dataset and a metric.

A taskpool is specified by a yaml configuration file, which often contains the following fields:

  • type: The type of the taskpool.
  • dataset_type: The type of the dataset used in the tasks.
  • tasks: A list of tasks, each task is dict with the following fields:
    • name: The name of the task.
    • dataset: The dataset used for the task.
    • metric: The metric used to evaluate the performance of the model on the task.

References

BaseTaskPool

Bases: BaseYAMLSerializableModel

Source code in fusion_bench/taskpool/base_pool.py 
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
class BaseTaskPool(BaseYAMLSerializableModel):
    _program = None

    @abstractmethod
    def evaluate(self, model):
        """
        Evaluate the model on all tasks in the task pool, and return a report.

        Take image classification as an example, the report will look like:

        ```python
        {
            "mnist": {
                "accuracy": 0.8,
                "loss": 0.2,
            },
            <task_name>: {
                <metric_name>: <metric_value>,
                ...
            },
        }
        ```

        Args:
            model: The model to evaluate.

        Returns:
            report (dict): A dictionary containing the results of the evaluation for each task.
        """
        pass
evaluate(model) abstractmethod

Evaluate the model on all tasks in the task pool, and return a report.

Take image classification as an example, the report will look like:

{
    "mnist": {
        "accuracy": 0.8,
        "loss": 0.2,
    },
    <task_name>: {
        <metric_name>: <metric_value>,
        ...
    },
}

Parameters:

  • model
    –

    The model to evaluate.

Returns:

  • report ( dict ) –

    A dictionary containing the results of the evaluation for each task.

Source code in fusion_bench/taskpool/base_pool.py 
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
@abstractmethod
def evaluate(self, model):
    """
    Evaluate the model on all tasks in the task pool, and return a report.

    Take image classification as an example, the report will look like:

    ```python
    {
        "mnist": {
            "accuracy": 0.8,
            "loss": 0.2,
        },
        <task_name>: {
            <metric_name>: <metric_value>,
            ...
        },
    }
    ```

    Args:
        model: The model to evaluate.

    Returns:
        report (dict): A dictionary containing the results of the evaluation for each task.
    """
    pass