Introduction to Algorithm Module¶

The Fusion Algorithm module is a core component of the FusionBench project, dedicated to the implementation and execution of various model fusion techniques. This module provides the mechanisms necessary to combine multiple models from the Model Pool, enabling nuanced and optimized model merging operations.

Key Points of the `Fusion Algorithm` Module¶

Adaptive Fusion: The module supports advanced fusion techniques, such as AdaMerging, that adaptively learn the best coefficients for model merging using sophisticated methods like entropy minimization.
Algorithm Configuration: Algorithms are defined and loaded based on configuration files, ensuring flexibility and ease of experimentation. This modular approach allows researchers to switch between different fusion methods seamlessly.
Model Integration: It facilitates the integration of multiple models, combining their strengths and mitigating individual weaknesses. The result is a single, merged model that ideally performs better than any individual model alone or have multitasking capability.
Evaluation Support: Once the model fusion process is completed, the merged model can interface with the TaskPool to evaluate the performance of the merged model across various tasks, providing a comprehensive assessment of its capabilities.

Example Capabilities¶

Entropy Minimization: Some algorithms in this module utilize entropy minimization on unlabeled test samples to refine merging coefficients, ensuring that the fusion process is data-driven and optimized.
Layer-wise and Task-wise Fusion: It allows both layer-wise and task-wise model fusion, where merging coefficients can be learned for individual layers or entire tasks, respectively.

Code Integration¶

The module is typically invoked through a configuration-driven approach in CLI scripts, enabling users to specify fusion algorithms and parameters via YAML configuration files. This method ensures reproducibility and ease of use. For more information, see the document of fusion_bench CLI.

ModelFusionAlgorithm is the base class for all fusion algorithms in the Fusion Algorithm module. It provides a common interface for different fusion techniques, allowing for seamless integration and execution of various algorithms.

`ModelFusionAlgorithm` ¶

Bases: ABC

Source code in fusion_bench/method/base_algorithm.py

class ModelFusionAlgorithm(ABC):
    def __init__(self, algorithm_config: Optional[DictConfig] = None):
        super().__init__()
        if algorithm_config is None:
            algorithm_config = DictConfig({})
        self.config = algorithm_config

    @abstractmethod
    def run(self, modelpool):
        """
        Fuse the models in the given model pool.

        Examples:
            >>> algorithm = SimpleAverageAlgorithm()
            >>> modelpool = ModelPool()
            >>> merged_model = algorithm.fuse(modelpool)

        Args:
            modelpool (_type_): _description_
        """
        pass

`run(modelpool)` `abstractmethod` ¶

Fuse the models in the given model pool.

Examples:

>>> algorithm = SimpleAverageAlgorithm()
>>> modelpool = ModelPool()
>>> merged_model = algorithm.fuse(modelpool)

Parameters:

modelpool (_type_) –

description

Source code in fusion_bench/method/base_algorithm.py

@abstractmethod
def run(self, modelpool):
    """
    Fuse the models in the given model pool.

    Examples:
        >>> algorithm = SimpleAverageAlgorithm()
        >>> modelpool = ModelPool()
        >>> merged_model = algorithm.fuse(modelpool)

    Args:
        modelpool (_type_): _description_
    """
    pass

Example Usage¶

from ..method import load_algorithm_from_config
from ..modelpool import load_modelpool_from_config

def run_model_fusion(cfg: DictConfig):
    modelpool = load_modelpool_from_config(cfg.modelpool)
    algorithm = load_algorithm_from_config(cfg.method)
    merged_model = algorithm.run(modelpool)

    if hasattr(cfg, "taskpool") and cfg.taskpool is not None:
        taskpool = load_taskpool_from_config(cfg.taskpool)
        taskpool.evaluate(merged_model)
    else:
        print("No task pool specified. Skipping evaluation.")

In summary, the Fusion Algorithm module is vital for the model merging operations within FusionBench, leveraging sophisticated techniques to ensure optimal fusion and performance evaluation of deep learning models. This capability makes it an indispensable tool for researchers and practitioners focusing on model fusion strategies.

References¶

`load_algorithm_from_config(method_config)` ¶

Loads an algorithm based on the provided configuration.

The function checks the 'name' attribute of the configuration and returns an instance of the corresponding algorithm. If the 'name' attribute is not found or does not match any known algorithm names, a ValueError is raised.

Parameters:

method_config (DictConfig) –

The configuration for the algorithm. Must contain a 'name' attribute that specifies the type of the algorithm.

Returns:

–

An instance of the specified algorithm.

Raises:

ValueError –

If 'name' attribute is not found in the configuration or does not match any known algorithm names.

Source code in fusion_bench/method/__init__.py

def load_algorithm_from_config(method_config: DictConfig):
    """
    Loads an algorithm based on the provided configuration.

    The function checks the 'name' attribute of the configuration and returns an instance of the corresponding algorithm.
    If the 'name' attribute is not found or does not match any known algorithm names, a ValueError is raised.

    Args:
        method_config (DictConfig): The configuration for the algorithm. Must contain a 'name' attribute that specifies the type of the algorithm.

    Returns:
        An instance of the specified algorithm.

    Raises:
        ValueError: If 'name' attribute is not found in the configuration or does not match any known algorithm names.
    """
    return AlgorithmFactory.create_algorithm(method_config)

Introduction to Algorithm Module¶

Key Points of the Fusion Algorithm Module¶