Structured Config Groups¶

Due to the modular design of FusionBench, it employs a powerful structured configuration system based on Hydra, which allows for modular, composable, and hierarchical configuration management. This document explains how to understand and use the structured config groups in FusionBench.

Overview¶

The configuration system is organized into several key groups, each serving a specific purpose in the model fusion pipeline:

Method: Defines the fusion algorithm and its hyperparameters
ModelPool: Manages the models involved in the fusion process
TaskPool: Specifies evaluation tasks and datasets
Hydra: Controls Hydra framework settings
Fabric: PyTorch Lightning Fabric configurations

Configuration Directory Structure¶

config/
├── method/            # Fusion algorithms
├── modelpool/         # Model pool configurations
├── taskpool/          # Task pool configurations
├── hydra/             # Hydra framework settings
├── fabric/            # Fabric configurations
└── *.yaml             # Top-level configuration files

Core Configuration Groups¶

1. Method Configuration (`method/`)¶

The method configuration defines which fusion algorithm to use and its hyperparameters. Each method configuration file specifies:

_target_: The Python class implementing the algorithm
Algorithm-specific parameters

Example: Simple Average (hyperparameter-free)

_target_: fusion_bench.method.SimpleAverageAlgorithm

Example: Task Arithmetic

_target_: fusion_bench.method.TaskArithmeticAlgorithm
scaling_factor: 0.3

2. ModelPool Configuration (`modelpool/`)¶

ModelPool configurations define the collection of models to be fused, including:

Base/pretrained models
Fine-tuned models for specific tasks
Model preprocessors and processors

Example: CLIP Vision Model Pool

_target_: fusion_bench.modelpool.CLIPVisionModelPool
_recursive_: False
processor: openai/clip-vit-base-patch32
models:
  _pretrained_: openai/clip-vit-base-patch32
  sun397: tanganke/clip-vit-base-patch32_sun397
  stanford-cars: tanganke/clip-vit-base-patch32_stanford-cars
platform: hf

3. TaskPool Configuration (`taskpool/`)¶

TaskPool configurations specify the evaluation tasks and their datasets:

Test datasets for evaluation
Task-specific processors

Example: CLIP Vision Task Pool

_target_: fusion_bench.taskpool.CLIPVisionModelTaskPool
test_datasets:
  sun397:
    _target_: datasets.load_dataset
    path: tanganke/sun397
    split: test
  stanford-cars:
    _target_: datasets.load_dataset
    path: tanganke/stanford_cars
    split: test
clip_model: openai/clip-vit-base-patch32
processor: openai/clip-vit-base-patch32

5. Hydra Configuration (`hydra/`)¶

Controls Hydra framework behavior:

config/hydra/default.yaml

defaults:
  - override help: fusion_bench_help
  - override job_logging: rich_logging
run:
  dir: outputs/${hydra.job.name}/${now:%Y-%m-%d_%H-%M-%S}
sweep:
  dir: outputs/${hydra.job.name}/${now:%Y-%m-%d_%H-%M-%S}
  subdir: ${hydra.job.num}

Complete Configuration Example¶

Here's a complete configuration that combines all groups:

config/_get_started/clip_task_arithmetic.yaml
_target_: fusion_bench.programs.FabricModelFusionProgram
_recursive_: false
method:
  _target_: fusion_bench.method.TaskArithmeticAlgorithm
  scaling_factor: 0.7
modelpool:
  _target_: fusion_bench.modelpool.CLIPVisionModelPool
  models:
    _pretrained_: openai/clip-vit-base-patch32
    sun397: tanganke/clip-vit-base-patch32_sun397
    stanford-cars: tanganke/clip-vit-base-patch32_stanford-cars
taskpool:
  _target_: fusion_bench.taskpool.CLIPVisionModelTaskPool
  test_datasets:
    sun397:
      _target_: datasets.load_dataset
      path: tanganke/sun397
      split: test
    stanford-cars:
      _target_: datasets.load_dataset
      path: tanganke/stanford_cars
      split: test
  clip_model: openai/clip-vit-base-patch32
  processor: openai/clip-vit-base-patch32

Using Structured Configs¶

1. With Hydra Defaults¶

You can use Hydra's defaults system to compose configurations:

defaults:
  - method: simple_average
  - modelpool: CLIPVisionModelPool/clip-vit-base-patch32_TA8
  - taskpool: CLIPVisionModelTaskPool/clip-vit-classification_TA8.yaml
  - _self_

_target_: fusion_bench.programs.FabricModelFusionProgram

2. Command Line Overrides¶

Override configuration values from the command line:

fusion_bench method=task_arithmetic method.scaling_factor=0.5

3. Config Groups¶

Specify different config groups:

fusion_bench \
  method=adamerging \
  modelpool=CLIPVisionModelPool/clip-vit-base-patch32_TA8 \
  taskpool=clip-vit-base-patch32_robustness_corrupted

Advanced Features¶

Recursive Configuration¶

Control recursive instantiation with _recursive_:

_target_: fusion_bench.modelpool.CLIPVisionModelPool
_recursive_: False  # Prevent recursive instantiation

Conditional Configurations¶

Leverage Hydra's conditional configuration syntax to handle advanced scenarios, such as dynamically selecting configurations based on specific conditions:

defaults:
  - /dataset/image_classification/train@train_datasets:
      - sun397
      - stanford-cars
      - resisc45

This structured approach makes FusionBench configurations highly flexible, maintainable, and easy to experiment with different combinations of methods, models, and tasks.