Skip to content

RegMean

Code Integration

Merge CLIP-ViT-B/32 models on eight image classification tasks

fusion_bench method=clip_regmean \
  modelpool=clip-vit-base-patch32_TA8 \
  taskpool=clip-vit-classification_TA8

Merge CLIP-ViT-L/14 models on eight image classification tasks

fusion_bench \
  method=clip_regmean \
    method.batch_size=8 method.num_workers=4 \
  modelpool=clip-vit-large-patch14_TA8 \
  taskpool=clip-vit-classification_TA8 \
    taskpool.clip_model=openai/clip-vit-large-patch14

Merge GPT-2 models for text classification tasks:

fusion_bench \
  method=gpt2_regmean \
  modelpool=gpt-2_glue \
  taskpool=gpt-2_glue

References

  1. Xisen Jin, et al. "Dataless Knowledge Fusion by Merging Weights of Language Models". http://arxiv.org/abs/2212.09849 ↩