BitDelta applies 1-bit quantization to the weight delta between fine-tuned and base models. For each weight matrix, we quantize its delta as its sign bits and a trainable high-precision scale factor. The scale factor is initialized to achieve the best approximation error in L2 norm and further refined with a few distillation steps. BitDelta shows minimal degradation in model performance and reduces memory consumption in multi-tenancy serving by representing multiple fine-tuned models with a single high-precision base model and multiple 1-bit deltas.
_target_:fusion_bench.method.bitdelta.BitDeltaAlgorithmsave_dir:nullsave_full_model:false# training argumentslr:1e-4batch_size:4num_steps:100# dataset argumentsdataset_name:c4subset:ensplit:trainmax_length:128
