Modality-Specialized Synergizers for Interleaved Vision-Language Generalists

This repository contains the Emu2 implementation of Modality-Specialized Synergizers (MoSS), accepted at ICLR 2025.

[Paper] [arXiv] [OpenReview]

Zhiyang Xu*, Minqian Liu*, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, and Lifu Huang *Equal contribution.

MoSS uses modality-specialized low-rank adapters in an autoregressive vision-language generalist:

• a Linear LoRA path models sequential text hidden states;
• a Convolutional LoRA path models the 2D local structure of image hidden states.

For Emu2, 64 image embeddings are reshaped to an 8×8 grid. The image adapter applies a causal 2×2 convolution, while text positions use a separate Linear LoRA. This repository retains the original research implementation and experiment files. The paper's Chameleon implementation is not included.

Code overview

The core paper implementation is located in:

emu2_code/multimodal_encoder/modeling_llama.py  Linear, MM-Linear, and ConvLoRA layers
emu2_code/multimodal_encoder/modeling_emu.py    Emu2 multimodal training and generation
llava/train/train_emu2.py                       Interleaved instruction tuning
llava/train/train_mem.py                        Memory-efficient training entry point
scripts/v1_5/finetune_emu2.sh                  Original parameterized Emu2 launcher
inference_emu2.py                               InterleavedBench generation
inference_magicbrush.py                         Image-editing generation
eval_geneval.py                                 GenEval generation
eval_mscoco.py                                  MS-COCO generation

The adapter implementation names are:

Paper configuration	lora_type	Implementation
LoRA	linear	One shared Linear LoRA
MoE-LoRA	mm_linear	Separate image and text Linear LoRAs
MoSS	mm_conv	Image ConvLoRA and text Linear LoRA

The research repository also retains earlier LLaVA/Emu experiments, analysis utilities, and evaluation code to preserve the original implementation history.

Installation

The original environment used Python 3.10, PyTorch 2.1.1, Transformers 4.31.0, DeepSpeed 0.9.5, and eight A100 GPUs.

conda create -n moss python=3.10 -y
conda activate moss
pip install --upgrade pip
pip install -e .

# Required by the retained Emu2 vision encoder and training launcher.
pip install xformers
pip install flash-attn --no-build-isolation

CUDA, PyTorch, xFormers, FlashAttention, and DeepSpeed versions must be mutually compatible. Emu2 is a 37B model and requires substantial GPU memory.

Prepare Emu2-Gen

Download the official BAAI/Emu2-Gen checkpoint outside this repository:

huggingface-cli download BAAI/Emu2-Gen \
  --local-dir checkpoints/Emu2-Gen

Back up the checkpoint's original remote-code files, then copy the MoSS-modified files into its multimodal_encoder directory:

cp -r checkpoints/Emu2-Gen/multimodal_encoder \
  checkpoints/Emu2-Gen/multimodal_encoder.upstream-backup

cp emu2_code/multimodal_encoder/configuration_emu.py checkpoints/Emu2-Gen/multimodal_encoder/
cp emu2_code/multimodal_encoder/constants.py checkpoints/Emu2-Gen/multimodal_encoder/
cp emu2_code/multimodal_encoder/modeling_emu.py checkpoints/Emu2-Gen/multimodal_encoder/
cp emu2_code/multimodal_encoder/modeling_llama.py checkpoints/Emu2-Gen/multimodal_encoder/
cp emu2_code/multimodal_encoder/visual.py checkpoints/Emu2-Gen/multimodal_encoder/

Replace YOUR_EMU2_MODEL_PATH in the launch scripts with the resulting checkpoint directory.

Data

Training uses a JSON list in the LLaVA conversation format. Image paths are interpreted relative to YOUR_IMAGE_FOLDER_PATH and follow the order of the <image> placeholders.

[
  {
    "id": "example-0001",
    "image": ["context.jpg", "target.jpg"],
    "conversations": [
      {
        "from": "human",
        "value": "Continue the story from this image: <image>"
      },
      {
        "from": "gpt",
        "value": "The next scene is: <image>"
      }
    ]
  }
]

LeafInstruct contains 184,982 interleaved instruction-tuning instances. Obtain the released annotations and images separately and follow the licenses of their constituent sources.

Configure public paths

Private machine paths have been replaced in place without removing the original scripts. Before running an experiment, replace the relevant placeholders:

Placeholder	Meaning
YOUR_EMU2_MODEL_PATH	Prepared Emu2-Gen checkpoint
YOUR_DATASET_PATH	Root containing training/evaluation datasets
YOUR_IMAGE_FOLDER_PATH	Root used to resolve training images
YOUR_CHECKPOINT_PATH	Training outputs and fine-tuned checkpoints
YOUR_REPO_PATH	Local repository checkout, where required by legacy scripts
YOUR_SLURM_ACCOUNT, YOUR_SLURM_PARTITION	Optional cluster settings

Additional YOUR_... placeholders occur only in retained legacy or analysis scripts and are named according to their purpose. Locate all of them with:

rg -n "YOUR_[A-Z0-9_]+" .

Training

After configuring scripts/v1_5/finetune_emu2.sh, run the three paper configurations:

# Shared Linear LoRA
bash scripts/v1_5/run_paper_lora.sh final

# Modality-specific Linear LoRAs
bash scripts/v1_5/run_paper_moe_lora.sh final

# MoSS: image ConvLoRA plus text Linear LoRA
bash scripts/v1_5/run_paper_moss.sh final

The wrappers call the original launcher without changing the training implementation. The retained launcher uses eight GPUs, one epoch, per-device batch size 1, learning rate 2e-5, LoRA rank 128, dropout 0.05, and gradient accumulation 8. In the original training code, lora_alpha is set to 2 × rank.

Other original presets (v1, v1_text, v2, v2_text, overfit, hq_edit, and blip_laion) remain available through:

bash scripts/v1_5/finetune_emu2.sh mm_conv YOUR_DATA_PRESET

Inference and evaluation

The original inference and benchmark scripts are retained. Replace their YOUR_... dataset, model, output, and device settings before execution.

InterleavedBench generation:

python inference_emu2.py YOUR_FINETUNED_MODEL_PATH

MagicBrush image editing:

python inference_magicbrush.py YOUR_FINETUNED_MODEL_PATH

GenEval and MS-COCO generation are implemented in eval_geneval.py and eval_mscoco.py. These scripts generate benchmark outputs; use the official benchmark repositories to compute final metrics.

Acknowledgements

This codebase is built on and retains code from several open-source projects:

• Emu and Emu2, including the Emu2-Gen multimodal encoder and diffusion pipeline.
• LLaVA, which provides the repository foundation, conversation preprocessing, training infrastructure, and evaluation utilities.
• FastChat and Stanford Alpaca, from which portions of the retained training pipeline were adapted.
• Hugging Face Transformers, which provides the LLaMA implementation and model-loading infrastructure.
• Microsoft's BEiT/UNILM and timm, on which the retained EVA vision-transformer code is based.
• PEFT, DeepSpeed, Diffusers, xFormers, and FlashAttention.

See NOTICE.md for detailed provenance and licensing boundaries. We thank all upstream authors and maintainers.

Citation

@inproceedings{xu2025modality,
  title={Modality-Specialized Synergizers for Interleaved Vision-Language Generalists},
  author={Xu, Zhiyang and Liu, Minqian and Shen, Ying and Rimchala, Joy and Zhang, Jiaxin and Wang, Qifan and Cheng, Yu and Huang, Lifu},
  booktitle={International Conference on Learning Representations},
  year={2025}
}

License

The repository retains the Apache License 2.0 used by its LLaVA-derived code; see LICENSE. This license applies only where the repository's contributors and upstream licensors have authority to grant it. It does not relicense Emu2/Emu2-Gen weights, LLaMA weights, SDXL components, benchmark data, LeafInstruct source data, or any other third-party artifact.

Users must obtain external models and datasets from their official distributors and comply with all corresponding licenses, access conditions, and acceptable-use policies. Files retaining upstream copyright or license headers remain governed by those notices.