Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning

Official training and analysis code for Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning.

Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang

[Paper] [Project page] [Data] [Models]

Overview

Vision-Flan is a human-labeled visual instruction-tuning dataset built from academic vision datasets. The paper studies a two-stage tuning framework:

1. Capability learning: fine-tune a pretrained LLaVA architecture on Vision-Flan to obtain Vision-Flan Base.
2. Human-preference alignment: further tune Vision-Flan Base on a small sample of GPT-4-synthesized LLaVA data to obtain Vision-Flan Chat.

The paper and the linked Hugging Face release contain 187 tasks and 1,664,261 instances. The original project website also documents an earlier 191-task, approximately 186K-instance subset with at most 1K examples per task; use the full Hugging Face release when reproducing the paper's full-data experiments.

Paper configuration

The main scripts retain the configuration used for the paper.

Component	Paper setting	Repository
Language model	Vicuna-13B v1.5	lmsys/vicuna-13b-v1.5
Vision encoder	CLIP ViT-L/14 at 336px	openai/clip-vit-large-patch14-336
Connector	Two-layer MLP	--mm_projector_type mlp2x_gelu
Trainable modules	LLM and MLP connector; vision encoder frozen	LLaVA training defaults used by the scripts
Stage 1	1 epoch, learning rate 2e-5, per-device batch size 16, 8 A100 GPUs	scripts/v1_5/finetune.sh
Stage 2	1K LLaVA examples, learning rate 1e-5, per-device batch size 8, 8 GPUs, 128 optimization steps	scripts/v1_5/human_align.sh; the original run used 8 epochs over the 1K-example set, equivalent to about 128 steps at global batch size 64

Both stages tune the language model and MLP connector. The stage-2 entry point, llava/train/2dn_train.py, loads the complete stage-1 multimodal checkpoint rather than reinitializing its connector.

Setup

This repository is based on the LLaVA v1.5 codebase and uses its environment. A typical setup is:

conda create -n vision-flan python=3.10 -y
conda activate vision-flan
pip install --upgrade pip
pip install -e .
pip install ninja
pip install flash-attn --no-build-isolation

For the dependency versions associated with this LLaVA snapshot, consult the upstream LLaVA installation documentation. Access to Vicuna weights is governed by the corresponding model license.

Data preparation

Download the full paper-scale data from the Vision-Flan dataset on Hugging Face. The original model links are collected on the Vision-Flan project website. Training annotations use the standard LLaVA conversation format and refer to images through their image fields.

Before running a script, replace its descriptive placeholders with local values:

• Your_VISION_FLAN_PATH: Vision-Flan annotations and images.
• Your_LLAVA_DATA_PATH: LLaVA instruction annotations and COCO images.
• Your_LLAVA_PATH: this repository and its checkpoints.
• Your_EVALUATION_PATH: prepared benchmark annotations and images.
• Your_TRANSFORMERS_CACHE_PATH: model cache.
• Your_CONDA_PATH: Conda installation.
• Your_SLURM_PARTITION and Your_SLURM_ACCOUNT: cluster scheduling configuration.
• Your_LLAVA_CONDA_ENV, Your_EVALUATION_CONDA_ENV, and Your_METRICS_CONDA_ENV: task-specific Conda environments.
• Your_SSH_KEY_PATH and Your_REMOTE_*_PATH: optional cluster-transfer settings in archival launch notes.

The scripts include several dataset names used for the ablations in the paper. Their filenames describe the corresponding subset or mixture. The canonical full-data stage-1 entry is vision-flan-scale-v6; llava_1k is the canonical stage-2 alignment set.

Training

Stage 1: Vision-Flan Base

The paper initializes from the LLaVA v1.5 pretrained projector, before LLaVA visual instruction tuning. Set the Vision-Flan and projector placeholders in scripts/v1_5/finetune.sh, then run:

bash scripts/v1_5/finetune.sh vision-flan-scale-v6

On a Slurm cluster, submit.sh is the corresponding eight-GPU launcher after its cluster placeholders are configured.

Stage 2: Vision-Flan Chat

Prepare a random 1K-example subset of the GPT-4-synthesized LLaVA instruction data, configure its annotation and COCO image paths in scripts/v1_5/human_align.sh, and run:

bash scripts/v1_5/human_align.sh \
  llava_1k \
  Your_VISION_FLAN_BASE_PATH \
  vision-flan-base \
  8

With 8 GPUs and per-device batch size 8, the original 8-epoch command over 1K examples gives approximately the 128 optimization steps reported in the paper.

Evaluation

The paper reports results on:

• comprehensive benchmarks: MM-Bench, MME, and MMMU;
• free-form generation: LLaVA-Bench and MM-Vet;
• hallucination: POPE;
• catastrophic forgetting: CIFAR-10, CIFAR-100, MNIST, and miniImageNet.

The upstream LLaVA-compatible evaluators are under scripts/v1_5/eval, and the paper-specific unified prediction utilities are under llava/instruct_eval. Configure Your_EVALUATION_PATH before use. Follow each benchmark's official evaluator for final scoring, as done in the paper. GPT-assisted benchmarks require an OPENAI_API_KEY supplied through the environment; no API credential is stored in this repository.

Additional analysis code for module freezing, connector/LLM correlation, information flow, task-count ablations, and LLM replacement is under analysis.

Results reported in the paper

Model	MM-Bench	MME	MMMU	LLaVA-Bench	MM-Vet	POPE	CF
Vision-Flan Base	69.8	1537.8	34.4	38.5	33.4	85.9	87.2
Vision-Flan Chat	67.6	1490.6	34.3	78.3	38.0	86.1	84.0

CF is the average performance on CIFAR-10, CIFAR-100, MNIST, and miniImageNet. These are the paper's full-data results.

Acknowledgements

This codebase is built directly on LLaVA, including its model architecture, training framework, serving code, and many evaluation utilities. We thank Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee, and the broader LLaVA community for making that work available. The implementation also depends on Vicuna/FastChat, CLIP, Hugging Face Transformers, DeepSpeed, and the official tools and datasets of the evaluation benchmarks.

Vision-Flan also builds on the lessons and task resources developed in MultiInstruct. We thank all Vision-Flan annotators and the creators of the 101 source datasets; please consult the project page for dataset-specific references and licenses.

The retained upstream documents in docs describe LLaVA functionality. When their generic LLaVA recipes differ from this page, use this README for the Vision-Flan paper configuration.

Citation

If this repository or dataset is useful in your research, please cite Vision-Flan:

@article{xu2024vision,
  title={Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning},
  author={Xu, Zhiyang and Feng, Chao and Shao, Rulin and Ashby, Trevor and Shen, Ying and Jin, Di and Cheng, Yu and Wang, Qifan and Huang, Lifu},
  journal={arXiv preprint arXiv:2402.11690},
  year={2024}
}

Please also cite the LLaVA works on which this implementation is based:

@article{liu2023improvedllava,
  title={Improved Baselines with Visual Instruction Tuning},
  author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae},
  journal={arXiv preprint arXiv:2310.03744},
  year={2023}
}

@inproceedings{liu2023llava,
  title={Visual Instruction Tuning},
  author={Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},
  booktitle={Advances in Neural Information Processing Systems},
  year={2023}
}

License

The code retains the upstream Apache 2.0 license. Dataset examples, source images, Vicuna weights, and evaluation assets remain subject to their respective licenses and terms. Vision-Flan is intended for research use; verify the license of every source dataset before redistributing or using its contents.