Introduction​

We propose Octavius, a unified, multimodal large language with a novel capability to comprehend various tasks across different modalities, including but not limited to 2D captioning, 2D detection, 3D VQA, and 3D dense captioning. Through combining well-known Mixture-of-Experts (MoE) and one of the representative PEFT techniques, i.e., LoRA, Octavius can efficiently be involved in more downstream tasks and more modalities by learning more LoRA modules, alleviating the potential task interference issuse arise from multimodal learning.

Usage​

1. Environment installation.

2. Prepare the instruction / benchmark dataset and required pretrained weights for LLMs and visual encoder.

3. Training scripts:

   • Image modality only

     cd src
     sh tools/Octavius/train_octavius_slurm.sh <YOUR_PARTITION>  <NUM_GPU> \
         config/Octavius/octavius_2d_e4_bs64.yaml octavius_2d_e4_bs64
     

   • Point cloud modality only

     cd src
     sh tools/Octavius/train_octavius_slurm.sh <YOUR_PARTITION>  <NUM_GPU> \
         config/Octavius/octavius_3d_e3_bs64.yaml octavius_3d_e3_bs64
     

   • Image & point cloud modality joint

     cd src
     sh tools/Octavius/train_octavius_slurm.sh <YOUR_PARTITION>  <NUM_GPU> \
         config/Octavius/octavius_2d+3d_e6_bs64.yaml octavius_2d +3d_e6_bs64
     

   We provide pretrained Octavius model here.

4. Evaluation

   We use ChEF to evaluate Octavius on both image and point cloud modalities, see here for details.

Citation​

@misc{chen2023octavius,
      title={Octavius: Mitigating Task Interference in MLLMs via MoE}, 
      author={Zeren Chen and Ziqin Wang and Zhen Wang and Huayang Liu and Zhenfei Yin and Si Liu and Lu Sheng and Wanli Ouyang and Yu Qiao and Jing Shao},
      year={2023},
      eprint={2311.02684},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License​

The project is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.