Tutorial

Our code is here.

Quickstart

Learn how to prepare the dataset, model, environment, and start training and evaluation.

Datasets

Prepare dataset for instruction tuning and benchmark.

Installation

Prepare the environment.

Training

Train MLLMs with provided scripts.

Benchmark

Evaluate MLLMs with LAMM-Benchmark or ChEF.

Once you have successfully configured the environment, the entire directory structure should be as follows:

LAMM
├── ckpt 
│   ├── lamm_2d               # saved checkpoints in training
│   └── ...
├── data                      # dataset folder, see `Dataset Preparation` section for detail
│   ├── LAMM                  # LAMM dataset
│   ├── Octavius              # Octavius dataset
│   ├── ChEF                  # ChEF dataset  
│   └── ...                   # your custom dataset
├── docs                      # document
├── images                    # readme assets
├── model_zoo                 # see `Model Preparation for Training` for detail 
│   ├── vicuna_ckpt           # Vicuna-7B/13B
│   ├── epcl_vit-L_256tokens  # EPCL pretraining checkpoints (Optional)
│   └── ...
├── requirements              # python environment requirements 
├── src
└── ...