Use Builtin Datasets¶
A dataset can be used by wrapping it into a torch Dataset. This document explains how to setup the builtin datasets so they can be used by X-modaler. The annotations for builtin datasets can be downloaded here.
X-modaler has builtin supports for a few datasets (e.g., MSCOCO or MSVD). The corresponding dataset wrappers are provided in xmodaler/datasets
:
xmodaler/datasets/
images/
mscoco.py
videos/
msvd.py
You can specify which dataset wrapper to use by DATASETS.TRAIN
, DATASETS.VAL
and DATASETS.TEST
in the config file.
Expected dataset structure for COCO:¶
mscoco_dataset/
mscoco_caption_anno_train.pkl
mscoco_caption_anno_val.pkl
mscoco_caption_anno_test.pkl
vocabulary.txt
captions_val5k.json
captions_test5k.json
# image files that are mentioned in the corresponding json
features/
up_down/
*.npz
Expected dataset structure for MSVD:¶
msvd_dataset/
msvd_caption_anno_train.pkl
msvd_caption_anno_val.pkl
msvd_caption_anno_test.pkl
vocabulary.txt
captions_val.json
captions_test.json
# videos files that are mentioned in the corresponding json
features/
resnet152/
*.npy
Expected dataset structure for MSR-VTT:¶
msrvtt_dataset/
msrvtt_caption_anno_train.pkl
msrvtt_caption_anno_val.pkl
msrvtt_caption_anno_test.pkl
vocabulary.txt
captions_val.json
captions_test.json
# videos files that are mentioned in the corresponding json
msrvtt_torch/
feature/
resnet152/
*.npy
When the dataset wrapper and data files are ready, you need to specify the corresponding paths to these data files in the config file. For example,
DATALOADER:
FEATS_FOLDER: '../open_source_dataset/mscoco_dataset/features/up_down' # feature folder
ANNO_FOLDER: '../open_source_dataset/mscoco_dataset' # annotation folders
INFERENCE:
VOCAB: '../open_source_dataset/mscoco_dataset/vocabulary.txt'
VAL_ANNFILE: '../open_source_dataset/mscoco_dataset/captions_val5k.json'
TEST_ANNFILE: '../open_source_dataset/mscoco_dataset/captions_test5k.json'