VIP-ANT: VIsually-Pivoted Audio and(N) Text

Code for the paper Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer @ NAACL 2022.

Data

AudioSet can be downloaded and preprocessed via this tool.

AudioSet Data

See AudioSet. It elaborates on our customized index files for pre-training on AudioSet.

Curated Audio-Text Data

See AudioTxt. It elaborates on our curation methods and customized index files for audio-text fine-tuning.

Vision-Audio (VA) Pre-training

Check out the running script bash/run_bimodal_va.sh.

Audio-Text (AT) Fine-tuning

Check out the running script bash/run_bimodal_at.sh. Fine-tuning starts with a VA pre-trained audio encoder.

Pre-trained Models

We provide a checkpoint that performs best for each task.

Pre-trained Models for Audio-Text Retrieval

Model	AudioCaps	Clotho (18s)	Clotho (10s)
VIP-ANT	00051623	00043681	00043681
+AT w/ GC	00006210	00006900	00004140

Pre-trained Models for Zero-shot Audio Classification

Model	ESC50 (w/ prompt)	US8K (w/ prompt)
VIP-ANT	00083391	00079420
+AT w/ GC	00004140	00004140

Dependencies

Dockerfile defines minimum dependencies of the repo.

License

MIT