Weakly Supervised Representation Learning for Audio-Visual Scene Analysis

Abstract : Audiovisual (AV) representation learning is an important task from the perspective of designing machines with the ability to understand complex events. To this end, we propose a novel multimodal framework that instantiates multiple instance learning. Specifically, we develop methods that identify events and localize corresponding AV cues in unconstrained videos. Importantly, this is done using weak labels where only video-level event labels are known without any information about their location in time. We show that the learnt representations are useful for performing several tasks such as event/object classification, audio event detection, audio source separation and visual object localization. An important feature of our method is its capacity to learn from unsynchronized audiovisual events. We also demonstrate our framework's ability to separate out the audio source of interest through a novel use of nonnegative matrix factorization. State-of-the-art classification results, with a F1-score of 65.0, are achieved on DCASE 2017 smart cars challenge data with promising generalization to diverse object types such as musical instruments. Visualizations of localized visual regions and audio segments substantiate our system's efficacy, especially when dealing with noisy situations where modality-specific cues appear asynchronously.
Document type :
Journal articles
Complete list of metadatas

Cited literature [78 references]  Display  Hide  Download

https://hal.telecom-paristech.fr/hal-02399993
Contributor : Gaël Richard <>
Submitted on : Monday, December 9, 2019 - 12:28:33 PM
Last modification on : Saturday, February 1, 2020 - 1:54:55 AM

File

2019-IEEE_TASLP_Parekh.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02399993, version 1

Collections

Citation

Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc Duong, Patrick Pérez, et al.. Weakly Supervised Representation Learning for Audio-Visual Scene Analysis. IEEE/ACM Transactions on Audio, Speech and Language Processing, Institute of Electrical and Electronics Engineers, 2019. ⟨hal-02399993⟩

Share

Metrics

Record views

42

Files downloads

107