Toggle Main Menu Toggle Search

Open Access padlockePrints

Robust Multi-Speaker Tracking via Dictionary Learning and Identity Modeling

Lookup NU author(s): Dr Mohsen Naqvi, Professor Jonathon Chambers


Full text for this publication is not currently held within this repository. Alternative links are provided below where available.


We investigate the problem of visual tracking of multiple human speakers in an office environment. In particular, we propose novel solutions to the following challenges: (1) robust and computationally efficient modeling and classification of the changing appearance of the speakers in a variety of different lighting conditions and camera resolutions; (2) dealing with full or partial occlusions when multiple speakers cross or come into very close proximity; (3) automatic initialization of the trackers, or re-initialization when the trackers have lost lock caused by e.g. the limited camera views. First, we develop new algorithms for appearance modeling of the moving speakers based on dictionary learning (DL), using an off-line training process. In the tracking phase, the histograms (coding coefficients) of the image patches derived from the learned dictionaries are used to generate the likelihood functions based on Support Vector Machine (SVM) classification. This likelihood function is then used in the measurement step of the classical particle filtering (PF) algorithm. To improve the computational efficiency of generating the histograms, a soft voting technique based on approximate Locality-constrained Soft Assignment (LcSA) is proposed to reduce the number of dictionary atoms (codewords) used for histogram encoding. Second, an adaptive identity model is proposed to track multiple speakers whilst dealing with occlusions. This model is updated online using Maximum a Posteriori (MAP) adaptation, where we control the adaptation rate using the spatial relationship between the subjects. Third, to enable automatic initialization of the visual trackers, we exploit audio information, the Direction of Arrival (DOA) angle, derived from microphone array recordings. Such information provides, a priori, the number of speakers and constrains the search space for the speaker's faces. The proposed system is tested on a number of sequences from three publicly available and challeng- ng data corpora (AV16.3, EPFL pedestrian data set and CLEAR) with up to five moving subjects.

Publication metadata

Author(s): Barnard M, Koniusz P, Wang W, Kittler J, Naqvi SM, Chambers JA

Publication type: Article

Publication status: Published

Journal: IEEE Transactions on Multimedia

Year: 2014

Volume: 16

Issue: 3

Pages: 864-880

Print publication date: 01/04/2014

Online publication date: 22/01/2014

Acceptance date: 27/11/2013

ISSN (print): 1520-9210

ISSN (electronic): 1941-0077

Publisher: IEEE


DOI: 10.1109/TMM.2014.2301977


Altmetrics provided by Altmetric