Toggle Main Menu Toggle Search

Open Access padlockePrints

Sensitivity Analysis for Deep Learning: Ranking Hyper-parameter Influence

Lookup NU author(s): Dr Varun OjhaORCiD

Downloads

Full text for this publication is not currently held within this repository. Alternative links are provided below where available.


Abstract

© 2021 IEEE.(DL). We present a novel approach to rank Deep Learning hyper-parameters through the application of Sensitivity Analysis (SA). DL hyper-parameter tuning is crucial to model accuracy however, choosing optimal values for each parameter is time and resource-intensive. SA provides a quantitative measure by which hyper-parameters can be ranked in terms of contribution to model accuracy. Learning rate decay was ranked highest, with model performance being sensitive to this parameter regardless of architecture or dataset. The influence of a model's initial learning rate was proven to be low, contrary to the literature. Additionally, the importance of a parameter is closely linked to model architecture. Shallower models showed susceptibility to hyper-parameters affecting the stochasticity of the learning process whereas deeper models showed sensitivity to hyper-parameters affecting the convergence speed. Furthermore, the complexity of the dataset can affect the margin of separation between the sensitivity measures of the most and the least influential parameters, making the most influential hyper-parameter an ideal candidate for tuning compared to the other parameters.


Publication metadata

Author(s): Taylor R, Ojha V, Martino I, Nicosia G

Publication type: Conference Proceedings (inc. Abstract)

Publication status: Published

Conference Name: 33rd International Conference on Tools with Artificial Intelligence (ICTAI 2021)

Year of Conference: 2021

Pages: 512-516

Online publication date: 21/12/2021

Acceptance date: 02/04/2018

ISSN: 2375-0197

Publisher: IEEE

URL: https://doi.org/10.1109/ICTAI52525.2021.00083

DOI: 10.1109/ICTAI52525.2021.00083

Library holdings: Search Newcastle University Library for this item

ISBN: 9781665408981


Share