ECCV Conference (2022)
Juseung Yun(KAIST, LG AI Research), Janghyeon Lee, Hyounguk Shon(KAIST, LG AI Research), Eojindl Yi(KAIST, LG AI Research), Seunghwan Kim, Junmo Kim(KAIST)
Abstract
If a network contains such normalization layers, the optimization objective is invariant to the scale of the neural network parameters. The scale-invariance induces the output of the neural network to be only affected by the direction of the weights and not the scale of the weights. We first attempt to find a common feature of good hyperparameter combinations, including learning rate, weight decay, number of data samples, and batch size, on such a scale-invariant network. Then we observe that hyperparameter setups which lead to good performance show similar degrees of angular update during one epoch. Inspired by this novel observation, we analyze the angular update using a stochastic differential equation and show how each hyperparameter is related to the angular update. On a scale-invariant neural network, we demonstrate that this relation can be applied to derive a simple hyperparameter tuning rule.