A Modified Stochastic Gradient Descent Optimization Algorithm With Random Learning Rate for Machine Learning and Deep Learning
- Authors
- Shim, Duk-Sun; Shim, Joseph
- Issue Date
- Nov-2023
- Publisher
- Institute of Control, Robotics and Systems
- Keywords
- Deep learning; machine learning; modified stochastic gradient descent; random learning rate; steepest descent algorithm
- Citation
- International Journal of Control, Automation and Systems, v.21, no.11, pp 3825 - 3831
- Pages
- 7
- Journal Title
- International Journal of Control, Automation and Systems
- Volume
- 21
- Number
- 11
- Start Page
- 3825
- End Page
- 3831
- URI
- https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/68183
- DOI
- 10.1007/s12555-022-0947-1
- ISSN
- 1598-6446
2005-4092
- Abstract
- An optimization algorithm is essential for minimizing loss (or objective) functions in machine learning and deep learning. Optimization algorithms face several challenges, one among which is to determine an appropriate learning rate. Generally, a low learning rate leads to slow convergence whereas a large learning rate causes the loss function to fluctuate around the minimum. As a hyper-parameter, the learning rate is determined in advance before parameter training, which is time-consuming. This paper proposes a modified stochastic gradient descent (mSGD) algorithm that uses a random learning rate. Random numbers are generated for a learning rate at every iteration, and the one that gives the minimum value of the loss function is chosen. The proposed mSGD algorithm can reduce the time required for determining the learning rate. In fact, the k-point mSGD algorithm can be considered as a kind of steepest descent algorithm. In a real experiment using the MNIST dataset of hand-written digits, it is demonstrated that the convergence performance of mSGD algorithm is much better than that of the SGD algorithm and slightly better than that of the AdaGrad and Adam algorithms. © 2023, ICROS, KIEE and Springer.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - College of ICT Engineering > School of Electrical and Electronics Engineering > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/68183)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.