Research Article
FARKLI ÇAPRAZLAMA TEKNİKLERİ KULLANAN DAĞINIK ARAMA ALGORİTMASI İLE EVRİŞİMLİ SİNİR AĞLARINDA HİPER PARAMETRE OPTİMİZASYONU
Abstract
Günümüzde yapay zekâ uygulamaları hayatın her alanında kullanılmaktadır ve gün geçtikçe daha uygulanabilir öneriler ve sonuçlar sunar hale gelmiştir. Evrişimli Sinir Ağları (ESA), birçok gerçek dünya probleminde başarılı sonuçlar veren, son yıllarda etkili ve yoğun bir şekilde uygulanan yapay zekâ algoritmalarından biridir. ESA’lar genellikle görsel bilginin analiz edilmesinde kullanılmaktadır. Görsel bilgi, ESA’larda bulunan evrişim, aktivasyon, havuzlama ve tam bağlantılı katmanlardan geçirilerek analiz edilmektedir. ESA eğitiminde kullanılan veri setine ve karşılaşılan probleme göre çeşitli parametreler kullanılmaktadır. Bu çalışmada en yüksek doğruluk değerini veren hiper parametrelerin seçilebilmesi için ESA eğitiminde kullanılan parametreler ve ağ yapısının oluşturulmasında kullanılan katmanlar optimize edilmiştir. Hiper parametrelerden kanal sayısı, evrişimsel katman, minimum parti boyutu ve aktivasyon fonksiyonu için ayrık değerler, öğrenme oranı için sürekli değerler belirlenmiştir. Bu çalışmada hiper parametre optimizasyonunu gerçekleştirmek için Dağınık Arama (DA) ve Genetik Algoritmalar (GA) yöntemleri tercih edilmiştir. DA yöntemi, GA ile kıyaslandığında ESA'lar için uygun hiper parametre değerlerinin kolaylıkla belirlenmesini sağlamıştır ve daha yüksek doğruluk değeri elde edilmiştir. Çalışmada elde edilen en yüksek doğruluk değerleri GA yöntemi ile %88.76 iken DA yöntemi ile % 93.24’tür. Bu değer 16 kanal sayısı, 5 x 5 evrişimsel katman, 64 minimum parti boyutu, 0.0052 öğrenme oranı ve reluLayer aktivasyon fonksiyonu parametreleri ile elde edilmiştir.
References
- Ait Amou, M., Xia, K., Kamhi, S., & Mouhafid, M. (2022, March). A novel MRI diagnosis method for brain tumor classification based on CNN and Bayesian Optimization. In Healthcare (Vol. 10, No. 3, p. 494). MDPI.
- Andonie, R., & Florea, A.-C. (2020). Weighted random search for CNN hyperparameter optimization. arXiv preprint arXiv:2003.13300.
- Anguita, D., Ghio, A., Oneto, L., Parra, X., & Reyes-Ortiz, J. L. (2013). A public domain dataset for human activity recognition using smartphones. Paper presented at the Esann.
- Aslan, M. F., Sabanci, K., Durdu, A., & Unlersen, M. F. (2022). COVID-19 diagnosis using state-of-the-art CNN architecture features and Bayesian Optimization. Computers in biology and medicine, 142, 105244.
- Atteia, G., Abdel Samee, N., El-Kenawy, E. S. M., & Ibrahim, A. (2022). CNN-hyperparameter optimization for diabetic maculopathy diagnosis in optical coherence tomography and fundus retinography. Mathematics, 10(18), 3274.
- Bochinski, E., Senst, T., & Sikora, T. (2017). Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. Paper presented at the 2017 IEEE international conference on image processing (ICIP).
- Bozkurt, F., & Yağanoğlu, M. (2021). Derin evrişimli sinir ağları kullanarak akciğer X-Ray görüntülerinden COVID-19 tespiti. Veri Bilimi, 4(2), 1-8.
- Fujino, S., Mori, N., & Matsumoto, K. (2017). Deep convolutional networks for human sketches by means of the evolutionary deep learning. Paper presented at the 2017 joint 17th world congress of international fuzzy systems association and 9th international conference on soft computing and intelligent systems (IFSA-SCIS).
- Geng, J. C., Cui, Z., & Gu, X. S. (2016). Scatter search based particle swarm optimization algorithm for earliness/tardiness flowshop scheduling with uncertainty. International Journal of Automation and Computing, 13(3), 285-295.
- Glover, F. (1977). Heuristics for integer programming using surrogate constraints. Decision sciences, 8(1), 156-166.
- Gülcü, A., & Kuş, Z. (2019). A survey of hyper-parameter optimization methods in convolutional neural networks. Gazi Üniversitesi Fen Bilimleri Dergisi, 7(2), 503-522.
- Hao, W., Yizhou, W., Yaqin, L., & Zhili, S. (2020, December). The role of activation function in CNN. In 2020 2nd International Conference on Information Technology and Computer Application (ITCA) (pp. 429-432). IEEE.
- Huang, D. S., Wunsch, D. C., Levine, D. S., & Jo, K. H. (Eds.). (2008). Advanced Intelligent Computing Theories and Applications. With Aspects of Artificial Intelligence: Fourth International Conference on Intelligent Computing, ICIC 2008 Shanghai, China, September 15-18, 2008, Proceedings (Vol. 5227). Springer Science & Business Media.
- İnik, Ö. (2023). CNN hyper-parameter optimization for environmental sound classification. Applied Acoustics, 202, 109168.
- Kıymaç, M. E. (2022). Hyper-parameter optimization of deep neural networks with metaheuristic algorithms. Yüksek Lisans Tezi, Alparslan Türkeş Bilim ve Teknoloji Üniversitesi.
- Lorenzo, P. R., Nalepa, J., Kawulok, M., Ramos, L. S., & Pastor, J. R. (2017). Particle swarm optimization for hyper-parameter selection in deep neural networks. Paper presented at the Proceedings of the genetic and evolutionary computation conference.
- MNIST Dataset. (2010). http://yann.lecun.com/exdb/mnist.
- Mooney, P. T. Retinal OCT images (Optical coherence tomography). Kaggle. (2018). https://www.kaggle.com/datasets/paultimothymooney/kermany2018
- Özbay, E., & Özbay, F. A. (2023). Parçacık Sürüsü Optimizasyon Algoritması ile Optimize Edilmiş Evrişimsel Sinir Ağı Kullanılarak Dermoskopik Görüntülerden Cilt Kanserinin Sınıflandırılması. Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 35(1), 261-273.
- Piczak, K. J. (2015). ESC: Dataset for environmental sound classification. Paper presented at the Proceedings of the 23rd ACM international conference on Multimedia.
- Porwal, P.; Pachade, S.; Kamble, R.; Kokare, M.; Deshmukh, G.; Sahasrabuddhe, V.; Meriaudeau, F. Indian Diabetic Retinopathy Image Dataset (IDRiD): A Database for Diabetic Retinopathy Screening Research. Data 2018, 3, 25.
- Raziani, S., & Azimbagirad, M. (2022). Deep CNN hyperparameter optimization algorithms for sensor-based human activity recognition. Neuroscience Informatics, 2(3), 100078.
- Sakib, S., Ahmed, N., Kabir, A. J., & Ahmed, H. (2019). An overview of convolutional neural network: Its architecture and applications.
- Salamon, J., Jacoby, C., & Bello, J. P. (2014). A dataset and taxonomy for urban sound research. Paper presented at the Proceedings of the 22nd ACM international conference on Multimedia.
- Sampson, J. R. (1976). Adaptation in natural and artificial systems (John H. Holland).
- Sethi, M., Ahuja, S., Rani, S., Bawa, P., & Zaguia, A. (2021). [Retracted] Classification of Alzheimer’s Disease Using Gaussian‐Based Bayesian Parameter Optimization for Deep Convolutional LSTM Network. Computational and Mathematical Methods in Medicine, 2021(1), 4186666.
- Shankar, K., Zhang, Y., Liu, Y., Wu, L., & Chen, C. H. (2020). Hyperparameter tuning deep learning for diabetic retinopathy fundus image classification. IEEE access, 8, 118164-118173.
- Tanyıldızı, E., & Demirtaş, F. (2019). Hiper Parametre Optimizasyonu Hyper Parameter Optimization. Paper presented at the 2019 1st International Informatics and Software Engineering Conference (UBMYK).
- Yurdakul, M. (2022). Meta-sezgisel algoritmalar ile konvolüsyonel sinir ağı mimarisinin hiper parametrelerinin optimizasyonu. Yüksek Lisans Tezi, Selçuk Üniversitesi.
- ZainEldin, H., Gamel, S. A., El-Kenawy, E. S. M., Alharbi, A. H., Khafaga, D. S., Ibrahim, A., & Talaat, F. M. (2022). Brain tumor detection and classification using deep learning and sine-cosine fitness grey wolf optimization. Bioengineering, 10(1), 18.
- Zhang, B., Rajan, R., Pineda, L., Lambert, N., Biedenkapp, A., Chua, K., ... & Calandra, R. (2021, March). On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics (pp. 4015-4023). PMLR.
- Zhang, M., Li, H., Pan, S., Lyu, J., Ling, S., & Su, S. (2021). Convolutional neural networks-based lung nodule classification: A surrogate-assisted evolutionary algorithm for hyperparameter optimization. IEEE Transactions on Evolutionary Computation, 25(5), 869-882.
- Xiao, X., Yan, M., Basodi, S., Ji, C., & Pan, Y. (2020). Efficient hyperparameter optimization in deep learning using a variable length genetic algorithm. arXiv preprint arXiv:2006.12703.
HYPERPARAMETER OPTIMIZATION IN CONVOLUTIONAL NEURAL NETWORKS WITH SCATTER SEARCH ALGORITHM USING DIFFERENT CROSSOVER TECHNIQUES
Abstract
Nowadays, artificial intelligence applications are used in all areas of life and have become more and more applicable to provide recommendations and results. Convolutional Neural Networks (CNN) is one of the most effective and intensively applied artificial intelligence algorithms in recent years, providing successful results in many real-world problems. CNNs are generally used to analyze visual information. Visual information is analyzed by passing it through convolution, activation, pooling and fully connected layers in CNNs. In CNN training, various parameters are used according to the data set used and the problem encountered. However, finding the best hyperparameter values for a CNN is still a challenging task. In this study, the parameters used in CNN training and the layers used in the network structure are optimized in order to easily select the hyperparameters that give the highest accuracy. Discrete values for the number of channels, convolutional layer, minimum batch size and activation function and continuous values for the learning rate were chosen as hyper-parameters. In this study, Scatter Search (SS) algorithm is preferred to perform hyper parameter optimization. With the SS method, appropriate hyperparameter values for CNNs were easily determined and a higher accuracy was achieved. The highest accuracy value obtained in the study is 93.24%. This value was obtained with 16 number of channels, 5 x 5 convolutional layers, 64 minimum batch size, 0.0052 learning rate and reluLayer activation function parameters.
References
- Ait Amou, M., Xia, K., Kamhi, S., & Mouhafid, M. (2022, March). A novel MRI diagnosis method for brain tumor classification based on CNN and Bayesian Optimization. In Healthcare (Vol. 10, No. 3, p. 494). MDPI.
- Andonie, R., & Florea, A.-C. (2020). Weighted random search for CNN hyperparameter optimization. arXiv preprint arXiv:2003.13300.
- Anguita, D., Ghio, A., Oneto, L., Parra, X., & Reyes-Ortiz, J. L. (2013). A public domain dataset for human activity recognition using smartphones. Paper presented at the Esann.
- Aslan, M. F., Sabanci, K., Durdu, A., & Unlersen, M. F. (2022). COVID-19 diagnosis using state-of-the-art CNN architecture features and Bayesian Optimization. Computers in biology and medicine, 142, 105244.
- Atteia, G., Abdel Samee, N., El-Kenawy, E. S. M., & Ibrahim, A. (2022). CNN-hyperparameter optimization for diabetic maculopathy diagnosis in optical coherence tomography and fundus retinography. Mathematics, 10(18), 3274.
- Bochinski, E., Senst, T., & Sikora, T. (2017). Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. Paper presented at the 2017 IEEE international conference on image processing (ICIP).
- Bozkurt, F., & Yağanoğlu, M. (2021). Derin evrişimli sinir ağları kullanarak akciğer X-Ray görüntülerinden COVID-19 tespiti. Veri Bilimi, 4(2), 1-8.
- Fujino, S., Mori, N., & Matsumoto, K. (2017). Deep convolutional networks for human sketches by means of the evolutionary deep learning. Paper presented at the 2017 joint 17th world congress of international fuzzy systems association and 9th international conference on soft computing and intelligent systems (IFSA-SCIS).
- Geng, J. C., Cui, Z., & Gu, X. S. (2016). Scatter search based particle swarm optimization algorithm for earliness/tardiness flowshop scheduling with uncertainty. International Journal of Automation and Computing, 13(3), 285-295.
- Glover, F. (1977). Heuristics for integer programming using surrogate constraints. Decision sciences, 8(1), 156-166.
- Gülcü, A., & Kuş, Z. (2019). A survey of hyper-parameter optimization methods in convolutional neural networks. Gazi Üniversitesi Fen Bilimleri Dergisi, 7(2), 503-522.
- Hao, W., Yizhou, W., Yaqin, L., & Zhili, S. (2020, December). The role of activation function in CNN. In 2020 2nd International Conference on Information Technology and Computer Application (ITCA) (pp. 429-432). IEEE.
- Huang, D. S., Wunsch, D. C., Levine, D. S., & Jo, K. H. (Eds.). (2008). Advanced Intelligent Computing Theories and Applications. With Aspects of Artificial Intelligence: Fourth International Conference on Intelligent Computing, ICIC 2008 Shanghai, China, September 15-18, 2008, Proceedings (Vol. 5227). Springer Science & Business Media.
- İnik, Ö. (2023). CNN hyper-parameter optimization for environmental sound classification. Applied Acoustics, 202, 109168.
- Kıymaç, M. E. (2022). Hyper-parameter optimization of deep neural networks with metaheuristic algorithms. Yüksek Lisans Tezi, Alparslan Türkeş Bilim ve Teknoloji Üniversitesi.
- Lorenzo, P. R., Nalepa, J., Kawulok, M., Ramos, L. S., & Pastor, J. R. (2017). Particle swarm optimization for hyper-parameter selection in deep neural networks. Paper presented at the Proceedings of the genetic and evolutionary computation conference.
- MNIST Dataset. (2010). http://yann.lecun.com/exdb/mnist.
- Mooney, P. T. Retinal OCT images (Optical coherence tomography). Kaggle. (2018). https://www.kaggle.com/datasets/paultimothymooney/kermany2018
- Özbay, E., & Özbay, F. A. (2023). Parçacık Sürüsü Optimizasyon Algoritması ile Optimize Edilmiş Evrişimsel Sinir Ağı Kullanılarak Dermoskopik Görüntülerden Cilt Kanserinin Sınıflandırılması. Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 35(1), 261-273.
- Piczak, K. J. (2015). ESC: Dataset for environmental sound classification. Paper presented at the Proceedings of the 23rd ACM international conference on Multimedia.
- Porwal, P.; Pachade, S.; Kamble, R.; Kokare, M.; Deshmukh, G.; Sahasrabuddhe, V.; Meriaudeau, F. Indian Diabetic Retinopathy Image Dataset (IDRiD): A Database for Diabetic Retinopathy Screening Research. Data 2018, 3, 25.
- Raziani, S., & Azimbagirad, M. (2022). Deep CNN hyperparameter optimization algorithms for sensor-based human activity recognition. Neuroscience Informatics, 2(3), 100078.
- Sakib, S., Ahmed, N., Kabir, A. J., & Ahmed, H. (2019). An overview of convolutional neural network: Its architecture and applications.
- Salamon, J., Jacoby, C., & Bello, J. P. (2014). A dataset and taxonomy for urban sound research. Paper presented at the Proceedings of the 22nd ACM international conference on Multimedia.
- Sampson, J. R. (1976). Adaptation in natural and artificial systems (John H. Holland).
- Sethi, M., Ahuja, S., Rani, S., Bawa, P., & Zaguia, A. (2021). [Retracted] Classification of Alzheimer’s Disease Using Gaussian‐Based Bayesian Parameter Optimization for Deep Convolutional LSTM Network. Computational and Mathematical Methods in Medicine, 2021(1), 4186666.
- Shankar, K., Zhang, Y., Liu, Y., Wu, L., & Chen, C. H. (2020). Hyperparameter tuning deep learning for diabetic retinopathy fundus image classification. IEEE access, 8, 118164-118173.
- Tanyıldızı, E., & Demirtaş, F. (2019). Hiper Parametre Optimizasyonu Hyper Parameter Optimization. Paper presented at the 2019 1st International Informatics and Software Engineering Conference (UBMYK).
- Yurdakul, M. (2022). Meta-sezgisel algoritmalar ile konvolüsyonel sinir ağı mimarisinin hiper parametrelerinin optimizasyonu. Yüksek Lisans Tezi, Selçuk Üniversitesi.
- ZainEldin, H., Gamel, S. A., El-Kenawy, E. S. M., Alharbi, A. H., Khafaga, D. S., Ibrahim, A., & Talaat, F. M. (2022). Brain tumor detection and classification using deep learning and sine-cosine fitness grey wolf optimization. Bioengineering, 10(1), 18.
- Zhang, B., Rajan, R., Pineda, L., Lambert, N., Biedenkapp, A., Chua, K., ... & Calandra, R. (2021, March). On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics (pp. 4015-4023). PMLR.
- Zhang, M., Li, H., Pan, S., Lyu, J., Ling, S., & Su, S. (2021). Convolutional neural networks-based lung nodule classification: A surrogate-assisted evolutionary algorithm for hyperparameter optimization. IEEE Transactions on Evolutionary Computation, 25(5), 869-882.
- Xiao, X., Yan, M., Basodi, S., Ji, C., & Pan, Y. (2020). Efficient hyperparameter optimization in deep learning using a variable length genetic algorithm. arXiv preprint arXiv:2006.12703.
There are 33 citations in total.
Details
| Primary Language | Turkish |
|---|---|
| Subjects | Deep Learning, Reinforcement Learning, Query Processing and Optimisation, Satisfiability and Optimisation |
| Journal Section | Computer Engineering |
| Authors | |
| Publication Date | December 3, 2024 |
| Submission Date | May 26, 2024 |
| Acceptance Date | October 5, 2024 |
| Published in Issue | Year 2024 |