Research Article
BibTex RIS Cite

REAL-TIME SEGMENTATION OF RAIL IMAGES WITH CONVOLUTIONAL NEURAL NETWORKS FOR UAV BASED RAIL INSPECTION

Year 2024, , 151 - 165, 03.03.2024

Selçuk Sinan Kırat , İlhan Aydın

https://doi.org/10.17780/ksujes.1367644

Abstract

Railways carry people and their cargo. Checking the rails is important for safe rail travel. Rails are usually controlled manually by humans. With the developing technology, UAVs are now replacing humans in many tasks. Manually checking the rails is time consuming and costly. Therefore, the rails can be controlled by UAVs. In order for UAVs to control the rails, they must fly autonomously on the rails. In order to do this, segmentation must be done on the ray images. Image segmentation is one of the studies in the field of computer vision. Deep learning is used in these studies. UNet, ICNet and BiSeNet V2, which are deep learning-based convolutional neural networks, are used in computer vision applications. These networks, which are used in real-time image segmentation tasks in the literature, were trained by customizing the publicly shared Railsem19 dataset. Networks, which reached 98% segmentation accuracy on 1024×512 pixel resolution images, reached approximately 15 fps on real-time images taken from the railway by UAV. Real-time segmentation video of the networks can be viewed at https://youtu.be/piVTdsDPzfg. A PID flight control system for autonomous UAV flight is also proposed in the study.

Keywords

Computer vision deep learning convolutional neural network real-time segmentation

Project Number

ADEP.22.02

References

  • Anadolu Ajansı. (2023). Deprem tren raylarını tel gibi büktü. https://www.ntv.com.tr/galeri/turkiye/deprem-tren-raylarini-tel-gibi-buktu,j6Y22jcDNk2TPmVE60ZCoA/JUO5LUJ0SkyT_r2jLbTUWg Erişim: 17.04.2023.
  • Aydin, I., Sevi, M., Sahbaz, K., & Karakose, M. (2021). Detection of Rail Defects with Deep Learning Controlled Autonomous UAV. 2021 International Conference on Data Analytics for Business and Industry, ICDABI 2021, 500–504. https://doi.org/10.1109/ICDABI53623.2021.9655796
  • Bayati, A. M. A. (2019). Evrişimsel Sinir Ağları Kullanarak Drone Tarafından Elde Edilen Görüntülerde Nesne Tanıma. Yüksek Lisans Tezi. Selçuk Üniversitesi Fen Bilimleri Enstitüsü Bilgisayar Müh. A.B.D., Konya 72s.
  • Bojarczak, P., & Lesiak, P. (2021). UAVs in rail damage image diagnostics supported by deep-learning networks. Open Engineering, 11(1), 339–348. https://doi.org/10.1515/eng-2021-0033
  • Çakmak, V., & Altaş, A. (2018). Sosyal Medya Etkileşiminde Tren Yolculukları: DOĞU EKSPRESİ İle İlgili Youtube Paylaşım Videolarının Analizi. Journal of Tourism and Gastronomy Studies, 6(1), 390–408. https://doi.org/10.21325/jotags.2018.194
  • Chakravarthy, A. S., Sinha, S., Narang, P., Mandal, M., Chamola, V., & Yu, F. R. (2022). DroneSegNet: Robust Aerial Semantic Segmentation for UAV-Based IoT Applications. IEEE Transactions on Vehicular Technology, 71(4), 4277–4286. https://doi.org/10.1109/TVT.2022.3144358
  • Chen, P., Wu, Y., Qin, Y., & Yang, H. (2022). All-in-One YOLO Architecture for safety Hazard Detection of Environment along High-Speed Railway. 2022 Global Reliability and Prognostics and Health Management (PHM-Yantai), 1–7. IEEE. https://doi.org/10.1109/PHM-Yantai55411.2022.9941973
  • Grandio, J., Riveiro, B., Soilán, M., & Arias, P. (2022). Point cloud semantic segmentation of complex railway environments using deep learning. Automation in Construction, 141. https://doi.org/10.1016/J.AUTCON.2022.104425
  • Guclu, E., Aydin, I., & Akin, E. (2021). Development of Vision-Based Autonomous UAV for Railway Tracking. 2021 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies, 3ICT 2021, 120–125. https://doi.org/10.1109/3ICT53449.2021.9581919
  • Güçlü, E., Aydın, İ., & Akın, E. (2022). Mask R-CNN Algoritmasını Kullanarak Demiryolu Travers Eksikliklerinin Tespiti İçin Otonom İHA Tasarımı. Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 34(1), 409–420. https://doi.org/10.35234/fumbd.1039995
  • International Union of Railways. (2022). 8th UIC Refugee Task Force meeting held on 6 May 2022. https://uic.org/com/enews/article/8th-uic-refugee-task-force-meeting-held-on-6-may-2022 Accessed: 10.05.2022.
  • Katar, O., & Duman, E. (2022). Automated Semantic Segmentation for Autonomous Railway Vehicles. TECHNICAL JOURNAL, 16, 484–490. https://doi.org/10.31803/tg-20220329114254
  • Kırat, S. S. (2023). Elazığ Demiryolu Segmentasyonu. https://youtu.be/piVTdsDPzfg Erişim: 09.04.2023.
  • Kupriyanovsky, V., Pokusaev, O., Klimov, A., Dobrynin, A., Lazutkina, V., & Potapov, I. (2020). BIM on the world’s railways-development, examples, and standards. International Journal of Open Information Technologies, 8(5), 57–80.
  • Mammeri, A., Jabbar Siddiqui, A., & Zhao, Y. (2021). UAV-assisted Railway Track Segmentation based on Convolutional Neural Networks. IEEE Vehicular Technology Conference, 2021-April, 1–7. IEEE. https://doi.org/10.1109/VTC2021-Spring51267.2021.9448887
  • Murat, S. (2021). İnsansız Hava Aracı Görüntülerinden Derin Öğrenme ile Nesne Tanıma. Yüksek Lisans Tezi. Maltepe Üniversitesi Lisansüstü Eğitim Enstitüsü Bilgisayar Mühendisliği Anabilim Dalı, İstanbul 108.
  • Ni, X., Liu, H., Ma, Z., Wang, C., & Liu, J. (2022). Detection for Rail Surface Defects via Partitioned Edge Feature. IEEE Transactions on Intelligent Transportation Systems, 23(6), 5806–5822. https://doi.org/10.1109/TITS.2021.3058635
  • Rahman, M. A., & Mammeri, A. (2021). Vegetation Detection in UAV Imagery for Railway Monitoring. Proceedings of the 7th International Conference on Vehicle Technology and Intelligent Transport Systems, 457–464. SCITEPRESS - Science and Technology Publications. https://doi.org/10.5220/0010439904570464
  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Içinde N. Navab, J. Hornegger, W. M. Wells, & A. F. Frangi (Ed.), Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-24574-4
  • Sevi, M., & Aydin, I. (2022). Rail Tracking and Detection with Drone in Gazebo Environment. 2022 International Conference on Decision Aid Sciences and Applications (DASA), 1450–1454. IEEE. https://doi.org/10.1109/DASA54658.2022.9765014
  • T.C. Ulaştırma ve Altyapı Bakanlığı. (2023). 2053 Ulaştırma ve Lojistik Ana Planı https://www.uab.gov.tr/uploads/pages/bakanlik-yayinlari/20221025-2053-ulastirma-ve-lojistik-ana-plani-tr.pdf Erişim: 05.04.2023.
  • Tiu, E. (2019). Metrics to Evaluate your Semantic Segmentation Model.
  • https://towardsdatascience.com/metrics-to-evaluate-your-semantic-segmentation-model-6bcb99639aa2 Accessed: 17.01.2022
  • Tong, L., Jia, L., Geng, Y., Liu, K., Qin, Y., & Wang, Z. (2023). Anchor-adaptive railway track detection from unmanned aerial vehicle images. Computer-Aided Civil and Infrastructure Engineering, 1–19. https://doi.org/10.1111/mice.13004
  • Tong, L., Wang, Z., Jia, L., Qin, Y., Wei, Y., Yang, H., & Geng, Y. (2022). Fully Decoupled Residual ConvNet for Real-Time Railway Scene Parsing of UAV Aerial Images. IEEE Transactions on Intelligent Transportation Systems, 23(9), 14806–14819. https://doi.org/10.1109/TITS.2021.3134318
  • Weng, Y., Li, Z., Huang, X., & Chen, X. (2023). Improved DeepLabV3+ based Railway Track Extraction to Enhance Railway Transportation Safety. https://doi.org/10.1007/978-981-99-0272-9_16
  • Wu, Y., Meng, F., Qin, Y., Qian, Y., Xu, F., & Jia, L. (2023). UAV imagery based potential safety hazard evaluation for high-speed railroad using Real-time instance segmentation. Advanced Engineering Informatics, 55(May 2022), 101819. https://doi.org/10.1016/j.aei.2022.101819
  • Yang, H., Li, X., Guo, Y., & Jia, L. (2022a). Discretization–Filtering–Reconstruction: Railway Detection in Images for Navigation of Inspection UAV. IEEE Transactions on Instrumentation and Measurement, 71, 1–13. https://doi.org/10.1109/TIM.2022.3220295
  • Yang, H., Li, X., Guo, Y., & Jia, L. (2022b). RT-GAN: GAN Based Architecture for Precise Segmentation of Railway Tracks. Applied Sciences, 12(23), 12044. https://doi.org/10.3390/app122312044
  • Yu, C., Gao, C., Wang, J., Yu, G., Shen, C., & Sang, N. (2021). BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation. International Journal of Computer Vision, 129(11), 3051–3068. https://doi.org/10.1007/s11263-021-01515-2
  • Zendel, O., Murschitz, M., Zeilinger, M., Steininger, D., Abbasi, S., & Beleznai, C. (2019). RailSem19: A Dataset for Semantic Rail Scene Understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Vienna: Austrian Institute of Technology.
  • Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018). ICNet for Real-Time Semantic Segmentation on High-Resolution Images. Içinde Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): C. 11207 LNCS (ss. 418–434). https://doi.org/10.1007/978-3-030-01219-9_25

İHA TABANLI RAY KONTROLÜ İÇİN EVRİŞİMLİ SİNİR AĞLARI İLE RAY GÖRÜNTÜLERİNİN GERÇEK ZAMANLI SEGMENTASYONU

Year 2024, , 151 - 165, 03.03.2024

Selçuk Sinan Kırat , İlhan Aydın

https://doi.org/10.17780/ksujes.1367644

Abstract

Demiryolları insan ve yükünü taşır. Güvenli bir demiryolu seyahati için rayların kontrol edilmesi önemlidir. Raylar genelde insanlar tarafından manuel olarak kontrol edilmektedir. Gelişen teknolojiyle artık İHA'lar birçok görevde insanın yerini almaktadır. Rayların manuel olarak kontrol edilmesi zaman alıcı ve maliyetli bir iştir. Bu nedenle raylar İHA'lar tarafından kontrol edilebilir. İHA'ların rayları kontrol edebilmesi için rayların üzerinde otonom olarak uçması gerekir. Bunu yapabilmesi için ray görüntüleri üzerinde segmentasyon yapılmalıdır. Görüntü segmentasyonu bilgisayarlı görü alanında yapılan çalışmalardandır. Bu çalışmalarda derin öğrenmeden faydalanılmaktadır. Derin öğrenme tabanlı evrişimsel sinir ağlarından olan UNet, ICNet ve BiSeNet V2, bilgisayarlı görü uygulamalarında kullanılmaktadırlar. Literatürde gerçek zamanlı görüntü segmentasyonu görevlerinde kullanılan bu ağlar halka açık olarak paylaşılan Railsem19 veri seti özelleştirilerek eğitilmiştir. 1024×512 piksel çözünürlüğündeki görüntüler üzerinde %98 segmentasyon doğruluğuna ulaşan ağlar İHA ile demiryolundan alınan gerçek zamanlı görüntülerde yaklaşık 15 fps hıza ulaşmışlardır. Ağların gerçek zamanlı segmentasyon videosu https://youtu.be/piVTdsDPzfg bağlantısından izlenilebilir. Çalışmada ayrıca otonom İHA uçuşu bir PID uçuş kontrol sistemi önerilmiştir.

Keywords

Bilgisayarlı görü derin öğrenme evrişimsel sinir ağı gerçek zamanlı segmentasyon.

Supporting Institution

Bu çalışma Fırat Üniversitesi Bilimsel Araştırma Projeleri tarafından ADEP.22.02 proje numarası ile desteklenmiştir.

Project Number

ADEP.22.02

Thanks

Bu çalışma Fırat Üniversitesi Bilimsel Araştırma Projeleri tarafından ADEP.22.02 proje numarası ile desteklenmiştir.

References

  • Anadolu Ajansı. (2023). Deprem tren raylarını tel gibi büktü. https://www.ntv.com.tr/galeri/turkiye/deprem-tren-raylarini-tel-gibi-buktu,j6Y22jcDNk2TPmVE60ZCoA/JUO5LUJ0SkyT_r2jLbTUWg Erişim: 17.04.2023.
  • Aydin, I., Sevi, M., Sahbaz, K., & Karakose, M. (2021). Detection of Rail Defects with Deep Learning Controlled Autonomous UAV. 2021 International Conference on Data Analytics for Business and Industry, ICDABI 2021, 500–504. https://doi.org/10.1109/ICDABI53623.2021.9655796
  • Bayati, A. M. A. (2019). Evrişimsel Sinir Ağları Kullanarak Drone Tarafından Elde Edilen Görüntülerde Nesne Tanıma. Yüksek Lisans Tezi. Selçuk Üniversitesi Fen Bilimleri Enstitüsü Bilgisayar Müh. A.B.D., Konya 72s.
  • Bojarczak, P., & Lesiak, P. (2021). UAVs in rail damage image diagnostics supported by deep-learning networks. Open Engineering, 11(1), 339–348. https://doi.org/10.1515/eng-2021-0033
  • Çakmak, V., & Altaş, A. (2018). Sosyal Medya Etkileşiminde Tren Yolculukları: DOĞU EKSPRESİ İle İlgili Youtube Paylaşım Videolarının Analizi. Journal of Tourism and Gastronomy Studies, 6(1), 390–408. https://doi.org/10.21325/jotags.2018.194
  • Chakravarthy, A. S., Sinha, S., Narang, P., Mandal, M., Chamola, V., & Yu, F. R. (2022). DroneSegNet: Robust Aerial Semantic Segmentation for UAV-Based IoT Applications. IEEE Transactions on Vehicular Technology, 71(4), 4277–4286. https://doi.org/10.1109/TVT.2022.3144358
  • Chen, P., Wu, Y., Qin, Y., & Yang, H. (2022). All-in-One YOLO Architecture for safety Hazard Detection of Environment along High-Speed Railway. 2022 Global Reliability and Prognostics and Health Management (PHM-Yantai), 1–7. IEEE. https://doi.org/10.1109/PHM-Yantai55411.2022.9941973
  • Grandio, J., Riveiro, B., Soilán, M., & Arias, P. (2022). Point cloud semantic segmentation of complex railway environments using deep learning. Automation in Construction, 141. https://doi.org/10.1016/J.AUTCON.2022.104425
  • Guclu, E., Aydin, I., & Akin, E. (2021). Development of Vision-Based Autonomous UAV for Railway Tracking. 2021 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies, 3ICT 2021, 120–125. https://doi.org/10.1109/3ICT53449.2021.9581919
  • Güçlü, E., Aydın, İ., & Akın, E. (2022). Mask R-CNN Algoritmasını Kullanarak Demiryolu Travers Eksikliklerinin Tespiti İçin Otonom İHA Tasarımı. Fırat Üniversitesi Mühendislik Bilimleri Dergisi, 34(1), 409–420. https://doi.org/10.35234/fumbd.1039995
  • International Union of Railways. (2022). 8th UIC Refugee Task Force meeting held on 6 May 2022. https://uic.org/com/enews/article/8th-uic-refugee-task-force-meeting-held-on-6-may-2022 Accessed: 10.05.2022.
  • Katar, O., & Duman, E. (2022). Automated Semantic Segmentation for Autonomous Railway Vehicles. TECHNICAL JOURNAL, 16, 484–490. https://doi.org/10.31803/tg-20220329114254
  • Kırat, S. S. (2023). Elazığ Demiryolu Segmentasyonu. https://youtu.be/piVTdsDPzfg Erişim: 09.04.2023.
  • Kupriyanovsky, V., Pokusaev, O., Klimov, A., Dobrynin, A., Lazutkina, V., & Potapov, I. (2020). BIM on the world’s railways-development, examples, and standards. International Journal of Open Information Technologies, 8(5), 57–80.
  • Mammeri, A., Jabbar Siddiqui, A., & Zhao, Y. (2021). UAV-assisted Railway Track Segmentation based on Convolutional Neural Networks. IEEE Vehicular Technology Conference, 2021-April, 1–7. IEEE. https://doi.org/10.1109/VTC2021-Spring51267.2021.9448887
  • Murat, S. (2021). İnsansız Hava Aracı Görüntülerinden Derin Öğrenme ile Nesne Tanıma. Yüksek Lisans Tezi. Maltepe Üniversitesi Lisansüstü Eğitim Enstitüsü Bilgisayar Mühendisliği Anabilim Dalı, İstanbul 108.
  • Ni, X., Liu, H., Ma, Z., Wang, C., & Liu, J. (2022). Detection for Rail Surface Defects via Partitioned Edge Feature. IEEE Transactions on Intelligent Transportation Systems, 23(6), 5806–5822. https://doi.org/10.1109/TITS.2021.3058635
  • Rahman, M. A., & Mammeri, A. (2021). Vegetation Detection in UAV Imagery for Railway Monitoring. Proceedings of the 7th International Conference on Vehicle Technology and Intelligent Transport Systems, 457–464. SCITEPRESS - Science and Technology Publications. https://doi.org/10.5220/0010439904570464
  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Içinde N. Navab, J. Hornegger, W. M. Wells, & A. F. Frangi (Ed.), Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-24574-4
  • Sevi, M., & Aydin, I. (2022). Rail Tracking and Detection with Drone in Gazebo Environment. 2022 International Conference on Decision Aid Sciences and Applications (DASA), 1450–1454. IEEE. https://doi.org/10.1109/DASA54658.2022.9765014
  • T.C. Ulaştırma ve Altyapı Bakanlığı. (2023). 2053 Ulaştırma ve Lojistik Ana Planı https://www.uab.gov.tr/uploads/pages/bakanlik-yayinlari/20221025-2053-ulastirma-ve-lojistik-ana-plani-tr.pdf Erişim: 05.04.2023.
  • Tiu, E. (2019). Metrics to Evaluate your Semantic Segmentation Model.
  • https://towardsdatascience.com/metrics-to-evaluate-your-semantic-segmentation-model-6bcb99639aa2 Accessed: 17.01.2022
  • Tong, L., Jia, L., Geng, Y., Liu, K., Qin, Y., & Wang, Z. (2023). Anchor-adaptive railway track detection from unmanned aerial vehicle images. Computer-Aided Civil and Infrastructure Engineering, 1–19. https://doi.org/10.1111/mice.13004
  • Tong, L., Wang, Z., Jia, L., Qin, Y., Wei, Y., Yang, H., & Geng, Y. (2022). Fully Decoupled Residual ConvNet for Real-Time Railway Scene Parsing of UAV Aerial Images. IEEE Transactions on Intelligent Transportation Systems, 23(9), 14806–14819. https://doi.org/10.1109/TITS.2021.3134318
  • Weng, Y., Li, Z., Huang, X., & Chen, X. (2023). Improved DeepLabV3+ based Railway Track Extraction to Enhance Railway Transportation Safety. https://doi.org/10.1007/978-981-99-0272-9_16
  • Wu, Y., Meng, F., Qin, Y., Qian, Y., Xu, F., & Jia, L. (2023). UAV imagery based potential safety hazard evaluation for high-speed railroad using Real-time instance segmentation. Advanced Engineering Informatics, 55(May 2022), 101819. https://doi.org/10.1016/j.aei.2022.101819
  • Yang, H., Li, X., Guo, Y., & Jia, L. (2022a). Discretization–Filtering–Reconstruction: Railway Detection in Images for Navigation of Inspection UAV. IEEE Transactions on Instrumentation and Measurement, 71, 1–13. https://doi.org/10.1109/TIM.2022.3220295
  • Yang, H., Li, X., Guo, Y., & Jia, L. (2022b). RT-GAN: GAN Based Architecture for Precise Segmentation of Railway Tracks. Applied Sciences, 12(23), 12044. https://doi.org/10.3390/app122312044
  • Yu, C., Gao, C., Wang, J., Yu, G., Shen, C., & Sang, N. (2021). BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation. International Journal of Computer Vision, 129(11), 3051–3068. https://doi.org/10.1007/s11263-021-01515-2
  • Zendel, O., Murschitz, M., Zeilinger, M., Steininger, D., Abbasi, S., & Beleznai, C. (2019). RailSem19: A Dataset for Semantic Rail Scene Understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Vienna: Austrian Institute of Technology.
  • Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018). ICNet for Real-Time Semantic Segmentation on High-Resolution Images. Içinde Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): C. 11207 LNCS (ss. 418–434). https://doi.org/10.1007/978-3-030-01219-9_25
There are 32 citations in total.

Details

Primary Language Turkish
Subjects Image Processing, Deep Learning, Machine Vision
Journal Section Computer Engineering
Authors

Selçuk Sinan Kırat 0000-0003-0106-6995

İlhan Aydın 0000-0001-6880-4935

Project Number ADEP.22.02
Publication Date March 3, 2024
Submission Date September 27, 2023
Published in Issue Year 2024

Cite

APA Kırat, S. S., & Aydın, İ. (2024). İHA TABANLI RAY KONTROLÜ İÇİN EVRİŞİMLİ SİNİR AĞLARI İLE RAY GÖRÜNTÜLERİNİN GERÇEK ZAMANLI SEGMENTASYONU. Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi, 27(1), 151-165. https://doi.org/10.17780/ksujes.1367644