IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES

Mücahit Cihan; Murat Ceylan

doi:10.17780/ksujes.1345020

TR EN

TERMAL VE GÖRÜNÜR GÖRÜNTÜLER KULLANILARAK GÖRÜNTÜ BİRLEŞTİRME VE DERİN ÖĞRENME TABANLI KULAK TANIMA

Abstract

Görüntüleme ve derin öğrenme alanındaki gelişmeler, kulağın yapısı benzersiz tanımlama özellikleri sunduğundan kulak biyometrisine olan ilgiyi artırmıştır. Termal ve görünür kulak görüntüleri bu özelliklerin farklı yönlerini yakalar. Termal görüntüler ışıktan bağımsızdır ve görünür görüntüler doku ayrıntılarını yakalamada mükemmeldir. Bu görüntülerin birleştirilmesi daha zengin özelliklere sahip kompozit görüntüler oluşturur. Bu çalışma, otomatik kulak tanımayı geliştirmek amacıyla farklı aydınlatma koşulları altında elde edilen termal ve görünür kulak görüntülerinin birleştirilmesini incelemektedir. Görüntü birleştirme işlemi üç farklı çok çözünürlüklü analiz yöntemini içermektedir: ayrık dalgacık dönüşümü, ridgelet dönüşümü ve curvelet dönüşümü. Ardından, kulak tanıma için özel olarak tasarlanmış derin öğrenme modeli kullanılmıştır. Bu çalışmanın sonuçları, karmaşık değerli curvelet dönüşümü ve termal görüntülerin kullanılmasının, diğer tüm yöntemleri geride bırakarak %96.82 gibi etkileyici bir tanıma oranı elde ettiğini ortaya koymaktadır. Buna karşılık, görünür görüntüler özellikle düşük ışık koşullarında %75.00 ile en düşük tanıma oranını sergilemiştir. Sonuç olarak, birden fazla veri kaynağının birleştirilmesi kulak tanıma etkinliğini önemli ölçüde artırmaktadır ve önerilen model, sınırlı sayıda birleştirilmiş kulak görüntüsüyle çalışırken bile tutarlı bir şekilde dikkate değer tanıma oranlarına ulaşmaktadır.

Keywords

IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES

Abstract

Advances in imaging and deep learning have fueled interest in ear biometrics, as the structure of the ear offers unique identification features. Thermal and visible ear images capture different aspects of these features. Thermal images are light-independent, and visible images excel at capturing texture details. Combining these images creates more feature-rich composite images. This study examines the fusion of thermal and visible ear images taken under varying lighting conditions to enhance automatic ear recognition. The image fusion process involved three distinct multiresolution analysis methods: discrete wavelet transform, ridgelet transform, and curvelet transform. Subsequently, a specially designed deep learning model was used for ear recognition. The results of this study reveal that employing the complex-valued curvelet transform and thermal images achieved an impressive recognition rate of 96.82%, surpassing all other methods. Conversely, visible images exhibited the lowest recognition rate of 75.00%, especially in low-light conditions. In conclusion, the fusion of multiple data sources significantly enhances ear recognition effectiveness, and the proposed model consistently achieves remarkable recognition rates even when working with a limited number of fused ear images.

Keywords

References

Abaza, A., & Bourlai, T. (2012, May). Human ear detection in the thermal infrared spectrum. In Thermosense: Thermal Infrared Applications XXXIV, 8354, 286-295. https://doi.org/10.1117/12.919285
Abd Almisreb, A., Jamil, N., & Din, N. M. (2018, March). Utilizing AlexNet deep transfer learning for ear recognition. In 2018 Fourth International Conference on Information Retrieval and Knowledge Management (CAMP), 1-5. DOI: 10.1109/INFRKM.2018.8464769
Alshazly, H., Linse, C., Barth, E., & Martinetz, T. (2019). Handcrafted versus CNN features for ear recognition. Symmetry, 11(12), 1493. https://doi.org/10.3390/sym11121493
AlZubi, S., Sharif, M. S., Islam, N., & Abbod, M. (2011, May). Multi-resolution analysis using curvelet and wavelet transforms for medical imaging. In 2011 IEEE international symposium on medical measurements and applications, 188-191. DOI: 10.1109/MeMeA.2011.5966687
Ariffin, S. M. Z. S. Z., Jamil, N., & Rahman, P. N. M. A. (2016, September). DIAST variability illuminated thermal and visible ear images datasets. In 2016 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), 191-195. DOI: 10.1109/SPA.2016.7763611
Ariffin, S. M. Z. S. Z., Jamil, N., & Rahman, P. N. M. A. (2017, May). Can thermal and visible image fusion improves ear recognition?. In 2017 8th International Conference on Information Technology (ICIT), 780-784. DOI: 10.1109/ICITECH.2017.8079945
Ashiq, F., Asif, M., Ahmad, M. B., Zafar, S., Masood, K., Mahmood, T., Mahmood, M. T., & Lee, I. H. (2022). CNN-based object recognition and tracking system to assist visually impaired people. IEEE Access, 10, 14819-14834. DOI: 10.1109/ACCESS.2022.3148036
Benzaoui, A., Kheider, A., & Boukrouche, A. (2015, October). Ear description and recognition using ELBP and wavelets. In 2015 International Conference on Applied Research In Computer Science And Engineering (Icar), 1-6. DOI: 10.1109/ARCSE.2015.7338146

Bertillon, A., & McClaughry, R. W. (1896). Signaletic instructions including the theory and practice of anthropometrical identification. Werner Company.
Candès, E. J., & Donoho, D. L. (1999). Ridgelets: A key to higher-dimensional intermittency?. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 357(1760), 2495-2509. https://doi.org/10.1098/rsta.1999.0444
Candes, E., Demanet, L., Donoho, D., & Ying, L. (2006). Fast discrete curvelet transforms. Multiscale Modeling & Simulation, 5(3), 861-899. https://doi.org/10.1137/05064182
Chen, D., Tang, J., Xi, H., & Zhao, X. (2021). Image Recognition of Modern Agricultural Fruit Maturity Based on Internet of Things. Traitement du Signal, 38(4). DOI: 10.18280/ts.380435
Choi, J., Hu, S., Young, S. S., & Davis, L. S. (2012, May). Thermal to visible face recognition. In Sensing Technologies for Global Health, Military Medicine, Disaster Response, and Environmental Monitoring II; and Biometric Technology for Human Identification IX, 8371, 252-261. https://doi.org/10.1117/12.920330
Cihan, M., & Ceylan, M. (2021). Fusion of CT and MR Liver Images Using Multiresolution Analysis Methods. Avrupa Bilim ve Teknoloji Dergisi, (30), 56-61. https://doi.org/10.31590/ejosat.1005858
Cihan, M., Ceylan, M., & Ornek, A. H. (2022a). Spectral-spatial classification for non-invasive health status detection of neonates using hyperspectral imaging and deep convolutional neural networks. Spectroscopy Letters, 1-14. https://doi.org/10.1080/00387010.2022.2076698
Cihan, M., Ceylan, M., Soylu, H., & Konak, M. (2022b). Fast Evaluation of Unhealthy and Healthy Neonates Using Hyperspectral Features on 700-850 Nm Wavelengths, ROI Extraction, and 3D-CNN. IRBM, 43(5), 362-371. https://doi.org/10.1016/j.irbm.2021.06.009
Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M., & Schmidhuber, J. (2011, June). Flexible, high performance convolutional neural networks for image classification. In Twenty-second international joint conference on artificial intelligence. DOI: 10.5591/978-1-57735-516-8/IJCAI11-210
Do, M. N., & Vetterli, M. (2003). The finite ridgelet transform for image representation. IEEE Transactions on image Processing, 12(1), 16-28. DOI: 10.1109/TIP.2002.806252
El-Naggar, S., & Bourlai, T. (2022). Exploring Deep Learning Ear Recognition in Thermal Images. IEEE Transactions on Biometrics, Behavior, and Identity Science, 5(1), 64-75. DOI: 10.1109/TBIOM.2022.3218151
Emeršič, Ž., Štepec, D., Štruc, V., & Peer, P. (2017). Training convolutional neural networks with limited training data for ear recognition in the wild. arXiv preprint arXiv:1711.09952. https://doi.org/10.48550/arXiv.1711.09952
Emeršič, Ž., Štruc, V., & Peer, P. (2017). Ear recognition: More than a survey. Neurocomputing, 255, 26-39. https://doi.org/10.1016/j.neucom.2016.08.139
Fadili, J. M., & Starck, J. L. (2009). Curvelets and ridgelets. https://doi.org/10.1007/978-0-387-30440-3_111
Fields, C., Falls, H. C., Warren, C. P., & Zimberoff, M. (1960). The ear of the newborn as an identification constant. Obstetrics & Gynecology, 16(1), 98-102.
Galdámez, P. L., Raveane, W., & Arrieta, A. G. (2017). A brief review of the ear recognition process using deep neural networks. Journal of Applied Logic, 24, 62-70. https://doi.org/10.1016/j.jal.2016.11.014
Guérin, J., Thiery, S., Nyiri, E., Gibaru, O., & Boots, B. (2021). Combining pretrained CNN feature extractors to enhance clustering of complex natural images. Neurocomputing, 423, 551-571. https://doi.org/10.1016/j.neucom.2020.10.068
Gutiérrez, L., Melin, P., & Lopez, M. (2010, July). Modular neural network integrator for human recognition from ear images. In The 2010 International Joint Conference on Neural Networks (IJCNN), 1-5. DOI: 10.1109/IJCNN.2010.5596633
Haghighat, M. B. A., Aghagolzadeh, A., & Seyedarabi, H. (2011). Multi-focus image fusion for visual sensor networks in DCT domain. Computers & Electrical Engineering, 37(5), 789-797. https://doi.org/10.1016/j.compeleceng.2011.04.016
Jain, A. K., Flynn, P., & Ross, A. A. (Eds.). (2007). Handbook of biometrics. Springer Science & Business Media.
Jamil, N., AlMisreb, A., & Halin, A. A. (2014). Illumination-invariant ear authentication. Procedia Computer Science, 42, 271-278. https://doi.org/10.1016/j.procs.2014.11.062
Kong, S. G., Heo, J., Boughorbel, F., Zheng, Y., Abidi, B. R., Koschan, A., Yi, M., & Abidi, M. A. (2007). Multiscale fusion of visible and thermal IR images for illumination-invariant face recognition. International Journal of Computer Vision, 71(2), 215-233. https://doi.org/10.1007/s11263-006-6655-0
Lannarelli, A. (1989). Ear identification. Forensic identification series.
Ma, Y., Huang, Z., Wang, X., & Huang, K. (2020). An overview of multimodal biometrics using the face and ear. Mathematical Problems in Engineering, 2020. https://doi.org/10.1155/2020/6802905
Maity, S., Abdel-Mottaleb, M., & Asfour, S. S. (2020). Multimodal biometrics recognition from facial video with missing modalities using deep learning. Journal of Information Processing Systems, 16(1), 6-29. DOI: 10.3745/JIPS.02.0129
Moreno, B., Sanchez, A., & Vélez, J. F. (1999, October). On the use of outer ear images for personal identification in security applications. In Proceedings IEEE 33rd Annual 1999 International Carnahan Conference on Security Technology (Cat. No. 99CH36303), 469-476. DOI: 10.1109/CCST.1999.797956
Morlet, J., Arens, G., Fourgeau, E., & Glard, D. (1982). Wave propagation and sampling theory—Part I: Complex signal and scattering in multilayered media. Geophysics, 47(2), 203-221. https://doi.org/10.1190/1.1441328
Nejati, H., Zhang, L., Sim, T., Martinez-Marroquin, E., & Dong, G. (2012, November). Wonder ears: Identification of identical twins from ear images. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), 1201-1204.
Pajares, G., & De La Cruz, J. M. (2004). A wavelet-based image fusion tutorial. Pattern recognition, 37(9), 1855-1872. https://doi.org/10.1016/j.patcog.2004.03.010
Pflug, A., Paul, P. N., & Busch, C. (2014, October). A comparative study on texture and surface descriptors for ear biometrics. In 2014 International carnahan conference on security technology (ICCST), 1-6. DOI: 10.1109/CCST.2014.6986993
Rane, M. E., & Bhadade, U. (2020, December). Face and palmprint Biometric recognition by using weighted score fusion technique. In 2020 IEEE Pune Section International Conference (PuneCon), 11-16. DOI: 10.1109/PuneCon50868.2020.9362433
Sarangi, P. P., Mishra, B. P., & Dehuri, S. (2018, March). Multimodal biometric recognition using human ear and profile face. In 2018 4th International Conference on Recent Advances in Information Technology (RAIT), 1-6. DOI: 10.1109/RAIT.2018.8389035
Sarangi, P. P., Nayak, D. R., Panda, M., & Majhi, B. (2022). A feature-level fusion based improved multimodal biometric recognition system using ear and profile face. Journal of Ambient Intelligence and Humanized Computing, 13(4), 1867-1898. https://doi.org/10.1007/s12652-021-02952-0
Seal, A., Bhattacharjee, D., Nasipuri, M., Gonzalo-Martin, C., & Menasalvas, E. (2017). Fusion of visible and thermal images using a directed search method for face recognition. International Journal of Pattern Recognition and Artificial Intelligence, 31(04), 1756005. https://doi.org/10.1142/S0218001417560055
Singh, S., Gyaourova, A., Bebis, G., & Pavlidis, I. (2004, August). Infrared and visible image fusion for face recognition. In Biometric technology for human identification, 5404, 585-596. https://doi.org/10.1117/12.543549
Starck, J. L., Donoho, D. L., & Candès, E. J. (2003). Astronomical image representation by the curvelet transform. Astronomy & Astrophysics, 398(2), 785-800. DOI: 10.1051/0004-6361:20021571
Toygar, Ö., Alqaralleh, E., & Afaneh, A. (2018). Person identification using multimodal biometrics under different challenges. Human-Robot Interaction-Theory and Application, 81-96. DOI: 10.5772/intechopen.71667
Victor, B., Bowyer, K., & Sarkar, S. (2002, August). An evaluation of face and ear biometrics. In 2002 International Conference on Pattern Recognition, 1, 429-432. DOI: 10.1109/ICPR.2002.1044746
Yu, H., Yang, L. T., Zhang, Q., Armstrong, D., & Deen, M. J. (2021). Convolutional neural networks for medical image analysis: state-of-the-art, comparisons, improvement and perspectives. Neurocomputing, 444, 92-110. https://doi.org/10.1016/j.neucom.2020.04.157

Details

Primary Language

English

Subjects

Image Processing, Pattern Recognition, Deep Learning

Journal Section

Research Article

Authors

Mücahit Cihan ^*
0000-0002-1426-319X
Türkiye

Murat Ceylan
0000-0001-6503-9668
Türkiye

Publication Date

December 3, 2023

Submission Date

August 17, 2023

Acceptance Date

September 21, 2023

Published in Issue

Year 2023 Volume: 26 Number: 4

DOI

https://doi.org/10.17780/ksujes.1345020

IZ

https://izlik.org/JA45JP32RP

Cite

RIS / Bibtex

APA

Cihan, M., & Ceylan, M. (2023). IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES. Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi, 26(4), 997-1009. https://doi.org/10.17780/ksujes.1345020

AMA

1.Cihan M, Ceylan M. IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES. KSU J. Eng. Sci. 2023;26(4):997-1009. doi:10.17780/ksujes.1345020

Chicago

Cihan, Mücahit, and Murat Ceylan. 2023. “IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES”. Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi 26 (4): 997-1009. https://doi.org/10.17780/ksujes.1345020.

EndNote

Cihan M, Ceylan M (December 1, 2023) IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES. Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi 26 4 997–1009.

IEEE

[1]M. Cihan and M. Ceylan, “IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES”, KSU J. Eng. Sci., vol. 26, no. 4, pp. 997–1009, Dec. 2023, doi: 10.17780/ksujes.1345020.

ISNAD

Cihan, Mücahit - Ceylan, Murat. “IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES”. Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi 26/4 (December 1, 2023): 997-1009. https://doi.org/10.17780/ksujes.1345020.

JAMA

1.Cihan M, Ceylan M. IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES. KSU J. Eng. Sci. 2023;26:997–1009.

MLA

Cihan, Mücahit, and Murat Ceylan. “IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES”. Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi, vol. 26, no. 4, Dec. 2023, pp. 997-1009, doi:10.17780/ksujes.1345020.

Vancouver

1.Mücahit Cihan, Murat Ceylan. IMAGE FUSION AND DEEP LEARNING BASED EAR RECOGNITION USING THERMAL AND VISIBLE IMAGES. KSU J. Eng. Sci. 2023 Dec. 1;26(4):997-1009. doi:10.17780/ksujes.1345020