DEEP LEARNING METHODOLOGIES FOR NUCLEI SEGMENTATION AND MITOSIS DETECTION IN HISTOPATHOLOGICAL IMAGES ANALYSIS

Nooshin Nemati; Refik Samet; Emrah Hançer; Serpil Dizbay Sak; Ayca Bilge Kirmizi; Zeynep Yildirim

doi:10.17780/ksujes.1613789

EN TR

DEEP LEARNING METHODOLOGIES FOR NUCLEI SEGMENTATION AND MITOSIS DETECTION IN HISTOPATHOLOGICAL IMAGES ANALYSIS

Abstract

Histopathological image analysis is a pivotal area of medical research that leverages deep learning to derive quantitative insights from Hematoxylin and Eosin (H\&E) stained images. This study aims to enhance the analysis of H\&E breast cancer histopathology images by developing deep learning methodologies focused on nuclei and mitosis. Nuclei provide essential information for disease diagnosis, while mitosis is crucial for cancer grading and prognosis prediction. We propose two methodologies: the first segments nuclei using a U-shaped semantic segmentation architecture called CompSegNet; the second detects and classifies mitotic cells through a hybrid approach combining object detection and fuzzy classification algorithms. To evaluate the effectiveness of these methodologies, we introduce two new publicly available datasets: NuSeC (Nuclei Segmentation and Classification) and MiDeSeC (Mitosis Detection, Segmentation, and Classification). These datasets not only validate our methodologies but also provide valuable resources for developing deep learning models in histopathological image analysis.

Keywords

Supporting Institution

TUBİTAK

Project Number

121E379.16

Thanks

This work is supported by Turkish Scientific and Research Council (TUBITAK) under Grant No.121E379.16

References

Aatresh, A. A., Yatgiri, R. P., Chanchal, A. K., Kumar, A., Ravi, A., Das, D., & Kini, J. (2021). Efficient deep learning architecture with dimension-wise pyramid pooling for nuclei segmentation of histopathology images. Computerized Medical Imaging and Graphics, 93, 101975. https://doi.org/10.1016/j.compmedimag.2021.101975.
Amgad, M., Atteya, L. A., Hussein, H., Mohammed, K. H., Hafiz, E., Elsebaie, M. A., & Cooper, L. A. (2022). NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer. GigaScience, 11, giac037. https://doi.org/10.1093/gigascience/giac037.
Aubreville, M., Stathonikos, N., Bertram, C. A., Klopfleisch, R., Ter Hoeve, N., Ciompi, F., & Breininger, K. (2023). Mitosis domain generalization in histopathology images—the MIDOG challenge. Medical Image Analysis, 84, 102699. https://doi.org/10.1016/j.media.2022.102699.
Aubreville, M., Wilm, F., Stathonikos, N., Breininger, K., Donovan, T. A., Jabari, S., & Bertram, C. A. (2023). A comprehensive multi-domain dataset for mitotic figure detection. Scientific data, 10(1), 484. https://doi.org/10.1038/s41597-023-02327-4.
Bankhead, P., Loughrey, M. B., Fernández, J. A., Dombrowski, Y., McArt, D. G., Dunne, P. D., & Hamilton, P. W. (2017). QuPath: Open source software for digital pathology image analysis. Scientific reports, 7(1), 1-7. https://doi.org/10.1038/s41598-017-17204-5.
Bertram, C. A., Veta, M., Marzahl, C., Stathonikos, N., Maier, A., Klopfleisch, R., & Aubreville, M. (2020). Are pathologist-defined labels reproducible? Comparison of the TUPAC16 mitotic figure dataset with an alternative set of labels. In Interpretable and Annotation-Efficient Learning for Medical Image Computing: Third International Workshop, iMIMIC 2020, Second International Workshop, MIL3ID 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4–8, 2020, Proceedings 3, 204-213. Springer International Publishing. https://doi.org/10.1007/978-3-030-61166-8_22.
Bonissone, P., Cadenas, J. M., Garrido, M. C., & Díaz-Valladares, R. A. (2010). A fuzzy random forest. International Journal of Approximate Reasoning, 51(7), 729-747. https://doi.org/10.1016/j.ijar.2010.02.003.
Chen, L. C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018). Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), 801-818. https://doi.org/10.48550/arXiv.1802.02611.

Chen, Y., Li, X., Hu, K., Chen, Z., & Gao, X. (2020). Nuclei segmentation in histopathology images using rotation equivariant and multi-level feature aggregation neural network. In 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 549-554. IEEE. https://doi.org/10.1109/BIBM49941.2020.9313413.
Dodballapur, V., Song, Y., Huang, H., Chen, M., Chrzanowski, W., & Cai, W. (2019). Mask-driven mitosis detection in histopathology images. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), 1855-1859. IEEE. https://doi.org/10.1109/ISBI.2019.8759164.
Gamper, J., Koohbanani, N. A., Benes, K., Graham, S., Jahanifar, M., Khurram, S. A., & Rajpoot, N. (2020). Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003.10778. https://doi.org/10.48550/arXiv.2003.10778.
Graham, S., Jahanifar, M., Azam, A., Nimir, M., Tsang, Y. W., Dodd, K., & Rajpoot, N. M. (2021). Lizard: A large-scale dataset for colonic nuclear instance segmentation and classification. In Proceedings of the IEEE/CVF international conference on computer vision, 684-693. https://doi.org/10.48550/arXiv.2108.11195.
Graham, S., Vu, Q. D., Raza, S. E. A., Azam, A., Tsang, Y. W., Kwak, J. T., & Rajpoot, N. (2019). Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical image analysis, 58, 101563. https://doi.org/10.1016/j.media.2019.101563.
Hamidinekoo, A., & Zwiggelaar, R. (2017). Stain colour normalisation to improve mitosis detection on breast histology images. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3, 213-221. Springer International Publishing. https://doi.org/10.1007/978-3-319-67558-9_25.
Hancer, E., Traore, M., Samet, R., Yıldırım, Z., & Nemati, N. (2023). An imbalance-aware nuclei segmentation methodology for H&E stained histopathology images. Biomedical Signal Processing and Control, 83, 104720. https://doi.org/10.1016/j.bspc.2023.104720.
Hassan, L., Abdel-Nasser, M., Saleh, A., A. Omer, O., & Puig, D. (2021). Efficient stain-aware nuclei segmentation deep learning framework for multi-center histopathological images. Electronics, 10(8), 954. https://doi.org/10.3390/electronics10080954.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778. https://doi.org/10.1109/CVPR.2016.90.
Hlavcheva, D., Yaloveha, V., & Podorozhniak, A. (2019). Application of convolutional neural network for histopathological analysis. Advanced Information Systems, 3(4), 69-73. https://doi.org/10.20998/2522-9052.2019.4.10.
Hoorali, F., Khosravi, H., & Moradi, B. (2022). Automatic microscopic diagnosis of diseases using an improved UNet++ architecture. Tissue and Cell, 76, 101816. https://doi.org/10.1016/j.tice.2022.101816.
Ilyas, T., Mannan, Z. I., Khan, A., Azam, S., Kim, H., & De Boer, F. (2022). TSFD-Net: Tissue specific feature distillation network for nuclei segmentation and classification. Neural Networks, 151, 1-15. https://doi.org/10.1016/j.neunet.2022.02.020.
Jahanifar, M., Shephard, A., Zamanitajeddin, N., Raza, S. E. A., & Rajpoot, N. (2022). Stain-robust mitotic figure detection for MIDOG 2022 challenge. arXiv preprint arXiv:2208.12587. https://doi.org/10.1016/j.media.2024.103132.
Jiang, S., & Li, J. (2022). TransCUNet: UNet cross fused transformer for medical image segmentation. Computers in Biology and Medicine, 150, 106-207. https://doi.org/10.1016/j.compbiomed.2022.106207.
Khan, M. S., Ali, S., Lee, Y. R., Kang, M. K., Park, S. Y., Tak, W. Y., & Jung, S. K. (2023). TransUNet-lite: A robust approach to cell nuclei segmentation. In Proceedings of the 2023 7th International Conference on Medical and Health Informatics, 251-258. https://doi.org/10.1145/3608298.3608344.
Khan, H. U., Raza, B., Shah, M. H., Usama, S. M., Tiwari, P., & Band, S. S. (2023). SMDetector: Small mitotic detector in histopathology images using faster R-CNN with dilated convolutions in backbone model. Biomedical Signal Processing and Control, 81, 104-414. https://doi.org/10.1016/j.bspc.2022.104414.
Khan, M. Z., Gajendran, M. K., Lee, Y., & Khan, M. A. (2021). Deep neural architectures for medical image semantic segmentation. IEEE Access, 9, 83002-83024. https://doi.org/10.1109/ACCESS.2021.3086530.
Kumar, N., Verma, R., Anand, D., Zhou, Y., Onder, O. F., Tsougenis, E., & Sethi, A. (2019). A multi-organ nucleus segmentation challenge. IEEE transactions on medical imaging, 39(5), 1380-1391. https://doi.org/10.1109/TMI.2019.2947628.
Kumar, N., Verma, R., Sharma, S., Bhargava, S., Vahadane, A., & Sethi, A. (2017). A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE transactions on medical imaging, 36(7), 1550-1560. https://doi.org/10.1109/TMI.2017.2677499.
Le Dinh, T., Lee, S. H., Kwon, S. G., & Kwon, K. R. (2022). Cell nuclei segmentation in cryonuseg dataset using nested unet with efficientnet encoder. In 2022 International Conference on Electronics, Information, and Communication (ICEIC), 1-4. IEEE. https://doi.org/10.1109/ICEIC54506.2022.9748537.
Li, C., Wang, X., Liu, W., & Latecki, L. J. (2018). DeepMitosis: Mitosis detection via deep detection, verification and segmentation networks. Medical image analysis, 45, 121-133. https://doi.org/10.1016/j.media.2017.12.002.
Liu, X., Guo, Z., Cao, J., & Tang, J. (2021). MDC-net: A new convolutional neural network for nucleus segmentation in histopathology images with distance maps and contour information. Computers in Biology and Medicine, 135, 104543. https://doi.org/10.1016/j.compbiomed.2021.104543.
Ludovic, R., Daniel, R., Nicolas, L., Maria, K., Humayun, I., Jacques, K., & Catherine, G. (2013). Mitosis detection in breast cancer histological images An ICPR 2012 contest. Journal of pathology informatics, 4(1), 8. https://doi.org/10.4103/2153-3539.112693.
Maarouf, C., Benomar, M. L., & Settouti, N. (2021). Pre-trained backbones effect on nuclei segmentation performance. In Mediterranean Conference on Pattern Recognition and Artificial Intelligence, 108-118. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-031-04112-9_8.
Macenko, M., Niethammer, M., Marron, J. S., Borland, D., Woosley, J. T., Guan, X., & Thomas, N. E. (2009). A method for normalizing histology slides for quantitative analysis. In 2009 IEEE international symposium on biomedical imaging: from nano to macro, 1107-1110. IEEE. https://doi.org/10.1109/ISBI.2009.5193250.
Mahbod, A., Schaefer, G., Bancher, B., Löw, C., Dorffner, G., Ecker, R., & Ellinger, I. (2021). CryoNuSeg: A dataset for nuclei instance segmentation of cryosectioned H&E-stained histological images. Computers in biology and medicine, 132, 104-349. https://doi.org/10.1016/j.compbiomed.2021.104349.
Maroof, N., Khan, A., Qureshi, S. A., ul Rehman, A., Khalil, R. K., & Shim, S. O. (2020). Mitosis detection in breast cancer histopathology images using hybrid feature space. Photodiagnosis and photodynamic therapy, 31, 101-885. https://doi.org/10.1016/j.pdpdt.2020.101885.
Naylor, P., Laé, M., Reyal, F., & Walter, T. (2017). Nuclei segmentation in histopathology images using deep neural networks. In 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017), 933-936. IEEE. https://doi.org/10.1109/ISBI.2017.7950669.
Naylor, P., Laé, M., Reyal, F., & Walter, T. (2018). Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE transactions on medical imaging, 38(2), 448-459. https://doi.org/10.1109/TMI.2018.2865709.
Nemati, N., Samet, R., Hancer, E., Yildirim, Z., & Akkas, E. E. (2023). A Hybridized Deep Learning Methodology for Mitosis Detection and Classification from Histopathology Images. Journal of Machine Intelligence and Data Science (JMIDS), 4(1), 35-43. https://doi.org/10.11159/jmids.2023.005.
Obeid, A., Mahbub, T., Javed, S., Dias, J., & Werghi, N. (2022). NucDETR: end-to-end transformer for nucleus detection in histopathology images. In International Workshop on Computational Mathematics Modeling in Cancer Analysis, 47-57. Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-17266-3_5.
Qin, J., He, Y., Zhou, Y., Zhao, J., & Ding, B. (2022). REU-Net: Region-enhanced nuclei segmentation network. Computers in Biology and Medicine, 146, 105-546. https://doi.org/10.1016/j.compbiomed.2022.105546.
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, 3(18), 234-241. Springer International Publishing. https://doi.org/10.48550/arXiv.1505.04597.
Roux, L., Racoceanu, D., Capron, F., Calvo, J., Attieh, E., Le Naour, G., & Gloaguen, A. (2014). Mitos & atypia. Image Pervasive Access Lab (IPAL), Agency Sci., Technol. & Res. Inst. Infocom Res., Singapore, Tech. Rep, 1, 1-8.
Sebai, M. (2020). Improved SegMitos framework for mitosis detection in breast cancer histopathology images. In 2020 IEEE International Conference on Artificial Intelligence and Information Systems (ICAIIS), 102-106. IEEE. https://doi.org/10.1109/ICAIIS49377.2020.9194877.
Sebai, M., Wang, T., & Al-Fadhli, S. A. (2020). PartMitosis: a partially supervised deep learning framework for mitosis detection in breast cancer histopathology images. IEEE Access, 8, 45133-45147. https://doi.org/10.1109/ACCESS.2020.2978754.
Sirinukunwattana, K., Raza, S. E. A., Tsang, Y. W., Snead, D. R., Cree, I. A., & Rajpoot, N. M. (2016). Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE transactions on medical imaging, 35(5), 1196-1206. https://doi.org/10.1109/TMI.2016.2525803.
Sohail, A., Khan, A., Wahab, N., Zameer, A., & Khan, S. (2021). A multi-phase deep CNN based mitosis detection framework for breast cancer histopathological images. Scientific Reports, 11(1), 6215. https://doi.org/10.1038/s41598-021-85652-1.
Swarts, D. R., van Suylen, R. J., den Bakker, M. A., van Oosterhout, M. F., Thunnissen, F. B., Volante, M., & Speel, E. J. M. (2014). Interobserver variability for the WHO classification of pulmonary carcinoids. The American journal of surgical pathology, 38(10), 1429-1436. https://doi.org/10.1097/PAS.0000000000000300.
Traoré, M., Hancer, E., Samet, R., Yıldırım, Z., & Nemati, N. (2024). CompSegNet: An enhanced U-shaped architecture for nuclei segmentation in H&E histopathology images. Biomedical Signal Processing and Control, 97, 106699. https://doi.org/10.1016/j.bspc.2024.106699.
Verdicchio, M., Brancato, V., Cavaliere, C., Isgrò, F., Salvatore, M., & Aiello, M. (2023). A pathomic approach for tumor-infiltrating lymphocytes classification on breast cancer digital pathology images. Heliyon, 9(3). https://doi.org/10.1016/j.heliyon.2023.e14371.
Verma, R., Kumar, N., Patil, A., Kurian, N. C., Rane, S., Graham, S., & Sethi, A. (2021). MoNuSAC2020: A multi-organ nuclei segmentation and classification challenge. IEEE Transactions on Medical Imaging, 40(12), 3413-3423. https://doi.org/10.1109/TMI.2021.3085712.
Veta M, Viergever MA, Pluim JPW, Stathonikos N, van Diest PJ. MICCAI Grand Challenge: Assessment of mitosis detection algorithms (AMIDA13). Computer Vision and Pattern Recognition 2014. https://doi.org/10.1016/j.media.2014.11.010.
Vu, Q. D., Graham, S., Kurc, T., To, M. N. N., Shaban, M., Qaiser, T., & Farahani, K. (2019). Methods for segmentation and classification of digital microscopy tissue images. Frontiers in bioengineering and biotechnology, 7, 53. https://doi.org/10.3389/fbioe.2019.00053.
Wilm, F., Marzahl, C., Breininger, K., & Aubreville, M. (2021). Domain adversarial RetinaNet as a reference algorithm for the MItosis DOmain generalization challenge. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 5-13. Cham: Springer International Publishing. https://doi.org/10.48550/arXiv.2108.11269.
Wolberg, W. (1992). Breast cancer Wisconsin (original). UCI Machine Learning Repository, 110. https://doi.org/10.24432/C5HP4Z.
Yancey, R. (2023). Parallel YOLO-based Model for Real-time Mitosis Counting. 10.24132/CSRN.3201.32.
Yancey, R. E. (2022). Deep Feature Fusion for Mitosis Counting. In 2022 3rd International Conference on Pattern Recognition and Machine Learning (PRML), 151-158. IEEE. https://doi.org/10.1109/PRML56267.2022.9882245.
Yang, G., Huang, J., He, Y., Chen, Y., Wang, T., Jin, C., & Sengphachanh, P. (2022). GCP‐Net: A Gating Context‐Aware Pooling Network for Cervical Cell Nuclei Segmentation. Mobile Information Systems, 2022(1), 7511905. https://doi.org/10.1155/2022/7511905.

Details

Primary Language

English

Subjects

Image Processing , Deep Learning , Artificial Intelligence (Other)

Journal Section

Research Article

Authors

Nooshin Nemati ^*
0000-0002-5306-0344
Türkiye

Refik Samet
0000-0001-8720-6834
Türkiye

Emrah Hançer
0000-0002-3213-5191
Türkiye

Serpil Dizbay Sak
0000-0003-3666-3095
Türkiye

Ayca Bilge Kirmizi
0000-0003-3192-1921
Türkiye

Zeynep Yildirim
0000-0001-5846-9256
Türkiye

Publication Date

June 3, 2025

Submission Date

January 5, 2025

Acceptance Date

March 18, 2025

Published in Issue

Year 1970 Volume: 28 Number: 2

DOI

https://doi.org/10.17780/ksujes.1613789

IZ

https://izlik.org/JA73NN33LM

Cite

RIS / Bibtex

APA

Nemati, N., Samet, R., Hançer, E., Dizbay Sak, S., Kirmizi, A. B., & Yildirim, Z. (2025). DEEP LEARNING METHODOLOGIES FOR NUCLEI SEGMENTATION AND MITOSIS DETECTION IN HISTOPATHOLOGICAL IMAGES ANALYSIS. Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi, 28(2), 785-801. https://doi.org/10.17780/ksujes.1613789

Cited By

HR-YOLOv8: an innovative model to detect mitosis and identify cancer regions in histopathological images

Neural Computing and Applications

https://doi.org/10.1007/s00521-025-11594-8

MAR: Cancer Grading Metric With AI‐Based Histopathological Image Assessment

International Journal of Imaging Systems and Technology

https://doi.org/10.1002/ima.70305

DEEP LEARNING METHODOLOGIES FOR NUCLEI SEGMENTATION AND MITOSIS DETECTION IN HISTOPATHOLOGICAL IMAGES ANALYSIS

DEEP LEARNING METHODOLOGIES FOR NUCLEI SEGMENTATION AND MITOSIS DETECTION IN HISTOPATHOLOGICAL IMAGES ANALYSIS

Abstract

Keywords

HİSTOPATOLOJİK GÖRÜNTÜ ANALİZİNDE ÇEKİRDEK SEGMENTASYONU VE MİTOZ TESPİTİ İÇİN DERİN ÖĞRENME YÖNTEMLERİ

Öz

Anahtar Kelimeler

Supporting Institution

Project Number

Thanks

References

Details

Primary Language

Subjects

Journal Section

Authors

Publication Date

Submission Date

Acceptance Date

Published in Issue

DOI

IZ

Cite

Cited By

HR-YOLOv8: an innovative model to detect mitosis and identify cancer regions in histopathological images

MAR: Cancer Grading Metric With AI‐Based Histopathological Image Assessment