Research Article
BibTex RIS Cite

EVALUATION OF PERFORMANCE METRICS USED IN RECOMMENDATION SYSTEMS ACCORDING TO FILTERING TECHNOLOGIES: A RESEARCH STUDY ON THE FIELD OF JOB RECOMMENDATION SYSTEMS

Year 2024, Volume: 27 Issue: 3, 706 - 725, 03.09.2024

Selin Bitirim , Duygu Çelik Ertuğrul

https://doi.org/10.17780/ksujes.1410926

Abstract

Thanks to Recommendation Systems (RSs), it has become possible to carry out existing processes/operations effectively in almost every sector (e.g. e-commerce, education, entertainment, healthcare, human resources, advertising, etc.) and to prioritize items that may interest the user. With the contribution of RSs, it is possible to effectively manage sectoral processes/services and produce personalized results for users. This study aims to review RS-related research, reveal a taxonomy of filtering techniques, and identify widely encountered performance metrics. In addition, Job Recommendation Systems, which are indispensable for Human Resources (HR) management, were chosen as the research area in this study and it was planned to determine performance metrics and item filtering approaches. Various studies from the literature on RS architecture and solutions, conducted between 2010 and 2023, were selected according to their relevance and reviewed. Filtering techniques in RSs are classified hierarchically and the majority evaluation metrics used in performance evaluations are determined and categorized. Additionally, the reflections of the gains learned from RSs on Job Recommendation Systems were investigated and RS solutions/metrics in the field of HR were presented. Finally, this study serves as a road map for researchers who want to conduct research, development and quality evaluations on RS solutions.

Keywords

Recommender systems job recommender systems taxonomy filtering performance metrics

References

  • Adomavicius, G., & Tuzhilin, A. (2011). Context-aware recommender systems. In Recommender systems handbook (pp. 217-253). Springer, Boston, MA. https://doi.org/10.1609/aimag.v32i3.2364
  • Al-Habaibeh, A., Watkins, M., Waried, K., & Javareshk, M. B. (2021). Challenges and opportunities of remotely working from home during Covid-19 pandemic. Global Transitions, 3, 99-108. https://doi.org/10.1016/j.glt.2021.11.001
  • Almalis, N. D., Tsihrintzis, G. A., Karagiannis, N., & Strati, A. D. (2015). FoDRA—A new content-based job recommendation algorithm for job seeking and recruiting. In 2015 6th International Conference on Information, Intelligence, Systems and Applications (IISA) (pp. 1-7). IEEE. https://doi.org/10.1109/IISA.2015.7388018
  • Al-Otaibi, S., & Ykhlef, M. (2017). Hybrid immunizing solution for job recommender system. Frontiers of Computer Science, 11(3), 511-527. https://doi.org/10.1007/s11704-016-5241-z
  • Al-Shamri, M. Y. H. (2016). User profiling approaches for demographic recommender systems. Knowledge-Based Systems, 100, 175-187. https://doi.org/10.1016/j.knosys.2016.03.006
  • Althbiti, A., Alshamrani, R., Alghamdi, T., Lee, S., & Ma, X. (2021). Addressing data sparsity in collaborative filtering-based recommender systems using clustering and artificial neural network. In 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC) (pp. 0218-0227). IEEE. https://doi.org/10.1109/CCWC51732.2021.9376008
  • Aouadni, I., & Rebai, A. (2017). Decision support system based on genetic algorithm and multi-criteria satisfaction analysis (MUSA) method for measuring job satisfaction. Annals of Operations Research, 256(1), 3-20. https://doi.org/10.1007/s10479-016-2154-z
  • Arita, S., Hiyama, A., & Hirose, M. (2017). Gber: A social matching app which utilizes time, place, and skills of workers and jobs. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 127-130). https://doi.org/10.1145/3022198.3026316
  • Avazpour, I., Pitakrat, T., Grunske, L., & Grundy, J. (2014). Dimensions and metrics for evaluating recommendation systems. In Recommendation systems in software engineering (pp. 245-273). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-45135-5_10
  • Ayub, M., Ghazanfar, M. A., Maqsood, M., & Saleem, A. (2018). A Jaccard base similarity measure to improve performance of CF based recommender systems. In 2018 International conference on information networking (ICOIN) (pp. 1-6). IEEE. https://doi.org/10.1109/ICOIN.2018.8343073
  • Beel, J., Gipp, B., Langer, S., & Breitinger, C. (2016). Paper recommender systems: a literature survey. International Journal on Digital Libraries, 17(4), 305-338. https://doi.org/10.1007/s00799-015-0156-0
  • Benabderrahmane, S., Mellouli, N., & Lamolle, M. (2017). Predicting the users' clickstreams using time series representation, symbolic sequences, and deep learning: application on job offers recommendation tasks. In 2017 IEEE International Conference on Information Reuse and Integration (IRI) (pp. 436-443). IEEE. https://doi.org/10.1109/IRI.2017.54
  • Bhat, S. S., Pranav, P., Shashank, K. V., Raghunandan, A., & Mohan, B. R. (2022). Comparative Performance Evaluation of Web-Based Book Recommender Systems. In 2022 6th International Conference on Trends in Electronics and Informatics (ICOEI) (pp. 985-991). https://doi.org/IEEE. 10.1109/ICOEI53556.2022.9777116
  • Birjali, M., Kasri, M., & Beni-Hssane, A. (2021). A comprehensive survey on sentiment analysis: Approaches, challenges and trends. Knowledge-Based Systems, 226, 107134. ). https://doi.org/10.1016/j.knosys.2021.107134
  • Bobadilla, J., Ortega, F., Hernando, A., & Gutiérrez, A. (2013). Recommender systems survey. Knowledge-based systems, 46, 109-132. https://doi.org/10.1016/j.knosys.2013.03.012
  • Bothmer, K., & Schlippe, T. (2022). Investigating natural language processing techniques for a recommendation system to support employers, job seekers and educational institutions. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium: 23rd International Conference, AIED 2022, Durham, UK, July 27–31, 2022, Proceedings, Part II (pp. 449-452). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-031-11647-6_90
  • Burke, R., Felfernig, A., & Göker, M. H. (2011). Recommender systems: An overview. Ai Magazine, 32(3), 13-18. https://doi.org/10.1609/aimag.v32i3.2361
  • Çano, E., & Morisio, M. (2017). Hybrid recommender systems: A systematic literature review. Intelligent data analysis, 21(6), 1487-152. https://doi.org/10.3233/IDA-163209
  • Çelik Ertuğrul, D., & Elçi, A. (2020). A survey on semanticized and personalized health recommender systems. Expert Systems, 37(4), e12519. https://doi.org/10.1111/exsy.12519
  • Chaaya, G., Métais, E., Abdo, J. B., Chiky, R., Demerjian, J., & Barbar, K. (2017, December). Evaluating non-personalized single-heuristic active learning strategies for collaborative filtering recommender systems. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) (pp. 593-600). IEEE. https://doi.org/10.1109/ICMLA.2017.00-96
  • Chai, Y., Wang, C., Wen, Y., & Yuan, X. (2016). A Hadoop-Based Database Querying Approach for Non-expert Users. In Asia-Pacific Web Conference (pp. 449-453). Springer, Cham.
  • Fang, D., Varshney, K. R., Wang, J., Ramamurthy, K. N., Mojsilovic, A., & Bauer, J. H. (2013). Quantifying and recommending expertise when new skills emerge. In 2013 IEEE 13th International Conference on Data Mining Workshops (pp. 672-679). IEEE. https://doi.org/10.1109/ICDMW.2013.33
  • Fkih, F. (2022). Similarity measures for Collaborative Filtering-based Recommender Systems: Review and experimental comparison. Journal of King Saud University-Computer and Information Sciences, 34(9), 7645-7669. https://doi.org/10.1016/j.jksuci.2021.09.014
  • Fusco, F., Vlachos, M., Vasileiadis, V., Wardatzky, K., & Schneider, J. (2019). RecoNet: An Interpretable Neural Architecture for Recommender Systems. In IJCAI (pp. 2343-2349). https://doi.org/10.24963/ijcai.2019/325
  • Goldberg, D., Nichols, D., Oki, B.M. and Terry, D. (1992). Using Collaborative Filtering to Weave an Information Tapestry. Communications of the ACM, 35, 61-70.http://dx.doi.org/10.1145/138859.138867.
  • González-Briones, A., Rivas, A., Chamoso, P., Casado-Vara, R., & Corchado, J. M. (2019). Case-based reasoning and agent-based job offer recommender system. In International Joint Conference SOCO’18-CISIS’18-ICEUTE’18: San Sebastián, Spain, June 6-8, 2018 Proceedings 13 (pp. 21-33). Springer International Publishing. https://doi.org/10.1007/978-3-319-94120-2_3
  • Guan, Z., Yu, B., & Liu, Y. (2019). Recruitment and Recommendation System Based on Intelligent Computing. In Proceedings of the 2019 5th International Conference on Computing and Data Engineering (pp. 77-80). https://doi.org/10.1145/3330530.3330532
  • Gunawardana, A., & Shani, G. (2009). A survey of accuracy evaluation metrics of recommendation tasks. Journal of Machine Learning Research, 10(12). https://doi.org/10.1145/1577069.1755883
  • Gunawardana, A., Shani, G., & Yogev, S. (2022). Evaluating recommender systems. In Recommender systems handbook (pp. 547-601). Springer, New York, NY. https://doi.org/10.1007/978-0-387-85820-3_8
  • Isinkaye, F. O., Folajimi, Y. O., & Ojokoh, B. A. (2015). Recommendation systems: Principles, methods and evaluation. Egyptian informatics journal, 16(3), 261-273. https://doi.org/10.1016/j.eij.2015.06.005
  • Kang, J. S., Shin, D. H., Baek, J. W., & Chung, K. (2019). Activity recommendation model using rank correlation for chronic stress management. Applied Sciences, 9(20), 4284. https://doi.org/10.3390/app9204284
  • Kumar, R., Verma, B. K., & Rastogi, S. S. (2014). Social popularity based SVD++ recommender system. International Journal of Computer Applications, 87(14). https://doi.org/10.5120/15279-4033
  • Kwieciński, R., Melniczak, G., & Górecki, T. (2023). Comparison of Real-Time and Batch Job Recommendations. IEEE Access, 11, 20553-20559. https://doi.org/10.1109/ACCESS.2023.3249356
  • Liang, F., & Wan, X. (2022). Job Matching Analysis Based on Text Mining and Multicriteria Decision-Making. Mathematical Problems in Engineering. https://doi.org/10.1155/2022/9245876
  • Liu, P., Ma, J., Wang, Y., Ma, L., & Huang, S. (2016). A context-aware method for top-k recommendation in smart TV. In Asia-Pacific Web Conference (pp. 150-161). Springer, Cham. https://doi.org/10.1007/978-3-319-45817-5_12
  • Lu, J., Wu, D., Mao, M., Wang, W., & Zhang, G. (2015). Recommender system application developments: a survey. Decision Support Systems, 74, 12-32. https://doi.org/10.1016/j.dss.2015.03.008
  • Lü, L., Medo, M., Yeung, C. H., Zhang, Y. C., Zhang, Z. K., & Zhou, T. (2012). Recommender systems. Physics reports, 519(1), 1-49. https://doi.org/10.1016/j.physrep.2012.02.006
  • Luo, Y., Xu, B., Cai, H., & Bu, F. (2014). A Hybrid User Profile Model for Personalized Recommender System with Linked Open Data. In 2014 Enterprise Systems Conference (pp. 243-248). IEEE. https://doi.org/10.1109/ES.2014.16
  • Malinowski, J., Keim, T., Wendt, O., & Weitzel, T. (2006). Matching people and jobs: A bilateral recommendation approach. In Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06) (Vol. 6, pp. 137c-137c). IEEE. https://doi.org/10.1109/HICSS.2006.266
  • Marlowe, A. N. (2021). Robot Recruiters: How Employers & Governments Must Confront the Discriminatory Effects of AI Hiring. J. High Tech. L., 22, 274.
  • Martin, F. J., Donaldson, J., Ashenfelter, A., Torrens, M., & Hangartner, R. (2011). The big promise of recommender systems. AI Magazine, 32(3), 19-27. https://doi.org/10.1609/aimag.v32i3.2360
  • Martinez-Gil, J., Paoletti, A. L., & Pichler, M. (2020). A novel approach for learning how to automatically match job offers and candidate profiles. Information Systems Frontiers, 22(6), 1265-1274. https://doi.org/10.1007/s10796-019-09929-7
  • Mathur, A., Juguru, S. K., & Eirinaki, M. (2019). A graph-based recommender system for food products. In 2019 First International Conference on Graph Computing (GC) (pp. 83-87). IEEE https://doi.org/10.1109/GC46384.2019.00020
  • McFee, B., & Lanckriet, G. R. (2010). Metric learning to rank. In Proceedings of the 27th international conference on machine learning (ICML-10) (pp. 775-782).
  • Mihalcea, R., Corley, C., & Strapparava, C. (2006). Corpus-based and knowledge-based measures of text semantic similarity. In Aaai (Vol. 6, No. 2006, pp. 775-780).
  • Mohamed, M. H., Khafagy, M. H., & Ibrahim, M. H. (2019). Recommender systems challenges and solutions survey. In 2019 international conference on innovative trends in computer engineering (ITCE) (pp. 149-155). IEEE. https://doi.org/10.1109/ITCE.2019.8646645
  • Negroponte, N. (1970). The architecture machine: toward a more human environment. The MIT Press. https://doi.org/10.7551/mitpress/8269.001.0001
  • Okfalisa, Siburian, R., Vitriani, Y., Rusnedy, H., Saktioto, & Yola, M. (2021). Job Training Recommendation System: Integrated Fuzzy AHP and TOPSIS Approach. In International Conference of Reliable Information and Communication Technology (pp. 84-94). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-98741-1_8
  • Painsky, A., & Wornell, G. (2018). On the universality of the logistic loss function. In 2018 IEEE International Symposium on Information Theory (ISIT) (pp. 936-940). IEEE. https://doi.org/10.48550/arXiv.1805.03804
  • Palomares, I., Porcel, C., Pizzato, L., Guy, I., & Herrera-Viedma, E. (2021). Reciprocal Recommender Systems: Analysis of state-of-art literature, challenges and opportunities towards social recommendation. Information Fusion, 69, 103-127. https://doi.org/10.1016/j.inffus.2020.12.001
  • Pan, Y., Zhang, Y., & Zhang, R. (2016). Combo-Recommendation Based on Potential Relevance of Items. In Asia-Pacific Web Conference (pp. 505-517). Springer, Cham. https://doi.org/10.1007/978-3-319-45817-5_55
  • Park, D. H., Kim, H. K., Choi, I. Y., & Kim, J. K. (2012). A literature review and classification of recommender systems research. Expert systems with applications, 39(11), 10059-10072. https://doi.org/10.1016/j.eswa.2012.02.038
  • Poriya, A., Bhagat, T., Patel, N., & Sharma, R. (2014). Non-personalized recommender systems and user-based collaborative recommender systems. Int. J. Appl. Inf. Sys, 6(9), 22-27. https://doi.org/10.5120/ijais14-451122
  • Portugal, I., Alencar, P., & Cowan, D. (2018). The use of machine learning algorithms in recommender systems: A systematic review. Expert Systems with Applications, 97, 205-227. https://doi.org/10.1016/j.eswa.2017.12.020
  • Prafajar, K. N., Vallyan, H., Candradewi, N. L. P. A., Edbert, I. S., & Suhartono, D. (2022). Multiclass job recommendation system in the IT field between classification and prediction method. In 2022 International Conference on Green Energy, Computing and Sustainable Technology (GECOST) (pp. 181-186). IEEE. https://doi.org/10.1109/GECOST55694.2022.10010659
  • Rimitha, S. R., Abburu, V., Kiranmai, A., Marimuthu, C., & Chandrasekaran, K. (2019). Improving Job Recommendation Using Ontological Modeling and User Profiles. In 2019 Fifteenth Int. Conference on Information Processing (ICINPRO) (pp. 1-8). IEEE. https://doi.org/10.1109/ICInPro47689.2019.9092271
  • Salton, G., and Buckley, C. (1997). Term weighting approaches in automatic text retrieval. In Readings in Information Retrieval. San Francisco, CA: Morgan Kaufmann Publishers. https://doi.org/10.1016/0306-4573(88)90021-0
  • Salton, G., and Lesk, M. (1971). Computer evaluation of indexing and text processing. Prentice Hall, Ing. Englewood Cliffs, New Jersey. 143–180. https://doi.org/10.1145/321439.321441
  • Salton, G., Singhal, A., Mitra, M., & Buckley, C. (1997). Automatic text structuring and summarization. Information processing & management, 33(2), 193-207. https://doi.org/10.1016/S0306-4573(96)00062-3
  • Schröder, G., Thiele, M., & Lehner, W. (2011). Setting goals and choosing metrics for recommender system evaluations. In UCERSTI2 workshop at the 5th ACM conference on recommender systems, Chicago, USA (Vol. 23, p. 53). https://doi.org/10.1007/s41870-018-0202-4
  • Selvi, C., & Sivasankar, E. (2018). A novel singularity based improved tanimoto similarity measure for effective recommendation using collaborative filtering. In 2018 8th International Conference on Cloud Computing, Data Science & Engineering (Confluence) (pp. 256-262). IEEE. https://doi.org/ 10.1109/CONFLUENCE.2018.8442697
  • Shakirova, E. (2017). Collaborative filtering for music recommender system. In 2017 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus) (pp. 548-550). IEEE. https://doi.org/ 10.1109/EICONRUS.2017.7910613
  • Shani, G., & Gunawardana, A. (2011). Evaluating recommendation systems. In Recommender systems handbook (pp. 257-297). Springer, Boston, MA. https://doi.org/10.1007/978-0-387-85820-3_8
  • Shervin Minaee. (2020). 20 Popular Machine Learning Metrics. Part 2: Ranking, & Statistical Metrics, Web Site: https://towardsdatascience.com/20-popular-machine-learning-metrics-part-2-ranking-statistical-metrics-22c3e5a937b6, LVD: 13-01-2023.
  • Silveira, T., Zhang, M., Lin, X., Liu, Y., & Ma, S. (2019). How good your recommender system is? A survey on evaluations in recommendation. International Journal of Machine Learning and Cybernetics, 10(5), 813-831. https://doi.org/10.1007/s13042-017-0762-9
  • Spearman, C. (1961). The proof and measurement of association between two things, The American Journal of Psychology 15 (1904) 72–101. https://doi.org/10.1037/11491-005
  • Tamburri, D. A., Van Den Heuvel, W. J., & Garriga, M. (2020). Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI) (pp. 391-394). IEEE. https://doi.org/10.48550/arXiv.2104.01966
  • Vijaysinh Lendave. (2021). How to Measure the Success of a Recommendation System?, in Developers Corner, October 24, , Web address: https://analyticsindiamag.com/how-to-measure-the-success-of-a-recommendation-system/, LVD: 13-01-2023. https://doi.org/10.1016/j.heliyon.2023.e15108
  • Vrontis, D., Christofi, M., Pereira, V., Tarba, S., Makrides, A., & Trichina, E. (2022). Artificial intelligence, robotics, advanced technologies and human resource management: a systematic review. The International Journal of Human Resource Management, 33(6), 1237-1266. https://doi.org/10.1080/09585192.2020.1871398
  • Wenxing, H., Yiwei, C., Jianwei, Q., & Yin, H. (2015). iHR+: A mobile reciprocal job recommender system. In 2015 10th International Conference on Computer Science & Education (ICCSE) (pp. 492-495). IEEE. https://doi.org/10.1109/ICCSE.2015.7250296
  • Wu, S., Sun, F., Zhang, W., Xie, X., & Cui, B. (2022). Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5), 1-37. https://doi.org/10.1145/3535101
  • Xin, X., Wang, D., Ding, Y., & Lini, C. (2016). FHSM: factored hybrid similarity methods for top-n recommender systems. In Asia-Pacific Web Conference (pp. 98-110). Springer, Cham. https://doi.org/10.1007/978-3-319-45817-5_8
  • Yao, Y. Y. (1995). Measuring retrieval effectiveness based on user preference of documents. Journal of the American Society for Information science, 46(2), 133-145. https://doi.org/10.1002/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z
  • Yi, P., Yang, C., Li, C., & Zhang, Y. (2016). A job recommendation method optimized by position descriptions and resume information. In 2016 IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC) (pp. 761-764). IEEE. https://doi.org/10.1109/IMCEC.2016.7867312
  • Yu, H., Liu, C., & Zhang, F. (2011). Reciprocal recommendation algorithm for the field of recruitment. Journal of Information & Computational Science, 8(16), 4061-4068.
  • Zhang, B., & Feng, Y. (2016). Improving temporal recommendation accuracy and diversity via long and short-term preference transfer and fusion models. In Asia-Pacific Web Conference (pp. 174-185). Springer, Cham. https://doi.org/10.1007/978-3-319-45817-5_14
  • Zhao, T., Wuyu, C., & Zhixiang, C. (2021). Summer Job Selection Model Based on Job Matching and Comprehensive Evaluation Algorithm. In 2021 2nd International Conference on Artificial Intelligence and Information Systems (pp. 1-5). https://doi.org/10.1145/3469213.3470394

ÖNERİ SİSTEMLERİNDE KULLANILAN PERFORMANS METRİKLERİNİN FİLTRELEME TEKNOLOJİLERİNE GÖRE DEĞERLENDİRİLMESİ: İŞ ÖNERİ SİSTEMLERİ ALANI ÜZERİNE BİR ARAŞTIRMA ÇALIŞMASI

Year 2024, Volume: 27 Issue: 3, 706 - 725, 03.09.2024

Selin Bitirim , Duygu Çelik Ertuğrul

https://doi.org/10.17780/ksujes.1410926

Abstract

Tavsiye Sistemleri (Recommendation Systems—RSs) sayesinde hemen hemen her sektörde (ör. e-ticaret, eğitim, eğlence, sağlık, insan kaynakları, reklamcılık, vb.) mevcut süreçlerin/operasyonların etkin bir biçimde yürütülebilmesi ve kullanıcının ilgisini çekebilecek öğelere öncelik verilmesi mümkün hale gelmiştir. RS'lerin katkısı ile, sektörel süreçlerin/hizmetlerin etkin şekilde yönetilmesi ve kullanıcılara kişiselleştirilmiş sonuçlar üretilmesi mümkündür. Bu çalışmada, RS ile ilgili araştırmaların gözden geçirilmesi, filtreleme teknikleri taksonomisinin ortaya çıkarılması ve geniş çapta rastlanan performans metriklerinin tespiti amaçlanmaktadır. Ayrıca, İnsan Kaynakları (İK) yönetiminin olmazsa olmazı olan İş Tavsiye Sistemleri bu çalışmada, araştırma sahası olarak seçilmiş olup performans metriklerinin ve öğe filtreleme yaklaşımlarının belirlenmesi planlanmıştır. RS mimarisi ve çözümleri üzerine, literatürden 2010-2023 yılları arasında yapılmış çeşitli çalışmalar ilgililik durumuna göre seçilmiş ve incelenmiştir. RS’lerde filtreleme teknikleri hiyerarşik olarak sınıflandırılmış ve performans değerlendirmelerinde kullanılan çoğunluk değerlendirme metrikleri saptanarak kategorize edilmiştir. Ayrıca, RS'lerden öğrenilen kazanımların İş Tavsiye Sistemleri’ndeki yansımaları araştırılmış ve IK alanındaki RS çözümleri/metrikleri ortaya konulmuştur. Son olarak, RS çözümleri üzerinde araştırma, geliştirme ve kalite değerlendirmeleri yapmak isteyen araştırmacılara, bu çalışmamız bir yol haritası niteliğindedir.

Keywords

Tavsiye Sistemleri İş Tavsiye Sistemleri Taksonomi Filtreleme Performans Metrikleri

Ethical Statement

Etik onay beyanı: Bu çalışma için resmi onay gerekli değildir.

Supporting Institution

DOĞU AKDENİZ ÜNİVERSİTESİ

References

  • Adomavicius, G., & Tuzhilin, A. (2011). Context-aware recommender systems. In Recommender systems handbook (pp. 217-253). Springer, Boston, MA. https://doi.org/10.1609/aimag.v32i3.2364
  • Al-Habaibeh, A., Watkins, M., Waried, K., & Javareshk, M. B. (2021). Challenges and opportunities of remotely working from home during Covid-19 pandemic. Global Transitions, 3, 99-108. https://doi.org/10.1016/j.glt.2021.11.001
  • Almalis, N. D., Tsihrintzis, G. A., Karagiannis, N., & Strati, A. D. (2015). FoDRA—A new content-based job recommendation algorithm for job seeking and recruiting. In 2015 6th International Conference on Information, Intelligence, Systems and Applications (IISA) (pp. 1-7). IEEE. https://doi.org/10.1109/IISA.2015.7388018
  • Al-Otaibi, S., & Ykhlef, M. (2017). Hybrid immunizing solution for job recommender system. Frontiers of Computer Science, 11(3), 511-527. https://doi.org/10.1007/s11704-016-5241-z
  • Al-Shamri, M. Y. H. (2016). User profiling approaches for demographic recommender systems. Knowledge-Based Systems, 100, 175-187. https://doi.org/10.1016/j.knosys.2016.03.006
  • Althbiti, A., Alshamrani, R., Alghamdi, T., Lee, S., & Ma, X. (2021). Addressing data sparsity in collaborative filtering-based recommender systems using clustering and artificial neural network. In 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC) (pp. 0218-0227). IEEE. https://doi.org/10.1109/CCWC51732.2021.9376008
  • Aouadni, I., & Rebai, A. (2017). Decision support system based on genetic algorithm and multi-criteria satisfaction analysis (MUSA) method for measuring job satisfaction. Annals of Operations Research, 256(1), 3-20. https://doi.org/10.1007/s10479-016-2154-z
  • Arita, S., Hiyama, A., & Hirose, M. (2017). Gber: A social matching app which utilizes time, place, and skills of workers and jobs. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (pp. 127-130). https://doi.org/10.1145/3022198.3026316
  • Avazpour, I., Pitakrat, T., Grunske, L., & Grundy, J. (2014). Dimensions and metrics for evaluating recommendation systems. In Recommendation systems in software engineering (pp. 245-273). Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-45135-5_10
  • Ayub, M., Ghazanfar, M. A., Maqsood, M., & Saleem, A. (2018). A Jaccard base similarity measure to improve performance of CF based recommender systems. In 2018 International conference on information networking (ICOIN) (pp. 1-6). IEEE. https://doi.org/10.1109/ICOIN.2018.8343073
  • Beel, J., Gipp, B., Langer, S., & Breitinger, C. (2016). Paper recommender systems: a literature survey. International Journal on Digital Libraries, 17(4), 305-338. https://doi.org/10.1007/s00799-015-0156-0
  • Benabderrahmane, S., Mellouli, N., & Lamolle, M. (2017). Predicting the users' clickstreams using time series representation, symbolic sequences, and deep learning: application on job offers recommendation tasks. In 2017 IEEE International Conference on Information Reuse and Integration (IRI) (pp. 436-443). IEEE. https://doi.org/10.1109/IRI.2017.54
  • Bhat, S. S., Pranav, P., Shashank, K. V., Raghunandan, A., & Mohan, B. R. (2022). Comparative Performance Evaluation of Web-Based Book Recommender Systems. In 2022 6th International Conference on Trends in Electronics and Informatics (ICOEI) (pp. 985-991). https://doi.org/IEEE. 10.1109/ICOEI53556.2022.9777116
  • Birjali, M., Kasri, M., & Beni-Hssane, A. (2021). A comprehensive survey on sentiment analysis: Approaches, challenges and trends. Knowledge-Based Systems, 226, 107134. ). https://doi.org/10.1016/j.knosys.2021.107134
  • Bobadilla, J., Ortega, F., Hernando, A., & Gutiérrez, A. (2013). Recommender systems survey. Knowledge-based systems, 46, 109-132. https://doi.org/10.1016/j.knosys.2013.03.012
  • Bothmer, K., & Schlippe, T. (2022). Investigating natural language processing techniques for a recommendation system to support employers, job seekers and educational institutions. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium: 23rd International Conference, AIED 2022, Durham, UK, July 27–31, 2022, Proceedings, Part II (pp. 449-452). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-031-11647-6_90
  • Burke, R., Felfernig, A., & Göker, M. H. (2011). Recommender systems: An overview. Ai Magazine, 32(3), 13-18. https://doi.org/10.1609/aimag.v32i3.2361
  • Çano, E., & Morisio, M. (2017). Hybrid recommender systems: A systematic literature review. Intelligent data analysis, 21(6), 1487-152. https://doi.org/10.3233/IDA-163209
  • Çelik Ertuğrul, D., & Elçi, A. (2020). A survey on semanticized and personalized health recommender systems. Expert Systems, 37(4), e12519. https://doi.org/10.1111/exsy.12519
  • Chaaya, G., Métais, E., Abdo, J. B., Chiky, R., Demerjian, J., & Barbar, K. (2017, December). Evaluating non-personalized single-heuristic active learning strategies for collaborative filtering recommender systems. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) (pp. 593-600). IEEE. https://doi.org/10.1109/ICMLA.2017.00-96
  • Chai, Y., Wang, C., Wen, Y., & Yuan, X. (2016). A Hadoop-Based Database Querying Approach for Non-expert Users. In Asia-Pacific Web Conference (pp. 449-453). Springer, Cham.
  • Fang, D., Varshney, K. R., Wang, J., Ramamurthy, K. N., Mojsilovic, A., & Bauer, J. H. (2013). Quantifying and recommending expertise when new skills emerge. In 2013 IEEE 13th International Conference on Data Mining Workshops (pp. 672-679). IEEE. https://doi.org/10.1109/ICDMW.2013.33
  • Fkih, F. (2022). Similarity measures for Collaborative Filtering-based Recommender Systems: Review and experimental comparison. Journal of King Saud University-Computer and Information Sciences, 34(9), 7645-7669. https://doi.org/10.1016/j.jksuci.2021.09.014
  • Fusco, F., Vlachos, M., Vasileiadis, V., Wardatzky, K., & Schneider, J. (2019). RecoNet: An Interpretable Neural Architecture for Recommender Systems. In IJCAI (pp. 2343-2349). https://doi.org/10.24963/ijcai.2019/325
  • Goldberg, D., Nichols, D., Oki, B.M. and Terry, D. (1992). Using Collaborative Filtering to Weave an Information Tapestry. Communications of the ACM, 35, 61-70.http://dx.doi.org/10.1145/138859.138867.
  • González-Briones, A., Rivas, A., Chamoso, P., Casado-Vara, R., & Corchado, J. M. (2019). Case-based reasoning and agent-based job offer recommender system. In International Joint Conference SOCO’18-CISIS’18-ICEUTE’18: San Sebastián, Spain, June 6-8, 2018 Proceedings 13 (pp. 21-33). Springer International Publishing. https://doi.org/10.1007/978-3-319-94120-2_3
  • Guan, Z., Yu, B., & Liu, Y. (2019). Recruitment and Recommendation System Based on Intelligent Computing. In Proceedings of the 2019 5th International Conference on Computing and Data Engineering (pp. 77-80). https://doi.org/10.1145/3330530.3330532
  • Gunawardana, A., & Shani, G. (2009). A survey of accuracy evaluation metrics of recommendation tasks. Journal of Machine Learning Research, 10(12). https://doi.org/10.1145/1577069.1755883
  • Gunawardana, A., Shani, G., & Yogev, S. (2022). Evaluating recommender systems. In Recommender systems handbook (pp. 547-601). Springer, New York, NY. https://doi.org/10.1007/978-0-387-85820-3_8
  • Isinkaye, F. O., Folajimi, Y. O., & Ojokoh, B. A. (2015). Recommendation systems: Principles, methods and evaluation. Egyptian informatics journal, 16(3), 261-273. https://doi.org/10.1016/j.eij.2015.06.005
  • Kang, J. S., Shin, D. H., Baek, J. W., & Chung, K. (2019). Activity recommendation model using rank correlation for chronic stress management. Applied Sciences, 9(20), 4284. https://doi.org/10.3390/app9204284
  • Kumar, R., Verma, B. K., & Rastogi, S. S. (2014). Social popularity based SVD++ recommender system. International Journal of Computer Applications, 87(14). https://doi.org/10.5120/15279-4033
  • Kwieciński, R., Melniczak, G., & Górecki, T. (2023). Comparison of Real-Time and Batch Job Recommendations. IEEE Access, 11, 20553-20559. https://doi.org/10.1109/ACCESS.2023.3249356
  • Liang, F., & Wan, X. (2022). Job Matching Analysis Based on Text Mining and Multicriteria Decision-Making. Mathematical Problems in Engineering. https://doi.org/10.1155/2022/9245876
  • Liu, P., Ma, J., Wang, Y., Ma, L., & Huang, S. (2016). A context-aware method for top-k recommendation in smart TV. In Asia-Pacific Web Conference (pp. 150-161). Springer, Cham. https://doi.org/10.1007/978-3-319-45817-5_12
  • Lu, J., Wu, D., Mao, M., Wang, W., & Zhang, G. (2015). Recommender system application developments: a survey. Decision Support Systems, 74, 12-32. https://doi.org/10.1016/j.dss.2015.03.008
  • Lü, L., Medo, M., Yeung, C. H., Zhang, Y. C., Zhang, Z. K., & Zhou, T. (2012). Recommender systems. Physics reports, 519(1), 1-49. https://doi.org/10.1016/j.physrep.2012.02.006
  • Luo, Y., Xu, B., Cai, H., & Bu, F. (2014). A Hybrid User Profile Model for Personalized Recommender System with Linked Open Data. In 2014 Enterprise Systems Conference (pp. 243-248). IEEE. https://doi.org/10.1109/ES.2014.16
  • Malinowski, J., Keim, T., Wendt, O., & Weitzel, T. (2006). Matching people and jobs: A bilateral recommendation approach. In Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS'06) (Vol. 6, pp. 137c-137c). IEEE. https://doi.org/10.1109/HICSS.2006.266
  • Marlowe, A. N. (2021). Robot Recruiters: How Employers & Governments Must Confront the Discriminatory Effects of AI Hiring. J. High Tech. L., 22, 274.
  • Martin, F. J., Donaldson, J., Ashenfelter, A., Torrens, M., & Hangartner, R. (2011). The big promise of recommender systems. AI Magazine, 32(3), 19-27. https://doi.org/10.1609/aimag.v32i3.2360
  • Martinez-Gil, J., Paoletti, A. L., & Pichler, M. (2020). A novel approach for learning how to automatically match job offers and candidate profiles. Information Systems Frontiers, 22(6), 1265-1274. https://doi.org/10.1007/s10796-019-09929-7
  • Mathur, A., Juguru, S. K., & Eirinaki, M. (2019). A graph-based recommender system for food products. In 2019 First International Conference on Graph Computing (GC) (pp. 83-87). IEEE https://doi.org/10.1109/GC46384.2019.00020
  • McFee, B., & Lanckriet, G. R. (2010). Metric learning to rank. In Proceedings of the 27th international conference on machine learning (ICML-10) (pp. 775-782).
  • Mihalcea, R., Corley, C., & Strapparava, C. (2006). Corpus-based and knowledge-based measures of text semantic similarity. In Aaai (Vol. 6, No. 2006, pp. 775-780).
  • Mohamed, M. H., Khafagy, M. H., & Ibrahim, M. H. (2019). Recommender systems challenges and solutions survey. In 2019 international conference on innovative trends in computer engineering (ITCE) (pp. 149-155). IEEE. https://doi.org/10.1109/ITCE.2019.8646645
  • Negroponte, N. (1970). The architecture machine: toward a more human environment. The MIT Press. https://doi.org/10.7551/mitpress/8269.001.0001
  • Okfalisa, Siburian, R., Vitriani, Y., Rusnedy, H., Saktioto, & Yola, M. (2021). Job Training Recommendation System: Integrated Fuzzy AHP and TOPSIS Approach. In International Conference of Reliable Information and Communication Technology (pp. 84-94). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-98741-1_8
  • Painsky, A., & Wornell, G. (2018). On the universality of the logistic loss function. In 2018 IEEE International Symposium on Information Theory (ISIT) (pp. 936-940). IEEE. https://doi.org/10.48550/arXiv.1805.03804
  • Palomares, I., Porcel, C., Pizzato, L., Guy, I., & Herrera-Viedma, E. (2021). Reciprocal Recommender Systems: Analysis of state-of-art literature, challenges and opportunities towards social recommendation. Information Fusion, 69, 103-127. https://doi.org/10.1016/j.inffus.2020.12.001
  • Pan, Y., Zhang, Y., & Zhang, R. (2016). Combo-Recommendation Based on Potential Relevance of Items. In Asia-Pacific Web Conference (pp. 505-517). Springer, Cham. https://doi.org/10.1007/978-3-319-45817-5_55
  • Park, D. H., Kim, H. K., Choi, I. Y., & Kim, J. K. (2012). A literature review and classification of recommender systems research. Expert systems with applications, 39(11), 10059-10072. https://doi.org/10.1016/j.eswa.2012.02.038
  • Poriya, A., Bhagat, T., Patel, N., & Sharma, R. (2014). Non-personalized recommender systems and user-based collaborative recommender systems. Int. J. Appl. Inf. Sys, 6(9), 22-27. https://doi.org/10.5120/ijais14-451122
  • Portugal, I., Alencar, P., & Cowan, D. (2018). The use of machine learning algorithms in recommender systems: A systematic review. Expert Systems with Applications, 97, 205-227. https://doi.org/10.1016/j.eswa.2017.12.020
  • Prafajar, K. N., Vallyan, H., Candradewi, N. L. P. A., Edbert, I. S., & Suhartono, D. (2022). Multiclass job recommendation system in the IT field between classification and prediction method. In 2022 International Conference on Green Energy, Computing and Sustainable Technology (GECOST) (pp. 181-186). IEEE. https://doi.org/10.1109/GECOST55694.2022.10010659
  • Rimitha, S. R., Abburu, V., Kiranmai, A., Marimuthu, C., & Chandrasekaran, K. (2019). Improving Job Recommendation Using Ontological Modeling and User Profiles. In 2019 Fifteenth Int. Conference on Information Processing (ICINPRO) (pp. 1-8). IEEE. https://doi.org/10.1109/ICInPro47689.2019.9092271
  • Salton, G., and Buckley, C. (1997). Term weighting approaches in automatic text retrieval. In Readings in Information Retrieval. San Francisco, CA: Morgan Kaufmann Publishers. https://doi.org/10.1016/0306-4573(88)90021-0
  • Salton, G., and Lesk, M. (1971). Computer evaluation of indexing and text processing. Prentice Hall, Ing. Englewood Cliffs, New Jersey. 143–180. https://doi.org/10.1145/321439.321441
  • Salton, G., Singhal, A., Mitra, M., & Buckley, C. (1997). Automatic text structuring and summarization. Information processing & management, 33(2), 193-207. https://doi.org/10.1016/S0306-4573(96)00062-3
  • Schröder, G., Thiele, M., & Lehner, W. (2011). Setting goals and choosing metrics for recommender system evaluations. In UCERSTI2 workshop at the 5th ACM conference on recommender systems, Chicago, USA (Vol. 23, p. 53). https://doi.org/10.1007/s41870-018-0202-4
  • Selvi, C., & Sivasankar, E. (2018). A novel singularity based improved tanimoto similarity measure for effective recommendation using collaborative filtering. In 2018 8th International Conference on Cloud Computing, Data Science & Engineering (Confluence) (pp. 256-262). IEEE. https://doi.org/ 10.1109/CONFLUENCE.2018.8442697
  • Shakirova, E. (2017). Collaborative filtering for music recommender system. In 2017 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus) (pp. 548-550). IEEE. https://doi.org/ 10.1109/EICONRUS.2017.7910613
  • Shani, G., & Gunawardana, A. (2011). Evaluating recommendation systems. In Recommender systems handbook (pp. 257-297). Springer, Boston, MA. https://doi.org/10.1007/978-0-387-85820-3_8
  • Shervin Minaee. (2020). 20 Popular Machine Learning Metrics. Part 2: Ranking, & Statistical Metrics, Web Site: https://towardsdatascience.com/20-popular-machine-learning-metrics-part-2-ranking-statistical-metrics-22c3e5a937b6, LVD: 13-01-2023.
  • Silveira, T., Zhang, M., Lin, X., Liu, Y., & Ma, S. (2019). How good your recommender system is? A survey on evaluations in recommendation. International Journal of Machine Learning and Cybernetics, 10(5), 813-831. https://doi.org/10.1007/s13042-017-0762-9
  • Spearman, C. (1961). The proof and measurement of association between two things, The American Journal of Psychology 15 (1904) 72–101. https://doi.org/10.1037/11491-005
  • Tamburri, D. A., Van Den Heuvel, W. J., & Garriga, M. (2020). Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI) (pp. 391-394). IEEE. https://doi.org/10.48550/arXiv.2104.01966
  • Vijaysinh Lendave. (2021). How to Measure the Success of a Recommendation System?, in Developers Corner, October 24, , Web address: https://analyticsindiamag.com/how-to-measure-the-success-of-a-recommendation-system/, LVD: 13-01-2023. https://doi.org/10.1016/j.heliyon.2023.e15108
  • Vrontis, D., Christofi, M., Pereira, V., Tarba, S., Makrides, A., & Trichina, E. (2022). Artificial intelligence, robotics, advanced technologies and human resource management: a systematic review. The International Journal of Human Resource Management, 33(6), 1237-1266. https://doi.org/10.1080/09585192.2020.1871398
  • Wenxing, H., Yiwei, C., Jianwei, Q., & Yin, H. (2015). iHR+: A mobile reciprocal job recommender system. In 2015 10th International Conference on Computer Science & Education (ICCSE) (pp. 492-495). IEEE. https://doi.org/10.1109/ICCSE.2015.7250296
  • Wu, S., Sun, F., Zhang, W., Xie, X., & Cui, B. (2022). Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5), 1-37. https://doi.org/10.1145/3535101
  • Xin, X., Wang, D., Ding, Y., & Lini, C. (2016). FHSM: factored hybrid similarity methods for top-n recommender systems. In Asia-Pacific Web Conference (pp. 98-110). Springer, Cham. https://doi.org/10.1007/978-3-319-45817-5_8
  • Yao, Y. Y. (1995). Measuring retrieval effectiveness based on user preference of documents. Journal of the American Society for Information science, 46(2), 133-145. https://doi.org/10.1002/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z
  • Yi, P., Yang, C., Li, C., & Zhang, Y. (2016). A job recommendation method optimized by position descriptions and resume information. In 2016 IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC) (pp. 761-764). IEEE. https://doi.org/10.1109/IMCEC.2016.7867312
  • Yu, H., Liu, C., & Zhang, F. (2011). Reciprocal recommendation algorithm for the field of recruitment. Journal of Information & Computational Science, 8(16), 4061-4068.
  • Zhang, B., & Feng, Y. (2016). Improving temporal recommendation accuracy and diversity via long and short-term preference transfer and fusion models. In Asia-Pacific Web Conference (pp. 174-185). Springer, Cham. https://doi.org/10.1007/978-3-319-45817-5_14
  • Zhao, T., Wuyu, C., & Zhixiang, C. (2021). Summer Job Selection Model Based on Job Matching and Comprehensive Evaluation Algorithm. In 2021 2nd International Conference on Artificial Intelligence and Information Systems (pp. 1-5). https://doi.org/10.1145/3469213.3470394
There are 77 citations in total.

Details

Primary Language Turkish
Subjects Business Process Management, Decision Support and Group Support Systems, Information Systems (Other), Performance Evaluation
Journal Section Computer Engineering
Authors

Selin Bitirim 0000-0002-3575-5855

Duygu Çelik Ertuğrul 0000-0003-1380-705X

Publication Date September 3, 2024
Submission Date December 27, 2023
Acceptance Date February 17, 2024
Published in Issue Year 2024Volume: 27 Issue: 3

Cite

APA Bitirim, S., & Çelik Ertuğrul, D. (2024). ÖNERİ SİSTEMLERİNDE KULLANILAN PERFORMANS METRİKLERİNİN FİLTRELEME TEKNOLOJİLERİNE GÖRE DEĞERLENDİRİLMESİ: İŞ ÖNERİ SİSTEMLERİ ALANI ÜZERİNE BİR ARAŞTIRMA ÇALIŞMASI. Kahramanmaraş Sütçü İmam Üniversitesi Mühendislik Bilimleri Dergisi, 27(3), 706-725. https://doi.org/10.17780/ksujes.1410926
 Download Cover Image