Login Register Cart 0
Generic placeholder image

Recent Advances in Computer Science and Communications

Editor-in-Chief

ISSN (Print): 2666-2558
ISSN (Online): 2666-2566

Back
Journal Home About Journal Editorial Board Current Issue Volumes/Issues
Subscribe
Research Article

Supervised Rank Aggregation (SRA): A Novel Rank Aggregation Approach for Ensemble-based Feature Selection

Author(s): Rahi Jain and Wei Xu*

Volume 17, Issue 3, 2024

Published on: 03 January, 2024

Article ID: e030124225206 Pages: 11

DOI: 10.2174/0126662558277567231201063458

Price: $65

conference banner
Abstract

Background: Feature selection (FS) is critical for high dimensional data analysis. Ensemble based feature selection (EFS) is a commonly used approach to develop FS techniques. Rank aggregation (RA) is an essential step in EFS where results from multiple models are pooled to estimate feature importance. However, the literature primarily relies on static rule-based methods to perform this step which may not always provide an optimal feature set. The objective of this study is to improve the EFS performance using dynamic learning in RA step.

Method: This study proposes a novel Supervised Rank Aggregation (SRA) approach to allow RA step to dynamically learn and adapt the model aggregation rules to obtain feature importance.

Results: We evaluate the performance of the algorithm using simulation studies and implement it into real research studies, and compare its performance with various existing RA methods. The proposed SRA method provides better or at par performance in terms of feature selection and predictive performance of the model compared to existing methods.

Conclusion: SRA method provides an alternative to the existing approaches of RA for EFS. While the current study is limited to the continuous cross-sectional outcome, other endpoints such as longitudinal, categorical, and time-to-event data could also be used.

Graphical Abstract

[1]
R. Bellman, "5. Dynamic programming", Mathematics in Science and Engineering, vol. 40, pp. 101-137, 1967.
[http://dx.doi.org/10.1016/S0076-5392(08)61063-2]
[2]
J. Fan, and R. Li, "Statistical challenges with high dimensionality: feature selection in knowledge discovery", In Proceedings of the International Congress of Mathematicians Madrid, Madridpp. 595-622
[http://dx.doi.org/10.4171/022-3/31]
[3]
S. Ayesha, M.K. Hanif, and R. Talib, "Overview and comparative study of dimensionality reduction techniques for high dimensional data", Inf. Fusion, vol. 59, pp. 44-58, 2020.
[http://dx.doi.org/10.1016/j.inffus.2020.01.005]
[4]
M. Piles, R. Bergsma, D. Gianola, H. Gilbert, and L. Tusell, "Feature selection stability and accuracy of prediction models for genomic prediction of residual feed intake in pigs using machine learning", Front. Genet., vol. 12, p. 611506, 2021.
[http://dx.doi.org/10.3389/fgene.2021.611506] [PMID: 33692825]
[5]
G. Healy, E. Salinas-Miranda, R. Jain, X. Dong, D. Deniffel, and and A. Borgida, ", "Pre-operative radiomics model for prognostication in resectable pancreatic adenocarcinoma with external validation",", Eur. Radiol., vol. vol. 32, no. no 4, pp. 2492-2505, 2022.
[6]
S. Walter, and H. Tiemeier, "Variable selection: Current practice in epidemiological studies", Eur. J. Epidemiol., vol. 24, no. 12, pp. 733-736, 2009.
[http://dx.doi.org/10.1007/s10654-009-9411-2] [PMID: 19967429]
[7]
G. Heinze, C. Wallisch, and D. Dunkler, "Variable selection-a review and recommendations for the practicing statistician", Biom. J., vol. 60, no. 3, pp. 431-449, 2018.
[http://dx.doi.org/10.1002/bimj.201700067] [PMID: 29292533]
[8]
R. Jain, and W. Xu, "HDSI: High dimensional selection with interactions algorithm on feature selection and testing", PLoS One, vol. 16, no. 2, p. e0246159, 2021.
[http://dx.doi.org/10.1371/journal.pone.0246159] [PMID: 33592034]
[9]
T.J. Mitchell, and J.J. Beauchamp, "Bayesian variable selection in linear regression", J. Am. Stat. Assoc., vol. 83, no. 404, pp. 1023-1032, 1988.
[http://dx.doi.org/10.1080/01621459.1988.10478694]
[10]
G. Zycinski, A. Barla, M. Squillario, T. Sanavia, B.D. Camillo, and A. Verri, "Knowledge Driven Variable Selection (KDVS)-a new approach to enrichment analysis of gene signatures obtained from high-throughput data", Source Code Biol. Med., vol. 8, no. 1, p. 2, 2013.
[http://dx.doi.org/10.1186/1751-0473-8-2] [PMID: 23302187]
[11]
N. Pudjihartono, T. Fadason, A.W. Kempa-Liehr, and J.M. O’Sullivan, "A review of feature selection methods for machine learning-based disease risk prediction", Frontiers in Bioinformatics, vol. 2, p. 927312, 2022.
[http://dx.doi.org/10.3389/fbinf.2022.927312] [PMID: 36304293]
[12]
G. Koop, and D. Korobilis, "Bayesian dynamic variable selection in high dimensions", Int. Econ. Rev., vol. 64, no. 3, pp. 1047-1074, 2023.
[http://dx.doi.org/10.1111/iere.12623]
[13]
P. Yang, H. Huang, and C. Liu, "Feature selection revisited in the single-cell era", Genome Biol., vol. 22, no. 1, p. 321, 2021.
[http://dx.doi.org/10.1186/s13059-021-02544-3] [PMID: 34847932]
[14]
P. Dhal, and C. Azad, "A comprehensive survey on feature selection in the various fields of machine learning", Appl. Intell., vol. 51, pp. 1-39, 2021.
[http://dx.doi.org/10.1007/s10489-021-02550-9]
[15]
A.R.S. Parmezan, H.D. Lee, N. Spolaôr, and F.C. Wu, "Automatic recommendation of feature selection algorithms based on dataset characteristics", Expert Syst. Appl., vol. 185, p. 115589, 2021.
[http://dx.doi.org/10.1016/j.eswa.2021.115589]
[16]
A. Bommert, T. Welchowski, M. Schmid, and J. Rahnenführer, "Benchmark of filter methods for feature selection in high-dimensional gene expression survival data", Brief. Bioinform., vol. 23, no. 1, p. bbab354, 2022.
[http://dx.doi.org/10.1093/bib/bbab354] [PMID: 34498681]
[17]
S. Chormunge, and S. Jena, "Correlation based feature selection with clustering for high dimensional data", JESIT, vol. 5, no. 3, pp. 542-549, 2018.
[http://dx.doi.org/10.1016/j.jesit.2017.06.004]
[18]
M. Dash, H. Liu, and and J. Yao, ", "Dimensionality reduction of unsupervised data", ", In Proceedings Ninth IEEE International Conference on Tools with Artificial Intelligence 1997pp., California, USApp. 532-9
[http://dx.doi.org/10.1201/9781315171401-9]
[19]
R. Zhang, F. Nie, X. Li, and X. Wei, "Feature selection with multi-view data: A survey", Inf. Fusion, vol. 50, pp. 158-167, 2019.
[http://dx.doi.org/10.1016/j.inffus.2018.11.019]
[20]
R. Kohavi, and G.H. John, "Wrappers for feature subset selection", Artif. Intell., vol. 97, no. 1-2, pp. 273-324, 1997.
[http://dx.doi.org/10.1016/S0004-3702(97)00043-X]
[21]
O. Tarkhaneh, T.T. Nguyen, and S. Mazaheri, "A novel wrapper-based feature subset selection method using modified binary differential evolution algorithm", Inf. Sci., vol. 565, pp. 278-305, 2021.
[http://dx.doi.org/10.1016/j.ins.2021.02.061]
[22]
M. Alweshah, S. Alkhalaileh, M.A. Al-Betar, and A.A. Bakar, "Coronavirus herd immunity optimizer with greedy crossover for feature selection in medical diagnosis", Knowl. Base. Syst., vol. 235, p. 107629, 2022.
[http://dx.doi.org/10.1016/j.knosys.2021.107629] [PMID: 34728909]
[23]
S. Kaur, Y. Kumar, A. Koul, and S. Kumar Kamboj, A Systematic Review on Metaheuristic Optimization Techniques for Feature Selections in Disease Diagnosis: Open Issues and Challenges., vol. 30. Springer Netherlands, 2023.
[http://dx.doi.org/10.1007/s11831-022-09853-1]
[24]
R. Tibshirani, "Regression shrinkage and selection via the lasso: A retrospective", J. R. Stat. Soc. Series B Stat. Methodol., vol. 73, no. 3, pp. 273-282, 2011.
[http://dx.doi.org/10.1111/j.1467-9868.2011.00771.x]
[25]
H. Chun, and and S. Keleş, ", "Sparse partial least squares regression for simultaneous dimension reduction and variable selection",", J. R. Stat. Soc. Series B Stat. Methodol.,, vol. vol. 72, pp. , pp. 3-25, .no. 1,
2010. [http://dx.doi.org/10.1111/j.1467-9868.2009.00723.x] [PMID: 20107611]
[26]
R. Jain, and W. Xu, "RHDSI: A novel dimensionality reduction based algorithm on high dimensional feature selection with interactions", Inf. Sci., vol. 574, pp. 590-605, 2021.
[http://dx.doi.org/10.1016/j.ins.2021.06.096]
[27]
"T.N. Lal, O. Chapelle, and J. Weston, "Embedded Methods", In: I. Guyon, M. Nikravesh, S. Gunn, and L.A. Zadeh, Eds.,", Feature Extraction: Foundations and Applications., vol. 165. Springer: Berlin, Heidelberg,, p. pp. 137-165, 2006.
[http://dx.doi.org/10.1007/978-3-540-35488-8_6]
[28]
Y. Guo, L. Duan, C. Xie, and W. Zhang, "A self-supervised feature selection algorithm for spectral embedding based on block HSIC lasso", J. Phys. Conf. Ser., vol. 2258, no. 1, p. 012026, 2022.
[http://dx.doi.org/10.1088/1742-6596/2258/1/012026]
[29]
M. Vatankhah, and M. Momenzadeh, "Self-regularized Lasso for selection of most informative features in microarray cancer classification", Multimedia Tools Appl., vol. 82, pp. 1-6, 2023.
[http://dx.doi.org/10.1007/s11042-023-15207-1]
[30]
Y. Li, U. Mansmann, S. Du, and R. Hornung, "Benchmark study of feature selection strategies for multiomics data", BMC Bioinformatics, vol. 23, no. 1, p. 412, 2022.
[http://dx.doi.org/10.1186/s12859-022-04962-x] [PMID: 36199022]
[31]
E. Hancer, B. Xue, and M. Zhang, "A survey on feature selection approaches for clustering", Artif. Intell. Rev., vol. 53, no. 6, pp. 4519-4545, 2020.
[http://dx.doi.org/10.1007/s10462-019-09800-w]
[32]
Z.J. Viharos, K.B. Kis, Á. Fodor, and M.I. Büki, "Adaptive, Hybrid Feature Selection (AHFS)", Pattern Recognit., vol. 116, p. 107932, 2021.
[http://dx.doi.org/10.1016/j.patcog.2021.107932]
[33]
M.A. Ganjei, and R. Boostani, "A hybrid feature selection scheme for high-dimensional data", Eng. Appl. Artif. Intell., vol. 113, p. 104894, 2022.
[http://dx.doi.org/10.1016/j.engappai.2022.104894]
[34]
K. Wang, Y. An, J. Zhou, Y. Long, and X. Chen, "A novel multi-level feature selection method for radiomics", Alex. Eng. J., vol. 66, pp. 993-999, 2023.
[http://dx.doi.org/10.1016/j.aej.2022.10.069]
[35]
E. Tasci, S. Jagasia, Y. Zhuge, K. Camphausen, and A.V. Krauze, "GradWise: A novel application of a rank-based weighted hybrid filter and embedded feature selection method for glioma grading with clinical and molecular characteristics", Cancers, vol. 15, no. 18, p. 4628, 2023.
[http://dx.doi.org/10.3390/cancers15184628] [PMID: 37760597]
[36]
B. Seijo-Pardo, I. Porto-Díaz, V. Bolón-Canedo, and A. Alonso-Betanzos, "Ensemble feature selection: Homogeneous and heterogeneous approaches", Knowl. Base. Syst., vol. 118, pp. 124-139, 2017.
[http://dx.doi.org/10.1016/j.knosys.2016.11.017]
[37]
M. Hosni, A. Idri, and A. Abran, "On the value of filter feature selection techniques in homogeneous ensembles effort estimation", J. Softw., vol. 33, no. 6, p. e2343, 2021.
[http://dx.doi.org/10.1002/smr.2343]
[38]
M. Mera-Gaona, D.M. López, R. Vargas-Canas, and U. Neumann, "Framework for the ensemble of feature selection methods", Appl. Sci., vol. 11, no. 17, p. 8122, 2021.
[http://dx.doi.org/10.3390/app11178122]
[39]
C.F. Tsai, and Y.T. Sung, "Ensemble feature selection in high dimension, low sample size datasets: Parallel and serial combination approaches", Knowl. Base. Syst., vol. 203, p. 106097, 2020.
[http://dx.doi.org/10.1016/j.knosys.2020.106097]
[40]
X. Song, L.R. Waitman, Y. Hu, A.S.L. Yu, D. Robins, and M. Liu, "Robust clinical marker identification for diabetic kidney disease with ensemble feature selection", J. Am. Med. Inform. Assoc., vol. 26, no. 3, pp. 242-253, 2019.
[http://dx.doi.org/10.1093/jamia/ocy165] [PMID: 30602020]
[41]
A.O. Balogun, S. Basri, S. Mahamad, L.F. Capretz, A.A. Imam, M.A. Almomani, V.E. Adeyemo, and G. Kumar, "A novel rank aggregation-based hybrid multifilter wrapper feature selection method in software defect prediction", Comput. Intell. Neurosci., vol. 2021, pp. 1-19, 2021.
[http://dx.doi.org/10.1155/2021/5069016] [PMID: 34868291]
[42]
UM. Khaire, and R. Dhanalakshmi, "“Stability of feature selection algorithm: A review”, J. King Saud Univ. -", Comput. Inf. Sci, vol. 34, no. 4, 2022.
[http://dx.doi.org/10.1016/j.jksuci.2019.06.012]
[43]
H. Tao, "A label-relevance multi-direction interaction network with enhanced deformable convolution for forest smoke recognition", Expert Syst. Appl., vol. 236, p. 121383, 2024.
[http://dx.doi.org/10.1016/j.eswa.2023.121383]
[44]
H. Tao, Q. Duan, M. Lu, and Z. Hu, "Learning discriminative feature representation with pixel-level supervision for forest smoke recognition", Pattern Recognit., vol. 143, p. 109761, 2023.
[http://dx.doi.org/10.1016/j.patcog.2023.109761]
[45]
H. Tao, Q. Duan, and J. An, "An Adaptive interference removal framework for video person re-identification", IEEE Trans. Circ. Syst. Video Tech., vol. 33, no. 9, pp. 5148-5159, 2023.
[http://dx.doi.org/10.1109/TCSVT.2023.3250464]
[46]
N. Noureldien, and S. Mohmoud, "The efficiency of aggregation methods in ensemble filter feature selection models", Transactions on Machine Learning and Artificial Intelligence, vol. 9, no. 4, pp. 39-51, 2021.
[http://dx.doi.org/10.14738/tmlai.94.10101]
[47]
R. Kolde, S. Laur, P. Adler, and J. Vilo, "Robust rank aggregation for gene list integration and meta-analysis", Bioinformatics, vol. 28, no. 4, pp. 573-580, 2012.
[http://dx.doi.org/10.1093/bioinformatics/btr709] [PMID: 22247279]
[48]
B. van Ginneken, "Fifty years of computer analysis in chest imaging: Rule-based, machine learning, deep learning", Radiological Phys. Technol., vol. 10, no. 1, pp. 23-32, 2017.
[http://dx.doi.org/10.1007/s12194-017-0394-5] [PMID: 28211015]
[49]
R.M. Cronin, D. Fabbri, J.C. Denny, S.T. Rosenbloom, and G.P. Jackson, "A comparison of rule-based and machine learning approaches for classifying patient portal messages", Int. J. Med. Inform., vol. 105, pp. 110-120, 2017.
[http://dx.doi.org/10.1016/j.ijmedinf.2017.06.004] [PMID: 28750904]
[50]
A. Lopez-Rincon, L. Mendoza-Maldonado, M. Martinez-Archundia, A. Schönhuth, A.D. Kraneveld, J. Garssen, and A. Tonda, "Machine learning-based ensemble recursive feature selection of circulating mirnas for cancer tumor classification", Cancers, vol. 12, no. 7, p. 1785, 2020.
[http://dx.doi.org/10.3390/cancers12071785] [PMID: 32635415]
[51]
R. Tibshirani, "Regression shrinkage and selection via the lasso", J. R. Stat. Soc. B, vol. 58, no. 1, pp. 267-288, 1996.
[http://dx.doi.org/10.1111/j.2517-6161.1996.tb02080.x]
[52]
A.E. Hoerl, and R.W. Kennard, "Ridge regression: Biased estimation for nonorthogonal problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
[http://dx.doi.org/10.1080/00401706.1970.10488634]
[53]
L. Breiman, "Random Forests", Mach. Learn., vol. 45, no. 1, pp. 5-32, 2001.
[http://dx.doi.org/10.1023/A:1010933404324]
[54]
S. Fahimifar, K. Mousavi, F. Mozaffari, and M. Ausloos, "Identification of the most important external features of highly cited scholarly papers through 3 (i.e., Ridge, Lasso, and Boruta) feature selection data mining methods", Qual. Quant., vol. 57, no. 4, pp. 3685-3712, 2023.
[http://dx.doi.org/10.1007/s11135-022-01480-z]
[55]
K. Koyama, K. Kiritoshi, T. Okawachi, and T. Izumitani, "Effective nonlinear feature selection method based on HSIC lasso and with variational inference", Proc. Mach. Learn. Res., vol. 151, pp. 10407-10421, 2022.
[56]
J. Jo, S. Jung, J. Park, Y. Kim, and M. Kang, "Hi-LASSO: High-performance python and apache spark packages for feature selection with high-dimensional data", PLoS One, vol. 17, no. 12, p. e0278570, 2022.
[http://dx.doi.org/10.1371/journal.pone.0278570] [PMID: 36455001]
[57]
M. Alduailij, Q.W. Khan, M. Tahir, M. Sardaraz, M. Alduailij, and F. Malik, "Machine-learning-based ddos attack detection using mutual information and random forest feature importance method", Symmetry, vol. 14, no. 6, p. 1095, 2022.
[http://dx.doi.org/10.3390/sym14061095]
[58]
H. Fei, Z. Fan, C. Wang, N. Zhang, T. Wang, R. Chen, and T. Bai, "Cotton classification method at the county scale based on multi-features and random forest feature selection algorithm and classifier", Remote Sens., vol. 14, no. 4, p. 829, 2022.
[http://dx.doi.org/10.3390/rs14040829]
[59]
H. Wang, and J. Wang, "Short-term wind speed prediction based on feature extraction with Multi-task Lasso and Multilayer Perceptron", Energy Rep., vol. 8, pp. 191-199, 2022.
[http://dx.doi.org/10.1016/j.egyr.2022.03.092]
[60]
B. Seijo-Pardo, V. Bolón-Canedo, and A. Alonso-Betanzos, "Testing different ensemble configurations for feature selection", Neural Process. Lett., vol. 46, no. 3, pp. 857-880, 2017.
[http://dx.doi.org/10.1007/s11063-017-9619-1]
[61]
A. Hashemi, M.B. Dowlatshahi, and H. Nezamabadi-pour, "Minimum redundancy maximum relevance ensemble feature selection: A bi-objective Pareto-based approach", J. Soft Comput. Inf. Technol., vol. 12, pp. 20-28, 2021.
[62]
R. Siva Subramanian, and D. Prabha, "Ensemble variable selection for Naive Bayes to improve customer behaviour analysis", Comput. Syst. Sci. Eng., vol. 41, no. 1, pp. 339-355, 2022.
[http://dx.doi.org/10.32604/csse.2022.020043]
[63]
J. Friedman, T. Hastie, and R. Tibshirani, "Regularization paths for generalized linear models via coordinate descent", J. Stat. Softw., vol. 33, no. 1, pp. 1-22, 2010.
[http://dx.doi.org/10.18637/jss.v033.i01] [PMID: 20808728]
[64]
A. Liaw, and M. Wiener, "Classification and regression by randomforest", R News, vol. 2, pp. 18-22, 2002.
[65]
"Centers for Disease Control and Prevention. Community Health Status Indicators (CHSI) to Combat Obesity, Heart Disease and Cancer", Available from: https://healthdata.gov/dataset/community-health-status-indicators-chsi-combat-obesity-heart-disease-and-cancer (Accessed on: August 5, 2020).
[66]
L.J. Waite, E.O. Laumann, and W.S. Levinson, ", S.T. Lindau, C.A. O’Muircheartaigh, LJ Waite, EO Laumann, WS Levinson, ST Lindau, and CA O’Muircheartaigh,"National social life, health, and aging project (NSHAP): Wave 1",", Inter-University Consortium for Political and Social Research, p. 20541, 2019.ICPSR, p. 20541,, 2019.
[http://dx.doi.org/10.3886/ICPSR20541.v9]
[67]
S. Numata, T. Ye, T.M. Hyde, X. Guitart-Navarro, R. Tao, M. Wininger, C. Colantuoni, D.R. Weinberger, J.E. Kleinman, and B.K. Lipska, "DNA methylation signatures in development and aging of the human prefrontal cortex", Am. J. Hum. Genet., vol. 90, no. 2, pp. 260-272, 2012.
[http://dx.doi.org/10.1016/j.ajhg.2011.12.020] [PMID: 22305529]
[68]
"Akalin", Available from: https://github.com/compgenomr/compGenomRData/blob/master/inst/extdata/CpGmeth2Age.rds
[69]
A. Onan, and S. Korukoğlu, "A feature selection model based on genetic rank aggregation for text sentiment classification", J. Inf. Sci., vol. 43, no. 1, pp. 25-38, 2017.
[http://dx.doi.org/10.1177/0165551515613226]
[70]
R. Wald, T.M. Khoshgoftaar, D. Dittman, W. Awada, and A. Napolitano, "An extensive comparison of feature ranking aggregation techniques in bioinformatics", C Zhang, J Joshi, E Bertino, and B Thuraisingham, Eds, In Proceedings of the 2012 IEEE 13th International Conference on Information Reuse and Integration, IRI 2012, 2012, p. 377-84
[http://dx.doi.org/10.1109/IRI.2012.6303034]
[71]
R. Salman, A. Alzaatreh, and H. Sulieman, "The stability of different aggregation techniques in ensemble feature selection", J. Big Data, vol. 9, no. 1, p. 51, 2022.
[http://dx.doi.org/10.1186/s40537-022-00607-1]
[72]
D. Pelegí-Sisó, P. de Prado, J. Ronkainen, M. Bustamante, and J.R. González, "methylclock: A bioconductor package to estimate DNA methylation age", Bioinformatics, vol. 37, no. 12, pp. 1759-1760, 2021.
[http://dx.doi.org/10.1093/bioinformatics/btaa825] [PMID: 32960939]
[73]
S. Horvath, "Erratum to: DNA methylation age of human tissues and cell types", Genome Biol., vol. 16, no. 1, p. 96, 2015.
[http://dx.doi.org/10.1186/s13059-015-0649-6] [PMID: 25968125]
[74]
Y. Wang, J.M. Franks, M.L. Whitfield, and C. Cheng, "BioMethyl: An R package for biological interpretation of DNA methylation data", Bioinformatics, vol. 35, no. 19, pp. 3635-3641, 2019.
[http://dx.doi.org/10.1093/bioinformatics/btz137] [PMID: 30799505]
[75]
R. Tacutu, D. Thornton, E. Johnson, A. Budovsky, D. Barardo, T. Craig, E. Diana, G. Lehmann, D. Toren, J. Wang, V.E. Fraifeld, and J.P. de Magalhães, "Human ageing genomic resources: New and updated databases", Nucleic Acids Res., vol. 46, no. D1, pp. D1083-D1090, 2018.
[http://dx.doi.org/10.1093/nar/gkx1042] [PMID: 29121237]

Rights & Permissions Print Cite
© 2024 Bentham Science Publishers | Privacy Policy