A Machine Learning Pipeline with Switching Algorithms to Predict Lung Cancer and Identify Top Features

Abstract

Lung cancer is the leading cause of cancer-related death around the world. Early detection is a critical factor for its effective treatment. To facilitate early-stage prediction, a Machine Learning (ML) pipeline has been built that uses inpatient admission data to train four ML models. The data is dynamically loaded into a database, cleaned, and passed through the SelectKBest selector to identify the top features influencing the prognosis, which are then fed into the pipeline and fitted to the ML models to make the forecast. Among the models used, Decision Tree provides the highest accuracy (97.09%), followed by Random Forest (94.07%). MLP and Logistic Regression reach an accuracy of 84.58% and 77.65% respectively. Some of the top 50 features include chronic obstructive pulmonary disease, pleural effusion, secondary and unspecified malignant neoplasm of intrathoracic lymph nodes, syndrome of inappropriate secretion of antidiuretic hormone, and neoplasm-related acute, chronic pain.