Journal: JMIRx Med

Article Title: Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

doi: 10.2196/45973

Figure Lengend Snippet: Design overview of the study. Nontemporal performance and drift (temporal) analyses were performed. Drifts in discrimination, calibration, clinical utility, data set, and variable importance were assessed. Time point assessments were performed for the clinical effectiveness metric (CEM). Drifts in component metrics of CEM were evaluated. AUC: area under the curve; ECE: expected calibration error; EuroSCORE: European System for Cardiac Operative Risk Evaluation; F1: F 1 -score; neuronetwork: neural network; SVM: support vector machine; Xgboost: extreme gradient boosting.

Article Snippet: The Dunn test showed strong evidence of XGBoost having the best overall performance (Table S8 in ; P <.05), followed by RF, LR, and NN (CEM difference to XGBoost: −0.0032, −0.0055, and −0.0108, respectively; P <.05).

Techniques: Plasmid Preparation

Journal: JMIRx Med

Article Title: Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

doi: 10.2196/45973

Figure Lengend Snippet: Geometric mean of individual metrics for each model in the holdout set. In all, 1000 bootstrap samples were used to derive the geometric mean of each metric. Adjusted ECE ^a and Brier score values are shown. Net benefit is the average absolute overall benefit across all thresholds.

Article Snippet: The Dunn test showed strong evidence of XGBoost having the best overall performance (Table S8 in ; P <.05), followed by RF, LR, and NN (CEM difference to XGBoost: −0.0032, −0.0055, and −0.0108, respectively; P <.05).

Techniques:

Journal: JMIRx Med

Article Title: Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

doi: 10.2196/45973

Figure Lengend Snippet: The Dunnett test with XGBoost ^a as a control and the rest of the models as comparisons.

Article Snippet: The Dunn test showed strong evidence of XGBoost having the best overall performance (Table S8 in ; P <.05), followed by RF, LR, and NN (CEM difference to XGBoost: −0.0032, −0.0055, and −0.0108, respectively; P <.05).

Techniques: Control

Journal: JMIRx Med

Article Title: Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

doi: 10.2196/45973

Figure Lengend Snippet: (A) Plot of CEM values by model and time. Geometric mean (95% CI) of 1000 bootstraps at each time point is shown. The horizontal line represents the CEM geometric mean of all models. (B) Box plot of difference in models’ CEM values across the first 3 months of 2017 and 2019. Kruskal-Wallis results for CEM across the time points are shown. (C) Paired-samples Wilcoxon test (Wilcoxon signed rank test) for the first 3 months of 2019 bootstrap CEM values. P values are adjusted using the Bonferroni method. **** P <.0001. CEM: clinical effectiveness metric; EuroSCORE: European System for Cardiac Operative Risk Evaluation; ns: not significant; neuronetwork: neural network; SVM: support vector machine; Xgboost: extreme gradient boosting.

Article Snippet: The Dunn test showed strong evidence of XGBoost having the best overall performance (Table S8 in ; P <.05), followed by RF, LR, and NN (CEM difference to XGBoost: −0.0032, −0.0055, and −0.0108, respectively; P <.05).

Techniques: Plasmid Preparation

Journal: JMIRx Med

Article Title: Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

doi: 10.2196/45973

Figure Lengend Snippet: Plots of CEM values by model and time: (A) XGBoost, (B) random forest, (C) logistic regression, and (D) EuroSCORE II. The geometric mean of 1000 bootstraps at each time point is shown. The red dotted line shows linear regression, and the blue line shows generalized additive model fit. Parameters and P values for the linear regressions are shown. (E) Discrimination (AUC) performance drift by time. Linear regression lines are plotted for each model, with slope, intercept, and P values displayed in the legend. (F) Calibration (adjusted ECE) performance drift by time. Linear regression lines are plotted for each model, with slope, intercept and P values displayed in the legend. SVM and EuroSCORE II are removed to enable a clearer separation of models with similar performance. AUC: area under the curve; CEM: clinical effectiveness metric; ECE: expected calibration error; EuroSCORE: European System for Cardiac Operative Risk Evaluation; neuronetwork: neural network; SVM: support vector machine; Xgboost: extreme gradient boosting.

Article Snippet: The Dunn test showed strong evidence of XGBoost having the best overall performance (Table S8 in ; P <.05), followed by RF, LR, and NN (CEM difference to XGBoost: −0.0032, −0.0055, and −0.0108, respectively; P <.05).

Techniques: Plasmid Preparation

Journal: JMIRx Med

Article Title: Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

doi: 10.2196/45973

Figure Lengend Snippet: (A) Clinical effectiveness (net benefit) performance drift by time. Linear regression lines are plotted for each model, with slope, intercept, and P values displayed in the legend. SVM and EuroSCORE II are removed to enable a clearer separation of models with similar performance. (B) SHAP variable importance drift for the holdout set over 27 months (EuroSCORE II and XGBoost). Solid dots show geometric mean values of 5-fold cross-validation. Smoothed locally estimated scatterplot lines are plotted, with green bands showing 95% CIs. (C) SHAP variable importance drift for the holdout set over 27 months for the top 6 most important variables (EuroSCORE II and XGBoost). The trends are unsmoothed. (D) Operative urgency data set drift across time for the holdout set. The percentages of each category are shown for each time point. CCS: Canadian Cardiovascular Society; CPS: critical preoperative state; EuroSCORE: European System for Cardiac Operative Risk Evaluation; ES: EuroSCORE; LV: left ventricle; MI: myocardial infarction; neuronetwork: neural network; NYHA: New York Heart Association; PA: pulmonary artery; PVD: peripheral vascular disease; SHAP: Shapley additive explanations; SVM: support vector machine; Xgboost: extreme gradient boosting.

Article Snippet: The Dunn test showed strong evidence of XGBoost having the best overall performance (Table S8 in ; P <.05), followed by RF, LR, and NN (CEM difference to XGBoost: −0.0032, −0.0055, and −0.0108, respectively; P <.05).

Techniques: Biomarker Discovery, Plasmid Preparation

Journal: JMIRx Med

Article Title: Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

doi: 10.2196/45973

Figure Lengend Snippet: The actual and projected net benefit drift for the NN and Xgboost models over time. NN: neural network; XGBoost: extreme gradient boosting.

Article Snippet: The Dunn test showed strong evidence of XGBoost having the best overall performance (Table S8 in ; P <.05), followed by RF, LR, and NN (CEM difference to XGBoost: −0.0032, −0.0055, and −0.0108, respectively; P <.05).

Techniques:

Journal: Sensors (Basel, Switzerland)

Article Title: A Cascade Ensemble Learning Model for Human Activity Recognition with Smartphones

doi: 10.3390/s19102307

Figure Lengend Snippet: Overview of HAR system. Handcrafted feature extraction based HAR contains data collection, signal processing, feature extraction and CELearning model. Automatic feature extraction based HAR contains data collection, FFT and CELearning model.

Article Snippet: Meanwhile, CELearning (XGBoost + Softmax Regression) and CELearning (XGBoost + Softmax Regression + Random Forest) achieved, respectively, the best performances with two and three classifiers when using automatic feature extraction based HAR.

Techniques: Extraction

Journal: Sensors (Basel, Switzerland)

Article Title: A Cascade Ensemble Learning Model for Human Activity Recognition with Smartphones

doi: 10.3390/s19102307

Figure Lengend Snippet: CELearning model. Each layer is composed of four basic classifiers which generate the probability vectors as augmented features for next layer’s learning.

Article Snippet: Meanwhile, CELearning (XGBoost + Softmax Regression) and CELearning (XGBoost + Softmax Regression + Random Forest) achieved, respectively, the best performances with two and three classifiers when using automatic feature extraction based HAR.

Techniques:

Journal: Sensors (Basel, Switzerland)

Article Title: A Cascade Ensemble Learning Model for Human Activity Recognition with Smartphones

doi: 10.3390/s19102307

Figure Lengend Snippet: Comparison of different methods based on handcrafted feature extraction.

Article Snippet: Meanwhile, CELearning (XGBoost + Softmax Regression) and CELearning (XGBoost + Softmax Regression + Random Forest) achieved, respectively, the best performances with two and three classifiers when using automatic feature extraction based HAR.

Techniques: Comparison, Extraction

Journal: Sensors (Basel, Switzerland)

Article Title: A Cascade Ensemble Learning Model for Human Activity Recognition with Smartphones

doi: 10.3390/s19102307

Figure Lengend Snippet: Comparison of different methods based on automatic feature extraction.

Article Snippet: Meanwhile, CELearning (XGBoost + Softmax Regression) and CELearning (XGBoost + Softmax Regression + Random Forest) achieved, respectively, the best performances with two and three classifiers when using automatic feature extraction based HAR.

Techniques: Comparison, Extraction

Journal: Sensors (Basel, Switzerland)

Article Title: A Cascade Ensemble Learning Model for Human Activity Recognition with Smartphones

doi: 10.3390/s19102307

Figure Lengend Snippet: Comparison of different combinations of four classifiers based on handcrafted feature extraction.

Article Snippet: Meanwhile, CELearning (XGBoost + Softmax Regression) and CELearning (XGBoost + Softmax Regression + Random Forest) achieved, respectively, the best performances with two and three classifiers when using automatic feature extraction based HAR.

Techniques: Comparison, Extraction, Standard Deviation

Journal: Sensors (Basel, Switzerland)

Article Title: A Cascade Ensemble Learning Model for Human Activity Recognition with Smartphones

doi: 10.3390/s19102307

Figure Lengend Snippet: Comparison of different combinations of four classifiers based on automatic feature extraction.

Article Snippet: Meanwhile, CELearning (XGBoost + Softmax Regression) and CELearning (XGBoost + Softmax Regression + Random Forest) achieved, respectively, the best performances with two and three classifiers when using automatic feature extraction based HAR.

Techniques: Comparison, Extraction, Standard Deviation

Journal: Sensors (Basel, Switzerland)

Article Title: A Cascade Ensemble Learning Model for Human Activity Recognition with Smartphones

doi: 10.3390/s19102307

Figure Lengend Snippet: Comparison of different methods for 12 categories of HAR.

Article Snippet: Meanwhile, CELearning (XGBoost + Softmax Regression) and CELearning (XGBoost + Softmax Regression + Random Forest) achieved, respectively, the best performances with two and three classifiers when using automatic feature extraction based HAR.

Techniques: Comparison