Reassessing CVSS Severity Classification with LightGBM and Operating System Metadata

摘要

CVSS severity and observed exploitation are different prediction targets. This study reassesses a LightGBM pipeline using an available 1,314-record vulnerability dataset and explicitly defines its label as a CVSS score greater than 9. The original 98.48% accuracy is reproducible, but the pipeline encodes complete score-defining vectors and fits text features before splitting. We therefore remove identifiers and complete vectors, fit preprocessing only on training records, and compare five classifiers using parsed attack vector, TF-IDF text and two operating-system metadata features. After description deduplication, 882 records with CVE identifier years up to 2023 are used for training and 390 records from identifier year 2024 for testing. This is an identifier-year robustness check, not a disclosure-time evaluation. LightGBM achieves 95.64% accuracy, 41.67% recall, 0.8412 ROC-AUC and 0.4821 average precision; the majority baseline already achieves 93.85% accuracy. Adding operating-system metadata changes average precision by −0.0303 (paired bootstrap 95% interval −0.1077 to 0.0409), providing no consistent improvement. TreeSHAP dependence plots and error cases reveal strong reliance on attack vector and lexical associations. The results support a limited severity-classification case study and demonstrate why these data cannot establish real-world exploitability prediction.