Abstract

The newfound rise of protein language models (PLMs) that leverage data and compute has introduced an interesting conflict: a trade-off between the high predictive performance of non-interpretable features and the scientific insight that can be gained from interpretable, hand-crafted ones. In this work, we highlight and study this conflict via the task of classifying protein domains into their CATH superfamilies. We train one-vs-all (OvA) linear SVM classifiers for 45 diverse CATH superfamilies, each characterised by significant class imbalance. Our analysis compares nine feature vector types, which are either non-interpretable embeddings from PLMs or interpretable hand-crafted features. We measure classification performance using the AM score, i.e., the arithmetic mean of sensitivity and specificity. Our results demonstrate that the PLM embedding-based feature ProtBERT-Emb achieves superior test AM scores of 90-99\% with low variability, outperforming hand-crafted features by 8-21\%. While PLM features yield high classification performance, their lack of interpretability obscures the underlying biological determinants. On the other hand, our novel structure-based Contact Separation Interval Composition (CSIC) feature strikes an optimal balance. CSIC achieves highly competitive performance ($\sim$88%) with low overfitting while providing rich structural information about contact sequence separation. Furthermore, we illustrate for two superfamilies, using the CSIC features and Marginal Contribution feature Importance (MCI) scores, that we can recover known structural characteristics of superfamilies such as characteristic long-range contacts and repeating amino acid motifs. This validates its utility for downstream applications, such as investigating protein-related diseases and guiding rational protein design.