Novel evaluation approach for molecular signature-based deconvolution methods (2023)

Since the therapeutic immuno-oncological approaches for treating tumors emerged as a solid way to eliminate cancer cells, there are concerns about why there is a considerable number of patients that do not benefit from such treatments. The interplay between tumor cells and the surrounding immune cells (i.e the Tumor Immune Microenvironment or TIME) has been largely acknowledged as a main determinant of tumor fate during treatments such as checkpoint blockade therapy and other therapeutic approaches that lead to immune activation. [1], [2], [3]. Despite flow cytometry [4] or single cell sequencing approaches being accepted as the preferred methods to interrogate the TIME, both are usually impractical to be massively applied on large cohorts. However, current availability of gene expression profiles from bulk tumor tissues on large data sets, allow catching up on the content and characteristics of the TIME by means of computational tumor immune cell deconvolution methods (DM) that, by predicting cell type content and proportions, will allow finding TIME association with prognosis and elucidating the intrinsic and acquired mechanism of resistance to immunotherapy [5].

Several computational approaches had been proposed to evaluate the TIME, each one with its own strengths and weaknesses. They can be divided into two groups, reference-free and reference-based models [6]. Reference-free models can be useful in situations when the cell types present in the samples to be quantified are unknown to the user, or when the cell types present in the samples have not been previously characterized [6]. On the other hand, reference-based models interrogate the presence/absence of specific cell types on well-studied tissues or tumors to further evaluate their cell-type diversity and/or proportions association with prognosis, or other important clinical factors. In this work, the focus is set on the latter ones, where for certain deconvolution algorithms it is assumed that the cell-type composition of a bulk sample can be inferred as a linear combination of the gene expression profiles (as suggested by Zhong et al.[7]). Such profiles are stored in a molecular signature (MS) matrix as specific cell-type transcriptomic templates, taken as a reference to identify each cell-type content on bulk samples by DMs.

The linear deconvolution of the cell types present in an MS, holding N genes for k cell types (${\mathit{MS}}^{\mathit{Nxk}}$), suspected to be the components of a mixture of cell types present in a tissue gene expression profile (Y), involves solving the following regression model equation, $Y=MS\bullet B$, where the proportions for all cell types in the admixture sample is represented by the column vector $B$ (the vector of “k” regression coefficients) which should satisfy both the non-negativity $\left(B=\left\{{\beta }_1,\cdots ,{\beta }_k\right\},{\beta }_j\ge 0\forall j=1...k\right)$ and sum-to-one constraints $\left({\sum }_{i=1}^k{\beta }_i=1\right)$. In order to preserve the linear relationship of the observed gene expression profile and the gene signature matrix, this equation should be addressed with non-log RNA-Seq data or one-color microarray data [7]. As a result, the DM provides an estimate of the cell type proportions as a $\widehat{B}$ vector (where “^” means estimated). In this sense, every DM output is composed by two terms: i) the cell type identification (i.e. those cell types for which ${\hat{\beta }}_j>0$ represents the detected cell types in a sample) and ii) their corresponding proportion values (i.e. ${\widehat{\beta }}_j$).

When analyzing cell type proportion prediction accuracy, current evaluations and comparisons were made using metrics like i) the correlation coefficient (ρ), ii) the goodness of fit or coefficient of determination from a regression technique (R²) and/or iii) the root mean squared error (RMSE) or mean absolute deviation (mAD), involving their predictions and the ground truth or, alternatively, the predicted proportions estimated by a competing method. However, such evaluation values may provide wrong clues about proportion prediction accuracy and bias since they only evaluate the linear association of the estimated proportion related to the expected one (i.e. the overall error), thus not giving any insight about how the error varies across different cell proportion scenarios (prediction-dependent bias trend).

In particular, correlation refers to the presence of a relationship between two different variables (measures the strength of the relation), whereas agreement is defined as the concordance between two measurements of one variable [8], [9], [10]). Highly correlated variables may have a low level of agreement and show significant systematic or prediction-dependent error bias with an ascending trend (see [9], [10]). Correlation analysis may result appropriate for comparing score-based methods like xCell [11] or imsig [12] where proportion estimates errors can not be evaluated, thus being advised for cell-type content analysis of score-based TIME predictors. On the other hand, the coefficient of determination, R², only gives information about the proportion of variance in common with the ground truth or, alternatively, a gold standard method [13]. RMSE and mAD are averaged measures of error’s standard deviation or the difference between one DM’s predicted proportions against the ground-truth-proportions. The RMSE may be useful when the differences between the ground truth and the predicted value are normally distributed, which may not always be the case, and two methods may have similar RMSE values but still have significant differences in the distribution of their error measurements (for instance, heteroscedasticity or prediction-dependent over/underestimation trends). Despite metrics like mAD can be used when the differences are not normally distributed, they do not provide, as the RMSE, information about the direction (over/underestimation) of the errors, thus not being fully suitable for evaluating the agreement between predicted and true proportions/cell-type coefficients. In any case, correlation, RMSE and mAD are just single values which does not allow the differentiation between systematic and prediction-dependent biased results (i.e. the first means a constant error across prediction values, and the second refers to predictions where the error or its variance augments or decreases depending on the predicted value). In order to qualitatively evaluate prediction-dependent patterns, graphical methods are crucial to provide information about the magnitude and direction of the differences between the prediction and the ground truth, helping to identify any kind of bias or variance variations.

Even though these metrics may still be used to evaluate MS-DM pairs, they are not sufficient to completely assess their performance nor to compare two or more of them. Any benchmarking protocol should necessarily include metrics and methods to explore the accuracy of a MS-DM pair estimations, not only comparing the averaged error between them or how their results correlate with the expected values, but measuring and displaying how the error evolves across different scenarios, if it has a homogeneous distribution, and their efficiency in detecting the presence of the questioned cell-types in the tumor sample.

In this sense, current proposed benchmark studies do not evaluate the cell-type identification abilities of each tested DM. This capability must be assessed by comparing the number of positively identified cell types with the true cell types present in the sample. Regarding this, measuring how accurately a DM detects which cell types are present in a sample is crucial since i) the $\widehat{\beta }$ coefficients that any DM estimates are constrained by the number of cell types identified at each sample and ii) they may provide cell types which are not truly present in the TIME. Consequently, over or under cell-type detection will lead to a biased estimation of $\widehat{\beta }$ coefficients (i.e predicted proportions).

In essence, the objective of any MS-DM pair is twofold: it should provide accurate cell type identification (low false discovery rates) present in the admixture sample and consequently provide unbiased $\widehat{\beta }$ coefficients. So, any benchmark protocol should appropriately evaluate these two MS-DM pair objectives defining appropriate evaluation scenarios, metrics and diagnostic plots as proposed in this work.

Independent of the evaluation scenario and/or evaluated feature (i.e. data scaling/normalization, distribution of generated noise, background prediction, minimal fraction detection, etc. [14], [15]), unbiased and objective metrics and methods are required to fairly benchmark MS-DM pairs. In this work, we propose a protocol and appropriate metrics to evaluate the proportion estimation accuracy and the cell-type detection capabilities of each MS-DM pair by defining true and false positive/negative predictions.

In order to evaluate overall estimation accuracy, an enhanced Bland-Altman method [13], [16] is proposed, providing a graphical support (diagnostic plots) and metrics for agreement evaluation. To perform a case-specific analysis, by defining which $\widehat{\beta }$ coefficients corresponded to cell types that were or were not present in a sample, the range of falsely detected $\widehat{\beta }$ coefficients can be evaluated and compared between DMs.

Cell-type detection performance can be evaluated after analyzing every tested DM’s output as a “binary” classification result, and then calculating classification accuracy metrics such as sensitivity (Se), specificity (Sp), positive/negative predictive values (PPV, NPV), distance to the optimal ROC point (DOP, evaluates detection capabilities and certainty) [17], F1-score (evaluates the amount and confidence of positive detections) [18], and the error rate (ER). The number of predicted cell types (NCT) detected in each sample across different scenarios must be evaluated too.

By means of these proposed methods and metrics, the accuracy of the predicted proportions, which is affected by false positive/negative cell type detections (using true negative and positive scenarios respectively) can be evaluated.

Discussion

Due to the massive availability of tumor gene expression data from both public repositories and from new clinical trials evaluating immunotherapies, the development of new techniques allowing the exploration of the complex interplay between the tumor cells and its immune microenvironment makes it crucial to decipher the mechanism driving therapy success, resistance and prediction response [31]. The deconvolution methods and the molecular signatures for tumoral-sample deconvolution are

Conclusion

Here it is demonstrated that by means of metrics not previously used in the field it is possible to assess the cell-type detection capabilities of each MS-DM pair, and how they play a key role in their proportion estimation accuracy. As a conclusion, we suggest the inclusion of this protocol’s metrics and methods when evaluating/developing MS-DM pairs for a fair and comprehensive comparison.

Statement of significance

• •

  Problem or issue: Current benchmark protocols for reference-based deconvolution methods do not adequately assess the cell-type detection capabilities, nor do they evaluate the accuracy of predicted proportions compared to the ground truth. Additionally, there is a lack of assessment for detecting patterns in the prediction error.

• •

  What is already known: Current literature estimates the proportion prediction accuracy of every method based on metrics (ρ/R²/RMSE/mAD) that only evaluate the linear

  See Also

  Design of multifunctional fluorescent self-assembly system for sensing Hg2+, volatile acids and being applied as anti-counterfeit materials through the transformation of ACQ-to-AIEDesign of multifunctional fluorescent self-assembly system for sensing Hg2+, volatile acids and being applied as anti-counterfeit materials through the transformation of ACQ-to-AIE(PDF) References - Springer978-1-349-18415-6/1.pdf · References 1 The Political ... ment, Africa versus Latin America, United Nations ... 54. 'Funds for Public Health Inadequate', Beijing - DOKUMEN.TIPSInferring the antigenic epitopes for highly pathogenic avian influenza H5N1 viruses

Funding

This work was supported by the Universidad Católica de Córdoba (80020180100029CC) to E.A.F., Universidad Nacional de Córdoba (33620180100993CB) to E.A.F and the Argentinean Agency for Promotion of Science and Technology (PCE-GLAXO-2020–00004) to G. M.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.