- To date, for the evaluation of DE methods, most attention has been paid on validity. - Yet another important aspect of DE methods, stability, is overlooked and has not been studied to the best of our knowledge.. - Results: In this study, we empirically show the need of assessing stability of DE methods and propose a stability metric, called Area Under the Correlation curve (AUCOR), that generates the perturbed datasets by a mixture distribution and combines the information of similarities between sets of selected features from these perturbed datasets and the original dataset.. - Conclusion: Empirical results support that AUCOR can effectively rank the DE methods in terms of stability for given RNA-seq datasets. - In addition, we explore how biological or technical factors from experiments and data analysis affect the stability of DE methods. - In the past few years, dozens of DE analysis meth- ods have been proposed in three mainstream strategies:. - While DE methods have been applied to identify features whose expression levels change between condi- tions and there have been many efforts to systematically compare these methods [10–12], an important question that has not been fully addressed is: how reliable is the selected set of features? Two aspects that are important and of interest to researchers about the reliability of the selected set of features are stability and validity:. - Validity measures the similarity between the sets of selected features by DE methods and the true collection of differentially expressed features. - In simulation studies, one can see a more complete picture of the validity of DE methods by several standard statistical metrics, such as precision, sensitivity, power and receiver operating characteristic (ROC) curves.. - The idealized result of DE methods is both high validity and high stability, i.e. - sets of selected features are consis- tent and close to the true set of DE features. - Currently, most evaluations of the reliability of DE methods in RNA- seq datasets are focusing on validity . - These evaluation procedures ignore the stability of results and may choose DE methods that are highly inconsistent when datasets have small perturbations, i.e. - sets of selected fea- tures are quite different from each other, but close to the true set of DE features in general.. - 1, DE methods may suffer a lack of sta- bility, i.e. - Figure 2 depicts a generic workflow for stability assessment of DE methods in microarray datasets that contains three steps: (1) Given a dataset Y, M perturbed samples are generated by either bootstrap or subsampling;. - (2) A DE method is applied to each perturbed sample and selects a set of DE features with some given threshold for adjusted p-values. - 1 Selection frequency of the Bottomly dataset [23] by edgeR-robust. - b Scatterplot of biological coefficient of variation (BCV) against average of log 2 of counts per million (CPM) of the first randomly selected sub-dataset. - Finally, the stability measure is computed by taking the average of similarities of all pairwise sets of DE features. - of DE features. - Second, by simply averaging the sim- ilarities of pairwise sets of DE features in step (3), the estimates of stability levels may heavily depend on the choice of the size of subsampled samples.. - The authors adopted the ratio of the area under the curve to the max- imal possible value K 2 / 2 as a measure of concordance.. - Yet, these two metrics can not be used to assess the similarity of two sets of DE features with different sizes.. - However, if a DE method tends to select large sets of DE features, the size of common features would be large.. - First, we propose a stability metric to quantify the stability of DE methods based on parametric data perturbations. - We demonstrate that the pro- posed metric could well rank the DE methods. - Second, we investigate which and how factors of RNA-seq data or DE analysis procedures influence the stability of DE methods in various simulation settings.. - where μ gi and σ gi 2 are the mean and variance of the Nega- tive Binomial (NB) distribution respectively. - The underlying idea of estimating the stability of DE methods for a specific dataset is simple: If the DE method is stable, then a minor perturbation of the data should not change the set of selected features drastically. - If the estimated mean and variance from the original dataset are close to the true mean and variance of the NB distribution, the mixture distribution f gi ( y ) is close to f 0 gi ( y ) no mat- ter how we choose α 1 . - The stability metric of DE methods. - At each perturbation size α 1 , compute the average sim- ilarities of the new set of selected DE features s α m 1 , m = 1. - However, the absence of the truth and limited flexibil- ity make the real datasets not suitable to assess the factors that may affect the stability of results of DE analysis. - fold changes of DE features are generated from the normal distribution N ( 3, 0.5 2. - DE methods. - We use a common threshold to call a set of DE features. - Yet, we can find a proxy of the true stability level by computing the average of Pearson’s correlation of DE results for indepen- dent samples. - Then, we apply DE methods to each random sample and compute the Pearson’s correlation coefficients for each pair of random samples. - The ranking of DE methods for both AUCOR and aver- age of correlation is generally consistent on both real RNA-seq and simulated datasets (Fig. - It is noted that the ranks of DE methods for the Cheung dataset and the simulated dataset are quite different. - To further show that the AUCOR values can rank the DE methods according to the stability, Fig. - Intuitively, stable DE methods select similar sets of features for different datasets. - large for stable DE methods, and small for unstable DE methods. - To understand how the methods perform in the sense of stability with different read count levels, the stability of DE methods is further analyzed. - In the absence of outliers, robust versions of DE methods, such as edgeR_robust and DESeq2, are more stable than other methods, except for the low expressed category. - A more comprehensive picture of the performance of different DE methods for the datasets with or without outliers under the basic simulation setting is presented in Fig. - Factors that affect stability of DE results. - chosen, it is also of interest to investigate which and how underlying factors affect the stability of DE analy- sis results. - 4 mFoldChange: mean of fold change of DE features, varies from 3 to 6, the default is 3.. - 5 rDisp: ratio that is multiplied to the estimated dispersion of the original dataset, varies from 0.6 to 2, the default is 1.. - 6 pUp: proportion of DE features that are up-regulated, varies from 0.1 to 0.7, the default is 0.5.. - We take the sets of DE genes for the verification dataset by all considered DE methods as truth (we exclude SAMseq, since it could hardly produce adjusted p-values less than 0.05). - Finally, we have 6 true sets from different DE methods. - a True set of DE genes from edgeR. - b True set of DE genes from edgeR_robust. - c True set of DE genes from DESeq_glm. - d True set of DE genes from DESeq2. - e True set of DE genes from EBSeq. - f True set of DE genes from Voom.. - So, we are particularly interested in the perfor- mance of DE methods on the RNA-seq dataset as the increasing of number of replicates. - We also observe that the precisions of these DE methods have sim- ilar patterns as AUCOR and ranks of methods according. - Fold changes of DE features are generated from the normal distribution with standard error 0.5 and the means of fold changes are set as 2, 2.5, 3. - d AUCOR against proportion of DE features that are up-regulated. - g AUCOR against proportion of DE features that is spread from 10% to 70%. - Different DE methods are represented by different symbols and colors. - Fold change and dispersion are two important factors that may affect the stability of DE methods, since these two fac- tors are the main parameters that all DE methods directly or indirectly want to estimate, and results of DE methods are largely determined by the qualities of the estimates of fold change and dispersion. - Different DE methods are represented by different colors. - of fold change, the difference between DE features and non-DE features are larger, and as a result, it is easier for DE methods to identify DE features. - By contrast, as the increasing of dispersion, the difference between DE features and non-DE features becomes vaguer, and it is more difficult to find DE features for DE methods. - In general, as the increasing of fold change or decreasing of dispersion, all DE methods exhibit higher stability (Fig. - When the number of replicates is 10 for each condition, DESeq and Voom show high stability and the stability trends of these two methods are similar to that of other DE methods (Additional file 1: Figures S11(a) and S11(b)).. - And the presence of out- liers may influence the estimates of parameters of DE methods and consequently the finally calling of DE genes.. - Figure 9f shows that the stability of DE methods may. - The proportion of DE features influences the stability slightly. - DE meth- ods seem less stable when the proportion of DE features is small (Fig. - The main goal of DE analysis is to find a set of features toward a task such as classification or identification of the top relevant features corresponding to a biological phenomenon of interest. - Regarding to the reliability of DE methods, there are two essential aspects: stability and validity. - To date, most attention has been paid on validity, while stability is overlooked during the evaluation of DE methods. - Thus, the current evaluation system for DE methods may prefer methods with low reproducibility.. - We have used three different datasets with large number of replicates, Bottomly, Cheung and PickMont datasets, to illustrate the stability of the DE methods. - of DE methods according to the AUCOR and averages of correlations from subsampling (Fig. - An advantage of the proposed stability metric is the suit- ability to RNA-seq datasets with small number of repli- cates under both conditions. - The overall trends of mean and dispersion for the perturbed datasets are very close to those of the original datasets (Additional file 1: Figure S2).. - In this study, we further employed simulations to explore which and how underlying factors affect the sta- bility of DE analyses via a broad range of possible settings.. - First, lev- els of fold change of truly differentially expressed features and dispersions of the dataset substantially affect the sta- bility of DE methods. - Specifically, as the decreasing of fold change or increasing of dispersion, DE methods tend to be less stable. - could make the results of DE methods more stable. - Further, it is worth mentioning that the perturbation of dataset is based on the assumption of the NB distribution.. - In conclusion, we developed a metric to measure the stability of DE methods for differential expression analy- ses of RNA-seq data. - Overall, the metric could rank DE methods according to the stability levels. - On one hand, we summarize stability performance of 6 popular DE methods based on our study (Table 1). - The practitioners can choose a method according to the table based on the information of the given RNA-seq dataset.. - backing in the literature, then estimate the stability levels of these candidate DE methods by AUCOR and select a DE method according to AUCOR values.. - In this paper, we focus on assessing the stability of selected sets of DE features based on a pre-set threshold for the ranking of features from DE methods. - Thus, this stability metric depends on the choice of the threshold and may have some potential drawbacks. - This may poten- tially affect the stability level of DE methods, although in general this is not a big issue. - Second, the proposed approach measures the stability of selected subsets of features, but not the ranking of features by DE methods. - Besides, there are other similarity measures for the results of DE methods other than Pearson’s correlation coefficient. - BQL and ZP contributed to the design of the study.
Xem thử không khả dụng, vui lòng xem tại trang nguồn hoặc xem
Tóm tắt