Two-Stage Testing in Microarray Analysis: What is Gained?
Allison, D.B., and Coffey, C.S. (2002). Journal of Gerontology: Biological Sciences (in press).
Microarray technology for gene expression studies offer powerful new technology for understanding changes in gene expression as a function of other observable or manipulable variables. However, microarrays also pose a number of new challenges. One of the most prominent of these is the difficulty in establishing a procedure for declaring whether a gene's expression level is associated with the independent variable that offers reasonable and specifiable false positive (type 1 error) and false negative (type 2 error) rates. Recently, Miller et al (2001) offered a two-stage design to address these goals. Miller et al's approach involves first testing for differences between two groups for all of k genes in one set of data (stage 1) using an alpha level greater than the alpha level that would be required by a Bonferroni correction. These k hypothesis tests will yield some number of genes, m (where m is <k), with significant effects at this first stage. Then, a second independent set of data is gathered at stage 2 and only those m genes found to be significant at stage 1 are tested at an alpha level of .05/m.
When introduced, it was stated, but not proven, that this method would alleviate the problem of false positives (type 1 errors) that would result from multiple significance testing without correction and false negatives (type 2 errors) that would result from the use of a Bonferroni correction. However, we (Allison & Coffey, in press) show that this two-stage design often fails to hold the overall experiment-wise type 1 error rate to some desired alpha level. In fact, many choices of the per-test alpha level at stage 1 will result in an experiment-wise type 1 error rate that is overly conservative. Moreover, this two-stage method can also exacerbate the false negative rate and hence decrease power relative to a single stage design. Thus, until and unless further refinements are made, we advocate that investigators not use the two-stage approach advocated by Miller et al.