In this guest editorial, we present our statistical perspective on how to interpret and report results that are not statistically significant. We believe that the best approach begins at the planning stages of a study. Good planning begins with carefully stated objectives and a plan for interpreting the possible outcomes of a study. An investigator can then estimate the number of subjects on the basis of statistical calculations, expert opinion, and research. This is the best opportunity to plan for the statistical analysis so that it addresses the research
questions.

We recognize that not all studies will have a comprehensive planning stage and that certain outcomes may not be fully anticipated. For this reason, we present some options for what to do in situations where results are not statistically significant and include a selection of user-friendly resources at the end of this editorial to provide additional background on all of the statistical concepts that form the basis of our perspective.

In part 2 of this editorial, which will appear in the July issue of JOSPT, we will provide additional comments on the topic of statistical power and the design of clinical studies. The motivation for this second part came from the questions that investigators have asked us on this topic in the course of our research collaborations. Part 2 will present our responses to these frequently asked questions.

J Orthop Sports Phys Ther. 2003; 33(6):303-306.

Key Words: nonsigificant results, statistics

In the first part of this guest editorial (JOSPT, June 2003 guest editorial), we presented our statistical perspective on how to interpret and report results that were not statistically significant. We emphasized that the best approach begins at the planning stages of a study. Carefully stated objectives, an appropriate study design, and statistical power calculations are all part of planning an effective research study. We also discussed how an analysis of post hoc statistical power can be used in the event that the results of a study are not statistically significant. A post hoc assessment of statistical power can identify whether or not the study was sensitive enough to detect an important clinical effect.

In this guest editorial, we provide additional comments on interpreting study results that are not statistically significant and the role that statistical power calculations have in supporting a specific interpretation. We present these comments in the format of responses to frequently asked questions that we have encountered in our collaborative work with investigators in a variety of disciplines within the life sciences.

J Orthop Sports Phys Ther. 2003; 33(7):367-368.

Key Words: nonsignificant results; statistics

‘‘Analysis of Paired Data in Physical Therapy Research: Time to Stop Double-Dipping?'' was a guest editorial that appeared in the August 2005 issue of the JOSPT. This editorial, written by Dr Hylton Menz, raised an important statistical issue about how to properly manage study designs that involve paired data. "Double-dipping'' refers to using an inappropriate statistical analysis for paired data. For example, if a study used 12 subjects and measured both feet of each subject, it would be inappropriate to calculate a standard error that is based on an n of 24 (2 × 12) independent feet. Double-dipping is a problem because the paired limbs within a subject are not likely to be statistically independent. This practice can result in a standard error that is smaller than it should be. Based on a spuriously small standard error and an inflated n, a statistical test may appear to be more ‘‘significant'' than it actually is. Dr Menz concluded his editorial by recommending several strategies to select only 1 of a paired limb per patient for measurement to maintain the statistical independence of the data.2 For example, we could randomly select and measure only 1 foot per subject, and then use an n of 12 (1 × 12) independent measurements in the statistical analysis. However, would we lose valuable information in doing so?

In his efforts to persuade readers to avoid the error of double-dipping, Dr Menz may have inadvertently discouraged the use of valid study designs and appropriate analysis methods for paired data. My purpose in writing this guest editorial is to discuss these designs and encourage investigators to consider all of the options available to them in planning their studies. This editorial discusses 3 situations where an appropriate design and analysis for paired data may be advantageous. I will use feet and ankles in my illustrations of paired data, but these comments also extend to other pairs of limbs and organs that may be the focus of research in physical therapy.

J Orthop Sports Phys Ther. 2006; 36(2):42-44. doi:10.2519/jospt.2006.0102 

Key Words: paired data, statistics