How to Report Nonsignificant Results: Planning to Make the Best Use of Statistical Power Calculations

Janice Derr

Laura Sears is a busy clinical investigator [this is a hypothetical study] involved in several research studies. In one of these studies, a new material for orthotic shoe inserts was compared to a standard material. Patients with foot pain were randomly assigned to 1 of the 2 types of orthotic insert. At the end of 6 weeks, the patients reported their level of foot pain on a Visual Analog Scale (VAS), a scale that ranges from 0 (no pain) to 100 (worst pain). The comparison between these 2 groups resulted in a difference that was not statistically significant. What can Dr. Sears conclude from this finding? How should she report these results?

In this guest editorial, we present our statistical perspective on how to interpret and report results that are not statistically significant. We believe that the best approach begins at the planning stages of a study. Good planning begins with carefully stated objectives and a plan for interpreting the possible outcomes of a study. An investigator can then estimate the number of subjects on the basis of statistical calculations, expert opinion, and research. This is the best opportunity to plan for the statistical analysis so that it addresses the research
questions.

We recognize that not all studies will have a comprehensive planning stage and that certain outcomes may not be fully anticipated. For this reason, we present some options for what to do in situations where results are not statistically significant and include a selection of user-friendly resources at the end of this editorial to provide additional background on all of the statistical concepts that form the basis of our perspective.

In part 2 of this editorial, which will appear in the July issue of JOSPT, we will provide additional comments on the topic of statistical power and the design of clinical studies. The motivation for this second part came from the questions that investigators have asked us on this topic in the course of our research collaborations. Part 2 will present our responses to these frequently asked questions.

What is the most appropriate way to interpret and report a nonsignificant result? This depends on the goals of the project. A well-designed study should support decisions that are relevant to the research goals. In our hypothetical example, Dr. Sears did all of this careful planning. She intended to recommend the new material for orthotic inserts to practitioners if patients experienced less foot pain by a clinically significant amount. If the study results did not support this outcome, she would then report that the study failed to detect a clinically important difference between the 2 materials.

View Full Article