Statistical significance and single arm studies

Jennifer Carroll

Is statistical significance required in the proposed future RWE guidance? If so, how can a single arm study can be presented?

We’ll start by building some context that supports the answer.

What is a single arm study?

A study type in which a sample of individuals with the medical condition are given the therapy and followed over time (either prospectively or retrospectively) to observe their response. This approach does not include a second sample of individuals to compare to.

Challenge of interpreting effectiveness from a single arm study

While this is a very simple trial design, the interpretation of effectiveness results is complicated by an inability to distinguish between a real treatment effect, a placebo effect, the natural course of the disease, or any combination of these. A robust interpretation of effectiveness generally involves a frame of reference for comparison.

Challenge mitigating the risk that apparent associations are due to chance alone

The preferred way to mitigate the risk that observed associations are due to chance alone is to perform statistical analysis on the difference or ratio between two groups that are very similar but that differ with regards to the independent variable. This sort of inter-group statistical analysis is not possible in a single arm study.

How does the above shape requirements pertaining to use of single arm studies in HCP advertising?

Single arm trials can be considered as the basis for data presentations pertaining to adherence/compliance, persistence/retention, safety and effectiveness. The explanatory statement (i.e., the statement next to the icon) must identify that this is a single arm study.

For such a study to be considered as the basis for presentations relating to safety and effectiveness, it must be published and peer-reviewed.

For effectiveness endpoints, the disclosure of key study limitations must also specify that the methodology may make it difficult to differentiate between:

drug effects and the natural history of the disease
drug effects and placebo effects

Please note that this requirement is in addition to the identification of the source as a single arm study in the explanatory statement.

Statistical analysis is not required in single arm studies. However, prominent disclosure of the sample size and a measure of dispersion in the outcome across the sample size is required.

For single arm studies, the PAAB will require that sample size and a measure of sample dispersion be presented within the body of the presentation (as opposed to among the study parameters). When a measure of sample dispersion is not available, the unavailability of this data must be included as a study limitation within the body.
This is intended to help HCPs evaluate how/whether they wish to use the study presentation in consideration of their view of:

the appropriateness of the sample size (n) in comparison to their understanding of the size of the corresponding patient population (N)
the degree of heterogeneity