“Safe and effective” is a short but powerful phrase, and it probably appears in most issues of ophthalmology journals. Authors may truly believe that they have proven something to be safe and effective and are eager to say so. There are powerful incentives to include the phrase in some manuscripts, as the words in some way are synonymous with “Food and Drug Adminstration (FDA) approved” or, at least, “approvable from a clinical perspective.” Like many other phrases, the strength of the words requires that the meaning be consistent for everyone who uses the phrase. Authors have an obligation to be sure that safe and effective is what is meant. Journal reviewers, when reading the phrase, ought to consider if it is appropriate. The editors have the final say for the journals, and when we err, letters to the editor should ask to reopen the discussions and offer other points of view.
Just what is meant by “safe and effective” often depends on who is saying it and in what context. The FDA has very specific views on what is meant, and the journals' are similar, although frequently more liberal. In this editorial, we summarize the FDA's interpretation and the journals'. Of course, the FDA's official interpretation is spelled out in legal code, publications, and regulations; our journals surely do not speak on their behalf.
“Safe and effective” is common language at the FDA, and its use is based on the Federal Food Drug and Cosmetic Act, as amended. Specifically, section 505(b)(1) explains that “any person may file … an application with respect to any drug … such persons shall submit … as part of the application … full reports of investigations which have been made to show whether or not such drug is safe for use and whether such drug is effective in use. …”
The regulations then define that the investigations need to be “adequate and well controlled studies”; the description of such studies is contained in section 21 CFR 314.126 of the Code of Federal Regulations. The text of the Federal Food Drug and Cosmetic Act includes the phrase “full reports of investigations” as plural words. Therefore, Congress's definition of safe and effective referred to more than one study. Rarely would any article in an eye journal meet this standard.
The size of the study may also make a difference. From another part of the regulations, there is a definition of phase 1, 2, and 3 studies (21 CFR 312.21). Phase 1 studies are “designed to determine the metabolism and pharmacologic actions of the drug in humans, the side effects associated with increasing doses, and, if possible, to gain early evidence on effectiveness.” The total number of patients in a phase 1 study is generally 20 to 80. Phase 2 includes the controlled studies conducted to evaluate the effectiveness of the drug for a particular indication “usually involving no more than several hundred subjects.” Phase 3 studies “are performed after preliminary evidence suggesting effectiveness of the drug has been obtained, and are intended to gather the additional information about effectiveness and safety that is needed to evaluate the overall benefit-risk relationship of the drug. …” Phase 3 studies usually include “several hundred to several thousand subjects.” These numbers are clearly higher than the numbers reported in most articles that claim safety or efficacy.
Our manuscripts typically are prepared to share information that might enhance and advance patient care, or that might spark an idea that leads to enhanced patient care in the future. When deciding what we mean by “safe and effective,” we can consider what the author's or subscriber's opinion is, but the journal editors must apply this term with patient expectations in mind. Different patients in different situations, with different needs, resources, and backgrounds, may evaluate an impartial summary of an identical data set, yet reach very different conclusions. A helicopter pilot with sophisticated needs for stereo acuity might accept a risk to improve vision from 20/40 to 20/20 in a second eye, whereas a retired banker might never dream of considering such a risk. So, whether something is safe or effective is ultimately judged by the health care consumer, the patient. The FDA will make its own determination, but the journals' requirements ought to be that the authors lay out the data with sufficient clarity so well-reasoned judgments can be made.
To allow readers to make informed judgments, authors should spell out their definitions of safety and efficacy. The reviewers and editors will decide if they are reasonable and, as needed, initiate negotiations with the authors to refine the definitions. With respect to efficacy, we prefer to see in the manuscript “efficacious compared to what?” A contemporaneous comparison group from a prospective randomized trial allows the greatest confidence, and two such trials may make believers of Congress as well. Other comparison groups may be acceptable, but confidence in the efficacy of a treatment or management approach diminishes quickly if not based on a prospective randomized trial. Randomization, comparability at baseline, adequate sample size, adequate duration of follow up, and attempts to minimize bias are all factors that influence the confidence with which we accept the authors' conclusions.
Authors are encouraged to spell out how they choose to keep score, and reviewers and editors will decide if the proposed rules are appropriate. So, if the authors view the safety of a new drug (e.g., an eye drop for itchy eyes) as meaning that patients are not likely to die from using the drop, they might report mortality data. If allergy to the drop were the key safety measure, that outcome would be reported. The possibilities to be considered are quite large, but significant guesswork could be eliminated. Again, the reviewers and the editors would be the initial arbiters.
Once the safety measures are enumerated, sample size becomes the key factor in considering safety claims. Is the study adequately powered to detect an appropriately selected safety outcome rate? So, for a severe disease, such as metastatic cancer, a 1% drug-related mortality rate might be acceptable. For itchy eyes, it is not. The question becomes “Is the sample size adequate to detect a 1% mortality rate (if a cancer study) or an extremely low mortality rate (if an itchy eye study)?” To know this, one needs a primer on statistics, a statistician, or a good rule to follow.
Luckily, we have a relatively easy rule to follow for a good approximation, known as the “rule of three.” This rule states that if none of n patients has the adverse event in question, then we can be reasonably confident (95%) that the true rate of this event in the population is no more than 3 in n (3/n).2 For example, if a study has 200 patients in it, and none of the patients has a heart attack, we can be reasonably confident that the true rate of heart attacks after using this drug is less than 3/200, or 1.5%. Similarly, if a study has 10 patients in it, and none of the patients develops ocular hyperemia, we can only be confident that the rate of red eyes from taking this drug product is no more than about 3/10, or 30%. Although these may each represent frighteningly high percentages, the calculation is easily performed and interpreted.
The first application or teaching of this rule of three is not known. A search of the medical literature yields reminders of the rule in different journals every few years. The mathematical formulas to support this rule, though left in the cited reference for ambitious readers, are relatively straightforward. The approximation of the rule to the full mathematical model is very close, even at relatively small values of n.
In conclusion, we ask that authors take care when they use the charged words “safe and effective,” and that reviewers should consider if the authors' proposed use is appropriate. The editors will cast the deciding vote. When we err, we should expect letters. Our readers need this information to assist their patients with complex medical decision making and deserve nothing less.
FULL TEXT