Clinical decision making is usually driven by some combination of scientific evidence, clinical judgment, financial constraints, and patient preference. In order to integrate scientific evidence from psychiatric outcomes research into clinical practice, psychiatrists need to make informed judgments about the relevance or generalizability of study findings to their actual clinical care. A concern that treatments do not seem to work as well in routine practice as they do in randomized controlled trials has motivated efforts to build features of usual practice into psychiatric outcomes research.
A new generation of studies, effectiveness research, seeks to shorten the distance separating data collection from clinical application. It has developed out of a growing awareness of the limitations of efficacy studies, which are conducted under carefully controlled but atypical conditions. In this article, I first discuss the characteristics of and differences between efficacy and effectiveness research. I then review some emerging methods in mental health effectiveness research, illustrating them with examples of the types of findings such studies can produce.
Several features of traditional efficacy research limit its clinical relevance and may account for why treatments do not appear to work as well in actual clinical practice as they do in clinical trials. First, clinical trials often have highly restrictive eligibility criteria that exclude patients who may be the most difficult to treat. These stringent eligibility criteria complicate efforts to generalize or apply findings to typical patients in routine settings. For example, a recent study of sertraline for panic disorder excluded patients with other anxiety disorders, major depression, dysthymia, personality disorders, bipolar disorder, schizophrenia, and organic mental disorders.
Because the vast majority of patients with panic disorder also meet criteria for other mental disorders,2 such restrictive eligibility criteria greatly compromise the extent to which findings can be safely generalized to clinical samples. The results of some efficacy studies, particularly those involving highly specialized psychological interventions, may also be difficult to transport or replicate in community practice. Second, clinical trials are typically limited to several weeks in duration and so focus on acute recovery rather than maintenance treatment. By contrast, most psychiatric visits are made by patients who are returning for maintenance treatment.3 Clinical improvements are typically use far less restrictive patient selection criteria than efficacy studies.
Antipsychotic medication was prescribed without restriction by attending psychiatrists. Over a 2-year follow-up, patients who received the social skills training achieved more independent community functioning. A tension exists between the “rigor” of efficacy studies and the “relevance” of effectiveness research. Outcome studies that provide the strongest and most scientifically credible findings are often the most difficult to translate to general psychiatric practice where patients present with complex diagnostic problems and treatment needs. Outcomes research conducted in more naturalistic settings generally yields more readily applicable but uncertain findings. Both types of studies are clearly important and, ideally, the strengths of each should complement the limitations of the other
The concepts of efficacy and effectiveness represent poles along a continuum. The best location for a given study on the efficacy/effectiveness continuum may depend upon the state of knowledge in a given area and the leading clinical concerns. As knowledge of an intervention increases, research generally moves in the direction from efficacy to effectiveness research. For example, the efficacy of antidepressants in the treatment of psychiatric patients with major depression is well established and there is little value in conducting randomized clinical trials (i.e., efficacy studies) of FDA-approved antidepressants.
However, the effectiveness of antidepressants in the treatment of major depression in other treatment settings, such as primary care clinics, is less well established. A study conducted by Schulberg et al. on this topic appropriately combined a mixture of efficacy and effectiveness elements. It was a randomized controlled trial that used narrow patient eligibility criteria and intensive follow-up assessments (common features of efficacy research) but also trained front line clinicians to prescribe the medications and included a cost analysis component (common features of effectiveness research).
THE SPECTRUM OF EFFECTIVENESS RESEARCH
Effectiveness research draws on a variety of research methods and addresses a wide range of topics. In the following section, I describe three general types of effectiveness research: Experimental effectiveness research Quality of care research Observational effectiveness research One important methodologic distinction among effectiveness research studies is whether the study applies random assignment to treatment, a feature of experimental effectiveness research, or observation of naturally occurring variations in treatment, a feature of quality of care and observational effectiveness research. Random assignment remains the sine qua non for establishing internal validity in outcomes research.
If the experimental conditions can be adequately maintained, random assignment is a powerful method for determining whether the observed effects are a consequence of the intervention under study as opposed to some other factor. In many situations, however, random assignment is not administratively or logistically feasible, so that other designs such as case-controls, time series, and historical controls must be used. In evaluating the clinical utility of research findings, it is important to balance the threats to internal validity common to observational research against the greater clinical applicability that comes from studying patients and treatments that broadly represent usual care without added restrictions on the nature of treatment.
Quality of care research concerns itself with describing aspects of practice variation without linking it to patient outcomes. At the most basic level, quality of care research involves documenting the extent to which treatment conforms with or deviates from accepted standards of care. In this type of research, investigators observe variations in clinical practice and rely on existing treatment outcome studies to evaluate the clinical significance of the observed variations.
Results often highlight areas of concern in the clinical delivery of care. A recent example of quality of care research is the Patient Outcomes Research Team (PORT) study of usual care for schizophrenia.11 By drawing on medical records and patient interviews, Lehman et al. determined that the rate at which outpatient treatment of schizophrenia conforms to recommendations in areas such as use of depot medication, use of adjunctive medication, family treatment, and vocational rehabilitation is below 50%.
The effects of aging on the metabolism of benzodiazepines are well known. Slower hepatic biotransformation and increased volume of distribution prolongs the half-life of long-acting benzodiazepines. For this reason, drugs with shorter half-lives are routinely recommended for the short-term treatment of insomnia in older adults. However, in one study, prescribing physicians showed little or no preference for shorter acting agents in elderly patients.13 This suggests that habit rather than sound pharmacokinetic principles may govern selection of benzodiazepine agents in the elderly. Such findings point out a need for targeted physician educational interventions. A higher level of quality of care research attempts to link variations in processes of care with specific service structures.
Service structures include physical facilities, providers, organization of services, and financing mechanisms. In this type of research, investigators attempt to uncover clinically significant variations in practice and attempt to relate this practice variation to some aspect of service structure.An illustration is provided by an analysis of discharge planning activities from a national survey of mental health facilities.14 It is well established that psychiatric inpatients who receive adequate discharge planning and transitional services are more likely to utilize outpatient mental health care and less likely to require rehospitalization in the months following hospital discharge.
In this survey, there was marked unevenness in the availability of transitional care services. Specifically, lower rates of telephone follow-up, staff assistance with referrals, visits to outpatient providers, and other critical services were found at general hospitals as compared with specialized psychiatric hospitals. These differences persisted after controlling for case mix, hospital ownership, and other factors. Structural or administrative differences between the psychiatric hospitals and general hospitals might account for the differences in transitional service availability. These results have implications for strengthening discharge planning services at general hospital psychiatric units.
Observational Effectiveness Research
Observational effectiveness research examines the degree, extent, and consequences of naturally occurring variation in the organization, financing, and delivery of mental health services as it relates to adherence or departure from established quality standards. Observational effectiveness research resembles quality of care research with the additional feature that variation in treatment processes is related to patient outcomes. In observational effectiveness research, a well defined group of patients is identified, differentially exposed to some key service in the course of usual care, followed over time, and examined for differences in outcome. This type of research tends to be expensive to conduct because relatively large sample sizes are required to capture sufficient variation in service delivery.
The internal validity of findings from observational effectiveness research is vulnerable to confounding. It occurs when a variable other than the treatment or outcome variables under study is independently related to the treatment and outcomes. Confounding can create apparent associations between treatment and outcomes or mask an association that actually exists. For example, an observational effectiveness study of intensive case management that fails to adequately adjust for the initial severity of patient illness might spuriously conclude that case management services lead to poorer patient outcomes when in fact the observation may be simply a consequence of more severely ill patients being selectively assigned case managers.
A recent study of hospital discharge planning illustrates observational effectiveness research methods. A group of adult short-term inpatients with schizophrenia were evaluated prior to hospital discharge and again 3 months later after being referred for outpatient care to clinicians who had not previously treated them. Patients who had telephone or face-to-face contact with the outpatient clinician before hospital discharge achieved significantly greater reductions in symptoms than patients who had no communication with outpatient staff prior to discharge.
Although it is possible that the differential outcome is related to group differences other than outpatient staff contact, the results nonetheless suggest that direct communication between inpatients and new outpatient clinicians helps smooth the transition to outpatient care. Changes in clinical policies provide opportunities to conduct observational effectiveness research. For example, when one community mental health center changed its rehabilitative day treatment program into a supported employment program, it offered a chance to make historical comparisons in the occupational outcomes of patients treated under the two rehabilitation models.
Author: MARK OLFSON
