| “Performance
measurement” is a relatively new concept in
medicine. Therefore considerable uncertainty exists
regarding its meaning and how the concept applies
to the daily lives of physicians. This article explains
how performance is measured, why it is measured
and what is being done nationally to measure physician
performance. In the context of physician performance
measurement, performance is defined as “the
processes a physician applies when rendering clinical
care and the outcomes resulting from applying those
processes.” Therefore measuring performance
is measuring processes and outcomes.
How Are Processes and Outcomes Measured?
The four steps in measuring a process are:
1) Define the process to be measured.
2) Identify the population of patients in whom
the process should be measured (the denominator).
3) Count the number of times a physician carries
out the process in daily practice (the numerator).
4) Divide the numerator by the denominator.
The process measure is the ratio of the number
of times the physician carried out the process (the
numerator) divided by the number of times carrying
out the process would be appropriate (the denominator).
For example documentation of the patient’s
end-tidal carbon dioxide tension (PCO2) during an
anesthetic is a process that could be measured.
The numerator might be defined as the number of
patients in whom the anesthesiologist documented
end-tidal PCO2 at least every 15 minutes
on the anesthetic record. The denominator would
be the total number of patients anesthetized by
the anesthesiologist, corrected for inclusion and
exclusion criteria. An inclusion criterion might
be “patients receiving general anesthesia”
(as opposed to regional anesthesia or monitored
anesthesia care). An exclusion criterion might be
“patients receiving general anesthesia by
mask” (as opposed to endotracheal tube or
laryngeal mask airway [LMA]). The process measure
would then be “percentage of patients receiving
general anesthesia by endotracheal tube or LMA in
whom the anesthesiologist documented end-tidal PCO2
at least every 15 minutes on the anesthesia record.”
Measuring a physician’s performance is meaningful
only to the extent that the physician has control
over the variables being measured. Measuring outcomes
is particularly problematic because many variables
contribute to patient outcomes that are not under
control of the physician. For example one could
measure “percentage of an anesthesiologist’s
patients who have suffered cardiac arrest during
anesthesia,” which is an outcome. To be meaningful,
that number would have to be corrected for a host
of variables that were not under control of the
anesthesiologist. That is, the outcome measure must
be “risk-adjusted.” In this example,
variables such as the patient’s cardiac risk
factors and the nature of the surgery being performed
would be included in the risk-adjustment model applied
to the data.
Data on a large number of risk factors must be collected
on every outcome requiring risk adjustment. This
can be a considerable burden because data on 10
or more risk factors may have to be collected for
every outcome data element collected. Furthermore
the science behind risk adjustment models is not
yet robust, and the models must be continually updated
as the data on which they are based expand and change
with time. Beyond problems associated with risk
adjustment, determining when a physician’s
performance on a given outcome measure deviates
significantly from the norm (that is, when the physician
becomes an “outlier”) requires statistical
analysis of large numbers of cases. In practice
the number of outcomes needed for valid statistical
analysis usually far exceeds the number of cases
available.
In contrast, adjusting process measures for variables
that are not under control of the physician is relatively
simple. The exclusion criteria for process measures
should be able to account for confounding variables.
For example, the denominator used to calculate “percentage
of patients with cardiac risk factors who receive
perioperative beta-blocker therapy for noncardiac
surgery,” which is a process measure, could
be adjusted to exclude “patients who are intolerant
or allergic to beta-blockers.” When exclusion
criteria are sufficiently rigorous, compliance with
the process should be 100 percent. Because processes
can generally be measured more easily and more accurately
than outcomes, most performance measurement sets
being developed today focus on processes rather
than on outcomes.
One important caveat, however, exists regarding
process measurement. Before a process is incorporated
into a performance measurement set, scientific evidence
should exist documenting that compliance with the
process improves patient outcome. Plausibility (“face
validity”) is a weak surrogate for scientific
evidence. Unfortunately many widely accepted processes
have not been scientifically validated to show that
they lead to improved patient outcomes.
Why Is Performance Measured?
Results of performance measurement are typically
used for two very different purposes: 1) to improve
quality of patient care and 2) to hold physicians
accountable. Certain types of performance measures
can serve both purposes. Most performance measures,
however, are appropriate only for quality improvement,
not for holding physicians accountable. For example
outcomes that are not risk-adjusted for variables
that are not under the physician’s control
should never be used to hold physicians accountable.
On the other hand, such outcome measurements, when
carefully analyzed in context, can be useful for
improving quality of patient care. Institutional
quality improvement committees must necessarily
work with small numbers of events that preclude
even rudimentary statistical analysis, let alone
application of sophisticated risk-adjustment models.
In contrast, data used to hold physicians accountable
must be capable of rigorous statistical analysis
and must be adjusted for risk. Several years ago,
the Health Care Financing Administration (HCFA),
now the Centers for Medicare & Medicaid Services
(CMS), released data on hospital mortality that
was not adjusted for patient characteristics that
were not under control of the hospitals. Much harm
was caused by the misleading data before HCFA withdrew
the report, finally recognizing that hospital mortality
usually has more to do with the degree of illness
of patients admitted than with the quality of patient
care provided. The same mistake should not be made
in reporting physician performance.
Except under rare circumstances, outcome data cannot
be used to hold individual physicians accountable
because the necessary tools for risk-adjustment
do not exist and the amount of data required to
ensure scientific validity is unavailable. Similarly,
using process data to hold physicians accountable
is hampered by small sample sizes and insufficient
numbers of process measures that are validated to
have a significant effect on patient outcomes. A
recently published comprehensive analysis concluded,
“At the current time, given the state of technology
and the existing infrastructure to support performance
assessment, broad-based mandatory clinical performance
assessment for individual physicians as means of
determining the competence of individual physicians,
whether for board certification or other reasons,
appears to be infeasible.”1
What Is Being Done Nationally to Measure
Physician Performance?
The current status of physician performance measurement
is reflected in the fact that national initiatives
in performance measurement are currently being focused
on developing the tools needed to measure performance
rather than on actually measuring performance. Application
of those tools is only being carried out in pilot
studies. The national efforts described below are
all related to improving quality of patient care,
not to holding physicians accountable.
Physician Consortium for Performance Improvement
(The Consortium). The Consortium, which
has been convened by the American Medical Association
(AMA), is currently made up of representatives from
54 medical specialty societies (including ASA) and
14 other medical organizations.
The following description of the Consortium’s
activities is extracted from an AMA brochure:
“The Consortium aims to provide performance
measurement resources for practicing physicians
to facilitate implementation of clinical quality
improvement programs… Consortium members
collectively seek to unify the medical profession’s
efforts to develop and identify effective performance
measures and to promote the appropriate use of
measures and measurement systems to address health
care quality and patient safety issues.”
2
To date, the Consortium has developed nine performance
measurement sets, primarily for use by primary care
physicians:
1) Asthma
2) Chronic stable coronary artery disease
3) Adult diabetes
4) Heart failure
5) Hypertension
6) Major depressive disorder
7) Osteoarthritis of the knee
8) Prenatal testing
9) Preventive care and screening3
The measurement sets are evidence-based, most of
them related to clinical practice guidelines developed
by the medical specialty societies that participated
in drafting the measurement sets.
Category II CPT Codes. To facilitate
the reporting of performance measures, AMA is incorporating
category II (performance measurement) codes into
its Current Procedural Terminology (CPT™)
system. To date, eleven category II CPT codes have
been approved 4 [Table
1].
 |
|
|
Table
1: Category II CPT Codes
|
| 0001F — Blood pressure,
measured |
|
| 0002F — Tobacco use,
smoking, assessed |
|
| 0003F — Tobacco use,
non-smoking, assessed |
|
| 0004F — Tobacco use
cessation intervention, counseling performed |
|
| 0005F — Tobacco use
cessation intervention, pharmacologic
therapy prescribed |
|
| 0006F — Statin therapy,
prescribed |
|
| 0007F — Beta-blocker
therapy, prescribed |
|
| 0008F — ACE inhibitor
therapy, prescribed |
|
| 0009F — Anginal symptoms
and level of activity, assessed |
|
| 0010F — Anginal symptoms
and level of activity, assessed using
a standardized instrument |
|
| 0011F — Oral antiplatelet
therapy, prescribed |
|
|
All of the above codes are for process measures
that have been clearly defined in the measurement
sets from which they were derived, including specification
of the populations of patients for whom the measures
are appropriate (denominator data adjusted with
inclusion and exclusion criteria).
Doctors Office Quality Project (DOQ). The
DOQ project is an initiative of CMS. Following are
descriptions of the project that appear on the CMS
Web site.5
Purpose: The goal of the
DOQ collaborative project is to define overall quality
measures that assess, and strategies that improve,
clinician performance in providing ambulatory care
for persons with chronic disease. The project will
enable clinicians to examine how well they are providing
chronic disease care to Medicare beneficiaries.
It also will provide tools that physicians and their
office staff can use to achieve excellence in care.
The three-year project will develop a model for
measurement and improvement of quality of care for
chronic disease and preventive services at the level
of the individual physician/medical office.
Three-State Pilot: The
testing phase of the DOQ pilot will test measures,
improvement strategies and incentives and will take
place in California, Iowa and New York. Generalist
physicians (internists, general practitioners and
family physicians) will be involved in each of the
three states.
The pilot project is scheduled to run from November
2002 through September 2005. Several performance
measurement sets developed by the AMA’s Physician
Consortium (described above) are being used in the
DOQ project.
How Can Anesthesiologists Measure Performance?
Anesthesiologists are much more likely than primary
care physicians to work in hospital settings in
which clinical data are collected in large relational
databases. Some tools have already been prepared,
and others are being prepared to facilitate the
collection of performance measurement data as a
part of routine collection of clinical and administrative
data.
ASA has developed “Guidelines for Database
Management,” which the ASA House of Delegates
approved in October 2000. The Anesthesia Patient
Safety Foundation (APSF) is leading an international
initiative (currently involving the United States,
Canada and the United Kingdom) to create a comprehensive
dictionary (taxonomy) of clinical and administrative
terms used in anesthesiology.6
Use of standardized terms is essential when comparing
performance data between individuals or between
organizations (benchmarking). Because APSF is working
closely with the vendors of automated anesthesia
records, all clinical and administrative terms used
in commercial, automated, anesthesia data collection
systems should eventually be mapped to the standardized
terms in the International Organization for Terminology
in Anesthesia data dictionary. Through this mechanism,
clinical and administrative data that are collected
by disparate anesthesia clinical information systems
should be capable of being compared across institutions.
Summary
Performance measurement is an emerging concept that
is destined eventually to become an integral part
of patient care. Performance measurement is not
yet capable of being used to determine physician
competence or to hold physicians accountable. However,
the value of performance measurement in quality
improvement looks promising and is currently being
tested in pilot projects.
| |
|
Ronald A. Gabel, M.D., is Professor Emeritus
of Anesthesiology, University of Rochester,
Rochester, New York. |
|
|