October 1997
Volume 61 |
Number 10
|
| |
|
| How Can Full Environment-Realistic
Patient Simulators Be Used for Performance Assessment? |
David M. Gaba, M.D.
Stephen D. Small, M.D.
Since their inception in the late 1980s, a recurrent question
about patient simulators has been, "Can (or will) they be
used for assessing performance?" Performance assessment has
always been important in training centers, but emerging realities
in health care delivery are adding new emphasis to this question
as pressure increases from a variety of sources for clinicians
to do more with less and, as demands increase, for more complete
performance assessment of clinicians as groups or individuals.
Additionally, problems of human performance are now recognized
as major contributors to preventable, undesirable outcomes in
medicine, just as in other complex domains. New tools and methods
will be needed to thoroughly describe the nuances of complex clinical
behaviors and to establish benchmarks to help guide quality improvement
and meaningful improvements in performance. In this article, we
will provide an overview of the issues concerning realistic patient
simulators and performance evaluation.
There are many different applications for performance assessment.
Research on the spectrum of skills and human vulnerabilities of
clinicians is one current application. Some applications relate
to measuring the impact of educational and training efforts for
students, residents or participants in continuing medical education.
Others relate to certification, competence assessment and remedial
evaluation of residents or the peer review of practitioners.
Advantages of Simulation for Performance
Assessment
Currently, there is no way to test the clinical abilities of
students, residents or practitioners comprehensively in the conduct
of patient care in anesthesia. Written and oral examinations can
assess one's knowledge base, reasoning ability and consultative
verbal skills, but do so in a controlled, isolated setting without
the true demands of the clinical environment. In reality, clinicians
simultaneously apply cognitive abilities and knowledge, perform
hands-on psychomotor skills and flexibly exercise a wide variety
of complex interpersonal behaviors in concert with other trained
practitioners. The very characteristics that make work so challenging
in the complex dynamic arena of the operating room or intensive
care unit are often missing from written or oral examinations.
Observing practitioners at work could capture some of these qualities,
but that is often impractical and does not provide a way to systematically
assess the ability to manage evolving critical
incidents.
Realistic simulation utilizing a computerized surrogate mannequin
patient, actual clinical equipment and staffing typical of the
clinical environment offers a number of advantages in evaluating
performance when compared to traditional testing. Such simulations
allow clinicians to behave as clinicians while carrying
out the management of standardized patients and situations. Standardized
"simulated patients" played by trained actors have been
used for performance assessment in "static" medical
domains like internal medicine. Realistic patient simulators can
provide the standardized patients and environment for equivalent
assessments of clinician behavior in anesthesiology. Moreover,
traditional methods of evaluation in anesthesiology are limited
in their ability to test performance "under fire," during
unexpected, risky, abnormal conditions.
Serious critical incidents are uncommon and occur unpredictably
in each practitioner's experience. The nature of the underlying
cause of any given event may puzzle even the experts, making it
difficult to evaluate a clinician's response. With real patients,
an evaluator cannot ethically allow an adverse situation to evolve
to its natural conclusion without intervening. In the simulator,
the clinician has a realistic opportunity to detect and correct
evolving problems, even if they are not picked up immediately.
New Methods of Performance Assessment
Traditional self-assessment or observations of actual clinical
behavior by faculty or peers are limited by the vagaries of memory
and subjectivity. Videotape recording enhances the objectivity
of performance evaluation by others, and it enables more detailed
and novel studies of complex behaviors, perception, integration,
decision-making and teamwork. Videotape recording is easy to conduct
during simulated cases but hard to arrange during real cases.
Viewing videotapes after simulated cases has proven to be a powerful
tool in self-assessment.
Other than ad hoc self-assessment, what performance assessment
techniques can be used during or after simulation sessions? One
question is: Can the "clinical outcome" of a simulated
case be used to judge the clinician's performance? How well does
the simulator predict how a real patient would have fared given
the actions of the clinician? Simulators use mathematical models
of "typical" physiology and pharmacology for given underlying
patient states, but they cannot predict how any individual patient
would behave. At the extremes of very good or very bad performance,
the simulator's physiologic behavior is likely to be a meaningful
predictor of outcome.
For example, the failure to defibrillate a simulated patient
with ventricular fibrillation will unequivocally lead to death
(excepting situations of cardiopulmonary bypass). The converse
is not true, in that even optimal treatment of this situation
may be unsuccessful. The mathematical models are not (and we believe
they will never be) sufficiently precise to predict what would
happen to any specific patient after complex sequences of therapy
and subtle patient care judgments.
Thus, the clinical outcome of the simulated patient is only one
datum that can be used in some cases to assess the performance
of the anesthesiologist. In the foreseeable future, any credible
performance measurement technique must involve subjective and
semi-objective judgments by clinical experts.
Skilled performance embodies both technical and behavioral aspects;
simulation offers benefits in assessing both. Unlike a real case,
the underlying nature and cause of a simulated critical incident
is known with certainty. This allows the definition of a continuum
of responses from contraindicated actions to failure to respond,
to partially effective actions, to a fully effective set of actions.
A participant's actions can be compared to this continuum in a
relatively objective fashion. Several investigations in the United
States, Canada and Europe indicate that evaluating the technical
performance of anesthesiologists during simulated management of
complex situations is possible.
On the other hand, the behavioral components of patient care
must be assessed in a qualitative fashion. The Veterans Affairs/Stanford
Simulation Center has described its experience using multiple
raters to conduct evaluations of crisis management behaviors from
videotapes of simulation runs. The evaluators used carefully anchored
ordinal scales of behaviors such as leadership, communication,
distribution of workload and overall performance. The evaluation
procedure clearly distinguished individuals and groups who performed
extremely well or extremely poorly in comparison to the average
behavior. They did find that any single rating could differ strongly
from the mean of the others, suggesting that more than one rater
should be used, as is currently practiced in board examinations.
Clearly there are many issues of validity to consider as well
as potentially confounding factors that could affect performance
on simulated cases. These include motivational problems, inability
to suspend disbelief concerning the "plastic" patient,
ambiguous or missing cues in the mannequin, and differing levels
of familiarity with clinical equipment and layout of the simulated
operating room.
Studies of performance during simulated cases show that performance
can differ strongly between individuals and, in some cases, within
individuals when tested on different scenarios or at different
times. Even highly experienced and skilled individuals and groups
do not perform flawlessly at all times. Perfection in performance
is unattainable. An important step in the development of simulation-based
performance assessment will be the identification of the actual
spectrum of performance of highly experienced clinicians to serve
as a benchmark in the evaluation of others.
A number of other improvements in the evaluation procedure have
been suggested, including developing better means to aggregate
a performance evaluation over time from dynamically changing behaviors,
providing multiple scenarios for evaluation and incorporating
data from the subject's verbal justification of the actions taken
as discussed in a post-simulation debriefing.
Anesthesiologists Will Lead the Application of Simulators
in Health Care
Until about 1993, there were only a handful of centers with a
realistic simulator in use. Today we are in the middle of a period
of greater acceptance and use of simulation, and there are now
more than 50 simulator centers worldwide, with quite a few taking
on the challenge of providing high fidelity training in anesthesia
care. It is a logical extension of the growing interest in these
tools that a number of approaches to simulation-based assessment
of performance will arise in a wide variety of contexts.
Anesthesiologists have been the leaders within medicine in applying
simulation technology and have gone on to develop training curricula
for simulation-based training for a number of nonanesthesia health
care domains (see accompanying article).
Considerable work remains to be done to determine when and how
realistic patient simulators can be used for the many different
applications of performance assessment, and it is likely that
here, too, anesthesiologists will lead the way.
Disclosure Statement
Dr. Gaba and his former research associate, John Williams,
M.D., have received license fees and royalties for the licensing
of simulation technology.
David M. Gaba, M.D., is Associate Professor
of Anesthesia at Stanford University School of Medicine, Stanford,
California, and Staff Anesthesiologist at Palo Alto Veterans Affairs
Medical Center, Palo Alto, California. He also is Director of
the VA/Stanford Simulation Center for Crisis Management Training
in Health Care.
E-mail the author.
Stephen D. Small, M.D., is an Instructor
in Anesthesia at Harvard Medical School, Boston, Massachusetts.
He also is Coordinator for Research and Development at the Boston
Anesthesia Simulator Center.
return to top
Home >Newsletters
>October 1997Home >Test |