hirteen days ago, as his computer crunched the mountain of
data he hoped would be his humble contribution to medical progress, the
researcher - he shall remain nameless - got a phone call he'd
It was Dr. John Halamka, the former emergency-room physician who runs
Beth Israel Deaconess Medical Center's gigantic computer network. He
told the professor that his flood of numbers was overwhelming the
system, threatening to freeze thousands of electronic medical records
and grind the hospital's network to a halt.
''He said, `Oh, my God!' and pulled the plug out of the wall,''
Halamka said last week.
It was too late. Somewhere in the web of copper wires and glass
fibers that connects the hospital's two campuses and satellite offices,
the data was stuck in an endless loop. Halamka's technicians shut down
part of the network to contain it, but that created a cascade of new
The entire system crashed, freezing the massive stream of information
- prescriptions, lab tests, patient histories, Medicare bills - that
shoots through the hospital's electronic arteries every day, touching
every aspect of care for hundreds of patients.
Within a few hours, Cisco Systems, the hospital's network provider,
was loading thousands of pounds of network equipment onto an airplane in
California, bound for a 2 a.m. arrival at Logan International Airport.
In North Carolina's Research Triangle area, computer experts were being
rousted out of bed to join a batallion of electronic shock troops who
would troubleshoot the situation. Closer to home, Cisco technicians were
converging on Boston from across Massachusetts.
The crisis began on a Wednesday afternoon, Nov. 13, and lasted nearly
four days. Before it was over, the hospital would revert to the paper
systems that governed patient care in the 1970s, in some cases reverting
to forms printed ''Beth Israel Hospital,'' from before its 1996 merger.
Hundreds of employees, from lab technicians to chief executive officer
Paul Levy, would work overtime running a quarter-million sheets of paper
from one end of the campus to the other.
And hospitals across the country - not to mention investment banks,
insurance companies and every other business that relies on a constantly
accessible stream of quickly-changing information - would get a scary
reminder of how dependent they are on their networks, and what would
happen if they disappeared.
''It's like the Y2K that never happened,'' said Dianne Anderson, vice
president for patient care services at Beth Israel Deaconess.
Now, Halamka - the hospital's chief information officer and a
networking addict who answers e-mails on his Blackberry device whether
he's at a meeting or a family dinner - is hustling to answer questions
from all over the country, from community hospitals in Western
Massachusetts and major medical centers such as Johns Hopkins
University, and financial-services companies that could lose millions in
''The message,'' he said, ''is make sure you're ready for a massive
disruption of your network - whether it's 9/11 or a natural disaster or
As a result of the crash, Beth Israel Deaconess plans to spend $3
million to replace its entire network - creating an entire parallel set
of wires and switches, double the capacity the medical center thought it
No other Massachusetts hospital has ever reported such a long-lasting
or disruptive network crash, said Elliot Stone, executive director of
the Massachusetts Health Data Consortium, a group that brings together
chief information officers from hospitals and health plans around the
state. He praised Beth Israel Deaconess for being open about the problem
and sharing lessons learned, both about technology itself and about
policy - such as the need to enforce rules against unauthorized
additions of new software onto the network. Not least, Stone said,
Halamka's counterparts see the incident as ammunition in their constant
quest to convince management to pay for network
The crash surprised experts in the field because most disaster
planners mainly worry about backing up hard drives and building
redundant servers. But in this case, it wasn't those repositories of
information that were in trouble. It was the network itself - the
''pipes'' that carry the information from one place to the other. It was
like when at busy times at the office, your e-mail slows down - only so
bad that everything ceased to function.
''Usually, when you think about backup, you're talking about backing
up hard drives. You don't think about the network itself,'' said Mark
Tuomenoksa, founder and chairman of Woburn-based OpenReach, a
network-security consulting company.
Halamka said that was the case at Beth Israel Deaconess: ''We
don't just have a backup generator, we have a backup-backup generator,
and then we have batteries. Servers are clustered; data writes on five
different hard drives.'' There is even a double ''pipeline'' between the
computer center on Tremont Street and Beth Israel Deaconess's main
campuses - but during the crash, both were clogged.
The crisis had nothing to do with the particular software the
researcher was using. The problem had to do with a system called
''spanning tree protocol,'' which finds the most efficient way to move
information through the network and blocks alternate routes to prevent
data from getting stuck in a loop. The large volume of data the
researcher was uploading happened to be the last drop that made the
Halamka said Beth Israel Deaconess's recent economic troubles were
not behind the problem. In fact, on Oct. 1, hospital officials had
approved a consultant's plan to overhaul the network - just not quite in
time. ''Now,'' he said, ''we're going to do it faster.''
The crisis also tapped into medicine's ambivalence about computers.
Yesterday, doctors at Brigham and Women's Hospital reported in the
Archives of Internal Medicine that 73 percent of medication-related
mistakes involved in malpractice claims are preventable and probably
could be averted through computerized prescription ordering - the latest
in a growing pile of evidence that computerization can cut medical
At the same time, clinicians have sometimes been wary of turning over
control to a computer, Tuomenesko said: ''When I enter something into a
computer, how do I know it got there?''
That was part of the problem Beth Israel Deaconess had: New
information could sometimes be entered, but since network function was
fading in and out, clinicians weren't sure whether that information was
being delivered. So, the hospital decided to shut down the computers -
taping handwritten ''Do Not Use'' notes to monitors - creating an
instant generation gap, said Anderson, the hospital's top nurse
''Nurses and doctors over the age of 35 were very much at ease,'' she
said. ''The younger nurses and doctors were very uncertain. We were
teaching residents how to write orders; we were showing nurses how to do
Meanwhile, the hospital was figuring out how to run at its usual pace
without the 100,000 e-mails it usually sends a day. The lab was dumping
3,000 results a day on paper into plastic bins, to be delivered by
runners who came by every 10 to 15 minutes. Microbiologists were
ferrying lab results. Cardiac fellows were digging through paper records
to find old cardiograms to compare to new ones. People at all levels of
the hospital hierarchy had to deal with each other face to face.
''The lab is usually anonymous until something goes wrong,'' said
Gina McCormack, technical director of the West Campus lab. ''A lot of
people realized we're here. People got to understand each other's
Anne Barnard can be reached, when the network is working, at