One Manhattan power room took down FAA air traffic control in 1991

At 4:40 p.m. on Tuesday, September 17, 1991, during the late-afternoon arrival rush across the Northeast, air traffic controllers at New York’s regional radar center and the towers at Kennedy, LaGuardia, and Newark lost their radar feeds, primary radios, and automated flight-handoff links at the same moment.123 With planes approaching all three airports, controllers had to guide commercial jets on short-range backup frequencies and manual tower-to-tower relays while the FAA halted airport traffic and 1,174 flights were canceled or delayed.234 Nothing had failed at the airports; instead, every primary and backup FAA landline for the New York metropolitan area ran through transmission bays powered by the 20th-floor DC power room of 33 Thomas Street, a 29-story windowless granite AT&T switching hub in Lower Manhattan.123 More than six hours earlier, a lead-acid battery reserve built to keep those lines running through power cuts had taken over when the room’s rectifiers tripped during a routine generator transfer, hiding the failure behind steady 48-volt output until the batteries ran dry.15 This post is part of an engineering safety series building on safety-standards work from a 2014 IEEE publication I co-authored.6

Figure 2: 33 Thomas Street in Lower Manhattan, housing the 20th-floor DC power plant and regional transmission trunks. Photo: Marcin Wichary (CC BY 2.0).

Between 6:00 and 7:00 a.m. EDT, Consolidated Edison contacted AT&T’s Building Operations Group under a voluntary peak-load power-sharing agreement and asked the facility to transfer off commercial street power for the day.15 At 10:10 a.m., the AC power supervisor threw the transfer switch, moving 33 Thomas Street from Con Edison street AC to the building’s internal diesel generators.15

During the transfer, a voltage spike on the AC feed reached the 20th-floor power room, which supplied direct current to the building’s high-capacity DS3 (45-megabit T3) fiber and coaxial transmission bays.1 That voltage spike tripped an overload protection relay that had been misadjusted during recent plant modernization, knocking the room’s AC-to-DC rectifiers offline.15

In a central office DC power plant, the rectifiers and the battery strings sit in parallel across the main 48-volt (-48V DC) busbars. Under normal operation, the rectifiers hold the bus slightly above the resting voltage of the lead-acid cells, keeping the batteries topped up while supplying the live current drawn by the switching bays. When the 20th-floor rectifiers opened at 10:10 a.m., bus voltage settled a few volts down onto the battery plateau and stopped falling.

No circuit dropped, and every trunk kept running at full capacity. The building’s three 4ESS toll switches ran on separate power plants and were unaffected, while the 20th-floor batteries carried all of the high-capacity DS3 transmission bays.1 The leased AT&T data lines, remote radio links, and voice trunks serving the FAA’s New York Terminal Radar Approach Control (TRACON) facility in Hempstead, Long Island, and the control towers at Kennedy, LaGuardia, and Newark continued operating normally.1234 Inside 33 Thomas Street, the 20th-floor batteries had begun a six-and-a-half-hour discharge.15

For a six-hour battery reserve to buy six hours of repair time, news of the rectifier trip has to reach an operator who can clear the relay and bring the rectifiers back onto the bus while the cells are still near full charge. At 33 Thomas Street, the fault path from the 20th-floor rectifiers to a human responder had to cross four physical and organizational links on the 20th and 14th floors. As AT&T Vice President of Network Operations William J. Carroll disclosed three days later, all four were open.57

An indicator bulb that would have signaled the fault was burned out.57 (AT&T’s October 2 synopsis added that a technician walking through the 20th-floor power room still would have seen a local POWER alarm inside the room, though no one walked the floor until the evening shift.1) Without a lamp-supervision check or a routine lamp test before a generator transfer, a burned-out bulb stays dark during a fault just as it does under normal operation.

Figure 4: A Bell System central office control and alarm frame (1AESS MCC), showing the indicator lamp rows, alarm keys, and wire-wrap bays typical of AT&T switching hubs. Photo: Sirgorpster (CC BY-SA 4.0).

On the 14th floor, a 25-inch metal strap was missing in the alarm control base, immobilizing the secondary alarm circuit.57 According to the FCC Common Carrier Bureau’s subsequent investigation, that alarm, scheduled for replacement in October, had not been checked since June 1990.17

Because the alarm wire carried current only during a fault, an open 25-inch strap produced the same electrical state at the receiving end as a healthy power plant. With the circuit left unchecked for fifteen months, the unbridged terminal posts reported zero fault current whenever the strap was absent, and the monitoring panel treated that silence as normal operation.

Even with the bulb dead and the 25-inch strap missing, a local audible alarm bell did receive current when the rectifiers tripped.5 As Carroll explained in his September 20 briefing, on the 14th floor “the alarm was clamped so that it was ringing lightly, but not loud enough to be audible.”5 A subsequent House subcommittee investigation of internal AT&T records reported that personnel had muted central-office alarm bells by tightening the audio alarm bell adjustment screw.5

Because the outage followed a string of 1990 and 1991 telephone network crashes caused by signaling-software bugs, later retellings frequently blame software for disabling the central-office warning horn.4 At 33 Thomas Street, no software was involved; personnel had muted the bell mechanically during earlier operations.5 When a local alarm bell rings continuously during maintenance or fault clearing without an automatic re-arming silence timer, tightening a bell screw or clamping a gong is a common floor workaround against constant noise.

With the indicator bulb dead, the 25-inch strap missing in the 14th-floor alarm control base, and the alarm bell muffled, the only remaining way to catch the rectifier trip was for a power technician to walk through the power rooms and inspect the rectifiers in person.54 Standard AT&T procedure required the direct-current power supervisor to dispatch technicians to check each power plant whenever the building switched from Con Edison street power to internal generators. The building had made that cutover four times earlier that summer without incident.154

On the morning of September 17, the DC power supervisor and the building’s dedicated power technicians were off-site at 811 10th Avenue in Midtown Manhattan, roughly 15 minutes away, attending a training class on a new computerized power alarm system scheduled for cutover at 33 Thomas Street in October.15 Fifty-two other technicians inside 33 Thomas Street were qualified to check the power rooms, and the AC power supervisor who threw the 10:10 a.m. transfer switch was in the building, but according to AT&T, the absent DC supervisor neither dispatched a substitute nor handed off the post-transfer walk-through.154

The larger gap was the handoff architecture. In September 1991, 33 Thomas Street had no remote power-plant telemetry wired to a regional network operations center.1 Every high-capacity trunk passing through the building depended on manual floor walk-throughs after a generator transfer, yet scheduling the entire DC power crew into an off-site class on the new alarm system triggered no automated handoff block and no remote watch transfer.154 The building’s 20th-floor power plant was running on battery reserve with nobody assigned to walk the floor and no wire leaving the building to report it.

For more than six hours after the 10:10 a.m. transfer, the 20th-floor batteries held the bus near 48 volts. Lead-acid cells do not lose voltage steadily as they drain; they hold a flat voltage plateau across almost their entire capacity and then drop off a steep cliff in the final minutes.

At 4:30 p.m. EDT, 6 hours and 20 minutes after the diesel cutover, a power-qualified communications technician coming on duty for the evening shift saw a visual low-voltage battery display in an unstaffed area on the 12th floor and attempted to cut the 20th-floor plant back from batteries to AC power.15 That restart attempt failed immediately.15

As AT&T explained in its post-incident synopsis, “the batteries had been discharged to a point where they would not physically accept recharging current without being disconnected from the facilities they were supporting.”1 Reconnecting offline rectifiers onto a deeply discharged 48-volt battery bank (KS-20472 / C&D flooded lead-acid cells) while full central-office load is still attached creates an electrical surge: the depleted cells demand a large charging current at the same moment the DS3 transmission bays draw their full steady-state load. Without automatic restart capability (which AT&T noted the existing 20th-floor plant lacked), technicians could not bring the plant back online while the transmission bays remained connected.1

Ten minutes after the failed cutback attempt, at 4:40 p.m. EDT, the 20th-floor battery reserve expired.15

Figure 6: Controllers at New York TRACON operating Sperry Univac ARTS IIIA (DEDS FA-8320) radar scopes. Source: Sperry Univac / J. ATC (1981).

Between 4:40 and 4:50 p.m. EDT, as the 20th-floor bus fell below operational cutoff, the DS3 and other high-capacity transmission facilities inside 33 Thomas Street shut down.13

The primary and backup FAA landlines for New York TRACON and the three airport towers ran through those bays, carrying remote radar and beacon targets into TRACON’s Sperry Univac ARTS IIIA Sensor Receiver and Processor (SRAP) and Communications Multiplexer Controller (CMC) units alongside voice and flight-plan links.234 The blackout also severed communications linking the New York, Boston, and Washington Air Route Traffic Control Centers, and the FAA at first believed it had lost all of the 90 or more radio and data links into New York Center, though a few were later restored. A backup system came up fairly quickly at the TRACON, but controllers there could do little with New York Center cut off.24 Controllers had to sequence approaching aircraft on short-range backup frequencies and manual tower-to-tower relays, halting departures and arrivals across Kennedy, LaGuardia, and Newark from roughly 5:00 p.m. until 9:00 p.m. EDT.2384 By the time traffic cleared, 1,174 flights had been canceled or delayed, affecting 85,000 passengers, and more than 5 million domestic and international calls had been blocked.274

Inside 33 Thomas Street, recovery required more than closing the 20th-floor rectifiers back onto the bus. Because the depleted battery strings and the connected transmission bays together drew more current than the rectifiers could accept from a cold start, engineers had to physically disconnect all supported equipment from the 20th-floor power plant before they could manually reset the rectifiers at 6:00 p.m.:1

Con Edison load-shed request. Utility asks AT&T’s Building Operations Group to transfer 33 Thomas Street off commercial street AC under a voluntary peak-load sharing agreement.15

Diesel cutover & rectifier trip. AC supervisor switches the building to internal diesels; a voltage spike trips a misadjusted 20th-floor overload relay, dropping the rectifiers offline while 48V batteries assume the full DS3 load.15

380 minutes of silent discharge. A burned-out indicator bulb, a missing 25-inch strap in the 14th-floor alarm control base, a clamped bell, and the absence of remote NOC telemetry leave the discharge undetected while the DC power crew attends an al