There is a video podcast over at Jupiter Broadcasting called TechSNAP that I submitted my best "war stories" to a few years back. Two senior members of the guild I was in were the main hosts and they got a kick out of some of the tales from tech. My personal favourite is this one but be prepared, it's a long one...(insert that's what she said comment here)...
I was running a problem management / critical situations team for a huge multi-national IT company at the time in 2003 I believe. There was a company wide celebration party for an anniversary that evening and I had the on call emergency phone so I became the de facto designated driver. A couple of hours into the festivities and I was at the bar to pick up a round where I ran into the on call manager for that evening and quickly noticed that she was partaking in some alcohol consumption which pissed me off at first but she advised me that a few drinks wouldn't be much of an issue and things had been quiet on the crit sit front all week. I followed her lead and switched over to having a few pints over the course of the rest of the evening and arranged for a replacement driver. The rest of the evening went very well and we all headed home around 3am. I was due into work at 8am so it was going to be one of those quick turnaround "mornings".
Somewhere around 4:20am the emergency phone started to ring. Once I figured out how to answer the damn thing I was informed by a tech support agent that a manager in the US had called our site asking to speak to the IT director (who was the on call manager's manager). The agent needed my approval to contact the on call manager to get her to then contact the director. I gave her the approval and planked my head back down onto the pillow. I entertained the thought that it was all over and then that little IT alarm in my brain starting putting bad thoughts to the forefront of my mind. I decided that it probably wouldn't be a bad idea to go get a coffee and see if any more calls came through...just in case.
With a warm coffee in my hand I sank into the sofa and switched on the news only to be greeted with "Eastern US seaboard suffers massive power outage". At that point only four letter expletives came to mind. Almost that same instant the emergency phone rang again and the agent informed me that the on call manager wasn't answering her phone. I tried to call her myself and had the same problem which probably meant that she was passed out drunk and / or was ignoring the phone. I bit the bullet and bypassed her and contact the IT director myself. During that conversion she informed me that she had already been contacted by management on the US side and she was heading in to the office. She requested that I do likewise. I wasn't drunk by any means but I was not going to drive to work. I tried to get a taxi but no car could get to me for 45 mins which was useless so I had to go back upstairs, wake up my girlfriend and convince her to drive me to work...needless to say that conversation had repercussions later on but at least I had a safe way to get to the office.
Once I made it to the office the magnitude of the problem became apparent as there were 5 US sites (even though 1 was in Canada) that were without power. That meant that within a few hours our site in Dublin would have to "go live" and take calls routed from the unavailable site. The primary problem there was that it would require us to get our staff into our building some 90 mins before they were expected to be at work. That was going to mean a lot of calls to our people in the middle of the night, a lot of taxi rides arranged and various other logistical nightmares. We started prepping for those eventualities while calls with US managers continued to see if any other options were possible. The best possible solution was to get the Canadian site onto emergency power via their diesel generator which would negate the need for our site to do anything. During the call my few remaining active brain cells prompted me to ask a potentially stupid question - "why didn't your generator kick in when the power went off?". I was then left dumbfounded when they informed us that there was no diesel in the diesel generator and nobody noticed until it failed to kick in.
The new priority became finding a way to get diesel to that generator. They arranged for a tanker to come out but it would take an hour to arrive. We agreed to have another call at that point to assess progress and to make the go / no go decision on dragging our Dublin staff in early. The hour passed, we were on the con call but nobody from the Canadian site joined. Another 20 mins passed and still no Canadians. I hunted out a mobile phone number for one of their managers and when he answered I could hear the noise of a truck near him. He explained that they were having some difficulty getting the diesel tanker into the generator compound and as such they hadn't begun filling up the generator yet. Those pesky brain cells piped up again and prompted me to ask - "what's the problem with getting the tanker to the generator exactly?".
Upon hearing the response I wished that I had never asked the question. "Er...eh...well you see....the gate to the compound....well, it's electric!". The emergency plans for these situations are mapped out in detail and reviewed multiple times before being set out as the procedures to be followed and yet somehow the one slight detail that the gate to an emergency diesel generator was electrically powered thus isolating that generator from being accessed during a power outage never dawned on anyone involved.
They eventually just broke down the electric gate and fence and got the generator running in time to avoid our staff having to be dragged in early. The sequence of unlikely events, poor decisions and damn near Benny Hill style antics on the Canadian site that night made me both simultaneously love and hate the whole problem management side of IT support.
I paid the price though, I didn't get to leave work that day until 3pm by which point I was so tired that I slept in my office slumped over my desk until it was time to start work again the next morning.
