Interview Question Flub
"Tell us about an outage you caused."
I stammered about how I'm really cautious (totally true) and mentioned the time I offlined the wrong datastore for an application. It was during off-hours and lasted less than an hour. Seriously, big freaking deal.
This tepid datastore story is not my "girl, you really fucked up" one. That story I don't usually drag out for interviews. It's equal parts shame and gratitude. And it still has a charge.
However, Brandi Pilot recently asked people to share real failure stories and I love this ask.
Here's mine. No anonymity.
A month or two into a new role, I spotted a "floor model" storage array during a walk-through. The array was actually on the floor, powered up and running production workloads. Instantly it became a wrong I just had to right.
Long before Becky arrived, getting that storage array off the floor was already on someone's to-do list, along with consolidating the virtual and storage systems spread across multiple racks. The work was scheduled for one fall weekend with little room to shift the timeline.
The problems weren't just meme-level snarled cables and network documentation that only a naive newbie would trust. I also didn't know what I didn't know. There was port security, network team coordination, and plenty of details I should have tracked down first.
This lack of due diligence here is at the top of my "you foolish girl, why?!" career moments.
"New rack" weekend arrived and I gracefully shut all of the systems that Friday night. My first uh-oh came when I spotted the once messy network cables snipped and now just stubs. It makes sense the cables would be cut, but I didn't expect it.
I felt this pit in my stomach and thought, I hope the network documentation is good. It was absolutely not.
Cabling was complete and the storage array looked stunning in its freshly racked position. The other systems looked good, too. Order to the chaos, or so I thought.
However, when I began powering up systems, nothing was reachable. Only the network switches were online.
Seriously. Epic fail.
During this debacle, I met the teammate who would become one of my biggest allies. He sat with me that weekend and later whenever things caught fire. And they did.
He had a knack for showing up right when I was about to give out. I hope everyone is lucky to have someone like that in their corner.
He's the grand door prize of teammates to work alongside. Still.
The network team got called in on Sunday morning. We spent the whole day trying to get everything back online. Six people had their weekends ruined in the service of righting my f*ck up.
Everything was back online before Monday morning. Technically it wasn’t an outage. The environment was even in a better state. So, yay?
But it was real professional shame shit. The kind that sticks.
If you've felt the pain of subpar network documentation, you'll become the person who creates really thorough docs. VLANs, MAC addresses, system port schematics and more. You'll champion accurate, verifiable documentation and update anything that doesn't meet that standard.
I later led efforts to move and re-rack many more systems than we tackled that weekend. Downtime was minimal. Those days were routine and ordinary thanks to good network documentation and real cross-team collaboration. Redemption for me, at least in my own self-judgy eyes.
Preparing for interviews is a strange, introspective process. You dig through the past and try to turn work experiences into safe, tidy stories.
Originally published on LinkedIn. I’m gradually moving my storage, infrastructure, and career-in-tech writing here so it has a permanent home.
