Are Big Mistakes That Big Of A Deal? Part 2

Are Big Mistakes That Big Of A Deal? Part 2

July 26, 2021 | Compiler team

Professional development

Compiler • • Are Big Mistakes That Big Of A Deal? Part 2 | Compiler

Are Big Mistakes That Big Of A Deal? Part 2 | Compiler

Subscribe here:

Transcript

Angela, Brent. I know you're both terrific at your jobs. But have you ever really, really messed up?

I've shut down production systems and databases. It would be one VM (virtual machine) and there would be a name that was super close to another one. And I would be rebooting it and it's like, "Oh, sugar foot." That was literally the database for the admissions. Oh God. Oh yeah, I've had my share. Yes.

What happened after though?

That particular one, it was like I knew immediately and I was like, "Oh God, I got to bring this system back up." So yes, there was an outage. It wasn't a widespread outage because if you weren't using it at that particular moment, you were unaware that I actually brought down the database server. Oh God, I've done that. I'm still here. I'm still here and I am not defined by my mistakes. Oh wait, I have to tell this story, please, please, please, please.

Oh yeah, go ahead.

Yeah, please do.

So there was a power outage at this university that I worked at. We were working with getting generators working. And so anyway, our whole data center went out, like boom! The entire data center in this college campus. So we're bringing things back up and we're all trying to do a postmortem, we're standing there. And this guy walks in and he's an electrician, and there's this big red button on the wall.

Oh no.

And he says, "What's this?" And he pushes it and the whole room went in slow motion, and we're like, "No." And the whole thing went dzzsh. The whole data center went down again.

Oh my God.

We call him buttons. And he is still there, so.

He's still there.

We're not defined by our mistakes.

But on the other hand, from what I've heard anyways, knowing people in the tech industry. Big mistakes doing something to a production system, it's almost like a rite of passage in this industry, right? Almost everyone has at least one story of doing exactly the same stuff that you were just talking about, right?

Mm-hmm.

So that led me to really wonder, are big mistakes that big of a deal?

This is Compiler, an original podcast from Red Hat.

We're your hosts.

I'm Brent Simoneaux.

And I'm Angela Andrews.

We're here to break down questions from the tech industry, big, small, and sometimes strange.

Each episode we go out in search of answers from Red Hatters and people they're connected to.

Today's question, are big mistakes that big of a deal?

Producer Johan Philippine is here to find out.

So today I've got three stories to share. Act one, I call it flying under the radar.

Okay. Ira Glass.

Oh God, Johan.

Look, it works. So I spoke with Ian Walker, he's a technical account manager here at Red Hat and he lives in Japan. Now I spoke to him first because he started an email thread a few months ago in response to a large social media outage that affected a lot of people and a lot of different websites. In his thread, he links to an article that describes effing up as part of the job of software development.

He's not lying.

As I was just looking at the news and stuff, I happened to cross the article from one of the writers of the Daily WTF, who mentioned that as software developers, screwing up is our job and that you need to screw up in order to get better. And screwing up allows you to get better at recovering from the screw ups and stuff like that. And so I thought, "Well, this is interesting." And there is a lot of stigma and stuff associated with making mistakes and things like that.

I thought what he was doing was really commendable, which was first of all, sharing the article, but trying to destigmatize the idea of messing up. Because Ian, well, he's got his own story about messing up. Early on in his career, he had an IT job for a big airline and his office was based in Los Angeles. Now this airline had flights across the Pacific Ocean and he was on the IT support team for airports in North America, Central America and South America.

Okay.

And that includes the airport in Kona, Hawaii. Now at the time of the story, the rest of his team had gone home and he was alone in the office.

So I had just learned about network switches and how you can log into them remotely and you enter some commands, and you can look at the configuration for all the different ports and all the different settings for the switch. And I'm not sure if I had been asked to gather this information or if I just decided to do it myself. So I was in our office in Los Angeles and I was accessing a switch that was a couple thousand miles away in Kona, Hawaii. So it was not something where I could just walk over there and plug it back in. But for some reason, I had decided I was going to log into one of the switches at an airport and I was going to check the settings to see what it had been set to. So at that time, I think either I telnetted or SSH'd into the switch, and I knew just enough to be dangerous. I knew that the command SH was supposed to show you the settings.

Now Angela, I take it you know where this is going.

Ooh yeah, I can see where this is going. And I guess I'm laughing about it.

Care to fill us in?

Wait, so there's a physical switch?

Somewhere.

Yeah. So at the airport, they have their own servers. Each airline had their own servers in the airport. And these servers handled things like check-in, and flight assignments and stuff like that. And they would have these physical servers and the network cables would come in and out of them to get their internet connections, right?

So what happened, Johan?

So he typed in SH thinking it would show him the settings because I assume that's what it does in some other contexts. But when he's logged into a particular port like that, it actually shuts down that port physically.

I'm sorry. I'm sorry. Oh my gosh, why do we do things like that?

So he's in Los Angeles, but the switch is in Hawaii?

That's right.

That's a problem.

That's a problem. His connection died, to this switch. But not only that, he killed that switch's connection outwards as well.

It's not like he could just walk down the hall and...

Yeah, exactly. So he basically shut down that server's access to the internet at the airport during business hours. So when he disabled the port, the airline operations department, they were unable to access their back-end airline systems, they weren't able to check-in, they weren't able to check the status of the flights that they were handling.

Wow.

Now luckily for him, he had just recently been to that airport a couple months before on a business trip to help them set up, I assume. And he had actually taken pictures of their setup.

So I knew what cable was plugged in where and how it was all set up. So I called up the operations department and I said, "Hey, it looks like your internet connection just went down." And they were like, "Yeah, everything just suddenly stopped working. It's weird, I can't access anything." So I was like, "Hmm. I think I might know what's going on."

"Yeah, that's really weird."

"How did this happen?" "I have no idea."

"Super weird."

Oh Ian, you definitely knew what was going on.

Yeah.

You really do have to play dumb for a second. You don't want to put yourself out there too fast too far.

"Can you go over to the switch?" And I explained what the switch was and said, "Can you take the cable out of number 14 and plug it into number 15 port just to see what happens?" So they did that and somehow it came up and I was able to connect to it.

He said somehow. It magically came back up. Oh my gosh, I love it. I love this is probably one of the best stories.

Oh, it gets better.

So I quickly logged back in and turned on the port that I had just shut down, and then asked them again to put this cable back to the original port. And they did, and everything came up and was working fine.

Wow.

Mm-hmm.

This is a good story.

It's great, I loved it. I loved hearing this from him.

So how long did this whole thing last?

Well, he was a little hazy on the details, but he estimated that it took about 30 minutes to an hour from start to finish, is what he remembers. I mean, time gets a little funny when you're in panic mode like that.

But I'm sure it felt like hours. Yeah.

For sure.

And another lucky break for him: It was early evening for him in Los Angeles, it was like mid afternoon in Kona, Hawaii at the time. So it all happened while the airline actually wasn't all that busy.

Okay.

Ooh, lady luck.

There weren't that many consequences. Lot of luck for him. So I asked him, what did he learn from his experience?

Well, I learned not to enter commands that you don't really understand.

I think that's pretty good advice.

The most sound advice anyone could ever give you.

Yeah.

I want to say it was an honest mistake. It was one of those mistakes like, "Bro, you know you messed up, right?" But it wasn't. He was curious, and curiosity is an amazing thing to have when you work in technology, just not on production systems.

Mm-hmm. So this is great advice, and it's advice that our next guest could have really used when she had a rough go on her first Linux job.

Oh, you would've just had to start over.

We're at act two. I call this one, ‘what is going on right now’? And I spoke to Joanna Delaporte.

Mistakes happen, that's what this is all about.

So that's her. She's been in the tech industry for about 15 years at this point. And about 10 years ago, she took a job as a Linux systems administrator for her local community college.

Okay.

Now, while she had some Windows administrative experience, she was learning a lot on the job how to handle the Linux system.

I mean, that's how I learned it.

Yeah. It forces you to learn it quickly, right? She had taken one course in college on Linux systems. So she had the basics down, but she had a lot more to learn.

Yeah. So I ran all of the Linux systems for my community college, and that was everything involved in the domain for Linux systems. So domain authentication, file sharing, managing the named DNS server, patching and configuring all of the lab systems for all the students. So if this machine went down, all of the other servers would go down as well.

So, pretty important system for her community college. It was located in a server room which she worked out of as well. It was about eight feet wide by maybe 18 feet long.

Okay.

Not a really big space. And she shared it with a half rack and then a few individual server towers. Now it was loud and it was cold to keep the servers cool.

Yeah.

Mm-hmm.

Yep. I was all alone in the closet.

Me too, girl.

Great.

So one day, Joanna was trying to figure out how to do a particular thing on a system. She doesn't really remember what it was she was trying to figure out, and she thought she could try and find that command by going through the log history because the previous administrator had surely done it before. Angela, if you had to go through a history of previously run commands, how would you go about that?

Besides up arrow? No. I always do a control R and maybe start typing in what I think some of the command could have been and it tries to do an auto complete for you. Like when you're in Google, and you start typing and it tries to fill in the spaces, that's one way to do it. That's two ways to do it, actually. That would be my go-to.

I see. Well, neither of those are what Joanna ended up doing.

Oh God.

Well, that's the funny thing. So I didn't actually know how to look at the commands. I was not familiar with the less command, or the more command, or the cat command. And what I wanted was one of those. Essentially, I wanted to see the commands. What I actually ended up typing was source of the root bash history, which was not a good move. It's definitely not something I should have done.

I heard a big sigh there.

Oh gosh, okay. So the source command is a really powerful, very powerful command and I only use it when I'm trying to do something very particular.

Mm-hmm.

Yeah.

Let me think for a second. When do I use the source command? If I'm installing something from maybe binaries or something like that.

Mm-hmm.

So it's like a shell command that executes something almost like the gospel. So you're going to source whatever this thing is, you're typing after the word source.

Okay.

So you just said that she did type source and then root, or?

Of the root bash history.

Oh sugar. Oh yeah. Well, so she did all of that, did she?

Yeah.

Okay. She did all the things.

She did all the things.

Okay.

So instead of just seeing the commands in a harmless way, I was actually executing every command in the bash history file.

Shut up.

And it fired off pretty rapidly as computers tend to do. It probably ran through at least 20 or 30 before I really understood what it was doing and that it was executing every command.

Girl, control C.

But even at that point, I wasn't sure yet how to stop it. I didn't even know how to use a PS command to find a process at that point, so it was something I had to figure out during this execution. I would say it probably ran somewhere between 50 and 200 commands before I finally managed to kill it. It's hard to say because so many of them happened so quickly that I wouldn't have seen them all necessarily.

I am sweating right now.

Me too. Okay, all right.

I am sweating.

I am so hot and nervous. And I was not the one who did the source command.

This happened 10 years ago, yeah.

Ooh. Yes. So just put yourself in this position where you have no idea. So this person, her predecessor may have been doing all types of things, installs, patching, removing software, changing config files, all these things. And she did a cut and paste and said, "Okay, I'm going to just do all the things that you've just done," not knowing what those things were. You can feel your soul leave your body when you watch those commands just run across the screen. And she didn't know how to stop it, oh poor thing.

Mm-hmm, yeah. So it was doing all those things. It was also SSHing into other machines, right, which as soon as these would see that pop up, she would kill it immediately.

Yeah.

Until eventually, she realized that the whole thing would pause when that new shell would come up.

Mm-hmm, that's right.

Right. Then she realized, "Okay, I'm going to leave it open. I'm not going to touch it because that's going to give me time to think and figure out how to stop this." Once that happened, she finally opened up another terminal to kill that process and the parade of terror was finally over.

The parade of terror.

It's literally a parade. They're marching across your street.

Right? Because it's one thing after the other.

Little marching band.

And you're just like, "Oh no."

Yeah. In the moment of course, daylight is funny when you're terrified and things are going wrong. It was probably somewhere between four and 10 minutes. When I eventually realized I had some slack, basically I got to the point where I was like, "I'm just going to let it get to the next point where it pauses because it has SSH'd into something or opened a file. And at that point, then I started doing the research I needed to figure out how to log in, find the process and kill the process.

So what did Joanna learn from all this?

I think it's going to sound very familiar.

Yeah.

I should have known what this command does, right? I'd heard of this command once, that's why I used it because I'd heard it once. But in a way, I felt like I should have known better, right? I should know not to use a command that I don't know what it does. I don't really know what it does. And I thought it was way more simple and harmless of a command than it really is.

Luckily for her, no really lasting and permanent damage was done to the system.

Yeah.

She looked back. She didn't have to wipe it and rebuild the system because that would've taken a long time, especially since she was still pretty new at this job. But she learned a valuable lesson from that.

I'm starting to pick up on a little theme here.

Do tell.

There's a common thread. What are you realizing?

There's a common thread which seems like a little bit of a golden rule here, which is, don't use commands that you don't understand.

Sometimes they sound like a good idea, I don't know. But you're right, this is literally a cautionary tale to anyone who's listening to this.

Several cautionary tales.

Exactly. If you're listening to this podcast, please make sure you know what command you're about to run before you type it and hit enter.

Mm-hmm.

Know the consequences of what you're about to do.

Mm-hmm.

Which is not to be preachy at all, right? Not to be preachy at all.

Oh gosh, no, wait a minute.

This is very common, right?

Mm-hmm.

It's common. It is common. We're humans.

Yeah.

And sometimes you could know, or at least you think you know, "Oh, I know what this command is going to do," and it does something, one, because it's really not the command that you think it is. And it does something totally unexpected.

On that note, we have one more story with a quick caveat that the person telling the story didn't cause the mistake, but she was part of the team that had to fix the mistake as it happened.

The cleanup crew, okay.

She was part of the cleanup crew. Act three. I call this one ‘syntax error’.

Okay.

It actually happened pretty recently. It was in 2018 at a massive tech company that we've all heard about.

We are not naming names.

Well, we're not naming company names. But I spoke to Ann Marie Fred, and at this point in her career, she had several years of experience as a developer. She was working in an open floor office with about 75 people in the room, that group was in charge of online sales and product information for this, again, massive tech company. And because it's fair to say that it was fairly well frequented, the website. Ann

I know that one of our bigger web engines would get 4 million hits a month.

Well frequented, okay.

Nothing to sneeze at, right?

Nope.

So they were running some AB testing on these pages. They're roughly half a million individual pages when counting all the content, which was also translated in multiple languages. They had a little snippet of JavaScript embedded in each of these pages to run experiments and gather data for analysis, to track conversion rates and things like that. And it worked pretty well until one of the consultants running the experiments, a consultant who was not a developer, made a critical coding mistake. Ann

Yeah. So the experiment itself, the little bit of code that was important, basically said if window.location.HRF = the URL for page A, then set window.location.HF to the URL for page B. Pretty simple.

I'm sorry. I had. To laugh because I have to wait until I hear exactly what happened, but it's literally pointing to another page.

Yeah. So yes.

Okay. Ann

And since this little snippet of code was embedded on all the web pages that our group was generating. Between the product pages, and the search pages and the 100+ languages that we were supporting, we're talking about at least a few hundred thousand webpages, maybe a half million webpages that had this little snippet on them.

So A/B testing. You randomly assign a user, either A or B at that point, that is the A version of a webpage or a B version of the webpage. There's going to be some differences between the two, and the idea is to determine which page out of those two is more effective at getting whatever desired outcome that you're trying to measure, right?

Oh, so you're running a little experiment.

You're running little experiments, right?

Yes.

But as a user, you don't really know.

As a user, you have no idea because you just either see page A or you see page B, you don't see both of them. You don't even know that an experiment's being run most of the time.

Yeah.

So they were running a particular experiment, or they're about to run a particular experiment and something goes terribly wrong.

Oh no. Ann

Well, in JavaScript, the single equal sign is used for assigning values to a variable. And then of course, a double equal sign lets you compare two variables irrespective of the data type. And then the triple equal sign compares two variables, but it checks the type strictly, right? Unfortunately, the person accidentally used the single equal sign. So instead of checking if the window location was logically equal to A, it was just immediately setting the window location to the new page, or it was actually setting the window location to A immediately. And so what happened is, as soon as that experiment went live, every single one of those pages started redirecting to the target page in an infinite loop.

Ooh.

So, half a million pages, give or take a few thousand. Instead of performing a check, instead they redirected to a single page. It's not too bad, right? That's the worst of it, right?

Is it though? Ann

So they launched the experiment and then immediately went into a multi-hour customer meeting and turned their phone off.

No. Ann

Of course, it's like the classic launch something on Friday evening scenario, right? But we noticed in our big, open office room, we had a lot of monitors on those webpages. And so what happened is, all the monitors that were checking for a specific content to render on a page, or for user journeys that could go through successfully started failing at roughly the same time within five to 15 minutes, depending on how sensitive they were. And so immediately, phones started ringing all over the place in our office from different teams that were monitoring their pages. And it very quickly became a... When one pager goes off, people would shake it off. But when 10 pagers go off, everybody in the room stops working and everybody wants to know what's going on.

And all these heads are popping up over their monitors like groundhogs like, "Wait a minute."

Yeah, Like groundhogs and meerkats. They're like, "What's going on here?"

Oh wow, that's a good one.

Wait. Paint this picture for us, Johan. What just happened here?

So a consultant who was running these A/B tests on the webpages. A consultant whose office, by the way, was in another city, not conveniently next door where they could just pop into her office and say like, "Hey, what's going on?" She started running an AB test experiment and it immediately started to redirect all of the pages for whatever that group was said to monitor to a single page, which would overload their system, I assume, is what happened.

Mm-hmm.

Everything stops working properly. All the hundreds of thousands of pages were no longer accessible, right, and they were all trying to get to one single page, which triggered all of these alarms. And all the teams who depended on the data from those pages, they'd noticed that something was wrong and they were calling into Anne Marie's office to be like, "Hey, something's up? Is something happening on your end?" And they didn't know what was going on because this was out of the blue for them, right, they didn't know that the A/B test had just been launched. It took them about half an hour to figure out, first of all, what was happening. Then they figured out that the pages were caught in a loop, but they didn't know why.

Wow. That's so stressful.

When they realized it was from the A/B testing platform, they went to try and shut it down only to find out that they didn't have the right permissions to do so, only the person who launched the experiment was able to do that, the consultant. And because she worked in another office and because her phone was off, they weren't able to turn it off right away. So Anne Marie was tasked with contacting this consultant and getting her to shut it down. Eventually she did so by calling other people who worked in that same office and to be like, "Hey, we really need to talk to this person right now. Can you get her on the phone and out of whatever meeting that she's in because this is a big deal."

"She's in a meeting. May I take a message?"

No.

Drag them out of that office.

But even though Anne Marie wasn't the cause of this problem, she and her team still learned a pretty valuable lesson. Ann

Well, we learned that the amazing power of an A/B testing framework could bring down a website if it's not configured correctly. So we got much more cautious after that. We worked with the vendor to put in an emergency kill switch so that we, as developers, could shut off any test or experiment with a single command.

Again, someone who didn't really know what they were doing caused a big problem, but Ann Marie and her team were able to put in a kill switch and a backup system so that they could intervene. They also implemented a code review so that anytime the A/B testers wanted to push something to production, they had a developer actually go in and check it to make sure that they wouldn't cause any more problems.

That's smart. It had more eyes on it.

It sure did. And after they implemented that code review, they didn't have the same mistake happen again. On that note, Anne Marie has got some advice about learning from those mistakes. Ann

But it's the same goal, right, that you learn from your mistakes and don't get angry about them. So I think that's really important to have a formal way to learn from those mistakes and also to fight for a culture where these things are treated blamelessly. Because you need people to trust the process and their coworkers enough that they will tell you the truth about what they know as opposed to getting into a defensive mode, right? And just to have a sense of humor about it because really, everybody makes mistakes.

Mm-hmm. Ain't that right?

She's right.

So Johan, we've just heard a few stories about people making big mistakes by doing things they don't quite understand. What are we to take away from this?

Well, mistakes happen. Big mistakes happen, especially when people are doing things that they don't fully understand. It is their fault in the end, right? But if you treat it in the right way, instead of pointing blame and try to learn from it, all of these people, they've learned from their mistakes and they're all still working in the tech industry, right? So big mistakes are going to happen, and sure, there are some situations where big mistakes are going to end a career. But from what I've heard from talking to people in the tech industry, that's pretty rare.

Does that line up with your experience, Angela?

It does. Because again, mistakes are all a part of the job. Because you're curious in your job and you're trying to do a better job, you shouldn't be penalized for your curiosity. Yes, you have to figure out what you're doing and what are their impacts, but none of this stuff was done in malice. None of it was done to bring the company down. No, it was just really people doing their job or just being curious, and mistakes are always going to happen. Sometimes you just have to know how to mitigate them, right, as quickly as possible.

And in my conversation with Ian Walker, from the top of the show, he was telling me how he really likes to create an environment where it's okay to make mistakes. He really tries to shield his junior developers from the consequences if there are any. Now, over the years, people have developed systems, they've developed ways in which mistakes can get caught or prevented before they have big consequences. As a preview for our next episode, which is part two of this ‘big mistakes’ episode.

Part two.

Sometimes the systems, they aren't enough. Sometimes they fail.

I only realized that something had gone horribly wrong when I got a call from the database admin an hour later, and he wasn't happy.

Ooh.

That's next time on Compiler.

This was such a great story, listeners, and I hope you had as much fun listening to it as we had talking about it. We want you to share your thoughts with us. Tweet us @Red Hat on Twitter. Use the hashtag #CompilerPodcast. We just want to hear about your F ups too, because we know they're out there. We know you've done them, now you just have to share them with us. We'd love to hear from you. And that does it for the first ‘eff-ups’ episode of Compiler.

Today's episode was produced by Johan Philippine and Caroline Craighead. Victoria Lawton makes sure we know what we're doing.

Our audio engineer is Kristie Chan. Special thanks to Sean Cole. Our theme song was composed by Mary-Ancheta.

A big thank you to our guest. Ian Walker, Joanna Delaporte and Anne Marie Fred.

Our audio team includes Leigh Day, Laura Barnes, Stephanie Wonderlick, Mike Esser, Claire Allison, Nick Burns, Aaron Williamson, Karen King, Boo Boo Howse, Rachel Ertel, Mike Compton, Ocean Matthews and Laura Walters.

If you like today's episode, please follow the show. Rate us, leave us a review and share it with someone you know, it really does help us out.

So glad you listened. Thank you. And we'll see you next time.

All right.

About the show

Compiler

Do you want to stay on top of tech, but find you’re short on time? Compiler presents perspectives, topics, and insights from the industry—free from jargon and judgment. We want to discover where technology is headed beyond the headlines, and create a place for new IT professionals to learn, grow, and thrive. If you are enjoying the show, let us know, and use #CompilerPodcast to share our episodes.

Learn more about Compiler

Are Big Mistakes That Big Of A Deal? Part 2

Are Big Mistakes That Big Of A Deal? Part 2 | Compiler

Subscribe

Transcript

More about professional development

About the show

Compiler

Products & portfolios

Tools

Try, buy, & sell

Communicate

About Red Hat

Select a language

Red Hat legal and privacy links

Red Hat legal and privacy links

Are Big Mistakes That Big Of A Deal? Part 2

Are Big Mistakes That Big Of A Deal? Part 2 | Compiler

Subscribe

Transcript

More about professional development

Red Hat boosts partner engagement with program updates and new Partner Demand Center

Friday Five — July 25, 2025 | Red Hat

Technically Speaking | The role of the OS in the Age of AI

How Should We Handle Failure? | Compiler

Compiler: Re:Role | The Designer And The Blueprint

About the show

Compiler

Products & portfolios

Tools

Try, buy, & sell

Communicate

About Red Hat

Select a language

Red Hat legal and privacy links

Red Hat legal and privacy links