Thursday, August 15, 2019

The Destroyer, Hello World

I try to keep an empty testing queue as much as possible. When something hits that sweet column o' mine on JIRA, I usually drop everything to knock it out right then and there so that it doesn't build up to Towering Inferno levels. Typically it only takes a few minutes per story to give it a good component testing once-over, and my team is adept enough at what they do that I very rarely have to kick anything back to them.

On this occasion, it was nearing the end of my self-imposed work day (hip hip huzzah for flexible hours and remote work) when I decided to quickly knock out four bug fixes that had crossed the threshold between Code Review and Testing. Yawn.

Bug fix 1: not in the environment, even after a hard refresh and cache/cookie clear. I left a note and kicked it back to In Progress.

Bug fix 2: ...also not in the environment. Hm. Okay, strange, given my team's typical success rate, but whatever, we've all been a little distracted lately due to shifting from a recently completed Giant Project back to our normal duties.

Bug fix 3: Nope. Okay. Something's amiss. I've worked closely with these devs for over a year now -- starting as solely QA and editorial, and moving into a junior dev position myself -- and there's no way they're going to miss 3/4.

To Slack I went. "Hey, these changes ARE supposed to be in the test environment and ready to go... right?"

"Should be. Check the pipelines."

Sure enough, two PR merges had stalled out due to queue order -- since the pipelines run after the merge itself, this was the day I discovered that JIRA will still show them as successfully merged even if there's a pipeline fail -- and wouldn't you know it, they were the PRs containing the fixes I needed. I clicked Rerun and prepared to resume my ez-mode work day.

That's when the yelling started.

Line after line of errors and warnings started showing up in the logs, which yes, I do keep up and stare intently at the entire time because we all know that a status bar never actually completes unless you're watching it. Hoooooo boy, so much for that quick end to the workday.

Once both pipelines had gloriously failed I went back through the logs to see what had happened. It looked like a bunch of packages related to our cloud functions were out of date. Nothing an npm update can't fix, I figured.

git checkout master
git pull
git checkout -b cloud-function-update

And now the magic could begin. It was supposed to begin. But attempting to update the outdated packages was triggering a cascade of failures in others. TSLint started shrieking about a laundry list of newly-deprecated code. This... was above my current pay grade.

I pulled my mentor-slash-boss Martine into a video conference so I could show her what was going on. The first issue was that I'd accidentally deleted a closing curly brace, reminding me of why I prefer Python to JavaScript in the first place. But that only fixed one of the many errors filling up my terminal.

I'd like to tell you that we figured out the issue immediately afterwards, but it was another two or three hours of ctrl-clicking file paths and looking for squiggles and beverage refills and swearing before we'd made significant process in whittling down the crisis -- no small feat considering that every time we smashed one error, another would take its place. Our eyes were starting to cross when we finally got the answer we were looking for.

"...Wait, what the hell is this hello-world.ts file it's talking about?"

cd functions/src

Sure enough, hello-world.ts was quietly waiting to be discovered in our sidebar hierarchy.

With all of the care usually reserved for opening the Ark of the Covenant, we cracked that sucker open to find a very important comment starting on line 1:

/* This code is generated as part of the Google Firebase cloud function setup and is intended for demo purposes only. It should never be deployed. */


According to GitLens, hello-world.ts and its equally useless cousin, add-message.ts, had been hidden away in our cloud functions since 2:36 a.m. on December 22nd, 2017, the date that it was finally set up and ready to go. It painted a very clear picture of my other boss, Michael, hammering at it all day and all night to get things working, until that glorious moment in the wee hours of the morning when all tests passed and he could finally get some sleep. There was no fanfare to indicate that these demo files had been added. They were two folders deep in the main project files, hardly noticeable unless you knew exactly what you were looking for. So there they sat, untouched, for two years, over three breaking changes to firebase-tools, until the third time was, as they say, the charm, and all Hell broke loose, because even though the demo files were commented out in index.ts, build was causing them to deploy anyway. And since the demo code wasn't being automatically updated with everything else...

Hello World, friend of beginning programmers everywhere, had absolutely wrecked our shit, badly enough to insert :clap: emojis between each word in that statement.

We very quietly deleted both files and re-ran npm run build. Pass. TSLint. Pass. Emulators. Pass. After a nerve-wracking git push, manual testing of functions in our test environment revealed everything in working order.

There was a moment of silence as we watched normal operations resume in our build and deploy processes.

"So... can I like, write this up for my blog?"

"Hell yes you can."