Not A Member? Who cares? (Click)
A couple of years ago, I used to work at a 15 member start-up. A very very early one.
I had hired an intern myself, probably the one who didn’t understand much.
But this isn’t about me. This is more about the smartest intern among those three :)
Mind you, we were a noob company. One dev environment and one prod. And most of it was directly shipped to production without testing much unless it was something critical.
A product was being developed, and to pay the bills, we were serving 4 clients by making ETL pipelines for them.
It must be around lunchtime, when two of them came running to the kid I had hired.
“We have been assigned to do this together”
Yk, they were all bright-eyed interns with six weeks of coding experience.
And they were all about to learn why senior engineers get twitchy about environment variables.
Alright, let me take you back to the scene of the crime.
What was the task?
A simple one. Writing a Python script to migrate some user data from a legacy PostgreSQL database to a new Redis cluster.
“Just connect, fetch rows, transform, and insert,” the senior or the lead had said.
“Use environment variables for credentials,” we had also emphasized.
But, they were in a hurry.
Why fuss with .env files when they could hardcode the database URL?
I’ll circle back to fix it later, the smart one told them, patting their egos on the back.
The script ran flawlessly on someone’s local machine.
They shipped the code, tagged it for production, and celebrated with a smug.
For few hours, nothing happened.
But as soon as It was everyone’s time to leave the office,
*boooooooom*
The script didn’t fail gracefully — it failed *loudly*, hammering the dead endpoint with retries until our monitoring tools lit up like a Christmas tree.
The “smart” intern almost-almost froze.
I just stared at my terminal.
The manager quietly closed their laptop and tried to run away.
Immediately, the process was killed, and the deployment was removed. And they were all called to the meeting room.
The lead and I sat with them to debug the issue. It wasn’t a big deal, I said. Our lead also just laughed and asked who wrote the code?
The code was written more or less by the smart guy, so they all just pointed at him.
It turned out, the script hadn’t just failed — it had gone mad.
The “smart” one’s genius retry logic (a while True loop with no exit condition) kept resurrecting the script. It was querying a database again and again, creating a denial-of-service spiral that flooded our alerts channel and took down half our analytics dashboard.
Basically,
the problem was:
- The script worked locally because the
.envhad the old DB URL (they had “forgotten” to update it). - Production had no idea about their hardcoded URL.
- Every retry spawned new threads. Our error logs looked like a blockchain gone wrong.
- And those alerts….grrrrrrrr..
Anyway, everything was fixed within 30 minutes or so. And I was asked to do the task the next day.
However, when I came back the next morning, only two out of the three interns were there. And they had sad faces.
The CTO had made a couple of memes on them and posted it in the public channel.
(“Database vs. Interns: 1–0”)
A flowchart titled “Should You Hardcode?” (Spoiler: The only arrow pointed to “NO”)
Honestly, It wasn’t the memes that made them sad or upset, It was the smart intern being fired.
In the evening, we all joined a meeting, and we were told how we can’t waste compute resources and everyone was here to take things seriously
New company policies were introduced.
- All scripts tested in a prod-mirrored Docker container or dev
- Mandatory pre-commit hooks to block secrets/URLs in code
- Interns banned from pushing anything without supervision
I left this company within the next 21 days, but that is a story for some other day.
In case we are meeting for the first time, come over here, it’ll be worth the roller coaster of articles that are gonna come up in the next few weeks.
Use this to get to know more about me — https://linktr.ee/shashwat_writes
Liked the story? Coffee☕ And Code💚
Discover the stories that will make your 🤍 beat!
This article was published on February 22th, 2024 in Coffee☕ And Code💚publication.
