Is it every intern’s or every developer’s worst nightmare? Isn’t it? You’ve just pushed a change to production, only to realize this is a seemingly small line of code.
There’s a reviewer.
You take a deep breath and try to be calm because your simple change cannot cause chaos until someone approves it. Until someday they decide to be super chill :’)
This is about 15th November 2024. I was on leave, chilling by the sea, thinking about my imaginary girlfriend ;) and I got a call. The Mario system is broken. Can you look into it too with your colleagues? Now, Mario is a central system in our company that deals with live data from multiple sources and goes through other APIs and is used by thousands of users daily.
The project is complex, with multiple integrations, and the production environment has always been sensitive. Everything here has to work smoothly — no downtime, no bugs, and certainly no interruptions :) And before you point fingers at us, we do run automated tests, review each other’s code, and make sure our commits follow the guidelines. We are a small team of four.
The code:
data = fetch_data_from_db()
data["status"] = "active"This was the change. Yes, just this. A senior developer and an intern, who had been working on a new feature, pushed this minor change to improve data handling. It seemed harmless enough — a single line of code designed to optimize a function.
So, we started looking at the older commits. But this was it. This was the issue :’(
Yeah, On the surface, it didn’t seem like a big deal.
Nothing obvious stood out in the error logs. But as we dug deeper, it became clear that the issue was our single line of code.
So, what was the Root Cause?
Answer: Immutability
Upon closer inspection, we realized that the data object was not a mutable dictionary, as the code had assumed.
So, what was it?
An immutable tuple.
The line of code was attempting to assign a new value to a key in a tuple, which is not possible because tuples are immutable.
In Python, once a tuple is created, its contents cannot be changed. This led to an AttributeError, which we had missed during code reviews because we didn’t check the data type carefully enough.
And then, since this error wasn’t clear enough — I’d also say the service had bad logging. I mean I am still super new to the company to point fingers, but it is what it is.
So, how did we miss something so basic? Both the intern and the senior developer, who had introduced the change, were confident that the optimization would work.
The code had passed unit tests, and the code review didn’t catch the issue because it focused on functionality and logic, rather than on the specifics of data types and immutability.
The problem wasn’t that we didn’t have a review process — we did. But we didn’t review it properly.
We assumed that the data returned from the fetch_data_from_db() function was mutable, when, in reality, it was immutable. This oversight was compounded by the fact that the developer hadn’t documented the behavior of the function properly, leading to confusion.
So, what next?
- Stricter Type Checks: We began adding explicit type checks and type hints in our code to ensure that developers knew exactly what data types they were working with.
- Stress Testing in Production Environments: We started implementing more rigorous testing in environments that simulated production more closely, especially for database operations.
Wanna Read More of My Python Articles?
In case we are meeting for the first time, come over here, it’ll be worth the roller coaster of articles that are gonna come up in the next few weeks. Enjoyed the read? You can support my writing journey here — Buy me a coffee?