Demo Data Nightmare, the beginning

- So what do you think about me taking this task and implementing it before anything else on that topic gets to the team's backlog?

- Sounds cool! 

- This means that I can help everyone, once I'm done, on similar tasks. 

- Yeah, that would be great. I think we should do that. 

This is the quick summary of the discussion that started it all, 7 months ago, with my manager, without anything, even slightly pointing to the months of endless work, rework, overwork and ultimately unhealthy stress that were going to hit me.

The task was pretty simple. Based on some existing custom framework, implement a new widget that is going to show some data as a graph. In theory, I shouldn't have done anything even remotely close to rocket science, as my job was to understand how the existing framework works and how it can be reused for different scenarios.

I started investigating, and while the existing framework wasn't as good as I would have expected it to be, especially for something that is to be used so many times and in so many places, it wasn't horrible. I had to copy some generic JSON that is going to define the overall widget configuration, change some parameters, tinker its configuration types to accept my scenario, put it in a specific file, fiddle with the new UI to-be-displayed parameters and voilà, task done. Best case scenario, a week of work, worst case scenario 2 to 3 weeks, depending on the feedback I should be getting from the Functional Architects (FAs), my load on helping the team with all they need and anything that might come along.

After several hours of wandering through project code, short calls with people that already used the framework and code owners, I found myself face to face against the normal situation in which I actually needed the data my widget will be based upon. The bummer about it was the fact that the data was provided by a fairly new and still in development feature and you had to spend many hours or even days in order to generate it, as it depended on some entity state from within the system that you had to actually change, from time to time. The more time you spend on it, the more data you get. There was no way to speed this up. Everyone was waiting to get the data, and so seemed should I. Many were complaining about this, with no effect.  

The whole thing was aggravated by the fact that we have 2 splits per week and usually they are not compatible. If you git pull main branch and then you try to rebase it into your feature branch, you need to recreate the whole database, and obviously everything you generated will be gone.

The prospect of wasting half of my day on generating data, two times a week, didn't sound very pleasing, and as such, the lazy programmer within me said:

- I cannot work like that. I need DEMO DATA!

So, let's snapshot the situation: An experienced engineer gets a task, with a worst case estimation of 3 weeks of work and probably 24 hours of wasting time on generating demo data. In Enterprise terms, 24 hours is basically nothing. But still, laziness in software engineering is not, usually, about time, but about doing the same repetitive task again and again knowing that it can be automated.

What can be automated, eventually will be automated. The only question is: "Who will do it and when?"

In this specific case it was me who thought and really considered that it is better to automate the data generation now, rather than "waste" those 24 hours, especially because I heard several colleagues complain about the same problem as myself.

On occasions like that you imagine yourself, at the end of the task, as Tony Stark presenting some new tech in front of the cheering crowd, bringing something to the table no one was able to, something that will improve everyone's professional day to day life.

(photo Copyright Marvel, Iron Man 2)

And just like that, it all began.