Incubation effect

What is incubation effect?

Have you experienced those moments when you take a break in the middle of solving a problem and that lightbulb moment happens during the break? Have you suddenly got new ideas when taking a shower? Well, that phenomenon has a name - it is called the incubation effect. This effect is based on the idea that the unconscious mind continues to work on the problem during breaks, allowing for new insights and solutions to emerge that might not have been accessible during continuous, focussed effort.

Rich Hickey, in his talk on “Hammock driven development” calls this the background mind. He talks about a wider set of things and not just the incubation effect in this video, do watch it fully!

Personally, this has happened to me many times and this article talks about how, as software engineers, we can leverage this effect to our benefit.

Create a conducive environment

In this age where we almost are always consuming some form of content or the other, we aren’t really giving true breaks to our minds. During work hours, whenever you take breaks, try not to check your phone unless necessary. Sip that coffee staring outside the window, let those thoughts wander during those restroom breaks, take those small walks around office or home periodically and give your mind a chance to do its thing.

In addition, I try to consume content when I work out only every other day / not even that. I work out in the evenings post office hours, and I’ve realized that, during the 60-90 minutes I spend at the gym, quite a lot of times my background mind suddenly comes up with better ideas for the work I did the very same day!

Take your time

It is very important you give yourself that time and mind-space to solve a problem. Do not rush into getting something that works. (I’m referring to production applications here, not proof of concepts). All of us are so used to working on time bound stories, estimating them based on story points, T-shirt sizes, whatever, that sometimes, finishing a story is our primary goal than solving the problem in hand. There is a difference. You can finish implementing a story in 10 different ways, but only a few of them will help you solve the next story, when you get your abstractions right and get that naturally fitting elegant solution. This process may sometimes take weeks though for you to realize what you could have done differently, but that is the beauty of building software. You can’t force your background mind to solve it immediately, it can take a while.

Use AI wisely

In this AI-driven times, the first cut solution provided by AI works most of the time. I personally feel that AI is going to increase the gap between the good and not so good software engineers. Do not accept the first solution provided! Have a longer conversation with ChatGPT or whichever you favorite model is, understand all the other ways you can go about the same problem, weigh the pros and cons suggested by AI using your experience, fully understand why you are doing what you are doing and how the decision you take today may impact what’s coming tomorrow, and then come up with a solution.

I’ve had instances where I realized I was asking the wrong question to the AI and once I presented the problem correctly, I got a beautiful solution. But it is on us to think and ask the correct questions. (I have a lot of hope for good software engineers in this AI driven age)

Parting thoughts