In the beginning, like all Software companies, our focus was to get our software running and into the market. In the beginning, we were creating new features every day. Our Product Manager and Customers loved it. A suggestion on Monday could be in Production on Friday. When you start, features come first, you have to get a Product to market, and you know that with success, performance requirements will be coming.
When we started to think about the US market, we knew that our “new normal” for any Enterprise we worked with would be at least ten times what we were used to in the UK. So performance went to the top of the list. At first we struggled – there was an unexpected lesson. We found we needed to re-organise our Engineering team.
What worked well for us developing feature and functions, wasn’t great for creating performant software.
There were 4 things we had to learn (or un-learn) to create performant software.
- We tried to include work effort for performant tuning into Sprints. The thinking was it was simply a matter of allocating some time to do “performance tuning”. This didn’t work.
- We thought that “performance tuning” was simply another type of coding, and the Dev team already had the skills and tools to do it. Simplistically you get a profiler running in Eclipse and you are done. This didn’t work.
- We thought we could add more memory and cores and that will buy us time. This didn’t work.
- We thought there was probably 2-3 things we need to change and that would give us 80% of the performance increase we needed. Increase all pool sizes, we make the database run really quickly with good indexing. We were wrong.
There were 4 things that we needed to change in our approach, to create performant Software.
- Find people in the team with the right mentality and aptitude, and create dedicated roles and give them the tools. We created a team who are tasked with nothing other than discovering, tuning, and testing application performance. We call them the “performance engineering team”, it signals that they are a distinct and specialist function in the engineering group.
- You need to go deep in to the code, and examine deeply your whole stack, not just the code you write. You need to understand how you application is using 3rd party libraries. It is entirely possible the way you are using 3rd party libraries is not the way they are intended to be used. It’s also possible that the reason why this 3rd party library was introduced no longer stands and this library should be removed.
- You need the right tools, grepping through jmap memory dumps is not very productive. We used Dynatrace – there are likely other tools that are good but we found Dynatrace very useful. (And we leave it running in Production so we have real world data, not lab test environment)
- We have developed a performant software philosophy, we say “The fasted query is the query you don’t run”. What it means 100 queries at 1ms each is always going to be slower than 1 query a 1ms. It was a profound insight into performant software for me. Less is more.
So in order to create performant software, we needed to restructure the engineering team. We needed to create a proper and distinct discipline within the Engineering team of “performance engineering”.