From big bang releases to a scale up

Read time: 7 mins

Usually, there were releases 2-3 times a year, and these were big events with lots of planning. Where there should have been excitement about releasing new features to customers, there was worry and concern about the pain involved in deploying the changes to production. We called this ‘release hell’ and now, being at Mention Me, it’s a breath of fresh air to work at a company with a more streamlined approach!

These were fairly large corporate companies that I previously worked for. That in itself isn’t an indicator of a lengthy release cadence or painful release process, but it was a painful experience trying to improve things. I worked with talented people who were equally frustrated, and even when we tried a bottom-up approach, we hit barriers that prevented a wider mindset shift.

I’ve enjoyed reading books like The DevOps Handbook, Accelerate, and the now classic Continuous Delivery. But I’ve been left frustrated when I’ve hit barriers pushing for this more continuous approach. An approach that I know will improve the developer experience and increase the frequency of releasing changes to the customers for us to learn from.

Breath of fresh air at Mention Me

Releasing changes and adding value for our customers is important! It’s great knowing that Mention Me respects this importance too. It’s a fast-paced environment and we’re constantly trying to lead innovations within the marketing space, which means we need the technology and processes to match this ambition. 

Of course, there are always improvements to make, but I enjoy being in an environment of like-minded people where we have the flexibility to set a high bar with our release process.

Our goal in Engineering is to reduce the number of code changes waiting for release — the more frequent you can release, the less buildup there will be. Making releases standard practice means they don’t turn into ‘big events’ that drain people's time and effort.

The DevOps Research and Assessment (DORA) team identified four key metrics that indicate the performance of a software development team: 

• Deployment Frequency - How often you can successfully release to production
• Lead Time for Changes - The time it takes for a commit to be deployed into production
• Change Failure Rate - How often changes lead to failure in production after deployment
• Time to Restore Service - The time from failure due to a deployment to full recovery in production

Using these metrics, we can measure our performance to see where we fall on the scale from low to elite performers.

Continuous Delivery

As an Engineering team, we enjoy releasing changes into production. I really do get a buzz when I see my changes have been deployed successfully!

We’ve been quite good at this, and our release processes enabled us to release into production when we like, which is typically a few times a week. Usually, when one of the engineers decides they want to release their change, they reach out to the team to give a heads up for a release. 

As long as there are no objections, they would manually ‘push the button’ to start the deployment into production. This would push their changes, as well as anyone else's that have been merged since the last release.

DORA metrics

You might think that releasing to production a few times a week isn’t bad, right?! It may not be, but as I mentioned earlier about the high bar, we wanted to do better.

The first two of DORA’s metrics, Deployment Frequency and Lead Time for Changes, mainly measure velocity. Using the calculations from the article, we would be a ‘High’ performer for our velocity.

We strive for continuous improvement, and as “Elite teams are twice as likely to meet or exceed their organizational performance goals”, we set our sights on becoming ‘Elite’ performers!

Move to Continuous Deployment

About a month ago, the final changes were made in our process to allow Continuous Deployment - exciting times! Now, any code change that’s merged into our main branch will automatically trigger a release through our pipeline. If all the tests pass, the change will go straight into production.

We’re now pushing 5 -10 deployments into production every day! And with our ‘lead time for changes’ currently at ~15 minutes, we are within the ‘Elite’ performer measurement for our release velocity. Happy days 🎉

Seeing a graph like this almost makes me want to cry with happiness!

Of course, not every change is a fully complete feature. We like to release small changes often. Most of these are behind feature flags that allow us to hide the functionality from users. Once a feature is ready, we open this up for internal users or specific customers for further testing and early feedback. 

Releasing small changes regularly also helps if any issues are found in our live sites. We’re able to easily identify each specific change that has been released to quickly narrow down the search for the cause.

When we do hit an issue, we usually roll forward, either by reverting a specific change out or releasing a fix if we’re able to turn this around quickly. We do this instead of rollback to make sure we're not going to corrupt data or leave our platform in an untested state.

Having Continuous Deployment in place has not only taken the hassle out of doing releases, but increased energy and excitement within the team. The satisfaction levels of the engineers has increased as they can see progression of their changes straight away, allowing them to focus on the next thing without having to remember which changes are or aren’t in production.

First day mission

From reading the books mentioned above, I had many ideas on how a frequent deployment process could help improve our onboarding and make it more exciting for the new hires. Recently at Mention Me, my dreams have become a reality!

Now, each new hire in the Engineering team has a mission: on their first day they’re tasked with pushing a new change into production. We have a ‘Teams’ page that lists each person who works on the platform, so the mission for the new hire is to add themselves to that (usually we throw in an additional small enhancement to the page too).

This helps with a few things…

1. It ensures we have a process where new hires can get their local setup in a reasonable amount of time with minimal input from others
2. New hires can get their hands on some code quickly and start finding their way around to identify what changes need to be made
3. They learn about our deployment process with creating PRs, getting them approved and merged, and seeing the release pipeline run through
4. Someone new to the team is able to push something to production on their first day - how exciting is that? Their photo is also now on the page, helping them feel part of the team.

What’s next?

Being part of a scale up where there is an innovative mindset means anything is possible! 

There are certainly areas we want to improve in, such as having a more robust E2E automation test suite in our pipelines. We’re also discussing options to see how viable breaking up the monolith is, and what value that would give us.

As the Engineering team scales up and the customer traffic grows, there will be plenty to keep us busy as we aim to keep our processes as efficient as possible.

Does this sound like a place where you would like to work? Have a look at our careers page to see our open roles.