The Tuenti Release and Development Process: Once Upon a Time

Víctor García, DevOps Engineer

Blog post series: part 1
The release process and development workflow at Tuenti has evolved from a slow, manual and unreliable process to a fast, fully automated and stable one. It’s now evolving and will keep evolving forever.

Some years ago, there was almost no control over the process, and the existing process was manual.

• Developers coded in branches, those branches were merged to the “release” branch with almost no testing by either automated or manual tests.
• The testing was only done in that “release branch” by CruiseControl (and later by Hudson) and, at the same time, manually by the QA team.
• Although there were dozens of broken tests, the release was deployed after someone manually ran those tests and determined that they were brittle tests.
• Nobody really knew what was going to be pushed.
• Huge release meetings took place the day before that wasted an hour of approximately 20 engineers’ time.
• The code building lasted more than an hour and we were doing a bunch of them every release.
• We didn't trust the process very much. We didn’t feel confident that everything was going to go smoothly, so the release was done very early in the morning to avoid disrupting our users and also because it could take several hours. We even had to wake up 4 hours earlier!
• A regular release started at 8am and ended at 12pm and required the Engineering Team’s attention the entire time.
• There were one or two releases per week at most.
• During the release, there were many code builds and deployments to production because some bugs were found and fixed right there.

• The bug tracking in the error log was a mess, the stacktraces were mixed, and tons probablu important of erros were ignored.

• In the end, www.tuenti.com had a lot of bugs and fixing any one of them was an expensive task that involved a lot of people.

Obviously, this was not the way to go so everything has changed. A lot!!

Everything Has Changed!
Today, everything is automated, and not just the specific release process, but even the development workflow.
Before going into details, some data:

• We perform about 10 releases per week plus small additional production deployments called hotfixes (urgent and small fixes that are deployed quickly and skip the full process).
• All branches are tested by Jenkins with more than 25,000 tests run each time.
• Many tests levels (unit, integration, acceptance, just JavaScript) and types (white/black box).
• Very few brittle tests.
• A full release (since the release candidate is chosen until the code is in production) can be done in half an hour.
• Every integrated changeset is a potential release candidate (“evergreen trunk”).
• Building the code takes about 20 minutes (taking into account that we also compile the HipHop binaries).
• The deployment to hundreds of servers takes 1- 2 minutes.
• Server resources are fully monitored.
• Development environment is reliable and more similar to production.
• Every server provisioned by Puppet.
• We have three deployment phases to reduce risks: alpha, staging, and production.
• Everything is orchestrated through Jira tickets.
• No manual intervention at all.

Process overview
Before going further, and to ease understanding, let’s explain how the branches’ tree is configured in our DCVS (Mercurial).

• There are two permanent branches (live and integration).
• Every development branch will be eventually merged into integration.
• From any of the integration changesets, a new release branch will be created.
• Once the release is finished, the branch is merged into the live branch and then, live back into integration to keep it updated.

Now let’s briefly list what we’ll cover in following blog posts.

2nd post: The Development Environment and Workflow
We’ll talk about:

• DCVS management
• The development and testing environment
• Tuenti-in-a-box: Vagrant+Puppet
• Webdriver
• Jenkins and alpha usage
• Ticket management with Jira
• Code reviews with Fisheye and Crucible

The development workflow must be as much frictionless as possible for the developer, and the integration process as transparent, fast and safe as possible.

And we’ve achieved that!!

3rd post: Development Branch Integration
We will talk about:

• Flow pull request system and dashboards
• Jira orchestration
• QA management

To make an integrated changeset a potential release candidate to be deployed to live, we wanted to have an “evergreen integration branch”.

And we’ve achieved that!!

4th post: Release Time!
We will talk about:

• Release branch selection
• Flow release management
• Alpha environment
• Staging environment
• Deploying to production
• The deployment: TuentiDeployer
• Build script
• HipHop compilation

Our goal is the deployment of any piece of code that is ready, without time constraints or risks. More frequent releases that are smaller is widely known as “continuous delivery”.

And we’ve achieved that!!

5th post: Deploying Configuration
We will talk about:

• Self-service
• ConfigCop
• Safe deployment assurance

The configuration changes are very frequent, and a bad one can create chaos. We didn’t want to waste much time performing and checking these changes, but somehow, we wanted to let everyone deploy their configurations to production on their own.

And we’ve achieved that!!

6h post: Conclusion and the Future
We will talk about:

• The benefits of these changes.
• Transparency for everyone in the company.
• Some tips that demonstrate success.
• And now what? Next steps and projects.

We want the ideal development workflow and environment.

And we’ll achieve that!