Víctor García, DevOps Engineer
Blog post series: part 2
Both infrastructures provide a full “dev” and “test” sub-environments:
Blog post series: part 2
The Development Environment
We maintain a couple of development environments: a shared one and a private self-hosted one, and we encourage developers to move to the private in order to ease and speed up the development workflow, getting rid of constraints and slowness caused by shared resources.Both infrastructures provide a full “dev” and “test” sub-environments:
- The dev has its own fake user data to play with while coding.
- The test environment is basically a full environment with an empty database to be filled in with test fixtures able to run any kind of tests.
- Unit and integration tests run in the same machine the code does, just using PHPUnit.
- Browser tests run in a Windows virtualbox machine we provide with Webdriver configured to run tests in Firefox, Chrome and Internet Explorer.
These two sub-environments are under two different Nginx virtual hosts, so a developer can use both at the same time.
The Shared Environment
The shared environment are virtual machines in a VMWare ESXi environment. Each development team is assigned to a single virtual machine. The management and creation of them is pretty straightforward as they are fully provisioned with Puppet. They provide full dev and testing environments with its own Nginx server, Memcache and Beanstalk instances, a HipHop PHP interpreter, MySQL database, etc. All these resources will be shared among all the virtual machine users. The development database, HBase, Sphinx and Jabber servers are hosted in a separated machine, so, these particular resources are shared by all users.
That’s why we’re currently moving everyone to the private environment. We’ve had problems with shared resources, such as when someone executes a heavy script that raises the CPU usage to 100%, makes the memory swaps, or slows the machine down due very high IO usage, sometimes even affecting users sharing the same physical machine.
The Private Environment: Tuenti-in-a-Box
Tuenti-in-a-box is the official name for this private environment. It’s a VirtualBox machine managed with Vagrant that runs on the developer’s personal laptop. The combination of Vagrant and Puppet is a fantastic thing to set this up since it let’s us provision virtual machines with Puppet very easily, every developer can launch as many private virtual machines as he wants just with a simple command. Thus, Tuenti-in-a-box is an autonomous machine that provides the above-mentioned full dev and test environments, ridding us of problems with shared resources. Every resource is within its virtual machine with one exception, the development database that is still being shared among all users.
Right now, project exists that lets every Tuenti-in-a-box user have its own database with a small and obfuscated subset of production data.
![]()
The Shared Environment
The shared environment are virtual machines in a VMWare ESXi environment. Each development team is assigned to a single virtual machine. The management and creation of them is pretty straightforward as they are fully provisioned with Puppet. They provide full dev and testing environments with its own Nginx server, Memcache and Beanstalk instances, a HipHop PHP interpreter, MySQL database, etc. All these resources will be shared among all the virtual machine users. The development database, HBase, Sphinx and Jabber servers are hosted in a separated machine, so, these particular resources are shared by all users.
That’s why we’re currently moving everyone to the private environment. We’ve had problems with shared resources, such as when someone executes a heavy script that raises the CPU usage to 100%, makes the memory swaps, or slows the machine down due very high IO usage, sometimes even affecting users sharing the same physical machine.
The Private Environment: Tuenti-in-a-Box
Tuenti-in-a-box is the official name for this private environment. It’s a VirtualBox machine managed with Vagrant that runs on the developer’s personal laptop. The combination of Vagrant and Puppet is a fantastic thing to set this up since it let’s us provision virtual machines with Puppet very easily, every developer can launch as many private virtual machines as he wants just with a simple command. Thus, Tuenti-in-a-box is an autonomous machine that provides the above-mentioned full dev and test environments, ridding us of problems with shared resources. Every resource is within its virtual machine with one exception, the development database that is still being shared among all users.
Right now, project exists that lets every Tuenti-in-a-box user have its own database with a small and obfuscated subset of production data.
The Code Workflow Until Integration
Developers work within their own branches. Each branch can be shared among other developers. These branches must be up-to-date, so developers frequently update them with the integration branch, which is always considered to be safe and with no broken tests (we will see how this is achieved in the next post).
Each team organizes its development in the manner they want, but to integrate the code, some steps must be followed:
Each team organizes its development in the manner they want, but to integrate the code, some steps must be followed:
- A code review must be carried out by your teammates and it must pass the review:
- In order to keep track of what a branch contains and for the sake of organization, every piece of code must be tied to a Jira ticket.
- Doing this lets us use Fisheye and Crucible to create code reviews.
- The code is good enough to be deployed to an alpha server.
- The code can’t break any tests, Jenkins will be in charge of executing all of them.
- A QA team member tests the branch and gives it the thumbs up.
- After all of that, a pull request is done and then the branch starts the integration process (more details about this in the next post).
Jenkins Takes Part!
At this stage, testing is one of the most important things. The development branches need to be tested. Giving feedback to developers while they are coding is useful for making them aware of the state of their branches.
Those branches can be tested in three different ways:
At this stage, testing is one of the most important things. The development branches need to be tested. Giving feedback to developers while they are coding is useful for making them aware of the state of their branches.
Those branches can be tested in three different ways:
- Development Jenkins builds that only run a subset of tests: faster and more frequent.
- ~14500 tests in ~10 minutes
- Nightly Jenkins builds, that run all tests: slower, but nightly, so developers can have all tests feedback the next day.
- ~26500 tests in ~60 minutes
- Preintegration Jenkins builds, that run all tests: slower but it's the final testing before merging to the integration branch.
- ~26500 tests in ~60 minutes
The creation of Jenkins builds is automated by scripts that anyone can execute by command line. No manual intervention in the interface is necessary.
Our Jenkins infrastructure has a master server with 22 slave servers. Each slave parallelizes the tests execution in 6 environments, and in order to be faster, a “pipeline mode” takes 6 slaves and make them run together to executes all tests. You might be interested in what the Jenkins architecture is like and how it executes and parallelizes such a large amount of tests in such a short time, but we’ll leave that for another blog post.
Our Jenkins infrastructure has a master server with 22 slave servers. Each slave parallelizes the tests execution in 6 environments, and in order to be faster, a “pipeline mode” takes 6 slaves and make them run together to executes all tests. You might be interested in what the Jenkins architecture is like and how it executes and parallelizes such a large amount of tests in such a short time, but we’ll leave that for another blog post.