Setting up a Jenkins CI Server (CI on a Budget, part 1)
This post is part of my "Continuous Integration on a Budget" series, and covers setting up a Jenkins CI server on a Linode Debian 7 instance.
In this post, I'll walk you through the steps of setting up a Jenkins CI server using a Debian 7 virtual server, set up at Linode.
Why Linode? Because that's my provider of choice for virtual servers. I've used them for almost 9 years, and they have provided, and continue to provide, exemplary service. If you haven't used them and are looking, I highly recommend that you give them serious consideration. Another good option to consider is Digital Ocean.
I also use Bitbucket for my online code repositories. You can set up an account with BitBucket and host unlimited private Git repositories with them at no cost, provided you have 5 people or less accessing those repositories. Since I'm a freelancer / small developer, this is well within my budget, and it gives me the ability to utilize some POST webhooks available at BitBucket to automate some of my testing, which I'll cover later. If you prefer to use GitHub, you can -- POST webhooks are available on GitHub as well, although GitHub charges for hosting for private repositories.
Jenkins CI / Testing Server for Drupal -- Minimum System Recommendations
- 2G RAM
- 30G HD space
- 2 CPU cores
- Linux based LAMP server running PHP 5.3+, Apache 2.2+, MySQL 5.1+
- Apache mod_rewrite
- php5-gd (required for Drupal)
- php5-curl (needed for PHPUnit and SimpleTest)
If you are using Linode or Digital Ocean, an instance that meets the minimum hardware requirements shown above can be set up for $20 a month (current pricing as of September 2014).
Linode has a very good "Getting Started" series on their documentation site, that will walk you step by step through the process of setting up and securing a LAMP stack server that will suffice for this purpose, so I won't cover all those steps here. One note: when you set up your firewall, make sure, in addition to opening ports 80 and 443, that you also open port 8080 for http connections, as this will be the port that the Jenkins server will be utilizing, and that Bitbucket/Github will use to POST commit notifications. To do this, add the following line to your firewall rules:
-A INPUT -p tcp --dport 8080 -j ACCEPT
Now we're ready to install Jenkins.
Installing Jenkins CI
Head on over to https://www.jenkinsci.org. They have great documentation on setting up Jenkins to run in a variety of environments, on a variety of operating systems. We'll be covering installation on a Debian server (which for the most part, applies to Ubuntu as well).
Jenkins is available as a Debian package, which will install all dependencies needed to run Jenkins on Debian. To use the Debian repository for Jenkins for installation purposes, first execute the following command on your server (as root, or "sudo"):
sudo wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
This adds the repository key to your system. Then add the repository to your available list by adding the following entry to /etc/apt/sources.list:
deb http://pkg.jenkins-ci.org/debian binary/
Update your package index:
sudo apt-get update
And install Jenkins:
sudo apt-get install jenkins ttf-dejavu xvfb
The last two entries install additional packages needed to display the graphs inside Jenkins on Debian/Ubuntu that our reporting will be using.
Jenkins is now installed on your server, running as a service on port 8080. Assuming the domain for your server is http://www.example.com, you can now find your Jenkins instance at http://www.example.com:8080. The home directory for the jenkins user is /var/lib/jenkins (important, you'll need this later).
The service is running under the "jenkins" user, belonging to the "jenkins" group. You can stop and restart the Jenkins server using the following commands:
sudo service jenkins stop #stops a running Jenkins service) sudo service jenkins start #(starts a non-running Jenkins service) sudo service jenkins restart #(restarts a Jenkins service)
Since I'm running Jenkins on a virtual server at Linode, I decided to acquire a domain name specifically for my Jenkins installation for a couple of different reasons. Primarily, I've got this domain configured to point to my Linode instance using a wildcard configuration, so that all subdomains of this domain automatically resolve to the same IP address unless I specifically direct them elsewhere. This plays nicely into my continuous integration builds for automated testing, as I just assign a subdomain for each project and Jenkins uses that subdomain to build a Drupal instance for testing purposes using the subdomain for that project.
Securing your Jenkins Installation
When Jenkins is first installed, it is wide open for all users to touch pretty much anything on the system, so first thing you need to do is secure your installation. Pull up the Jenkins homepage (http://www.example.com:8080) and click on the link to "Manage Jenkins", then click "Configure Global Security".
Under "Security Realm", select "Jenkins’ own user database" and check "Allow users to signup" (we'll turn this off in a minute). Under "Authorization", select "Logged-in users can do anything". Click "Save".
You should now see a "Log In" and a "Sign Up" link in the top menubar of your site. Go through the signup process, selecting a username, password, etc. for the user that will be your "administrator" for the Jenkins server. Once you have established your account, log back in as that user, and go to "Manage Jenkins > Configure Global Security".
From there, under "Security Realm" uncheck "Allow users to signup", and under "Authorization" change your selection to "Matrix-based security".
Add your username, which you established when you registered your initial account above, and give that user access to all aspects of the Jenkins system. Grant the "anonymous" user Overall "read" access, and Job "read" access. Save your changes and log out.
When you visit the site, you should see a listing of Jenkins jobs and the status of those jobs (which is currently empty because this is a new install).
Installing Jenkins Plugins
There are some plugins (for Drupal developers, think "modules") that you'll want to install to make Jenkins work effectively with some of the tools you will be using to implement your CI workflow. We won't use all of these plugins to start with, but you will use all of them eventually:
Log into your Jenkins server, and go to "Manage Jenkins > Manage Plugins" and click the "Available" tab. Select the following plugins for installation (if you don't see them listed, check the "Installed" tab -- they may already be installed):
- Git Plugin
- Git Client Plugin
- Checkstyle Plugin
- Duplicate Code Scanner Plugin
- JUnit Plugin
- Phing Plugin
- Plot Plugin
- PMD Plugin
- Static Analysis Collector Plugin
- Static Analysis Utilities Plugin
and click "Download now and install after restart".
Installing toolsets on your server
Now to install the toolsets you need on your server that Jenkins will use to run tests and analysis on your codebase. You will need to shell into your server at the command line.
We will be using PEAR to install these tools. Most of these tools are also available to be installed using Composer; however, there is a bug in Phing that prevents it from finding packages installed using Composer, so for now, you'll have less trouble if you just install them using PEAR.
To install, you first need to add the following PEAR package repositories to your system, using "sudo pear channel-discover
sudo pear channel-discover pear.phpunit.de sudo pear channel-discover pear.symfony.com sudo pear channel-discover pear.netpirates.net sudo pear channel-discover pear.phing.info sudo pear channel-discover pear.pdepend.org sudo pear channel-discover pear.phpmd.org sudo pear channel-discover components.ez.no sudo pear channel-discover pear.symfony-project.com
Once the above channels have been discovered by PEAR, you can install your toolsets as follows:
sudo pear install phing/phing-2.4.12 sudo pear install -a phpmd/PHP_PMD sudo pear install phpunit/phpcpd sudo pear install phpunit/phploc sudo pear install PHPDocumentor sudo pear install PHP_CodeSniffer sudo pear install HTTP_Request2 sudo pear install -a phpunit/PHP_CodeBrowser
After installing the above packages, you'll want to install Drush, and make sure your drush installation is on the environment path for all users.
At the command line, enter the following:
cd /usr/bin sudo wget https://github.com/drush-ops/drush/archive/master.zip sudo unzip master.zip
This should install the most current version of Drush in /usr/bin/drush-master
After you install Drush, you need to run an update to get all dependencies for Drush. You can do this with
cd /usr/bin/drush-master sudo composer update
After you install Drush, you will need to edit your system-wide profile to place Drush in the environment path. You can (as root) edit the file "/etc/profile" and add ":/usr/local/drush-master" to the PATH settings at the beginning of this file. Once finished, execute the following command:
sudo source /etc/profile
to reload the new PATH settings.
Some Miscellaneous Preparations before installing "the" build template
A couple of changes you will want to make to just make life easier and allow you to fully leverage the automated build process you are about to implement are:
Change the user running apache from "www-data" to "jenkins"
- Stop apache with "sudo service apache stop"
- Edit the Apache environment variables config file at /etc/apache2/envvars, and change "APACHE_RUN_USER" and "APACHE_RUN_GROUP" to "jenkins"
- Change the ownership of /var/lock/apache2 (if it exists) to "jenkins" with sudo chown jenkins:jenkins /var/lock/apache2"
- Change the ownership of any website script files running on the server from "www-data" to "jenkins" (by default, located at /var/www)
- Restart Apache with "sudo service apache2 restart"
Generate a public-private keypair for the "jenkins" user and put that keypair on Bitbucket.org
- Shell into your server over ssh
- Change to the root user with "su", and supply the root password
- Change to the jenkins user with "su jenkins"
- Generate a public/private keypair using "ssh-keygen"
- copy the public key for the jenkins user (located at /var/lib/jenkins/.ssh/id_rsa.pub), and install it on your Bitbucket.org (or GitHub) account
- Configure Jenkins to use the private key for authentication
To configure Jenkins to use the private key, keep your command line open in a terminal window, and log into Jenkins via the browser.
- Go to "Manage Jenkins > Manage Credentials" and add "SSH Username with Private Key".
- Set the scope to "Global"
- Input your username for the account you have on Bitbucket (or GitHub)
- Give the configuration a description to help identify what the credential is for (such as "Jenkins Key for BitBucket")
- Select "Enter directly" and paste in the contents of /var/lib/jenkins/.ssh/id_rsa into the textarea provided.
- Save these settings.
Whew! That's alot. However, this is a one-time process that you will undertake that will embark you on a journey to improving your developer skills and workflow. It will be well worth the journey, I promise.