Use Git Hook to Automate the Deployment of Code to Staging or Production Servers

Continuous integration is an integral part of software development today. Git hooks help us accomplish continuous integration, and I can’t imagine how difficult it would have been without git hooks.

If you use git for development then following can help you setup automated deployment of code to your staging and production servers. I used the following to setup staging environment for one of my Django projects, but it can be used with other type of web applications as well.

I wanted to be able to run the app as it will run in production, so I wanted to use Apache with WSGI to host the application. So when I push the code I wanted following to happen in order:

  1. Update the application code hosted on server.
  2. Restart apache.

Staging or Production Server Setup

You need to host a bare repository on your server. If you already have that you can skip this section, below I will list down the steps to replicate how my server is setup and how git bare repository is hosted.

  1. Create a user to host git repository.
    useradd -m git
  2. Set up the password for new user or if you use key based ssh logins set it up for new user and make sure you are able login to remote machine from your development machine.
  3. Login as git on your server and clone your git repository.
    git clone --bare

    If you run the above command in home directory of git user you will have a new directory created named “<RepositoryName>.git”. Git will automatically add ‘.git’ at the end of bare repository to differentiate them from the non-bare repositories.

  4. Goto your development machine and setup new git remote, I am using ‘staging’ in the following command but you can name it what ever you want.
    git remote add staging git@server.com:.git
  5. Do a test push to verify everything is working and you can successfully push to your repository on staging.
    git push staging +master:refs/heads/master

Setup Git Hooks For Automated Deployment

If you followed the steps above then your bare git repo will be at “/home/git/<RepositoryName>.git”. But for simplicity we will use following variable names.

Bare Repository Path on Remote Server: <RepositoryPath>
Application deployment path (document root): <AppPath>

  1. Goto your <RepositoryPath>/hooks
  2. Create a script named: post-receive
    Note: It can be a executable script in any language, but I prefer bash for such trivial tasks.
    Note 2: Don’t forget to change the variables in the script as per your setup.

    #!/bin/bash
    while read oldrev newrev ref
    do
        if [[ $ref =~ .*/master$ ]];
        then
            echo "Master ref received.  Deploying master branch to production..."
            git --work-tree= \
                    --git-dir= checkout -f
            echo "Restarting Aapche"
            sudo /usr/sbin/apache2ctl restart
        else
            echo "Ref $ref successfully received."
            echo " Doing nothing: only the master \
    branch may be deployed on this server."
        fi
    done
  3. Make script executable.
    chmod +x post-receive
  4. Do a git push and you should see output like. Notice the lines starting with “remote:”. These are the outputs from our script.
    Counting objects: 4, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (4/4), done.
    Writing objects: 100% (4/4), 375 bytes | 0 bytes/s, done.
    Total 4 (delta 3), reused 0 (delta 0)
    remote: Master ref received. Deploying master branch to production...
    remote: Project update successful... Restarting Aapche
    To git@.com:.git
       50244ea..e46a9f0 master -&gt; master

Django Specific Post-receive script

For deployment of Django app there are some extra django specific commands which you might want to run before restarting your apache, so I modified the script that I am using to do following as well before actually restarting apache, and the apache is only restarted if all the commands are successfully completed.

  1. export Python virtual environment path.
  2. source virtualenvwrapper.sh
  3. load the virtual environment
  4. make sure pip-tools is installed
  5. run pip sync to make sure all project dependencies are installed and upto date
  6. run db migrations
  7. collect static files
  8. django check to verify everything is good
  9. restart apache

Below is the script as it is.

#!/bin/bash
while read oldrev newrev ref
do
    if [[ $ref =~ .*/master$ ]];
    then
        echo "Master ref received.  Deploying master branch to production..."
        git --work-tree=/var/www/MySite \
		--git-dir=/home/git/MySite.git checkout -f
	cat &lt;

OAuthApi, an OAuth Extension to python-twitter

If you are a python developer and you tried to develop some application for twitter, you must have came acrossed python-twitter. Which provides you a pythonic interface to twitter, by allowing you to access almost all twitter objects as python objects. python-twitter is still in devleopment, and hence still does not have the OAuth compatibility. I am being a oauth pro, deicided to write a derived class from python-twitter, to allow OAuth to be used as the authentication mechnism for python-twitter.

python-twitter is being developed by our beloved google’s employee: DeWitt Clinton

You can get the python-twitter at: python-twitter

You can get the oauth extension to python-twitter at: oauth-python-twitter

In the next following posts I will posting about how I am using it with Django specifically and how I have writter my TwitterAuthBackend for Django. The whole purpose of publishing the code is to get improvements and reviews on the code, so I have allowed non-members to review the above project on code.google.com, you are more then welcome to post your coments here or review the code on the project. Let me know if you don’t understand anything by asking questions in comments.

P.S I have used the same license as was used for python-twitter.

Happy twittering and developing 🙂

Rapid Development with Python, Django, and Google App Engine

Google I/O the developer event that happened in SanFransisco, many were able to join it but there are thousands of people who were not able to join it Google has not forgotten them. The videos of almost all the talks has been posted but the best one is “Rapid Development wiht Python, Django and Google App Engine“.

Its a must watch video so I decided to share it here. I hope you will benefit from it. You can get the presentation slides from here.

If you like the above video then there are few other good videos too related to Google App Engine, so if you ware interested in learning Google App Engine and not only that but there are videos for other Google technologies which include Gears, Gadgets, OpenSocial and gData. Have look at Google I/O session videos posted with slides

Now YOU can sign up for Google App Engine

Wow! Great news for python developer out there, Google has open the signup for their App Engine. Now every one can signup for the App Engine account. If you haven’t got your account yet and want to be on the bleeding edge of web development, then go and signup for App Engine and have some fun with App Engine SDK then you can upload you app.

Google App Engine has also introduced to new API which includes Image manipulation and our all time favourite memcache API. So, its time to put the fun back into web development and lets develop some useful apps based on App Engine.

If you are Django Developer then you can put your Django Apps on Google App Engine too. Happy developing.

Note: If you are not a developer and want to see something developed on App Engine, then leave a comment and may be some of us will do it for fun.

Rietveld: Guido’s dream app released

Guido Van Rossum, who has been working with team working on Google App Engine, has released the open source version of Mondrian. Mondrian is the code review tool heavily used at Google by Google Engineers, Mondrian was developed by Guido as his first project at Google. Guido said:

My first project as a Google engineer was an internal web app for code review. According to Wikipedia, code review is “systematic examination (often as peer review) of computer source code intended to find and fix mistakes overlooked in the initial development phase, improving both the overall quality of software and the developers’ skills.” Not an exciting topic, perhaps, but the internal web app, which I code-named Mondrian after one of my favorite Dutch painters, was an overnight success among Google engineers (who evidently value software quality and skills development :-). I even gave a public presentation about it: you can watch the video on YouTube.

Guido always wanted to release Mondrian as open source, but due to its popularity amongst Google Engineers he couldn’t released it as Mondrian is heavily tied to Google internal development tools. According to Guido:

I’ve always hoped that we could release Mondrian as open source, but so far it hasn’t happened: due to its popularity inside Google, it became more and more tied to proprietary Google infrastructure like Bigtable, and it remained limited to Perforce, the commercial revision control system most used at Google.

But After joining Google App Engine Team Guido got a chance to write the clone of Mondrian which he named Rietveld and released as OpenSource. Rietveld is available at: codereview.appspot.com

According to Guido:

The Rietveld app serves several purposes at once: it is a demo of fairly large-scale use of the popular web framework Django with App Engine.

To learn more about Rietveld read the article on Google Code!

Google App Engine now have a Django helper

Oh yes! Finally. Google has releast a Django helper for their Google Apps Engine. The helper gives you the blank project the same that would be provided by django-admin’s startproject command. From the README file:

The helper is provided in the context of a blank Django project, very
similar to what would be provided by the django-admin.py startproject command.
This project contains minor customisations to manage.py and settings.py that
demonstrate how to integrate the helper with a Django project.

Helper helps you use your Django framework almost natively and build your Django apps that would be runnable on Google App Engine. The Django helper provides following functionality:

  • The ability to use most manage.py commands
  • A BaseModel class that appears the same as the standard Django Model class.
  • The ability to serialize and deserialize model instances to JSON, YAML and XML.
  • Access to Django’s test framework with a test datastore and support for fixtures.

You can read about the announcement on Google App Engine Helper for Django

The steps Google has taken will really help Django Community to get their hands dirty with Google Apps Engine. As now they have BaseModel class that is same as Model class provided by the Django framework. The people who have used Django would agree with me that Django DB Models are very easier to manage and use then any other frameworks (I have used CakePHP, Django and Google Apps Engine). But I am fan of Django DB Models as they provide the Pythonic interface to your Database.

As summer is near and I am committed with my self to get my hands dirty on Ruby this summer and Google App Engine Team is working hard to add other languages support like PHP, Ruby & perl. So hopefully by this summer they will be able to add Ruby until then I would be familiar with Ruby on Rails framework, then I might be in better position to decide which framework gives your the easiest DB API and DB Models.

For all of you Django developers tighten your seat belts for DjangoDash

DjangoDash, let see who is the fastest and coolest. A competition of its kind which is based on Django Frame work is about to start. Every team can have 2 participants and they will need to build a Django app in a very tight timeframe of 48 hours.

This competition is going to prove that “Django is the webframe work for perfectionists with deadline”. So a 48 hours deadline and you need to build a coolest app, with good code quality and everything a perfect django app should have. See DjangoDash judging for more information.

Note: If you don’t know Django and want to participate in DjangoDash and you think you a good/innovative coder, please grab your copy of Django now and start learning and coding in Django. There is still time in competition to start.