The CodebaseHQ Blog

I thought I'd write a post about how things work here at Codebase. I'm going to start by detailing a bit about what we had before and how it worked, not only from a hardware perspective but from a technology/software one. Before we begin, Codebase is a Ruby on Rails application with a number of smaller Ruby modules thrown in - everything else is pretty much outlined below...

Why was Codebase made?

The inspiration for the application came from our own requirements. When we started moving from Subversion to Git in March 2008, we had a private GitHub account, however due to the large number of projects we have, both past and present, it would not have been economical to import everything into their system (Codebase does not limit the number of repositories per project on paid plans). Not only that, we were also looking for a ticketing system and a way to work more closely with our clients, meaning we’d need to sign up for another application for ticketing and yet another for collaboration.

The testing phase

Before we even started working with an interface, we wanted to make sure we could do everything we needed to, so we mocked up some basic tests using Rails and Grit and you wouldn't believe how happy we were with very basic outputs!

The screenshot above was, effectively, Codebase v0 although it did rather suck.

Version 1 - April 2008 until August 2008

We launched Codebase into beta back in April 2008 and hosted everything on a single virtual machine with an external MySQL server. This was all located on our own in-house Rails hosting cluster which was powered by another of our products, Radar. This worked nicely because we didn't have any large numbers of users and traffic was low.

• We used Tom Preston-Werner's Grit library for all our Git access.
• We used the same in-house task runner we'd written for Radar for processing all background jobs.
• The application ran on a collection of Mongrels with an nginx front-end server.
• The SSH connections were handled by Gitosis and we simply exported a gitosis config file & authorized_keys file whenever was appropriate.
• All our views were written in ERB.
• We used the Prototype/Scriptaculous javascript frameworks

This is what version 1 (and 2) looked like for those of you who can't remember:

Version 2 - September 2008 - January 2009

As we came out of beta, we continued to host on our virtual infrastructure however we needed to make a significant change to the way we served out Git repositories.

• We pulled away from Gitosis. This was never going to be an acceptable long-term solution for hosting any number of repositories or even supporting more than one directory level's worth of repos. Replacing Gitosis was our own connector which would process all incoming SSH git requests and execute the appropriate git-receive|upload-pack command. This removed the need for us to generate a gitosis config file and we could call our database directly - a massive performance boost for the application and users could assign permissions in real time rather than having to wait for the file to be generated by a background task.

This was pretty much it for this version. The reason we made the jump from 1 to 2 was simply because this marked us removing our beta flag and starting to accept any signups. At the time, we had plans to launch another of our web applications, so our entire signup process was handled through our main aTech Media website which managed all user data and simply used API calls to create actual accounts in Codebase.

Version 3 - February 2009 - present

Version 3 saw a complete rewrite of the entire application - from the ground up we re-wrote every view, controller, model and internal library. The only thing which remained the same from v2 was our SSH git connector (which, incidentally, has now gone too). With almost a years experience of running the app we'd found a number of things which we could have done better and the interface left much to be desired. The principle changes between v2 and v3 included:

• A completely new interface
• Multiple repositories per project
• Complete account separation - the early days saw user accounts shared between accounts but this was confusing and we had many enquiries from users who didn't understand this.
• Ability to take payments from clients and start charging for the service
• Signups happened directly in the application rather than being farmed out to the aTech Media website

Hosting wise when we released version 3 we migrated everything onto its own dedicated hardware which included a single application server (which also doubled up as the background task runner) and another box for storage. Never underestimate the power of a single bare-metal server - they are extremely powerful and you can run a lot of traffic through just one and it's often preferable than adding another and dealing with clustering them (which will always add extra network latency to your file system requests).

• Removed our own task runner and replaced it with delayed job
• Moved from an Nginx/Mongrel stack to an Apache/Passenger one
• Removed much of the Grit usage in favour of our own internal library
• Removed all view logic and moved into specific presenters which offer a simple class to a view
• Added a number of "containers" to provide inherited functionality to controllers.
• Added a gravatar cache
• All views are written in HAML
• We use the jQuery & jQuery-UI javascript frameworks

Since this version went live, we've been adding new features all the time including the wiki, time tracking, commit comments and much much more - you can see by looking back in the blog or using the application itself.

Version 3.3 - Summer 2009

Version 3.3 saw a major milestone in Codebase's history - adding support for two extra repository types in the form of Subversion & Mercurial. It saw the final demise of Grit - which, although Grit is awesome, was a great feeling for us.

Since we launched in April 2008, we always thought I'd be cool to support Subversion in our application but ruled this out early on as "too much" and we didn't want to overburden ourselves + our infrastructure would never have supported it. In July 2009, we finally bit the bullet and started planning how we'd add support for Subversion & Mercurial.

Tripod - our repository library

Tripod is our library which supports unified access to any git, subversion or mercurial repository. As an example example:

repo = Tripod.initialize_repository('path/to/my/repo')
repo.scm_type           #=> 'Git'
repo.commits            #=> [<Tripod::Git::Commit: be71f1>, <Tripod::Git::Commit: 3e910e, ...]
repo.commit('f368ef')   #=> <Tripod::Git::Commit: f368ef>

repo = Tripod.initialize_repository('path/to/my/svn_repo')
repo.scm_type           #=> 'Subversion'
repo.commits            #=> [<Tripod::Subversion::Commit: 4>, <Tripod::Subversion::Commit: 3, ...]
repo.commit(2)          #=> <Tripod::Subversion::Commit: 2>

That only gives a tiny taste of what we can run with Tripod and, because the API is exactly the same for any repository type, it makes our lives so much easier when making our presenters & controllers.

Tripod is completely separate to Codebase's application so can easily be re-used in any other system. I'm sure we'll be posting more about Tripod in the future.

SCAM - Source Control Access Manager

SCAM is another of our libraries which works with the Codebase database to establish whether or not users have access to a specific requested repository. We use for authenticating all Git & Mercurial requests over SSH as well as when authenticating Subversion requests over HTTP. Git & Mercurial support was easy because it was much the same as what we were already doing, however Subversion posed a slight issue. For Subversion support, we'd already decided to support HTTP as the official protocol because we wanted a low barrier to entry for SVN users - no public key nonsense etc... We ended up writing an Apache module which accepted the user's normal Codebase username & password and used SCAM to establish whether or not the connection should be accepted - it works very well (if I do say so myself) - kudos to Charlie for getting this working so well.

Next time on "Codebase: the technical story"...

• What we did when we upgraded to our new multi-server cluster
• Details about our new partnership with VPS.NET
• Information about our new background runner: WorkHorse
• Our new SSH key server