Software deployment logging and the unexpected benefits
Codeweavers, as any previous blog post readers will know are an agile software house. As an agile software house we offer our clients a fast turn around and as part of this we have to deploy multiple times a day. While some people believe that this is a bad thing, we see it as a hugely positive thing. We cannot deny that we have been caught out by it once or twice but the advantages to the client of deploying often outweigh the rare occasions that we have issues. On the rare occasions that we do get caught by it, we sit down and come up with a solution to try and prevent the problem ever reoccurring. During the last issue we had with a deploy, we found it difficult to track down the issue as it did not show itself immediately. In the end it turns out that a number of deploys had gone out not long before the issue showed itself and and we found it difficult to track down which of the deployed services contained the problem. The main issue we had while trying to track down the issue was due to not knowing which services had been deployed around the time that the issue occurred.
A few weeks before this issue I had watched a great video regarding Facebook's deploys and their disaster recovery
This video contained a number of good ideas that we could implement at Codeweavers, the main one being a log that Facebook has of every major event which takes place (deploys, data imports etc). After watching this video I wanted to introduce a similar log into Codeweavers, alas I never found the time to do this and this ended up biting us as the deploy log would have benefited us greatly in tracing which services had been deployed around the time the issue showed itself. After we had resolved the issue the deployment log became my number one priority.
The deployment log is stored in a database and we just have a small web page to allow us to filter the deploys in an effective way for us. We felt that it would be useful to be able to sort the deploys by environment, date and application.