San Francisco Office and Datacenter
Continuing our efforts from 2007 towards making the Burning Man information systems more reliable and available, the Systems Administration Team started working on projects that will provide a better environment for our development and maintenance operations. Thanks to some focus over the last couple years towards reliability, and thorough preventive maintenance, there were no major outages on our production servers from 2007 to 2008.
The new datacenter location at 200 Paul with service provider United Layer proved to be very reliable, despite two short outages due to power issues that were resolved quickly by adding more power to our cage.
Burning Man’s deficient firewall systems in the San Francisco and Gerlach offices were replaced with new Juniper units. The legacy of the old firewalls had been a periodic issue for several years, but the new systems perform better and don’t require as much attention.
The main backup system for our servers was completely redesigned and implemented, and dedicated backup servers with large storage capacity were installed in San Francisco and Gerlach. These improvements provided more storage capacity, making it possible to catch up on the growth of our data without having to decrease the volume of our backups, and allow us to have two backup servers to rely on in case of failure or disaster in one area.
The mail system, which handles all Burning Man email and mailing lists, is an ongoing project. The mail server manages more than 400 lists, including the Jack Rabbit Speaks, which is the largest one, having passed the 80,000 subscribers mark in the spring. The mail server configuration has been optimized to be able to handle such massive mailings without disrupting or slowing down regular email.
In order to preserve our production systems and to create a more robust and shareable development environment, it was decided to dedicate a new server to configuration management and software development. The new machine was installed last spring, and implementation and transition to the new environment will happen in the next year.
Nevada and Playa Operations
The Nevada network is comprised of several components, one of which is supporting Internet connectivity for Gerlach and the Nevada Properties. Getting a fast and reliable connection all the way out to Gerlach is no easy task! The backbone and many smaller components make up the whole network, including the Gerlach Office network, the Black Rock Saloon network, the Black Rock Station network, the Town Wireless, and the Black Rock City (event) networks.
Burning man installed a new tower in Gerlach for both year around, and event Internet and radio communications.
After many false starts on the project, a 100-foot tower was installed in Gerlach to provide the main backbone link. A new building was acquired to support the tower and serve as a geek work center for both Internet and radio gear use. After the tower and building were set in place, the main link point from the Gerlach Office was moved there. A new backhaul link from the tower to the office was then created. This link will be upgraded in the future to create a more stable network for the office and saloon. After this link was created, the microwave drum that we use for the event, as well as all the other gear we use for Internet communications, were installed on the new tower. The main advantage of this tower and the new setup is that they are permanent and quite solid. Our Gerlach ISP (internet service provider) upgraded the last bottleneck in the link to the net, and it is now possible to pull a full 9Mb full duplex (tested) from the 10Mb link. After years of unstable Internet connectivity in Black Rock City, having a solid and sturdy structure made a world of difference for our on playa network reliability.
An improvement in 2008 was implementing traffic shaping on the BRC network. The enhancement has been needed for many years, and a number of attempts have been made in the past. Between the solid connection to the Internet, the upgraded bottleneck in Gerlach, and some new Cisco gear, most customers within BRC noticed a far better Internet connection.
The Internet connectivity to Black Rock Station, Burning Man’s storage and staging facility, experienced few issues this year, just a few glitches from the satellite uplink, but nothing could be done on our side to fix that. The main change was the installation of a new node at the common shop. There are plans in the future to run Internet to Black Rock Station from the main link in Gerlach. This has had a few delays but will hopefully be implemented in 2009.
In conclusion, this year had its ups and downs but it can be said yet again, that it was the best year to date. 2008 worked better than 2007! The System Administration Team strives to improve things every year, and work out the problems as they occur, but overall it’s always a blast.
Submitted by
Cat Fougere, Linux Demoness
Chris Petrell (Taz), Nevada Technical Lead