The Systems Administration team had a pretty smooth year, while implementing exciting changes. Some of the most noticeable improvements are a new mail server in San Francisco, the relocation of the cage in our collocation facility, the doubling of speed of the Internet link in Gerlach for the event, the implementation of a microwave cloud across Black Rock City, and more…
San Francisco Servers and Datacenter
Many changes to our existing infrastructure were implemented this year to increase performance and reliability of our systems.
At the datacenter, intelligent power strips were added in the cage, allowing systems administrators to remotely power cycle any server, and to remotely connect to the console. This allows all administrative tasks to be performed from anywhere off-site.
Several power outages happened in the datacenter in early 2010, which were due to issues with a failing battery unit on the provider side. To avoid more outages while the recurring power issues were being resolved, it was decided to move the Burning Man cage to a different floor of the building, where power is much more reliable. Even though the move hadn’t been planned far in advance, it went very smoothly, incurred only a couple hours downtime, and the operation was a success with no power issues experienced since then.
The mail server was upgraded to a new, more powerful machine, which can now handle our increasing email and list traffic more smoothly. This mission critical server processes all of Burning Man email and around 500 Mailman lists; the biggest list is the Jack Rabbit Speaks, which reached 116,000 subscribers. There was a serious performance increase of the local mail clients and list interfaces.
More systems were upgraded to the latest version of Debian Linux (aka Lenny) and the rest of the upgrades will be completed in 2011.
A new multimedia server was installed, and a new Media Gallery was successfully launched during the event.
More groups are using the development server that was rolled out in 2009 to maintain department-related websites or to work on experimental projects.
Migration of projects from older servers to new ones has been started, and will be completed in 2011, along with more exciting improvements.
Nevada and Playa operations
Many changes were made for the 2010 Burning Man event. The first was a 250% increase in the speed of the main link to the Internet in Gerlach. In addition to faster connectivity, the distribution method within BRC was changed. In the past, point-to-point microwave was used only for key links, and WiFi for secondary links. This year, a microwave cloud was created over the entire city, and used for all the infrastructure and public customers. With the extra speed, we didn’t need to implement traffic shaping as we did last year, though we could have turned it on if necessary. We used more of the cost-efficient new gear that we started using in 2009. We purchased new gear to use as a backup between Gerlach and BRC, but ended up using it as the primary link. We never got it transmitting to its full potential and our previous main link was not operational. We’ve concluded that we need to test the long-range link in advance before event setup. Overall, this year’s wireless network at BRC was the best to date. We will most likely keep this setup for many years, saving money in the long run.
We changed strategies for our local area network that serves the infrastructure of BRC. Instead of individual point-to-point wifi connections, we distributed hard line connections from centralized microwave connections to various customers in Center Camp. Last year we experimented with fiber and shielded Cat5e. This year we skipped the labor-intensive fiber, and went with only shielded Cat5e laid in the same trenches as the power grid, and it worked great. We will be recycling the Cat5e we pulled post-event to use on other various year-round projects. This was a win-win situation for our Center Camp customers!
Size does matter! One great change this year was increasing the height of the DPW depot tower from 30’ to 60’. This made the pre- and post-event connection to the Internet much better than ever before. For connections in 2011, we are considering some microwave link changes that will integrate the Depot tower with the NOC tower for a more reliable and stable long distance link.
Meanwhile off the playa, a link to the Black Rock Saloon that was installed in 2009 became unstable a couple of weeks before the event, so it was rapidly changed over, along with the Gerlach office, to a new microwave system. These changes greatly improved the network speeds and reliability. In the office the phone system still needs an upgrade, but the old system held on this year. It is likely we will be integrating the Gerlach phone system with the VOIP system in our San Francisco Headquarters in 2011.
The Weather station has been challenged by all the harsh Nevada weather, and will need to be updated again in the future. The Black Rock Station and Gerlach networks remain the same, with some changes planned for the coming year.
| And here is a cat. Posted by order of our Burning Man Sysadminatrix |  |
Submitted by,
Cat Fougere & Chris Petrell