Wednesday, August 19, 2009

Data Center Move Part 1: Why I did it

This past Saturday night, we did what I hope we only have to do every 5 or more years - we switched data centers. We had to physically move all of JangoMail's servers out of one data center and into another, while minimizing the impact to our customers. JangoMail was offline during the entire move, which makes it a stressful experience. The worst case scenario would be for JangoMail to go offline and never come back online. A lot would have to go wrong to realize the worst case scenario, but still, taking your life’s work offline for multiple consecutive hours while trying to move and reconfigure equipment as fast as possible in the wee hours of a Sunday morning is a large undertaking. I decided to write this article 1. As an account for myself of what we did right and wrong and 2. To help other sysadmins navigate the messy waters of switching data centers.

Why I moved data centers

Because this article isn’t meant to be an endorsement or complaint against any organization, I won’t mention the names of the old data center and the new data center. I’ll simply refer to them as Old Data Center and New Data Center.

I moved because I was unhappy with Old Data Center. Old Data Center has colocation facilities all over the country and if the cabinet space you need is under a certain size, you can’t buy from them directly – you must go through one of their reseller partners. We rent two full cabinets to house our equipment.

My issues with Old Data Center were:

  1. Every 6 months for the last 2 years, they increased their prices.
  2. I was unimpressed with the onsite staff at Old Data Center. Whenever I needed to see them in person for a simple issue of getting a new access card made for a new employee, I’d show up at the assigned time and ALWAYS have to wait for the right person to be available. They weren’t dressed well, nor were they articulate.
  3. 18 months ago, when we expanded from 1 to 2 cabinets, as part of the setup for the second cabinet, a data center employee physically cut the Ethernet cable connecting our main cabinet #1 to the WAN. We were down for 7 hours, and no explanation was provided by Old Data Center as to why that employee thought cutting that cable was a good idea. I requested a credit, and 12 days later Old Data Center responded that we would get a credit of $46.38. I protested, citing the damage that had been done to our business, and the credit was increased to $324.66. Our monthly fee for the one cabinet was approximately $1,200 and it was my opinion that for such a boneheaded mistake, one full month’s credit should’ve been issued.
  4. Setup fees were, in my opinion, too high. When we needed to increase the Amps for our power circuit, along with paying for the increased Amps, a $500 setup fee was incurred. When we wanted to move from regular cabinet doors to mesh cabinet doors for better heat dispersion, we were quoted $750 per cabinet.
  5. On the few times that there were connectivity issues with Old Data Center, we would call the support number of the reseller reporting the outage, and then they would contact Old Data Center on our behalf to get an explanation. I didn’t like the multiple layers involved.
  6. After four years of having been a customer of reseller of Old Data Center, and never being late on an invoice payment, they suddenly informed us that we owe them a security deposit of $2,315. The reason? Apparently Old Data Center was suddenly charging reseller a security deposit. Again, I wasn’t pleased with this multi-vendor relationship.
  7. I didn’t feel like a valued customer.

The reasons I picked New Data Center were:

  1. It was a single, independent company that owned, managed, and supported the datacenter.
  2. The monthly fee was less than that of Old Data Center, and already included mesh doors.
  3. New Data Center also provided Internet-connected PDUs in each of our two cabinets.
  4. When I toured New Data Center, all employees were dressed professionally, courteous, gave me their business cards, explained their positions, and offered help above and beyond what I expected.

Stay tuned for Part II, in which I’ll detail the planning of the data center move, and Part III, where I’ll detail the actual move operation.