Thursday 7 February 2013

The Server Hardware Misconfiguration

Umm, it doesn't seem to be working correctly...
It's almost comical that nearly every technological addition or upgrade to our server room is fraught with one mindless defect or another upon arrival. Something may be missing, something may be broken, or the object itself is the wrong one, wrong size or even out of date.

I remember on one occasion we needed a simple power strip for a server rack to power some switches. Nothing fancy, but it needed to have certain connectors for it to do it's job. 5 months and 3 power strips (all wrong) later and we finally get one that we have to change the plug type on ourselves, because we couldn't afford to wait any longer for yet another product exchange.

We take every care to 'spec' out the equipment we need. We research what's available, we choose carefully to meet our needs and budget. Then we pass it on to the bean counters to mess it up in their own special way. Saving money is the name, buying crap, defective hardware is the game (as long as it's cheap).

We have had servers arrive without mounts. Hell, we have had servers arrive without power supplies.
Cables have been ordered too short to use, connectors have been the wrong type, network cards missing. It's a wonder we have never had any explode into raging fireballs before we even switch them on.

But nothing could have prepared us for the next round of world class purchases.

You see, bean counters are somewhat suckers for deals that seem too good to be true. They seem too good to be true for one simple reason: THEY ARE TO GOOD TO BE TRUE! And they never learn!

Instead of ordering from a manufacturers outlet or middle company supplier, our management team have found a company that supplies 'refurbished' equipment. A bargain basement, Chuckle Brothers outfit called "Discount Server Hardware".
"Can we interest you in the deal of a lifetime?"
To make matters worse, DSH periodically phones our management up with their latest defects, sorry, I meant deals. So it is no surprise that when they have something we don't need in stock, we usually end up with it on a shelf somewhere for the next five years, gathering dust until we are able to sneak it into a skip out the back door.

Discount Server Hardware's Latest Stock? Nope, Our Latest Stock!
Some three years ago we installed a disaster roll-over system on another site. To cut a long story short, should our main site be wiped off the map in a nuclear attack, or should an alien race decide our company would be a good place to begin recolonisation, everyone else at our other sites will still be able to get their emails. In order for that to be achieved, our essential servers (which means, all of them) along with the data they contain are synced to the remote site.

The main problem here is that as the systems grow in HQ, the remote site must be increased in capacity to cope with it. The trouble is the hardware at the remote site is older than the hardware in HQ, and so to increase it's capacity further requires a major upgrade.

Cue purchase number one: 2 years ago, two main boards and a box of amazing looking cables that would make the US government gasp in wonder turn up on our doorstep. Upgrades for the remote SAN, with NO instructions on how fit them or get them working.

The SAN, any SAN is a complicated machine. It can't just have it's hardware changed and then be expected to work as it did before. It has to have diagnostics run, and firmware upgraded to cope with the new boards. It is a major operation which leaves little room for mistakes. When we are presented with the box of goodies, this all lands on deaf ears as expected. The training coarse on how to perform this miracle is too expensive, the tools and software are only available to certified specialists, and one error and the entire SAN will need to be replaced. All has to be completed without any downtime at all!

The final nail in the coffin is that the entire chassis will need to be replaced as well, as the current one cannot support the new boards. If you buy a new chassis and fit the new boards, with new firmware etc, you may as well just invest in a new SAN.

The parts are still in their box under a bench, cunningly disguised by a . We have since discovered that the parts they sent are not the parts that were ordered, and are no more main boards for a SAN than a Domino's Stuff-crust Pizza. Needless to say DSH saw us coming.


Purchase number two: One morning a band spanking new, second hand, discontinued (so discontinued it takes over half an hour to find the user manual online in the 'archive' section of the manufacturers website) KVM lands on my desk to replace the old, clunky one that hasn't worked properly since we got it 4 or 5 years ago. KVMs are control boxes that allow us to control multiple servers from one set of monitor, mouse and keyboard. Or in this case from two sets.

We were a little dubious, after all we didn't ask for it. When we find out that they are replacing our current Monitor/Mouse/Keyboard unit as well we are immediately suspicious. We get the KVM mounted in the rack and a couple of servers connected as the new screen arrives in an ominous brown cardboard box.

It is then we discover the ulterior motive for this new found generosity. The old stuff we pull out will be sent to 'The Lab'.

The Lab is where they teach computer hardware/troubleshooting to the noobs. This is not as exciting as it may sound, as they are taught by people who have no idea what it's like to work in a professional IT Support environment.These are people that only know IT in theory. They firmly believe that everything they have read  in the books, been told at a seminar, or in a YouTube video is the 100% gospel truth. So when things don't go to plan, they come and ask (yes ask) the professionals. This time is usually spent arguing with us that we are wrong, and trying to recall and recount every piece of useless information they have ever heard about the subject. Everything that is except the information they need to complete their project.


To be honest, why give them any technical equipment in the first place? We may as well give them a plastic tin opener and tell them its the latest in wireless technology, as long as we include instructions from an unknown source that can confirm it. We are not sure why they want our old equipment, but it's top priority we get it over to them. The old KVM is whisked away before we have even finished fishing the old cabling out from between the racks.

They are in for a shock with regards to the screen though, as we only have one other and three racks. We are keeping it. Taking advantage of it, it allows us to have two set ups. We move the older one over and hook it up to the new KVM, leaving a space for the new screen.

We unpack the new screen. Before we even get it out of the box we groan in dismay, but not surprise. It's some third party rubbish that weighs a ton and looks like it was built in the 1980's. With it finally in the rack we start to connect this to the KVM. We start... We can't finish because the connectors on the KVM are not a compatible match with the connectors on the new screen.

The screen is NOT made for our systems. Plain and simple, it even says so on the box. We request it is returned, and one more like our older screen is purchased, only more up to date. For some reason the management is fixated on one that doesn't fit and we are told we have to get it working. "Besides they are all like that now and they don't make the others any more."

One Google search and 10 seconds later: "Urrrmmmm, no they are not, and yes they do!!!!"

But it's OK, Discount Server Hardware to the rescue. They have rarely sent us anything that we actually ask for and never one in full working order, but that never seems to stop them unloading on to us every piece of crap they have in their warehouse. They send us an adaptor, which doesn't fit, so they send us another adapter to attach to that adaptor to connect to the screen. Before we know it we have three adaptors and an extra meter of cable before the screen is finally connected to the KVM.

Months later, it's still in the rack, it's still connected, and it still doesn't work. But it's OK, don't panic! The Lab has got our old hardware, we are worse off than before, and it's only a matter of time before DSH has yet another obscure, faulty piece of hardware they can't sell to anyone else to force upon us...

Good Job!

1 comment:

  1. oh man that really is a nightmare situation! Doing your research and proper maintence goes a long way. Some top notch server cabinets can go a long way in helping

    ReplyDelete

Popular Posts