Like the t-shirt says: “That’s what I do: I fix stuff and I know things”…
My time in the USN was on a submarine, and on a submarine you’re expected to be able to do everything and fix anything – so while I was a Radioman managing top secret comms, I studied and trained on everything from nuclear and diesel AC/DC power plant operation to the various electrical, hydraulic, and refrigeration systems used onboard.
Accordingly I’m fairly well versed in a plethora of technical ‘things’ and get called upon at work to do all manner of stuff. Take this morning for example:
When I arrived at work at 0615 I commenced unlocking everything and turning on the lights – and noticed the server room was louder than usual. Upon opening the door I instantly figured out why; it was about a hundred degrees in there and all of the fans in the servers were screaming.
A quick analysis pointed out a high head pressure situation with the one working CRAC (Computer Room AC), which had shut it down about an hour before I got there.
High head pressure is a ‘condenser-end’ issue where, for some reason, the freon isn’t changing states from gas to liquid appropriately anymore. This is typically caused by it being really cold outside, the condenser coils being dirty, or the fan has stopped working. Most AC systems will detect this and turn off the compressor to keep it from being damaged.
Anyway, once I had the backup portable AC units running and things cooled down enough I wasn’t worried about immediate hardware failures, I ran up to the roof to take a look… And sure enough, the condenser fan had seized.
I have two of these and both units are a bit over 30 years old, so this is kinda expected… The backup unit’s compressor failed two years ago, but with the transition to “the cloud” I’ve only got a dozen or so systems running local now so I just tagged out the backup and carried on with the primary.
I fiddle with the fan a bit and get it to go again, definitely a bad bearing.
It sounds horrible and I really need to replace it, so time to call my HVAC company…
So I head back down to the office and call the HVAC guys who don’t seem to be too happy to answer the phone and tell me that even for an emergency the best they can do is get someone out Thursday.
By Thursday I may lose a quarter million in hardware if the portable units die (as they tend to do as soon as you rely on them 24/7).
So the obvious solution is to on-the-fly disassemble the backup unit’s rooftop condenser, scavenge the fan and associated controls out of it, and transplant them into the primary unit’s condenser… I can do that, no problem!
Given the rusted nature of every nut and bolt in the thing, the fact I got the old one apart in about 15 minutes is pretty impressive – and I don’t even need a tetanus shot after the fact!
I decided to wait for my roommate to arrive before shutting down the limping primary because the two of use would be much faster. So an hour later my roommate arrived and helped me disassemble the primary unit, get the failed fan out of it, and transplant the backup fan into it.
So the whole surgical process of remove / replace / rewire only took ten minutes and didn’t overheat the servers again…
After the surgery everything was tested and worked better than ever, and being as no one was in the office (it’s the day before a holiday, so everyone is taking today off to make it a 4-day weekend) we decided to close up shop and go get lunch.
Which is right about the time the HVAC company decided to show up to fix things. 😀
Ya snooze ya loose guys.
Listening to "Le Mirage" by Dana Jean Phoenix