Stories from tech 

Icee vs My Network

I am the Head of IT for a multi-site car wash. The infrastructure has come a long way since I started, but I still have to deal with less than ideal ways of connecting my networking devices. So on Thursday, I'm sitting in the office of our most profitable car wash on a bright and sunny day. Perfect for washing cars. Suddenly, our cashier computer loses connection to the server. Then down goes the manager's connection. Within a few seconds every computer on site had lost connection, which meant people who had already paid for washes were stuck in line before the tunnel. I immediately whip out my Surface and start pinging things to get to the bottom of it. I pull out my phone and access the offsite network controller to see if any of my networking devices are offline, or if somehow my vlan configuration shat itself. Everything pointed towards my router having failed. Inter-vlan routing seemed to have ceased. I grab an extra router I keep in my car and port the configs over and stick it in the network… difference.

By this time I had the Owner and General Manager breathing down my neck asking why I bought crappy networking hardware. I'm getting desperate. I make sure I have a backup of my network config and then move everything onto a single vlan to eliminate the need for inter vlan routing…… Still nothing. Crap.

"Maybe some wires spontaneously broke? Maybe there is a bad connection somewhere. I know I probably should have started with Layer 1, but honestly, how often do wires just stop working?"

As I trace the supply cable from one of my switches to the wall, it gets really sticky towards the last few inches. A long sticky string of God knows what follows it as I unplug it from the wall. I let out an audible "What the Hell?!". The entire wall jack is covered in sticky residue from some sugary drink. The sticky residue has bridged the connections, soaked the punchdowns, and glued the entire faceplate to the sheetrock. Come to find out someone bought Icees for all of the employees the other day, and they all drank them in the office by the super important computers they aren't supposed to be near in the first place. As soon as I put a new jack on, everything came back up, and the Owner told the manager to immediately open back up instead of letting me restore the VLAN config or put the networking hardware back where it's supposed to go. so now everything is on the same subnet and temporary Ethernet cables are crisscrossing the office without me being allowed to turn everything off and put it all back where it goes. I unplugged the guest WiFi though, so there's that.

Now this week I have to figure out how I'm going to alter the infrastructure to make it more resillient to rogue Icees.

TL;DR: Some ass hat spilled blue Icee all over a very important wall jack in the office and then failed to notify anyone. It took a few days for the damage to finally occur, but when it did, it brought down our most profitable car wash for several hours.

submitted by /u/Firemanz
[link] [comments]Click here to read the original post

Related posts

Leave a Comment