Ask HN: Strange bug workarounds?
This isn&#x27;t a software bug, but since a lot of these aren&#x27;t, I thought I&#x27;d share because it was a fun one with an unexpected cause.<p>I worked on a floor with about 10 people that was entirely occupied by phone switch equipment (raised floor, wires&#x2F;racks, Halon fire suppression and big enough to seat several hundred people were it not for the equipment).  For two weeks, about every 3 days or so in the middle of the night, the power would cut off.  This was particularly surprising since the entire floor had dedicated battery and diesel backup (regularly checked&#x2F;tested) and they never kicked on.  Our facilities guy was going bald troubleshooting it -- brought in electricians and had the techs checking <i>everything</i>.  There was just no explanation.<p>In a last ditch effort to try to get some information, he setup a laptop with a built in webcam and placed it high enough in the air so as to get most of the site[0].<p>A little history is necessary for the facility&#x27;s design to make sense. At one point this room was used for our mainframe -- we were a local phone company and had a <i>ton</i> of data.  This necessitated having a very elaborate near-line storage device custom built for the company.  It consisted of a multi-million dollar robot (the exact kind you see on commercials building CARS, an arm about the size of an adult man coming out of the floor which ran on a track from one wall to about the middle of the space).  It was enclosed in glass and would move tapes from a large shelving unit into drives and back but it was an open loop system: it never truly knew if it got a tape or if the tape made it to the drive and back and being an imperfect mechanical device, every once in a while it dropped a tape and someone would have to disable it, go in and pick the tape up off the floor (or, more often, the pieces of what was once a tape in some cases).  This robot moved <i>very fast</i> and was <i>very powerful</i> so in a scenario where it&#x27;s a person vs. &quot;big moving robot&quot;... well, there&#x27;d be pieces of person on the floor instead of tape.  Since we liked our employees (and OSHA probably mandated it), the interior of the robot housing was filled with exposed &quot;big red buttons&quot; that would cut the power in an emergency.  The exterior walls of the switch room had the same switches, though these buttons had a large acrylic cover with a hole in it so that you couldn&#x27;t <i>accidentally</i> power down anything.  A choice few of these killed power to the entire site and had a sign indicating that with something along the lines of &quot;OH PLEASE GOD DON&#x27;T TOUCH THIS BUTTON&quot;<p>Janitorial staff had been used to turning the lights out on their way out if they were left on and a new member of janitorial staff discovered, at some point, that hitting that big red button took care of all of the lights at once (along with all of the normally blinking LEDs on the thousands of switch cards, but hey -- it got dark at least!).  So on his way out the door, he&#x27;d walk over to it, look at it for a second, then push it ... powering down ... everything.<p>The workaround was easy.  We were now responsible for taking care of our garbage, dusting and cleaning from that point forward (which I think during my 7 or so years on that floor happened <i>once</i>) and a permanent camera was installed in the ceiling which was powered on a circuit not affected by the buttons.  The buttons remained, though.<p>[0] I think after ruling out everything else he had suspected sabotage of some kind was responsible.  Our doors used RFID badges and visitor logs were accessible, but at that time the doors that were interior to our office space didn&#x27;t require badge access and there were no entries for the doors that one would have checked.