Got a stupid coworker quote last week.
I used to be a developer in my (UK) company before moving to DevOps, more of an engineering role. Due to previous experience, the Jenkins cluster and the Subversion (code version-control system) server have become my responsibility, and generally run without trouble.
Yesterday, a colleague got an error with their SVN update, so I hopped onto the server and checked the logs. After giving them their help, I switched programs and forgot I had the server open. After lunch, I flicked through my windows and found the shell session still open, still showing the logs, and surprisingly the error log file is growing.
I can see one entry roughly every 3 seconds (alarm bells ringing) from one IP address (uh-oh, compromised machine?). 404s are being thrown like Hallowe'en candy. This is a very strange thing because navigating an SVN repository through a browser will not show you files that do not exist, and the regular errors suggest something is spider'ing the repository, very badly, rather than a human doing so manually. Some kind of script out of control?
I track down the IP address to our US office on the East Coast. Due to the time difference, the day hasn't started yet. However, we have a second office in the same timezone where people habitually get into work an hour earlier. So I start an IM with one of my technically-minded friends in this office and mention the bizarre stream of SVN errors.
Alexa*: "Oh, you mean the thing Carlos* set up yesterday?"
Carlos is a developer I used to work with before I switched teams, and works from the other US office. He is aware that the internal infrastructure is managed exclusively from the UK. She forwards me an email where he has proudly set up a third-party system to index all files in our repositories so they can be searched, and of course it's running from his workstation. He's shared this information only with people he currently works with.
I put the clues together and conclude the idiots who wrote the software have decided to spider every file at every revision in the repository through the web interface, moving on whenever they receive a 404. This is an insane approach because SVN natively supports providing a log of all revisions of any file. Also, we already have web interfaces to the repositories that let developers view the file history (although, fair enough, it doesn't have a search function).
With nothing to do until the hapless developer gets in, I send a reply to the email and use iptables to drop all incoming traffic from his workstation so the logs aren't filling up with noise, meaning I can diagnose any genuine issues. Add to, his indexing service is running across a VPN link across the Atlantic, slurping down the whole contents of the repository. We have 4 of them, and I realise he's probably set up all 4, and it's going through in alphabetical order. The second repository is more than 10x the size and contains 10 years of history - definitely not the sort of thing you want to spider. Thankfully I drop the traffic before it finishes the first.
A long-winded setup, but I think it helps to frame what happens when Carlos gets into the office and signs into the company IM:
Carlos: "So Alexa told me you had to disable my SVN access?"
me: "Only because that service you're running is generating tonnes of errors and is clearly doing it wrong."
Carlos: "Can we tell the server to ignore the errors for a specific user and I'll get the service to run under that?"
Er, what? You're a developer, don't you recognise bad practise when it's being presented to you from a log file?
me: "...no, this service is badly designed if it's really doing what I think, and there is no way I'm comfortable with it polluting my logs with this much noise."
Carlos: "Well how are we supposed to trial it?"
Yeah, I think the trial just ended, pal. Took me a long time to impress upon him that he's playing with critical infrastructure and needs to inform me or the rest of the IT teams whenever he decides to plug something that heavy into a system used by the entire company...