Would love to hear funny stories about your ML implementations. What kind of bugs have you encountered?
One of my favorites – not sure if it’s true or not – was that a few years back the US Army wanted to build a computer vision model to detect camouflaged tanks. They got some data scientists to build a model, and these folks got to borrow a tank for a couple of days, drive it around the woods, and take lots of pictures. One day they did it without camo and labelled it. The next day they got the camo nets and did the same. They built a model with these pictures, and did everything right – holding back a portion for testing and validation. When they were done, their model was incredibly accurate. They had succeeded!
Then they went into the field to test it. And it failed, miserably. They couldn’t figure out why. Their test set and validation sets were properly selected and randomized. It should work.
And then somebody pointed out that the weather was sunny on day 1 (no camo), and cloudy on day 2 (with camo) , so instead of a camo detector, they had actually built a cloudy sky detector instead…