Designing an audio adblocker
Great article, thank you for sharing.<p>About the machine learning model; forgive me if any of these questions were covered in the article:<p>1)  Why classify as three categories (music, talk, ads) versus just two (ads, content)?  Seems like that might help simplify things, since I assume all we care about is whether something is an ad or not.<p>2)  So based on the description you&#x27;re feeding 4 seconds worth of buffer into the model for classification, every second; the model being a stateless RNN.  I&#x27;m curious, did you try just using a (1D) CNN instead?  That would allow you to use a more robust architecture, versus RNNs which tend to be finicky.  (And RNN doesn&#x27;t seem to provide benefit, since you aren&#x27;t using its state, other than potentially being a smaller model).<p>3)  AFAIK there are loss functions which penalize false positives (in this case, penalizing incorrectly tagging content as ads).  Was this experimented with?<p>4)  This one is more of a curious idea:  4 seconds worth of buffer might lack enough context, which we can see play out in some of the failings alluded to in &quot;Future improvements&quot;.  So I&#x27;m curious...<p>Suppose, if your architecture isn&#x27;t set up this way already, that you&#x27;ve got a layer before your final layer that&#x27;s just a little bit bigger than the number of categories.  Say, 32 features.  (A lot of RNN architectures are built with an output matrix that&#x27;s HiddenState =&gt; Outputs (n=3).  So I&#x27;m suggesting HiddenState =&gt; Embedding [n=32] =&gt; Outputs [n=3]).<p>Train that architecture like before.  Now, take that trained model and chop off the output layer so that its output is those 32 element vectors.  We can now build a secondary, context model on top of it.  It takes as input those 32 element vectors (one vector per second) and outputs the same classifications (ad&#x2F;not-ad).  But since its input is much smaller, you can train an RNN with a much, much deeper BPTT.<p>Hopefully this model can not only be more intelligent by using context, but also take the place of your hysteresis.<p>Since the underlying model is fixed, you can pre-compute its outputs on your dataset, so there&#x27;s absolutely no cost while training the higher order model (and the pre-computed dataset will be really tiny; 32 elements every second would be only 100mb for your 10 day corpus.)<p>Now, this is somewhat dependent on your corpus being of the nature &quot;here&#x27;s a long section of audio where the ad is tagged precisely&quot;.  i.e. it can just be a mixed bag of &quot;this whole audio segment is ad&#x2F;not-ad&quot;.  If you don&#x27;t have that kind of data you&#x27;d need to create it (assuming my crazy idea is worth the effort).  Hopefully you wouldn&#x27;t need much.  This higher order model shouldn&#x27;t need to be very complex, which means it doesn&#x27;t need a big corpus.<p>Or you could try to use a CTC loss function, in which case you just need a dataset that&#x27;s large chunks of audio and vague labels like &quot;this length of audio has 1 ad somewhere in it.&quot;<p>By the way, this isn&#x27;t intended as advice; I&#x27;m by no means a domain expert here.  I&#x27;m really just thinking out loud.  And since none of the other comments at the time of this writing are discussing anything other than the morality of this software, I thought I&#x27;d take the time to inject some more technical discussion.<p>&gt; hip-hop music, easily mispredicted as advertisements<p>Well given the endemic use of name dropping and product placement in popular hip-hop, I&#x27;m not sure that&#x27;s a misprediction ... :P<p>Joking aside, one thing that might be useful is to find a stream with positively no ads (e.g. Spotify Premium, Apple Music, etc).  Play that through a trained classifier.  If the classifier ever detects an ad, add that audio sample to your corpus.