We built a system like Apple’s to flag CSAM and concluded the tech was dangerous
One additional issue that I haven&#x27;t really seen discussed is how to handle a situation when a false accusation is made.<p>If a person knows the right people who work at these companies, things get sorted out, but I imagine sometimes a person is forced to just handle the consequences.<p>Stepping away from CSAM and going back to something like developer account and apps getting banned on platforms for violating vague &quot;guidelines&quot;. It&#x27;s someone&#x27;s livelihood that&#x27;s sometimes destroyed. Demonetization, apps getting banned, payment processors freezing accounts are mostly black box events and most situations aren&#x27;t even related to crimes dealing with CSAM.<p>If it was something the government made a mistake with, there&#x27;s legal ways to fight for your rights. There&#x27;s generally a level of transparency that is afforded to you.<p>It is concerning that people flagged for handling CSAM will not know if they have been manually reviewed. The need to keep the forwarding to authorities a secret is understandable, but a human review before forwarding is only necessary if you expect false positives to begin with. Keeping that flag secret seems like another black box you can&#x27;t fight as a user.<p>I don&#x27;t deny the value of catching these criminals, but it throws the idea of due process out the window when the only assurance so far has been &quot;trust us to do the right thing&quot;.<p>It&#x27;s also weird how Apple has chosen to intentionally insert itself into the investigation pipeline rather than just let NCMEC handle it like all other cloud providers.<p>I am glad this hasn&#x27;t flown under the radar just because it is Apple who is making these promises. I have heard non-tech people talk about this but there&#x27;s a lot of misunderstanding.