Facebook's Download-Your-Data Tool Is Incomplete
This is a topic that I&#x27;m intimately familiar with, thanks to a bizarre set of circumstances (and a ton of reverse engineering). Story above, technical details below:<p>Part 1:<p>A couple years ago, I noticed that the number of photos I was tagged in kept going up and down, as a couple of people I knew would disable their accounts occasionally, and re-enable them a couple weeks later.<p>I manually the images from them, but wanted a way to automatically scrape any images I was tagged in, so I wouldn&#x27;t need to do this manually.<p>I got myself a Facebook Graph API key and created a sample app with full account permissions, only to discover that Facebook won&#x27;t let you export photos you&#x27;re tagged in (that you didn&#x27;t take). The numbers the API reports are wrong, and there&#x27;s no indication that it&#x27;s being purposely redacted.<p>As a result, I wrote a tool that crawls a profile given a set of authenticated cookies, and essentially clicks the download link automatically on every photo. This worked decently well for a couple years, and continues to work to this day.<p>Part 2:<p>I had some spare time on my hands in December 2019, and wanted to write a tool to browse chat logs from across a variety of services (Facebook Chat, Hangouts, SMS), such that you&#x27;d be able to click a name and see a chronological discussion, regardless of what service it was on. I downloaded the Facebook data dump, figuring that was the easiest way to get access to my Messenger data.<p>The Messenger dump revealed a few things that surprised me:
* The character encoding is messed up, and requires decoding as Latin1, then re-encoding as UTF-8<p>* Some messages are straight up missing, despite being in the UI. The dump is supposed to include attachments (images are included), but is missing audio messages &#x2F; voice snippets, presumably among others.<p>* If a user has deleted their Facebook account, the username will appear solely as &#x27;Facebook User&#x27;, so now you need to figure out who you were actually talking to. Some conversations were very obvious, but others involved wasting a ton of time on and involved dumb techniques (like finding Adium logs of the same chat from an old computer).<p>To identify certain conversations, I started scrolling back through certain Facebook posts (which I wrote), to figure out who had been at certain events with me (to narrow things down). I read a bunch of comment threads that didn&#x27;t appear to make much sense to me, until I realized that anyone who deletes their account also has their comments removed, so basically all old comment threads are somewhat nonsensical if anyone in the conversation has since deleted their account. For comparison, deleting a reddit account changes the ownership of a comment&#x2F;post to [deleted], which seems much more appropriate.<p>Presumably wall posts (including happy birthday messages) from people who have since deleted their accounts are also removed, which is exceedingly shitty - if someone sends you a greeting card and then dies several years later, it&#x27;s not like the post office comes to your house to take your cards back in the middle of the night.<p>Part 3:<p>Because of this, I figured that the only way to mitigate future data loss on Facebook is to consistently archive things. Since the &#x27;download your data&#x27; tool is basically useless, I started work on a tool that scrapes the site and &quot;decompiles&quot; pages into raw directed graph DB rows, which can be re-rendered into a new version of the site. It features a reasonably complete implementation of Facebook&#x27;s TAO (<a href="https:&#x2F;&#x2F;www.facebook.com&#x2F;notes&#x2F;facebook-engineering&#x2F;tao-the-power-of-the-graph&#x2F;10151525983993920" rel="nofollow">https:&#x2F;&#x2F;www.facebook.com&#x2F;notes&#x2F;facebook-engineering&#x2F;tao-the-...</a>) on top of PostgreSQL, and works decently well - notably, it also maintains things like proper links to profiles and stores all assets offline.<p>Writing a bug-compatible &quot;decompiler&quot;&#x2F;&quot;recompiler&quot; taught me several things about how the site works (or rather, doesn&#x27;t). Here&#x27;s a small list of errata I&#x27;ve discovered along the way:<p>* Objects can have multiple FBIDs<p>* FBIDs can contain comments&#x2F;reactions<p>* Since there may exist multiple FBIDs for a given object, it&#x27;s quite common for multiple comment threads to exist for a given item, such that commenters on one don&#x27;t see the responses on the other (and vice versa). Several of my friends have confirmed finding disjointed discussions on their posts after discovering this bug.<p>* Facebook has several types of deprecated reactions that they store in the DB, which cannot traditionally be viewed from the site anymore. Sucks to be you if you reacted to something that way.<p>* Certain objects can get lost in their UI, with no easy way to find them. Uploading a photo in a post will put it in your Timeline Photos album, but uploading a photo as a comment to someone else&#x27;s post will basically make it impossible to find again.<p>* The number of reactions&#x2F;comments on a given post is often wrong - this isn&#x27;t the traditional bug due to eventual consistency, but rather is due to not adjusting the counts for items when a person deletes their account. To a certain degree, this will show you how many people that interacted on something have departed the site.