Building Sentry, a service to process native crash reports and minidumps
I built Yahoo&#x27;s in-house mobile app crash reporting tool a few years ago (still in use). I used an on-premise install of Sentry as the UI. At the time, Sentry didn&#x27;t really support mobile error reporting, so I built something much like what&#x27;s detailed in this post and called it the Processor.<p>I regret never having made the time to open-source what I built. The Processor is written in Python, takes reports from mobile devices, unwinds, symbolicates, retraces, unminifies, etc as needed, then generates a Sentry &quot;event&quot; and forwards that to our on-prem Sentry instance.<p>I also built the SDKs. For iOS, I used PLCrashReporter. These days I&#x27;d probably use KSCrash. An important point here. On iOS, the unwinding is done on the device. So all you have to do on the backend is symbolicate it. Another point: it&#x27;s relatively easy to get iOS system symbols. Plug an iOS device into a Mac running Xcode and the symbols are transferred from the device to the Mac. You can then harvest them however you need. In fact, Apple has apparently stopped encrypting OTA updates so you no longer need an iOS device to get the symbols:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;Zuikyo&#x2F;iOS-System-Symbols" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Zuikyo&#x2F;iOS-System-Symbols</a><p>For Android NDK crashes I&#x27;ve tried a few approaches and still don&#x27;t have a satisfying solution. Originally I went with breakpad + minidumps on the device. On the backend, the Processor runs the breakpad stackwalker on the minidump. Another important point: the unwinding is occurring on the backend in this case, unlike iOS where it&#x27;s done on the phone. (A minidump is basically just a snapshot of all the thread stack memory, plus some extra diagnostic info.) But to unwind reliably off-device you need the Android system symbols (in addition to the app&#x27;s symbols obviously). Well good luck with that. Google makes the original Nexus Android OS images available so you can harvest those but you&#x27;ll never get symbols for all the various Android devices. I built a tool that can harvest symbols off a device and tried to crowdsource them from Yahoo&#x27;s developers but it&#x27;s not been very successful (there&#x27;s a lot of flavors of Android).<p>Another issue is that minidumps are relatively largish to deal with. So my second approach was two-fold. I&#x27;m still using breakpad&#x27;s crash handler on the device, but I now have it generating the much smaller microdump format. In addition, I&#x27;ve added libunwind to our Android SDK so that after capturing the microdump, I attempt to unwind on the device (also collecting function names during unwinding) and add that info to the report. The Processor then only needs to unwind the microdump if the unwinding on the device failed. Otherwise it just needs to symbolicate. This hasn&#x27;t been wildly successful though. Unwinding on an Android device is trickier than on an iOS device. Also, it&#x27;s almost impossible (well I haven&#x27;t figured out how) to unwind through the ART&#x2F;Java frames that called into the native code.<p>Of course the vast majority of Android crashes are in Java code and this is much easier to deal with these. They are unwound just find on the device so on the backend you only need to deal with deobfuscating the ProGuard minification which is easily done using the mapping file generated by ProGuard.<p>What&#x27;s really annoying with native mobile crashes is that both Android and iOS have their own services for both capturing crashes and unwinding on the device. And because these are integrated with the OS and work out-of-process, they are much more reliable than anything you can do in-process using something like PLCR, KSCrash, libunwind, etc.<p>But, neither OS gives an app access to its own system generated reports. All you get is the lame reports the devices upload to Google Play Console &#x2F; iTunes Connect.<p>Anyway, thank you to Sentry for providing such a great product and I&#x27;m sorry again I wasn&#x27;t able to contribute more. I&#x27;m not sure what I built would work at your scale. It&#x27;s interesting we ended up with similar designs.