title: How to debug "watchdog timeout" crash logs?
body: <p>I'm somewhat new to using iOS crash logs - I'm pretty familiar with app-based crash logs, since they're pretty easy to symbolicate, but I'm still trying to figure out how to read logs produced from a watchdog timeout. To be clear, these timeouts are caused by a Notification Center widget I'm writing, which runs as a part of the SpringBoard process. The timeouts are usually caused by an error in my code (ie. accessing a deallocated object), but instead of getting a regular crash and crash log, SpringBoard simply hangs for 2 minutes before the watchdog triggers a respring.</p>
  
  <p>So I guess my question is two-fold. First, why is it that SpringBoard is simply hanging when my code does something wrong, rather than crashing (and therefore giving me a more useful crash log)? And second, how can I make use of the watchdog timeout logs to figure out the source of the issue in my code? I've tried compiling my code with the <code>-g</code> flag and <code>DEBUG=1</code> option set in my makefile (using theos), and enabling <code>syslogd</code>, but none of these have affected the output of the watchdog timeout logs, unfortunately. Maybe I'm just missing something, but I can't figure out how to make use of the information in these logs, or how to make them provide more useful information than what is there.</p>
  
  <p>Here's a sample log from a watchdog timeout, for reference. I've tried to grab only the relevant sections of the log, but I can provide the entire log if necessary.</p>
  
  <pre><code>Hardware Model:      iPhone3,1
  OS Version:          iPhone OS 5.0.1 (9A405)
  Kernel version:      Darwin Kernel Version 11.0.0: Tue Nov  1 20:33:58 PDT 2011; root:xnu-1878.4.46~1/RELEASE_ARM_S5L8930X
  Date:                2012-01-25 17:38:35 -0800
  Exception Code:      0xfaded321
  Reason:              Watchdog timeout: 120.021790s since last successful ping: 3506u0 3406m0 3406u0 3306m0 3306u0 3206m0 3206u0 3106m0 3106u0 3006m0 3006u0 2903m0 2903u0 2800m0 2800u0 2700m0 2700u0 2600m0 2600u0 2500m0 2500u0 2400m0 2400u0 2300m0 2300u0 2200m0 2200u0 2100m0 2100u0 2000m0 2000u0 1900m0 1900u0 1800m0 1800u0 1700m0 1700u0 1600m0 1600u0 1500m0 1500u0 1400m0 1400u0 1300m0 1300u0 1200m0 1200u0 1000m10004003 1000u0 1000u0 800m10004003 800u0 800u0 600m10004003 600u0 600u0 400m10004003 400u0 400u0 200m10004003 200u0 200u0 0m10000004 0c1328 
  
  Thermal Level:       0
  Thermal Sensors:    2435 2353 5146 2839 2348 2821 2544 2406 2935 2635 32768 2 2736 2703 2430 2435 2421 2436 1973 1834 1971
  
  Frontmost process PID:    4904
  Jetsam Level:             20
  Free Pages:            15086
  Active Pages:           7397
  Inactive Pages:         3351
  Purgeable Pages:        1992
  Wired Pages:           15818
  Speculative Pages:      1772
  Throttled Pages:       85667
  Busy Buffer Count:         0
  
  Process 0 info:
      resident memory bytes:  44748800
      page faults:               3189
      page-ins:                     0
      copy-on-write faults:         0
      user   time in task: 151641.651106 seconds
      system time in task:     0.000000 seconds
  
  Process 0 kernel_task threads:
  thread 0x1 TH_WAIT|TH_UNINT 0x802e0158
      thread priority:          92
      thread sched flags:     none
      kernel cont 8005aa29
      user   time in thread:    11.050610 seconds
      system time in thread:     0.000000 seconds
  // Continues in this format for a while...
  
  ...
  
  // SpringBoard process log
  Process 4904 info:
      Frontmost
      resident memory bytes:  63995904
      page faults:             932726
      page-ins:                 17055
      copy-on-write faults:      5881
      user   time in task:   883.320096 seconds
      system time in task:     0.000000 seconds
  
  Process 4904 binary images:
   0xf6000 &amp;lt;7506c20d86da3f1dbe9bf38f8bda253d&amp;gt;
   0x386000 &amp;lt;cf2cce379dcd3a4c970e3196b908b0b6&amp;gt;
   0x3cd000 &amp;lt;30381ec9e24c3c289f447bf428bda2c1&amp;gt;
   0x500000 &amp;lt;8a7d931c6871371e8c011cb2b9d60a4b&amp;gt;
   0x511000 &amp;lt;4cd6f668c82c3232a60da2f40737bb7b&amp;gt;
   0x3f1000 &amp;lt;fd9fcb38af01393f888320f2a67d9d8d&amp;gt;
   0x3f5000 &amp;lt;d375337d03a7324c9cfb608b7231eeea&amp;gt;
   0x639000 &amp;lt;ee905d084b7a3ee1b8d5ddf3a0d2dc2d&amp;gt;
   0x3fc000 &amp;lt;72f867f586cb34bd94ea52e2b1582b8d&amp;gt;
   0x642000 &amp;lt;87085dc54e1f324cad66b1e07d504fee&amp;gt;
   0x64e000 &amp;lt;25608ce0a9e23bd5a78415481897f8bc&amp;gt;
   0x653000 &amp;lt;af0bbe7579b13d14aee2fc8967e3c933&amp;gt;
   0x1993000 &amp;lt;afb6041043ca3935b08e00630906d9b2&amp;gt;
   0x1997000 &amp;lt;c755a0ab2ec43860931bc844b358df2d&amp;gt;
   0x1a35000 &amp;lt;b2f2268e166838f286a5838cc243bd01&amp;gt;
   0x1a42000 &amp;lt;0a8f131d01f63458846f0b016a88f080&amp;gt;
   0x1a58000 &amp;lt;97daf7f3c6cd34359e7c63ce16b33e6e&amp;gt;
   0x1aa8000 &amp;lt;bdc6016e1f9e3552b41fc6feced31050&amp;gt;
   0x3a3000 &amp;lt;36a26644e27f3bb7b5b2c8157df7dba2&amp;gt;
   0x3a7000 &amp;lt;a9101c6d02d33f128837bb355e8bfc4b&amp;gt;
   0x3aa000 &amp;lt;aa58ea5c933232d99b51b4192dc8f180&amp;gt;
   0x3ad000 &amp;lt;f5f9138532993ca1959bf0973a5de0fd&amp;gt;
   0x1aba000 &amp;lt;943484dfa081c192ec89f3f2329b085a&amp;gt;
   0x3b4000 &amp;lt;aa0b4733baee386ab29bfcbd74beaa57&amp;gt;
   0x3c0000 &amp;lt;dfda867ef28e362bb67a734199725a72&amp;gt;
   0x1ad2000 &amp;lt;bba657e1bd09bab5f888545db2928786&amp;gt;
   0x1ad7000 &amp;lt;fe21097b2cb83481a70c9ab877af55e3&amp;gt;
   0x1ae1000 &amp;lt;ad65601afce130f3abf98b3279fad9ce&amp;gt;
   0x1ae5000 &amp;lt;d138ce2db8b033aba6119acf3d561535&amp;gt;
   0x4dab000 &amp;lt;45d7c264810c364b976dba254572d73d&amp;gt;
   0x4ee8000 &amp;lt;cf8b322a0143350eb7bb7146c90c72d9&amp;gt;
   0x4f7f000 &amp;lt;d203aed2bda137bb9cd5799e8d8fd480&amp;gt;
   0x7325000 &amp;lt;199f52167001328a8e76735cd1f6c12a&amp;gt;
   0x67f5000 &amp;lt;dc67fb590cb49f466ef7e82a3f44de28&amp;gt;
   0x6f62000 &amp;lt;b00c29872f767a3a993ef2e7724af2e8&amp;gt;
   0x681f000 &amp;lt;c4d71529a1d51704706fffa892c4f5d4&amp;gt;
   0x6834000 &amp;lt;9d55da6026ab3f9574a9c9950794f0c0&amp;gt;
   0x6867000 &amp;lt;9f0179854929b62ee16f5aeed620c303&amp;gt;
   0x7375000 &amp;lt;892ebb5b3528fa70720361182fe71fe8&amp;gt;
   0x6894000 &amp;lt;d7a840f7a771adea76922c6ecd09de95&amp;gt;
   0x7379000 &amp;lt;1d40433bbdf09c726cc36edd0894bc03&amp;gt;
   0x737e000 &amp;lt;9ef7d722cc04353aad958c6e2f4a3cf7&amp;gt;
   0x738f000 &amp;lt;4c8273a6631135ee817d250488e2c3a8&amp;gt;
   0x7b41000 &amp;lt;2f8d756ef0223c02b924ddd707315c5b&amp;gt;
   0x7b48000 &amp;lt;734a26aa647d3b76b9f2c497f787bde2&amp;gt;
   0x7b5a000 &amp;lt;563cf37a7e5b39de8c2fe7e7428a8b0e&amp;gt;
   0x7c10000 &amp;lt;ede34c56929f3722b575b295964ddbcb&amp;gt;
   0x7d7a000 &amp;lt;ccec053d8f2a3893840565eb19e2401f&amp;gt;
   0x7ec2000 &amp;lt;4d06e894046a33818ac5458a29a744f1&amp;gt;
   0x619a000 &amp;lt;6525fe796904390bb5a10d71d3ecd11f&amp;gt;
   0x61a5000 &amp;lt;3fc0a7cfe9593de39f1295fd34e3c3a8&amp;gt;
   0x61a8000 &amp;lt;15c155566d4637de9db0fca6a4a4d909&amp;gt;
   0x61bb000 &amp;lt;f3f4ff89e340da2ceb563577d11c6701&amp;gt;
   0x6a26000 &amp;lt;b1c8b77244c8317493bdb5db5adba8bb&amp;gt;
   0x61c1000 &amp;lt;46f5d93c6ba0382891f35b04740e7a77&amp;gt;
   0x6ad8000 &amp;lt;303c7dd64de03ce1a1d34f055c38f5c9&amp;gt;
   0x662f000 &amp;lt;402f53a8b8963879a352a1c799d04536&amp;gt;
   0x69bd000 &amp;lt;dea93838279539519f94dc6ee85e74ce&amp;gt;
   0x69a9000 &amp;lt;54c6fb4190093744b2096fb99c6ac6ca&amp;gt;
   0x568000 &amp;lt;8826a6e7f75f80ec2d99607e8d702d8e&amp;gt;
  
  Process 4904 SpringBoard threads:
  thread 0x2f43e TH_WAIT 0
      thread priority:          22
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x376818cb 0x376802f7 0x376809ad 0x3767ed03 0x3767ebcd 0x36dc5bf9 0x36dc574d 0x33d5aab3 0x33cb5680 0x36dbc73d 0x3f21c4 0x33cb7803 0x3572d50f 0x33d24577 0x33cb00cf 0x356a13fb 0x36dbd7cb 0x36dbd6f9 0x36dc7183 0x376723e1 0x33d2c553 0x33d2c4f5 0x33d2b343 0x33cae4dd 0x33cae3a5 0x30532fcd 0x3712f743 0xf9a33 0xf9074
      user   time in thread:   393.531235 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f44b TH_WAIT 0
      thread priority:          33
      thread sched flags:     none
      kernel cont 80199e39
      user 0x35c633b4 0x3698de7f 0x3698db9d 0
      user   time in thread:     3.898458 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f450 TH_WAIT 0
      thread priority:          55
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x348ca5bb 0x34919e6d 0x32afec1d 0x32afead8
      user   time in thread:    15.476818 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f451 TH_WAIT 0
      thread priority:          54
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33d2b47b 0x348f5573 0x34919e6d 0x32afec1d 0x32afead8
      user   time in thread:     0.000361 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f452 TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x356a2b85 0x356bc533 0x3740b3c7 0x357425a1 0x32afec1d 0x32afead8
      user   time in thread:     3.802697 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f453 TH_WAIT 0
      thread priority:          30
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x3618212f 0x32afec1d 0x32afead8
      user   time in thread:     0.781208 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f45c TH_WAIT 0
      thread priority:          54
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33d2b47b 0x348f5573 0x34919e6d 0x32afec1d 0x32afead8
      user   time in thread:   343.767086 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f460 TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x35c643e3 0x302e2f85 0x32afec1d 0x32afead8
      user   time in thread:     0.050931 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f461 TH_WAIT 0
      thread priority:          63
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33d2b47b 0x302e0a17 0x32afec1d 0x32afead8
      user   time in thread:    58.461374 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f468 TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x169175 0x32afec1d 0x32afead8
      user   time in thread:     0.497826 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f469 TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x16902f 0x32afec1d 0x32afead8
      user   time in thread:     0.000296 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f46c TH_WAIT 0x86369178
      thread priority:          47
      thread sched flags:     none
      kernel cont 801d89e9
      user 0x35c73068 0x32afef35 0x32afecb1 0x31c5a007 0x34abdfc1 0x31c79bcd 0x32afec1d 0x32afead8
      user   time in thread:     0.000439 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f4f7 TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x356a2b85 0x356bc533 0x356aea91 0x357425a1 0x32afec1d 0x32afead8
      user   time in thread:     0.067711 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f502 TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x32dba405 0x32afec1d 0x32afead8
      user   time in thread:     0.010444 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f503 TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x32db96ef 0x32afec1d 0x32afead8
      user   time in thread:     0.022591 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f53c TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x356a2b85 0x356bc533 0x7d7bdbd 0x356aea91 0x357425a1 0x32afec1d 0x32afead8
      user   time in thread:     0.061308 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f55d TH_WAIT 0
      thread priority:          33
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x356aebc9 0x356aea91 0x357425a1 0x32afec1d 0x32afead8
      user   time in thread:     3.892466 seconds
      system time in thread:     0.000000 seconds
  thread 0x2f567 TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 801aac09
      user 0x35c73570 0x33d30671 0x32afec1d 0x32afead8
      user   time in thread:     0.294812 seconds
      system time in thread:     0.000000 seconds
  thread 0x37423 TH_WAIT 0
      thread priority:          47
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x34ac3cf9 0x31c79bcd 0x32afec1d 0x32afead8
      user   time in thread:     0.014877 seconds
      system time in thread:     0.000000 seconds
  thread 0x3b204 TH_WAIT 0
      thread priority:          31
      thread sched flags:     none
      kernel cont 8000f401
      user 0x35c63010 0x35c6320d 0x33d2c423 0x33d2b15b 0x33cae4dd 0x33cae3a5 0x356a2b85 0x356c7b5f 0x33a84a6b 0x356aea91 0x357425a1 0x32afec1d 0x32afead8
      user   time in thread:     0.087839 seconds
      system time in thread:     0.000000 seconds
  thread 0x3d554 TH_WAIT 0x87024e80
      thread priority:          33
      thread sched flags:     none
      kernel cont 801dc8a1
      user 0x35c73cd4 0x32af9311 0x32af90a4
      user   time in thread:     0.000450 seconds
      system time in thread:     0.000000 seconds
  </code></pre>