Corruptions within Calendar: duplicates. How to analyse their cause and fix? I am using Calendar on a Mac running MacOS X Mountain Lion (10.8.5).

This Mac is manually regularly synchronized with an iPhone running iOS 7.1.2.

This Calendar is storing 15 years of events organized within 9 "calendars". Since some of this information is highly sensitive, professionnal or private, I don't synchronize them on any form of public store calendar (iCloud, Google Calendar…). On the other hand I have many Time Machine backups and full backups.

Recently I discovered quite unexpectedly that since summer 2001 I have duplicated full day events within Calendar on my Mac. I have been able to see these ones quickly since their duplicate nature is directly visible. This isn't a general case: most of my full day events aren't
duplicated. But all my 9 "calendars" are hit by this corruption.
I estimate that I have about a few hundreds of events in this case.
I see the same corruption on my iPhone.

I exported one of my calendar, and extracted one of the duplicated entries. Here is the output of a diff on the 2 .ics extracts:

••My_Mac••$ diff duplicate.[12].ics
2c2
&lt; UID:74FC7CC1-016C-4A74-9E02-7ECDD82C8129
---
&gt; UID:9B6BC4CD-5859-4DC2-8DEA-9158CB8F9B0D
10,11c10,11
&lt; X-WR-ALARMUID:D0FE4A14-981C-4409-84C1-B11107F7EC31
&lt; UID:D0FE4A14-981C-4409-84C1-B11107F7EC31
---
&gt; X-WR-ALARMUID:48141767-C3C6-4131-9984-0DD080833D9F
&gt; UID:48141767-C3C6-4131-9984-0DD080833D9F
••My_Mac••$


Notation: the string ••name•• means that "name" was redacted.

Here is what I found within /var/log/system.log and which might be related:


Sep 13 10:08:32 ••My_Mac•• SyncServer[93677]: [0x7fbe60c0bdd0] |SyncServer|Warning| Refreshing watchdog because of a calendar time change alert.
Sep 13 16:09:10 ••My_Mac•• SyncServer[94189]: [0x7fd25a40bdd0] |SyncServer|Warning| Refreshing watchdog because of a calendar time change alert.
Sep 14 03:21:15 ••My_Mac•• SyncServer[94351]: [0x7f9e1ac0bdd0] |SyncServer|Warning| Refreshing watchdog because of a calendar time change alert.
Sep 14 08:56:41 ••My_Mac•• SyncServer[94351]: [0x7f9e1ac0bdd0] |SyncServer|Warning| Refreshing watchdog because of a calendar time change alert.
Sep 15 14:11:39 ••My_Mac•• SyncServer[94351]: [0x7f9e1ac0bdd0] |SyncServer|Warning| Refreshing watchdog because of a calendar time change alert.
Sep 16 00:25:17 ••My_Mac•• SyncServer[95764]: [0x7faf92c0bdd0] |SyncServer|Warning| Refreshing watchdog because of a calendar time change alert.
Sep 16 13:36:27 ••My_Mac•• SyncServer[96213]: [0x7f9470c0bdd0] |SyncServer|Warning| Refreshing watchdog because of a calendar time change alert.
Sep 16 13:51:33 ••My_Mac•• CalendarAgent[90827]: Invalid char _ for PropertyName in line 7
Sep 16 13:51:33 ••My_Mac•• CalendarAgent[90827]: Unexpected EOF, returning last token as fallback



How may I analyse where these duplicates events are coming from?

How may I find the date and time an event might have started such a corruption of my agendas? Without a date of the beginning of the damage, my backups are of little help. Moreover, they will imply a total rebuild of the correct events which occured after the damage.

How may I get a correct vision of this corruption of all my "calendars"?

And moreover how may I fix this huge and apparently random data corruption?
I haven't yet a correct explanation of these corruptions, but at least I wrote a solution to have a clear vision of the extent of the damage and to fix it.

Here is a perl script: duplicate.pl:

$ cat &lt;&lt;'eof' &gt;duplicate.pl
#!/usr/bin/perl
use strict ;
use warnings ;

# program reading on its standard input a file under
# ics format exported by iCal or Calendar
# Both only export one calendar at a time

# file = name of created file for a given calendar

my %file = () ;

# filedesc = file descriptor of the created calendar file

my %filedesc = () ;

# hash of all unduplicated events local_event_id
my %events = () ;

# current event storage
my @event = () ;
my $dtstart = '' ;
my $dtend = '' ;

# number of events analysed
my $num_event = 0 ;
my $duplicate = 0 ;
my $calendar = '' ;

# state booleans
my $in_header = 1 ;
my $in_event = 0 ;
my $in_summary = 0 ;
my $line = '' ;
my $summary = '' ;

# local event identifier :      summary;dtstart;dtend
# because ';' is never used within a name
my $local_event_id = '' ;

while (&lt;STDIN&gt;) {
        $line = $_ ;

# header :      BEGIN:VCALENDAR
#               ...
#               BEGIN:VEVENT

        if ( $in_summary ) {

#               continuation line of summary

                if ( $line =~ /^ (.+)\r\n$/ ) {
                        $summary .= $1 ;
                } else {

#                       end of summary continuation lines analysis

                        $in_summary = 0 ;
                }
        }
        if ( $line =~ /^SUMMARY[^:]*:(.+)\r\n$/ ) {
                $summary = $1 ;
                $in_summary = 1 ;
        } elsif ( $line =~ /^BEGIN:VEVENT/) {
                if ( $in_header ) {
                        $in_header = 0 ;

#                       print every lines of event or header

                        foreach $line (@event) {
                                printf {$filedesc{$calendar}} "%s", $line ;
                        }
                }
                $in_event = 1 ;
                @event = () ;
        } elsif ( $line =~ /^X-WR-CALNAME:(.+)\r\n$/) {
                $calendar = $1 ;

#               create .ics file

                if ( ! defined $file{$calendar} ) {
                        $file{$calendar} = $calendar . ".ics" ;
                        if ( -e $file{$calendar} ) {
                                die "$file{$calendar} already exists\n" ;
                        }
                        open ($filedesc{$calendar}, "&gt;", $file{$calendar}) ;

#                       print every lines of header

                        foreach $line (@event) {
                                printf {$filedesc{$calendar}} "%s", $line ;
                        }
                        @event = () ;
                        $in_event = 0 ;
                }
#               printf STDOUT "calendar = %s\n", $calendar ;
#               printf STDOUT "file = %s\n", $file{$calendar} ;
#               printf STDOUT "fh = %d\n", $filedesc{$calendar} ;

        } elsif ( $line =~ /^DTSTART[^:]*:(.*)\r\n$/ ) {
                $dtstart = $1 ;

#               printf STDOUT "DTSTART = %32s\n", $dtstart ;

        } elsif ( $line =~ /^DTEND[^:]*:(.*)\r\n$/ ) {
                $dtend = $1 ;

#               printf STDOUT "DTEND = %32s\n", $dtend ;

        } elsif ( $line =~ /^END:VEVENT/ ) {

# it's only on closing an event definition that we have
# a complete local event identifier

                $local_event_id = "$summary" . ";" . "$dtstart" . ";" . "$dtend" ;


                if ( defined ( $events{"$local_event_id"} )) {

#                       duplicate event

                        printf STDOUT "\n\tduplicate\t%s\n", $local_event_id ;

#                       free event storage

                        @event = () ;
                        $in_event = 0 ;
                        $duplicate++ ;
                } else {

#                       new event
                        $events{$local_event_id} = 1;

                        if ($in_event) {

#                               print every lines of stored event

                                foreach $line (@event) {
                                        printf {$filedesc{$calendar}} "%s", $line ;
                                }
                                @event = () ;
                                $in_event = 0 ;
                                $num_event++ ;
                                $in_header = 1 ;

#                               show progress
                                if (($num_event % 100) == 0) {
                                        printf STDOUT "\n%8d", $num_event ;
                                } else {
                                        print STDOUT "." ;
                                }
                        }
                }
        } elsif ( $in_event == 0 ) {
                $in_header = 1 ;
        }

#       store every line of event or header

        if ($in_event || $in_header) {
                push (@event, $line) ;
        }

}

printf STDOUT "\nevents:%12d\n", $num_event ;
printf STDOUT "duplicates:%8d\n", $duplicate ;

#                       print every lines of ending

                        foreach $line (@event) {
                                printf {$filedesc{$calendar}} "%s", $line ;
                        }

close ($filedesc{$calendar}) ;
exit 0 ;
eof
$ chmod u+x duplicate.pl


And here is the way I used it:


From Calendar, export a given calendar through:

File &gt; Export &gt; Export...

let's say it is: Documents/Calendar/2015/professionnal.ics

NB.: this will be a backup in case of problem, and a much more practical one than any file manipulation with the help of Time Machine
Install the above script as:

~/Documents/Calendar/src/duplicate.pl

Go in directory to test for fixed version, for example:

mkdir ~/Documents/Calendar/2015.fixed
cd ~/Documents/Calendar/2015.fixed

Run duplicate.pl:

../src/duplicate.pl &lt;../2015/professionnal.ics


which will  display the number of events read and the number of duplicate found and create here a fixed version of the calendar:

~/Documents/Calendar/2015.fixed/professionnal.ics

Compare the result with the original corrupted version to check everything is OK:

 diff ../2015/professionnal.ics .

Within Calendar select the calendar professionnal and delete it
through:

Edit &gt; Delete
Import the fixed one through:

File &gt; Import &gt; Import...