Flight Plan Reception Suite Automated (Fprsa-R) Sub-System Incident 28th August [pdf]
This is a preliminary report on the incident that grounded pretty much all planes in the UK for the best part of a day last month. I absolutely, positively <i>love</i> the trigger for the entire thing:<p>&gt; Although there has been work by ICAO and other bodies to eradicate non-unique waypoint names there are duplicates around the world.<p>I mean, this is like seeing the shark fin in Jaws: even ignoring the suspenseful music, you just <i>know</i> there is going to be trouble...<p>Because, it&#x27;s just perfect, right? A duplicate key, that does <i>usually</i> not occur, so you will happily process your entire triple-ISO-certified test suite, then a million live flight plans, until one day, someone files a flight plan that just happens to include a random collision of the same name for different waypoints on two opposite ends of the world (as an aside, I would <i>love</i> to see the meetings about that: <i>you</i> change your waypoint name! No, this waypoint name is part of our ancient traditions, <i>you</i> change it!).<p>But once that duplicate key is in the system, it goes down <i>hard</i> and keeps doing the exact same thing on restarts, because, well, have to process <i>all</i> the data! The report&#x27;s conclusion about that whole thing is a masterpiece of British understatement:<p>&gt; Clearly a better way to handle this specific logic error would be for FPRSA-R to identify and remove the message and avoid a critical exception.<p>But anyway, without further ado, here&#x27;s how <i>you</i> can avoid having similarly embarrassing 18-page PDFs being dedicated to you -- I call it the blast-door pattern, and it works whenever there is a concept of a &#x27;work item&#x27; anywhere in your code. Trigger warning: C#, but you can probably manage:<p><pre><code>    enum ItemState
    {
        Created,
        Pending,
        Completed,
        Cancelled,
        Failed
    }

    void ItemProcessor(ItemQueue queue, ItemProcessor processor, ItemStore store)
    {
        var item = queue.Dequeue();
        if (item?.State == ItemState.Pending)
        {
            try
            {
                var cts = new CancellationTokenSource();
                cts.CancelAfter(TimeSpan.FromMinutes(5));
                processor.Process(cts);
                item.State = ItemState.Completed;
            }
            catch (OperationCanceledException)
            {
                item.State = ItemState.Cancelled;
            }
            catch (Exception ex)
            {
                item.Exception = ex;
                item.State = ItemState.Failed;
            }
        }
        if (item != null) store.Add(item);
    }
</code></pre>
Now, they key takeaways here are:<p>1. Your state enum has an initial value that does <i>not</i> cause it to be processed: that is only done once the value is explicitly set to Pending. This saves you from &#x27;oops, the item creation did not go entirely right, so it was picked up prematurely by the processor, which then crashed&#x27;;<p>2. Items come from a queue, go back into a different store. You monitor both queue length and the number of items with a bad state in the store; if either number goes up, you have something to look into;<p>3. Set a time limit on processing, which should be entirely ridiculous (so, if item processing normally takes a few milliseconds, make the timeout at least a minute);<p>4. Yes, you catch <i>every</i> exception other than that, because that&#x27;s like the entire point: crashing work items should be set aside to be looked at.<p>You&#x27;ll also want some reliable tooling to reprocess failed items as needed, and <i>one last tip</i> before I go: don&#x27;t wire up individual failed item notifications to your JIRA or something, unless you want to add a &#x27;and then the ticket ID field had an integer overflow&#x27; war story to your arsenal...