text:
<!DOCTYPE HTML>

		<meta http-equiv="Content-Type" content="html/html; charset=utf-8" />
		<title>NSRConfig Class Reference</title>
		<meta id="xcode-display" name="xcode-display" content="render"/>
		<meta name="viewport" content="width=550" />
		<link rel="stylesheet" type="text/css" href="../css/styles.css" media="all" />
		<link rel="stylesheet" type="text/css" media="print" href="../css/stylesPrint.css" />
		<meta name="generator" content="appledoc 2.0.5 (build 789)" />

NSRails

				Dan Hassin

NSRConfig Class Reference

					<button aria-label="Show Table of Contents" role="checkbox" class="open" id="table_of_contents">Table of Contents</button>

					<select id="jumpTo">
	<option value="top">Jump To…</option>

	<option value="overview">Overview</option>

	<option value="tasks">Tasks</option>

	<option value="properties">Properties</option>

	<option value="//api/name/appOAuthToken">    appOAuthToken</option>

	<option value="//api/name/appPassword">    appPassword</option>

	<option value="//api/name/appURL">    appURL</option>

	<option value="//api/name/appUsername">    appUsername</option>

	<option value="//api/name/autoinflectsClassNames">    autoinflectsClassNames</option>

	<option value="//api/name/autoinflectsPropertyNames">    autoinflectsPropertyNames</option>

	<option value="//api/name/dateFormat">    dateFormat</option>

	<option value="//api/name/ignoresClassPrefixes">    ignoresClassPrefixes</option>

	<option value="//api/name/managedObjectContext">    managedObjectContext</option>

	<option value="//api/name/managesNetworkActivityIndicator">    managesNetworkActivityIndicator</option>

	<option value="//api/name/performsCompletionBlocksOnMainThread">    performsCompletionBlocksOnMainThread</option>

	<option value="//api/name/succinctErrorMessages">    succinctErrorMessages</option>

	<option value="//api/name/timeoutInterval">    timeoutInterval</option>

	<option value="//api/name/updateMethod">    updateMethod</option>

	<option value="class_methods">Class Methods</option>

	<option value="//api/name/contextuallyRelevantConfig">    + contextuallyRelevantConfig</option>

	<option value="//api/name/defaultConfig">    + defaultConfig</option>

	<option value="instance_methods">Instance Methods</option>

	<option value="//api/name/dateFromString:">    - dateFromString:</option>

	<option value="//api/name/end">    - end</option>

	<option value="//api/name/initWithAppURL:">    - initWithAppURL:</option>

	<option value="//api/name/stringFromDate:">    - stringFromDate:</option>

	<option value="//api/name/use">    - use</option>

	<option value="//api/name/useAsDefault">    - useAsDefault</option>

	<option value="//api/name/useIn:">    - useIn:</option>

</select>

Overview

Tasks

Properties

Routing

Authentication

Server-side settings

CoreData

Retrieving global configs

Using configs in specific locations

Date conversions

Initializing a config

Properties

appOAuthToken

appPassword

appURL

appUsername

autoinflectsClassNames

autoinflectsPropertyNames

dateFormat

ignoresClassPrefixes

managedObjectContext

managesNetworkActivityIndicator

performsCompletionBlocksOnMainThread

succinctErrorMessages

timeoutInterval

updateMethod

Class Methods

contextuallyRelevantConfig

defaultConfig

Instance Methods

dateFromString:

end

initWithAppURL:

stringFromDate:

use

useAsDefault

useIn:

Index

Hierarchy

NSRConfig Class Reference

	Inherits from
	NSObject

	Conforms to
	NSCoding

	Declared in
	NSRConfig.h

Overview

The NSRails configuration class is NSRConfig, a class that stores your Rails app’s configuration settings (server URL, etc) for either your app globally or in specific instances. It also supports basic HTTP authentication and very simple OAuth authentication.

Universal Config

This should meet the needs of the vast majority of Rails apps. Somewhere in your app setup, set your server URL (and optionally a username and password) using the defaultConfig singleton:

//AppDelegate.m

#import "NSRConfig.h"

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
 [[NSRConfig defaultConfig] setAppURL:@"http://myapp.com"];

 //Similarly, if using HTTP Authentication, you can set your app's username and password like this:
 //[[NSRConfig defaultConfig] setAppUsername:@"username"];
 //[[NSRConfig defaultConfig] setAppPassword:@"password"];
}

Using several configs in one project

When NSRails needs to retrieve configuration settings, it calls the NSRRemoteObject config method. The default behavior is to return contextuallyRelevantConfig, but this can be overridden by your NSRRemoteObject subclass. Simply initialize your own NSRConfig instance and return it, and base URLs, autoinflection, date formats, and any other NSRConfig configurations will be used for NSRails actions called on this class or its instances.

If specific actions must be called using a separate config, an NSRConfig instance can be used to define a context block in which to call those config-specific methods:

  NSRConfig *myConfig = [[NSRConfig alloc] initWithAppURL:@"http://secondrailsapp.com/"];

  [myConfig use];
  NSArray *peopleFromOtherServer = [Person getAllRemote];
  [myConfig end];

  Or, using block notation:

  NSArray *peopleToTransfer = [Person getAllRemote];
  NSRConfig *myConfig = [[NSRConfig alloc] initWithAppURL:@"http://secondrailsapp.com/"];
  [myConfig useIn:^
      {
          for (Person *p in peopleToTransfer)
          {
              [p remoteCreate];
          }
      }
  ];

  In these examples, everything within the blocks will be called using the config context specified, regardless of defaultConfig. The config for the current context can be retrieved using the contextuallyRelevantConfig class method.

  You can nest several config contexts within each other.

Tasks

Properties

		  appURL

Root URL for your Rails server.

	property

		  performsCompletionBlocksOnMainThread

When true, the completion blocks passed into asynchronous remote methods will be called on the main thread.

	property

		  managesNetworkActivityIndicator

The network activity indicator (gray spinning wheel on the status bar) will automatically turn on and off with requests.

	property

		  succinctErrorMessages

Cleaner error messages when generating NSError objects.

	property

		  timeoutInterval

Timeout interval for HTTP requests.

	property

Routing

		  autoinflectsClassNames

When true, all Objective-C class names will have a default equivalence to their under_scored versions.

	property

		  autoinflectsPropertyNames

When true, all Objective-C property names will have a default equivalence to their under_scored versions.

	property

		  ignoresClassPrefixes

When converting class names to their Rails equivalents, prefixes will be omitted.

	property

Authentication

		  appUsername

Username for basic HTTP authentication (if used by server.)

	property

		  appPassword

Password for basic HTTP authentication (if used by server.)

	property

		  appOAuthToken

Token for OAuth authentication (if used by server.)

	property

Server-side settings

		  updateMethod

HTTP method used for updating objects.

	property

		  dateFormat

Date format used if a property of type NSDate is encountered, to encode and decode NSDate objects.

	property

CoreData

		  managedObjectContext

Managed object context for CoreData support. (Required if CoreData is enabled)

	property

Retrieving global configs

		+ defaultConfig

Returns the current default configuration.

		+ contextuallyRelevantConfig

Returns the contextually relevant configuration.

Using configs in specific locations

		– use

Begins a context block of code to use the receiver as the default config.

		– end

Ends a context block of code to use the receiver as the default config.

		– useIn:

Executes a given block with the receiver as the default config in that block.

		– useAsDefault

Specifies for all NSRails actions to use the receiver.

Date conversions

		– stringFromDate:

Returns a string representation of a given date formatted using dateFormat.

		– dateFromString:

Returns a date representation of a given string interpreted using dateFormat.

Initializing a config

		– initWithAppURL:

Initializes a new NSRConfig instance with an app URL.

Properties

appOAuthToken

Token for OAuth authentication (if used by server.)

@property (nonatomic, strong) NSString *appOAuthToken

Declared In

		NSRConfig.h

appPassword

Password for basic HTTP authentication (if used by server.)

@property (nonatomic, strong) NSString *appPassword

Declared In

		NSRConfig.h

appURL

Root URL for your Rails server.

@property (nonatomic, strong) NSString *appURL

Declared In

		NSRConfig.h

appUsername

Username for basic HTTP authentication (if used by server.)

@property (nonatomic, strong) NSString *appUsername

Declared In

		NSRConfig.h

autoinflectsClassNames

When true, all Objective-C class names will have a default equivalence to their under_scored versions.

@property (nonatomic) BOOL autoinflectsClassNames

Discussion

For instance, the class DataType in Objective-C will change to data_type when sending/receiving to/from Rails.

When false, names must be identical to their corresponding models in Rails.

If there are just a few cases where you don’t want this, you can override model names like this.

Default: YES.

Declared In

		NSRConfig.h

autoinflectsPropertyNames

When true, all Objective-C property names will have a default equivalence to their under_scored versions.

@property (nonatomic) BOOL autoinflectsPropertyNames

Discussion

For instance, myProperty in Obj-C will change to my_property when sending/receiving to/from Rails. the_id = theID; the_ids = theIDs.

When false, names must be identical to their corresponding attributes in Rails.

If there are just a few cases where you don’t want this, see NSRRemoteObject’s overridable methods.

Default: YES.

Declared In

		NSRConfig.h

dateFormat

Date format used if a property of type NSDate is encountered, to encode and decode NSDate objects.

@property (nonatomic, strong) NSString *dateFormat

Discussion

This should not be changed unless the format is also changed server-side; the default is the default Rails format.

Default: "yyyy'-'MM'-'dd'T'HH':'mm':'ss'Z'" (Rails default).

Declared In

		NSRConfig.h

ignoresClassPrefixes

When converting class names to their Rails equivalents, prefixes will be omitted.

@property (nonatomic) BOOL ignoresClassPrefixes

Discussion

Example: NSRClass will simply become class, instead of nsr_class.

Default: YES.

Declared In

		NSRConfig.h

managedObjectContext

Managed object context for CoreData support. (Required if CoreData is enabled)

@property (nonatomic, strong) NSManagedObjectContext *managedObjectContext

Discussion

This is the context into which NSRails will insert any new objects created by various internal methods, or search for any existing objects.

Should stay nil (default) if CoreData is not being used.

Declared In

		NSRConfig.h

managesNetworkActivityIndicator

The network activity indicator (gray spinning wheel on the status bar) will automatically turn on and off with requests.

@property (nonatomic) BOOL managesNetworkActivityIndicator

Discussion

This is only supported for asynchronous requests, as otherwise the main thread is blocked.

Also only supported in iOS development.

Default: NO.

Declared In

		NSRConfig.h

performsCompletionBlocksOnMainThread

When true, the completion blocks passed into asynchronous remote methods will be called on the main thread.

@property (nonatomic) BOOL performsCompletionBlocksOnMainThread

Discussion

This can be useful if you wish to update view elements from this block (where iOS would otherwise lock up).

Not sure when this couldn’t be useful, but leaving disabling it as an option. Maybe performance?

Default: YES.

Declared In

		NSRConfig.h

succinctErrorMessages

Cleaner error messages when generating NSError objects.

@property (nonatomic) BOOL succinctErrorMessages

Discussion

For example, simply Couldn't find Person with id=15, instead of this mess.

May not be effective with non-Rails servers.

Default: YES.

Declared In

		NSRConfig.h

timeoutInterval

Timeout interval for HTTP requests.

@property (nonatomic) NSTimeInterval timeoutInterval

Discussion

The minimum timeout interval for POST requests is 240 seconds (set by Apple).

Default: 60.

Declared In

		NSRConfig.h

updateMethod

HTTP method used for updating objects.

@property (nonatomic, strong) NSString *updateMethod

Discussion

Rails is currently at 3.2.3 and using PUT, but 4.0 will use PATCH by default.

Default: @"PUT"

Warning: When Rails 4.0 is released, this default value will be changed to @"PATCH".

Declared In

		NSRConfig.h

Class Methods

contextuallyRelevantConfig

Returns the contextually relevant configuration.

+ (NSRConfig *)contextuallyRelevantConfig

Return Value

The contextually relevant configuration, or default configuration is no explicit context is set.

Discussion

This is like defaultConfig, but will return any configuration currently being used in a use/end or useIn: block before going to the default.

Declared In

		NSRConfig.h

defaultConfig

Returns the current default configuration.

+ (NSRConfig *)defaultConfig

Return Value

The default configuration.

Declared In

		NSRConfig.h

Instance Methods

dateFromString:

Returns a date representation of a given string interpreted using dateFormat.

- (NSDate *)dateFromString:(NSString *)string

Parameters

string

The string to parse.

Discussion

This method is used internally to convert a Rails-sent datetime string representation into an NSDate for an object property.

See Also

– stringFromDate:.

@property dateFormat.

Declared In

		NSRConfig.h

end

Ends a context block of code to use the receiver as the default config.

- (void)end

Discussion

Highest config precedence.

See Also

– use.

Declared In

		NSRConfig.h

initWithAppURL:

Initializes a new NSRConfig instance with an app URL.

- (id)initWithAppURL:(NSString *)url

Parameters

url

App URL to be set to the new instance.

Declared In

		NSRConfig.h

stringFromDate:

Returns a string representation of a given date formatted using dateFormat.

- (NSString *)stringFromDate:(NSDate *)date

Parameters

date

The date to format.

Discussion

This method is used internally to convert an NSDate property in an object to a Rails-readable string.

See Also

– dateFromString:.

@property dateFormat.

Declared In

		NSRConfig.h

use

Begins a context block of code to use the receiver as the default config.

- (void)use

Discussion

Highest config precedence.

See Also

– end.

Declared In

		NSRConfig.h

useAsDefault

Specifies for all NSRails actions to use the receiver.

- (void)useAsDefault

Discussion

defaultConfig will now return the receiver.

Declared In

		NSRConfig.h

useIn:

Executes a given block with the receiver as the default config in that block.

- (void)useIn:(void ( ^ ) ( void ))block

Parameters

block

Block to be executed with the default config context of receiver.

See Also

– use.

– end.

Declared In

		NSRConfig.h

Index

Hierarchy

© 2012 Dan Hassin. All rights reserved. (Last updated: 2012-07-23)

						Generated by appledoc 2.0.5 (build 789).


Construction Apparel | Garden State Highway Products, Inc.
Safety apparel that ensures maximum visibility when it matters most.
With our high visibility safety apparel from ML Kishigo, others can see you while working in poor weather and low light conditions.
Whether you're looking for construction safety apparel, equipment or safety gear for other jobs, we have a wide selection. Select from a wide variety of jackets, high visibility t-shirts, sweatshirts, eyewear, gloves, rain gear, vests and more! Even when you're covered from head to toe in dust and dirt your clothes can still do the talking.
So go ahead, we're giving you permission to be the best-dressed guy or gal on site. Check out our premium black series where raising the bar, meets style. Custom imprinting available on most styles.
What garments meet the specified High Visibility requirements?
Only garments certified and labeled as ANSI/ISEA 107 and ANSI/ISEA 207, or in some cases certain firefighter standards, meet standard specifications for high visibility personal protective safety apparel in the U.S. On or near roadways, performance Class 2 or 3 ANSI/ISEA 107 garments are required for compliance with the worker high visibility regulation in the Federal Highway Administration (FHWA) Manual on Uniform Traffic Control Devices (MUTCD) Section 6D.03. The FHWA recognizes ANSI/ISEA 107 and 207 revisions starting in 2004 and going through the current ANSI/ISEA 107-2015 revision. To find out more on high-visibility warning garments by construction workers, visit OSHA requirements at https://www.osha.gov.
According to the U.S. Bureau of Labor Statistics (BLS), "The most common event associated with fatal occupational injuries incurred at a road construction site was worker struck by vehicle, mobile equipment. Of the 639 total fatal occupational injuries at road construction sites during the 2003–07 period, 305 were due to a worker being struck by a vehicle or mobile equipment." The BLS article further reports that more workers are struck and killed by construction equipment (38 percent) than by cars, vans and tractor-trailers (33 percent). As such, work zone "runovers" and "backovers" are clearly the greatest hazard to roadway construction workers and, by far, the leading cause of death.
The use of high visibility clothing is one important strategy in reducing the number of "struck-by" deaths on road construction sites.


text:
Interview: Fedde Le Grand
By Chris Barker 2011-11-18T10:47:00.321Z
Fedde in action at Ministry Of Sound, London
His name became synonymous with the stop, drop electro-house sound of 2006, until a change in style saw him emerge from the underground in to a fully rounded, superstar DJ.
He recently supported Coldplay as part of their world tour after being asked to remix their smash hit Paradise and continues to play the biggest venues around the globe. We caught up with after his recent set at Ministry Of Sound, London.
It feels like you resisted the temptation to continue solely on the back of Put Your Hands Up and Let Me Think About It… and headed back to the underground club style with your sound. Tell us about this decision and what it was like at the time, did you actively turn your back on the radio?
"It wasn't something I did actively, I never make a conscious decision to do my music one way or another, I just make what feels right for me at the time. If a more commercial sound had been buzzing round my head after Let Me Think About It then that's what I would have gone with, but it wasn't, so I didn't."
What have been the main changes in the way you produce now?
"The main change has been my new studio which I installed at the beginning of this year. I've built a totally perfect acoustic studio that gives me more control of the sound and it's such an inspirational place for me. I'm getting closer to the ideal mix every day."
Tell us about the process in creating the Coldplay remix.
"I use Logic Pro. Before I start with a remix I make sure I have a certain idea where I want to go. With this one I wanted to have the main hook but still make it danceable."
Do you have a special master plug-in chain?
"I'm a big fan of the Sonoxx Oxford range - they have some quality plug-ins including limiters, compressors and everything I need. Plus, they're really easy for me to use. I also like UAD, I get a really warm analog sound from using them which sounds so much better and gets the rounded effects that I want in my music."
What are your three favorite plug-in synths/ effects right now, and why?
"1. Sylenth: it's highly editable; it's got an easy interface and good sound quality
"2. Nexus: not so editable but has amazing useful presets
"3. NI Massive: super highly editable, pretty much makes any sound you have in mind, not the easiest interface but worth working on it."
What's on your DJ rider right now, including tech!
I don't really need anything too outlandish, just four CDJs and a Pioneer DJM 800 mixer. And enough Barcardi and coke to last the night.
Are you going down the DJ 'show' route, like SHM, Deadmau5, Mason etc, with lights / visuals / pyros?
I've definitely upped things as far as putting visuals and VJ shows into my own Takeover parties go. I think it's important to give clubbers a full experience when they come to see you at a showcase party, to really push the boat out so that they walk away buzzing about everything that they've encountered on the dance floor. Having said that, for me the most important thing is always the music; as much as lights and visuals and pyros can add a certain 'wow' factor, it's the music that has to blow people away. Sometimes the best nights don't have the best production but the music just blows your mind and you walk away feeling like you've been lifted to another level.
What are your biggest tracks right now on the dance floor and what producers are you most impressed by from a technical point of view?
My own private Faithless - Insomnia edit, my Coldplay remix of 'Paradise' and a new track by Deniz Koyu. For me still most amazing producer soundwise is Eric Prydz, what he does in terms of production would take a lifetime to learn but it just seems to come so naturally to him, he's the producer's producer.
Do you think it's easier or harder to break through now as an up and coming producer/DJ? Got and advice for people trying to break through?
"I think it's definitely harder, for quite a few reasons. There's a LOT of competition out there. I'm not saying that everyone wants to be a DJ, and it's a good job otherwise there would be no-one on the dance floor, but if you're starting out and trying to get a name for yourself there are a lot of other people in the same spot, trying to get the same gigs. And I think the reason for that is that we've moved into a digital era, meaning it's a lot easier to buy tunes, it's far more convenient. In the 'old days' of vinyl, you had to get out there, get into the record shops, get hold of the records and the promos before they sold out or stopped getting pressed up, it almost had to be something of an obsession.
"These days you can just log in, download as many tunes as you like without leaving your chair. It's opened up mixing to many more people, which can be a good thing or a bad thing, depending on how you view it. To anyone that wants to break through, my advice would be to work as hard as you can and get yourself out there talking to promoters, agents, management companies. And be prepared to work your way up from the bottom. Don't pass up a gig just because you're not getting paid for it or you're at the bottom of the bill - that's the way that people hear your sound, that you get a following, that other promoters get to hear about you."
What's been the hardest decision you've had to make in your career since you started?
"Sometimes you have to acknowledge the fact that some people you work with and you have a good bond with, you have to let go so that you can move on yourself. It's a hard decision to make but there comes a time when you need to be, not calculating I think but put your career head on and recognize that to go further you have to take a different road than other people. Thank God I don't have to make that decision too often."
"I've got some more tunes in the pipeline, I've got a tour of the States and Canada coming up in the middle of December and I'm also playing in LA for NYE which will be a big one. I'm taking a bit of a break after New Year just because if I don't, I think I'll collapse! And of course I'm also thinking about a new album so hopefully I'll get to schedule that in some time soon."
Catch up with all the latest Fedde news at his website: www.feddelegrand.com

redpajama_set_name: RedPajamaCommonCrawl


Born in Baltimore, MD; married; children: three. Education: Peabody Conservatory at Johns Hopkins University, B.A., M.A.
Opera singer with the Hartford Opera Company, Hartford County, MD, Baltimore Opera, Baltimore, MD, and Washington Opera, Washington, DC; worked in public relations and for a music academy in PA.
(As Libby Sternberg) Uncovering Sadie's Secrets: A Bianca Balducci Mystery (young adult novel), Bancroft Press (Baltimore, MD), 2003.
(As Libby Sternberg) Finding the Forger: A Bianca Balducci Mystery (young adult novel), Bancroft Press (Baltimore, MD), 2004.
Loves Me, Loves Me Not (novel), Red Dress Ink (Don Mills, Ontario, Canada), 2005.
Libby Malin first wrote under her married name, Libby Sternberg, when she penned her "Bianca Balducci" mysteries for young adults. The first, Uncovering Sadie's Secrets: A Bianca Balducci Mystery, finds the protagonist, high school sophomore Bianca, pursuing Doug and befriending the new girl, Sadie, whose life is complicated by two stalkers. Bianca, whose older sister is an actual private investigator, delves into Sadie's past as she attempts to unravel the mystery. "While Sternberg emphasizes mystery and action elements, she doesn't ignore deeper issues," observed a Kirkus Reviews writer about the novel.
In Finding the Forger: A Bianca Balducci Mystery, Bianca deals with jealousy when her best friend, Kerrie, seems to be spending an inappropriate amount of time with Doug. She then meets Neville, the British son of a board member of the local art museum, where art is being replaced with forgeries. Reviewers particularly praised the character of Bianca. For instance, School Library Journal contributor Michelle Capozzella wrote: "The author does a great job of capturing her voice." Ilene Cooper in Booklist noted "Bianca's first-person voice—droll, yet totally teen." A Kirkus Reviews contributor commented that Sternberg includes "a couple of nice red herrings along with some action sequences that make the story satisfying."
Loves Me, Loves Me Not is the author's first book under her maiden name, Libby Malin, and is written for adult readers. Amy Sheldon's fiancé, Rick, was killed in a car accident for which she feels responsible, and Amy devotes herself to managing a flower shop to cope with her grief. One of her clients, Henry Castle, comes into the shop, upset because an order for one of his many women was improperly filled. Henry, who is described as a "bed-hopping Latino," energizes her love life, but Amy is worried that he may not be right for her. Booklist critic Kristine Huntley felt that Malin's "clever debut" in the chick-lit genre "offers quite a few surprises along the way." "Lightweight but cute, Malin's plot moves along steadily enough toward its inevitable happy ending," wrote a Publishers Weekly reviewer.
Malin told CA: "I always loved writing and wrote short stories (none of them published) for many years before seriously applying myself to trying to get my work published. I think every writer's life and world-view provide influences for his or her work and I'm no exception. I tend to be optimistic so even my gloomiest manuscripts will have a ray of sunshine at the end. I set myself a daily page quota when I'm trying to finish a manuscript and usually proceed through the novel in a linear fashion, sketching out some ideas first in either a bullet-form outline (very rough) or a narrative synopsis. I sometimes jump ahead to write critical scenes or scenes that are so vibrant in my head I can't not write them down.
"The business of writing has surprised me. I was one of those writers who imagined that once you sold your first book, you had 'arrived.' Little did I know how steep the hill to true writing success really is! Learning about building audiences and reaching out to readers who might enjoy my work has been a challenge. I've also found it challenging to determine what editors are looking for and how to persuade them that my book is what they should buy! My favorite book is probably one that has yet to be published, a retelling of the Jane Eyre story.
"I hope readers will come away from my books feeling less lonely, feeling there are others out there who have shared in their experiences and have found love or happiness or redemption along the way."
Booklist, January 1, 2003, Sally Estes, review of Uncovering Sadie's Secrets: A Bianca Balducci Mystery, p. 872; February 1, 2005, Ilene Cooper, review of Finding the Forger: A Bianca Balducci Mystery, p. 955; October 15, 2005, Kritine Huntley, review of Loves Me, Loves Me Not, p. 35.
Kirkus Reviews, December 15, 2002, review of Uncovering Sadie's Secrets, p. 1857; December 15, 2004, review of Finding the Forger, p. 1209.
Publishers Weekly, August 15, 2005, review of Loves Me, Loves Me Not, p. 31.
School Library Journal, March 23, 2003, Delia Fritz, review of Uncovering Sadie's Secrets, p. 240; April, 2005, Michelle Capozzella, review of Finding the Forger, p. 143.
Libby Malin Home Page,http://www.libbymalin.com (July 22, 2006).
Libby Sternberg Home Page,http://www.libbysternberg.com (July 22, 2006).
"Malin, Libby." Contemporary Authors. . Encyclopedia.com. 20 Apr. 2019 <https://www.encyclopedia.com>.


text:
\section{Additional Results}
\label{sec:additional-results}

Related to Table \ref{tab:results-pi}, Table \ref{tab:results-non-aggregated}
shows the evaluation of non-aggregated PIs (not accounting for epistemic
uncertainty) for the QD+ and QD models.

\begin{table}[h]

  \scriptsize
  \centering
  \caption{Evaluation of non-aggregated PIs (2-layer NNs).}
  \label{tab:results-non-aggregated}
  \vskip 1.em
  \setlength\tabcolsep{4pt}
  \begin{tabular}{l|*2c|*2c}
    \toprule
    \multirow{2}{*}{\textbf{Dataset}}
    & \multicolumn{2}{c|}{\textbf{PICP}}
    & \multicolumn{2}{c}{\textbf{MPIW}} \\

    & QD+ & QD
    & QD+ & QD \\
    \midrule
    Boston
    & $0.85\pm0.01$ & $0.88\pm0.01$
    & $1.22\pm0.04$ & $1.12\pm0.02$ \\
    Concrete
    & $0.86\pm0.01$ & $0.89\pm0.00$
    & $0.79\pm0.03$ & $1.03\pm0.01$ \\
    Energy
    & $0.90\pm0.01$ & $0.93\pm0.00$
    & $0.21\pm0.01$ & $0.27\pm0.01$ \\
    Kin8nm
    & $0.91\pm0.00$ & $0.92\pm0.00$
    & $0.95\pm0.01$ & $1.01\pm0.01$ \\
    Naval
    & $0.98\pm0.00$ & $0.94\pm0.00$
    & $0.08\pm0.00$ & $0.25\pm0.00$ \\
    Power
    & $0.94\pm0.00$ & $0.93\pm0.00$
    & $0.79\pm0.00$ & $0.75\pm0.00$ \\
    Protein
    & $0.93\pm0.00$ & $0.93\pm0.00$
    & $2.03\pm0.01$ & $2.11\pm0.01$ \\
    Wine
    & $0.91\pm0.00$ & $0.91\pm0.00$
    & $2.31\pm0.03$ & $1.81\pm0.01$ \\
    Yacht
    & $0.76\pm0.01$ & $0.89\pm0.01$
    & $0.07\pm0.00$ & $0.15\pm0.01$ \\
    Year
    & $0.93\pm0.00$ & $0.93\pm0.00$
    & $2.27\pm0.00$ & $2.07\pm0.02$ \\
    \bottomrule
  \end{tabular}
  \vskip 1.em
\end{table}

In Table \ref{tab:qde-results}, we show the reproduced results of the original
QDE method following the exact experimental setting as implemented in
\citeauthor{pearce:2018} \yrcite{pearce:2018}. These results are included to
offer a comparison between the 1-layer and 2-layer models (Table
\ref{tab:qde-results} and Table \ref{tab:results-pi}, respectively). It also
serves as a verification of our re-implementation of the method by
\citet{pearce:2018}.

\begin{table}[h]

  \scriptsize
  \centering
  \caption{Original QDE with 1-layer NNs as a reference.}
  \label{tab:qde-results}
  \vskip 1.em
  \setlength\tabcolsep{4pt}
  \begin{tabular}{l|*2c|*2c}
    \toprule
    \multirow{2}{*}{\textbf{Dataset}}
    & \multicolumn{2}{c|}{\textbf{PICP}}
    & \multicolumn{2}{c}{\textbf{MPIW}} \\

    & SEM-QD & QD
    & SEM-QD & QD \\
    \midrule
    Boston
    & $0.90\pm0.01$ & $0.80\pm0.01$
    & $1.01\pm0.01$ & $0.81\pm0.01$ \\
    Concrete
    & $0.92\pm0.01$ & $0.84\pm0.00$
    & $1.01\pm0.01$ & $0.83\pm0.00$ \\
    Energy
    & $0.97\pm0.00$ & $0.91\pm0.00$
    & $0.45\pm0.01$ & $0.38\pm0.01$ \\
    Kin8nm
    & $0.96\pm0.00$ & $0.89\pm0.00$
    & $1.24\pm0.00$ & $1.02\pm0.00$ \\
    Naval
    & $0.99\pm0.00$ & $0.96\pm0.00$
    & $0.25\pm0.02$ & $0.17\pm0.00$ \\
    Power
    & $0.95\pm0.00$ & $0.94\pm0.00$
    & $0.85\pm0.00$ & $0.81\pm0.00$ \\
    Protein
    & $0.95\pm0.00$ & $0.94\pm0.00$
    & $2.26\pm0.00$ & $2.17\pm0.00$ \\
    Wine
    & $0.93\pm0.01$ & $0.91\pm0.00$
    & $2.31\pm0.02$ & $2.04\pm0.01$ \\
    Yacht
    & $0.95\pm0.01$ & $0.86\pm0.01$
    & $0.15\pm0.00$ & $0.10\pm0.00$ \\
    Year
    & $0.95\pm\text{NA}$ & $0.93\pm0.00$
    & $2.41\pm\text{NA}$ & $2.22\pm0.01$ \\
    \bottomrule
  \end{tabular}

\end{table}

Table \ref{tab:dataset-and-sem-qd-pe} considers a simple baseline with point estimates constructed as the mean of PIs from SEM-QD (annotated SEM-QD*). This simple baseline is underperforming compared to the results in Table \ref{tab:results-pe}.

\begin{table}[h]

  \scriptsize
  \centering
  \caption{Extension of Table \ref{tab:results-pe} provides a simple baseline with point estimates constructed as the mean of PIs from SEM-QD (annotated SEM-QD*). Also, the sizes $|\mathcal{D}|$ and input dimensions $d$ of the datasets are presented.}
  \label{tab:dataset-and-sem-qd-pe}
  \vskip 1.em
  \setlength\tabcolsep{4pt}
  \begin{tabular}{l*2c|*1c}
    \toprule
    \multicolumn{3}{c|}{\textbf{Dataset}}
    & \multicolumn{1}{c}{\textbf{MSE}} \\

    Name
    & $|\mathcal{D}|$ & $d$
    & SEM-QD* \\
    \midrule
    Boston
    & $506$ & $13$
    & $0.209\pm0.030$ \\
    Concrete
    & $1030$ & $8$
    & $0.102\pm0.005$ \\
    Energy
    & $768$ & $8$
    & $0.028\pm0.008$ \\
    Kin8nm
    & $8192$ & $8$
    & $0.067\pm0.001$ \\
    Naval
    & $11934$ & $16$
    & $0.012\pm0.001$ \\
    Power
    & $9568$ & $4$
    & $0.054\pm0.001$ \\
    Protein
    & $45730$ & $9$
    & $0.666\pm0.004$ \\
    Wine
    & $1599$ & $11$
    & $0.804\pm0.021$ \\
    Yacht
    & $308$ & $6$
    & $0.003\pm0.001$ \\
    Year
    & $515345$ & $90$
    & $0.686\pm\text{NA}$ \\
    \bottomrule
  \end{tabular}

\end{table}

\section{Robustness}
\label{sec:robustness}

The robustness of training process of QD+ vs. the fragility of training process of QD is shown in Table \ref{tab:robustness}. The contribution of the penalty function $\mathcal{L}_{P}$ to the robustness is shown in Figure \ref{fig:sensitivity-analysis} of the following section.

\begin{table}[h]

  \scriptsize
  \centering
  \caption{Extension of Tables \ref{tab:results-pi} and \ref{tab:results-pe} showing the number of failed/repeated training attempts of QD+ and QD. The targeted model count equals to an ensemble size $m$ times number of trials $t$.}
  \label{tab:robustness}
  \vskip 1.em
  \setlength\tabcolsep{6pt}
  \begin{tabular}{l|*2c|*1c}
    \toprule
    \multirow{2}{*}{\textbf{Dataset}}
    & \multicolumn{2}{c|}{\textbf{Failures/Retries}}
    & \multicolumn{1}{c}{\textbf{Target Model Count}} \\

    & QD+ & QD
    & $m \cdot t$ \\
    \midrule
    Boston
    & $0$ & $3$
    & $5 \cdot 20$ \\
    Concrete
    & $0$ & $2$
    & $5 \cdot 20$ \\
    Energy
    & $0$ & $22$
    & $5 \cdot 20$ \\
    Kin8nm
    & $0$ & $2$
    & $5 \cdot 20$ \\
    Naval
    & $0$ & $10$
    & $5 \cdot 20$ \\
    Power
    & $0$ & $6$
    & $5 \cdot 20$ \\
    Protein
    & $0$ & $22$
    & $5 \cdot 5$ \\
    Wine
    & $0$ & $29$
    & $5 \cdot 20$ \\
    Yacht
    & $0$ & $33$
    & $5 \cdot 20$ \\
    Year
    & $0$ & $18$
    & $5 \cdot 1$ \\
    \bottomrule
  \end{tabular}

\end{table}

\section{Sensitivity Analysis}
\label{sec:sensitivity-analysis}

Figure \ref{fig:sensitivity-analysis} provides a sensitivity analysis with respect to hyper-parameters $\lambda_1$, $\lambda_2$ and $\xi$. It visualizes the measures PICP, NMPIW and MSE in relation to changing hyper-parameters of the QD+ model. The analysis was performed on the Yacht dataset.
The sensitivity analysis is also an ablation study since the border values, i.e. $\lambda_1, \lambda_2 \in \{0, 1\}$ and $\xi = 0$, disable certain parts of the $\mathcal{L}_{QD+}$ loss function (eq. \ref{eq:loss-qd-plus}).

Figure \ref{fig:with-penalty-function} would guide us to choose $\lambda_1$ around $0.975$ and $\lambda_2$ around $0.05$. Leaving out the penalty function from the loss function (i.e. $\xi = 0$) leads to a non-converging training process for $\lambda_2$ below approx. $0.7$ and more noisy measures in the remaining hyper-parameter space (Figure \ref{fig:without-penalty-function}). It demonstrates the critical importance of the penalty function $\mathcal{L}_{P}$ (eq. \ref{eq:penalty-function}), thus, supporting the claims about the robustness together with the evidence in Table \ref{tab:robustness}.

\begin{figure*}[p]
  \begin{subfigure}[t]{.5\textwidth}
    \vskip 0.2in
    \begin{center}
    \centerline{\includegraphics[width=.9\columnwidth]{sensitivity_analysis_ksi_10_vectorized}}
    \caption{$\xi = 10$}
    \label{fig:with-penalty-function}
    \end{center}
    \vskip -0.2in
  \end{subfigure}
  \begin{subfigure}[t]{.5\textwidth}
    \vskip 0.2in
    \begin{center}
    \centerline{\includegraphics[width=0.9\columnwidth]{sensitivity_analysis_ksi_0_vectorized}}
    \caption{$\xi = 0$}
    \label{fig:without-penalty-function}
    \end{center}
    \vskip -0.2in
  \end{subfigure}
  \caption{Sensitivity analysis of QD+ on Yacht dataset with respect to hyper-parameters $\lambda_1$ and $\lambda_2$ with (\subref{fig:with-penalty-function}) and without (\subref{fig:without-penalty-function}) an active penalty function $\mathcal{L}_{P}$. Gray color stands for a failed training process, i.e. high loss or outputs violating the semantic integrity (PI crossing or point estimates
outside the PI bounds).}
  \label{fig:sensitivity-analysis}
\end{figure*}

\end{appendices}

\section{INTRODUCTION}
\label{sec:introduction}

Quantifying predictive uncertainty of machine learning models is crucial in applications, e.g., mission-critical systems, where it is essential to know when the model is not able to provide accurate predictions.
In decision support systems, providing the human with an additional information
about the uncertainty of a prediction may decrease the response time and
increase the accuracy of an action. It can also positively contribute in
building trust and understanding towards the machine learning system and its correct facilitation.
In this work, we focus on the quantification of predictive uncertainty for
the regression task, specifically in the form of prediction intervals which have
probabilistic interpretation and which are interpretable for both humans and
machines.

Prediction interval (PI) is an estimate representing predictive uncertainty in
the form of two values between which a future observation will fall with a
certain probability.
Well-calibrated/high-quality prediction intervals are as narrow as possible
while attaining the desired coverage probability.

\begin{figure}[t]
  \vskip 0.2in
  \begin{center}
  \centerline{\includegraphics[width=\columnwidth]{toy_example_2_vectorized}}
  \caption{A toy example demonstrating our method for generating PIs from a split normal mixture of quality-driven deep ensembles. The final PIs are accounting for aleatoric and epistemic uncertainty. The synthetic dataset is a sinusoid with Gaussian noise and sparsified samples. The magnitude of Gaussian noise and sparsification is increasing with distance from the center ($x=0$).}
  \label{fig:toy-example}
  \end{center}
  \vskip -0.2in
\end{figure}

While neural networks (NNs) are powerful function approximators, they are poor at representing
predictive uncertainty. Previously, methods adopting a Bayesian approach in the
context of neural networks \citetext{e.g., \citealp{graves:2011, blundell:2015,
hernandez:2015, krueger:2017, louizos:2017, pawlowski:2017, wu:2019, izmailov:2019}} represented the
state-of-the-art for providing predictive uncertainty estimates. See \citeauthor{yao:2019} \yrcite{yao:2019} for an overview. Recently, a number of non-Bayesian yet probabilistic approaches provide competitive predictive
uncertainty estimates~\cite{lakshminarayanan:2017, pearce:2018,
tagasovska:2019}. In this paper, we focus on and extend the latter branch of research.

Our work builds on the findings of \citeauthor{pearce:2018}
\citeyearpar{pearce:2018} which is inspired by works of
\citeauthor{khosravi:2011} \citeyearpar{khosravi:2011} and
\citeauthor{lakshminarayanan:2017} \citeyearpar{lakshminarayanan:2017}.
\citeauthor{pearce:2018} \citeyearpar{pearce:2018} presented a method based on
aggregating ensembled PIs from a set of NNs optimized with respect to a so-called
quality-driven loss function (see Section~\ref{sec:original-quality-driven-ensembles}).

Although the aforementioned work has achieved promising results of well-calibrated PIs, it has several limitations that need to be addressed. In our continuation, we primarily address three main limitations of the state-of-the-art method: (1) The inability to generate point estimates, (2) the theoretically weakly justified method for aggregation of ensembled PIs, and (3) the fragile training process. With this in mind, the main contributions of this work can be summarized as follows:

\begin{itemize}
  \item{
  We retrofit the quality-driven loss function:
  \begin{itemize}
    \item{
    With a point estimate loss (particularly MSE or mean  squared error) to be
    able to draw point estimates and PIs from the same generative distribution.
    }
    \item{
    With a penalty function adding a constraint and thus enforcing the integrity of
    the results, i.e. avoiding crossing of PI boundaries or point estimates out
    of PI bounds, and consequently stabilizing/strengthening the training process.
    }
  \end{itemize}
  }
  \item{
  We propose a new aggregation method for ensembles of PIs with
  point estimates. It is fitting a split normal mixture~\cite{wallis:2014} providing tighter and theoretically well-founded aggregates of PIs.
  }
  \item{
  We propose an analytical approach to parameter initialization for the
  parameter fitting process of a split normal probability density function,
  thus increasing the success of fitting and accelerating it compared to
  random initialization.
  }
\end{itemize}

In addition to the above, we provide important insights and guidelines for hyper-parameter
search contributing to reproducibility and reliable model fitting, and we
suggest directions for future research.

\section{BACKGROUND}
\label{sec:background}

In this section, we formally introduce the predictive uncertainty and
prediction intervals, we consider methods for quantifying predictive uncertainty
in NNs, especially in the form of prediction intervals, and we provide insights into the work of \citeauthor{pearce:2018} \yrcite{pearce:2018} which we build on.

\subsection{Predictive Uncertainty}
\label{sec:predictive-uncertainty}

The sources of predictive uncertainty can be categorized into aleatoric and
epistemic uncertainty~\cite{kiureghian:2009}.

The aleatoric or aleatory or data uncertainty is also known as the irreducible
uncertainty, i.e. it can not be reduced either through model or data. It arises
from an inherent and irreducible data noise. The aleatoric uncertainty can be
captured by learning the conditional distribution between the target and the input variables.
The aleatoric uncertainty can further be characterized as homoskedastic (homoscedastic) if the irreducible noise is constant across random variables or as heteroskedastic (heteroscedastic) if contrary.

In contrast to aleatoric uncertainty, the epistemic uncertainty is reducible, and it can be further decomposed into model uncertainty and distributional uncertainty.
Model uncertainty can be caused by model bias or parameter uncertainty due to
insufficient data. Distributional uncertainty can be caused by the mismatch
between training and test set~\cite{malinin:2018}, and it is often described
as a part of the model uncertainty.

Generally, the aim is to reduce epistemic uncertainties. However, constraints
emerging from model or data usually do not allow it.

Please note that predictive uncertainty is a broad concept, and the literature is
inconsistent in the terminology. We follow the taxonomy found in
\cite{kiureghian:2009, pearce:2018, malinin:2018}.

\subsection{Prediction Intervals}
\label{sec:prediction-intervals}

Given an input $\mathbf{x}^{(i)}$, a prediction interval $[\hat{y}_{L}^{(i)}, \hat{y}_{U}^{(i)}]$ of a sample $i$ captures the future observation (target variable) $y^{(i)}$ with the probability equal or greater than $\gamma \in [0, 1]$ (eq.~\ref{eq:prediction-interval}). The value of $\gamma$ is commonly set to $0.95$ or $0.99$. Common in the literature is an alternative notation with $\alpha$.
\begin{equation}
  \label{eq:prediction-interval}
  \operatorname{Pr}\left(\hat{y}_{L}^{(i)} \leq y^{(i)} \leq \hat{y}_{U}^{(i)}\right) \geq \gamma = (1-\alpha)
\end{equation}
Given $n$ samples, the quality of the generated prediction intervals is assessed
by measuring the prediction interval coverage probability (PICP)
\begin{equation*}
  \label{eq:picp}
  \mathit{PICP} = \frac{c}{n}
\end{equation*}
where
\begin{equation}
  \label{eq:c}
  c = \sum_{i=1}^{n} k_i
\end{equation}
for
\begin{equation}
  \label{eq:k_i}
  k_i =
  \begin{cases}
    1 & \text{if}\ \hat{y}_{L}^{(i)} \leq y^{(i)} \leq \hat{y}_{U}^{(i)}, \\
    0 & \text{otherwise},
  \end{cases}
\end{equation}
and by measuring the mean prediction interval width (MPIW)
\begin{equation*}
  \label{eq:mpiw}
  \mathit{MPIW} = \frac{1}{n}\sum_{i=1}^{n} \hat{y}_{U}^{(i)} - \hat{y}_{L}^{(i)}
\end{equation*}
or its normalized version NMPIW
\begin{equation}
  \label{eq:nmpiw}
  \mathit{NMPIW} = \frac{\mathit{MPIW}}{r}
\end{equation}
where $r = max(\vectorsym{y}) - min(\vectorsym{y})$. Please note that MPIW assumes $\hat{y}_{U}^{(i)} \geq \hat{y}_{L}^{(i)}$, i.e. no crossing of PI bounds. Ignoring this fact can lead to misleading results.

It is desired to achieve $\mathit{PICP} \geq \gamma$ while having MPIW as small
as possible.

\subsection{Related Work}
\label{sec:related-work}

Following is a concise exploration of methods based on NNs capable of quantifying predictive uncertainty in a regression analysis. Methods adopting the Bayesian
approach (such as BNN or Bayesian neural networks) are a prominent group \citetext{\citealp{graves:2011, blundell:2015, hernandez:2015, krueger:2017,
louizos:2017, pawlowski:2017, wu:2019, izmailov:2019}; etc.} and represented for long the
state-of-the-art to estimate predictive uncertainty. We explore alternative
non-Bayesian yet probabilistic methods representing competitive and possibly a
state-of-the-art alternative.

Interpreting Monte Carlo dropout (MC-dropout) at test time as approximate
Bayesian inference~\cite{gal:2016} has been a widely used method to quantify
predictive uncertainty, mainly due to its scalability and simplicity.
Interpretation of MC-dropout as an ensemble model combination motivated
consecutive research on ensemble-based methods which have been shown to be
superior in generating predictive uncertainty to MC-dropout or even
Bayesian methods~\cite{lakshminarayanan:2017}. Recently, it has been shown that
deep ensembles with random initialization may explore different modes in
function space, and therefore perform well in exploring model uncertainty. That
is in contrast to subspace sampling methods (e.g. MC-dropout, weight
averaging) which may generate a set of diverse functions but still in the vicinity of
the starting point and thus generating insufficiently diverse predictions
\cite{fort:2019}. Also recently, the robustness of uncertainty quantification
methods under dataset shift was investigated, and although all compared methods
have been deceptible to increasing dataset shifts, deep ensembles have shown the
greatest robustness~\cite{ovadia:2019}.

More recently, an alternative line of work employs a loss function to
optimize for generating specifically prediction intervals.
A method based on quantile regression is utilizing the so-called pinball
loss function~\cite{koenker:2005} to optimize a single NN for the desired PIs
and so-called Orthonormal Certificates to account for epistemic uncertainty
\cite{tagasovska:2019}.
A method called LUBE~\cite{khosravi:2011} constructs prediction intervals using
quality-driven loss function utilizing quality metrics PICP and NMPIW. LUBE
applies simulated annealing for training a NN with respect to the quality-driven
loss function. Inspired by \citeauthor{khosravi:2011} \yrcite{khosravi:2011} and
\citeauthor{lakshminarayanan:2017} \yrcite{lakshminarayanan:2017}, a slightly
modified quality-driven loss function optimized for gradient descent was proposed
\cite{pearce:2018} and extended with ensembles to account for epistemic
uncertainty. Although presenting favorable results, both methods deliver only
prediction intervals without point estimates.

\subsection[Quality-Driven Ensembles by Pearce et al. (2018)]{Quality-Driven Ensembles by \mbox{\citeauthor{pearce:2018} \yrcite{pearce:2018}}}
\label{sec:original-quality-driven-ensembles}

Our work builds on~\cite{pearce:2018} henceforth referred to as the original
quality-driven ensembles or shortly original QDE (in results annotated as SEM-QD). First, the quality-driven loss function is described (annotated as QD). Second, the aggregation method is described (annotated as SEM).

\subsubsection{Loss Function}
\label{sec:original-loss-function}

Parameters of each NN in an ensemble of size $m$ are optimized with respect to
the loss function $\mathcal{L}_{QD}$ (eq.~\ref{eq:original-qde}). The loss
function operates on mini-batches of size $n$.
\begin{equation}
  \label{eq:original-qde}
  \mathcal{L}_{QD} = \mathcal{L}_{MPIW} + \lambda \frac{n}{\alpha (1 - \alpha)} \mathcal{L}_{PICP}
\end{equation}
$\mathcal{L}_{MPIW}$ defined in eq.\ (\ref{eq:loss-mpiw}) is optimizing the width of the PIs
capturing an observation. Due to variables $c$ and $k_i$ from eq.\ (\ref{eq:c})
and (\ref{eq:k_i}), $\mathcal{L}_{MPIW}$ only includes those samples for which the observation is inside the PI.
\begin{equation}
  \label{eq:loss-mpiw}
  \begin{aligned}
  \mathcal{L}_{MPIW} = \frac{1}{c}\sum_{i=1}^{n} (\hat{y}_{U}^{(i)} - \hat{y}_{L}^{(i)}) \cdot k_i
 \end{aligned}
\end{equation}
$\mathcal{L}_{PICP}$ defined in eq.\ (\ref{eq:loss-picp}) is optimizing the coverage probability of the PIs penalizing only if the PICP is below the desired $\gamma$.
\begin{equation}
  \label{eq:loss-picp}
  \begin{aligned}
  \mathcal{L}_{PICP} = \max (0, (1 - \alpha) - PICP)^2
 \end{aligned}
\end{equation}
Expanded, we get
\begin{equation*}
  \label{eq:original-qde-expanded}
  \begin{aligned}
  \mathcal{L}_{QD} ={} & \frac{1}{c}\sum_{i=1}^{n} (\hat{y}_{U}^{(i)} - \hat{y}_{L}^{(i)}) \cdot k_i \\
  & + \lambda \frac{n}{\alpha (1 - \alpha)} \max (0, (1 - \alpha) - PICP)^2.
  \end{aligned}
\end{equation*}
The hyper-parameter (Lagrangian) $\lambda$ controls the importance of
$\mathcal{L}_{PICP}$ with respect to $\mathcal{L}_{MPIW}$. The intuition behind
the fraction $\frac{n}{\alpha (1 - \alpha)}$ is that it should reflect the
confidence of $\mathcal{L}_{PICP}$ with respect to $n$ and $\alpha$.

\subsubsection{Aggregation Method}
\label{sec:original-aggregation-method}

Given an ensemble of $m$ NN models fitted with respect to $\mathcal{L}_{QD}$
(\ref{eq:original-qde}), we acquire prediction intervals $[\hat{y}_{L}^{(ij)},
\hat{y}_{U}^{(ij)}]$ for sample $i \in \{ 1 \isep n \}$ and model $j \in \{1
\isep m\}$. The following calculations are taken to acquire the final predictions
intervals $[\tilde{y}_{L}^{(i)}, \tilde{y}_{U}^{(i)}]$ that should account also
for the epistemic uncertainty.
\begin{equation*}
  \label{ref:mu-l}
  \begin{aligned}
    \mu_{L}^{(i)}=& \frac{1}{m} \sum_{j=1}^{m} \hat{y}_{L}^{(ij)},
  \end{aligned}
\end{equation*}
\begin{equation*}
  \label{ref:mu-u}
  \begin{aligned}
    \mu_{U}^{(i)}=& \frac{1}{m} \sum_{j=1}^{m} \hat{y}_{U}^{(ij)},
  \end{aligned}
\end{equation*}
\begin{equation*}
  \label{ref:sigma-l}
  \begin{aligned}
    {\sigma_{L}^{(i)}}^{2}=\frac{1}{m-1} \sum_{j=1}^{m}\left(\hat{y}_{L}^{(ij)} - \mu_{L}^{(i)}\right)^{2}
  \end{aligned}
\end{equation*}
\begin{equation*}
  \label{ref:sigma-u}
  \begin{aligned}
    {\sigma_{U}^{(i)}}^{2}=\frac{1}{m-1} \sum_{j=1}^{m}\left(\hat{y}_{U}^{(ij)} - \mu_{U}^{(i)}\right)^{2}
  \end{aligned}
\end{equation*}

These quantities are then combined to generate the final PIs.
In the paper~\cite{pearce:2018}, the aggregation is done as follows:
\begin{equation}
  \label{eq:original-aggregation-method-paper-l}
  \begin{aligned}
    \tilde{y}_{L}^{(i)}=\mu_{L}^{(i)} - 1.96 \cdot \sigma_{L}^{(i)},
  \end{aligned}
\end{equation}
\begin{equation}
  \label{eq:original-aggregation-method-paper-u}
  \begin{aligned}
    \tilde{y}_{U}^{(i)}=\mu_{U}^{(i)} + 1.96 \cdot \sigma_{U}^{(i)}.
  \end{aligned}
\end{equation}

The actual implementation differs in using a standard error of the mean (SEM)
$\sigma_{\bar{y}_{L}}^{(i)}$ (\ref{eq:original-aggregation-method-impl-l}) and
$\sigma_{\bar{y}_{U}}^{(i)}$ (\ref{eq:original-aggregation-method-impl-u})
instead of in the paper presented standard deviation $\sigma_{L}^{(i)}$
(\ref{eq:original-aggregation-method-paper-l}) and $\sigma_{U}^{(i)}$
(\ref{eq:original-aggregation-method-paper-u}) respectively; notice the introduction of the scalar $1/\sqrt{m}$ in both equations.
\begin{equation}
  \label{eq:original-aggregation-method-impl-l}
  \begin{aligned}
    \tilde{y}_{L}^{(i)}=\mu_{L}^{(i)} - 1.96 \cdot \sigma_{L}^{(i)} \cdot \mathbf{\frac{1}{\sqrt{m}}}
  \end{aligned}
\end{equation}
\begin{equation}
  \label{eq:original-aggregation-method-impl-u}
  \begin{aligned}
    \tilde{y}_{U}^{(i)}=\mu_{U}^{(i)} + 1.96 \cdot \sigma_{U}^{(i)} \cdot \mathbf{\frac{1}{\sqrt{m}}}
  \end{aligned}
\end{equation}

Aggregating PIs using the latter equations
(\ref{eq:original-aggregation-method-impl-l}) and
(\ref{eq:original-aggregation-method-impl-u}) results in narrower PIs. Both
aggregation methods lack any theoretical justification:
The lower and upper PI boundaries are aggregated independently, i.e. without mutual consideration, and we consider this as a flaw of the method.
In fact, under a simple assumption of normal posterior distribution that is correctly
captured by $m$ NN models but with shifted PIs with correct $\gamma$ coverage
probability (PICP), it can be shown that both original aggregation methods in
the above equations yield PIs resulting in PICP greater than $\gamma$, and thus
greater MPIW. In the results, the latter aggregation function (eq.~\ref{eq:original-aggregation-method-impl-l} and \ref{eq:original-aggregation-method-impl-u}) was evaluated (annotated as SEM).

\section{METHOD}
\label{sec:method}

In this section, we describe our method annotated as SNM-QD+ that provides a point prediction along with prediction interval as output. First, an ensemble
of neural network models is trained with an extended quality-driven loss
function annotated as QD+ (eq.\ \ref{eq:loss-qd-plus}). Second, the results from each model in the ensemble are aggregated to provide a final result (incorporating the epistemic uncertainty)
by a fitting split normal density functions~\cite{wallis:2014} from each NN's point estimate and PI, and aggregating them into a split normal mixture (annotated as SNM) from which the final PI of coverage probability $\gamma$ is calculated.

\subsection{Quality-Driven Loss Function}
\label{sec:loss-function}

By having a single model for prediction intervals and point estimates, we achieve a
coherency of the results and so avoid the case of two disjoint models learning
different function approximations for prediction intervals and point estimates.
In other words, providing prediction intervals and point estimates as a result
of two disjoint models may not capture the predictive uncertainty of the point
estimate model.

We therefore propose a new loss function
\begin{equation}
  \label{eq:loss-qd-plus}
  \begin{aligned}
  \mathcal{L}_{QD+} ={} & (1 - \lambda_{1}) (1 - \lambda_{2}) \cdot \mathcal{L}_{MPIW} \\
  & + \lambda_{1} (1 - \lambda_{2}) \cdot \mathcal{L}_{PICP} \\
  & + \lambda_{2} \cdot \mathcal{L}_{MSE} \\
  & + \xi \cdot \mathcal{L}_{P}
 \end{aligned}
\end{equation}
where point estimates $\hat{y}^{(i)}$ are optimized by
\begin{equation}
  \label{eq:loss-mse}
  \begin{aligned}
  \mathcal{L}_{MSE} = \frac{1}{n}\sum_{i=1}^{n} \left(\hat{y}^{(i)} - y^{(i)}\right)^{2}
 \end{aligned}
\end{equation}
and the penalty function
\begin{equation}
  \label{eq:penalty-function}
  \mathcal{L}_{P} = \frac{1}{n} \sum_{i=1}^{n} \left[\operatorname{max}(0, \hat{y}_{L}^{(i)} - \hat{y}^{(i)}) + \operatorname{max}(0, \hat{y}^{(i)} - \hat{y}_{U}^{(i)})\right]
\end{equation}
is adding a constraint to enforce their integrity.

We retrofit a slightly simplified version of the loss function by
\citeauthor{pearce:2018} \citeyearpar{pearce:2018}. We introduce an auxiliary
loss $\mathcal{L}_{MSE}$ driving the point estimates, the mean squared error
(MSE) in our particular use case. However, our empirical results showed rather
difficult training process (exhibited already by the original loss function
$\mathcal{L}_{QD}$) and issues with the integrity of the generated output, i.e.
interval crossing and point estimates out of PI bounds. Therefore a constraint
violation penalty function $\mathcal{L}_{P}$ was added which mitigated both
issues significantly.

The hyper-parameter $\lambda_{1} \in (0,1)$ controls the mutual influence between losses $\mathcal{L}_{MPIW}$ (eq.\ \ref{eq:loss-mpiw}) and $\mathcal{L}_{PICP}$ (eq.\ \ref{eq:loss-picp}).
The hyper-parameter $\lambda_{2} \in (0,1)$ controls the influence of the $\mathcal{L}_{MSE}$ (eq.\ \ref{eq:loss-mse}) in relation to the aforementioned $\mathcal{L}_{MPIW}$ and $\mathcal{L}_{PICP}$.
The hyper-parameter $\xi$ of the penalty function $\mathcal{L}_{P}$ (eq.\ \ref{eq:penalty-function}) controls the degree of penalization in case the constraint is violated.

\subsection{Split Normal Aggregation Method}
\label{sec:aggregation-method}

The prediction intervals and point estimates retrieved from the ensemble need
to be aggregated into a final prediction interval, capturing both
the aleatoric and epistemic uncertainty, and a final point estimate.

\begin{figure}[t]
  \vskip 0.2in
  \begin{center}
  \centerline{\includegraphics[width=\columnwidth]{split_normal_mixture_2_vectorized}}
  \caption{An example illustrating split normal PDFs (dashed) fitted from an ensemble ($m=5$) of PIs and point estimates, and subsequently aggregated into a split normal mixture (black) from which the final PI is calculated.}
  \label{fig:split-normal-mixture}
  \end{center}
  \vskip -0.2in
\end{figure}

To this end, we assume that the posterior predictive distribution from a single model is a split normal distribution~\cite{wallis:2014}. The split normal distribution or two-piece normal distribution is a result of joining halves of normal distributions with the same mode but different variances. The split normal probability density function (PDF) is defined as
\begin{equation*}
  f_{\mathcal{SN}}\left(x ; \mu, \sigma_{1}, \sigma_{2}\right)=
  \begin{cases}
  A \exp \left(-\frac{(x-\mu)^{2}}{2 \sigma_{1}^{2}}\right) &
  \text{if } x < \mu, \\
  A \exp \left(-\frac{(x-\mu)^{2}}{2 \sigma_{2}^{2}}\right) &
  \text{otherwise},
\end{cases}
\end{equation*}
where $ A=\sqrt{2 / \pi}\left(\sigma_{1}+\sigma_{2}\right)^{-1}$.

Given the lower and upper PI bounds $[\hat{y}_{L}^{(ij)}, \hat{y}_{U}^{(ij)}]$ and the point estimates $\hat{y}^{(ij)}$ from an ensemble, we can fit parameters of a split normal distribution for each ensemble result by optimizing the loss
\begin{equation}
  \label{eq:split-normal-loss}
  \begin{aligned}
  \mathcal{L}_{\mathcal{SN}} = {} & [F_{\mathcal{SN}}(\hat{y}_{L}^{(ij)}; \cdot) - \alpha / 2]^2 \\
  & + [F_{\mathcal{SN}}(\hat{y}_{U}^{(ij)}; \cdot) - (1 - \alpha / 2)]^2
  \end{aligned}
\end{equation}
where the split normal cumulative density function (CDF) is defined as
\begin{equation*}
  F_{\mathcal{SN}}\left(x ; \mu, \sigma_{1}, \sigma_{2}\right) =
  \begin{cases}
  \frac{\sigma_{1} + \operatorname{erf}\left(\frac{x - \mu}{\sqrt{2} \sigma_{1}}\right) \sigma_{1}}{\sigma_{1} + \sigma_{2}} &
  \text{if } x < \mu, \\
  \frac{\sigma_{1} + \operatorname{erf}\left(\frac{x - \mu}{\sqrt{2} \sigma_{2}}\right) \sigma_{2}}{\sigma_{1} + \sigma_{2}} &
  \text{otherwise}.
\end{cases}
\end{equation*}

Through a gradient descent (GD) optimization of eq. (\ref{eq:split-normal-loss}) we estimate the parameters $\sigma^{(ij)}_{1}$ and $\sigma^{(ij)}_{2}$ with respect to $[\hat{y}_{L}^{(ij)}, \hat{y}_{U}^{(ij)}]$ (variable $x$), $\hat{y}^{(ij)}$ (variable $\mu$) and $\alpha$.

When fitting the parameters $\sigma^{(ij)}_{1}$ and $\sigma^{(ij)}_{2}$ with eq.\ (\ref{eq:split-normal-loss}) as the objective, initialization of $(\sigma^{(ij)}_{1}, \sigma^{(ij)}_{2})$ plays an important role for finding the optimal solution. A random initialization does not always yield the optimal solution. Therefore, following the
hypothesis that the parameters of a split normal distribution are close to those of normal distribution, we initialize the $\sigma^{(ij)}_{1}$ and $\sigma^{(ij)}_{2}$ as follows:
\begin{equation*}
  \label{eq:sigma_1-initialization}
  \sigma^{(ij)}_{1} = \frac{\hat{y}_{L}^{(ij)} - \hat{y}^{(ij)}}{\sqrt{2} \operatorname{erf}^{-1}(2 p_{L}-1)},
\end{equation*}
\begin{equation*}
  \label{eq:sigma_2-initialization}
  \sigma^{(ij)}_{2} = \frac{\hat{y}_{U}^{(ij)} - \hat{y}^{(ij)}}{\sqrt{2} \operatorname{erf}^{-1}(2 p_{U}-1)},
\end{equation*}
where $p_{L} = \alpha / 2$ and $p_{U} = 1 - \alpha / 2$. The above is derived from the inverse CDF (quantile function) of a normal distribution:
\begin{equation*}
    F_{\mathcal{N}}^{-1}(p) = \mu + \sigma \sqrt{2} \operatorname{erf}^{-1}(2p - 1)
\end{equation*}
with $p \in (0,1)$.

In our experiments, this has been proven as superior compared to random
initialization.

As a result of the above steps, we acquire an ensemble of $m$ split normal PDFs $\left\{f_{\mathcal{SN}}\left(x; .\right)\right\}_{j=1}^{m}$ (for a single sample $i$) from which we create a mixture PDF
\begin{equation*}
    \label{eq:split-normal-mixture}
    f^{(i)}(x)=\frac{1}{m} \sum_{j=1}^{m} f_{\mathcal{SN}}\left( x; \theta^{(ij)} \right).
\end{equation*}
See Figure \ref{fig:split-normal-mixture} for illustration. The final PIs $[\tilde{y}_{L}^{(i)}, \tilde{y}_{U}^{(i)}]$ are calculated from the mixture distribution $f^{(i)}(x)$ by numerically solving
\begin{equation*}
    \label{eq:final-prediction-intervals}
    \tilde{y}_{L}^{(i)} = {F^{-1}}^{(i)}(\alpha / 2) \quad \text{ and } \quad \tilde{y}_{U}^{(i)} = {F^{-1}}^{(i)}(1 - \alpha / 2)
\end{equation*}
where ${F^{-1}}^{(i)}(p)$ is the inverse CDF of the mixture $f^{(i)}(x)$.

The final point estimates are calculated as an equally weighted combination (mean) of point estimates from an ensemble.
\begin{equation*}
  \tilde{y}^{(i)} = \frac{1}{m} \sum_{j=1}^{m} \hat{y}^{(ij)}
\end{equation*}

\section{EXPERIMENTS}
\label{sec:experiments}

\begin{table*}[t]
  \scriptsize
  \centering
  \caption{Ultimately, PIs are compared between SNM-QD+, SEM-QD and MVE, and point estimates are compared between SNM-QD+, MSE and MVE in Table \ref{tab:results-pe}. Additionally, aggregation methods are compared in SNM-QD+ vs. SEM-QD+. All methods are ensembles of 2-layer NNs. The values stand for mean $\pm$ standard error of the mean. The best results (with standard error of the mean taken in consideration) are shown in bold.}
  \label{tab:results-pi}
  \vskip 1.em
  \setlength\tabcolsep{5pt}
  \begin{tabular}{l|*4c|*4c}
    \toprule
    \multirow{2}{*}{\textbf{Dataset}}
    & \multicolumn{4}{c|}{\textbf{PICP}}
    & \multicolumn{4}{c}{\textbf{MPIW}} \\

    & SNM-QD+ & SEM-QD+ & SEM-QD & MVE
    & SNM-QD+ & SEM-QD+ & SEM-QD & MVE \\
    \midrule
    Boston
    & $0.95\pm0.01$ & $0.97\pm0.01$ & $\mathbf{0.95\pm0.01}$ & $0.88\pm0.01$
    & $1.58\pm0.06$ & $1.82\pm0.07$ & $\mathbf{1.52\pm0.06}$ & $0.89\pm0.01$ \\
    Concrete
    & $\mathbf{0.94\pm0.01}$ & $0.96\pm0.01$ & $0.97\pm0.00$ & $0.96\pm0.00$
    & $\mathbf{0.99\pm0.04}$ & $1.14\pm0.05$ & $1.36\pm0.02$ & $1.14\pm0.02$ \\
    Energy
    & $0.99\pm0.00$ & $0.99\pm0.00$ & $0.99\pm0.01$ & $\mathbf{0.98\pm0.01}$
    & $0.29\pm0.01$ & $0.33\pm0.02$ & $0.48\pm0.03$ & $\mathbf{0.16\pm0.00}$ \\
    Kin8nm
    & $0.97\pm0.00$ & $0.98\pm0.00$ & $0.99\pm0.00$ & $\mathbf{0.97\pm0.00}$
    & $1.07\pm0.01$ & $1.21\pm0.01$ & $1.29\pm0.01$ & $\mathbf{1.04\pm0.01}$ \\
    Naval
    & $1.00\pm0.00$ & $1.00\pm0.00$ & $0.99\pm0.00$ & $\mathbf{1.00\pm0.00}$
    & $0.09\pm0.00$ & $0.10\pm0.00$ & $0.36\pm0.02$ & $\mathbf{0.05\pm0.00}$ \\
    Power
    & $\mathbf{0.95\pm0.00}$ & $0.96\pm0.00$ & $0.96\pm0.00$ & $0.96\pm0.00$
    & $\mathbf{0.80\pm0.00}$ & $0.85\pm0.00$ & $0.87\pm0.00$ & $0.87\pm0.01$ \\
    Protein
    & $\mathbf{0.95\pm0.00}$ & $0.97\pm0.00$ & $0.95\pm0.00$ & $0.98\pm0.00$
    & $\mathbf{2.12\pm0.01}$ & $2.31\pm0.01$ & $2.24\pm0.01$ & $2.82\pm0.10$ \\
    Wine
    & $\mathbf{0.94\pm0.01}$ & $0.95\pm0.01$ & $0.92\pm0.01$ & $0.94\pm0.00$
    & $\mathbf{2.62\pm0.06}$ & $2.92\pm0.07$ & $2.06\pm0.03$ & $2.94\pm0.02$ \\
    Yacht
    & $\mathbf{0.94\pm0.01}$ & $0.96\pm0.01$ & $1.00\pm0.00$ & $0.99\pm0.00$
    & $\mathbf{0.12\pm0.00}$ & $0.13\pm0.00$ & $0.25\pm0.02$ & $0.40\pm0.04$ \\
    Year
    & $0.94\pm\text{NA}$ & $0.96\pm\text{NA}$ & $\mathbf{0.95\pm\text{\textbf{NA}}}$ & $0.96\pm\text{NA}$
    & $2.34\pm\text{NA}$ & $2.54\pm\text{NA}$ & $\mathbf{2.29\pm\text{\textbf{NA}}}$ & $2.75\pm\text{NA}$ \\
    \bottomrule
  \end{tabular}

\end{table*}

\begin{table}[t]
  \scriptsize
  \centering
  \caption{Extension of Table \ref{tab:results-pi} evaluating point estimates of SNM-QD+, MSE and MVE.}
  \label{tab:results-pe}
  \vskip 1.em
  \setlength\tabcolsep{5pt}
  \begin{tabular}{l|*3c}
    \toprule
    \multirow{2}{*}{\textbf{Dataset}}
    & \multicolumn{3}{c}{\textbf{MSE}} \\

    & MSE & SNM-QD+ & MVE \\
    \midrule
    Boston
    & $\mathbf{0.112\pm0.013}$ & $0.115\pm0.013$ & $0.127\pm0.019$ \\
    Concrete
    & $0.056\pm0.003$ & $\mathbf{0.053\pm0.003}$ & $0.077\pm0.004$ \\
    Energy
    & $0.001\pm0.000$ & $\mathbf{0.001\pm0.000}$ & $0.001\pm0.000$ \\
    Kin8nm
    & $0.060\pm0.001$ & $\mathbf{0.059\pm0.001}$ & $0.062\pm0.001$ \\
    Naval
    & $\mathbf{4.8\text{e-}5\pm0.000}$ & $1.3\text{e-}4\pm0.000$ & $1.2\text{e-}4\pm0.000$ \\
    Power
    & $\mathbf{0.042\pm0.001}$ & $0.050\pm0.001$ & $0.048\pm0.001$ \\
    Protein
    & $\mathbf{0.310\pm0.002}$ & $0.361\pm0.004$ & $0.445\pm0.023$ \\
    Wine
    & $\mathbf{0.597\pm0.015}$ & $0.616\pm0.018$ & $0.621\pm0.015$ \\
    Yacht
    & $0.002\pm0.000$ & $\mathbf{0.001\pm0.000}$ & $0.003\pm0.000$ \\
    Year
    & $0.637\pm\text{NA}$ & $\mathbf{0.636\pm\text{\textbf{NA}}}$ & $0.639\pm\text{NA}$ \\
    \bottomrule
  \end{tabular}

\end{table}

Our proposed loss function QD+ (Section~\ref{sec:loss-function}) and aggregation method SNM (Section~\ref{sec:aggregation-method}) are compared with the original QDE~\cite{pearce:2018}, specifically with the loss function QD (Section~\ref{sec:original-loss-function}) and aggregation method SEM (Section~\ref{sec:original-aggregation-method}). We also compare against an ensemble of NNs optimizing the parametrization of a normal distribution~\cite{lakshminarayanan:2017}, referred as a mean-variance estimator (MVE). Additionally, we compare point estimates generated by our method SNM-QD+ with an ensemble of NNs optimizing MSE (eq.\ \ref{eq:loss-mse}) for point estimates solely.

We follow a similar experimental setup initially set by \citeauthor{hernandez:2015}
\yrcite{hernandez:2015} and also followed in related works~\cite{gal:2016,
lakshminarayanan:2017, pearce:2018, tagasovska:2019}.

\subsection{Datasets}
\label{sec:datasets}

Ten open-access benchmark datasets from the UCI dataset repository are used
\cite{dua:2019}. For each dataset we have created 20 shuffled versions across which we have evaluated the methods, i.e. 20 trials. Exceptions are the Year dataset with 1 trial (without shuffling) and the Protein dataset with 5 trials. The datasets are standardized (to zero mean and unit variance) based on the
training set. The input of evaluation measures is standardized based on the full dataset for comparability across different trials.

\subsection{Hyper-Parameters}
\label{sec:hyper-parameters}

We optimize for 95\% prediction intervals, i.e. $\alpha = 0.05$.

To not add any competitive advantage to our method, we keep the NN sizes identical across different models. Note that this may be a disadvantage for our model, given the complexity of the loss function QD+.
This potentially sub-optimal setting should be kept in mind as our comparison's objective is feasibility when compared to models specialized either to generate point estimates or PIs (and not both at the same time).

The ensemble size is $m=5$. The size of mini-batches is $n=100$ ($n=1000$ for the Year dataset). All NNs have 2 hidden layers with 50 units (100 units for Protein and Year datasets) and ReLU activation functions.

For the hyper-parameter search (HPS), we do not follow the legacy of \citeauthor{hernandez:2015} \yrcite{hernandez:2015}. The HPS is performed only on a single concrete shuffled version of a dataset with excluded 10\% test set, i.e. training set remains. Given the training set, the hyper-parameters (HPs) are validated on 5 shuffled 90\% and 10\% splits (i.e. 81\% and 9\% of the complete dataset) for training and validation, respectively. This deviates opposed to the originally suggested single 80\% and 20\% split (i.e. 72\% and 18\% of the complete dataset) without cross-validation. The change of the hyper-parameter search setup was motivated by the difficulty to find good HPs using the original setup, especially in smaller datasets (such as Boston, Concrete, Energy, Wine and Yacht).

A random HPS was performed on the following hyper-parameters (depending on the model): learning rate, decay rate, $\lambda_{1}$, $\lambda_{2}$, epochs. $\xi$ is set to $10$ for all experiments. The remaining settings are consistent with those of \citeauthor{pearce:2018} \yrcite{pearce:2018}.

The applied heuristic for selecting the best hyper-parameters was as follows: If mean PICP is equal to $\gamma \pm 0.01$, then HPs with the lowest MPIW and MSE were selected. Alternatively, also heuristic considering mean and standard deviation of PICP proved good results. If neither of the criteria were fulfilled, the best achieved PICP was selected. Also, the training process on the validation set was analyzed for steady convergence and overfitting.
Note that the loss cannot be used to select HPs because $\lambda_{1}$, $\lambda_{2}$ and $\xi$ scale parts of the loss function; hence the loss is associated with particular values of $\lambda_{1}$, $\lambda_{2}$ and $\xi$.

Due to a smaller number of HPS trials (max. 300), we did not always find well-performing HPs. Therefore, some experiments (QD+ for the datasets Energy, Kin8nm, Naval, Protein and Year) were manually fine-tuned. Adjusting hyper-parameters ($\lambda_1$ and $\lambda_2$, learning rate, decay rate and number of epochs) has proven to be intuitive by analyzing PICP, MPIW and MSE on a validation set. We resorted to manual fine-tuning only if a model was considerably under-performing compared to the original QDE or compared to MVE. There was no attempt to optimize to the model's full potential.

\subsection{Evaluation}
\label{sec:evaluation}

The input of evaluation measures is standardized based on the full dataset for comparability across different trials. The following measures are used to assess and compare models (QD+, QD and MVE) and aggregation methods (SNM and SEM): PICP, MPIW and mean squared error (MSE).

\subsection{Results}
\label{sec:results}

Tables~\ref{tab:results-pi} and \ref{tab:results-pe} show the results of comparing the two models QD+ and QD, combined with the aggregation methods SNM and SEM. We have also included a model that generates only point estimates (denoted MSE), and MVE generating both PIs and point estimates. Consider SNM-QD+ vs. SEM-QD for a comparison between our method and the original QDE method~\cite{pearce:2018}, respectively. SNM-QD+ vs. SEM-QD+ gives a comparison between our split normal aggregation method and the original aggregation method, respectively. SNM-QD+ vs. MVE provides a comparison of methods generating both PIs and point estimates. Finally, Table \ref{tab:results-pe} shows the evaluation of the point estimates between SNM-QD+, MSE and MVE. The best results (with standard error of the mean taken in consideration) are presented using bold font. Supplement~\ref{sec:additional-results} provides the evaluation of non-aggregated PIs.

Overall, the results demonstrate that a complex multi-objective quality-driven loss function (QD+) can deliver well-calibrated PIs when the split normal mixture (SNM) is employed as the aggregation method. The proposed method SNM-QD+ performs best with respect to PIs and competitively with respect to point estimates. We argue that the quality of the point estimates can be improved by increasing the capacity of NNs or by hyper-parameter fine-tuning.

The results clearly show that the split normal aggregation method (SNM) yields well-calibrated PIs, while the original aggregation method (SEM) generally tends to generate wider PIs.
Contrary to SEM, SNM aggregation method does not treat PI boundaries independently. It fits flexible enough (asymmetric) distributions within a versatile mixture distribution that is, as the results show, a sufficient approximation of the posterior predictive distribution.
Note that the SNM aggregation method is not necessarily tied to the QD+ loss, but can be applied in other situations as well.
A potential drawback of SNM is the apparent computational overhead of fitting the split normal mixture when the number of samples is large. However, this learning task is parallelizable and distributable; hence the wall-clock time spent on the task is negligible.

The training process of QD+ is, contrary to the original QD, quite robust.
To give a particular example with the Protein dataset, training 2-layer NNs with the original QDE required approximately one retry per run due to interval crossing or high loss value (stuck in local minimum). No retries were required when using QD+. Details are given in Supplement~\ref{sec:robustness}.
Furthermore, the original QD is sensitive to parameter initialization. Overall, the penalty function $\mathcal{L}_{P}$ (eq.~\ref{eq:penalty-function}) in QD+ results in significant improvements of the stability of the training process and strengthens the integrity of the output (mitigates the undesired interval crossing and point estimates out of the PI bounds).
Supplement~\ref{sec:sensitivity-analysis} provides a sensitivity analysis together with an ablation study clearly demonstrating the importance of the penalty function.

Since we compare our results with \citeauthor{pearce:2018} \yrcite{pearce:2018} and \citeauthor{lakshminarayanan:2017} \yrcite{lakshminarayanan:2017}, our work is also comparable with Bayesian approach by \citeauthor{hernandez:2015} \yrcite{hernandez:2015}, and partly also with \citeauthor{tagasovska:2019} \yrcite{tagasovska:2019}.

\subsection{Implementation}
\label{sec:implementation}

Our implementation is shared\footnote{\url{https://github.com/tarik/pi-snm-qde}} and results fully reproducible. To embrace reproducibility, we use the Python package {Sacred}~\cite{sacred:2017}. PyTorch~\cite{pytorch:2019} is used as the main NN framework and the fitting of a split normal mixture is implemented using JAX~\cite{jax:2018}.

\section{CONCLUSIONS}
\label{sec:conclusions}

The main finding of this paper is that models delivering point estimates together with prediction intervals are competitive to models providing only one or the other.
Motivated by the results of the quality-driven deep ensembles~\cite{pearce:2018} as an alternative to Bayesian methods, we endeavored to address the three key limitations: (1) The inability to generate point estimates; (2) the weakly justified method for aggregation of prediction interval ensembles; (3) the fragile training process.

We propose a new quality-driven loss function generating both prediction intervals and point estimates, and we dramatically increase the robustness of the training process by integrating a penalty function.

A unique and well-founded method fitting a split normal mixture as the aggregate of ensembled neural network output generates well-calibrated prediction intervals accounting for aleatoric and epistemic uncertainty. Moreover, an analytical approach for initializing parameters of a split normal probability density function is proposed that leads to acceleration and dramatically increased success of the fitting process.

With this work, we extend the practitioners toolset for quantifying predictive uncertainty in the regression task.

\section{FUTURE WORK}
\label{sec:future-work}

We end the paper with the following suggestions for future research or possible improvements:

\begin{itemize}
    \item
    We have shown that a single model architecture can be used to generate both point estimates and prediction intervals. We therefore envision a two-step learning process, where one first optimizes the model to generate point estimates and consequently learns to generate prediction intervals. To avoid catastrophic forgetting, it seems appropriate to treat this as a transfer learning problem. The key benefit of this operation is that it will allow us to reuse (pre-trained) NN models that currently only generate point estimates.

    \item
    It seems reasonable to use the NMPIW measure (eq.\ \ref{eq:nmpiw}) in place of MPIW in the definition of the loss function $\mathcal{L}_{MPIW}$ (eq.\ \ref{eq:loss-mpiw}). While it is already relatively intuitive to find HPs manually, and the HPS can be automated efficiently, we hypothesise that using NMPIW may lead to better-scaled hyper-parameters and possibly also to smaller variations across different datasets. The current work uses MPIW to ease the comparison between QD and QD+.

    \item
    The current approach learns to output a reasonable PI that is later used to fit a split normal PDF.
    This could alternatively be simplified so that the output of the neural network was the parameterization of the split normal PDF (or a mixture of split normal PDFs) directly.
    This approach is relevant for use-cases where a full predictive distribution is needed and would require a training objective defined for such situations instead of QD+. Again, the implemented approach was chosen to extend the non-Bayesian branch of research and ease comparison with the work of \citet{pearce:2018}.
\end{itemize}

redpajama_set_name: RedPajamaArXiv


Neurology services at Avita are available by specialty physicians who provide care and treatment for disorders of the nervous system. This includes diseases of the brain, spinal cord, nerves, and muscles. Some of the more common problems that neurologists treat include strokes, epilepsy (seizures), headaches, dementia (Alzheimer's disease), multiple sclerosis, Parkinson's disease, and various forms of pain. A neurologist may serve as a consultant to other physicians or provide long-term care to patients with chronic neurological disorders.


text:
'We are thrilled': Businesses rejoice as Superior Street reopens to traffic
Back to News Items
By Brittany Lind
After a long, tedious four-year-long wait, Superior Street is officially re-opened.
Duluth Mayor Emily Larson announced its reopening Friday morning, and signs blocking 3rd and 4th Ave E were taken down around 6 p.m.
"This is really a project we are reset and we are ready this is truly a once in a generation project, it been a tremendous lift for so many people but most importantly it's open and it's ready," Larson said.
The 31.5 million dollar project began in 2018, with more than 130,000 hours put into it by crews.
Sidewalks and roads were redone along with some beautification efforts such as new flower planters.
Local businesses like Hucklebeary faced many obstacles during construction, as closed roads caused problems.
"Without having people drive down Superior Street, I mean our sidewalks even got replaced as well so having people walk into your business its been challenging. And then throw a pandemic on top of that it's been kind of a wild ride the past 4 years," said owner, Emily Ekstrom.
After four years, many business obstacles are now solved, which is why Duluth's Greater Downtown Council is celebrating.
"There's been a lot of challenges for our businesses, but they are resilient, and our downtown is resilient and we are so excited to have this street opening later on today," said GDC President Kristi Stokes.
Meanwhile, Superior Street businesses couldn't be happier.
"We are thrilled, like thrilled to have the street back open," Ekstrom said.
Construction crews are going to be doing some finishing touches, but they said it won't affect the flow of traffic.
For more information visit: https://kbjr6.com/2021/09/03/we-are-thrilled-businesses-rejoice-as-superior-street-reopens-to-traffic/
Email Facebook LinkedIn Twitter Pinterest


No design skills required. Large selection of Ayah logos to choose from. Create amazing logos quick and easily.
These Ayah logos you can use for all occasions. Both for personal or professional use. For example you can use them to create: menus, post labels, websites, presentation folders, magnets, ads and photo posters. Customize the logo for your needs in seconds!
This Ayah logo may be used anywhere. Your imagination is the limit. Create compelling presentations at work or at school. A few usage examples: classified ads, image processing, spreadsheets, film editing, blogs, Facebook messages and menus. All logos can be easily customized to your needs!
Create exciting school projects or cool promotional materials in seconds. This Ayah logo can be used in endless contexts. Just a sample of use: film editing, place cards, stickers, websites, party invitations, Google ads and stationary. Creating logos can not be any easier?
Our Ayah logos can be used for whatever you need. You can use them at work, home or school projects. Just a small example of usage: websites, presentation folders, signs, business cards, photo wallpaper, documents and menus. Customize the logo for your needs in seconds!


text:
This account will serve as a means for me to host my personal website.  The code provided here was taken via Bootstrap, and I used the template to build upon.  The copyright and licensing are provided.

##  Original Creator of Template

Start Bootstrap was created by and is maintained by **[David Miller](http://davidmiller.io/)**, Owner of [Blackrock Digital](http://blackrockdigital.io/).

* https://twitter.com/davidmillerskt
* https://github.com/davidtmiller

Start Bootstrap is based on the [Bootstrap](http://getbootstrap.com/) framework created by [Mark Otto](https://twitter.com/mdo) and [Jacob Thorton](https://twitter.com/fat).

## Copyright and License

Copyright 2013-2016 Blackrock Digital LLC. Code released under the [MIT](https://github.com/BlackrockDigital/startbootstrap-landing-page/blob/gh-pages/LICENSE) license.

redpajama_set_name: RedPajamaGithub


The Trap Pack Vocal Pack includes 23 High Quality Vocal Tags created by producer Rob B Beatz to function easily in any DAW… These Vocals are for use with trap music to really bring you the edge you've been looking for. Stand out with these deep boisterous vocals which will turn heads when your track plays. The vocals in this pack were mainly inspired for trap productions but can also be used in EDM and other Genres.


text:
Home > Storyboard > Anniversary reflections
We asked Kristjan Järvi, Thomas Hummel and Dr Dirk von Ameln to look back at the first ten years of the Baltic Sea Philharmonic and share their memories and highlights, as well as their aspirations for the future. Kristjan is the orchestra's Founding Conductor and Music Director. Thomas is the orchestra's Executive Director and also Artistic Director of the Usedom Music Festival. Dirk is Chairman of the Board of the Baltic Sea Music Education Foundation.
What was the idea behind the creation of the ensemble in 2008?
Thomas Hummel Creating the orchestra was a process in itself. The Usedom Music Festival had been working with different youth orchestras from the Baltic Sea region, such as the youth orchestra of the Mariinsky Theatre and an orchestra of young players from Finland called Vivo. We decided with Nord Stream, a company established at that time to plan and build a gas pipeline through the Baltic Sea, that we wanted to jointly do a regional project, and our first idea was to form an orchestra of music academy students from just the three Baltic States – Estonia, Latvia and Lithuania. This Baltic Youth Orchestra gave a concert with cellist David Geringas at the Almedalen Week in Visby, on the island of Gotland, in June 2007.
From this starting point, the next step was to create an orchestra from all the ten Baltic Sea countries, which was the Baltic Sea Youth Philharmonic. We were all excited about the idea of spreading out across the region, and involving young musicians from more countries. Our challenge was that we had to bring everything together for two concert dates in 2008: 4 June in Riga on the occasion of the summit of the Baltic Sea states, and 21 September when the orchestra would open the Usedom Music Festival. So we really needed an inspirational conductor to pull this off. Kristjan had a lot of concerts in his schedule that year but happened to be free on those two days. With him on board we were able to make those first two concerts happen.
The vision of growing into an ensemble with international tours that we have today was not there at the start. In the beginning the main idea was to bring young people together. At the 2008 Usedom Music Festival we also initiated a Youth Forum involving high school students from Denmark, Poland and Germany. They came to the orchestra's concert and got to experience great music.
Dirk von Ameln I had the pleasure of attending the concert in Gotland in 2007 with the young musicians from the three Baltic States. At that time I was the Permitting Director of Nord Stream AG and our approach was to become a good neighbour in the region. We were sponsoring environmental and scientific projects, but we also wanted to do something cultural. Those first concerts were a big success: people could see that here were young musicians from different countries working together and contributing to society in a very positive way. That was our message, the idea we wanted to share.
Kristjan Järvi When I became involved in 2008 we only had two and a half months to put together an orchestra for the first concert in Riga. For the concert programme, we didn't just want traditional repertoire, we wanted to do something different. Indeed, we did a world premiere – a commission from Danish composer Niels Marthinsen, Burning Fiery Furnace. Everything came together so quickly and I think we were extremely lucky to be in the right place at the right time, and to have a sponsor who was willing to take the germ of an idea that emerged in Gotland, and then expand it into something more substantial. After the first two concerts I was just eager to know what we were going to do next. I thought, 'This is too good to be a one-off!'
What are your memories of the first concerts in 2008? What was the atmosphere like in the orchestra and in the audience?
TH The atmosphere was terrific. Everyone had the feeling that something great was starting. And that feeling was there in Riga and also in Usedom, where we had young people coming together for the Youth Forum and listening to the music. The orchestra played Dvořák's Ninth Symphony, and also Beethoven's Triple Concerto with soloists from Denmark, Latvia and Lithuania – violinist Søren Elbæk, pianist Lauma Skride and cellist David Geringas. The musicians clearly shared in the enjoyment of making great music together, and understood that this was also an opportunity to be great neighbours and learn from one another. That's what came across to me in the rehearsals, too, and it was a moving experience for me.
KJ Yes, right from the beginning this project was about creating a community through music. The players are an extension of all the different communities and nationalities of the Baltic Sea region. The orchestra is all about the people of the region, and the primary goal is to unify people, and to create a powerful symbol of unity.
DvA For me, the excitement and energy that comes off the players and jumps out into the auditorium is one of the defining things about the Baltic Sea Philharmonic. It was there in those early concerts and comes through strongly today.
What have been your highlights from the first ten years?
KJ I always say my highlight is the last thing we did. Every year's a highlight, and every year has many highlights. I can't really say that one concert or one project is more special than another. It's like seeing a baby grow up: it's wonderful all the way along.
TH It's certainly been an incredible progression over the last 10 years. To think we started just in Riga and Usedom, and now we've spread out beyond the Baltic Sea region, and toured to Italy, Switzerland, Austria and France, and this year we're going to the Middle East. Being able to work with the same conductor so consistently for so many years has been fantastic. It was amazing, also, that we were able to have Kurt Masur conduct the orchestra on two occasions at the Usedom Music Festival.
KJ Kurt Masur was one of the biggest heroes and role models we've had the honour and fortune to work with in the orchestra. Our project is about unifying people, and he was instrumental in unifying voices from the East and West in Germany, so for me, having him involved was like a big stamp of approval, an endorsement that what we're doing is right.
DvA I remember some very special concerts, one of them at the Beethoven Festival in Bonn in 2014. When the main programme ended, an usher from the concert hall brought down a walking frame for an old lady who was sitting in front of me. Then the encores came and people in the audience were standing up, and this lady stood up and pushed away the frame, raised her arms and clapped her hands. She was so excited by the young musicians, and it was so inspiring to see how that spark went over, and how she just cast aside her frame.
Which of the orchestra's achievements are you most proud of?
TH I'm very proud of what we've created, the contacts between communities that we've made, and where we've got to in just ten years, especially with all the recent developments such as performing from memory and with light and projection art. There has been a lot of progress. Other organisations might have taken 20 or 30 years to come to this point.
KJ Many orchestras take big jumps organisationally or creatively, but they don't have a strategic vision or a grand plan. We have a grand plan which is about constantly empowering and developing our musicians, but that plan changes according to current influences, such as geopolitical and social influences but also influences from different musical styles and genres. That's why we're doing things like performing with lights and playing from memory, and doing these things give us a unique, singular identity as an orchestra.
We've developed great partnerships. The Usedom Music Festival is a partner, and my production company Sunbeam Productions is a partner. People see us now as a network, not just one orchestra but a network of like-minded thinkers. This approach will gain us more partners in the future. We're also expanding our geographical reach, touring beyond Europe. We hope to go to more countries, to the US and Asia, because the word is spreading about us. And why is it spreading? Because we're consistent, we have a vision and a strategy, and we're constantly developing and growing.
DvA If there is a landmark achievement for me, it is the orchestra winning the European Cultural Prize in 2015. But in another sense, when I look back, the orchestra was a start-up, and we have developed it into a well-working entity. It's organised and efficient but we have managed to create a framework that allows for creativity; it's not a rigid organisation. It's still very much open to allow new things to happen and changes to take place, and for things to be done differently. And that may be one of our biggest achievements.
What has the orchestra not done yet that you would like it to do?
TH We talk a lot, together with Kristjan, about the next steps. We don't want to do the same things as other orchestras. Going on tour and doing ten concerts is good but we're looking at how we might extend our stays in certain cities. It would be great if we could base ourselves somewhere for a few days, and bring a special vibe to that place, and make the people there feel that they simply have to come and experience what we are doing. And if we can do that not just in European cities but in other parts of the world too, and thus carry the unifying spirit of the Baltic Sea far and wide, that would be wonderful.
KJ Thomas is right about this vision. When we go and play in a city, we're not changing the environment, we're creating the environment. Our message to the people of that city is, 'Even though this orchestra is international and is made up of musicians from all across the Baltic Sea region, it is something singularly for your community. We will come into your community, and connect with you and spend time with you, and show you a world of possibilities through music and culture, and through our education and outreach projects, such as the school concerts we do.' So we go into a community which we may not know much about, and they enter a fresh world which we bring to them. There is so much potential impact from that experience. For me, I actually can't think of a better project that I have done in my life.
Do you think that being a voice of international understanding, a movement for bringing people together, is more important now that it was ten years ago?
DvA My answer to that is unfortunately yes. When we started out, there was some tension between the West and the East, with the Georgian crisis and the 2008 war in South Ossetia. But the situation stabilised and improved. Now, however, we are back to a world where barriers are put up, and not just psychological barriers but also physical barriers. So it is even more important to have an entity that overcomes barriers, breaks down borders, and tells the story of a united community. We will continue to do our bit to improve the situation.
KJ Most political analysts will say that relations between Russia and America have never been as bad as they are now. Even when they were bad in the Cold War, at least then everybody knew the rules of the game. No good comes of such tension today, but if there's one community where Russia's not going to be excluded it's our community – the community of the Baltic Sea Philharmonic.
What do you hope that Baltic Sea Philharmonic musicians will learn from their time in the orchestra and take through into their professional lives?
TH We've always encouraged openness and fearlessness in our musicians, telling them that nothing is impossible. When we first talked about playing The Firebird from memory, some of us thought it would be very difficult. To see the coaches work with the players, and then at the end everyone performing on stage with no music stands, playing with amazing energy, with connection, with pride in their achievement – that was a very moving experience for me. For the musicians, having that confidence, that knowledge that they can play a full symphonic piece without sheet music – and if they can do that then anything is possible – is very important for them going forward.
KJ We had the situation where musicians came up to us after The Firebird saying, 'Hey, we want to memorise everything. We just want to play from memory now.' That's a fantastic attitude.
DvA All the things we do come down to empowering people. We empower our musicians, and my hope is that they carry that forward and help other people to believe in themselves and put energy and effort into achieving their goals.
KJ I'm very happy that there are chamber groups, in places like Berlin and Estonia, and ensembles like 1B1 in Norway and NICO in Lithuania, which are doing similar things to us. We are all inspiring each other, and as we develop new ideas we create a movement, a momentum for doing things differently, based on the understanding that there's nothing to fear, that everything is possible. My hope for the future is that this entrepreneurial mentality takes hold of more and more people, not just in the world of classical music but in other genres and other areas of society. I want people to look at what we are doing, how we connect people and connect people with the environment, and be so moved and motivated by it that they want to do it themselves, only better.