text:
'Puss in Boots: The Last Wish': So Much More Than Just a Sequel
Producer Mark Swift talks about the development of DreamWorks Animation's blockbuster adventure-comedy now in theaters, and why the long-awaited swashbuckling feline's return is his favorite of all the films he's worked on.
By Jon Hofferman | Thursday, January 5, 2023 at 8:57am
In 3D, CG, Films, People, Voice Acting | ANIMATIONWorld | Geographic Region: All
When you've played a key role in bringing to fruition the epic movie that is Captain Underpants: The First Epic Movie, it's understandable that you may feel your work in this life is finished. Regarding that 2017 release, producer Mark Swift says, "I read the books with my kids, and I loved the idea of a movie. I knew DreamWorks had the rights and were talking about making it, but [CEO of DreamWorks Animation] Jeffrey Katzenberg wasn't a huge fan. But they'd have these meetings, and I would just walk in and offer ideas and thoughts, and eventually someone said, 'Oh, you must be the producer on this film,' and I was like, 'yeah.' I willed my way onto it.'"
While that redoubtable accomplishment may have been enough for some of us, Swift fearlessly carried on with other adventures in animation, including The Croods: A New Age (2020) and, most recently, DreamWorks' new adventure-comedy, Puss in Boots: The Last Wish. Featuring a phenomenal cast and a glowing, painterly design that goes beyond anything DreamWorks has done before, the long-awaited sequel to the 2011 spinoff from the Shrekaverse finds the eponymous adventurer facing his mortality –after years of reckless carousing, he's used up eight of his nine lives, and restoring them will require a daunting journey into the Black Forest. It will also require that he seek help from his former partner and nemesis Kitty Softpaws, and keep one step ahead of the Three Bears Crime Family, "Big" Jack Horner, and the big, bad Wolf.
Among the many pleasures of Puss in Boots: The Last Wish is that the above-mentioned characters are played by Salma Hayek, Ray Winstone, Olivia Colman, Samson Kayo, John Mulaney, and Wagner Moura. The dream cast, which also includes Antonio Banderas as Puss, Harvey Guillén (What We Do in the Shadows) as the therapy dog Perrito, and Florence Pugh as Goldilocks, came together over the sequel's many years of development. When Swift joined the party, he says, some actors were already attached, but he and the other creators did a lot of rejigging.
"When we came in, Florence was already being discussed for Goldi, but we changed up her character and changed up the character of Three Bears," he recounts. "We went with a down-market, Cockney version of them. So they're still a crime family, but you get the sense they're not a very good crime family. They're a little low-rent. Immediately, Ray Winstone, who always plays that kind of gruff Cockney on the wrong side of the law, came to mind. For Baby Bear, they did a great job casting Samson, who's fantastic and so joyful in the movie, and such a contrast to Florence, where you really get that sibling dynamic."
He continues, "The actors add so much to the Goldi and the mother storyline, which is where the heart is in that story. Some of Florence and Olivia's readings are so beautiful, so perfect. I've seen this movie thousands of times and, every time when Goldi says, 'I'm not a bear. You're not my family. I'm looking for a proper family," and the mother says, "I knew it was too good to last, but you need that star, we'll help you get the star,' it brings a lump to my throat. I'm really loving how many people in the reviews are bringing them up because, when you think about it, they don't get a huge amount of screen time."
As for Guillén, Swift says that, as fans of What We Do in the Shadows, the producers felt the choice was obvious once they realized they wanted to cast a Spanish-speaking actor.
"At the start of the movie, we're in some kind of Spanish or Mexican town, and we felt that everyone in that town should be a native Spanish speaker," he explains. "We were looking for a sweet, kind of naive voice, and I think three of us said his name at the same time. He actually came in and tested, and we were like, 'yeah, he's awesome.'"
Perhaps the most surprising character in the film is the Wolf, who, as the physical embodiment of Death, has what you might call a seriously dark side, as well as a complex role to play in the narrative. At the same time, there's a fun aspect to the character that makes him more than simply a one-dimensional incarnation of evil. Again, Swift credits the actor with helping to realize the character's multifaceted personality.
"Wagner Moura was a great partner in finding him," he relates. "We had to move very fast on that character. It was the first sequence we were actually putting into animation and Wagner came to it late. So we were still trying to find that character when we met with him. And one of the things we discovered is that Wolf can kill Puss at any minute. Once we realized that he's playing with him, he's having fun, it gave him a little more personality. But it was important for us that the wolf thinks of himself as funny – if you met him in a bar and it was his day off, he'd be saying, 'I'm a funny guy. In my job, I like to mess around with people.' This is a big deal to him. He wants to make a statement. He wants to let Puss know that he's playing with him. But he waits a while to reveal who he is."
As mentioned above, the animation design in Puss in Boots: The Last Wish – which integrates a number of different styles, and aspires to a kind of storybook aesthetic – is a significant departure for DreamWorks Animation. Swift explains that one of the major challenges for production designer Nate Wragg and his team in developing the look was finding the right balance between audience expectations and the desire to forge a new style.
"The thing they were bumping into initially is that this is not an original movie," he says. "The audience comes into this movie with an expectation of how Puss and Kitty are going to look because they've seen them in the previous movie. Puss is a cat – he's furry, he's cuddly, he's very three-dimensional, and you want to keep that. But we were also flattening the backgrounds, giving it a little bit more of a 2D look, and trying to integrate our 3D characters into that was where we were bumping and where things were popping a little bit. In the end, they came up with a really good balance. When you're seeing Puss in close-up, it's still the Puss you know and love. I think it's a better looking Puss than the last movie, but it's not completely toon-shaded out so it's a different version of him. And, at the start, when he picks up an object, it was looking like it wasn't all within the same film. So that took a while to figure out. We tried to give the action sequences an anime kind of vibe, but that didn't work throughout, because for the more emotional moments, we want to be in there with Puss and see it in a more traditional way."
Swift previously worked with director Joel Crawford on The Croods: A New Age, and the two have developed a close working relationship. This served them especially well in the new film, since, as Swift recounts, there were significant time constraints and they had to move quickly. While he and Crawford had primary responsibility for figuring out what the movie was going to be, it was very much a group effort that also involved co-director Januel Mercado, head of story Heidi Jo Gilbert, editor Jim Ryan, and writer Paul Fisher.
"We work a little bit like a writers' room, where everyone's pitching in," Swift elaborates. "You get disagreements in there, but indecision is worse than a decision in some ways, so we just make a decision. A lot of times, we can revert that decision because we're going to put it up in a screening and storyboard. If it doesn't work, we'll have another go at it. But I think my job sometime is to say, 'okay, we've talked enough, now we're moving this way.' Nine times out of 10, I'm going to do whatever Joel says he wants to do. He has the deciding vote, and I just have to let everyone in the room know that's where we're going."
If working against a punishing deadline while taking into account the thoughts and opinions of a daunting number of stakeholders – including Executive Producer Chris Meledandri, DreamWorks Animation Chief Creative Officer Kristin Lowe and President Margie Cohn, Universal Pictures, not to mention the development team – was the biggest challenge on the film, Swift cites the interaction with his co-workers as the biggest pleasure.
"I've been in animation 30-something years now, and this is my favorite movie ever that I've worked on, both in terms of the result and in terms of how we got there," he enthuses. "And it comes down to people. When you have Joel Crawford, Januel Mercado, Nate Wragg, Jim Ryan, all these people who we very carefully selected – everyone worked together so well. We laughed every day. We joked about it. Even when things were tense, we still kept laughing and joking through it. And I think we created an environment and an atmosphere that allowed creative ideas and notes to come up from everyone."
"Puss in Boots 2 started in a negative place for a lot of people," Swift explains. "It's a sequel to a spinoff of a franchise that has maybe seen better days. So you're coming in and you're having to dig out of that hole. We wanted to demonstrate that this wasn't a lazy kind of cash grab. This was something that we were passionate about. We thought it could be an artistic triumph. And so just to feel that love back from people, it's great. It's really rewarding."
Jon Hofferman is a freelance writer and editor based in Los Angeles. He is also the creator of the Classical Composers Poster, an educational and decorative music timeline chart that makes a wonderful gift.
Meow!
feature film animation
Captain Underpants: The First Epic Movie
GC animation
Voice Cast
Ray Winstone
Samson Kayo
Anthony Mendez
Joy Randolph
Mark Swift
Ludo Bouancheau
The Croods: A New Age
Joel Crawford
Chris Meledandri
Januel Mercado
Heidi Jo Gilbert
Paul Fisher
Kristin Lowe
Margie Cohn
Production Pipelines
painterly animation design
character arcs
More From ANIMATIONWorld:
Superprod Dreams Big with Angoulême Studio ExpansionPrevious Post
Animated Characters: Sequence 2 – My New York Times Part 2
Crime and Funishment: A Feline Hero Returns in 'Puss in Boots: The Last Wish'
'Puss in Boots: The Last Wish' Returns to its Fairy-Tale Illustration Roots
DreamWorks Animation Drops 'Puss in Boots: The Last Wish' Trailer
DreamWorks to Present 'Puss in Boots: The Last Wish' and 'The Bad Guys' at VIEW 2022
DreamWorks Teases 'Puss in Boots: The Last Wish' at Annecy 2022
DreamWorks Animation Drops 'Puss in Boots: The Last Wish' Trailer and Art
DreamWorks' 'The Bad Guys' and 'Puss in Boots: The Last Wish' Get Release Dates
Universal and DreamWorks Rebooting 'Shrek' and 'Puss in Boots'


Charles H.Y. Wei, General Manager, Ensemble Urban Innovation Co.
Hongson Huang, Chair Man, Central Engineering Consultants and Research Inc.
Hsuhui Su, General Manager, Metropolitan Engineering Consulting Co. Ltd.
Looking forward to seeing you in Taipei, 2015.


Coal crusher house for power plant and Maintenance of Crusher House for Coal Handling in of Critical Equipments for Coal Powerpoint Presentation For Crusher Plant.
free powerpoint template of coal crushers cement or coal coal trading companies gap between coal crusher hammer amp breaker block pics of coal mines Check price.
coal crusher house ppt. coal crusher house ppt . Coal Handling Ppt Presentation Author Stream. 28 Dec 2009 Coal Handling Ppt Presentation A PowerPoint presentation.
coal crusher in pp. coal crusher in pp. March 27, 2018 admin. We are an oldest industry of crushing machines. We have experienced manufacturers of Mining Machinery.
Title: PowerPoint Presentation Author: User Last modified by: user Created Date: portable rock crusher rentals oregon Coal Surface Mining.
impact type jaw crusher PPT Powerpoint ball mill ppt slides. Powerpoint Presentations and Slides coal crusher used in chp of power plant.
Crushing properties of coal. Authors Hardgrove grindability index coal quality crusher power Queensland Coal, 1998, March, pp. 26–38. Google Scholar. 15.
crusher powerpoint Power Plant Coal Handling Plant Maintenance Pdf details crusher machine company give coal crusher operation Made For A Crushing Plant Ppt.
Powerpoint(.ppt) Slide Presentations : coal handling plant Download Coal crushers Ring Type Coal Crusher Hammer Mill Coal Crusher Brad Ford Breaker. Get More Info.


Suspension, brakes, wheels, tyres, etc.
Preventative maintenance - 2nd hand coil springs?
17s on my Mk2 1.4 16v?
Replacing Spax coilovers with OEM, which extra parts needed?
the stock centre wheel to top of arch measurement.
Can't get the rear wheels to lock up.
Wheel bearing issue please help!
Clio Ph2 1.4 16v Rear Brake swap. Drum to Disc. Possible?


text:
Q: Uniformly Most Powerful Test and Rejection Region of Poisson Distribution Let $X_1, \dots,X_n$ be a random sample from a Poisson$(\lambda)$ distribution where $\lambda > 0$.

(1) Find the Uniformly Most Powerful (UMP) level $\alpha$ test for the following set of hypotheses: $H_0: \lambda\leq1$ versus $H_A: \lambda >1$.

$\bf{My \ thoughts:}$ I know I can use Karlin-Rubin to help me with this part of the problem. Using Factorization Theorem, I get that $\sum^n_{i=1}X_i$ is a sufficient and complete statistic.  Checking the MLR and using Karlin-Rubin,I get $\alpha = P_{\lambda} \Bigl(\sum^n_{i=1}X_i > 1 \Bigr).$

(2) Using the CLT, provide an expression for the rejection region for this test.

$\bf{My \ thoughts:}$ Using the CLT, I know I need to start by finding the asymptotic distribution of $\sum^n_{i=1}X_i$ and go from there.  Just not sure on setting that up.

(3) Find the power function for this test.

$\bf{My \ thoughts:}$ This doesn't seem like it would be too difficult and should come from having what I need from 1 and 2.
Any help is greatly appreciated.

A: *

*To construct the UMP test, we have to construct the corresponding MP test. Hence, the LR is given by
\begin{align}
\frac{L_{1}(X_1,..,X_n|\lambda_1)}{L_{1}(X_1,..,X_n|\lambda_0)} = \frac{\prod_{i=1}^n \frac{e^{-\lambda_1} \lambda_1^{x_i}}{x_i!}}{\prod_{i=1}^n \frac{e^{-\lambda_0} \lambda_0^{x_i}}{x_i!}} &= \exp\{n(\lambda_0 - \lambda_1)\}\left(\frac{\lambda_1}{\lambda_0}\right)^{\sum_i^nx_i}\\
&= \exp\{n(\lambda_0 - \lambda_1)\}\left(\frac{\lambda_1}{\lambda_0}\right)^{n\bar{x}_n} > c\,.
\end{align}
This statistic depends on the distribution of $\bar{X}_n$, hence for large enough $n$ we can use the CLT to approximate the rejection region. Note that the MP is $\Psi(\mathrm{X}) = \mathcal{I}\left( \bar{X}_n >c' \right)$.
2.

\begin{align}
\alpha = \mathbb{E}_{\lambda_0}\Psi(\mathrm{X}) &= \mathbb{P}_{\lambda_0}\left( \bar{X}_n >c' \right)\\
&a\approx 1-\phi\left(\frac{c'-\lambda_0}{\sqrt{\lambda_0/n}} \right)\\
&c' = \lambda_0 + Z_{1-\alpha}\sqrt{\lambda_0/n}.
\end{align}
For $\lambda_0 = 1$,
$$
c' = 1 + Z_{1-\alpha}\sqrt{1/n}\,.
$$

*Power function for the parametric space $\Lambda = \mathbb{R}^+$.

\begin{align}
\pi(\Psi(\mathrm{X})|\Lambda) &= \mathbb{E}_{\Lambda}\Psi(\mathrm{X}) = \mathbb{P}_{\Lambda}(\bar{X}_n>c')\\
&=1-\phi\left( \frac{c'-\lambda}{\sqrt{\lambda/n}} \right),&  \forall \lambda \in \Lambda.
\end{align}

redpajama_set_name: RedPajamaStackExchange


text:
Mac Games > Puzzle Games > Myths of the World: Spirit Wolf Collector's Edition
Myths of the World: Spirit Wolf Collector's Edition
The wolf spirit still haunts… will you heed its call? You're an expert in Native American symbolism who's been called to investigate some glowing glyphs. Your amulet, which has been in your family for generations, begins to glow as soon as you arrive, as if it's part of this place… As you begin to investigate, you encounter strange ghosts from the past. Find out what caused their rift all those generations ago and how you're connected to this place and its spirit. This is a special Collector's Edition release full of exclusive extras you won't find in the standard version.
Break the symbol barriers using your interactive amulet
Play the bonus game to unlock your amulet's secrets
Get soundtracks, wallpapers, screensavers, and concept art
Try the available Strategy Guide
Free download Myths of the World: Spirit Wolf Collector's Edition 1382.9 Mb
Buy Myths of the World: Spirit Wolf Collector's Edition for Mac $13.99
OS: Mac OS X 10.9/10.8/10.7/10.6
CPU: 1.6 GHz (INTEL ONLY)
Free Hard Drive: 1421 MB

redpajama_set_name: RedPajamaCommonCrawl


text:
Home Historic Charlotte Postcards from the Mary Boyer Collection Presbyterian College, Charlotte, NC
Presbyterian College, Charlotte, NC
Title Presbyterian College, Charlotte, NC
Subject Presbyterian College for Women (Charlotte, N.C.)
Education, Higher--North Carolina--Charlotte
Description View of Presbyterian College in Charlotte, NC at North College and Ninth Streets. [Published by The Washington News Co., Washington, D.C. - 91246; color; undivided back; fine] From 1857 to 1891, the building shown was occupied by the Charlotte Female Institute. In 1891, the school closed and was replaced by the Charlotte Seminary for Girls and relocated to 510 North Tryon Street. In 1896, the Seminary changed names to the Presbyterian College for Women and moved back to the campus shown on the card. In 1912, the name changed again and it became Queens College, moving to Selwyn Avenue in Myers Park in 1912. The building at College and Ninth became College Apartments and operated as such for many years. The Neoclassical building had three-story sections and a dome centered over a Corinthian portico. It had a large entry rotunda, a formal parlor, dormitories, dining room that seated 250 people, a gym and 1000-seat auditorium. The Apartments allowed the auditorium to be used by the Little Theater up into the 1930s. The building was demolished in 1948.
Subject - Location Charlotte (N.C.)
Date of material 1899-1979
Publisher Charlotte, N.C.: J. Murrey Atkins Library University of North Carolina at Charlotte
Genre Picture postcards
Identifier mb121_3_10_51f
Source Mary Boyer Collection. J. Murrey Atkins Library University of North Carolina at Charlotte
Relation Forms part of: Mary Boyer Collection, 1899 - 1979
Contact J. Murrey Atkins Library Special Collections, University of North Carolina at Charlotte: http://specialcollections.uncc.edu
Rights Materials in the Digital Collections at Atkins Library are provided free for educational use under fair use as outlined by current U.S. Copyright law and accompanying guidelines. Written permission from the J. Murrey Atkins Library or the rights holder must be obtained before using an item for publishing or commercial purposes.
Note Atkins Library has provided the description and subject analysis for these images. We encourage you to contact us at spec-coll@uncc.edu with your comments and feedback.
Category Schools
Add tags for Presbyterian College, Charlotte, NC
Post a Comment for Presbyterian College, Charlotte, NC


A few weeks ago, Ausillio flew to England to monitor Manchester City midfielder İlkay Gündoğan who looks to be one of the main targets for the upcoming transfer window.
Gündoğan contract expires in 2020 and today Guardiola said that the German will not be renewing his contract. Inter want to take advantage of this and are ready to offer him a four-year contract worth 6 million per season (similar to what Godin will be earning).
Manchester City are asking for 25 million to sell him this summer but the price is negotiable since the player wants to try a new experience and his contract will be expiring in 12 months.


text:
\section{Introduction}\label{sec:intro}

Inference for spatial random effects models is notoriously difficult, due to the dimensionality of the parameter space and the dependencies in the likelihood surface. Inference can proceed via either Markov chain Monte Carlo \citep{margossian:etal:20} or analytic approximation methods, including variational methods \citep{blei:etal:17} and those based on the Laplace approximation (LA). In this paper, we focus on the latter, since they are fast and are widely-used in practice. In particular, we consider the integrated nested Laplace approximation (INLA) method implemented in the R-INLA package (\cite{rue2009}, \cite{martins2013}) and the LA within the Template Model Builder (TMB) package (\cite{kristensen2016}).

Since its introduction, INLA has increasingly grown in popularity and is now the method of choice for many spatial and spatio-temporal analyses. The method is very well-documented (\cite{rue2009}, \cite{blangiardo2013}, \cite{lindgren2015}, \cite{rue2017}, \cite{martino2020}) and its popularity is evidenced by a number of book-length treatments specifically for spatial data (\cite{blangiardo2015}, \cite{krainski2018}, \cite{moraga2019}, \cite{gomez2020}) along with numerous applications. However, the {\tt R} implementation of INLA does not offer complete flexibility with respect to the likelihood specification.

In this paper, we discuss the utility of the existing random effects modeling tool TMB, developed by \cite{kristensen2016}. TMB is exceptionally flexible, allowing the user to define custom models within a {\tt C++} template, is computationally efficient, and is applicable to a wide class of sampling models. While base TMB, the focus of this paper, provides deterministic approximations to marginal densities, the release of {\tt tmbstan} by \cite{monnahan:kristensen:18} provides an approach to stochastically sample from target distributions defined with TMB templates using no U-turn sampling (NUTS) MCMC allowing for fully Bayesian inference. We focus on TMB since it is less well known to statisticians, though is popular in the ecological literature (\cite{thorson2016}, \cite{bolstad2017}, \cite{free2019}). We provide a detailed description of the methodology and source code to implement a variety of the most popular spatial models. The spatial simulation studies we use to assess TMB and R-INLA adds to a growing body of literature showing the strengths and limitations of these methods (\cite{taylor2014}, \cite{ferkingstad2015}, \cite{auger2017}). They also extend the limited number of studies validating the stochastic partial differential equations (SPDE) approach to fitting continuous spatial models (\cite{teng2017}, \cite{righetto2020}).

The structure of this paper is as follows. In Section \ref{sec:inferential} we describe the models that we are considering, along with an overview of inferential approaches. Sections \ref{sec:inla} and \ref{sec:TMB} and \ref{sec:differences} describe INLA and TMB and summarize their differences, respectively.  Section \ref{sec:sim} compares the two approaches, via extensive simulations, and Section \ref{sec:euro.bc} considers the modeling of European breast cancer data using an incidence/mortality model that can be fitted in TMB, but not in INLA.  We conclude with a discussion in Section \ref{sec:discussion}.

\section{Inferential Overview}\label{sec:inferential}

We will consider the model:
\begin{eqnarray}
  \mbox{\boldmath $y$} | \mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}, \mbox{\boldmath $\phi$}_1 &\sim & p_1(\mbox{\boldmath $y$} | \mbox{\boldmath $\beta$} , \mbox{\boldmath $b$}, \mbox{\boldmath $\phi$}_1 )\label{eq:sampling}\\ \mbox{\boldmath $b$} | \mbox{\boldmath $\phi$}_2 &\sim & p_2(\mbox{\boldmath $b$} | \mbox{\boldmath $\phi$}_2)\label{eq:randomeffects}
\end{eqnarray} where $\mbox{\boldmath $y$}$ represent data, $p_1$ and $p_2$ are the likelihood and random effects distributions, respectively, $\mbox{\boldmath $\beta$}$ represent fixed effects, $\mbox{\boldmath $b$}$ random effects and $\mbox{\boldmath $\phi$}=[\mbox{\boldmath $\phi$}_1,\mbox{\boldmath $\phi$}_2]$ variance-covariance parameters, with $\mbox{\boldmath $\phi$}_1$ appearing in the likelihood and $\mbox{\boldmath $\phi$}_2$ in the prior. The random effects may be split into a set of spatial random effects $\mbox{\boldmath $u$}$ and non-spatial random effects $\mbox{\boldmath $v$}$ so that $\mbox{\boldmath $b$}=[\mbox{\boldmath $u$},\mbox{\boldmath $v$}]$. We consider Gaussian spatial random effects so that $\mbox{\boldmath $b$}|\mbox{\boldmath $\phi$}_2$ falls within the class of latent Gaussian models (LGMs).

A Bayesian approach to inference completes the model (\ref{eq:sampling}) and (\ref{eq:randomeffects}) by adding the hyperprior, $p_3(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$})=p_3(\mbox{\boldmath $\beta$})p_3(\mbox{\boldmath $\phi$})$, with the prior for $\mbox{\boldmath $\beta$}$ assumed to be Gaussian. INLA is generally used to carry out fully Bayesian inference. The automatic differentiation at the heart of TMB allows various inferential approaches. We let,
$$f(\mbox{\boldmath $\beta$},\mbox{\boldmath $b$},\mbox{\boldmath $\phi$}) = \log p_1(y | \mbox{\boldmath $\beta$} , \mbox{\boldmath $b$}, \mbox{\boldmath $\phi$}_1 ) + \log p_2(\mbox{\boldmath $b$} | \mbox{\boldmath $\phi$}_2).$$ The simplest approach is maximum likelihood estimation (MLE) for the fixed effects and variance-covariance parameters (REML is also available for $\mbox{\boldmath $\phi$}$). The MLEs maximize the marginal likelihood,
\begin{equation}
  \label{eq:tmb.marg.lik} \mathcal{L}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$}) = \int \mbox{exp}\left[f(\mbox{\boldmath $\beta$},\mbox{\boldmath $b$},\mbox{\boldmath $\phi$}) \right] ~\dif\mbox{\boldmath $b$}.
\end{equation} Inference may proceed via the asymptotic normal distribution of the MLE with uncertainty expressed via the observed information evaluated at the MLE, as in TMB's methodological ancestor, ADMB (\cite{fournier2012}). Within TMB one may also maximize the marginal posterior,
\begin{equation}
  p( \mbox{\boldmath $\beta$} , \mbox{\boldmath $\phi$} | \mbox{\boldmath $y$} ) \propto \mathcal{L}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$}) \times p_3(\mbox{\boldmath $\beta$})\times p_3(\mbox{\boldmath $\phi$}), \label{eq:marg.post}
\end{equation}
to produce marginal maximum a posteriori (MMAP) estimates for the fixed effects and variance parameters, and  where inference will be based on the asymptotic distribution of the posterior.  In either setting, inference for the random effects then occurs through empirical Bayes by maximization of $f(\widehat\mbox{\boldmath $\beta$},\mbox{\boldmath $b$},\widehat\mbox{\boldmath $\phi$})$ having conditioned on the parameter estimates, $\widehat\mbox{\boldmath $\beta$}$ and $\widehat\mbox{\boldmath $\phi$}$

\section{Integrated Nested Laplace Approximation}\label{sec:inla}

INLA has been well described in a number of statistical venues including recently in \cite{martino2020} and \cite{rue2017}. We provide a brief overview of its details here for comparison against the details of TMB presented in Section \ref{sec:TMB} and to aid in comparing the INLA and TMB simulation results presented in Section \ref{sec:sim}.

We first note that INLA does not use the ``standard" Laplace approximation (which is used in TMB, as described in Section \ref{sec:TMB}), but rather implements the Laplace ratio approximation (LRA) described in Tierney and Kadane (1986, Section 4.1)\nocite{tierney1986} which benefits from some cancellation of approximation error.

INLA was introduced by \cite{rue2009} to provide a quick option for Bayesian computation for the class of additive LGMs (ALGMs). In an ALGM, the general hierarchical model formulation described in (\ref{eq:sampling}) and (\ref{eq:randomeffects}) is restricted to models where the conditional expectation of the observations can be related to a linear combination of the fixed and random effects via a known link function $g$:
\begin{equation}
  \mbox{E}[y_i| \mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}] = g(\eta_i) = g\left(\beta_0 + \sum_{j=1}^J \beta_j z_{ij} + \sum_{k=1}^K b_i^{(k)}\right) \label{eq:lin.pred}
\end{equation}
for observed covariates, $z_{ij}$, associated with the fixed effects, $\beta_j, j=1,\ldots, J$, random effects, $\{b_i^{(k)},\ k=1, \ldots, K\}$, and with $\eta_i$ the linear predictor for each observation $i$. All the parameters of the linear predictor are assumed to be Gaussian, completing the LGM definition.

We introduce a slight change of model formulation, to accommodate the consolidation of the like terms. Specifically, we collect together the Gaussian fixed and random effects, from (\ref{eq:randomeffects}), to write
$$p_2(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$} |\mbox{\boldmath $\phi$}_2) = p_2( \mbox{\boldmath $b$} | \mbox{\boldmath $\phi$}_2 ) \times p_3(\mbox{\boldmath $\beta$}).$$ Defining these terms to be $\mbox{\boldmath $B$} = \left[\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}^{(1)},\ldots,\mbox{\boldmath $b$}^{(K)}\right]$, in the LGM setting we then have
\begin{equation}
  \mbox{\boldmath $B$}|\mbox{\boldmath $\phi$}_2 \sim \mbox{N}\left(\mbox{\bf 0}, \mbox{\boldmath $Q$}^{-1}(\mbox{\boldmath $\phi$}_2)\right) \label{eq:lat.gauss}
\end{equation}
where $\mbox{\boldmath $Q$}^{-1}(\mbox{\boldmath $\phi$}_2)$ is the precision matrix for the Gaussian field. This ensures that the linear predictor $\mbox{\boldmath $\eta$}$ is Gaussian as well. To complete the Bayesian model specification, the hyperprior, $p_3(\mbox{\boldmath $\phi$})$ is specified.

The primary targets of inference for the INLA algorithm are univariate posterior densities for the latent field parameters, $p(B_i|\mbox{\boldmath $y$})$, and the joint posterior of the hyperparameters, $p(\mbox{\boldmath $\phi$}|\mbox{\boldmath $y$})$. INLA approximates these in three steps:
\begin{enumerate}
\item Explore and discretize $\mbox{\boldmath $\phi$}$-space via an approximation,
  \begin{equation} \widetilde{p}(\mbox{\boldmath $\phi$}|\mbox{\boldmath $y$})\propto \frac{p_1(\mbox{\boldmath $y$}|\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}, \mbox{\boldmath $\phi$})p_2(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}|\mbox{\boldmath $\phi$})p_3(\mbox{\boldmath $\phi$})}{\widetilde{p}_G(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}|\mbox{\boldmath $\phi$}, \mbox{\boldmath $y$})}\bigg|_{\scriptsize\mbox{\boldmath $\beta$}=\mbox{\boldmath $\beta$}^*(\mbox{\boldmath $\phi$}), \mbox{\boldmath $b$}=\mbox{\boldmath $b$}^*(\mbox{\boldmath $\phi$})},\label{eq:inla.hyp.approx}
  \end{equation} where $\widetilde{p}_G(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}|\mbox{\boldmath $\phi$}, \mbox{\boldmath $y$})$ is the Gaussian approximation to the conditional distribution obtained by numerically finding and matching the mode, $\{\mbox{\boldmath $\beta$}^*(\mbox{\boldmath $\phi$}), \mbox{\boldmath $b$}^*(\mbox{\boldmath $\phi$})\}$, and curvature at the mode, for given $\mbox{\boldmath $\phi$}$.
  \end{enumerate}

  \noindent The approximation in (\ref{eq:inla.hyp.approx}) is equivalent to the Laplace Ratio Approximation (LRA) for the posterior marginal proposed by \cite{tierney1986}. As the hyperparameter space is explored, (\ref{eq:inla.hyp.approx}) is evaluated at $L$ high-density points to generate an approximate discretization: $\widetilde{p}(\mbox{\boldmath $\phi$}^{(l)}|\mbox{\boldmath $y$})$ at points $\{\mbox{\boldmath $\phi$}^{(1)}, \ldots, \mbox{\boldmath $\phi$}^{(L)}\}$. R-INLA has three available approaches for selecting $\mbox{\boldmath $\phi$}^{(l)}$: the empirical Bayes (EB) option uses only the modal value as a single integration point, the grid method develops a regular grid on the primary orthogonal axis of the hyperparameter space, and the central composite design (CCD) approach which efficiently selects the modal value and a group of `star points' surrounding the center. The default within R-INLA selects the grid option for small numbers of hyperparameters and otherwise selects the CCD method

  \begin{enumerate}[2.]
  \item Approximate $p(B_i|\mbox{\boldmath $\phi$}^{(l)}, \mbox{\boldmath $y$})$ for $l=1, \ldots, L$ using one of three approximations: Gaussian, Laplace, or Simplified Laplace (SL).
  \end{enumerate}

 \noindent   In the Gaussian approximation, $\widetilde{p}_G(B_i|\mbox{\boldmath $\phi$}^{(l)}, \mbox{\boldmath $y$})$ is calculated directly as the marginal of $\widetilde{p}_G(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}|\mbox{\boldmath $\phi$}, \mbox{\boldmath $y$})$, from (\ref{eq:inla.hyp.approx}). While very fast, this approximation is often not particularly good \cite[Section 4.7.2]{blangiardo2015}. In the Laplace approximation, a computationally optimized version of the LRA of \cite{tierney1986} is used,
\begin{equation} \widetilde{p}_L(B_i|\mbox{\boldmath $\phi$}^{(l)}, \mbox{\boldmath $y$}) \propto \frac{p_1(\mbox{\boldmath $y$}|\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}, \mbox{\boldmath $\phi$}^{(l)})p_2(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}|\mbox{\boldmath $\phi$}^{(l)})p_3(\mbox{\boldmath $\phi$})}{\widetilde{p}_G(\mbox{\boldmath $B$}_{-i}|B_i,\mbox{\boldmath $\phi$}^{(l)}, \mbox{\boldmath $y$})}\bigg|_{\scriptsize\mbox{\boldmath $B$}_{-i}=\mbox{\boldmath $B$}_{-i}^*(B_i, \mbox{\boldmath $\phi$}^{(l)})},\label{eq:inla.param.lap.approx}
\end{equation}
 \noindent where $\widetilde{p}_G(\mbox{\boldmath $B$}_{-i}|B_i,\mbox{\boldmath $\phi$}, \mbox{\boldmath $y$})$ is the Gaussian Laplace approximation to $p(\mbox{\boldmath $B$}_{-i}|B_i,\mbox{\boldmath $\phi$}, \mbox{\boldmath $y$})$ with mode $\mbox{\boldmath $B$}_{-i}^*(B_i, \mbox{\boldmath $\phi$})$. This approximation often works very well since the conditional distribution of the latent field parameters are generally close to Gaussian, but it is computationally expensive since it must be recomputed for all desired combinations of $\mbox{\boldmath $B$}$ and $\mbox{\boldmath $\phi$}$. The SL approximation, $\widetilde{p}_{SL}(B_i|\mbox{\boldmath $\phi$}^{(l)}, \mbox{\boldmath $y$})$, uses a Taylor-series approximation of $ \widetilde{p}_L(B_i|\mbox{\boldmath $\phi$}^{(l)}, \mbox{\boldmath $y$}) $. This approximation is quick and accurate for many applications and is the default option within R-INLA. More details on this approximation can be found in \cite[Section 3.2]{rue2009}.

\begin{enumerate}[3.]

\item Approximate the marginal using numerical integration,
\begin{equation} \widetilde{p}(B_i|\mbox{\boldmath $y$}) = \sum_{l=1}^L\widetilde{p}(B_i|\mbox{\boldmath $\phi$}^{(l)}, \mbox{\boldmath $y$})\times\widetilde{p}(\mbox{\boldmath $\phi$}^{(l)}|\mbox{\boldmath $y$})\times\Delta_l,\label{eq:inla.num.int}
\end{equation} over the integration points, $\mbox{\boldmath $\phi$}^{(l)}$, appropriately scaled by their associated weights, $\Delta_l$.
\end{enumerate}

Although INLA returns the univariate marginals, in general we may be interested in functions of the parameters and R-INLA provides a method to sample from the approximate joint posterior, implemented in their sampling function, {\tt inla.posterior.sample()}, using the mixture:
\[
  \widetilde{p}(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}, \mbox{\boldmath $\phi$} |\mbox{\boldmath $y$})\approx\sum_{l=1}^L\widetilde{p}_G(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}, |\mbox{\boldmath $y$}, \mbox{\boldmath $\phi$}^{(l)})\times\widetilde{p}(\mbox{\boldmath $\phi$}^{(l)}|\mbox{\boldmath $y$}).
\]
For each draw, $d$, first a sample, $\mbox{\boldmath $\phi$}^{(d)}$ is drawn from the discretized hyperparameter posterior, $\widetilde{p}(\mbox{\boldmath $\phi$}^{(l)}|\mbox{\boldmath $y$})$, and then a sample, $\{\mbox{\boldmath $\beta$}^{(d)},\mbox{\boldmath $b$}^{(d)}\}$, is drawn from a Gaussian approximation to the joint conditional latent distribution, $\widetilde{p}_G(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}, |\mbox{\boldmath $y$}, \mbox{\boldmath $\phi$}^{(d)})$ which is found to match the mode and curvature at the mode conditional on the specific  hyperparameter draw.  While there is no guarantee that this joint approximation will lead to the same approximate univariate marginals from the full INLA algorithm, by default R-INLA corrects both the mean and the skew of the Gaussian marginals sampled from the joint posterior my mapping to a SkewNormal distribution using the better-approximated marginal posteriors (\cite{wakefield2016}). We note that even when the EB method for the hyperparameter integration is chosen in INLA and there is only one `integration point' for $\mbox{\boldmath $\phi$}$, this approximate joint distribution can still account for skew.

R-INLA only allows for a limited set of built-in likelihoods, and while INLA theoretically can be used on any ALGM, it does have some pragmatic suggestions to ensure reasonable computing times. In particular, a reasonably small dimension of $\mbox{\boldmath $\phi$}$ is required (\cite{rue2017} suggest 2-5 and not more than 20) to minimize the computational burden of the numerical integration in (\ref{eq:inla.num.int}). Crucially, sparsity in the precision of the latent field parameters can be leveraged at numerous points in the algorithm by R-INLA to greatly reduce the computational cost and speed up the approximation. The class of ALGMs is the target of the INLA method because they often permit sparsity and because the posteriors can often be well approximated with the LRA. Spatial statistics applications commonly use Gaussian Markov Random Field (GMRF) model specifications which maintain high levels of sparsity in the precision. A multivariate Gaussian random variable is a GMRF on an undirected graph $G=\{V, E\}$ with vertices, $V$, and edges $E$ if non-zeros in the precision matrix correspond to the edges of $G$. GMRF models are used in the following simulations and in the cancer application.

\section{Template Model Builder}\label{sec:TMB}

TMB, rooted in frequentist inference, requires the user to differentiate between random and non-random effects.  We focus our description of TMB in the context of a frequentist treatment while noting the differences for Bayesian inference.  In contrast to INLA, TMB uses a single LA to integrate out the random effects from the full joint distribution to obtain an approximation to the marginal likelihood in (\ref{eq:tmb.marg.lik}). Defining the conditional mode,
\begin{equation}
  \widehat{\mbox{\boldmath $b$}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$}) := \underset{\mbox{\boldmath $b$}}{\mbox{argmin}}~ -f(\mbox{\boldmath $\beta$}, \mbox{\boldmath $\phi$}, \mbox{\boldmath $b$}),  \label{eq:tmb.re.est}
\end{equation}
TMB approximates (\ref{eq:tmb.marg.lik}) using the LA to marginalize over the random effects:
\begin{eqnarray}
  \mathcal{L}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$}) \approx \tilde{\mathcal{L}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$}) = (2\pi)^{n/2}|\bm{\mathcal{H}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$})|^{1/2}\exp[-f(\mbox{\boldmath $\beta$},\widehat{\mbox{\boldmath $b$}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$}),\mbox{\boldmath $\phi$})], \label{eq:tmb.laplace}
\end{eqnarray} where $ \bm{\mathcal{H}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$})$ is the Hessian of $f(\mbox{\boldmath $\beta$},\widehat{\mbox{\boldmath $b$}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$}) ,\mbox{\boldmath $\phi$})$, with $(j,k)$-th element
\begin{eqnarray}
  \frac{\partial^2}{\partial b_j \partial b_k} f( \mbox{\boldmath $\beta$}, \widehat{\mbox{\boldmath $b$}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$}),\mbox{\boldmath $\phi$}). \label{eq:tmb.hess}
\end{eqnarray}

Estimation in TMB is performed through a two-stage nested optimization procedure which searches for the vector $(\widehat{\mbox{\boldmath $\beta$}}, \widehat{\mbox{\boldmath $\phi$}} )$ maximizing $\tilde{\mathcal{L}}(\mbox{\boldmath $\beta$}, \mbox{\boldmath $\phi$})$ as defined in (\ref{eq:tmb.laplace}). To evaluate $ \tilde{\mathcal{L}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$})$, we need to evaluate both $\widehat{\mbox{\boldmath $b$}}$ in (\ref{eq:tmb.re.est}) and $\bm{\mathcal{H}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$})$ in (\ref{eq:tmb.hess}), but neither are usually available in closed-form. While $\widehat{\mbox{\boldmath $b$}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$})$ may be evaluated through nonlinear optimization of $f$, the crux of the work is to evaluate $\bm{\mathcal{H}}(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$})$. TMB computes gradients and the Hessian using automatic differentiation (\ref{app:ad} presents an overview) performed via {\tt CppAD} \citep{bell2007} and at the same time can auto-detect sparsity in the model in order to leverage sparse matrix routines. TMB reuses the LA and the automatically generated Hessian to produce estimates of the joint covariance between the fixed effects estimates, $\widehat{\mbox{\boldmath $\beta$}}$ and $\widehat{\mbox{\boldmath $\phi$}}$, and the random effects predictors, $\widehat{\mbox{\boldmath $b$}}$, as described in Section \ref{sec:tmb.var}. Inference then proceeds assuming asymptotic normality for the joint distribution of the estimated fixed effects and predicted random effects.

\subsection{The TMB Estimation Algorithm}\label{sec:tmb.alg}
TMB implements a two-step nested optimization routine to iteratively search for the fixed effects estimates with random effects predictions also being produced. Given initial starting values, the routine performs the following steps at each evaluation in the search:

\begin{enumerate}
\item Given current values of the fixed effects parameters, $(\mbox{\boldmath $\beta$}^\star, \mbox{\boldmath $\phi$}^\star)$, perform nonlinear optimization to find updated modal values of the random effects, $\widehat{\mbox{\boldmath $b$}}(\mbox{\boldmath $\beta$}^\star, \mbox{\boldmath $\phi$}^\star)$ as in (\ref{eq:tmb.re.est}) and set this to be the current value of the random effects, $\mbox{\boldmath $b$}^\star$.
\item Given current values of the random effects parameters, $\mbox{\boldmath $b$}^\star$, find the modal values of the Laplace approximation to the marginal likelihood, $\tilde{\mathcal{L}}(\mbox{\boldmath $\beta$}, \mbox{\boldmath $\phi$})$, shown in (\ref{eq:tmb.laplace}) and set them to the current value of the fixed effects, $(\mbox{\boldmath $\beta$}^\star, \mbox{\boldmath $\phi$}^\star)$. In addition, evaluate the gradient of the marginal likelihood to assess stopping conditions.
\item If the maximum gradient component (MGC) of the marginal likelihood is below a stopping threshold the routine stops, otherwise go to step 1.
 \end{enumerate}

 If a stopping criteria was reached, then the final values of the fixed effect and random effects are returned as the fixed effects estimates, $(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}) = (\mbox{\boldmath $\beta$}^\star, \mbox{\boldmath $\phi$}^\star)$, and the random effects predictions are updated one final time to yield $\widehat\mbox{\boldmath $b$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$})$.

 \subsection{The TMB Estimators}\label{sec:tmb.est}

In models without hyperpriors, TMB produces the marginal MLEs of the $\mbox{\boldmath $\beta$}, \mbox{\boldmath $\phi$}$, from the approximate marginal likelihood defined in (\ref{eq:tmb.laplace}). The random effects predictions in this setting, taken as the mode of their conditional distribution, $p(\mbox{\boldmath $b$}|\widehat{\mbox{\boldmath $\beta$}}, \widehat{\mbox{\boldmath $\phi$}}_1, \mbox{\boldmath $y$})$, using `plug-in' estimates of the fixed effects MLEs, are empirical Bayes predictors (\cite{devalpine2009}).

In models with hyperpriors, TMB produces the marginal maximum a posteriori estimates of $\mbox{\boldmath $\beta$}, \mbox{\boldmath $\phi$}$, again using the Laplace approximation from (\ref{eq:tmb.laplace}) applied to the marginal posterior in (\ref{eq:marg.post}). With priors, basing random effects inference on the estimated posterior, $p(\mbox{\boldmath $b$}|\widehat{\mbox{\boldmath $\beta$}},\widehat{\mbox{\boldmath $\phi$}}_1,\mbox{\boldmath $y$})$, where the hyperparameters have been estimated using the data, TMB provides parameteric empirical Bayes (PEB) estimates \cite[Chapter 3.3]{carlin2000}. In essence, this replaces the posterior, $$p(\mbox{\boldmath $b$}|\mbox{\boldmath $y$}) = \int p(\mbox{\boldmath $b$}|\mbox{\boldmath $y$}, \mbox{\boldmath $\beta$}, \mbox{\boldmath $\phi$})p(\mbox{\boldmath $\beta$},\mbox{\boldmath $\phi$}|\mbox{\boldmath $y$})~\dif\mbox{\boldmath $\beta$}\dif\mbox{\boldmath $\phi$},$$ with $p(\mbox{\boldmath $b$}|\widehat{\mbox{\boldmath $\beta$}},\widehat{\mbox{\boldmath $\phi$}}_1,\mbox{\boldmath $y$})$ where $\widehat{\mbox{\boldmath $\beta$}}$ and $\widehat{\mbox{\boldmath $\phi$}}_1$ are MMAP estimates. The EB option for the numerical integration in (\ref{eq:inla.num.int}) in the INLA algorithm makes the same tradeoff: skipping the computational complexity of the integration by not accounting for the uncertainty in the (hyper)parameters.

While the random effects predictors are fast to compute via optimization having conditioned on the fixed effects estimates, with the number of REs increasing proportionally to the number of data observations, EB predictions of random effects generally have no guarantees of consistency \citep{thorson2016}. In order to improve the predictions for the random effects and differentiable functions of the mixed effects, $\mbox{\boldmath $d$}(\mbox{\boldmath $\beta$}, \mbox{\boldmath $\phi$}, \mbox{\boldmath $b$})$, TMB implements a generic bias-correction adjustment termed the ``epsilon'' method in \cite{thorson2016}. The correction is applicable to quantities which depend on random effects, $\mbox{\boldmath $d$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$})$, and has no effect on the fixed effects estimates. This correction is based an approximation to moment-generating functions, using $M(\mbox{\boldmath $\epsilon$})=E[\text{exp}\{\mbox{\boldmath $\epsilon$} \mbox{\boldmath $s$}\cdot \mbox{\boldmath $d$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$},\mbox{\boldmath $b$})\}]$, with $\mbox{\boldmath $s$}\cdot$ denoting the dot product, proposed by \cite{tierney1989} (Section 3.1).  Introducing an auxiliary parameter vector, $\mbox{\boldmath $\epsilon$}$, of dimension equal to the dimension of $\mbox{\boldmath $d$}()$, consider a new function with the auxiliary parameters:
\begin{eqnarray}
  \label{eq:tmb.nuisance}
  \mbox{\boldmath $e$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$}, \mbox{\boldmath $\epsilon$} |\mbox{\boldmath $y$}) =  \text{log} \Big( \int \text{exp}\big(f(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$}) + \mbox{\boldmath $\epsilon$}\mbox{\boldmath $s$}\cdot\dif(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$}) \big)~\dif\mbox{\boldmath $b$} \Big).
\end{eqnarray}
Employing the namesake method of moment-generating functions, the form of the bias-corrected estimator is found by differentiating with respect to $\mbox{\boldmath $\epsilon$}$ and then setting $\mbox{\boldmath $\epsilon$}=\mbox{\boldmath $s$} 0$:
\begin{eqnarray}
  \frac{\partial}{\partial\mbox{\boldmath $\epsilon$}}\left(\mbox{\boldmath $e$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$}, \mbox{\boldmath $\epsilon$} |\mbox{\boldmath $y$}) \right)\biggm\lvert_{\mbox{\boldmath $\epsilon$} = \mbox{\boldmath $s$} 0} &= &\frac{\int\text{exp}\left(f(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$},\mbox{\boldmath $b$})  + \mbox{\boldmath $\epsilon$}\mbox{\boldmath $s$}\cdot\mbox{\boldmath $d$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$})\right)\mbox{\boldmath $d$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$})~\dif\mbox{\boldmath $b$}}{\int\text{exp}\left(f(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$},\mbox{\boldmath $b$}) + \mbox{\boldmath $\epsilon$}\mbox{\boldmath $s$}\cdot\mbox{\boldmath $d$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$})\right)~\dif\mbox{\boldmath $b$}} \biggm\lvert_{\mbox{\boldmath $\epsilon$} = \mbox{\boldmath $s$} 0}\nonumber \\
&= &\frac{\int\text{exp}\left(f(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$},\mbox{\boldmath $b$}) \right) \mbox{\boldmath $d$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$})~\dif\mbox{\boldmath $b$}}{\int\text{exp}\left(f(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$},\mbox{\boldmath $b$}) \right)~\dif\mbox{\boldmath $b$}}  \label{eq:tmb.bias.correct2} \\
  &= &\text{E}[\mbox{\boldmath $d$}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $b$})\lvert \mbox{\boldmath $y$}]. \nonumber
\end{eqnarray}

\noindent This identity allows TMB to improve the predictions for the random effects ($\mbox{\boldmath $d$}(\cdot)$ is the identity) and nonlinear functions of the mixed effects using the gradient of the approximate marginal likelihood with respect to the auxiliary parameters, $\mbox{\boldmath $\epsilon$}$, leveraging its available tools: {\tt CppAD} to automatically evaluate derivatives and the LA to approximate the integrals. \cite{thorson2016} note that this is only a bias correction algorithm because the LA used to evaluate both numerator and denominator in (\ref{eq:tmb.bias.correct2}) will be inexact unless the likelihood, conditional on the fixed effects, is multivariate Gaussian.

\subsection{Variance of TMB Estimators}\label{sec:tmb.var}

TMB approximates the covariance of the fixed effects using the inverse of the observed Hessian of the log-likelihood:

\begin{equation}
   \mbox{\boldmath $\Sigma$}_{\scriptsize \widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}}= \text{Cov}(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}) = \left(-\nabla^2 \text{log}\tilde{\mathcal{L}}(\widehat\mbox{\boldmath $\beta$},\widehat\mbox{\boldmath $\phi$})\right)^{-1}.
\end{equation}

In models that include random effects, the joint covariance of the fixed and random effects is approximated using an application of the law of total variance and a linearization:
\begin{eqnarray}
  \label{eq:tmb.joint.var}
   \mbox{\boldmath $\Sigma$}_{\scriptsize \widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \widehat\mbox{\boldmath $b$}}= \text{Cov}\begin{pmatrix} \widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}\\ \hat{\mbox{\boldmath $b$}}\end{pmatrix}
  &=&\text{E}\left[\text{Cov}\left(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \hat{\mbox{\boldmath $b$}} \middle\vert \, \widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$} \right)\right] +
    \text{Cov}\left[\text{E}\left(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \hat{\mbox{\boldmath $b$}} \middle\vert \, \widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$} \right)\right] \nonumber\\
  &\approx&
    \begin{pmatrix}
      0 & 0\\
      0 & \bm{\mathcal{H}}_{bb}^{-1}(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}, \mbox{\boldmath $\phi$})
    \end{pmatrix} +
    \mbox{\boldmath $s$} J \mbox{\boldmath $\Sigma$}_{\scriptsize \widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}}\mbox{\boldmath $s$} J^{\text{T}}
\end{eqnarray}

\noindent where $\bm{\mathcal{H}}_{bb}(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}, \mbox{\boldmath $\phi$})$ is the random effects sub-matrix of the full joint Hessian of $f(\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$}, \mbox{\boldmath $\phi$})$, and $\mbox{\boldmath $J$}$ is the Jacobian of the vector $\left(\mbox{\boldmath $\theta$}, \hat{\mbox{\boldmath $b$}}(\mbox{\boldmath $\theta$}) \right)^{\text{T}}$ with respect to $\mbox{\boldmath $s$}\theta$. The $\delta$-method is used to find the joint covariance of differentiable functions of the mixed effects:
  \begin{eqnarray}
  \label{eq:tmb.joint.delta}
  \text{Cov}\left(\mbox{\boldmath $d$} (\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \hat{\mbox{\boldmath $b$}})\right)
    &= \nabla \mbox{\boldmath $d$}\,\mbox{\boldmath $\Sigma$}_{\scriptsize \widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \hat{\mbox{\boldmath $b$}}} \nabla \mbox{\boldmath $d$}^{\text{T}},
  \end{eqnarray}
\noindent where $\nabla \mbox{\boldmath $d$}$ is the Jacobian of $\mbox{\boldmath $d$}$. For models with only fixed effects, this simplifies to
\begin{eqnarray}
  \label{eq:tmb.fe.cov}
  \text{Cov}\left(\mbox{\boldmath $s$} d(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$})\right)=\nabla \mbox{\boldmath $d$}\    \mbox{\boldmath $\Sigma$}_{ \scriptsize \widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}} \nabla \mbox{\boldmath $d$}^{\text{T}}.
\end{eqnarray}

Much like the correction for the random effects predictions, TMB can improve the covariance estimator of the random effects prediction and functions of the mixed effects shown in (\ref{eq:tmb.joint.delta}) using the second derivative of $\mbox{\boldmath $s$} e(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $u$}, \mbox{\boldmath $\epsilon$} |\mbox{\boldmath $y$})$ from (\ref{eq:tmb.nuisance}). The form of the improved variance estimator again uses the law of total variance:

\begin{eqnarray}
  \label{eq:tmb.var.correct}
  \text{Cov}\left(\mbox{\boldmath $s$} d (\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \widehat{\mbox{\boldmath $b$}})\right) &=&
  \biggl[\frac{\partial^2}{\partial^2\mbox{\boldmath $s$}\epsilon}\left(\mbox{\boldmath $s$} e(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $s$} b, \mbox{\boldmath $s$}\epsilon |\mbox{\boldmath $s$} y) \right) + \nonumber \\
&&\frac{\partial}{\partial\mbox{\boldmath $s$}\theta} \frac{\partial}{\partial\mbox{\boldmath $s$}\epsilon}\left(\mbox{\boldmath $s$} e(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $s$} b, \mbox{\boldmath $s$}\epsilon |\mbox{\boldmath $s$} y) \right)^{\text{T}}~
       \mbox{\boldmath $\Sigma$}_{\scriptsize \widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}}~
    \frac{\partial}{\partial\mbox{\boldmath $s$}\theta} \frac{\partial}{\partial\mbox{\boldmath $s$}\epsilon}\left(\mbox{\boldmath $s$} e(\widehat\mbox{\boldmath $\beta$}, \widehat\mbox{\boldmath $\phi$}, \mbox{\boldmath $s$} b, \mbox{\boldmath $s$}\epsilon |\mbox{\boldmath $s$} y) \right)\biggr]_{\mbox{\boldmath $s$}\epsilon = \mbox{\boldmath $s$} 0},
\end{eqnarray}

\noindent where the first term on the right-hand side works out to be the standard variance estimator for random effects conditional on the fixed effects (demonstrated analogously to the derivation in (\ref{eq:tmb.bias.correct2})), and the second term accounts for having conditioned on the fixed effects estimators. This is an improvement over the naive EB variance estimators which ignore the conditioning on the fixed effects \cite[Chapter 3.5]{carlin2000}.

\section{Contrasting TMB and INLA}
\label{sec:differences}

For clarity, Table~\ref{tab:differences} provides a summary of the primary differences - and similarities - between TMB and INLA that were discussed in detail in the previous two sections.

TMB and INLA will yield the most similar results under the following conditions:
\begin{itemize}
\item TMB is coded to optimize the marginal posterior using the same priors and with the same internal parameter representations used by R-INLA,
\item TMB uses both bias and variance corrections to functions of REs,
\item R-INLA uses the Gaussian approximations and the empirical Bayes `integration' strategy,
\item R-INLA does not use the bias and skew corrections when drawing posterior samples.
\end{itemize}

\begin{table}[h!]
  \begin{centering}

    \begin{tabular}{p{.31\textwidth}|p{.34\textwidth}|p{.34\textwidth}|}
      \cline{2-3}
                                            & \hspace{4em} TMB                           & \hspace{3.7em} INLA \\
      \hline

      \multicolumn{1}{|l|}{\multirow{1}[15]{*}{Inferential Method: FE}}  & Frequentist: MLEs from marginal likelihood (or MMAP from marginal posterior) & \multirow{2}[15]{*}{Full Bayesian (or PEB)} \\
      \cline{1-2}
      \multicolumn{1}{|l|}{\multirow{1}{*}{Inferential Method: RE}} & Frequentist: EB (or PEB) &  \\

      \hline \hline

      \multicolumn{1}{|l|}{\multirow{1}[5]{*}{Laplace approximation}} & Standard Gaussian approximation & LRA from \cite{tierney1986}\\
      \hline
      \multicolumn{1}{|l|}{\multirow{1}[5]{*}{Bias Corrections}} & Yes, to functions of REs & Applied to align joint posterior samples with marginal distributions\\
      \hline
      \multicolumn{1}{|l|}{\multirow{1}{*}{Var. Corrections}} & Yes, to functions of REs  & No \\
      \hline
      \multicolumn{1}{|l|}{\multirow{1}{*}{Skew Corrections}} & No & Applied to align joint posterior samples with marginal distributions \\
      \hline
      \multicolumn{1}{|l|}{\multirow{1}{*}{Hyperpar. Integration}} & No                            & Yes (no) \\

      \hline \hline

      \multicolumn{1}{|c|}{\multirow{1}[18]{*}{Inferential sampling}} & Gaussian centered at point estimates and covariance from observed information and linearization & Gaussian approximation to joint posterior with mean and skew corrections applied to marginals\\

      \hline \hline

      \multicolumn{1}{|c|}{\multirow{1}{*}{Hessian Evaluation}} & Automatic Differentiation & Finite Differences \\
      \hline
      \multicolumn{1}{|c|}{\multirow{1}{*}{Sparse Matrices}} & Yes & Yes \\
      \hline
      \multicolumn{1}{|c|}{\multirow{1}{*}{Parallelization}} & Yes & Yes \\
      \hline
    \end{tabular}

  \end{centering}
  \caption{Summarization of primary differences between TMB and INLA. Entries in (parentheses) indicate outcomes from TMB for models that include priors and indicate outcomes from INLA under eb `numerical integration' over the hyperparameters. The table is split into sections corresponding to methods, approximations, post-model sampling, and computation. FE=fixed effects, RE=random effects, MMAP=marginal maximum a posterior estimates, PEB = parametric Empirical Bayes, LRA = Laplace ratio approximation.}  \label{tab:differences}
\end{table}

\section{Spatial Simulation Study}
\label{sec:sim}

We performed two simulation studies on popular continuous and discrete spatial smoothing models to assess the ability of TMB and R-INLA to estimate the total spatial field effects, the parameters and hyperparameters, and how their computational performance scales as data volume and random effects dimension increase.

The second purpose of the continuous simulation study was to perform a thorough assessment the popular stochastic partial differential equations (SPDE) representation fitting GPs which uses a finite element method over a triangulation to solve a particular SPDE whose solution is known to have Mat\'ern covariance \citep{stein:1999}. For details on the SPDE approximation, see \cite{lindgren2010} and \cite{miller2020}. The discrete simulation study was included in part to assess and verify inference in TMB using hard constraints on the random effects parameters.

For each study, a grid of experiments, shown in Tables~\ref{tab:sim_params} and \ref{tab:disc_sim_params}, was defined. Each level of each experiment was replicated 25 times to obtain Monte Carlo errors on the validation metrics. For each replicate within each experiment, completely new spatial fields, sampling locations, and observations were generated. For each of these, TMB and R-INLA algorithms were run, and inference and validation was performed using 500 joint samples drawn from each model, projected to 5$\times$5 km$^2$ raster grids in the case of the continuous simulation, and compared against against the truth. For both models, the internal representation of parameters in TMB were coded to align with those used by R-INLA. TMB was always run using its bias correction method and the improved variance estimates, and R-INLA joint estimates were generated using its available mean bias correction. All of the simulation analyses use the empirical distribution taken across the 25 replicates of each experimental level.

\subsection{Continuous Spatial Simulation}
\label{sec:sim.cont}

This simulation was designed with respect to three governing motivations that dictated the choice of models, covariates, and true parameter ranges: (1) vetting TMB and INLA inference while (2) assessing the SPDE approximation in a variety of settings (3) using simulated risk fields and data akin to those commonly see in public health settings. Although motivated by public health applications, the broad range the parameters, such as the maximum number of simulated data locations, greatly extend this study's applicability beyond any one applied domain.

We consider Gaussian process (GP) models with mean $\mu(\mbox{\boldmath $s$})$ at location $\mbox{\boldmath $s$}$, and the Mat\'ern function where the covariance between two spatial locations distance $||\mbox{\boldmath $s$}_i - \mbox{\boldmath $s$}_j||$ from one another is:
\begin{equation}
  C(u(\mbox{\boldmath $s$}_i), u(\mbox{\boldmath $s$}_j)) = \frac{\sigma_\text{m}^2 }{2^{\nu-1}\Gamma(\nu)}\left( \kappa||\mbox{\boldmath $s$}_i - \mbox{\boldmath $s$}_j|| \right)^{\nu}K_{\nu}\left(\kappa||\mbox{\boldmath $s$}_i - \mbox{\boldmath $s$}_j|| \right), \label{eq:matern.app}
\end{equation} where $\sigma_\text{m}^2$ is the variance, $\kappa > 0$ is a scaling parameter related to the range, $r_m=\frac{\sqrt{8\nu}}{\kappa}$, defined to be the distance at which the spatial correlation drops to 0.1, $\nu > 0$ is related to the smoothness of the field, and $K_\nu$ is the modified Bessel function of the second kind. For any finite collection of locations within the domain, $\{\mbox{\boldmath $s$}_1, \ldots, \mbox{\boldmath $s$}_{n}\}\in\mathcal{D}$, the random vector $\mbox{\boldmath $u$} = [u(\mbox{\boldmath $s$}_1),\ldots,u(\mbox{\boldmath $s$}_{n})]$ has multivariate Gaussian distribution with precision matrix $\mbox{\boldmath $Q$}$.

The simulated data, observed at locations $\mbox{\boldmath $s$}_i,\ i=1, \ldots, n_s,$ within Nigeria and selected using a stratified spatial sampling design, arise from the following hierarchical model:
\begin{eqnarray*}
  \mbox{\boldmath $y$} |\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$} ,\mbox{\boldmath $\phi$}_1&\sim& p_1(\mbox{\boldmath $y$} |\mbox{\boldmath $\beta$}, \mbox{\boldmath $b$},\mbox{\boldmath $\phi$}_1) \label{eq:sim_hier} \\
  \mbox{E}[y_i|\mbox{\boldmath $\beta$},u_i , v_i] &=& g^{-1}\left(\alpha+\mbox{\boldmath $z$}^{\tiny{T}}_i \mbox{\boldmath $\beta$}+ u_i + v_i\right)\nonumber\\
  \mbox{\boldmath $u$} &\sim &\mbox{N}(\mbox{\bf 0}, \mbox{\boldmath $Q$}(r_\text{m}, \sigma^2_\text{m}))\nonumber\\
  \mbox{\boldmath $v$} &\sim& \mbox{N}(\mbox{\bf 0},\mbox{\boldmath{I}}_{n_s} \sigma^2_\text{clust}). \nonumbe
\end{eqnarray*} where $\alpha$ is the intercept and is fixed to $-1$ across all simulations, $\mbox{\boldmath{I}}_{n_s}$ is the $n_s \times n_s$ identity matrix and the last two lines correspond to $p_2(\mbox{\boldmath $b$} | \mbox{\boldmath $\phi$}_2)$ with $\mbox{\boldmath $b$}=[\mbox{\boldmath $u$},\mbox{\boldmath $v$}]$ and $\mbox{\boldmath $\phi$}_2=[\sigma^2_\text{m},r_\text{m},\sigma^2_\text{clust}]$.  The precision for the spatial GP, $\mbox{\boldmath $Q$}$ is Mat\'ern with range $r_\text{m}$ and standard deviation $\sigma_\text{m}$. In some models, two spatially varying covariates were included: access time to heath care (\cite{weiss2018}) and malaria incidence (\cite{weiss2019}). While the SPDE representation is used to fit the spatial fields (using three different triangulation resolutions shown in Figure \ref{fig:meshes}), the true fields are simulated directly on a high resolution regular grid using the {\tt RandomFields R} package \citep{schlather2015}.

The domain, covariates and stratified sampling scheme were chosen to represent public health datasets, such as the Demographic and Health Surveys (DHS) and UNICEF Multiple Indicator Cluster Surveys, which are increasingly being used to predict continuous spatial(-temporal) maps of health outcomes. Our stratified cluster sampling design mimics the one used by DHS which stratifies by regions (administrative level 1 units) and urban/rural status and usually collects observations at 250-750 clusters, with typically 25-35 households sampled within each cluster. The covariates, access (travel time) to health care and malaria incidence, are both reasonable choices to be correlated with heath risks and were further chosen for their different spatial characteristics, and the magnitude of the GP and total field were selected to yield  moderately rare outcomes. Both Binomial and Gaussian data are commonly collected in a variety of applications (Poisson likelihoods are examined in the discrete simulation), and the binary data provide an extra challenge for both TMB and INLA. The Mat\'ern range was selected to represent medium-small and medium-large spatial ranges over the approximate $10\times 10$ (degrees latitude-longitude) domain. The Mat\'ern variance takes two values representing small- and large-scale spatial effects relative to the covariate effects and the small, medium, and large values of the $iid$ cluster variance and Gaussian observation variance. The number of vertices in the SPDE triangulation were selected to represent medium, fine, and very fine meshes, relative to the domain and resolution of the $5\times 5$km raster representation, and to assess the computational scaling of TMB and R-INLA. While the INLA method with empirical Bayes integration and Gaussian approximations are closest to the methods in TMB, we also evaluate some of the more accurate options in order to asses the default (and better) approximations available in R-INLA.

The (nearly) full combinatorial grid of simulation parameters shown in Table~\ref{tab:sim_params} comprised the set of 16128 experiments (R-INLA with 8000 clusters and the full Laplace approximation was excluded due to computational time constraints and the Gaussian variances were not varied for Binomial observations). We note that the internal representation of parameters in TMB were coded to align with those used by R-INLA. Additional details including prior specification, the stratified spatial sampling scheme, and SPDE mesh generation may be found in \ref{app:sim.cont}.

\begin{table}[htb!]
  \begin{footnotesize}
    \begin{tabular}{ll}
      \begin{tabularx}{1.0\textwidth}{|l|X|}
        \hline
        Parameter & Simulation Values \\
        \hline \hline
        Data Observations & Binomial, Gaussian \\ \hline
        Gaussian Observation Variance, $\sigma_{obs}^2$ & $0.1^2, 0.2^2, 0.4^2$\\ \hline
        Covariates & None, $(-.25\times \mbox{access} + .25\times\mbox{Malaria Incid.})$ \\ \hline
        Number of Clusters, $n_{\mbox{\boldmath $s$} s}$ & 250, 500, 750, 1000, 2000, 4000, 8000 \\ \hline
        Expected Samples per Cluster, E$[n_i]$ & 35\\ \hline
        Spatial Range (lat-lon degrees) & 1, $\sqrt{8}$ \\ \hline
        Spatial Variance & $0.25^2, 0.5^2$\\ \hline
        Cluster Variance, $\sigma_{clust}^2$  & 0, $0.1^2, 0.2^2, 0.4^2$\\ \hline
        Num. Nodes in SPDE Mesh & 3631, 7922, 13869 (low, medium, high resolution)\\ \hline
        R-INLA Integration Strategy & Empirical Bayes ({\tt EB}), Central Composite Design ({\tt CCD})\\ \hline
        R-INLA Approximation Strategy & {\tt G}aussian, {\tt S}implified Laplace, {\tt L}aplace\\ \hline
      \end{tabularx}
    \end{tabular}
    \caption{ \footnotesize{ Parameters varied across the continuous simulation experiments. The total number of experiments was 16128 (Gaussian variance was not varied for Binomial experiments), each replicated 25 times.
        \label{tab:sim_params}}}
  \end{footnotesize}
\end{table}

\subsubsection*{Selected Continuous Simulation Results}
\label{sec:sim.cont.results}

The simulation comparisons presented in this section compare results from TMB against those from R-INLA using the default options: the simplified Laplace approximation and the CCD numerical integration scheme with mean and skew corrections to the marginals of the joint posterior sampling distribution (see Section~\ref{sec:inla}). While using the grid numerical integration scheme and the full Laplace approximations could offer an overall better approximation, and the empirical Bayes integration with Gaussian approximations would be closest to the approximations in TMB, the default INLA options provide a nice balance between computation and accuracy and are what many users explicitly or implicitly choose to use. For these reasons we felt this setting to be a useful and fair benchmark for which to contrast TMB. The extended online results also include comparisons with other combinations of INLA approximations and a number of additional selections are included in \ref{app:sim.cont}.

Additional continuous simulation results for all experiments in Table~\ref{tab:sim_params} can be explored via an interactive R-Shiny web application by visiting \href{https://faculty.washington.edu/jonno/software.html}{faculty.washington.edu/jonno/software.html}.

The following four figures display:
\begin{itemize}
\item Figure \ref{fig:binom.par.bias}: Binomial scenarios' parameter bias for the intercept, access and malarian incidence coefficients, cluster variance, and the Mat\'ern standard deviation and range.
\item Figure \ref{fig:binom.dist.cov}: Binomial, medium-resolution mesh, scenarios' mean pixel coverage, stratified by the value of the true GP, and faceted by cluster variance and number of spatial observations.
\item Figure \ref{fig:normal.dist.cov}: Normal $\sigma_{obs}^2=0.04$, medium-resolution mesh, scenarios' mean pixel coverage, stratified by the value of the true GP, and faceted by cluster variance and number of spatial observations.
\end{itemize}

\begin{figure}[hb!] \centering \includegraphics[width=\linewidth]{figures/cont_sim_res/plot_03_binom_param_bias_summary__TMB_and_INLA_CCD_S_80_intervals.png}
\caption{ \footnotesize{ Comparison of the estimated parameter bias from TMB (dashed lines) and R-INLA using CCD hyperparameter integration and simplified Laplace approximations (solid lines) plotted against the number of cluster observations for Binomial observation experiments. Colors represent different cluster (i.i.d nugget) variances used in an experiment. Each point is the median bias of 3 experiments (coarse, medium, and fine SPDE triangulation), calculated across 75 replicates, and the bars represent the middle 80\% quantile range of the bias across replicates. \label{fig:binom.par.bias}}}
\end{figure}

\begin{figure}  \vspace{-3em} \centering \includegraphics[width=\linewidth]{figures/cont_sim_res/plot_04_binom_7922_meshnodes_gp_coverage_decile_summary__TMB_and_INLA_CCD_S_80_intervals.png}
  \caption{ \footnotesize{ Comparison of the average estimated field coverage of the simulated truth, faceted by cluster (i.i.d. nugget) variance and the number of clusters, from TMB (dashed lines) and R-INLA using CCD hyperparameter integration and simplified Laplace approximations (solid lines) plotted against the target nominal coverage, $\alpha$, for Binomial observation experiments with the medium resolution SPDE triangulation. Colors stratify pixels included in the average coverage calculation by the decile of the true GP for the experiment replicate. Each point is the median average coverage of an experiment, calculated across 25 replicates, and the bars represent the middle 80\% quantile range of the average coverage across replicates.\label{fig:binom.dist.cov}}}
\end{figure}

\begin{figure} \vspace{-3em}
  \centering \includegraphics[width=\linewidth]{figures/cont_sim_res/plot04trouble01.png}
   \caption{ \footnotesize{ Comparison of the average estimated field coverage of the simulated truth, faceted by cluster (i.i.d. nugget) variance and the number of clusters, from TMB (dashed lines) and R-INLA using CCD hyperparameter integration and simplified Laplace approximations (solid lines) plotted against the target nominal coverage, $\alpha$, for Gaussian observation experiments with $\sigma^2 =0.04$ and the medium resolution SPDE triangulation. Colors stratify pixels included in the average coverage calculation by the decile of the true GP for the experiment replicate. Each point is the median average coverage of an experiment, calculated across 25 replicates, and the bars represent the middle 80\% quantile range of the average coverage across replicates.\label{fig:normal.dist.cov}}}
\end{figure}

\clearpage

While the overall results are quite similar, TMB generally has larger bias in the fixed effects estimators, particularly hyperparameters which may deviate further from Gaussianity, as shown in the continuous Binomial experiments, Figures \ref{fig:binom.par.bias}, and the continuous Gaussian experiments, Figure \ref{fig:normal.par.bias}. This trend appears consistently across a variety of INLA options, Figures \ref{fig:binom.par.bias.eb.g}-\ref{fig:binom.par.bias.ccd.l}.

In spatial statistics settings, the hyperparameters (and sometimes all parameters) may not be of inferential interest. Figures \ref{fig:binom.dist.cov} and \ref{fig:normal.dist.cov} demonstrate that TMB consistently yields results very similar to those from R-INLA at the spatial field level, and that the results are similar across all ranges of the spatial effect. Figures \ref{fig:binom.pix.cov} and \ref{fig:normal.pix.cov} show these results collapsed across the GP magnitude. In contrast to R-INLA, TMB seems to consistently have slightly lower coverage which could be attributed to the lack of integration over the hyperparameters even though the covariance estimator in (\ref{eq:tmb.var.correct}) attempts to account for this.

In the Binomial experiment, we saw no notable differences in the spatial field coverage across the different resolutions of the SDDE triangulation suggesting that the approximation was appropriately resolved. In the Gaussian data setting, we observed that the coarser meshes undercovered the field estimates in experiments with small $\sigma^2_{clust}$ and large sample sizes - but this was mostly remedied at the finer triangulation resolution. Interestingly, we saw more severe field undercoverage for larger numbers of spatial observations. These patterns were observed in results from both TMB and INLA. See the lower left plots of Figures \ref{fig:normal04.dist.cov.c}-\ref{fig:normal04.dist.cov.f}.

\clearpage
\subsection{Discrete Spatial Simulation}
\label{sec:sim.disc}

For the discrete simulation study, we considered the BYM2 model (a modern formulation of the classic Besag-York-Mollie model developed by \cite{riebler2016}). This model requires a sum-to-zero constraint a hard constraint, using appropriate conditional densities \cite[Section 12.1.7.4]{gelfand2010}, was implemented in TMB to match the linear constraint used in the R-INLA BYM2 model formulation.

The discrete model was implemented over the 37 regions (first-level administrative units) of Nigeria, and neighbors were defined by to be regions with shared boundaries. Within each region, the population, $n_s$, for that area was first sampled from iid Poisson distributions. Conditional on the population, poisson observations were simulated from the following hierarchical model:
\begin{eqnarray}
  y_i|n_s, \eta_i &\sim &\text{Poisson}(n_s\times \eta_i)   \label{eq:disc.sim.hier} \\
  \eta_i &= &\text{exp}\left(\alpha + \mbox{\boldmath $s$} b_i \right)\nonumber\\
  \mbox{\boldmath $s$} b &= &\frac{1}{\sqrt{\tau}}\left( \sqrt{1-\varphi}\boldsymbol v + \sqrt{\varphi}\boldsymbol u_{\star} \right) \nonumber \\
  \mbox{\boldmath $s$} v &\sim &N\left(\mbox{\boldmath $s$} 0, \mbox{\boldmath $s$} I\right)\nonumber \\
  \mbox{\boldmath $s$} u_\star &\sim &N\left(\mbox{\boldmath $s$} 0, \mbox{\boldmath $s$} Q_{\star}^{-1} \right), {\text s.t. }   \sum_{i=1}^{37} u_{\star_i} = \mbox{\boldmath $s$} 0, \nonumber
\end{eqnarray}
\noindent with $\alpha$ the GMRF intercept, fixed at -3 across simulations, BYM2 field $\mbox{\boldmath $s$} b$ with total variance $\tau^{-1}$, mixing parameter $\varphi$ controlling the contribution of $\mbox{\boldmath $s$} v$, the unstructured i.i.d. portion of the BYM2 field, and $\mbox{\boldmath $s$} u_{\star}$, the scaled spatially structured component of the BYM2. The structured portion of the BYM2 is specified with precision $\mbox{\boldmath $s$} Q_\star$, a scaled version of the precision from the classic BYM ICAR model, and is constrained to sum to zero.

The full combinatorial grid of simulation parameters is shown in Table~\ref{tab:disc_sim_params} consisted of a set of 20 experiments, with each experiment replicated 25 times to obtain Monte Carlo errors on the validation metrics. Additional details including prior specification and the constrained random effects density may be found in \ref{app:sim.disc}.

\begin{table}[h!]

\makebox[1 \textwidth][c]{
\resizebox{1\textwidth}{!}{

  \begin{tabularx}{1.\textwidth}{|l|X|}
    \hline
    Parameter    &  Simulation Values \\
    \hline
    \hline
    Data Observations & Poisson \\
    \hline
    Mean Observations per Region, $\text{E}[n_{\mbox{\boldmath $s$} s}]$ & 16, 36, 49, 100, 400\\
    \hline
    BYM2 $\varphi$ & 1, $0.25, 0.5, 0.75, 0.9$ \\
    \hline
    BYM2 Variance, $(\tau^{-1})$ & $0.5$\\
    \hline
    GMRF Intercept & -3\\
    \hline
    R-INLA Integration Strategy  & Central Composite Design ({\tt CCD})\\
    \hline
    R-INLA Approximation Strategy  & {\tt S}implified Laplace\\
    \hline
  \end{tabularx}
}}
  \renewcommand{\baselinestretch}{1.0} \caption{ \footnotesize{ Parameters varied across the discrete simulation
      experiments. The total number of experiments was 20, and each
      was replicated 25 times.
      \label{tab:disc_sim_params}}}
\end{table}

\subsubsection*{Discrete Simulation Results}
\label{sec:sim.disc.results}

\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{figures/disc_sim_res/avg_param_bias.png}
  \renewcommand{\baselinestretch}{1.0} \caption{ \footnotesize{ Comparison of the estimated
      parameter bias from TMB (red) and R-INLA (blue) plotted
      against the number of observations per region for Poisson data
      experiments with varying values of the true BYM2 $\varphi$. Each point
      is the median bias of 1 experiments, calculated across 25
      replicates, and the bars represent the middle 80\% quantile
      range of the bias across replicates. \label{fig:disc.bias}}}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{figures/disc_sim_res/avg_region_coverage.png}
  \renewcommand{\baselinestretch}{1.0} \caption{ \footnotesize{ Comparison of the average
      estimated region coverage of the simulated truth, faceted by
      values of the true BYM2 $\varphi$ and number of observations per
      region, from TMB (red) and R-INLA (blue), plotted against
      the target nominal coverage, $\alpha$, for Poisson observation
      experiments. Each point is the median average coverage
      of an experiment, calculated across 25 replicates, and the bars
      represent the middle 80\% quantile range of the average coverage
      across replicates. \label{fig:disc.cov}}}
\end{figure}

The results from the discrete simulations, summarized in term of parameter bias in Figure \ref{fig:disc.bias} and in terms of spatial field coverage in Figure \ref{fig:disc.cov}, share many similarities to those observed in the continuous simulations of Section \ref{sec:sim.cont.results}. Both methods produce very similar spatial field estimates though TMB appears more likely to have larger bias for some parameters.

\clearpage

\section{European Breast Cancer Application}
\label{sec:euro.bc}

Data quality for cancer monitoring vary significantly across the European Union (EU) and range from complete registry coverage and high quality mortality estimates from vital registration to countries with no data. We use TMB to fit a similar model to one previously detailed by \cite{mercer2016} and fitted with user-written MCMC code. We describe a two-level nonlinear model which first assumes a Poisson model for cancer incidence and then, conditional on cancer counts, models deaths as a binomial outcome. This model appropriately handles the variety of cancer data present in the EU in part because the conditional two-level Poison-Binomial form induces a Poisson model for unconditional mortality to account for countries without incidence data from registries. Using data provided by the International Agency for Research on Cancer (IARC), we implement a Bayesian spatial smoothing model to borrow strength between countries to provide estimates of national incidence and mortality, along with measures of uncertainty.

The approach directly models mortality (which is more universally available) and the mortality-incidence (MI) ratio, to estimate both incidence and mortality for all countries. The model synthesizes four types of country data: type I countries with national incidence and mortality data, type II countries with sub-national incidence and mortality data (from registries) in addition to national mortality, type III countries with only national mortality, and type IV countries with no available data. While there is reason to think cancer incidence may be spatially correlated across countries, for example due to environmental and lifestyle risks, different preventions and screening strategies may result in large variability between nearby countries. For this reason, the BYM2 model presented in Section \ref{sec:sim.disc} was used to model a combination of spatial and iid country effects. We would also expect some smoothness in mortality over space due to similarities in GDP, and therefore healthcare, in close by countries.

For countries, $c$, that have both national mortality and incidence data (type I) we assume a Poisson process with rate $p_c$ for cancer incidence, where $Y_c$ is the number of reported individuals with breast cancer from a population of $N_c$. Conditional on having cancer, we model total mortality, $Z_c$, as a binomial outcome with probability of death $r_c$ for each of the $Y_c$ individuals with cancer. This induces a Poisson process for mortality when incidence is unobserved with rate $p_c\times r_c$. For illustration, we work with only with data from 2008 in women aged 50-54. Our base model for type I countries is:
\begin{eqnarray} Y_c|N_c,p_c &\sim& \mbox{Poisson}(N_c\times p_c),\hspace{1em} p_c = \mbox{exp}(\alpha_c^I)\label{eq:1} \\
  Z_c|Y_c,r_c &\sim& \mbox{Binomial}(Y_c, r_c),\hspace{1em} r_c = \frac{\mbox{exp}(\alpha_c^{MI})}{1+\mbox{exp}(\alpha_c^{MI})}\label{eq:2}
  \end{eqnarray} which implies the unconditional mortality model:
  \begin{eqnarray}
    Z_c|N_c,p_c &\sim \mbox{Poisson}(N_c q_c),\hspace{1em} q_c = p_c\times r_c\label{eq:3}.
  \end{eqnarray}
  We assume a log- and logit-linear model for incidence and conditional mortality and we assign the following forms:
  \begin{eqnarray}
    \alpha_c^I &=& \alpha^I + b_c^I \label{eq:4}\\ \alpha_c^{MI} &=& \alpha^{MI} + b_c^{MI} \label{eq:5}
  \end{eqnarray}
  where $p_c$ is the reported incidence, $r_c$ is the reported mortality, $\alpha^I$ and $\alpha^{MI}$ are global intercepts, and $b_c^I$ and $b_c^{MI}$are country random effects that are assumed to have BYM-2 structure comprising of a spatially correlated term as well as an unstructured (iid) country specific term. Additional details on the data and the model may be found in \ref{app:euro.bc}. Maps of $\hat{b}_c^I$, $\hat{b}_c^{MI}$, $\widehat{p}_c$, and $\widehat{r}_c$, with measures of uncertainty, are presented in Figure~\ref{fig:euro.bc.maps}.

The nonlinear unconditional mortality rate, $q_c = p_c\times r_c$, is necessary to include countries with incomplete or missing mortality data and prohibits this model from being fit within INLA. A similar model that used country-level fixed effects without the spatial random effect would be possible in INLA \citep{meehan20} but without complete data in each country this is not feasible.

\begin{figure}[h!]  \centering \includegraphics[width=\linewidth]{figures/euro_bc/posterior_res.png}
   \caption{ \footnotesize{ Each row consists of a pair of posterior median and 95\% credible interval widths for the estimated quantity. Row 1: the BYM2 country random effects for incidence, row 2: the BYM2 country random effects for MI ratio, row 3: estimates country incidence rates, and row 4: estimated country mortality|incidence probabilities.}}
\label{fig:euro.bc.maps}
\end{figure}

\clearpage

\section{Discussion}\label{sec:discussion}

We were pleasantly surprised to find near concurrence in spatial field estimator distributions in TMB and R-INLA -- in both continuous and linearly constrained discrete model settings -- across a wide range of simulation parameters. The generally smaller parameter bias in INLA results may be due to the integration over the fixed effects, similar to restricted maximum likelihood (REML) inference which is known to reduce bias, in contrast to the ML inference performed in TMB. One possible remedy for this could be to effectively enable REML in TMB by adding all linear fixed effects parameters to the list of `random' parameters input by the modeler.

The field coverage figures clearly demonstrate that random effects coverage is a function of the magnitude of the effect (\cite{yu2018}) and should serve as a warning for those trying to interpret coverage of specific portions of estimated spatial fields. The discrete Poisson experiment showed excellent recovery of the BYM2 field and the continuous models, with the SPDE approximation to the GP performed nearly as well. We see that the SPDE approximation in the binomial scenarios appears more robust than in the Gaussian and the undercoverage under extremely large data volumes warrants further investigation. Even so, the SPDE approximation performs very well across a wide range of experiments and by increasing the density of the mesh until the results no longer change the user should be able to confidently determine when the approximation is resolved. We believe these to be the largest-scale simulation studies on the SPDE approach to fitting GPs in TMB or in R-INLA and we hope that the demonstrated success encourages continued use of this convenient approach.

The simulations also provided an opportunity to assess the relative computational burdens of TMB and R-INLA. \ref{app:timing} describes both the serial (one CPU thread) and parallelized timing experiments. Figures \ref{fig:time} and \ref{fig:time.parallel} demonstrate that TMB scales extremely well, even in comparison to R-INLA which was designed, in part, to be a computationally efficient and quick alternative to MCMC sampling. We note that timing tests attempting to replicate `real world' experience are quite difficult and these timing results only provide some crude measures of computational cost. Nonetheless, TMB appears to perform quite favorably and, at minimum, it could be used to quickly test and iterate models before final inference is performed using the modeler's method of choice.

One of the main limitations of this study is the lack of tuning of any particular simulation and inference. Due to the sheer number of experiments, a set of default model parameters (for example, the starting values) were used across simulations. The simulation code was built to catch and relaunch convergence issues in both TMB and R-INLA that may, with human interaction, have been remedied. While sensible tuning decisions were attempted, this also implies a lack of sensitivity analysis on the prior selection for any one simulated dataset or experiment. Furthermore, these simulation studies use correctly specified models and do not attempt to study either tool under model misspecification.

The novel incidence-mortality model from Section \ref{sec:euro.bc} demonstrates the utility of the flexibility TMB provides. Simulation results for this model shown in Figure \ref{fig:simres} indicate that TMB is capable of recovering parameter estimates from nonlinear mixed effects models that cannot be fit within R-INLA. One of the main tradeoffs in return for this flexibility is relative difficulty of constructing the {\tt C++} templates. There has already been efforts, such as the {\tt glmmTMB} {\tt R}-package (\cite{glmmTMB}) which allows users to run {\tt TMB} from within {\tt R} using standard {\tt R} notation for mixed effects models, to streamline the use of TMB, but none will likely be able to offer the freedom of directly coding within the {\tt C++} templates. We note that there are numerous tutorial documents and examples available on the TMB github page and the TMB authors have worked to make it easier for {\tt R} coders to use. Example code for fitting spatial GPs via the SPDE approach is available in \ref{app:code} to provide a sense of modeling in TMB and R-INLA, and the full code used in this study is available online to serve as starting points for those interested in spatial modeling in TMB. Furthermore, others have previously used TMB in spatial settings, such as classic spatial models (for example, \cite{dwyer2016}), and more recent work to account for missing spatial information (\cite{wilson:wakefield:20}, \cite{wilson:wakefield:21} and \cite{marquez:wakefield:21}), and their code is available too.

The primary restriction of TMB's applicability lies in the assumptions underlying the LA. Although many models may be fit within the TMB framework, it is clear that the LA will not perform equally well on all of them - though intelligently chosen reparameterization, like those used for the hyperparameters in both simulations, can help significantly. While the quality of the LA may be difficult to assess, the recent availability of the {\tt tmbstan} package permits MCMC sampling from TMB models - with or without the integration of the random effects performed by LA. By comparing results run with TMB and the LA against those run with MCMC without the LA, it is possible to assess the quality of the LA in particular applications. The speed of TMB makes it a useful option for iterating through models during exploration phases of research and it may provide opportunities to fit models that could not otherwise be fit in reasonable amounts of time or at all.

After conducting these extensive simulations, we have found TMB to perform more than adequately in comparison to existing well-trusted tools. We hope that this unified and detailed explanation of TMB and the additional new model demonstrations will improve understanding, instill confidence, and generate further interest and use in a compelling and generally unknown statistical computational tool.

\section*{Acknowledgments}
\label{sec:ack}
We thank the International Agency for Research on Cancer (IARC) for access to the cancer data used in this study. This work was partially supported by the Institute for Health Metrics and Evaluation (IHME), and we thank them for use of the computing cluster needed to run the continuous spatial simulations and for their support.

\bibliographystyle{format_style/chicago}

redpajama_set_name: RedPajamaArXiv