\section{Introduction}
\label{sec:int}
Thermodynamics is one of the most general and successful branches of physics, in the sense that its applicability covers a wide range of physical systems with different properties and structure. However, it is a purely phenomenological theory, since it is essentially based on a set of empirical laws which are the result of far-reaching observational and experimental data. It is widely accepted that, within its range of applicability, thermodynamics well describes any macroscopic system. Despite this success, alternative approaches to thermodynamics have been constructed in the past decades, which also have proven to be useful. In particular, different geometric representations of thermodynamics have been explored \cite{1948Gibb,1975Weina,1979Rupp,2007Quev}.
These formulations are constructed by promoting the thermodynamic space of equilibrium states, ${\cal E}$, to a differential manifold endowed with a metric structure, usually of Riemannian type.

Weinhold \cite{1975Weina} and, a few years later, Ruppeiner \cite{1979Rupp} proposed to endow the space of equilibrium states $\cal{E}$ with metrics whose components are defined via the Hessian matrices of a given thermodynamic potential or fundamental thermodynamic relation which describes the thermodynamic system under consideration. If the potential is the internal energy $U$, the metric obtained is called the Weinhold metric $g^W$ \cite{1975Weina}; if the selected thermodynamic potential is (minus) the entropy $S$, the corresponding associated metric is called the Ruppeiner metric $g^R$ \cite{1979Rupp}. These metrics can be written as
\begin{equation} \label{hessian}
	g^W = \frac{\partial^2 U}{\partial E^a\partial E^b} dE^a
	\otimes dE^b \, \quad \text{and} \quad g^R = - \frac{\partial^2 S}
	{\partial E^a\partial E^b} dE^a \otimes dE^b \,.
\end{equation}
In this work, we use the convention that repeated indices indicate a sum over all their values. The $E^a$ ($a=1,...,n$) represent the set of extensive variables necessary to describe a system with $n$ thermodynamic degrees of freedom in either representation. It can be shown that $g^R$ is conformally related to $g^W$ via the inverse of the temperature as the conformal factor \cite{1984Salamon}.

The thermodynamic fundamental relation as a function of the extensive variables, either in the entropic or energetic representation, is completely sufficient to describe any given thermodynamic system. Any other thermodynamic potential can be obtained from the internal energy or entropy via a Legendre transformation, and thus a metric based on the Hessian of any thermodynamic potential is a suitable structure for the equilibrium space manifold \cite{2010Liu}. Another important property of equilibrium thermodynamics is thus Legendre invariance, i.e., the well-known fact that the physical properties of any thermodynamic system do not depend on the selected thermodynamic potential \cite{Callen}. It is thus self-evident to incorporate this feature into the metric structure as well, and the geometric formulation which is constructed around this invariance is known as Geometrothermodynamics (GTD) \cite{2007Quev}.

GTD considers the invariance under Legendre transformations as the main condition to be satisfied by all the geometric structures in this formalism. The metric of the space of equilibrium states emerges as a consequence of imposing this invariance. This is achieved by representing Legendre transformations as coordinate transformations \cite{1980Arnold} and obtaining the most general solution for a metric invariant under these transformations. The $n$ extensive variables $E^a$, the $n$ intensive variables $I_a$ and the entropy $S$, as the thermodynamic potential, represent the set of all thermodynamic variables, which is denoted by $z^A=(S, E^a, I_a)$.
Then, a Legendre transformation can be represented as a coordinate transformation $z^A\rightarrow z'^{A}=(\phi', E'^a, I _a')$ defined by
\begin{equation} \label{legendretransf}
	S  = \phi' -  E'^k I'_k \,,\quad
	E^i = -  I'_i \,, E^j =  I'_j, \quad
	I_i  = E'^i \,, I_j  = I'_j \,,
\end{equation}
where $i \in I$, $j \in J$, $k= 1,...,i$, and $I\cup J$ is any disjoint decomposition of the set of indices $\{1,...,n\}$. In particular, for $I=\{1,...,n\}$ or $I=\emptyset$ we obtain the total Legendre transformation and the identity, respectively. It is easy to show that this transformation is well-defined \cite{2007Quev} in the sense that it leads to a non-vanishing Jacobian with $|\partial z^A /\partial \tilde z ^{A}|=1$.

In the GTD formalism, the thermodynamic phase space (TPS) is denoted by $\mathcal{T}$, has $(2n+1)$ dimensions and coordinates $z^A$, and  is equipped with a Riemannian structure via the introduction of the metric $G=G_{AB} \d z ^A \d z^B$. We call a metric Legendre invariant if the functional form of $G$ is not modified by a Legendre transformation. This condition leads to a set of algebraic equations relating the components of the original metric $G_{AB}$ with those of the Legendre transformed metric $G' _{AB}$ \cite{2007Quev}. Solving this set of algebraic equations allows us to find several possible solutions, and one of them is, besides $G^{I}$ and $G^{II}$ established for other purposes, the metric
\begin{equation} \label{GIII}
	G^{III} = (\d S - I_a \d E^a)^2+(E^a I_a)^{2K+1}
	\d E^a \d I_a\ \,,
\end{equation}
where $K$ is an integer. The above metric is invariant with respect to partial and total Legendre transformations. It should be mentioned that it is not at all trivial to find Legendre invariant metrics. Since Legendre transformations do not form a group, it is not possible to apply the standard Killing approach to derive invariant geometric objects. The only possible path to follow is to solve the aforementioned set of algebraic equations, which leads to the most general Legendre invariant metric found so far as given by Eq.\,\eqref{GIII}.

Additionally, the thermodynamic phase space ${\cal T}$ can be canonically endowed with a contact structure by means of the one-form
\begin{equation} \label{eq:eta}
	\Theta = \d S - I_a \d E^a \,,
\end{equation}
satisfying the condition $\Theta \wedge (d\Theta)^n \neq 0$, which distinguishes $\Theta$ as a contact one-form on $\cal T$. According to Darboux' theorem \cite{1882Darboux}, this contact one-form is defined modulo a conformal factor $\Theta\rightarrow f\Theta$, where $f$ is a non-vanishing function $f:{\cal T} \rightarrow \mathbb{R}$. The contact one-form $\Theta$ is also Legendre invariant, since under a Legendre transformation it preserves the same functional form. Consequently, the phase space is defined in GTD by means of the triad $({\cal T}, \Theta, G^{III})$, which is a Riemannian contact manifold with the important property of being Legendre invariant.

The geometry of the space $\mathcal{T}$ can be related in a consistent way with the equilibrium space $\mathcal{E} \subset \cal T$ via the introduction of the smooth embedding map, $\varphi: {\cal E}\rightarrow {\cal T}$ such that $\varphi ^*(\Theta) = 0$, where the asterisk denotes the pullback of the map. This means that ${\cal E}$ is a maximal Legendre sub-manifold, on which the relationship
\begin{equation} \label{eq:firstlaw}
	\d S = I_a \d E^a
\end{equation}
holds, which corresponds exactly to the first law of thermodynamics where the equilibrium condition
\begin{equation} \label{eq:equil}
	I_a =\frac{\partial S}{\partial E^a}
\end{equation}
is satisfied. Moreover, the Riemannian structure on $\mathcal{T}$ also induces a canonical Riemannian structure in the equilibrium space $\mathcal{E}$ by means of $\varphi^*(G^{III}) = g^{III}$. This pullback is explicitly written as
\begin{equation} \label{gGTD}
	g^{III} =  \left(E^a \frac{\partial S}{\partial E^a}
	\right)^{2K+1} \frac{\partial^2 S}{\partial E^a \partial E^b}
	\, \d E^a \otimes \d E^b \,.
\end{equation}
The above metric is in general different from the Hessian metrics $g^W$ and $g^R$, presented in Eq.\,\eqref{hessian}, and the additional terms in $g^{III}$  guarantee the invariance under Legendre transformations as defined on the phase space $\mathcal{T}$.

As shown in \cite{1990Mrug}, the geometric structure determined by the Hessian metrics \eqref{hessian} can be understood as emerging from a statistical approach associated with the family of Boltzmann-Gibbs (BG) canonical probability distributions $\rho$.
In the same work, the geometric structures of the TPS are shown to be related to the descriptions of equilibrium and fluctuations on the manifold of equilibrium states, and the physical consequences of the microscopic statistical approach for the Ruppeiner metric \cite{1995Rupp} are explored.

In a previous work \cite{reparametrizations}, we investigated the reparametrizations of the thermodynamic variables and proposed a geometric description in the TPS. It was also found that the reparametrizations of the thermodynamic variables can be described geometrically  in the TPS through a two-rank tensor, whose pullback to the space of equilibrium states gives a metric, which in principle is related to  thermodynamic fluctuations. In this work, based on the results of \cite{reparametrizations}, we use the aforementioned two-rank tensor to find the general form of Legendre invariant metrics, establishing in this way the statistical origin of the metric \eqref{gGTD} in the formalism of GTD.

This paper is organized as follows. In Sec. \ref{sec:GenDistFunc}, a brief discussion of the most relevant results found in \cite{reparametrizations} are presented. It is shown how, via generalized reparametizations of the thermodynamic variables, the Riemannian structure of the TPS is modified. In Sec. \ref{sec:genGTD}, we discuss how these changes on the TPS lead to a metric structure, which can be connected with Legendre invariant metrics and, in particular, with the metrics found in GTD \cite{2007Quev}. Sec. \ref{sec:conclusions} is devoted to presenting the conclusions and perspectives of this work.

\section{Generalized probabilistic distribution function and Riemannian contact manifold}
\label{sec:GenDistFunc}

Reparametrizations of the extensive and intensive thermodynamic variables can be geometrically described in the TPS \cite{reparametrizations}. In fact, they are represented by distinct TPS's which have different contact and Riemannian structures, while leaving invariant the geometric structure of the space of equilibrium states. Indeed, as mentioned above, the TPS is a ($2n+1$)-dimensional contact manifold characterized by the Gibbs (contact) one-form, which in coordinates $S$, $E^a$ and $I_a$ takes the canonical form \eqref{eq:eta}. Moreover, the space of equilibrium states $\mathcal{E}$ is defined by  the  smooth embedding map $\varphi:\mathcal{E} \to \mathcal T$, along with conditions \eqref{eq:firstlaw} and \eqref{eq:equil}. Alternatively, in the TPS one can apply a total Legendre transformation, defined as in Eq.\,\eqref{legendretransf}, which transforms the Gibbs one-form into
	\begin{equation} \label{gibbsM}
	\Theta = \d \phi - E^a \d I_a,
	\end{equation}
and the embedding conditions $\varphi^*(\Theta) = 0$ become
	\begin{equation} \label{embedding1}
	\d \phi = E^a \d I_a, \quad E^a = \frac{\partial \phi}{\partial I_a}.
	\end{equation}
It is only in $\mathcal{E}$ that the variables $\phi$, $E^a$ and $I_a$ acquire a thermodynamic meaning, namely the total Massieu potential (the total Legendre transform of the entropy potential) and the extensive and intensive variables, respectively. Therefore, the expressions in \eqref{embedding1} correspond to the first law and the equations of state for a thermodynamic system described by the fundamental relation $\phi(I_a)$, where $\phi$ is the total Legendre transform of the entropy. Regarding the Riemannian structure, a metric can be defined for the TPS as
\begin{equation} \label{Gfr}
	G = \Theta \otimes \Theta + t \,,
\end{equation}
where in local coordinates $\Theta$ is given by \eqref{gibbsM} and $t$ is a tensor of rank two taking the form
\begin{equation}  \label{t}
	t = \d I_a \overset{s}{\otimes} \d E^a \,,
\end{equation}
where  the product $\overset{s}{\otimes}$ is defined as
\begin{equation} \label{sotimes}
	\d x \overset{s}{\otimes} \d y \equiv \frac{1}{2} \left(
	\d x \otimes \d y + \d y \otimes \d x \right) \,.
\end{equation}
The pullback of $G$ induces a metric $g = \varphi^*(G)$ on the space of equilibrium states $\mathcal{E}$, which in local coordinates reduces to the Hessian of $\phi$,
	\begin{equation}  \label{gi}
	g = \varphi^*(G) = \frac{\partial^2 \phi}{\partial I_a \partial I_b}\ \d I_a \otimes \d I_b.
	\end{equation}
In \cite{1990Mrug}, it was shown that under a total Legendre transformation, the induced metric $g$ on $\mathcal{E}$ coincides with Ruppeiner's metric, i.e., a metric whose components can be expressed in local coordinates as the negative Hessian of the entropy.

The statistical origin of the Gibbs one-form and the metric $G$ is tied to the BG  distribution $\rho = \exp(-\phi + I_a H^a)$, where $\{H^a\}$ is a set of $n$ stochastic functions on the mechanical phase space $\Gamma$, e.g., the Hamiltonian of a system of particles. In the expression for the BG entropy,
\begin{equation} \label{boltzmann}
	S = -\int \rho \ln \rho\  \d \Gamma \,
\end{equation}
the microscopic entropy defined as
\begin{equation}
	s = - \ln \rho = \phi - I_a H^a
\end{equation}
can be introduced. If the normalization of the distribution is not imposed, the microscopic entropy can be understood as a function of the variables $\{\phi, I_a\}$. If, in addition, we consider the variables $\{\phi, E^a, I_a\}$ as a set of independent variables for a higher-dimensional space, we can relate the above geometric objects to statistical quantities as follows. The Gibbs one-form is related to the statistical average of the differential of the microscopic entropy,
\begin{equation} \label{ds}
	\langle \d s \rangle \equiv \Theta = \d \phi - E^a \d I_a \,,
\end{equation}
while the metric $G$ is related to its variance
\begin{equation} \label{vards}
	\langle (\d s - \langle \d s \rangle )^2 \rangle \equiv t =
	\d E^a \overset{s}{\otimes} \d I_a \,.
\end{equation}
Therefore, the Gibbs one-form geometrically encodes the information about equilibrium, while the metric $G$ is associated with the statistical fluctuations of the system. For a detailed description of the connection between geometry in the TPS and statistical mechanics we refer the reader to Refs. \cite{1990Mrug, HamThermo}.

Turning our attention to the description of reparametrizations in $\mathcal{E}$, in \cite{reparametrizations} it was shown that different geometric structures on the TPS can be used to determine the effects of such reparametrizations. In particular, a new contact one-form $\tilde{\Theta}$ can be chosen that is not related by a contactomorphism to \eqref{gibbsM} and a new tensor $\tilde{t}$ can be defined to account for the reparametrizations at the level of the Riemannian structure. In spite of having a different TPS, when the pullback of these new structures to the space of equilibrium states is done, the Riemannian manifold $(\mathcal{E},g)$ turns out to remain unchanged. The new geometric structures in the TPS comprising the information about the reparametrization can be expressed as follows. The new Gibbs one-form in terms of the new, tilded variables is
	\begin{equation} \label{Thetat}
	\tilde{\Theta} = \d \tilde{\phi} - \tilde{E}^a \d \tilde{I}_a,
	\end{equation}
and the new metric is
	\begin{equation} \label{Gt}
	\tilde{G} = \tilde{\Theta} \otimes \tilde{\Theta} + \tilde{t} \, ,
	\end{equation}
where in the local coordinates the tensor $\tilde{t}$ takes the form
	\begin{equation} \label{ttilde}
	\tilde{t} = \d \tilde{E}^a \overset{s}{\otimes} \d \tilde{I}_a.
	\end{equation}
Restricting the reparametrizations to the form $\tilde{\phi} = \phi$, $\tilde{I}_a = \tilde{I}_a(I_a)$ and $\tilde{E}^a = \tilde{E}^a(E^a)$, implying that the Jacobian of the transformation is diagonal, the new Gibbs one-form can be expressed in terms of the original variables as
\begin{equation} \label{Thetanew}
	\tilde{\Theta} = \langle \d \tilde{s} \rangle_{\tilde{I}} = \d \phi
	- \tilde{E}^a(E^a) \frac{d \tilde I_a}{d I_a} \d I_a \,,
\end{equation}
where $\tilde{s} = \phi - \tilde{I}_a H^a$ is the microscopic entropy in terms of the new variables. The tensor $\tilde{t}$, which geometrically encodes the information about the statistical fluctuations and the reparametrizations in $\mathcal T$, can be expressed in terms of the original variables as
\begin{equation} \label{tensort}
	\tilde{t} = \frac{d \tilde I_a}{d I_a} \ \d \tilde{E}^a\ \overset{s}{\otimes} \d I_a
	= \frac{d \tilde I_a}{d I_a} \frac{d \tilde E_a}{d E_a} \ \d E^a\ \overset{s}{\otimes} \d I_a \,.
\end{equation}
Therefore, we have two different contact structures \eqref{gibbsM} and \eqref{Thetanew}, as well as two different Riemannian structures \eqref{Gfr} and \eqref{Gt} on the TPS. The important feature is that these different structures have no effect on the geometric description in the space of equilibrium states -- when projected to $\mathcal{E}$ they result in the same Legendre submanifold.

In this work, we are interested in the role that the tensor $\tilde{t}$ might play in the geometric description of statistical fluctuations. To this end, we propose to endow the TPS with a different Riemannian structure constructed as $(\mathcal{T}, \Theta, \mathcal{G})$, where the metric $\mathcal{G}$ is defined as
	\begin{equation} \label{Gprime}
	\mathcal{G} = \Theta \otimes \Theta + \tilde{t} \,,
	\end{equation}
which in the local coordinates $\{\phi, E^a, I_a\}$ takes the form
\begin{equation} \label{calGcoord}
	\mathcal{G} = (\d \phi - E^a \d I_a) \otimes (\d \phi - E^b \d I_b) + \frac{d \tilde I_a}{d I_a} \ \frac{d \tilde E_a}{d E_a} \ \d E^a \overset{s}{\otimes} \d I_a \,.
\end{equation}
This Riemannian structure, when projected to the space of equilibrium states $\mathfrak{g} = \varphi^*(\mathcal{G})$, will furnish $\mathcal{E}$ with a different Riemannian structure, in particular, different to the Hessian family of metrics widely used in the geometry of thermodynamics. It is worth emphasizing that, as we are using the Gibbs one-form \eqref{gibbsM}, the first law of thermodynamics will be expressed in its regular form in terms of the canonical variables $\{E^a, I_a\}$, i.e., as given by \eqref{embedding1}. However, the information about the fluctuations contained in the metrics $\mathcal{G}$ and $\mathfrak{g}$ differs from the information in the original Riemannian structure on $\mathcal T$.

\section{Generalized Geometry of Thermodynamics}
\label{sec:genGTD}
In the context of thermodynamics and statistical mechanics, the parameters $\{I_a\}$ of the BG probability density are identified with intensive thermodynamic variables and $\phi$ with the total Massieu potential (the total Legendre transformation of the entropy), whereas the metric \eqref{gi} gives a notion of thermodynamic length for curves connecting different states and relates the extent of the fluctuations with the geometric distance \cite{crooks}. Therefore, it is reasonable to look for a similar interpretation for the induced metric on $\mathcal{E}$ obtained from \eqref{Gprime}.

As we mentioned in the previous section, the tensor field $\tilde{t}$
and consequently $\mathcal{G}$, captures the geometrical information of a generalized probability distribution. The objective is to translate this information into the geometric description of thermodynamic systems, i.e., to determine how this \emph{statistical} information is inherited by the geometrical structures on the submanifold of \emph{thermodynamic} states.

In the submanifold $\mathcal{E}$, determined by the condition $\varphi^*(\Theta) = 0$, each point represents an equilibrium state for the system described by the thermodynamic fundamental relation $\phi = \phi(I_a)$, in which the first law of thermodynamics \eqref{embedding1} is satisfied. A fundamental feature of thermodynamics is its invariance under Legendre transformations. In a geometric language, this property is enclosed in the fact that the one-form \eqref{gibbsM} is invariant under such transformations on $\mathcal{T}$.

As shown in \cite{1990Mrug}, the second law of thermodynamics is satisfied by the submanifold $\cal E$ by construction, since it is derived directly from a maximum entropy principle. In our particular case, we guarantee the second law of thermodynamics to be satisfied since $\phi(I_a)$ is the total Massieu potential. This potential is a convex function of the intensive variables $I_a$, or equivalently, its Legendre transform, the entropy, is a concave function of the extensive variables $E^a$, except at the points of macroscopic phase transitions. However, in the geometric description, the contact structure does not guarantee the second law and it is necessary to endow the contact manifold with a Riemannian structure. In the canonical case, i.e., using the family of BG probability distributions, the metric induced on $\cal E$ by the metric \eqref{Gfr} is given by Eq.\,\eqref{gi},
which under a total Legendre transformation coincides with the entropic metric $g^R$ of the geometric thermodynamic formulation of Ruppeiner \cite{1979Rupp,1995Rupp}, defined as the negative Hessian of the entropy as given by Eq.\,\eqref{hessian}. The metric \eqref{gi} allows us to relate the local thermodynamic stability conditions for the corresponding thermodynamic potential to a geometric structure. Moreover, as the metric is also a measure of distance between points in the manifold, the length of curves in $\cal E$ is related to the number of fluctuations along the path followed during a quasi-static thermodynamic process, larger fluctuations corresponding to a smaller distance between points \cite{crooks}.

\subsection{Generalized metrics for thermodynamics}
\label{subsec:metricG}
We recall that the generalized metric \eqref{Gprime} on the Riemannian contact manifold $(\mathcal{T},\Theta, \mathcal{G})$ is defined by means of the Gibbs form $\Theta$ and the tensor field $\tilde{t}$, whose expressions in local coordinates are given by \eqref{gibbsM} and \eqref{ttilde}, respectively. In terms of local coordinates, $\mathcal{G}$ takes the form given in \eqref{calGcoord}. Then, the induced metric $\mathfrak{g} = \varphi^*(\mathcal{G})$ on $\mathcal{E}$ is
\begin{equation} \label{gcalmetric}
	\mathfrak{g} = \varphi^*(\mathcal{G}) = \frac{d \tilde I_a(I_a)}{d I_a}
	\frac{d \tilde E^a(E^a)}{d E^a}{}\bigg|_{E^a = \partial \phi/\partial I_a}
	 \ \frac{\partial^2 \phi}{\partial I_a \partial I_b}\ \d I_a \overset{s}
	 {\otimes} \d I_b \,,
\end{equation}
Equations \eqref{Gprime} and \eqref{gcalmetric} are in fact a family of metrics for $\mathcal{T}$ and $\mathcal{E}$, respectively. Indeed, for each set $\{\tilde{I}_a = \tilde{I}_a(I_a), \tilde{E}^a = \tilde{E}^a(E^a)\}$ we have a different metric structure on $\mathcal{T}$ as well as on $\mathcal{E}$. Thus, \eqref{gcalmetric} represents a family of thermodynamic metrics on $\mathcal{E}$ and each one is related to the statistical fluctuations of the system and contains information about the specific reparametrization.

\subsection{Legendre invariant metrics}
\label{subsec:g3}
The invariance of the contact structure under a Legendre transformation incorporates the well-known thermodynamic fact that the thermodynamic information of any physical system described by a fundamental relation is preserved under any such  transformations. In geometric terms, this corresponds to the invariance of the Gibbs one-form in the TPS under such a transformation. That is, the equilibrium description is Legendre invariant by construction. On the contrary, the metric \eqref{Gfr} is not invariant on $\mathcal{T}$, in the sense that it does not preserve its form under the action of a Legendre transformation, a property that the metric \eqref{gi} inherits. Therefore, we conclude that the geometric description of fluctuations on $\mathcal{E}$ is not invariant under Legendre transformations. In fact, it is known that Hessian metrics in the geometry of thermodynamics obtained from different (Legendre related) potentials correspond to different behaviors in the various ensembles \cite{2010Liu, termometrica}.

The behavior of the family of metrics \eqref{Gprime} is now analyzed with respect to a general Legendre transformation, in order to determine the requirements for $\mathcal{G}$ to remain invariant under such transformations. This family of metrics has a freedom in the choice of the functions $\{\tilde{I}_a = \tilde{I}_a(I_a), \tilde{E}^a = \tilde{E}^a(E^a)\}$ and unlike \eqref{Gfr}, there is the possibility of finding an invariant metric under Legrendre transformations. We can say that this requirement amounts to asking for a invariant geometric description of fluctuations in terms of Riemannian structures on the manifolds $\mathcal{T}$ and $\mathcal{E}$.

A partial Legendre transformation for the set of coordinates $\{\phi,I_a,E^a\}$ is given by Eq.\,\eqref{legendretransf},
where the same conventions for the set of indices $\{i,j\}$ are applied, and $\phi'$ becomes the new thermodynamic potential on $\mathcal{E}$. The generalized metric $\mathcal{G}$ transforms as
\begin{equation} \label{Gtildetrans}
	\mathcal{G} = \Theta \otimes \Theta + \frac{d \tilde I_i(E'{}^i)}{d E'{}^i}
	\frac{d \tilde E ^i(-I'{}_i)}{d I'{}_i} \d E'{}^i \overset{s}{\otimes}
	\d I'{}_i +\frac{d \tilde I_j(I'{}_j)}{d I'{}_j} \frac{d \tilde
	E^j(E'{}^j)}{d E'{}^j} \d E'{}^j \overset{s}{\otimes} \d I'{}_j \,,
\end{equation}
with $\Theta = \d \phi' - I'{}_a \d E'{}^a$. It is straightforward to show that in general this metric is not covariant under a general Legendre transformation. The fulfillment of  this property will clearly depend on the functions $\tilde I_a(I_a)$ and $\tilde E^a(E^a)$. In order to obtain a form of $\mathcal{G}$ which is covariant under the full set of Legendre transformations, two conditions must be satisfied by the set of functions $\{\tilde I_a, \tilde E^a\}$: (i) all of $\tilde I_a$ and $\tilde E^a$ must have the same functional form, and (ii) $\tilde I_a$ and $\tilde E^a$ must be even functions, i.e., $\tilde E^a(-E^a) = \tilde E^a(E^a)$ and $\tilde I_a(-I_a) = \tilde I_a(I_a)$. A particular choice for the set of functions satisfying the above conditions is
\begin{equation} \label{choicegf}
	\tilde I_a(I_a) = \frac{(I_a)^{2K +2}}{2K+2}, \quad \text{and}
	\quad \tilde E^a(E^a) = \frac{(E^a)^{2K+2}}{2K+2}\,,
\end{equation}
with $K$ an integer. This choice for $\tilde I_a$ and $\tilde E^a$ yields the following metric for the thermodynamic phase space $\cal T$,
\begin{equation} \label{G3}
	\mathcal{G} = \Theta \otimes \Theta + \left(I_a E^a\right)^{2K +1}
	\d E^a \overset{s}{\otimes} \d I_a\,.
\end{equation}
This metric was proposed within the formalism of GTD and coined as $G^{III}$; its main feature is to be invariant under any Legendre transformation.
The corresponding induced metric on the space of thermodynamic equilibrium states $\cal E$, \emph{after a total Legendre transformation}, is then
\begin{equation} \label{g3metric}
	\mathfrak{g} =\varphi^*(\mathcal{G})= \left( E_a \frac{\partial S}
	{\partial E_a} \right)^{2K +1} \frac{\partial^2 S}{\partial E_a
	\partial E_b} \ \d E_a \otimes \d E_b \,.
\end{equation}
This is the same metric as the one induced via the pullback $g^{III} = \varphi^*(G^{III})$, given in Eq.\,\eqref{gGTD}. It provides a notion of distance between thermodynamic states on the submanifold $\cal E$, which is different from the one given by the Hessian thermodynamic metrics. However, each component of the above metric is still proportional to the second derivatives of the thermodynamic potential with respect to its corresponding variables, thus providing a connection between distances in $\cal E$ and thermodynamic fluctuations, as well as between curvature singularities and phase transitions; nonetheless, these relations between geometric and thermodynamic features are not completely understood yet, nor is it clear how to establish them from the general geometric expressions. Nonetheless, there is enough evidence and a large number of results supporting these claims \cite{2015Quev,2013Brava,2011Quevb,2012Aviles,2014Quevedo,2010Vasq,2015Quevedo}.

\section{Conclusions}
\label{sec:conclusions}

To develop a geometric description of equilibrium thermodynamics two different classes of Riemannian metrics have been applied, namely, Hessian metrics and Legendre invariant metrics. The first class was created by introducing metrics into the equilibrium space, whose components coincide with the Hessian matrix of specific thermodynamic potentials. In particular, the Hessian metric derived from the entropy (the Ruppeiner metric) has been shown to have a very interesting interpretation in terms of statistical quantities derived from the BG  entropy.

The class of Legendre invariant metrics have been introduced in the formalism of GTD with the aim of taking into account the well-known property that in equilibrium thermodynamics the physical properties of a system do not depend on the thermodynamic potential used to describe the system. This class of metrics has been introduced by using a purely geometric approach in which the thermodynamic phase space is defined as a Riemannian contact manifold.

In this work, we have shown that Legendre invariant metrics can also be interpreted in terms of statistical quantities, namely, the average and the variance of the differential of the microscopic entropy. To this end, we introduce a particular reparametrization of the extensive and intensive variables, which are used as coordinates in the thermodynamic phase space. These reparametrizations are encoded in a two-rank tensor, which is shown to contain information about the statistical fluctuations of the system. We conclude that Legendre invariant metrics can also be interpreted in terms of statistical quantities that are related to the thermodynamic fluctuations of the system under consideration.

The approach presented here can be applied not only to Legendre invariant metrics, but also to any metric of the phase space that can be obtained by the particular parametrization, which involves only the extensive and intensive variables, separately. It would be interesting to analyze the properties of this new class of metrics and explore their applicability in the context of the  geometric descriptions of thermodynamics.

\section*{Acknowledgments}

L. F. Escamilla-Herrera and F. Nettel thank the financial support from the Consejo Nacional de Ciencia y Tecnolog\'ia (CONACyT, Mexico). C. Gruber acknowledges support by a Junior Fellowship of the Hanse-Wissenschaftskolleg Delmenhorst, and from the University of Oldenburg and the Research Training Group "Models of Gravity". This work was partially supported  by UNAM-DGAPA-PAPIIT, Grant No. 111617.

\bibliographystyle{ieeetr}


\section{Conclusion}
\label{conclusion}
In this paper, we proposed three multimodal CNN architectures in different fusion strategies, which can process jointly the image and audio data for bird classification. Experimental results verified that the two-stream multimodal CNN in late fusion strategy outperforms the others. In addition, we proposed summed fusion method to combine multiple CNNs, which shows better performance comparing against several existing fusion methods. Moreover, with the help of two-stage fine-tuning, our method can be more effective. However, there still exist several drawbacks of our method: (1) Choosing the suitable duration based on the vocal features of birds to be recognized, is essential ingredients of improvement, is missed in our current work. (2) Our method is based on the raw image data, thus part detection and extracting features from pose-normalized regions may improve the classification performance.
\par
As the future work, we aim to apply multiple kernel learning to combine multiple modalities, which is able to learn optimal composite kernel through combining basis kernels constructed from different features of modalities. We are also interested in designing new CNN architectures by increasing the number of modalities.
\section{Database}
\label{database}
We evaluate our models on the popular fine-grained CUB-200-2011 \cite{wah2011caltech} bird dataset and our originally collected sound dataset from sharing bird sound database Xeno-Canto\footnote{\url{http://www.xeno-canto.org}}.

\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.45\textwidth}
   \includegraphics[width=\linewidth]{images/albatross_font_changed.png}
   \caption{}
\end{subfigure}
\begin{subfigure}[b]{0.4\textwidth}
   \centering {\includegraphics[width=0.8\linewidth]{images/albatross_10_changed.png}}
   \caption{}
\end{subfigure}
\caption[Spectrogram of the black-footed albatross]{(a) Radio (top) and spectrogram (bottom) of the black-footed albatross. (b) The spectrogram of 10 seconds duration, which will be fed into the bird classification model.}
\label{spectrogram}
\end{figure}

CUB-200-2011 \cite{wah2011caltech} dataset contains 11788 images of 200 species of birds, with each image downscaled to 227 $\times$ 227 pixel. Spectro-temporal features (spectrogram) are extracted from audio recordings that we collected from the Xena-Canto to be used as the audio representation. Based on the 200 species of the CUB-200-2011, we try to harvest at least 10 different audio recordings for each species. As a result, audio recordings over 178 species were collected completely ($\mathrm{\#recordings}\geq10$), audio recordings from 19 species were collected deficiently ($0<\mathrm{\#recordings}<10$), and 3 species could not be collected. The spectrograms of the audio are obtained by using short-time fourier transform (STFT) over 10 seconds audio frames, windowed with Hanning window (size 512, 50\% overlap). The reason is, the sounds of birds are usually contained in a small portion of the frequency range (mostly around 2-8 kHz) as stated in \cite{cakir2017convolutional}, so we only extract features from the range of (0, 10) kHz. In order to focus only sounds produced in the vocal organ of birds (i.e. calls and songs), first we obtained the maximum amplitude of the audio and removed a frame which contains only amplitude less than $1/4$ of the maximum amplitude. Finally, the spectrograms are saved as 227$\times$227 pixel color images, and the dataset contains 4807 images of 194 species of birds.
Several examples including both images and spectrograms over different bird species are shown in Fig.\ref{dataset}. We follow the standard training/test split of CUB-200-2011 dataset suggested in \cite{wah2011caltech}. The sound dataset is split into two halves for training and test set respectively.
\par
The multimodal CNNs are trained in a supervised manner, thus we create integrated dataset by matching two data sets and corresponding labels using HDF5 \footnote{\url{https://www.hdfgroup.org/HDF5/}} file format. Because the HDF5 file can contain any collection of data entities (images, videos, audio recording, text, etc.,) in a single file, and used to organize  heterogeneous collections consisting of very large and complex datasets.

\begin{figure}[t]
	\centering
	\begin{subfigure}{.25\linewidth}
		\centering
		\includegraphics[width=0.95\textwidth]{images/43.jpg}
		\caption{}
	\end{subfigure}%
	\begin{subfigure}{.25\linewidth}
		\centering
		\includegraphics[width=0.95\textwidth]{images/128.jpg}
		\caption{}
	\end{subfigure}%
	\begin{subfigure}{.25\linewidth}
		\centering
		\includegraphics[width=0.95\textwidth]{images/53.jpg}
		\caption{}
	\end{subfigure}%
	\begin{subfigure}{.25\linewidth}
		\centering
		\includegraphics[width=0.95\textwidth]{images/86.jpg}
		\caption{}
	\end{subfigure}%
	\caption{An example of CUB-200-2011 and audio dataset: (a) the yellow bellied flycatcher, (b) the seaside sparrow (c), the western grebe, and (d) the pacific loon.}
	\label{dataset}
\end{figure}
\section{Experiments and Results}
\label{experiment}
The experiments in this section are conducted to evaluate the effectiveness of our proposed architectures (Net1, Net2, Net3). In order to evaluate an advantage of our late fusion approach, we conduct a comparative experiment between Net3 model and two different existing fusion approaches \cite{eitel2015multimodal,simonyan2014two}. Besides, we have fine-tuned a pretrained model for all the models (including comparative models) in order to improve the performance. To improve the repeatability and only focus on the evaluation of fusion step, we use the well-known CaffeNet model \cite{jia2014caffe} to extract features, train, and fine-tune the CNN with default structure and parameter setting fixed except the base learning rate and batch size. We set the learning rate to 0.001 for single modality and 0.0001 for multimodal learning. The batch size is set to 32 for single modality and 1 for multimodal learning due to limited resources.

\subsection{Quantitative result: Single-modality v.s. Multi-modality}
The core idea of this paper is to address the integration of the image and audio. Therefore, it is necessary to compare the performance between single-modality (image or sound) and multi-modality (both image and sound: Net1, Net2, Net3). To focus on the evaluation of fusion models, in this experiment, we do not introduce any transfer learning techniques (e.g., fine-tune the pretrained model to help differentiate between gains from the proposed architectures). Table \ref{table1} summarizes the results of single modality and proposed models. We can observe that combining two modalities using CNN improves the performance of those only using one modality, extracting features separately from image and audio and fusing them at the late stage performs better with significant gain. Interestingly, the performance of low-level and mid-level fusion models slightly better than the performance of single-modality. One possible reason is because CNN learns features for the predominant modality. In contrast, learning features separately for different modalities results in more independent features, which leads to achieve better performance. Let us mention that the result obtained by multi-modality is different from simply combining the results of two CNNs trained separately. Indeed, the two modalities' parameters are jointly estimated and thus can be mutually influenced. We provide a detailed discussion in following section.
\begin{table}[!t]
	\caption{Comparative results between individual modality and multimodal CNNs.}
	\label{table1}
	\renewcommand{\arraystretch}{1.2}
	\centering
	\begin {tabular}{cl|c}
	\hline \hline%

	\multicolumn {2}{c}{Method} & \parbox{2cm}{\centering Accuracy (\%)}
	\\\hline
	\multirow{2}{*} {Single modality} 		& Image	& 16.2		\\\cline{2 - 3}
	& Audio & 46.4	\\\hline
	\multirow{3}{*} {Multi modality} 	& Net1 	& 50.0		\\\cline {2 -3}
	& Net2 	& 49.9		\\\cline {2 -3}
	& \bf Net3 (summation)	& \bf 53.8	\\\hline
	\end {tabular}
\end{table}

Figure \ref{net3_all} plots learning process of single modality (image only and audio only) and Net3 model over learning epochs. It can be observed that Net3 significantly improves the results.

\begin{figure}[b]
	\centering
	\includegraphics[width=\linewidth]{images/curve.pdf}
	\caption{Test accuracy vs. Epoch.}
	\label{net3_all}
\end{figure}

\subsection{Qualitative result: Single-modality v.s. Multi-modality} \label{ssec:qualitative}
We perform a qualitative study to analyze effects of multimodal learning models by comparing single-modality and multi-modality networks. First, we select some classes where the multi-modality models provides the correct answer while the single-modality model produces the wrong classification. Then, we study why the multi-modality models provides right answers while the single-modality model failed to produce the correct classification. Figure \ref{singlevsmulti} shows some examples of single-modality vs. multi-modality classification. In the first column, the single-modality models predict the input image and spectrogram as the 'barn shallow' and the 'ring-billed gull' respectively rather than the 'red-bellied woodpecker'. However, the multi-modality models are able to predict the right answer, because those models provide joint features of different modalities. We observe that when single-modality model provide right answer for spectrogram, the probability of providing the right answer of multimodality models is higher than when single-modality classificaion is correct for image. In the second column of Fig. \ref{singlevsmulti}, single-modality model is misclassified the 'clark nutcracker' image as the 'great grey shrike', where other models provide correct answers. Lastly, the Net3 model is able to provide right answers while the other models provide misclassification on the 'belted kingfisher' (last column).
\par
To better understand the difference between the models, we analyze the feature learned from the each networks by visualizing the filters of the first convolutional layers shown in Fig. \ref{filter}. We see that each network's filters of different models have similar pattern. Secondly, we see that the single-modality's filters have more meaningful patterns than the multi-modality filters. As we mentioned before its seems that the learning features separately for different modalities results more independent features. Finally, it can be seen that the filters of early fusion model has combined patterns from both networks, but most filters are similar to the spectrogram network. It reveals that the multimodal model (early fusion and mid-level fusion) learns features for the predominant modality.
\begin{figure}[t]
\centering
\begin{tabular}{m{2.5em}ccc}
\multirow{2}{*}{\rotatebox[origin=c]{90}{Input}} & \includegraphics[width=0.23\linewidth]{images/Red_Bellied_Woodpecker_0050_180751.jpg}& \includegraphics[width=0.23\linewidth]{images/Clark_Nutcracker_0075_85715.jpg}& \includegraphics[width=0.23\linewidth]{images/Belted_Kingfisher_0084_70399.jpg
\\
& \includegraphics[width=0.23\linewidth]{images/4521.png} & \includegraphics[width=0.23\linewidth]{images/2016.png} & \includegraphics[width=0.23\linewidth]{images/1764.png}
\\
Image & \ding{55} &  \ding{55}  &  \ding{55}\\
Sound & \ding{55} &  \ding{51} &  \ding{55}\\
Net1 & \ding{51} & \ding{51} & \ding{55}\\
Net2 & \ding{51} & \ding{51} & \ding{55}\\
Net3 & \ding{51} & \ding{51} & \ding{51}\\
\end{tabular}
\caption{Effects of combining image and spectrogram. Top two rows show sample image and spectrogram of different bird species where are fed into single-modality models and multi-modality models. The bottom rows show the resulting classification, where multimodal networks provide a correct classification while the single-modality classification are incorrect.}
\label{singlevsmulti}
\end{figure}

\subsection{Quantitative result: Net3 v.s. existing late fusion approaches}
To evaluate the effectiveness of our late fusing approach, we conduct comparative experiments on Net3 with existing late fusing methods presented in \cite{simonyan2014two,eitel2015multimodal}. The differences between the late fusion approaches is shown in Fig. \ref{fig:fusion_difference}.

\begin{figure}[!b]
\begin{tabular}{m{1em}m{3cm}m{3cm}}
\rotatebox[origin=c]{90}{Single-modality} &\includegraphics[width=\linewidth]{images/image_conv1_X.png}&
\includegraphics[width=\linewidth]{images/sound_conv1_Y.png} \\
\rotatebox[origin=c]{90}{\centering{Net1}} & \multicolumn{2}{c}{\includegraphics[width=.38\linewidth]{images/net1_conv1_1.png}} \\
\rotatebox[origin=c]{90}{Net2} &\includegraphics[width=\linewidth]{images/net2_conv1_X.png}&
\includegraphics[width=\linewidth]{images/net2_conv1_Y.png}\\
\rotatebox[origin=c]{90}{Net3} &\includegraphics[width=\linewidth]{images/net3_conv1_X.png} &
\includegraphics[width=\linewidth]{images/net3_conv1_Y.png}
\end{tabular}
\caption{Visualization of 96 filters of the first convolutional layer. Left side shows the filters related to the image network, while right side shows the filters related to the spectrogram network. It can be seen that the filters (left or right side) of different models have similar pattern. However, the filters of Net1 seems to have mixed filters of both networks.}
\label{filter}
\end{figure}

\begin{itemize}
\item \textbf{FC7 concat\cite{eitel2015multimodal}}: FC7 layers (green and blue) of each networks are concatenated and merge into the fusion layer, which performs tensor multiplication of two vectors. The resulting fusion vector is then passed through one additional fully-connected layer for classification. This means this fusion methods is a linear combination of pair-wise interactions between two features. However, this method is not suitable when the features are in different sizes.
\item \textbf{Averaging scores\cite{simonyan2014two}}: Each networks focus on learning features from images and spectrograms, respectively, and the final classification is computed as an average of the softmax scores of the two networks. In this fusion method, they do not consider pair-wise interactions between the features. However, this method is suitable when the model consists of different structured network streams.

\end{itemize}
In terms of pair-wise interactions, our method is similar to FC7 concat method. However, we fused final output of each networks. Figure \ref{net3_all} plots learning curve of Net3 model with different late fusing methods for each epoch, indicating that averaging the Softmax scores gives lowest performance and the our fusing approach performs best.

\begin{figure}[b]
    \centering
    \includegraphics[width=.9\linewidth]{images/late_fusions.pdf}
    \caption{Differences between the late fusion approaches.}
    \label{fig:fusion_difference}
\end{figure}

\subsection{Fine-tuning the pretrained model}
Combining multi-modalities at the late stage of CNN has proven to be more effective, thus we show additional results with fine-tuning CaffeNet pretrained CNN under Net\ref{net3} model in this section. One natural idea for fine-tuning is to train the model by initializing both image and audio CNNs with the weights and biases of the first seven layers derived from CaffeNet pretrained network, discarding the last fully connected layer. Instead of last fully connected layer of the pretrained model, we randomly place the initialized new fully connected layer for 200-class bird classification (in our experiment, 194 classes due to the lack of audio dataset).
\par
\begin{figure}[p]
\centering
\begin{tabular}{m{1em}m{3cm}m{3cm}}
\rotatebox[origin=c]{90}{Input} &\includegraphics[width=\linewidth]{images/Pacific_Loon_0006_75386.jpg} &
\includegraphics[width=\linewidth]{images/1840.png} \\
\rotatebox[origin=c]{90}{Pool1} &\includegraphics[width=\linewidth]{images/pool1_X.png} &
\includegraphics[width=\linewidth]{images/pool1_Y.png}\\
\rotatebox[origin=c]{90}{Pool2} &\includegraphics[width=\linewidth]{images/pool2_X.png} &
\includegraphics[width=\linewidth]{images/pool2_Y.png} \\
\rotatebox[origin=c]{90}{Pool5} &\includegraphics[width=\linewidth]{images/pool5_X.png} &
\includegraphics[width=\linewidth]{images/pool5_Y.png} \\
\multicolumn{3}{c}{(a) Pooling layer}  \\
 & \multicolumn{2}{c}{\includegraphics[width=0.7\linewidth]{images/FC_copy.png}} \\
\multicolumn{3}{c}{(b) Fully connected layer} \\
\end{tabular}
\caption{Feature visualization of network layer. These are examples of features at different levels in Net3, where (a) shows the features of different pooling layer of image (left) and audio (right) network, and (b) top to bottom shows the features of last fully connected layer of image only, spectrogram only, Fused layer using FC7 concat, and fused layer using summation. Here, the red rectangle shows incorrect answer, the green rectangle shows correct answer of the classification.}
\label{activation_visualization}
\end{figure}
Another method of fine-tuning the model is to train Net3 in two stages. First, training the two stream individually followed by a joint fine-tuning. We train the image and audio CNNs separately, adapting the weights of pretrained model and learn the weights of the new 194-class output layer. After this training, the networks can be used to perform separate classification with respect to each modality. After then, we train an entire model by setting their learning rate to zero and only training the fusion part of the network to freeze the individual stream networks. As shown in Table \ref{table2}, two stage training is resulted best performance, and it proves that additional modality can improve the performance. We found that the fine-tuning pretrained model to two modalities and training them simultaneously, is significant worse than two stage training. We think the problem relates to the difference between the size of two datasets and batch size of single-modality and multi-modalities. In this experiment, we used net surgery\footnote{\url{https://github.com/BVLC/caffe/blob/master/examples/net_surgery.ipynb}} to fine-tune two different CNN using one pretrained model and to fine-tune two pretrained model into our model.

To confirm that the features extracted from sample image and spectrogram during the fine-tuning were meaningful, we visualized the activations of different layers, especially the fused layers in Net\ref{net3}. The results shown in Fig.\ref{activation_visualization}, that allowed us to confirm the learned features were meaningful and qualitatively resembled the sample image and spectrogram. Moreover, the activations of our fused layer takes advantage by incorporating the features from each streams, when each network failed to produce write answer.
\begin{table}[t]
	\caption{Classification performance of fine-tuned Net\ref{net3} model with different fusion and fine-tuning method.}
	\label{table2}
	\renewcommand{\arraystretch}{1.2}
	\centering
	\begin {tabular}{cl|c}
	\hline \hline%

	\multicolumn {2}{c}{Method} & \parbox{1.5cm}{\centering Accuracy (\%)}
	\\\hline
    \multicolumn {2}{c|}	{Fine-tuning weights of pretrained model (summation)}	& {65.0}	\\\hline
	\multirow{4}{*} {Two stage fine-tuning} 	& \textbf{Summation (ours, Net3)} 	& \textbf{78.9}		\\\cline {2-3}
	& Multiplication (ours, Net3) 	& 75.0		\\\cline {2 -3}
    & Simonyan et al. \cite{simonyan2014two}	& 70.0		\\\cline {2 -3}
	&  Eitel et al. \cite{eitel2015multimodal}	&  72.5	\\\hline
	\end {tabular}
\end{table}
\section{Introduction}
\label{sec:intro}
Identifying the species of a bird is a widely-studied problem to ornithologists, and an important task in ecosystem monitoring and biodiversity preservation. Recognition of bird species in images is a challenging task due to various appearances, backgrounds, and environmental changes. Despite of this, this fine-grained recognition task has received a significant amount of attention in the computer vision community \cite{zhang2014part, branson2014bird, cui2016fine, gavves2015local} because of its potential widespread applications. Compared to generic object recognition, fine-grained recognition benefits more from learning critical parts of the object that can help align objects of same class and discriminate between neighboring classes. Current state-of-the-art methods (e.g., \cite{zhang2014part,branson2014bird,guo2018fine, lebedev2018impostor,he2017fine}) adopt CNN-based architectures that learn representations directly from the raw data and can be used to extract set of discriminative features.
\par
On the other hand, sound also provide us important information about the world around us. Many animals make sounds either for communication or their living activities such as moving, flying, mating etc. Although sound is in some case complementary to visual information, such as when we listen to something out of view, vision and hearing are often informative about the same structures in the world \cite{owens2016ambient}. As a consequence, numerous efforts have been devoted to recognize bird species based on auditory data \cite{kahl2017large,cakir2017convolutional} in recent years. Adapting CNN architectures for the purpose of audio event detection has become a common practice and generating deep features based on visual representations of audio recordings has proven to be very effective \cite{takahashi2018aenet} such as in bird sounds \cite{piczak2016recognizing,cakir2017convolutional}. To improve the affinity between images and sounds, we utilize the CNN to extract features from spectrogram of audio recordings.
\par
In the real world, human are able to handle information consist of different information from multiple plural modalities. In the filed of machine learning, multimodal recognition can improve performance compared with unimodal recognition by utilizing complementary sources of information \cite{srivastava2012multimodal,ngiam2011multimodal}. Multimodal learning have been used for tasks such as image and sentence matching \cite{tatulli2017feature}, RGB-D object recognition \cite{eitel2015multimodal}, action detection \cite{simonyan2014two} and specially speech recognition \cite{tatulli2017feature,noda2015audio,meutzner2017improving,torficoupled}, fusing different modalities. The results have shown that one modality can enhance the performance of the other by providing relevant information. Furthermore, authors of \cite{ngiam2011multimodal,huang2013audio} proposed fusion schemes for multimodal learning with considering the architectures of neural networks.
\\
Our main contributions are summarized as follows:
\begin{itemize}
    \item We propose that the combination of image and sound provide richer training signal for bird species classification under CNN framework, which is the first attempt to the best of our knowledge.
    \item Three strategies are investigated for fusing audio and image modalities using CNN.
\item We collect at least 10 audio recordings for each bird over 178 species, corresponding to the image dataset CUB-200-2011 \cite{wah2011caltech}.
\end{itemize}
Specifically, we adopt CNN to process jointly the two modalities for bird species classification. Three strategies are investigated for fusing audio and image modalities using CNN: (1) an early fusion strategy in which the feature vectors related to each modality are concatenated together and input to the CNN. (2) A middle fusion strategy. Features learned by each single modality are combined at the mid-level of the CNN. (3) A late fusion strategy. Outputs of single modality are fused to determine a final classification.
Our experimental results show that the architecture with late fusion strategy outperforms among the proposed architectures, which indicates that combining decisions of the classifiers from two modalities is superior. In addition, we apply a two-stage training procedure, which improves the classification accuracy.
\par
The rest of the paper organized as follows. In section 2, we review related works on bird species identification with CNN and multimodal learning algorithms. Section 3 describes several architectures among which a multimodal CNN processing jointly the two modalities. In section 4, we describe the integrated dataset utilized for the evaluation experiments of our architectures. Section 5 describes the experimental setup and results. In section 6, we conclude this paper.
\section{Methodology}
\label{sec:method}
Our three multimodal architectures extend conventional CNN for large-scale image classification \cite{krizhevsky2012imagenet}. Our implementation is based on CaffeNet , and can be treated as \cite{jia2014caffe} a variation of the structure proposed by Krizhevsky et al. \cite{krizhevsky2012imagenet}.
\subsection{Feature extraction}
CNN uses hierarchical features in its processing pipeline. The features from initial layers are primitive while late layers are high-level abstract features made from combinations of lower-level features. The CaffeNet consists of five convolutional layers (with max pooling layers following the first, second and fifth convolution layer) followed by three fully connected (FC) layers and a softmax classifier. Rectified linear unit is applied to every convolutional layer and fully connected layer and  local normalization is applied in first and second convolutional layer. The process through this 8-layer CNN network can be treated as a process from low to mid to high-level features. We hypothesize, that combining the features of different layers in this pipeline can lead to achieve better performance.
\subsection{Feature combination}
We propose our method in three strategies to fuse features: early fusion, middle fusion and late fusion. Early fusion, also known as feature level fusion, is a feature combination scheme that features from multiple modalities concatenated to form a merged feature vector. Middle fusion, also called mid-level combination, combines the high-level features learned by single network. Late fusion, also called decision-level fusion, combines the decisions of the unimodal classifier and determine the final classification. In this paper, we use concatenation to combine low and high-level features, and summation or multiplication to combine decisions of the classifiers.
The feature combination layers can be trained with standard back-propagation and stochastic gradient descent.

\begin{figure}[tb]
\includegraphics[width=\linewidth]{images/net2_alone.pdf}
\caption{The architecture of early fusion model (Net1). $ 227 \times 227$ pixel RGB images of two modalities are concatenated at merging layer, which produces $ 227 \times 454 \times 3$ output volume, and the convolution layers will extract joint features from this merged volume. We use HDF5 format to manage datasets of two modalities, because of it's flexible data storage and unlimited data types.}
\label{net1}
\end{figure}

\subsection{Architectures of proposed models}
\label{sec:apm}

The proposed multimodal learning models which combines audio and image using CNN by different fusion approaches are described in this section. We exploit the same architecture for both audio and image modalities to focus on evaluating the effectiveness of the feature combination approaches.
\subsubsection{Early fusion model}
One direct approach for combining audio and image is to train a CNN over the concatenated audio and image data as shown in Fig. \ref{net1}. In this strategy, the input vectors related to each modality are concatenated together and then processed together throughout the rest of the CNN pipeline. This model is most computational efficient comparing to the middle and late fusion models, because the number of learnable weight parameters is almost half times less than late fusion model.

\subsubsection{Middle fusion model}
In the middle fusion strategy, unimodal features is extracted independently from audio and images, then combined into a multimodal representation by concatenating the activations of the last pooling layers of two modalities. The multimodal representation then learned in the following fully connected layers. The middle fusion model is shown in Fig.\ref{net2}.

\begin{figure}[tb!]
	\includegraphics[width=\linewidth]{images/net1_alone.pdf}
	\caption{The architecture of middle fusion model (Net2). The activations of the pool5 layers of the two modalities are concatenated at merging layer, and feeding it into the three fully connected layers with softmax at the end. }
	\label{net2}
\end{figure}

\subsubsection{Decision or late fusion model}
In contrast to the middle fusion model, extracted unimodal features are separately learned to compute unimodal scores, then these scores are integrated to determine a final scores. The late fusion model consists of two-streams processing audio and image data (green and blue) independently, which are fused after last fully connected layers as shown in Fig.\ref{net3}. Among the various ways of combining CNNs with different modalities, one straightforward way is to add additional fully connected layers to combine the outputs of the each streams as presented in \cite{eitel2015multimodal}. Instead of using FC layer to combine the two streams, we applied element-wise summation and element-wise multiplication to fuse the outputs of the each streams.

\begin{figure}[t]
	\includegraphics[width=\linewidth]{images/net3_alone.pdf}
	\caption{The architecture of late fusion model (Net3). The last fully connected layers of each model holds the unimodal scores for each class, which fused at fusing layer by summing and multiplying the unimodal scores. }
	\label{net3}
\end{figure}

\section{Related work}
\label{sec:re}
\par
Our work is related to the problem of recognition from multimodal data as well as convolutional neural networks for image classification. We will briefly highlight connections and differences between our work and existing works.
\par
Deep neural networks (DNN) have successfully applied for single modality such us text \cite{collobert2011natural,zhang2015character,kim2014convolutional}, images \cite{le2013building,lecun1990handwritten,lecun1989backpropagation} and audio \cite{hinton2012deep,petetin2015deep} showing their ability to learn representations directly from raw data and can be used to extract a set of discriminative features. CNN is one powerful deep architecture of DNN commonly utilized for image classification \cite{krizhevsky2012imagenet, szegedy2015going, he2016deep}. The use of CNN for distinguishing between fine grained categorization such as bird species categorization has been proposed in many studies \cite{zhang2014part,branson2014bird,guo2018fine,lebedev2018impostor,he2017fine}, which employ part/pose based approaches to achieve good performance. Besides, CNNs have also been applied for speech processing \cite{sainath2013deep,abdel2014convolutional}. In \cite{takahashi2018aenet,espi2015exploiting}, authors utilized CNNs to extract features from spectral representations of audio recordings. It has proved to be efficient when extracting features based on spectrograms of audio recordings such as bird sound \cite{kahl2017large,piczak2016recognizing}, thus we used spectrogram representations for audio data.
In this paper, instead of employing part/pose based approaches we will focus on integrating raw image an audio using conventional CNN, in order to know which fusion strategy performs best.
\par
Multimodal learning algorithms have been used for tasks such as image sentence matching \cite{ma2015multimodal}, action recognition \cite{simonyan2014two}, RGB-D object recognition \cite{eitel2015multimodal}, and speech recognition (audio-visual speech recognition \cite{noda2015audio} and visual-only speech recognition \cite{tatulli2017feature}). Among the many approaches for multimodal learning, multimodal integration is commonly realized by three different categories of approaches. First, in early fusion approach, feature vectors from multiple modalities are concatenated and transformed to acquire a multimodal feature vector. For example, Ngiam et al. \cite{ngiam2011multimodal} utilized DNN to extract fused representations directly from multimodal signal inputs. Likewise, in middle fusion approach, Huang et al. \cite{huang2013audio} employed deep belief network (DBN) to combine mid-level features learned by single modality. Lastly, in late fusion approach, outputs of unimodal classifiers are merged to determine a final classification. For example, in RGB-D object recognition, Eitel et al. \cite{eitel2015multimodal} proposed two separate CNN streams processing RGB and depth data independently are combined with late fusion approach. Therefore,  Simonyan et al. \cite{simonyan2014two} proposed two-stream (one stream processing spatial features from RGB image inputs, while the other stream processing temporal features from optical flow inputs) network architecture designed to recognize action for videos. They combined two streams by concatenating features and by averaging prediction scores from two CNNs, respectively. In contrast to these works, we propose simple yet effective concatenation, summation and multiplication based fusion methods with respect to three strategies.

\section{Conclusion}
\label{conclusion}
In this paper, we proposed three multimodal CNN architectures in different fusion strategies, which can process jointly the image and audio data for bird classification. Experimental results verified that the two-stream multimodal CNN in late fusion strategy outperforms the others. In addition, we proposed summed fusion method to combine multiple CNNs, which shows better performance comparing against several existing fusion methods. Moreover, with the help of two-stage fine-tuning, our method can be more effective. However, there still exist several drawbacks of our method: (1) Choosing the suitable duration based on the vocal features of birds to be recognized, is essential ingredients of improvement, is missed in our current work. (2) Our method is based on the raw image data, thus part detection and extracting features from pose-normalized regions may improve the classification performance.
\par
As the future work, we aim to apply multiple kernel learning to combine multiple modalities, which is able to learn optimal composite kernel through combining basis kernels constructed from different features of modalities. We are also interested in designing new CNN architectures by increasing the number of modalities.
\section{Database}
\label{database}
We evaluate our models on the popular fine-grained CUB-200-2011 \cite{wah2011caltech} bird dataset and our originally collected sound dataset from sharing bird sound database Xeno-Canto\footnote{\url{http://www.xeno-canto.org}}.

\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.45\textwidth}
   \includegraphics[width=\linewidth]{images/albatross_font_changed.png}
   \caption{}
\end{subfigure}
\begin{subfigure}[b]{0.4\textwidth}
   \centering {\includegraphics[width=0.8\linewidth]{images/albatross_10_changed.png}}
   \caption{}
\end{subfigure}
\caption[Spectrogram of the black-footed albatross]{(a) Radio (top) and spectrogram (bottom) of the black-footed albatross. (b) The spectrogram of 10 seconds duration, which will be fed into the bird classification model.}
\label{spectrogram}
\end{figure}

CUB-200-2011 \cite{wah2011caltech} dataset contains 11788 images of 200 species of birds, with each image downscaled to 227 $\times$ 227 pixel. Spectro-temporal features (spectrogram) are extracted from audio recordings that we collected from the Xena-Canto to be used as the audio representation. Based on the 200 species of the CUB-200-2011, we try to harvest at least 10 different audio recordings for each species. As a result, audio recordings over 178 species were collected completely ($\mathrm{\#recordings}\geq10$), audio recordings from 19 species were collected deficiently ($0<\mathrm{\#recordings}<10$), and 3 species could not be collected. The spectrograms of the audio are obtained by using short-time fourier transform (STFT) over 10 seconds audio frames, windowed with Hanning window (size 512, 50\% overlap). The reason is, the sounds of birds are usually contained in a small portion of the frequency range (mostly around 2-8 kHz) as stated in \cite{cakir2017convolutional}, so we only extract features from the range of (0, 10) kHz. In order to focus only sounds produced in the vocal organ of birds (i.e. calls and songs), first we obtained the maximum amplitude of the audio and removed a frame which contains only amplitude less than $1/4$ of the maximum amplitude. Finally, the spectrograms are saved as 227$\times$227 pixel color images, and the dataset contains 4807 images of 194 species of birds.
Several examples including both images and spectrograms over different bird species are shown in Fig.\ref{dataset}. We follow the standard training/test split of CUB-200-2011 dataset suggested in \cite{wah2011caltech}. The sound dataset is split into two halves for training and test set respectively.
\par
The multimodal CNNs are trained in a supervised manner, thus we create integrated dataset by matching two data sets and corresponding labels using HDF5 \footnote{\url{https://www.hdfgroup.org/HDF5/}} file format. Because the HDF5 file can contain any collection of data entities (images, videos, audio recording, text, etc.,) in a single file, and used to organize  heterogeneous collections consisting of very large and complex datasets.

\begin{figure}[t]
	\centering
	\begin{subfigure}{.25\linewidth}
		\centering
		\includegraphics[width=0.95\textwidth]{images/43.jpg}
		\caption{}
	\end{subfigure}%
	\begin{subfigure}{.25\linewidth}
		\centering
		\includegraphics[width=0.95\textwidth]{images/128.jpg}
		\caption{}
	\end{subfigure}%
	\begin{subfigure}{.25\linewidth}
		\centering
		\includegraphics[width=0.95\textwidth]{images/53.jpg}
		\caption{}
	\end{subfigure}%
	\begin{subfigure}{.25\linewidth}
		\centering
		\includegraphics[width=0.95\textwidth]{images/86.jpg}
		\caption{}
	\end{subfigure}%
	\caption{An example of CUB-200-2011 and audio dataset: (a) the yellow bellied flycatcher, (b) the seaside sparrow (c), the western grebe, and (d) the pacific loon.}
	\label{dataset}
\end{figure}
\section{Experiments and Results}
\label{experiment}
The experiments in this section are conducted to evaluate the effectiveness of our proposed architectures (Net1, Net2, Net3). In order to evaluate an advantage of our late fusion approach, we conduct a comparative experiment between Net3 model and two different existing fusion approaches \cite{eitel2015multimodal,simonyan2014two}. Besides, we have fine-tuned a pretrained model for all the models (including comparative models) in order to improve the performance. To improve the repeatability and only focus on the evaluation of fusion step, we use the well-known CaffeNet model \cite{jia2014caffe} to extract features, train, and fine-tune the CNN with default structure and parameter setting fixed except the base learning rate and batch size. We set the learning rate to 0.001 for single modality and 0.0001 for multimodal learning. The batch size is set to 32 for single modality and 1 for multimodal learning due to limited resources.

\subsection{Quantitative result: Single-modality v.s. Multi-modality}
The core idea of this paper is to address the integration of the image and audio. Therefore, it is necessary to compare the performance between single-modality (image or sound) and multi-modality (both image and sound: Net1, Net2, Net3). To focus on the evaluation of fusion models, in this experiment, we do not introduce any transfer learning techniques (e.g., fine-tune the pretrained model to help differentiate between gains from the proposed architectures). Table \ref{table1} summarizes the results of single modality and proposed models. We can observe that combining two modalities using CNN improves the performance of those only using one modality, extracting features separately from image and audio and fusing them at the late stage performs better with significant gain. Interestingly, the performance of low-level and mid-level fusion models slightly better than the performance of single-modality. One possible reason is because CNN learns features for the predominant modality. In contrast, learning features separately for different modalities results in more independent features, which leads to achieve better performance. Let us mention that the result obtained by multi-modality is different from simply combining the results of two CNNs trained separately. Indeed, the two modalities' parameters are jointly estimated and thus can be mutually influenced. We provide a detailed discussion in following section.
\begin{table}[!t]
	\caption{Comparative results between individual modality and multimodal CNNs.}
	\label{table1}
	\renewcommand{\arraystretch}{1.2}
	\centering
	\begin {tabular}{cl|c}
	\hline \hline%

	\multicolumn {2}{c}{Method} & \parbox{2cm}{\centering Accuracy (\%)}
	\\\hline
	\multirow{2}{*} {Single modality} 		& Image	& 16.2		\\\cline{2 - 3}
	& Audio & 46.4	\\\hline
	\multirow{3}{*} {Multi modality} 	& Net1 	& 50.0		\\\cline {2 -3}
	& Net2 	& 49.9		\\\cline {2 -3}
	& \bf Net3 (summation)	& \bf 53.8	\\\hline
	\end {tabular}
\end{table}

Figure \ref{net3_all} plots learning process of single modality (image only and audio only) and Net3 model over learning epochs. It can be observed that Net3 significantly improves the results.

\begin{figure}[b]
	\centering
	\includegraphics[width=\linewidth]{images/curve.pdf}
	\caption{Test accuracy vs. Epoch.}
	\label{net3_all}
\end{figure}

\subsection{Qualitative result: Single-modality v.s. Multi-modality} \label{ssec:qualitative}
We perform a qualitative study to analyze effects of multimodal learning models by comparing single-modality and multi-modality networks. First, we select some classes where the multi-modality models provides the correct answer while the single-modality model produces the wrong classification. Then, we study why the multi-modality models provides right answers while the single-modality model failed to produce the correct classification. Figure \ref{singlevsmulti} shows some examples of single-modality vs. multi-modality classification. In the first column, the single-modality models predict the input image and spectrogram as the 'barn shallow' and the 'ring-billed gull' respectively rather than the 'red-bellied woodpecker'. However, the multi-modality models are able to predict the right answer, because those models provide joint features of different modalities. We observe that when single-modality model provide right answer for spectrogram, the probability of providing the right answer of multimodality models is higher than when single-modality classificaion is correct for image. In the second column of Fig. \ref{singlevsmulti}, single-modality model is misclassified the 'clark nutcracker' image as the 'great grey shrike', where other models provide correct answers. Lastly, the Net3 model is able to provide right answers while the other models provide misclassification on the 'belted kingfisher' (last column).
\par
To better understand the difference between the models, we analyze the feature learned from the each networks by visualizing the filters of the first convolutional layers shown in Fig. \ref{filter}. We see that each network's filters of different models have similar pattern. Secondly, we see that the single-modality's filters have more meaningful patterns than the multi-modality filters. As we mentioned before its seems that the learning features separately for different modalities results more independent features. Finally, it can be seen that the filters of early fusion model has combined patterns from both networks, but most filters are similar to the spectrogram network. It reveals that the multimodal model (early fusion and mid-level fusion) learns features for the predominant modality.
\begin{figure}[t]
\centering
\begin{tabular}{m{2.5em}ccc}
\multirow{2}{*}{\rotatebox[origin=c]{90}{Input}} & \includegraphics[width=0.23\linewidth]{images/Red_Bellied_Woodpecker_0050_180751.jpg}& \includegraphics[width=0.23\linewidth]{images/Clark_Nutcracker_0075_85715.jpg}& \includegraphics[width=0.23\linewidth]{images/Belted_Kingfisher_0084_70399.jpg
\\
& \includegraphics[width=0.23\linewidth]{images/4521.png} & \includegraphics[width=0.23\linewidth]{images/2016.png} & \includegraphics[width=0.23\linewidth]{images/1764.png}
\\
Image & \ding{55} &  \ding{55}  &  \ding{55}\\
Sound & \ding{55} &  \ding{51} &  \ding{55}\\
Net1 & \ding{51} & \ding{51} & \ding{55}\\
Net2 & \ding{51} & \ding{51} & \ding{55}\\
Net3 & \ding{51} & \ding{51} & \ding{51}\\
\end{tabular}
\caption{Effects of combining image and spectrogram. Top two rows show sample image and spectrogram of different bird species where are fed into single-modality models and multi-modality models. The bottom rows show the resulting classification, where multimodal networks provide a correct classification while the single-modality classification are incorrect.}
\label{singlevsmulti}
\end{figure}

\subsection{Quantitative result: Net3 v.s. existing late fusion approaches}
To evaluate the effectiveness of our late fusing approach, we conduct comparative experiments on Net3 with existing late fusing methods presented in \cite{simonyan2014two,eitel2015multimodal}. The differences between the late fusion approaches is shown in Fig. \ref{fig:fusion_difference}.

\begin{figure}[!b]
\begin{tabular}{m{1em}m{3cm}m{3cm}}
\rotatebox[origin=c]{90}{Single-modality} &\includegraphics[width=\linewidth]{images/image_conv1_X.png}&
\includegraphics[width=\linewidth]{images/sound_conv1_Y.png} \\
\rotatebox[origin=c]{90}{\centering{Net1}} & \multicolumn{2}{c}{\includegraphics[width=.38\linewidth]{images/net1_conv1_1.png}} \\
\rotatebox[origin=c]{90}{Net2} &\includegraphics[width=\linewidth]{images/net2_conv1_X.png}&
\includegraphics[width=\linewidth]{images/net2_conv1_Y.png}\\
\rotatebox[origin=c]{90}{Net3} &\includegraphics[width=\linewidth]{images/net3_conv1_X.png} &
\includegraphics[width=\linewidth]{images/net3_conv1_Y.png}
\end{tabular}
\caption{Visualization of 96 filters of the first convolutional layer. Left side shows the filters related to the image network, while right side shows the filters related to the spectrogram network. It can be seen that the filters (left or right side) of different models have similar pattern. However, the filters of Net1 seems to have mixed filters of both networks.}
\label{filter}
\end{figure}

\begin{itemize}
\item \textbf{FC7 concat\cite{eitel2015multimodal}}: FC7 layers (green and blue) of each networks are concatenated and merge into the fusion layer, which performs tensor multiplication of two vectors. The resulting fusion vector is then passed through one additional fully-connected layer for classification. This means this fusion methods is a linear combination of pair-wise interactions between two features. However, this method is not suitable when the features are in different sizes.
\item \textbf{Averaging scores\cite{simonyan2014two}}: Each networks focus on learning features from images and spectrograms, respectively, and the final classification is computed as an average of the softmax scores of the two networks. In this fusion method, they do not consider pair-wise interactions between the features. However, this method is suitable when the model consists of different structured network streams.

\end{itemize}
In terms of pair-wise interactions, our method is similar to FC7 concat method. However, we fused final output of each networks. Figure \ref{net3_all} plots learning curve of Net3 model with different late fusing methods for each epoch, indicating that averaging the Softmax scores gives lowest performance and the our fusing approach performs best.

\begin{figure}[b]
    \centering
    \includegraphics[width=.9\linewidth]{images/late_fusions.pdf}
    \caption{Differences between the late fusion approaches.}
    \label{fig:fusion_difference}
\end{figure}

\subsection{Fine-tuning the pretrained model}
Combining multi-modalities at the late stage of CNN has proven to be more effective, thus we show additional results with fine-tuning CaffeNet pretrained CNN under Net\ref{net3} model in this section. One natural idea for fine-tuning is to train the model by initializing both image and audio CNNs with the weights and biases of the first seven layers derived from CaffeNet pretrained network, discarding the last fully connected layer. Instead of last fully connected layer of the pretrained model, we randomly place the initialized new fully connected layer for 200-class bird classification (in our experiment, 194 classes due to the lack of audio dataset).
\par
\begin{figure}[p]
\centering
\begin{tabular}{m{1em}m{3cm}m{3cm}}
\rotatebox[origin=c]{90}{Input} &\includegraphics[width=\linewidth]{images/Pacific_Loon_0006_75386.jpg} &
\includegraphics[width=\linewidth]{images/1840.png} \\
\rotatebox[origin=c]{90}{Pool1} &\includegraphics[width=\linewidth]{images/pool1_X.png} &
\includegraphics[width=\linewidth]{images/pool1_Y.png}\\
\rotatebox[origin=c]{90}{Pool2} &\includegraphics[width=\linewidth]{images/pool2_X.png} &
\includegraphics[width=\linewidth]{images/pool2_Y.png} \\
\rotatebox[origin=c]{90}{Pool5} &\includegraphics[width=\linewidth]{images/pool5_X.png} &
\includegraphics[width=\linewidth]{images/pool5_Y.png} \\
\multicolumn{3}{c}{(a) Pooling layer}  \\
 & \multicolumn{2}{c}{\includegraphics[width=0.7\linewidth]{images/FC_copy.png}} \\
\multicolumn{3}{c}{(b) Fully connected layer} \\
\end{tabular}
\caption{Feature visualization of network layer. These are examples of features at different levels in Net3, where (a) shows the features of different pooling layer of image (left) and audio (right) network, and (b) top to bottom shows the features of last fully connected layer of image only, spectrogram only, Fused layer using FC7 concat, and fused layer using summation. Here, the red rectangle shows incorrect answer, the green rectangle shows correct answer of the classification.}
\label{activation_visualization}
\end{figure}
Another method of fine-tuning the model is to train Net3 in two stages. First, training the two stream individually followed by a joint fine-tuning. We train the image and audio CNNs separately, adapting the weights of pretrained model and learn the weights of the new 194-class output layer. After this training, the networks can be used to perform separate classification with respect to each modality. After then, we train an entire model by setting their learning rate to zero and only training the fusion part of the network to freeze the individual stream networks. As shown in Table \ref{table2}, two stage training is resulted best performance, and it proves that additional modality can improve the performance. We found that the fine-tuning pretrained model to two modalities and training them simultaneously, is significant worse than two stage training. We think the problem relates to the difference between the size of two datasets and batch size of single-modality and multi-modalities. In this experiment, we used net surgery\footnote{\url{https://github.com/BVLC/caffe/blob/master/examples/net_surgery.ipynb}} to fine-tune two different CNN using one pretrained model and to fine-tune two pretrained model into our model.

To confirm that the features extracted from sample image and spectrogram during the fine-tuning were meaningful, we visualized the activations of different layers, especially the fused layers in Net\ref{net3}. The results shown in Fig.\ref{activation_visualization}, that allowed us to confirm the learned features were meaningful and qualitatively resembled the sample image and spectrogram. Moreover, the activations of our fused layer takes advantage by incorporating the features from each streams, when each network failed to produce write answer.
\begin{table}[t]
	\caption{Classification performance of fine-tuned Net\ref{net3} model with different fusion and fine-tuning method.}
	\label{table2}
	\renewcommand{\arraystretch}{1.2}
	\centering
	\begin {tabular}{cl|c}
	\hline \hline%

	\multicolumn {2}{c}{Method} & \parbox{1.5cm}{\centering Accuracy (\%)}
	\\\hline
    \multicolumn {2}{c|}	{Fine-tuning weights of pretrained model (summation)}	& {65.0}	\\\hline
	\multirow{4}{*} {Two stage fine-tuning} 	& \textbf{Summation (ours, Net3)} 	& \textbf{78.9}		\\\cline {2-3}
	& Multiplication (ours, Net3) 	& 75.0		\\\cline {2 -3}
    & Simonyan et al. \cite{simonyan2014two}	& 70.0		\\\cline {2 -3}
	&  Eitel et al. \cite{eitel2015multimodal}	&  72.5	\\\hline
	\end {tabular}
\end{table}
\section{Introduction}
\label{sec:intro}
Identifying the species of a bird is a widely-studied problem to ornithologists, and an important task in ecosystem monitoring and biodiversity preservation. Recognition of bird species in images is a challenging task due to various appearances, backgrounds, and environmental changes. Despite of this, this fine-grained recognition task has received a significant amount of attention in the computer vision community \cite{zhang2014part, branson2014bird, cui2016fine, gavves2015local} because of its potential widespread applications. Compared to generic object recognition, fine-grained recognition benefits more from learning critical parts of the object that can help align objects of same class and discriminate between neighboring classes. Current state-of-the-art methods (e.g., \cite{zhang2014part,branson2014bird,guo2018fine, lebedev2018impostor,he2017fine}) adopt CNN-based architectures that learn representations directly from the raw data and can be used to extract set of discriminative features.
\par
On the other hand, sound also provide us important information about the world around us. Many animals make sounds either for communication or their living activities such as moving, flying, mating etc. Although sound is in some case complementary to visual information, such as when we listen to something out of view, vision and hearing are often informative about the same structures in the world \cite{owens2016ambient}. As a consequence, numerous efforts have been devoted to recognize bird species based on auditory data \cite{kahl2017large,cakir2017convolutional} in recent years. Adapting CNN architectures for the purpose of audio event detection has become a common practice and generating deep features based on visual representations of audio recordings has proven to be very effective \cite{takahashi2018aenet} such as in bird sounds \cite{piczak2016recognizing,cakir2017convolutional}. To improve the affinity between images and sounds, we utilize the CNN to extract features from spectrogram of audio recordings.
\par
In the real world, human are able to handle information consist of different information from multiple plural modalities. In the filed of machine learning, multimodal recognition can improve performance compared with unimodal recognition by utilizing complementary sources of information \cite{srivastava2012multimodal,ngiam2011multimodal}. Multimodal learning have been used for tasks such as image and sentence matching \cite{tatulli2017feature}, RGB-D object recognition \cite{eitel2015multimodal}, action detection \cite{simonyan2014two} and specially speech recognition \cite{tatulli2017feature,noda2015audio,meutzner2017improving,torficoupled}, fusing different modalities. The results have shown that one modality can enhance the performance of the other by providing relevant information. Furthermore, authors of \cite{ngiam2011multimodal,huang2013audio} proposed fusion schemes for multimodal learning with considering the architectures of neural networks.
\\
Our main contributions are summarized as follows:
\begin{itemize}
    \item We propose that the combination of image and sound provide richer training signal for bird species classification under CNN framework, which is the first attempt to the best of our knowledge.
    \item Three strategies are investigated for fusing audio and image modalities using CNN.
\item We collect at least 10 audio recordings for each bird over 178 species, corresponding to the image dataset CUB-200-2011 \cite{wah2011caltech}.
\end{itemize}
Specifically, we adopt CNN to process jointly the two modalities for bird species classification. Three strategies are investigated for fusing audio and image modalities using CNN: (1) an early fusion strategy in which the feature vectors related to each modality are concatenated together and input to the CNN. (2) A middle fusion strategy. Features learned by each single modality are combined at the mid-level of the CNN. (3) A late fusion strategy. Outputs of single modality are fused to determine a final classification.
Our experimental results show that the architecture with late fusion strategy outperforms among the proposed architectures, which indicates that combining decisions of the classifiers from two modalities is superior. In addition, we apply a two-stage training procedure, which improves the classification accuracy.
\par
The rest of the paper organized as follows. In section 2, we review related works on bird species identification with CNN and multimodal learning algorithms. Section 3 describes several architectures among which a multimodal CNN processing jointly the two modalities. In section 4, we describe the integrated dataset utilized for the evaluation experiments of our architectures. Section 5 describes the experimental setup and results. In section 6, we conclude this paper.
\section{Methodology}
\label{sec:method}
Our three multimodal architectures extend conventional CNN for large-scale image classification \cite{krizhevsky2012imagenet}. Our implementation is based on CaffeNet , and can be treated as \cite{jia2014caffe} a variation of the structure proposed by Krizhevsky et al. \cite{krizhevsky2012imagenet}.
\subsection{Feature extraction}
CNN uses hierarchical features in its processing pipeline. The features from initial layers are primitive while late layers are high-level abstract features made from combinations of lower-level features. The CaffeNet consists of five convolutional layers (with max pooling layers following the first, second and fifth convolution layer) followed by three fully connected (FC) layers and a softmax classifier. Rectified linear unit is applied to every convolutional layer and fully connected layer and  local normalization is applied in first and second convolutional layer. The process through this 8-layer CNN network can be treated as a process from low to mid to high-level features. We hypothesize, that combining the features of different layers in this pipeline can lead to achieve better performance.
\subsection{Feature combination}
We propose our method in three strategies to fuse features: early fusion, middle fusion and late fusion. Early fusion, also known as feature level fusion, is a feature combination scheme that features from multiple modalities concatenated to form a merged feature vector. Middle fusion, also called mid-level combination, combines the high-level features learned by single network. Late fusion, also called decision-level fusion, combines the decisions of the unimodal classifier and determine the final classification. In this paper, we use concatenation to combine low and high-level features, and summation or multiplication to combine decisions of the classifiers.
The feature combination layers can be trained with standard back-propagation and stochastic gradient descent.

\begin{figure}[tb]
\includegraphics[width=\linewidth]{images/net2_alone.pdf}
\caption{The architecture of early fusion model (Net1). $ 227 \times 227$ pixel RGB images of two modalities are concatenated at merging layer, which produces $ 227 \times 454 \times 3$ output volume, and the convolution layers will extract joint features from this merged volume. We use HDF5 format to manage datasets of two modalities, because of it's flexible data storage and unlimited data types.}
\label{net1}
\end{figure}

\subsection{Architectures of proposed models}
\label{sec:apm}

The proposed multimodal learning models which combines audio and image using CNN by different fusion approaches are described in this section. We exploit the same architecture for both audio and image modalities to focus on evaluating the effectiveness of the feature combination approaches.
\subsubsection{Early fusion model}
One direct approach for combining audio and image is to train a CNN over the concatenated audio and image data as shown in Fig. \ref{net1}. In this strategy, the input vectors related to each modality are concatenated together and then processed together throughout the rest of the CNN pipeline. This model is most computational efficient comparing to the middle and late fusion models, because the number of learnable weight parameters is almost half times less than late fusion model.

\subsubsection{Middle fusion model}
In the middle fusion strategy, unimodal features is extracted independently from audio and images, then combined into a multimodal representation by concatenating the activations of the last pooling layers of two modalities. The multimodal representation then learned in the following fully connected layers. The middle fusion model is shown in Fig.\ref{net2}.

\begin{figure}[tb!]
	\includegraphics[width=\linewidth]{images/net1_alone.pdf}
	\caption{The architecture of middle fusion model (Net2). The activations of the pool5 layers of the two modalities are concatenated at merging layer, and feeding it into the three fully connected layers with softmax at the end. }
	\label{net2}
\end{figure}

\subsubsection{Decision or late fusion model}
In contrast to the middle fusion model, extracted unimodal features are separately learned to compute unimodal scores, then these scores are integrated to determine a final scores. The late fusion model consists of two-streams processing audio and image data (green and blue) independently, which are fused after last fully connected layers as shown in Fig.\ref{net3}. Among the various ways of combining CNNs with different modalities, one straightforward way is to add additional fully connected layers to combine the outputs of the each streams as presented in \cite{eitel2015multimodal}. Instead of using FC layer to combine the two streams, we applied element-wise summation and element-wise multiplication to fuse the outputs of the each streams.

\begin{figure}[t]
	\includegraphics[width=\linewidth]{images/net3_alone.pdf}
	\caption{The architecture of late fusion model (Net3). The last fully connected layers of each model holds the unimodal scores for each class, which fused at fusing layer by summing and multiplying the unimodal scores. }
	\label{net3}
\end{figure}

\section{Related work}
\label{sec:re}
\par
Our work is related to the problem of recognition from multimodal data as well as convolutional neural networks for image classification. We will briefly highlight connections and differences between our work and existing works.
\par
Deep neural networks (DNN) have successfully applied for single modality such us text \cite{collobert2011natural,zhang2015character,kim2014convolutional}, images \cite{le2013building,lecun1990handwritten,lecun1989backpropagation} and audio \cite{hinton2012deep,petetin2015deep} showing their ability to learn representations directly from raw data and can be used to extract a set of discriminative features. CNN is one powerful deep architecture of DNN commonly utilized for image classification \cite{krizhevsky2012imagenet, szegedy2015going, he2016deep}. The use of CNN for distinguishing between fine grained categorization such as bird species categorization has been proposed in many studies \cite{zhang2014part,branson2014bird,guo2018fine,lebedev2018impostor,he2017fine}, which employ part/pose based approaches to achieve good performance. Besides, CNNs have also been applied for speech processing \cite{sainath2013deep,abdel2014convolutional}. In \cite{takahashi2018aenet,espi2015exploiting}, authors utilized CNNs to extract features from spectral representations of audio recordings. It has proved to be efficient when extracting features based on spectrograms of audio recordings such as bird sound \cite{kahl2017large,piczak2016recognizing}, thus we used spectrogram representations for audio data.
In this paper, instead of employing part/pose based approaches we will focus on integrating raw image an audio using conventional CNN, in order to know which fusion strategy performs best.
\par
Multimodal learning algorithms have been used for tasks such as image sentence matching \cite{ma2015multimodal}, action recognition \cite{simonyan2014two}, RGB-D object recognition \cite{eitel2015multimodal}, and speech recognition (audio-visual speech recognition \cite{noda2015audio} and visual-only speech recognition \cite{tatulli2017feature}). Among the many approaches for multimodal learning, multimodal integration is commonly realized by three different categories of approaches. First, in early fusion approach, feature vectors from multiple modalities are concatenated and transformed to acquire a multimodal feature vector. For example, Ngiam et al. \cite{ngiam2011multimodal} utilized DNN to extract fused representations directly from multimodal signal inputs. Likewise, in middle fusion approach, Huang et al. \cite{huang2013audio} employed deep belief network (DBN) to combine mid-level features learned by single modality. Lastly, in late fusion approach, outputs of unimodal classifiers are merged to determine a final classification. For example, in RGB-D object recognition, Eitel et al. \cite{eitel2015multimodal} proposed two separate CNN streams processing RGB and depth data independently are combined with late fusion approach. Therefore,  Simonyan et al. \cite{simonyan2014two} proposed two-stream (one stream processing spatial features from RGB image inputs, while the other stream processing temporal features from optical flow inputs) network architecture designed to recognize action for videos. They combined two streams by concatenating features and by averaging prediction scores from two CNNs, respectively. In contrast to these works, we propose simple yet effective concatenation, summation and multiplication based fusion methods with respect to three strategies.