MANILA, Philippines – A young Filipina recently went viral for a unique 'blouse' she wore to one of the most popular tourist destinations in Asia: Ocean Park Hong Kong.
A lot of netizens had a good laugh over her strange OOTD but others were impressed that she was able to carry this look like she owned the style. She's certainly a model and genius 'designer' by her own right, netizens claimed.
The plastic bag in question was a rather unappealing neon green piece from SM hypermarket but Angelie Janelle Orleans was able to pull off the look, making the plastic piece look more glamorous than it was intended to be.
The reason behind her wearing that outfit also impressed a lot of people.
Her claim to fame started when her friend, Jerwyn Buenaventura, posted her OOTD on Facebook last Wednesday.
Some people might be puzzled about the 'Henry Sy' part but this was just a joke. The 'Who are you wearing today?' part refers to the designer of the outfit she's wearing, something that fashion magazines often use. But because she was wearing the SM plastic bag, Angelie's supposed designer was no less than the owner of the SM malls!
The neon green SM plastic bag 'blouse' was perhaps a silly thing to wear but Angelie had actually done this to save 40HKD (US$5.10 or PHP270.85) on a poncho being sold in the theme park so her clothes wouldn't get wet.
Jerwyn revealed that they were actually lining up to take The Rapids ride in Ocean Park Hong Kong and Angelie was concerned that her clothes might get wet and ruined in the ride. They had the option to buy the poncho but the wise young lady realized that the plastic bag just might work for her, instead of spending money on the poncho she would only use once.
But did the outfit actually work? Well, yes! Jerwyn revealed that the plastic bag 'blouse' actually made it through the rapids – and it did turn heads at Ocean Park Hong Kong. What a wise young lady!


text:
Panning for gold at Sovereign Hill (Courtesy Sovereign Hill)
Sovereign Hill By WhereTraveler ®
Sovereign Hill, an internationally acclaimed tourist attraction, brings to life the colour and excitement of Australia's 1850s gold rush. It's just like stepping back in time to gold rush Ballarat—from the hustle and bustle of Main Street where costumed ladies and gents parade their new-found wealth, to the excitement of the diggings where visitors can pan for real gold! There's also an underground mine tour, a spectacular $150,000 gold pour, the Redcoat soldiers who fire their powerful muskets, and much more. Convenient on-site accommodation is available at Comfort Inn Sovereign Hill.
Bradshaw St.
http://sovereignhill.com.au/
Map of Sovereign Hill

redpajama_set_name: RedPajamaCommonCrawl


text:
Karlovský tunel I je železniční jednokolejný tunel na železniční trati 086 Liberec – Česká Lípa. Nachází se v nadmořské výšce 455 m mezi stanicí Karlov pod Ještědem a zastávkou Kryštofovo Údolí v km 134,632–134,950. Tunel dlouhý 318 m byl dán do provozu v roce 1900.

Tunel byl ražen pod Srnčí horou v Ještědsko-kozákovském hřbetu firmou podnikatele Ing. Luigi Pelli. Ražba probíhala anglickou metodou a byly při ní poprvé použity elektrické stacionární vrtačky Siemens-Halske a elektrické ventilátory. Výrobu elektrického proudu zabezpečovala dynama poháněná parním strojem.

Tunel prochází územím národní přírodní památky Karlovské bučiny.

Odkazy

Reference

Související články
 Seznam železničních tunelů v Česku

Externí odkazy

Železniční tunely v okrese Liberec
Železniční trať Liberec – Česká Lípa
Ještědsko-kozákovský hřbet
Dopravní stavby v Liberci
Železniční doprava v Liberci

redpajama_set_name: RedPajamaWikipedia


Image Title: Kitchen Hanging Lighting Fixtures For Pendant Light Over Regarding Island Remodel 5. Post Title: Lighting For Kitchen Island. Filename: kitchen-hanging-lighting-fixtures-for-pendant-light-over-regarding-island-remodel-5.jpg. Image Dimension: 600 x 415 pixels. Images Format: jpg/jpeg. Publisher/Author: Marlin Bruen. Uploaded Date: Monday - September 24th. 2018 08:06:10 AM. Category: Architecture. Image Source: bhg.com. Kitchen Island Lighting You Ll Love Wayfair Pertaining To For Ideas 0. Kitchen Island Lighting You Ll Love Wayfair Within For Decor 9. Best Kitchen Drop Lights For Island 1000 Ideas Inside Lighting 3. Kitchen Island Pendant Lighting In A Cozy California Ranch For Remodel 2. Kitchen Hanging Lighting Fixtures For Pendant Light Over Regarding Island Remodel 5. Awesome Design Kitchen Island Pendant Lighting AWESOME HOUSE LIGHTING Throughout For Plan 13. Kitchen Island Lighting Is Hudson Valley 2623 PN Pertaining To For Designs 10. Nautical Pendant Lights For Kitchen Island Glass AWESOME HOUSE Intended Lighting Plan 6. Kitchen Island Lighting You Ll Love Wayfair With For Ideas 4. Kitchen Island Lighting Pictures Lettuceveg Com For Ideas 18.


An "intellectual, accepting, beautiful" liberal arts college in northeast Los Angeles, Occidental is "perfect for hard-working and involved students who not only want to be challenged academically, but also want to be pushed to learn more about the world around them. The faculty and staff "really encourage students to take a proactive role in their education," giving them the freedom to "experiment with a wide range of courses. In the classroom, Oxy professors "find ways to connect the lectures to the real world," and many are "very willing to have students help them with their research," providing valuable hands-on experience to undergraduates.


text:
\section{1. Introduction}

Automatic differentiation is an extremely powerful tool that is at the core of the current explosion of deep-learning~\cite{Goodfellow16}.
It allows the optimization of complex models without the \emph{burden} of the explicit computation of the derivatives~\cite{Martinez18}.
The flexibility of the resulting models explains their success in visual neuroscience~\cite{Kriegeskorte15,Yamins16,Cadena19}.
However, more important than the accuracy of the specific models,
the advent of machine learning descriptions of neural function stress
the need to clarify one central point~\cite{Kriegeskorte19}:
the strong interaction between \emph{goal} and \emph{architecture}.
Marr's \emph{computational} and \emph{algorithmic} levels of analysis of  vision~\cite{Marr76,Marr82} are not that independent: a biologically sensible goal (at the computational level) may lead to non-biological behaviors if the wrong architecture (at the algorithmic level) is chosen~\cite{Martinez19}.
As a result, it is not surprising that the behavior of current deep artificial neural networks (ANNs) departs from human behavior~\cite{WichmannHVEI17,GeirhosICLR19} despite they achieve human-like accuracy in the goal~\cite{Geirhos20beyond}.
For instance, relying on the universal approximation property of deep architectures
in a specific goal may lead to the proposition of too deep (unrealistic) models that display non-human behavior in other tasks.
This could be avoided by using more realistic layers/neurons that may
require shallower architectures~\cite{SciRep20}.
This interaction between goal and architecture may complicate the discussion
about the organization principles of the visual system.
In this context, the analysis of the similarities and differences between biological and deep-learning systems is interesting because it may reveal fundamental limitations of the conventional ANN analogy to study vision.

This work follows~\cite{Gomez18,Gomez20b}, where we did the first systematic study of color illusions in ANNs by subjecting the networks to classical psychophysical experiments. In order to explain the human-like nature of some of the illusions found in artificial systems, in~\cite{Gomez20b} we proposed a novel eigenanalysis of the networks. This analysis revealed the emergence of human-like chromatic channels and filters. That work on visual illusions of ANNs was focused on Convolutional Neural Networks (CNNs) and specifically on CNN-Autoencoders.
The goal of autoencoders is minimizing the error of the decoded signal in the representation domain of the input~\cite{Goodfellow16}.
Such simple low-level goal in the spatial domain may be reasonable for early vision stages like the Lateral Geniculate Nucleous (LGN)~\cite{MartinezNeuron14},
and it is known that spatio-chromatic contrast sensitivity functions (CSFs) can be derived from the responses of the LGN cells~\cite{InglingUriegas,Uriegas94,Shapley11}.

Our aims here are (1)~checking how the spatio-chromatic CSFs of CNN autoencoders may depend on the architecture even if the organization principle is fixed,
and (2)~the eventual emergence of similar human-like CSFs in 3D-CNNs trained with spatio-temporal signals. In order to do so we connect the eigenanalysis of the networks proposed in~\cite{Gomez20b} with classical psychophysical definitions of the CSF, and we assess the nonlinear response of CNNs for gratings of different contrasts.

The exploration of contrast response of CNNs trained on natural scenes updates classical statistical accounts of the CSFs~\cite{Atick92a,Atick92b}. These classical explanations were based on clever observations
about the 2nd order properties of natural images, but relied on linear filtering models.
Therefore, in principle, deep-learning models have two advantages:
(i) the use of actual image data instead of (idealized) models of images, and
(ii) the use of flexible networks that overcome the linear constraint and hence may develop masking saturations.

The eventual advantages of current deep-learning models seem obvious, but one may ask:
\emph{do more general (nonlinear) architectures necessarily lead to more human CSFs?},
or \emph{does better accuracy in the goal imply more human CSFs and masking behavior?}.
These are the kind of questions addressed by our experiments.

The structure of the paper is as follows.
Section~2 extends the theory proposed in~\cite{Gomez20b} to obtain the CSFs
of artificial systems with a low-contrast analysis based on classical psychophysics.
Section~3 shows the main empirical findings of the work: the emergence of human-like CSFs and contrast saturation in CNN-autoencoders with shallow architectures under the considered error minimization goal.
Interestingly, resemblance with human behavior reduces for deeper architectures.
Finally, Section~4 discusses the implications of the results:
on the one hand, statements about the goal or organization principles at an abstract level are problematic because the final behavior very much depends on the algorithmic level (or selected architecture).
On the other hand, special care has to be taken in using deep models in vision
science: their ability for function approximation without the appropriate architecture constraints does not guarantee human behavior.
This suggests a word of caution against premature interpretations of the behavior obtained through deep architectures in biological problems.

\vspace{0.4cm}
\section{2. Methods: Estimating Contrast Sensitivity in Autoencoders}
\label{Method}

Here we consider different linear characterizations of the autoencoders including the eigenanalysis proposed in~\cite{Gomez20b}.
In this way we propose a procedure to estimate the autoencoder CSF that is more connected to the conventional definition
of the CSF in human observers.

\vspace{0.0cm}
\subsection{Autoencoders}

Autoencoders are ANNs that transform the signal into an inner representation through an \emph{encoder} and a \emph{decoder} transforms this inner representation back in the input domain.
\begin{equation}
  \xymatrixcolsep{2pc}

  \xymatrix{ \vect{x} \ar@/^0.8pc/[r]^{\scalebox{0.80}{$\,\,\,\,\,S(\vect{x},\theta)\,\,\,\,$}} & \vect{y}  }
  \label{autoencoder}
\end{equation}

In Eq.~\ref{autoencoder} we do not made explicit the encoding and decoding operations, i.e. $\vect{x}$ and $\vect{y}$ are in the image space.
In this work we will not make any assumption on the nature of the inner representation of the autoencoder.
This is because the basic goal function in autoencoders (reconstruction error) is defined in the image domain, shared by input and response.
Moreover, with the appropriate stimuli, the CSF characterization can be defined in this image domain.

Following~\cite{Gomez18,Gomez20b} we focus on Convolutional Autoencoders trained to enhance blurred and noisy signals coming from
natural scenes. In this context, given a clean image, $\vect{x}_c$, the input to the neural system would be a distorted version of the clean signal:
$\vect{x} = H \cdot \vect{x}_c + \vect{n}$, where $H$ is blurring operator related to the optics of the eye and $\vect{n}$ is the noise associated to the response of the photodetectors. Both $H$ and $\vect{n}$ are unknown to the neural system.
The goal of the network at this early stage is inferring $\vect{x}_c$ from $\vect{x}$.
Accurate models of LGN cells show that this may be one of the goals of the biological processing after retinal detection~\cite{MartinezNeuron14}.

In the supervised learning setting of ANNs, the parameters $\theta$ of the network are selected so that the reconstruction error
$\varepsilon = |\vect{x}_c - S(\vect{x},\theta)|_2$ is minimized over a set of training images~\cite{Goodfellow16}.
Of course, supervised learning and parameter updates using backpropagation may not be biologically plausible~\cite{AgainstHinton}.
However, our initial aim here is looking for statistical explanations of human frequency sensitivity and hence ANNs can be seen as convenient tools to optimize the selected goal.
With this focus on the \emph{goal}, the specific learning algorithm is not as important as ensuring that the final network
actually fulfills the goal. We will see that the situation may not be that simple because networks optimizing the same goal with equivalent performance may display \emph{human} or \emph{non-human} CSFs depending on their architecture.

\subsection{Contrast Sensitivity Function and Linearized Autoencoders}

The CSF describes the linear response of human viewers for low-contrast sinusoids~\cite{Campbell68,Mullen85,Kelly79}.
In that linear setting, the CSF describes an input-output mapping where an input sinusoid of frequency $f$, the basis function $\vect{b}^f$, leads to an output, $\vect{y}^f$, with attenuated contrast (or standard deviation, $\sigma$) given by, $\sigma(\vect{y}^f) = \textrm{CSF}(f) \, \sigma(\vect{b}^f)$. In the case of humans the slope is obtained from contrast thresholds because there is no access to the output. However, for autoencoders the computation of the output is straightforward, $\vect{y}^f = S(\vect{b}^f,\theta)$.
Therefore, given an input sinusoid, one could check the attenuation factor by comparing the standard deviation of output and input:
\begin{equation}
       \textrm{CSF}(f) = \frac{\sigma(S(\vect{b}^f,\theta))}{\sigma( \vect{b}^f )}
       \label{csf1}
\end{equation}
This ratio should be computed for low-contrast sinusoids to keep the (eventually) nonlinear autoencoder in the low-energy range.
For chromatic sinusoids the deviations have to be computed separately over the achromatic, red-green, and blue-yellow color channels~\cite{Mullen85}.
In this work we use the Jameson and Hurvich opponent color space~\cite{Jameson59} to generate achromatic and purely chromatic gratings and to decompose the corresponding responses.

Of course, plain attenuation for sinusoids in Eq.~\ref{csf1} may not provide a full description of the action of nonlinear systems.
In principle it is not obvious why we should perform the analysis in a specific basis. Therefore one should check to what extent sinusoids are indeed eigenfunctions of the system.

A way to test this point is linearizing the response of the autoencoders in the low-contrast regime and check that it is shift invariant.
Using a Taylor expansion, the response for low-contrast images can be approximated by the Jacobian around the origin (the zero-contrast image, $\vect{0}$):
\vspace{-0.5cm}
\begin{eqnarray}
      \vect{y} & = & S(\vect{x},\theta) \nonumber \\
      \vect{y}_{\textrm{low}} & = & S(\vect{0} + \vect{x}_{\textrm{low}},\theta) \approx S(\vect{0},\theta) + \nabla_{\!\vect{x}} S(\vect{0},\theta) \cdot \vect{x}_{\textrm{low}} \\
      \vect{y}_{\textrm{low}} & \approx & \nabla_{\!\vect{x}} S(\vect{0},\theta) \cdot \vect{x}_{\textrm{low}} \nonumber
\end{eqnarray}
\noindent where we assumed that the response for zero-contrast images is zero.
If the behavior of the system at this low-energy regime is shift invariant, the Jacobian matrix can be diagonalized as $\nabla_{\!\vect{x}} S(\vect{0},\theta) = B\cdot \lambda \cdot B^{-1}$, with extended oscillatory basis functions in the columns of $B$ (and rows of $B^{-1}$).
Fourier basis and cosine basis are examples of extended (non-local) oscillatory functions that diagonalize shift invariant systems.
The reason for this result is equivalent to the emergence of cosine basis
when computing the principal components of stationary signals (shift invariant autocorrelation)~\cite{Clarke81}.
As a result, the slope of the response for low-contrast sinusoids (the CSF) will be related to the eigendecomposition of the Jacobian of the system at $\vect{0}$. Let's compute the response for a sinusoid in this Taylor/Fourier setting to see the relation.
A basis function $\vect{b}^f$ with specific frequency $f$ is orthogonal to all rows (sinusoids) in $B^{-1}$ except that of the same frequency, i.e. $B^{-1} \cdot \vect{b}^f = \vect{\delta}^{f'f}$. And this delta selects the corresponding column (of frequency $f$) among all the columns in B:
\begin{eqnarray}
      \vect{y}^f & = & S(\vect{b}^f,\theta) \nonumber \\
                 & \approx & B\cdot \lambda \cdot B^{-1} \cdot \vect{b}^f \nonumber \\
                 & \approx & B\cdot \lambda \cdot \vect{\delta}^{f'f} \nonumber \\
                 & \approx & \lambda_f \, \vect{b}^f
                 \label{eq_eigen}
\end{eqnarray}
So the slope of the response for basis functions of frequency $f$ is $\lambda_f$ (the corresponding eigenvalue of the Jacobian of the autoencoder).
As a result, for systems with shift invariance in the low-contrast regime, the eigenvalues of the linear approximation of the system (eigenvalues of the Jacobian) are \emph{conceptually} similar to the CSF.
Direct comparison of the eigenvalue spectrum with the CSF may not be simple because the eigenfunctions may differ from Fourier sinusoids and they are given in a specific order depending on the diagonalization algorithm.

Nevertheless, if the linearized version of the system (the Jacobian at $\vect{0}$) is shift invariant, which can be seen from a convolutional structure in the Jacobian matrix,
oscillatory waves are eigenfunctions of the system, and hence Eq.~\ref{csf1} may provide a good description of the behavior of the system.

\subsection{Alternative linear characterizations of the autoencoders}

A 2D cartoon of the degradation and restoration process can illustrate alternative characterizations of the neural networks optimized to enhance the retinal signal (see Fig.~\ref{fig_degrad_restore}).
In this diagram, two-pixel natural scenes (the left panel) follow a PDF obtained from independent t-student sources mixed by a matrix that introduces strong correlation between the luminance of the pixels.
This kind of two-pixel representations is common to describe the statistics of natural images~\cite{Olshausen01},
and mixtures of sparse components is a widely accepted model for natural scenes~\cite{Hyvarinen09,JMLR14,Malo20}, and
appropriate enough for this illustration.
In this diagram the low-frequency direction corresponds to the main diagonal (where the two pixels have the same luminance)
and the high-frequency direction is orthogonal (for images where one of the pixels is brighter than the other).
The zero-contrast image is at the crossing point of the frequency axes.

\begin{figure*}[t]
\begin{center}
\includegraphics[width=0.9\linewidth]{figure_1.JPG}
   \vspace{-0.2cm}
   \caption{Degradation of visual data and linear and nonlinear strategies for enhancement in the spatial domain.
   The axes of the plots represent the luminance in each of the photodetectors of two-pixel images \blue{(Olshausen01)}.
   The left panel represents the PDF of natural scenes: marginal heavy tailed distributions of oscillatory functions
   mixed to have strong correlation in the pixel domain. Optical blur (second panel) implies contraction of the high
   frequencies. Additive retinal noise implies the convolution by the PDF of the noise leading to the PDF in the third panel.
   The restoration problem may include (i) generic nonlinear solutions such as the one represented by the PDF at the right-top,
   and (ii) linear solutions as the one at the right-bottom.}
   \label{fig_degrad_restore}
\end{center}
\end{figure*}

Optical blur implies the attenuation of high-frequency versus low-frequency components and hence the contraction of the dataset
as shown in the second panel. Assuming linear and noisy photoreceptors, the PDF of the retinal response results from the convolution
of the PDF of the blurred images with the PDF of the noise (the function with circular support).
The result (third panel) is the input to the autoencoder, whose goal is recovering the distribution at the first panel.
Linear solutions are limited to global scaling of the domain (for instance by inverting the contraction introduced by the blur),
while nonlinear solutions may twist the domain in arbitrary ways.

In this setting, the computation of the CSF according to Eq.~\ref{csf1} means putting low-contrast sinusoids (e.g. the samples
highlighted in green in the third panel) through the system, and checking the amplitude of the output (green dots at
the panels at the right) over the directions of the input.
This nonlinear example illustrates the fact that the behavior can be contrast dependent (see the different twist in the concentric contours).
This graphical view illustrates the difference between three possible linear characterizations, with $\vect{y} = M\cdot \vect{x}$:
\vspace{0.0cm}
\begin{itemize}
     \item \textbf{The optimal linear solution:} the matrix $M$ that better relates the input $\vect{x}$ with the desired output $\vect{x}_c$.
           This is the $M$ that minimizes the expected value
           $E\{|\vect{x}_c - M \cdot \vect{x}|_2\}$.
           Assuming a representative set of $N$ clean/distorted pairs stacked in the matrices $\vect{X_c} = [\vect{x}_c^{(1)} \vect{x}_c^{(2)} \ldots \vect{x}_c^{(N)}]$ and $\vect{X} = [\vect{x}^{(1)} \vect{x}^{(2)} \ldots \vect{x}^{(N)}]$, the optimal RMSE solution is given by the pseudoinverse:
           \begin{equation}
                  M = \vect{X_c} \cdot \vect{X}^\dag
                  \label{linear_solution}
           \end{equation}
     \item \textbf{Globally linearized network:} the matrix $M$ that better describes the nonlinear behavior of the
           network \emph{over the whole set of natural images}.
           This is the $M$ that minimizes $E\{|S(\vect{x},\theta) - M \cdot \vect{x}|_2\}$.
           Assuming a representative set of $N$ input/output pairs stacked in the matrices $\vect{X} = [\vect{x}^{(1)} \vect{x}^{(2)} \ldots \vect{x}^{(N)}]$, and $\vect{Y} = [\vect{y}^{(1)} \vect{y}^{(2)} \ldots \vect{y}^{(N)}]$, the solution is given by the pseudoinverse:
           \begin{equation}
                  M = \vect{Y} \cdot \vect{X}^\dag
                  \label{global_linear_approx}
           \end{equation}
     \item \textbf{Locally linearized network at $\vect{0}$:} the matrix $M$ that better describes the nonlinear behavior of the
           network \emph{for low-contrast images}. This is the $M$ that minimizes $E\{|\nabla_{\!\vect{x}}S(\vect{0},\theta)\cdot\vect{x}_{low} - M \cdot \vect{x}_{low}|_2\}$. Of course, this could be empirically approximated by $M = \vect{Y}_{low} \cdot \vect{X}_{low}^\dag$, but in this case
           the obvious exact solution is:
           \begin{equation}
                  M = \nabla_{\!\vect{x}}S(\vect{0},\theta)
                  \label{local_linear_approx}
           \end{equation}
\end{itemize}

While the optimal linear solution (or the optimal linear network) is a convenient reference to describe the problem, the other two options are different characterizations of the autoencoder. Eq.~\ref{global_linear_approx} summarizes the behavior of the network in a single matrix,
and Eq.~\ref{local_linear_approx} is a description only valid around $\vect{0}$, and hence more closely connected to the low-contrast regime of the CSF. The eigenanalysis cited for $\nabla_{\!\vect{x}}S$ in Eq.~\ref{eq_eigen} can be applied for the three matrix characterizations, but it is important to note the differences between them.

The Jacobian of cascades of linear+nonlinear layers (as in autoencoders based on Convolutional Neural Networks) can be obtained
either analytically~\cite{Martinez18}, or it can be obtained via automatic differentiation or alternative methods
based on system identification~\cite{Laparra17}.
However, the above procedures are tedious, so in~\cite{Gomez20b} we took the more straightforward approach represented by Eq.~\ref{global_linear_approx}.

The different linear characterizations considered in this section and the example in Fig.~\ref{fig_degrad_restore} illustrate that
the behavior of a nonlinear autoencoder over high contrast images may be substantially different from the threshold behavior.
Therefore, the eigenanalysis used in~\cite{Gomez20b} and the corresponding attenuation of sinusoids through the matrix given by Eq.~\ref{global_linear_approx},
will also be compared with the result of Eq.~\ref{csf1}.

\vspace{0.4cm}
\section{3. Results}

Here we analyze the behavior of a range of autoencoders trained for the same statistical goal (enhancement of blurred/noisy retinal signals) using different architectures.
First we show that they indeed fulfill the goal.
Then, we analyze their global linear approximation according to Eq.~\ref{global_linear_approx} by showing their shift-invariant nature, their spatio-chromatic and spatio-temporal receptive fields, and their extended oscillatory eigenfunctions.
Finally, we apply Eq.~\ref{csf1} both to the nonlinear systems and to their linearized versions to get the corresponding CSFs.

\subsection{3.1 Architectures and performance}

In this work we consider 2D CNNs that act on spatio-chromatic signals and 3D CNNs that act on spatio-temporal signals (color images and achromatic videos).

The spatio-chromatic models follow the networks studied in~\cite{Gomez18,Gomez20b}:
autoencoders with 2D convolutional layers made of 8 feature maps with kernels of spatial size $5\times5$ and sigmoids or Rectified Linear Units (ReLU) as activation functions.
In every case, the last (reconstruction) layer has 3 features (the RGB color channels) so that the input and output domains are the same.

We train a range of progressively deeper architectures: 2 layers, 4 layers, 6 layers, and 8 layers.
We also consider a really deep architecture made of pre-trained State-Of-The-Art (SOTA) solutions to the considered problems: first we apply the denoising network~\cite{Zhang17}, and then the deblurring network~\cite{Tao18}. This cascade represents a sophisticated network with 17+21 = 38 layers.
Finally, we consider the best linear solution, Eq.~\ref{linear_solution}, as a convenient reference.

The spatio-temporal models follow the same spirit, in this case also including convolution in the temporal dimension: autoencoders with 3D convolutional layers made of 8 feature maps with kernels of size $5\times5\times5$ and sigmoid or ReLU activation functions. In every case, the last (reconstruction) layer has 1 feature map (the luminance). We consider architectures of different depth (2 layers, 4 layers, 6 layers, and 8 layers), and also the optimal linear network as useful reference.

Training of the spatio-chromatic models is done over the same dataset used in~\cite{Gomez18,Gomez20b}: the Large Scale Visual Recognition Challenge 2014 CLS-LOC validation dataset
(which contains 50 k images), leaving 10 k images for validation purposes. This dataset is a subset of the whole ImageNet dataset \cite{Russakovsky2015}.
We take $128\times128$ images and assume a sampling frequency $fs = 70 cpd$, i.e. we assume images subtend 3.6 deg. The cut off frequency of the Gaussian blur kernel is $fs/4 = 17 cpd$, and the variance of the zero-mean Gaussian noise is $\sigma = 15$ in the [0,255] scale of RGB channels.

Implementation and training is done in the same way as in~\cite{Gomez18,Gomez20b}:
mean squared error is used as loss function in all cases and all the models are implemented using Tensorflow~\cite{Abadi2015}.
We train our models using ADAM stochastic gradient descent~\cite{Kingma} with a batch size of 64 examples, momentum of 0.9, weight decay of 0.001. The maximum number of epochs is set to 100 and we stop the optimization early if there is no improvement in the validation set after 10 consecutive evaluations.

The spatio-temporal models are trained over patches of size $32\times32\times25$ from the movie \emph{The Stranger}~\cite{Welles46}. We use 10000 patches for training and 1000 patches for validation. We assume spatial sampling of 30 cpd and temporal sampling of 25 Hz.
Degradation used is the same as in the spatio-chromatic case, here applied frame-wise to the luminance channel. Loss function, implementation and learning method are the same as in the spatio-chromatic case.
Training data and code are available at \blue{http://isp.uv.es/code/visioncolor/autoencoderCSF.html}

\vspace{0.25cm}

Table~1 and Figures~\ref{fig_spatio_chrom} and~\ref{fig_spatio_temp}
show that the considered networks indeed learn to solve the spatio-chromatic and spatio-temporal problems.
While the table shows the values of the goal function over the whole validation set, the figures show representative visual results of the transmitted signals together with the goal values for specific stimuli of the validation set.
Results include the original stimuli, the input (the degraded retinal responses), the responses of the considered nonlinear networks, and the optimal linear solutions.

\begin{figure*}[t!]
\begin{center}
\includegraphics[width=0.9\linewidth]{figure_3.JPG}
   \caption{Representative responses of CNNs for two spatio-chromatic stimuli. Numbers indicate the values of the goal function (RMSE distance in RGB with regard to the original stimulus).
   }
   \label{fig_spatio_chrom}
\end{center}
\end{figure*}

In both cases (spatio-chromatic and spatio-temporal) the optimal linear network computed from the pseudoinverse, Eq.~\ref{linear_solution}, clearly improves the retinal signal in RMSE and the
high frequency noise is (visually) less apparent.
The nonlinear networks (regardless of the activation functions) further improve the linear results both in RMSE and visually (edges are sharper and noise has been reduced).
The table shows that the performance of the linearized versions of the networks, Eq.~\ref{global_linear_approx}, is
better than the optimal linear network but worse than the nonlinear networks.
This means that the response of the nonlinear networks introduces a sort of regularization that makes the resulting matrices more appropriate to solve the problem. Linearized responses are not shown in the figures because all reconstructions are visually similar and do not convey much qualitative information.

Increasing the depth of the CNN architectures improves the performance with regard to the shallower configuration: note the improvement from 2 to 4 layers both in RMSE and visually for every activation function. This is true for both the spatio-chromatic and the spatio-temporal cases.
However, the improvement saturates and adding extra layers (with no modification of the training strategy) does not necessarily reduce the reconstruction error. This is noticeable in the RMSE scores, but not that relevant in the visual results.
In some specific cases (e.g. 6 and 8 layers architectures with sigmoid activation) the networks introduce a shift in the chromatic content as seen in Fig.~\ref{fig_spatio_chrom} (top row).
This specific artifact will induce a clear distortion in one of the chromatic CSFs as shown below.

\begin{figure*}[t!]
\begin{center}
\includegraphics[width=0.9\linewidth]{figure_4.JPG}
   \caption{Representative responses of CNNs to a spatio-temporal stimulus. Numbers indicate the values of the goal function (RMSE distance in luminance with regard to the original sequence).  Data from \emph{The Stranger}~\blue{(Welles, 1946)}, including this image, can be freely used since the movie is in the Public Domain, and online available at \blue{https://archive.org/details/TheStranger720p}.}
   \label{fig_spatio_temp}
\end{center}
\end{figure*}

\begin{table}[b!]
\centering
\begin{tabular}{|c|cccc|cccc|}
\hline
                 &         \textbf{Spatio-Chrom}. & &  &   &   \textbf{Spatio-temp.}& &  & \\
                 &          RMSE in RGB  & &  &   &        RMSE in Lumin. & &  & \\ \hline\hline
Degradation      &          $30.5 \pm 0.2$ &         &  &   &        $16.12 \pm 0.05$ & &  & \\ \hline\hline
Linear Net   &         $22.7 \pm 0.3$  &        &  &   &        $11.1 \pm 0.1$ & &  & \\ \hline\hline
CNNs             & Sigmoid &               &  ReLU  &  & Sigmoid &               &  ReLU  &  \\
                 & Nonlinear & Linearized  &  Nonlinear  & Linearized & Nonlinear  & Linearized & Nonlinear  & Linearized \\ \hline
2-Layers         & $21.7 \pm 0.4$ & $21.0 \pm 0.3$  &  $20.6 \pm 0.5$  & $20.4 \pm 0.2$ & $8.3 \pm 0.1$ & $10.2 \pm 0.1$  &   $7.7 \pm 0.1$  & $10.0 \pm 0.1$ \\
4-Layers         & $19.2 \pm 0.5$ & $20.2 \pm 0.3$  &  $18.8 \pm 0.5$  & $20.3 \pm 0.3$ & $7.6 \pm 0.1$ & $10.0 \pm 0.1$  &   $7.8 \pm 0.1$  & $10.2 \pm 0.1$ \\
6-Layers         & $20.9 \pm 0.4$ & $21.9 \pm 0.3$  &  $19.1 \pm 0.5$  & $20.5 \pm 0.3$ & $7.9 \pm 0.1$ & $10.0 \pm 0.1$  &   $7.2 \pm 0.2$  & $10.2 \pm 0.1$ \\
8-Layers         & $20.0 \pm 0.4$ & $21.0 \pm 0.3$  &  $19.1 \pm 0.5$  & $20.4 \pm 0.4$ & $7.8 \pm 0.1$ & $10.1 \pm 0.1$  &   $8.9 \pm 0.1$  & $10.5 \pm 0.1$ \\ \hline \hline
SOTA             & Nonlinear  & Linearized  &    & & &  & &  \\
                 & $17.9 \pm 0.4$ & $20.2 \pm 0.3$  &  & &  & & &  \\ \hline
\end{tabular}
\label{table_RMSE}
\caption{Performance of the networks for the spatio-chromatic problem (left) and the spatio-temporal problem (right). Initial value of the goal function (RMSE), and result obtained by different architectures and activation functions. All scores refer to the average over 100 realizations
over randomly selected 80\% of the validation set, with the corresponding standard deviation.}
\end{table}

The saturation of the performance with the depth of the architecture is a known effect in deep learning~\cite{Glorot10,Priyanka19}. It is the reason for the development of architectural/training tricks
to keep the performance high in validation when increasing the number of free parameters~\cite{Komatsu20-U-architect,He16-Residuals,Bengio06-layer-wise}.
We didn't explore these specific tricks because of two reasons:
(1) the aim of the paper is not pure goal optimization but studying the eventual emergence of human behavior in the networks, and (2) the considered cases already show improvement of the performance from the shallower configurations to deeper configurations and the posterior decay in performance is small (and negligible in visual terms in most of the cases).

As shown below, these cases are enough to show how human-like properties may appear in the networks, and how sensitive this result may be with regard to modifications in the architecture.

Finally, the considered State-Of-The-Art (SOTA) system with a total of 38 layers does get the better RMSE and visual results.
These SOTA architectures were carefully handcrafted to this specific end.
Note that in Table 1 the global linearization of the SOTA network is not that different from the linearized versions of the other networks.
The key difference (in terms of the goal) is not in the linear approximation of the networks but in the different nonlinear behavior they have.

We didn't use the spatio-chromatic SOTA system for the enhancement of spatio-temporal signals because a frame-wise application of the 2D network would not exploit the 3D structure of natural movies.
Naive frame-wise application of 2D networks could introduce artifacts when studying the response to spatio-temporal sinusoids so we preferred to exclude 2D models from the spatio-temporal study focused on 3D CNNs.
Again, the considered 3D architectures reveal the fulfillment of the goal and display a range of performance values which are enough to discuss the emergence of human-like spatio-temporal bandwidths.

\subsection{3.2 Linearization and Eigenanalysis}

Figure~\ref{Linearization} shows representative results for the linearization of the considered networks.
The top row recalls the results reported in~\cite{Gomez20b} for the 2D spatio-chromatic networks with two convolutional layers and sigmoid activation, and the bottom row shows the results for the 3D spatio-temporal networks with two convolutional layers and sigmoid activation.
The equivalent results for the other networks are qualitatively similar.

\begin{figure*}[b!]
\begin{center}
\includegraphics[width=1.0\linewidth]{figure_2xy.JPG}
   \caption{Linearization of spatio-chromatic networks (top) and spatio-temporal networks (bottom).
   The matrices $M$ (in panels A and E), the receptive fields (panels B and F), the eigenfunctions (in panels C and G), and the eigenvalues (panels D and H).}
   \label{Linearization}
\end{center}
\end{figure*}

Following the discussion in Section 2, the matrices $M$ that result from Eq.~\ref{global_linear_approx} (panels A and E) have a clear convolutional structure: rows with the same shape up to a shift in the spatial or the spatio-temporal domain. Note that in this matrix notation the input and output have been vectorized, as shown in panel A.
This structure in $M$ reveals shift invariance in the receptive fields.
Representative receptive fields are shown in panels B and F, where the row highlighted in green in the matrix has been reshaped to be represented back into the spatial or the spatio-temporal domain.
All the linearized networks present these kind of center-surround receptive fields (a red-green example in panel B,
and an on-center/off-surround with distinct temporal responses in different spatial locations in panel F).
These receptive fields are replicated with the corresponding shift along the rows of the matrices $M$.
As anticipated in Section 2, following~\cite{Clarke81}, convolutional-like matrices lead to extended oscillatory eigenfunctions as illustrated in panels C and G.
The spatio-chromatic functions in panel C are similar to Cosine basis and interestingly, as noted in~\cite{Gomez20b}, the basis display oscillations in human-like red-green and blue-yellow opponent directions. See the original report for further analysis of the chromatic directions and color matching functions.
The spatio-temporal functions in panel G include extended textures (both in space and time), with flicker and motion, but they cannot be identified with Fourier or Cosine waves. However, it is clear that they display characteristic spatio-temporal frequencies (medium to low spatial frequencies at the top versus high spatio-temporal frequencies at the bottom).
The functions in panels C and G are ordered according to the eigenvalues (in panels D and H). This means that the systems favour medium and low spatio-temporal frequencies.

As stressed in~\cite{Gomez20b}, this eigenanalysis makes no other assumption than the linear approximation of the networks.
The structure of the matrices found and the order in the oscillating eigenfunctions reveal the suitability of Fourier-like basis and the higher relevance of medium and low frequencies.
Fourier basis are not imposed: they just become evident after the eigenanalysis of the linearized versions of the networks.
This justifies further study of the nets (both the original nonlinear networks and the corresponding linearized versions) through the conventional ratio of output/input signals assumed in psychophysics, Eq.~\ref{csf1}.

Following~\cite{Gomez20b} we computed the fraction of the nonlinear response of the CNNs that can be explained by the linear approximation, i.e. $1 - \frac{E\{|S(\vect{x}) - M\cdot \vect{x}|\}}{E\{|S(\vect{x})|\}}$.
This linear fraction, which is 100\% for a linear system and decreases with the intrinsic nonlinearity of the network, is given in Table~2.
Results show that the considered problems (and the corresponding networks) are mostly linear. Moreover, increasing the depth (increasing the number of parameters and potential flexibility) does not mean increasing the effective nonlinearity of the networks. This suggests that the networks are not overfitting the data.

\begin{table}[t!]
\centering
\begin{tabular}{|c|cc|cc|}
\hline
                 &         \textbf{Spatio-Chrom}. & & \textbf{Spatio-temp.}&  \\
                 &          Linear Fract.         & & Linear Fract.       &  \\
                 &          (in \%)         & & (in \%)       &  \\ \hline\hline
CNNs             & Sigmoid &                ReLU  &  Sigmoid &   ReLU    \\ \hline
2-Layers         & $90.6$  &  $92.9$  & $92.2$  & $92.1$    \\
4-Layers         & $94.1$  &  $94.8$  & $92.1$  & $91.7$ \\
6-Layers         & $94.3$  & $94.6$   & $92.7$  & $91.6$  \\
8-Layers         & $94.4$  & $94.8$   & $92.4$  & $92.6$  \\ \hline \hline
SOTA             & $93.9$     &   &    &    \\ \hline
\end{tabular}
\label{table_Linearfract}
\caption{Linear fraction of the networks for the spatio-chromatic problem (left) and the spatio-temporal problem (right). Results are computed for the validation set.}
\end{table}

\subsection{3.3 Stimulation with gratings}

In the experiments below we stimulate the considered architectures with purely achromatic, and isoluminant red-green and blue-yellow gratings according to the cardinal directions of a psychophysically meaningful color opponent space~\cite{Jameson59}.
See Fig.~\ref{gratings} for a representative subset of the gratings used to feed the networks for the estimation of the spatio-chromatic CSFs.
The full set included gratings of 36 spatial frequencies linearly spaced in the range [0.5, 20.5]~cpd and 9 contrasts linearly spaced in the range [0.07,0.6]. The average color was the white of the system with 30 $cd/m^2$.

\begin{figure*}[t]
\begin{center}
\includegraphics[width=0.8\linewidth]{figure_5.png}
   \caption{Representative spatio-chromatic stimuli to feed the networks.}
   \label{gratings}
\end{center}
\end{figure*}

The stimuli for the computation of the spatio-temporal CSFs were moving sinusoids with 16 spatial frequencies in
the range [0,15]~cpd, 10 temporal frequencies in the range [0,10] Hz, and 9 contrasts in the range [0.1,1].
The average luminance was set to 45~$cd/m^2$.

\subsection{3.4 Spatio-chromatic CSFs and contrast responses}

Figure~\ref{nonlinearCSFs} shows the attenuation factors suffered by the different gratings (achromatic and chromatic) depending on their frequency and contrast.
Results are shown for all the nonlinear architectures with either sigmoid or ReLU activation functions.

While some of the frequency responses (those for shallower networks) share features with human CSFs, the result strongly depends on the architecture.

The result of the 2-layer networks is particularly interesting: the opponent chromatic channels display
a narrow low-pass behavior, while the achromatic channel has substantially wider bandwidth and attenuates a bit the low frequencies (particularly for low contrasts).
Another human-like feature is the fact that the attenuation increases with contrast.
This means that the contrast response curves reduce their slope (saturate) with contrast, as is the
case in human observers for achromatic~\cite{Legge80,Legge81} and chromatic~\cite{Uriegas97} gratings.
Note that the slope (attenuation factor) decreases with contrast in both 2-layer architectures.
In the sigmoid case the effect is strong in all channels but in the ReLU case it is also apparent in the achromatic channel.
Of course, in these two-layer examples the agreement with human behavior is not complete:
the maximum of the achromatic CSF is shifted towards high frequencies, and the decay at low frequencies is smaller than in humans.

Nevertheless, the resemblance of the behavior in the shallower architectures is remarkable in a learning scenario
where no biological constraint has been introduced.
As a result, considering only the 2-layer networks one could think that the selected goal (enhancement of retinal signals) is actually guiding the biological organization.
However, the human-like features present in the 2-layers result disappear for other architectures that have similar or better performance in the goal.

For instance, in deeper architectures with ReLU the qualitative differences between the achromatic and chromatic channels go away: all sensitivities turn low-pass and a clearly non-human peak appears at high frequencies for high contrast sinusoids.
This artifact in the achromatic channel is stronger in deeper architectures with sigmoid activation.
Moreover, for 6- and 8-layers nets with sigmoids the sensitivity of the red-green channel is so small that does not appear in the plots. This explains the chromatic shift for these networks in the colors of the images in Fig.~\ref{fig_spatio_chrom}.

More interesting is the non-human nature of the CSFs of the SOTA system despite it is the network that excels in the goal. This is a compelling example of the fact that a better performance in terms of a sensible statistical goal does not necessarily imply the emergence of human-like behavior.

\begin{figure*}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{figure_6b.png}
   \caption{Spatio-chromatic CSFs for gratings of different contrast. Different CNN autoencoders and the State-Of-The-Art (SOTA) system are considered. No normalization to the output of Eq.~\ref{csf1} has been applied.}
   \label{nonlinearCSFs}
\end{center}
\end{figure*}

The presence of different sensitivity curves for different contrasts in Fig.~\ref{nonlinearCSFs} implies
that the contrast response for a sinusoid may be nonlinear. Note for instance the points highlighted with
circles (that correspond to sinusoids with $f = 2$ cpd). Variation of the attenuation factor with the contrast for the different shades of the circles implies that the slope changes. This is more clearly illustrated in  Fig.~\ref{nonlinear_resp_color}, that explicitly reproduces representative examples of these curves for achromatic sinusoids (in black), red-green sinusoids (in red), and blue-yellow sinusoids (in blue).

\begin{figure*}[t]
\begin{center}
\includegraphics[width=0.9\linewidth]{figure_8.JPG}
   \caption{Representative examples of nonlinear responses for achromatic (top row) and chromatic (middle and bottom rows) sinusoids with $f=2$ cpd. The linear behavior at the low-contrast regime has been plotted with dashed line as useful reference to check if the response is expansive or compressive with regard to the threshold behavior.}
   \label{nonlinear_resp_color}
\end{center}
\end{figure*}

The 2-layer architectures (e.g. left column in Fig.~\ref{nonlinear_resp_color}) display a saturating behavior similarly to the compressive contrast response in humans~\cite{Legge80,Legge81,Uriegas97}.
However, deeper architectures display non-human (expansive) nonlinearities (as in the chromatic cases of the 8-layers ReLU) or roughly linear behavior (as in the 6-layer sigmoid). Note that in this case, the small gain of the red-green channel explains the chromatic distortion shown in Fig.~\ref{fig_spatio_chrom}.
Note that the contrast response of the SOTA system also departs from the linear behavior happening at the low-energy regime.
The pattern found in the nonlinearities is not correlated with the achievement of higher accuracies in the goal.

\begin{figure*}[t!]
\begin{center}
\includegraphics[width=1.0\linewidth,height=9cm]{figure_7b.png}
   \caption{Spatio-chromatic CSFs for linearized networks. The optimal linear network (top-left) and the human CSFs from (Watson02,Mullen85) at the bottom-left are also included as reference. The artificial results (from the output/input ratios) were in the same range as in Fig.~\ref{nonlinearCSFs}. However, in each case, the three CSFs were normalized by the maximum of the achromatic channel for a better comparison with the human data, also normalized in that way.}
   \label{Linearized_chrom_CSFs}
\end{center}
\end{figure*}

The last spatio-chromatic result shows the CSFs obtained from the stimulation of the linearized models with gratings. This means applying the output/input ratios, similarly to Eq.~\ref{csf1}, to the matrices obtained from the nonlinear models via Eq.~\ref{global_linear_approx}. The result corresponding to the optimal linear network (obtained from Eq.~\ref{linear_solution}) is included as a convenient reference.
Figure~\ref{Linearized_chrom_CSFs} shows these results together with the achromatic CSF of the OSA Standard Spatial Observer~\cite{Watson02}, and classic results of the RG and BY chromatic CSFs~\cite{Mullen85}.

The linear network, the 2-layer networks and the SOTA system are the cases where the achromatic CSF displays a more clear band-pass result. The reduction of sensitivity for achromatic low frequencies is not as strong as in humans but there is a rough resemblance of the results in these cases.
In the other CNN cases the achromatic channel becomes low-pass. Again, the 6- and 8-layers architectures with sigmoid activation display the artifact in red-green channel. In summary the linearized models corresponding to deeper CNNs display qualitatively less-human CSFs.

\subsection{3.5 Spatio-temporal results}

Figure~\ref{nonlinear_spatio_tempCSFs} shows the attenuation factors found for moving sinusoids of low and high contrast in the plane $(f_x,f_t)$.
Similarly to the spatio-chromatic case we find a contrast-dependent behavior (which is very strong in deeper networks), so we also provide the average attenuation factors over contrast.
The CSF result of the optimal linear network and the human CSF in~\cite{Kelly79} have also been included as reference.

The main feature of the spatio-temporal human window of visibility is its diamond shape, with smaller spatial bandwidth for higher temporal frequencies (or speeds)~\cite{Kelly79,Watson13}.
The linear solution and the 2-layer architectures have an average result with human-like diamond shape of human-like bandwidth.
Deeper architectures, particularly with ReLU activations have less-human CSFs (non-diamond shape or reduced bandwidth).

\begin{figure*}[t!]
\begin{center}
\includegraphics[width=0.83\linewidth]{fig9a.JPG}\\
\includegraphics[width=0.83\linewidth]{fig9b.JPG}\\
\includegraphics[width=0.83\linewidth]{fig9c.JPG}
   \caption{Spatio-temporal CSFs for gratings of different contrast.}
   \label{nonlinear_spatio_tempCSFs}
\end{center}
\end{figure*}

The contrast dependence of the attenuation of sinusoids may reflect the saturation of
human motion sensors tuned to specific $(f_x,f_t)$~\cite{Simoncelli98,Morgan06}.
In order to check this point, Fig.~\ref{nonlinearities_spatio_temp} shows the responses of
the networks for two spatio-temporal frequencies compared to the linear responses derived from the CSF.

In every case, the response for the lower spatial frequency is bigger, as in humans, and the nonlinear responses are mainly saturating. However, for deeper ReLU architectures (those departing from the diamond shape in the CSF) there is a strong deviation from the low-energy behavior.

\begin{figure*}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{figure_10.JPG}
   \caption{Representative examples of nonlinear responses for the different networks.
   In each architecture, the curves have been normalized to the maximum of the nonlinear response
   (i.e. the response for $f_x = 3.75$ cpd and $f_t = 3$ Hz).}
   \label{nonlinearities_spatio_temp}
\end{center}
\end{figure*}

\vspace{0.4cm}
\section{4. Discussion}

\subsection{Summary of results}
In the experiments we trained CNN autoencoders to minimize the reconstruction error
of blurred and noisy retinal signals over a set of natural scenes.
We considered the reconstruction of spatio-chromatic signals and
spatio-temporal achromatic sequences with architectures made of 2D and 3D layers with conventional point-wise activation functions.
We analyzed linearized versions of the networks and we found shift-invariant
center-surround receptive fields and extended waves as eigenfunctions.
These eigenfunctions show chromatic oscillations in cardinal directions of human-like color opponent spaces.
The corresponding eigenvalues show that the networks are focused on medium and low
spatio-temporal frequencies.
This suggests a human-like frequency sensitivity in the CNNs and confirm that it is possible to
represent their response through the ratio of the output/input amplitudes
of spatio-chromatic and spatio-temporal gratings.

The stimulation of the networks with achromatic, RG and BY isoluminant sinusoids and moving sinusoids shows that:
(i)~2D~shallow autoencoders display narrow low-pass behavior in the chromatic channels and wider band-pass behavior in the achromatic channel.
(ii)~3D~shallow autoencoders display a diamond-shaped bandwidth in the spatio-temporal Fourier domain.
(iii) Shallow autoencoders (both 2D and 3D) display saturating contrast responses for spatio-chromatic and spatio-temporal sinusoids.
These human-like features of 2-layer architectures disappear for more complicated architectures despite the fact that
these deeper architectures may have better performance in terms of the statistical goal.
Particularly interesting is the case of a deep (38 layers) State-Of-The-Art system~\cite{Zhang17,Tao18} that excels in the goal but gets really noisy and non-human CSFs.
Only after global linearization it shows human-like CSFs at the cost of a significant drop in the performance.

\subsection{Relation to previous work}
Our results revisit classical work on the statistical grounds of the CSFs~\cite{Atick92a,Atick92b} in light of the new possibilities provided by automatic differentiation.

From the technical point of view,
a number of assumptions that had to be done in the 90's, either have been confirmed with the use of large data sets, or are not necessary with the use of more flexible models.
In particular, regarding the signal, Atick et al. assumed translation invariance, independence between color and space-time, and 2nd order relations (autocorrelation with $1/|f|^2$ decay).
Moreover, regarding the model, they restricted themselves to linear solutions similar to Wiener filters.
More recent studies with colorimetrically calibrated scenes~\cite{Gutmann14} have confirmed
the correctness of the color/space independence assumption.
However, the focus on the power spectrum and the linear solutions has proven to be too limited for denoising~\cite{Gutierrez06}. Adaptive (nonlinear) models that take into account additional features of the signal are required.
Nevertheless, the nonlinear networks considered here turn out to be roughly translation invariant (see Fig.~\ref{Linearization}) as expected from their convolutional
architectures.

Other technical difference is in the formulation of the statistical goal:
Atick et al. maximized the mutual information between the clean signal and the response $I(\vect{x}_c,\vect{y})$; while here we minimized the RMSE between the clean signal and the response, $|\vect{X}_c - \vect{Y}|_2$.
These goals are exactly equivalent when the difference between clean signal and the response is Gaussian, which is not the case in general.
However, note that these goals are always related because the limit $|\vect{X}_c - \vect{Y}|_2 \rightarrow 0$ implies $I(\vect{x}_c,\vect{y}) \rightarrow \infty$.

Beyond the spatio-chromatic case, in our work we check the emergence of the CSF with spatio-temporal signals, which was mentioned but not addressed by Atick et al. Finally, the consideration of nonlinear models allows us to explore the consequences of the error-minimization goal in the saturation of the contrast responses which, of course, was not possible in the linear framework of Atick et al.

Simultaneously with our results, other group has been working independently on the emergence of CSFs in ANNs, which they will  present at the next VSS meeting~\cite{Arash21}.
Their results are restricted to the spatial CSF (no chromatic nor spatio-temporal cases) and are based on networks trained for higher-level goals such as classification. Therefore, their results are complementary to ours, focused on lower-level signal enhancement at earlier visual stages such as the LGN.
Bandwidth reduction at early stages may be a necessary preprocessing that simplifies downstream decision making, and hence it appears when imposing higher level goals.

\subsection{Goal and architecture are not independent}

More important than the technical generalizations over~\cite{Atick92a,Atick92b} is that the current freedom to explore the different linear and nonlinear architectures stresses the relevance of the architectural constraints.
The conventional interpretation of of the efficient coding hypothesis~\cite{Barlow1961} is the following: obtaining human-like results from certain statistical goal seems to suggest that the human visual system has been shaped by this goal.
However, it is important to realize that the results have been obtained via the optimization of certain model. In the case of Atick et al. it was a single model (the linear filter), but in our case here we tried a range of models (architectures).
Since the results for the different architectures is markedly different, the conclusion can not be about the \emph{goal}, but about specific combinations of \emph{goal-and-architecture}.
Our results illustrate the fact that the computational and the algorithmic levels of analysis of visual processing systems~\cite{Marr76,Marr82} are not independent.
This prevents about premature conclusions about the organization principles at the abstract computational level if sensible architectures are not adopted.

\subsection{Beyond accuracy}

Human-like CSFs are obtained for shallow autoencoders (2-layers), or even linear networks, despite deeper architectures achieve similar or better performance in the goal. Previous literature has warned about the limitations of a single accuracy/performance measure to identify human-like behavior. Achieving similar performance on a task does not guarantee that two models actually use the same strategy (or algorithm)~\cite{Firestone20}.
For instance, different strategies may become evident if performance degrades in different ways when changing
the experimental setting~\cite{WichmannHVEI17,GeirhosICLR19}.
Therefore, additional checks different from the optimization goal have to be done in order
to confirm the human-like behavior of a model.
Examples include verifying additional psychophysics not included in the goal~\cite{Martinez19},
or disaggregating the results checking the consistency between model and humans
in individual trials, not on averages over the data set~\cite{Geirhos20beyond}.

\subsection{Individual Non-Euclidean distances from the optimization of average Euclidean distance}

Another interesting consequence of our results is that a Euclidean measure (the RMSE) aggregated over the set of natural images leads to systems that measure individual differences in non-Euclidean ways.
Note that (in the systems that we trained) given two input signals $\vect{x}$ and $\vect{x}+\Delta \vect{x}$, the difference between the corresponding responses is $\Delta \vect{y} \approx M \cdot \Delta\vect{x}$.
As a network should assess the difference between the two signals from $\Delta \vect{y}$,
the \emph{perceived} difference for the system will depend on $\lambda$ (the eigenspectrum of $M$) or on
the CSF. In this way, when measuring differences between individual images, the perceptual metric of the
network will be non-Euclidean. For instance, high-frequency distortions will be less relevant for the network than medium or low-frequency distortions. Even though here we did not check the correlation between the image distortions perceived by humans and networks, the observed CSFs in the networks are consistent with the fact that RMSE is not a good representation of human distortion metrics~\cite{Wang09,Laparra10,Hepburn20}.

The emergence of a non-Euclidean distance from the minimization of the average Euclidean distance is a counter intuitive consequence of the highly nonuniform distribution of natural scenes: departures in less populated regions of the image space (e.g. in high frequency directions) have to be underrated to favour the average match to the data in highly populated regions.

\subsection{Final remarks}

In visual neuroscience, deep models are emerging as the new standard to reproduce the
activity of visual areas under natural scene stimulation.
On the one hand, conventional deep models driven by object recognition goals reproduce the response
from V1~\cite{Kriegeskorte15,GucluJN15} to IT~\cite{Cadieu14,Yamins14}.
On the other hand, deep networks are powerful enough to fit the mappings between stimuli and measured responses~\cite{Gallant04,Antolik16,Batty17}.
These two approaches (goal-driven and measurement-driven deep models)
have been thoroughly compared in V1 and were found to be superior to
linear filter-banks and simple linear-nonlinear models~\cite{Cadena19}.
However, more recently, the same team has shown that linear-nonlinear models with general Divisive Normalization
make a significant step towards the performance of state-of-the-art CNN with interpretable parameters~\cite{Cadena20}.

In our goal-driven case for visual psychophysics, the emergence of human-like spatio-chromatic and spatio-temporal CSFs for certain CNN autoencoders generalizes in different ways previous statistical explanations of the CSF based on linear models~\cite{Atick92a,Atick92b}.
However, we find a strong dependence of the CSFs on the architecture for similar performance measures.
Our results suggest that when checking a statistical goal, one should be careful about the architecture of the considered model.
Similarly to the move from conventional CNNs in~\cite{Cadena19}
to more realistic divisive normalization models in~\cite{Cadena20},
we think that future goal-driven derivations of visual psychophysics should include more realistic architectures too. Examples include
divisive normalization with interpretable interaction between parametrized features~\cite{Martinez18,Martinez19} and generalizations of Wilson-Cowan interactions~\cite{SciRep20}.
The deep learning framework is excellent to consider additional factors in the goal that shapes the organization of the system. For instance, \emph{what would be the impact of information constraints on contrast responses and CSFs?}.
Learning frameworks with rate-distortion constraints are already available~\cite{Balle17}, and we advocate for the study of their artificial psychophysics using realistic and interpretable architectures as opposed to conventional CNNs.

\vspace{0.5cm}

\bibliographystyle{jovcite}
\input{jov_CSF_arxiv.bbl}

\end{document}

redpajama_set_name: RedPajamaArXiv


text:
[![Build Status](https://travis-ci.org/irishdan/ResponsiveImageBundle.svg?branch=master)](https://travis-ci.org/irishdan/ResponsiveImageBundle)
[![Scrutinizer Code Quality](https://scrutinizer-ci.com/g/irishdan/ResponsiveImageBundle/badges/quality-score.png?b=master)](https://scrutinizer-ci.com/g/irishdan/ResponsiveImageBundle/?branch=master)

## Overview:

The ResponsiveImageBundle adds the ability to easily create styled and responsive images (scaled, cropped, greyscale) in Symfony3.
Responsive images are generated as <picture> tags or as <mg> tags with sizes and scrset attributes.

A crop and focus area widget is also provided to ensure that cropped images always include the most important part of the image.

The bundle uses [FlySystem](http://flysystem.thephpleague.com/) filesystem abstraction layer giving you control over where images are stored.
Events are used to drive the system, giving more flexibiltiy and the ability to implement queuing of intensive image processing.

## Features

- Image objects are stored via Doctrine ORM
- Handles uploading images to a "[FlySystem](http://flysystem.thephpleague.com/)" filesystem
- Images styles are defined in configuration.
- Breakpoints and pictures sets are defined in configuration
- Handles creation of styled images on the fly (as they are viewed) if using a local filesystem
- Includes a widget to define an image's crop and focus areas giving art direction to styled images

## Quick and basic setup

Out of the box, ResponsiveImage bundle should work with minimal configuration.

### Step 1: Download, enable the bundle and import its services and configuration

Download with composer
```
composer require irishdan/responsive-image-bundle
```
Enable the bundle and OneupFlysystem in the kernel
```php
// app/AppKernel.php

public function registerBundles()
{
    $bundles = array(
        // ...
        new IrishDan\ResponsiveImageBundle\ResponsiveImageBundle(),
        new Oneup\FlysystemBundle\OneupFlysystemBundle(),
    );
}
```

### Step 2: Import its services, default configuration and the local image routing

Import responsive image services, default and filesystem configuration.
```
imports:
    - { resource: "@ResponsiveImageBundle/Resources/config/services.yml" }
    - { resource: "@ResponsiveImageBundle/Resources/config/config.responsive_image_defaults.yml" }
    - { resource: "@ResponsiveImageBundle/Resources/config/config.responsive_image_filesystem.yml" }
```

Import the routing for local on the fly image generation.

```yml
responsive_image:
    resource: "@ResponsiveImageBundle/Resources/config/routing.yml"
    prefix:   /
```

### Step 3: [Install](Resources/doc/installation.md) the bundle and [generate](Resources/doc/commands.md) a [ResponsiveImage entity](Resources/doc/entities.md) and it's CRUD.

```php
php bin/console responsive_image:generate:entity
php bin/console responsive_image:generate:crud
```
With the generated image [entity](Resources/doc/entities.md) and CRUD you can now, create and [upload](Resources/doc/uploading.md) images, apply '[art direction](Resources/doc/art-direction.md)' to images.

### Step 4: Define some image styles in your [configuration](Resources/doc/configuration.md) file. (Usually config.yml)

```yml
responsive_image:
    image_styles:
        groovy_thumbnail_style:
            effect: crop
            width: 180
            height: 180
        groovy_thumbnail_style_base:
            effect: scale
            width: 240
        groovy_thumbnail_style_mobile:
            effect: scale
            height:480
        groovy_thumbnail_style_mobile:
            effect: crop
            width: 200
            height: 300
            greyscale: true

```
You can now [render](Resources/doc/rendering.md) a styled in your twig template like so:
 ```
    {{ styled_image(image, 'groovy_thumbnail_style') }}

 ```
### Step 5: Define some breakpoints and "picture sets"

```yml
breakpoints:
    base:
        media_query: 'min-width: 0px'
    mobile:
        media_query: 'min-width: 300px'
    desktop:
        media_query: 'min-width: 1100px'
    groovey_picture_set:
        fallback: 'groovy_thumbnail_style'
        sources:
            base: groovy_thumbnail_style_base
            mobile: groovy_thumbnail_style_mobile
            desktop: groovy_thumbnail_style_desktop

```
You can now render <picture> [responsive images](Resources/doc/rendering.md) or render [responsive background image css](Resources/doc/rendering.md) in twig templates

```
<head>
    {{ background_responsive_image(image, 'picture_set_name', '#header') }}
</head>
<body>
    {{ picture_image(image, 'groovey_picture_set') }}
</body>
```

### Step 6: Define some size sets

```yml
responsive_image:
    size_sets:
        blog_sizes:
            fallback: 'groovy_thumbnail_style_base'
            sizes:
                10vw:
                    breakpoint: 'mobile'
            srcsets: [ 'groovy_thumbnail_style_mobile', 'groovy_thumbnail_style_desktop' ]

```
You can now render <img> [responsive images](Resources/doc/rendering.md) width srcset and sizes attributes in twig templates.

```
    {{ sizes_image(image, 'blog_sizes') }}
```

## Documentation

- [Installation and Setup](Resources/doc/installation.md)
- [Filesystems](Resources/doc/filesystem.md)
- [Image entities](Resources/doc/entities.md)
- [Uploading](Resources/doc/uploading.md)
- [Styled image generation](Resources/doc/styled-image-generation.md)
- [Image rendering](Resources/doc/rendering.md)
- [Art Direction](Resources/doc/art-direction.md)
- [Commands](Resources/doc/commands.md)
- [Configuration](Resources/doc/configuration.md)
- [Urls](Resources/doc/urls.md)
- [Events](Resources/doc/events.md)
- [Tests](Resources/doc/test.md)

## Attribution

- [Intervention](http://image.intervention.io/) is the standalone PHP Imagine Library is used by this bundle for image transformations
- [OneupFlysystemBundle](https://github.com/1up-lab/OneupFlysystemBundle) which used [Flysystem](https://flysystem.thephpleague.com/) filesystem astraction library, is required by this bundle
- The CropFocus art direction widget javascript was created by following this [TutsPlus tutorial](http://code.tutsplus.com/tutorials/how-to-create-a-jquery-image-cropping-plugin-from-scratch-part-i--net-20994)

redpajama_set_name: RedPajamaGithub