text:
\section{Introduction}\label{sec:intro}

\begin{figure}[t]
    \centering
    \includegraphics[width=1.0\textwidth]{figs/fig1.pdf}
    \captionsetup{font=small}
    \caption{
    Overview of the method and
        latent space representation.
        We start from an original image $I_o$ that can be edited $t(\cdot)$ in various ways: its feature extraction $f(t(I_o))$ spawns the shaded region in the embedding space.
        The edited versions should be recoverable by nearest neighbor search on quantized representations.
        In the regular (non-active) case, $f(I_o)$ is quantized by the index as \includegraphics[width=0.5em]{figs/assets/c1.pdf}.
        When the image is edited, $t(I_o)$ switches cells and the closest neighbor returned by the index is the wrong one \includegraphics[width=0.5em]{figs/assets/c2.pdf}.
        In active indexing: $I_o$ is modified in an imperceptible way to generate $I^\star$ such that $f(I^\star)$ is further away from the boundary.
        When edited copies $f(t(I^\star))$ are queried, retrieval errors are significantly reduced.
    \label{fig:fig1}}
\end{figure}

The traceability of images on a media sharing platform is a challenge:
they are widely used, easily edited and disseminated both inside and outside the platform.
In this paper, we tackle the corresponding task of Image Copy Detection (ICD), \textit{i.e}.\@  finding whether an image already exists in the database; and if so, give back its identifier.
ICD methods power reverse search engines, photography service providers checking copyrights, or media platforms moderating and tracking down malicious content (\textit{e.g}.\@  Microsoft's \cite{photodna} or Apple's \cite{neuralhash}).
Image identification systems have to be robust to identify images that are edited (cropping, colorimetric change, JPEG compression \ldots) after their release~\citep{douze2021disc, wang2022benchmark}.

The common approach for content-based image retrieval reduces images to high dimensional vectors, referred to as \emph{representations}.
Early representations used for retrieval were hand-crafted features such as color histograms~\citep{swain1991color}, GIST~\citep{oliva2001modeling}, or Fisher Vectors~\citep{perronnin2010large}.
As of now, a large body of work on self-supervised learning focuses on producing discriminative representations with deep neural networks, which has inspired recent ICD systems.
In fact, \emph{all} submissions to the NeurIPS2021 Image Similarity challenge~\citep{papakipos2022results} exploit neural networks.
They are trained to provide invariance to potential image transformations, akin to data augmentation in self-supervised learning.

Scalability is another key requirement of image similarity search: searching must be fast on large-scale databases, which exhaustive vector comparisons cannot do.
In practice, ICD engines leverage approximate neighbor search algorithms, that trade search accuracy against scalability.
Approximate similarity search algorithms speed up the search by \emph{not} computing the exact distance between all representations in the dataset~\citep{johnson2019faiss, guo2020scann}.
First they lower the number of scored items by partitioning the representation space, and evaluate the distances of only a few subsets.
Second, they reduce the computational cost of similarity evaluation with quantization or binarization.
These mechanisms make indexing methods subject to the curse of dimensionality.
In particular, in high-dimensional spaces, vector representations lie close to boundaries of the partition~\citep{bohm2001searching}.
Since edited versions of an original image have noisy vector representations, they sometimes fall into different subsets or are not quantized the same way by the index.
All in all, it makes approximate similarity search very sensitive to perturbations of the edited image representations, which causes images to evade detection.

In this paper, we introduce a method that improves similarity search on large databases, provided that the platform or photo provider can modify the images before their release (see Fig.~\ref{fig:fig1}).
We put the popular saying ``attack is the best form of defense'' into practice by applying image perturbations and drawing inspiration from adversarial attacks.
Indeed, representations produced with neural networks are subject to \emph{adversarial examples}~\citep{szegedy2013intriguing}:
small perturbations of the input image can lead to very different vector representations, making it possible to
create adversarial queries that fool image retrieval systems~\citep{liu2019whos, tolias2019targeted,dolhansky2020adversarial}.
In contrast, we modify an image to make it \emph{more} indexing friendly.
With minimal changes in the image domain, the image representation is pushed towards the center of the indexing partition, rising the odds that edited versions will remain in the same subset.
This property is obtained by minimizing an indexation loss by gradient descent back to the image pixels, like for adversarial examples.
For indexing structures based on product quantization~\citep{jegou2010pq}, this strategy amounts to pushing the representation closer to its quantized codeword, in which case the indexation loss is simply measured by the reconstruction error.
Since the image quality is an important constraint here, the perturbation is shaped by perceptual filters to remain invisible to the human eye.

Our contributions are:
\begin{itemize}[leftmargin=1cm,itemsep=0cm,topsep=-0.1cm]
    \item a new approach to improve ICD and retrieval, when images can be changed before release;
    \item an adversarial image optimization scheme that adds minimal perceptual perturbations to images in order to reduce reconstruction errors, and improve vector representation for indexing;
    \item experimental evidence that the method significantly improves index performance.
\end{itemize}

\section{Related Work} \label{sec:related}

\paragraph{Image watermarking}
hides a message in a host image, such that it can be reliably decoded even if the host image is edited.
Early methods directly embed the watermark signal in the spatial or transform domain like DCT or DWT~\citep{cox2007digital}.
Recently, deep-learning based methods jointly train an encoder and a decoder to learn how to watermark images~\citep{zhu2018hidden,ahmadi2020redmark,zhang2020udh}.

Watermarking is an alternative technology for ICD.
Our method bridges indexing and watermarking, where the image is modified before publication.
Regarding retrieval performance, active indexing is more robust than watermarking.
Indeed, the embedded signal reinforces the structure naturally present in the original image, whereas watermarking has to hide a large secret keyed signal independent of the original feature.
App. \ref{sec:watermarking} provides a more thorough discussion and experimental results comparing indexing and watermarking.

\paragraph{Active fingerprint} is more related to our work.
As far as we know, this concept was invented by Voloshynovskiy \textit{et al}.\@ ~\cite{voloshynovskiy2012active}.
They consider that the image $I\in \mathbb{R}^N$ is mapped to $x \in \mathbb{R}^N$ by an invertible transform $W$ such that $WW^\top$.
The binary fingerprint is obtained by taking the sign of the projections of $x$ against a set of vectors $b_1,., b_L \in \mathbb{R}^N$ (à la LSH).
Then, they change $x$ to strengthen the amplitude of these projections so that their signs become more robust to noise.
They recover $I^\star$ with $W^\top$.
This scheme is applied to image patches in~\citep{7533094} where the performance is measured as a bit error rate after JPEG compression.
Our paper adapts this idea from fingerprinting to indexing, with modern deep learning representations and state-of-the-art indexing techniques.
The range of transformations is also much broader and includes geometric transforms.

\section{Preliminaries: Representation Learning and Indexing}

For the sake of simplicity, the exposure focuses on image representations from SSCD networks~\citep{pizzi2022sscd} and the indexing technique IVF-PQ~\citep{jegou2010pq}, since both are typically used for ICD.
Extensions to other methods can be found in Sec.~\ref{sec:generalization}.

\subsection{Deep descriptor learning}

Metric embedding learning aims to learn a mapping $f: \mathbb{R}^{c\times h\times w} \to \mathbb{R}^d$, such that measuring the similarity between images $I$ and $I'$ amounts to computing the distance $\norm{f(I) - f(I')}$.
In recent works, $f$ is typically a neural network trained with self-supervision on raw data to learn metrically meaningful representations.
Methods include contrastive learning~\citep{chen2020simclr}, self-distillation~\citep{grill2020bootstrap, caron2021dino}, or masking random patches of images~\citep{he2022masked, assran2022masked}.
In particular, SSCD~\citep{pizzi2022sscd} is a training method specialized for ICD.
It employs the contrastive self-supervised method SimCLR~\citep{chen2020simclr} and entropy regularization~\citep{sablayrolles2018catalyser} to improve the distribution of the representations.

\subsection{Indexing}
Given a dataset $\mathcal X = \{x_i\}_{i=1}^{n}\subset \mathbb{R}^d$ of $d$-dimensional vector representations extracted from $n$ images and a query vector $x_q$, we consider the indexing task that addresses the problem:
\begin{align}
    x^* := \mathop{\mathrm{argmin}}_{x \in \mathcal X}
    \;  \norm{x - x_q}.
\end{align}
This exact nearest neighbor search is not tractable over large-scale databases.
Approximate search algorithms lower the amount of scored items thanks to space partitioning and/or accelerate the computations of distances thanks to quantization and pre-computation.

\paragraph{Space partitioning and cell-probe algorithms.}
As a first approximation,
nearest neighbors are sought only within a fraction of $\mathcal{X}$:
at indexing time, $\mathcal{X}$ is partitioned into $\mathcal X = \bigcup_{i=1}^{b} \mathcal{X}_i$.
At search time, an algorithm $Q: \mathbb{R}^d \to \{1,..,b\}^{k'}$ determines a subset of ${k'}$ buckets in which to search, such that ${k'}=|Q(x_q)| \ll b$, yielding the approximation:
\begin{align}
    \mathop{\mathrm{argmin}}_{x \in \mathcal X} \; \norm{x-x_q}
    \approx
    \mathop{\mathrm{argmin}}_{x \in \bigcup_{i\in Q(x_q)} \mathcal{X}_i} \; \norm{x-x_q}.
\end{align}
A well known partition is the KD-tree~\citep{bentley1975kdtree} that divides the space along predetermined directions.
Subsequently, locality sensitive hashing (LSH)~\citep{indyk1998lsh, gionis1999lsh} and derivative~\citep{datar2004lsh,pauleve2010locality} employ various hash functions for bucket assignment, which implicitly partitions the space.

We focus on the popular clustering and Inverted Files methods~\citep{sivic2003video}, herein denoted by IVF.
They employ a codebook $\mathcal{C} = \{c_i\}_{i=1}^{k}\subset\mathbb{R}^d$ of $k$ centroids (also called ``visual words'' in a local descriptor context), for instance learned with k-means over a training set of representations.
Then, $Q$ associates $x$ to its nearest centroid $q_\mathrm{c}(x)$ such that the induced partition is the set of the $k$ Voronoï cells.
When indexing $x$, the IVF stores $x$ in the bucket associated with $c_i=q_\mathrm{c}(x)$.
When querying $x_q$, IVF searches only the ${k'}$ buckets associated to centroids $c_i$ nearest to $x_q$.

\paragraph{Efficient metric computation and product quantization.}
Another approximation comes from compressed-domain distance estimation.
Vector Quantization (VQ) maps a representation $x \in \mathbb{R}^d$ to a codeword $q_\mathrm{f}(x) \in \mathcal{C} = \{C_i\}_{i=1}^{K}$.
The function $q_\mathrm{f}$ is often referred to a \emph{quantizer} and $C_i$ as a \emph{reproduction value}.
The vector $x$ is then stored as an integer in $\{1, .., K\}$ corresponding to $q_\mathrm{f}(x)$.
The distance between $x$ and query $x_q$ is approximated by $\norm{q_\mathrm{f}(x) - x_q}$, which is an ``asymmetric'' distance computation (ADC) because the query is not compressed.
This leads to:
\begin{align}
    \mathop{\mathrm{argmin}}_{x \in \mathcal X} \;  \norm{x-x_q}
    \approx
    \mathop{\mathrm{argmin}}_{x \in \mathcal X} \;  \norm{q_\mathrm{f}(x)- x_q} .
\end{align}
Binary quantizers ({a.k.a}.\@  sketches, \cite{charikar2002similarity} lead to efficient computations but inaccurate distance estimates~\citep{weiss2008spectral}.
Product Quantization (PQ)~\citep{jegou2010pq} or derivatives \cite{ge2013optimized} offer better estimates.
In PQ, a vector $x\in \mathbb{R}^d$ is split into $m$ subvectors in $\mathbb{R}^{d/m}$: $x=(x^1, \ldots, x^m)$.
The product quantizer then quantizes the subvectors: $q_\mathrm{f}: x \mapsto (q^1(x^1), \ldots, q^m(x^m))$.
If each subquantizer $q^j$ has $K_s$ reproduction values, the resulting quantizer $q_\mathrm{f}$ has a high  $K=(K_s)^m$.
The squared distance estimate is decomposed as:
\begin{align}
    \norm{q_\mathrm{f}(x)-x_q}^2 = \sum_{j=1}^m \norm{q^j(x^j)-x_q^j}^2.
\end{align}
This is efficient since $x$ is stored by the index as $q_\mathrm{f} (x)$ which has $m\log_2 K_s$ bits, and since summands can be precomputed without requiring decompression at search time.

\section{Active Indexing}\label{section:method}

Active indexing takes as input an image $I_o$, adds the image representation to the index and outputs an activated image $I^\star$ with better traceability properties for the index.
It makes the feature representation produced by the neural network more compliant with the indexing structure.
The activated image is the one that is disseminated on the platform, therefore the alteration must not degrade the perceived quality of the image.

Images are activated by an optimization on their pixels. The general optimization problem reads:
\begin{align}
    I^\star := \mathop{\mathrm{argmin}}_{I \in \mathcal{C}(I_o)}
    \; \mathcal{L}\left(I;I_o\right),
    \label{eq:active_image}
\end{align}
where $\mathcal{L}$ is an indexation loss dependent on the indexing structure, $\mathcal{C}(I_o)$ is the set of images perceptually close to $I_o$.
Algorithm~\ref{alg:1} and Figure~\ref{fig:fig1} provide an overview of  active indexing.

\begin{wrapfigure}{R}{0.45\textwidth}
\vspace{-0.7cm}
\resizebox{1.0\linewidth}{!}{
\begin{minipage}{0.5\textwidth}
    \begin{algorithm}[H]
    \caption{Active indexing for IVF-PQ}
    \label{alg:1}
        \begin{algorithmic}
            \State \textbf{Input}: $I_o$: original image; $f$: feature extractor;
            \State Add $x_o = f(I_o)$ to Index, get $q(x_o)$;
            \State Initialize $\delta_0 = 0_{(c\times h\times w)}$;
            \For{$t = 0, ..., N-1$}
            \State $I_t \gets I_o + \alpha \,.\, H_{\mathrm{JND}}(I_o) \odot \mathrm{tanh}(\delta_t)$
            \State $x_{t}\gets f(I_{t})$
            \State $\mathcal{L} \gets  \mathcal L _{\mathrm{f}} (x_{t}, q(x_o)) + \lambda  \mathcal L _{\mathrm{i}} (\delta_t)$
            \State $\delta_{t+1} \gets \delta_t + \eta \times \mathrm{Adam}(\mathcal{L})$
            \EndFor
            \State \textbf{Output}: $I^\star=I_N$ activated image
        \end{algorithmic}
    \end{algorithm}
\end{minipage}
}
\end{wrapfigure}

\subsection{Image optimization dedicated to IVF-PQ (``activation'')}

The indexing structure IVF-PQ involves a coarse quantizer $q_\mathrm{c}$ built with k-means clustering for space partitioning, and a fine product quantizer $q_\mathrm{f}$ on the residual vectors, such that a vector $x \in \mathbb{R}^d$ is approximated by $q(x) = q_\mathrm{c}(x) + q_\mathrm{f}\left( x-q_\mathrm{c}(x) \right)$.

We solve the optimization problem~\eqref{eq:active_image} by iterative gradient descent, back-propagating through the neural network back to the image.
The method is classically used in adversarial example generation~\citep{szegedy2013intriguing, carlini2017c&w} and watermarking~\citep{vukotic2020classification, fernandez2022sslwatermarking}.

Given an original image $I_o$, the loss is an aggregation of the following objectives:
\begin{align} \label{eq:objective}
    &  \mathcal L _{\mathrm{f}} (x,q(x_o)) = \norm{x - q(x_o)}^2
    \textrm{\qquad  with } x_o = f(I_o) ,\, x = f(I) \\
    &  \mathcal L _{\mathrm{i}} (I,I_o) = \norm{I - I_o}^2.
\end{align}
$ \mathcal L _{\mathrm{i}} $ is a regularization on the image distortion.
$ \mathcal L _{\mathrm{f}} $ is the indexation loss that operates on the representation space.
$ \mathcal L _{\mathrm{f}} $ is the Euclidean distance between $x$ and the target $q(x_o)$ and its goal is to push the image feature towards $q(x_o)$.
With IVF-PQ as index, the representation of the activated image gets closer to the quantized version of the original representation, but also closer to the coarse centroid.
Finally, the losses are combined as
$\mathcal{L}(I;I_o) =  \mathcal L _{\mathrm{f}} (x,q(x_o)) + \lambda  \mathcal L _{\mathrm{i}} (I,I_o)$.

\subsection{Perceptual attenuation}
It is common to optimize a perturbation $\delta$ added to the image, rather than the image itself.
The adversarial example literature often considers perceptual constraints in the form of an $\ell_p$-norm bound applied on $\delta$ (\cite{madry2017towards} use $\norm{\delta}_\infty < \varepsilon = 8/255$).
Although a smaller $\varepsilon$ makes the perturbation less visible, this constraint is not optimal for the human visual system (HVS), \textit{e.g}.\@  perturbations are more noticeable on flat than on textured areas of the image (see App.~\ref{subsec:linf}).

We employ a handcrafted perceptual attenuation model
based on a Just Noticeable Difference (JND) map~\citep{wu2017enhanced}, that adjusts the perturbation intensity according to luminance and contrast masking.
Given an image $I$, the JND map $H_{\mathrm{JND}}(I)\in \mathbb{R}^{c\times h \times w}$ models the minimum difference perceivable by the HVS at each pixel and additionally rescales the perturbation channel-wise since the human eye is more sensible to red and green than blue color shift (see App.~\ref{sec:perceptual} for details).

The relation that links the image $I$ sent to $f$, $\delta$ being optimized and the original $I_o$, reads:
\begin{align}
    I = I_o + \alpha \,.\, H_{\mathrm{JND}}(I_o) \odot \mathrm{tanh}(\delta),
    \label{eq:scaling}
\end{align}
with $\alpha$ a global scaling parameter that controls the strength of the perturbation and $\odot$ the pointwise multiplication.
Coupled with the regularization $ \mathcal L _{\mathrm{i}} $~\eqref{eq:objective}, it enforces that the activated image is perceptually similar, \textit{i.e}.\@  $I^\star\in \mathcal{C}(I_o)$ as required in~\eqref{eq:active_image}.

\subsection{Impact on the indexing performance}
Figure~\ref{fig:fig1} illustrates that the representation of the activated image gets closer to the reproduction value $q(f(I_o))$, and farther away from the Voronoï boundary.
This is expected to make image similarity search more robust because
(1) it decreases the probability that $x=f(t(I_o))$ ``falls'' outside the bucket; and
(2) it lowers the distance between $x$ and $q(x)$,  improving the PQ distance estimate.

Besides, by design, the representation stored by the index is invariant to the activation.
Formally stated, consider two images $I$, $J$, and one activated version $J^\star$ together with their representations $x,y,y^\star$.
When querying $x=f(I)$, the distance estimate is $\norm{q(y^\star)- x} = \norm{q(y)- x}$, so the index is oblivious to the change $J\rightarrow J^\star$.
This means that the structure can index passive and activated images at the same time.
Retrieval of activated images is more accurate but the performance on passive images does not change.
This compatibility property makes it possible to select only a subset of images to activate, but also to activate already-indexed images at any time.

\section{Experimental Results}

\subsection{Experimental setup}\label{sec:experimental}

\paragraph{Dataset.}
We use DISC21~\citep{douze2021disc} a dataset dedicated to ICD.
It includes 1M reference images and 50k query images, 10k of which are true copies from reference images.
A disjoint 1M-image set with same distribution as the reference images is given for training.
Images resolutions range from 200$\times$200 to 1024$\times$1024 pixels (most of the images are around 1024$\times$768 pixels).

The queries used in our experiments are \emph{not} the queries in DISC21, since we need to control the image transformations in our experiments, and most transformations of DISC21 were done manually so they are not reproducible.
Our queries are transformations of images \emph{after active indexing}.
These transformations range from simple attacks like rotation to more realistic social network transformations which created the original DISC21 queries (see App. \ref{subsec:dataset}).

\paragraph{Metrics.}
For retrieval, our main metric is Recall $1$@$1$ ($R$@$1$\@  for simplicity), which corresponds to the proportion of positive queries where the top-1 retrieved results is the reference image.

For copy detection, we use the same metric as the NeurIPS Image Similarity Challenge~\citep{douze2021disc}.
We retrieve the $k=10$ most similar database features for every query; and we declare a pair is a match if the distance is lower than a threshold $\tau$.
To evaluate detection efficiency, we use the 10k matching queries above-mentioned together with 40k negative queries (\textit{i.e}.\@  not included in the database).
We use precision and recall, as well as the area under the precision-recall curve, which is equivalent to the micro average precision (\emph{$\mu$AP}).
While $R$@$1$\@  only measures ranking quality of the index, $\mu$AP takes into account the confidence of a match.

As for image quality metric, we use the Peak Signal-to-Noise Ratio (PSNR) which is defined as $10\log_{10} \left( 255^2 / \mathrm{MSE}(I, I')^2 \right)$, as well as SSIM~\citep{wang2004ssim} and the norm $\norm{I-I'}_\infty$.

\paragraph{Implementation details.}\label{par:details}
The evaluation procedure is: (1) we train an index on the 1M training images, (2) index the 1M reference images, (3) activate (or not) 10k images from this reference set.
(4) At search time, we use the index to get closest neighbors (and their distances) of transformed versions from a query set made of the 10k images.

Unless stated otherwise, we use a IVF4096,PQ8x8 index (IVF quantizer with 4096 centroids, and PQ with 8 subquantizers of $2^8$ centroids), and use only one probe on IVF search for shortlist selection ($k'=1$).
Compared to a realistic setting, we voluntarily use an indexing method that severely degrades learned representations to showcase and analyze the effect of the active indexing.
For feature extraction, we use an SSCD model with a ResNet50 trunk~\citep{he2016resnet}.
It takes image resized to 288$\times$288 and generates normalized representations in $\mathbb{R}^{512}$.
Optimization~\eqref{eq:active_image} is done with the Adam optimizer~\citep{kingma2014adam}, the learning rate  is set to $1$, the number of iterations to $N=10$ and the regularization to $\lambda=1$.
In~\eqref{eq:scaling}, the distortion scaling  is set to $\alpha=3$ (leading to an average PNSR around $43$~dB).
In this setup, activating 128 images takes around 6s ($\approx$ 40ms/image) with a 32GB GPU.
It can be sped-up at the cost of some accuracy (see App.~\ref{sec:speedup}).

\subsection{Active vs. Passive}\label{sec:act_vs_passive}
This section compares retrieval performance of active and passive indexing.
We evaluate $R$@1 when different transformations are applied to the 10k reference images before search.
The ``Passive'' lines of Tab.~\ref{tab:act_vs_pas_retrieval} show how the IVF-PQ degrades the recall.
This is expected, but the IVF-PQ also accelerates search 500$\times$ and the index is 256$\times$ more compact, which is necessary for large-scale applications.
Edited images are retrieved more often when they were activated for the index:
increase of up to $+60$ $R$@$1$\@  for strong brightness and contrast changes, close to results of the brute-force search.
We also notice that the performance of the active IVF-PQ$^{k'=1}$ is approximately the same as the one of the passive IVF-PQ$^{k'=16}$, meaning that the search can be made more efficient at equal performance.
For the IVF-PQ$^\dagger$ that does less approximation in the search (but is slower and takes more memory), retrieval on activated images is also improved, though to a lesser extent.

\begin{table}[t]
    \centering
    \captionsetup{font=small}
    \caption{
        Comparison of the index performance between activated and passive images.
        The search is done on a 1M image set and $R$@1 is averaged over 10k query images submitted to different transformations before search.
        \textbf{Random}: randomly apply 1 to 4 transformations.
        \textbf{Avg.}: average on the transformations presented in the table (details in App. \ref{subsec:transformations}).
        \textbf{No index}: exhaustive brute-force nearest neighbor search. \textbf{IVF-PQ}: \textsc{IVF4096,PQ8x8} index with $k'$=1 (16 for \textbf{IVF-PQ}$^{16}$).
        \textbf{IVF-PQ}$^\dagger$: \textsc{IVF512,PQ32x8} with $k'=32$.
    }
    \label{tab:act_vs_pas_retrieval}
    \vspace{-0.3cm}
    \resizebox{1.0\linewidth}{!}{
    \begingroup
        \setlength{\tabcolsep}{4pt}
        \def1.1{1.1}
        \begin{tabular}{ l |l| l| c| *{15}{p{0.04\textwidth}}}
            \multicolumn{1}{c}{} & \multicolumn{1}{c}{\rot{Search (ms)}} & \multicolumn{1}{c}{\rot{Bytes/vector}} & \multicolumn{1}{c}{\rot{Activated}} & \rot{Identity} & \rot{Contr. 0.5} & \rot{Contr. 2.0} & \rot{Bright. 0.5} & \rot{Bright. 2.0} & \rot{Hue 0.2} & \rot{Blur 2.0}  & \rot{JPEG 50}  & \rot{Rot. 25} & \rot{Rot. 90} & \rot{Crop 0.5} & \rot{Resi. 0.5} & \rot{Meme} & \rot{Random} & \rot{Avg.} \\ \midrule

            No index & 252 & 2048 & \ding{55} & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 1.00 & 0.90 & 0.99 \\ \midrule

            & & & \ding{55}  & 1.00 & 0.73 & 0.39 & 0.73 & 0.28 & 0.62 & 0.48 & 0.72 & 0.07 & 0.14 & 0.14 & 0.72 & 0.14 & 0.13 & 0.45 \\

            \rowcolor{apricot!30} \cellcolor{white!0} \multirow{-2}{*}{IVF-PQ} & \multirow{-2}{*}{0.38} \cellcolor{white!0} & \multirow{-2}{*}{8} \cellcolor{white!0}
            & \checkmark & 1.00 & 1.00 & 0.96 & 1.00 & 0.92 & 1.00 & 0.96 & 0.99 & 0.10 & 0.50 & 0.29 & 1.00 & 0.43 & 0.32 & 0.75 \\
            \midrule

            & & & \ding{55} & 1.00 & 1.00 & 0.90 & 1.00 & 0.78 & 0.99 & 0.95 & 0.99 & 0.35 & 0.57 & 0.57 & 1.00 & 0.56 & 0.39 & 0.79 \\

            \rowcolor{apricot!30} \cellcolor{white!0} \multirow{-2}{*}{IVF-PQ$^{16}$} & \multirow{-2}{*}{0.42} \cellcolor{white!0} & \multirow{-2}{*}{8} \cellcolor{white!0} &
            \checkmark & 1.00 & 1.00 & 1.00 & 1.00 & 0.98 & 1.00 & 1.00 & 1.00 & 0.43 & 0.88 & 0.75 & 1.00 & 0.84 & 0.50 & 0.88 \\
            \midrule
            & & & \ding{55} & 1.00 & 1.00 & 0.99 & 1.00 & 0.95 & 1.00 & 0.99 & 1.00 & 0.72 & 0.87 & 0.88 & 1.00 & 0.87 & 0.61 & 0.92 \\

            \rowcolor{apricot!30} \cellcolor{white!0} \multirow{-2}{*}{IVF-PQ$^\dagger$} & \multirow{-2}{*}{1.9} \cellcolor{white!0} & \multirow{-2}{*}{32} \cellcolor{white!0} &
            \checkmark & 1.00 & 1.00 & 0.99 & 1.00 & 0.98 & 1.00 & 1.00 & 1.00 & 0.75 & 0.92 & 0.91 & 1.00 & 0.92 & 0.63 & 0.94 \\
            \bottomrule
        \end{tabular}
    \endgroup
    }
    \vspace{-0.3cm}
\end{table}

As for copy detection, Figure~\ref{fig:prc} gives the precision-recall curves obtained for a sliding value of $\tau$, and corresponding $\mu$AP.
Again, we observe a significant increase ($\times 2$) in $\mu$AP with active indexing.
Note that the detection performance is much weaker than the brute-force search even in the active case because of the strong approximation made by space partitioning (more details in Sec.~\ref{sec:space_partitioning}).

Example of activated images are given in Fig.~\ref{fig:qualitative_short} (more in App. \ref{sec:more_qualitative}), while the qualitative image metrics are as follows: PSNR$=43.8\pm 2.2$~dB, SSIM$=0.98 \pm 0.01$, and $\norm{I-I'}_\infty=14.5 \pm 1.2$.
These results are computed on 10k images, the $\pm$ indicates the standard deviation.

\subsection{Image quality trade-off}

For a fixed index and neural extractor, the performance of active indexing mainly depends on the scaling $\alpha$ that controls the activated image quality.
In Fig. \ref{fig:psnr}, we repeat the previous experiment for different values of $\alpha$ and plot the $\mu$AP against the average PSNR.
As expected, lower PSNR implies better $\mu$AP. For instance, at PSNR 30~dB, the $\mu$AP is augmented threefold compared to the passive case.
Indeed, for strong perturbations the objective function of \eqref{eq:objective} can be further lowered, reducing even more the gap between representations and their quantized counterparts.

\begin{figure}[b]
   \begin{minipage}{0.4\textwidth}
        \centering
        \includegraphics[width=0.95\textwidth]{figs/exp/psnr_loss_muap.pdf}
        \captionsetup{font=small}
        \caption{PSNR trade-off. As the PSNR decreases, the {\color{orange}$\mu$AP\@  (orange)} gets better, because the {\color{blue} distance (blue)} between activated representations $x$ and $q(x)$ decreases.}
        \label{fig:psnr}
   \end{minipage}\hfill
   \begin{minipage}{0.55\textwidth}
        \centering
        \includegraphics[width=0.95\textwidth]{figs/exp/qualitative_short.jpg}
        \captionsetup{font=small}
        \caption[Caption]{Activated images. \emph{Left:} reference from DISC (\href{http://www.flickr.com/photos/11805179@N04/1816673349/}{R000643.jpg} and \href{http://www.flickr.com/photos/36449457@N00/8585081884/}{R000761.jpg}), \emph{middle:} activated image, right: pixel-wise difference.}
        \label{fig:qualitative_short}
   \end{minipage}
\end{figure}

\subsection{Generalization}\label{sec:generalization}

\paragraph{Generalization to other neural feature extractors.}

We first reproduce the experiment of Sec.~\ref{par:details} with different extractors, that cover distinct training methods and architectures.
Among them, we evaluate a ResNext101~\citep{xie2017aggregated} trained with SSCD~\citep{pizzi2022sscd}, a larger network than the ResNet50 used in our main experiments ;
the winner of the descriptor track of the NeurIPS ISC, \textsc{Lyakaap}-dt1~\citep{yokoo2021isc}, that uses an EfficientNetv2 architecture~\citep{tan2021efficientnetv2} ; networks from DINO~\citep{caron2021dino}, either based on ResNet50 or ViT~\citep{dosovitskiy2020vit}, like the ViT-S model~\citep{touvron2021training}.

Table~\ref{tab:extractors} presents the $R$@$1$\@  obtained on 10k activated images when applying different transformations before search.
The $R$@$1$\@  is better for activated images for all transformations and all neural networks. The average improvement on all transformations ranges from $+12\%$ for DINO ViT-s to $+30\%$ for SSCD ResNet50.

\begin{table}[t]
    \centering
    \captionsetup{font=small}
    \caption{
        $R$@$1$\@  for different transformations before search.
        We use our method to activate images for indexing with IVF-PQ, with different neural networks used as feature extractors.
    }
    \label{tab:extractors}
    \vspace{-0.3cm}
    \resizebox{1.0\linewidth}{!}{
    \begingroup
        \setlength{\tabcolsep}{4pt}
        \def1.1{1.1}
        \begin{tabular}{ l| l| c| *{15}{p{0.04\textwidth}}}
            \multicolumn{1}{c}{\rot{Name}} & \multicolumn{1}{c}{\rot{Arch.}} & \multicolumn{1}{l}{\rot{Activated}} & \rot{Identity} & \rot{Contr. 0.5} & \rot{Contr. 2.0} & \rot{Bright. 0.5} & \rot{Bright. 2.0} & \rot{Hue 0.2} & \rot{Blur 2.0}  & \rot{JPEG 50}  & \rot{Rot. 25} & \rot{Rot. 90} & \rot{Crop 0.5} & \rot{Resi. 0.5} & \rot{Meme} & \rot{Random} & \rot{Avg.} \\ \midrule

             &  & \ding{55} & 1.00 & 0.73 & 0.39 & 0.73 & 0.28 & 0.62 & 0.48 & 0.72 & 0.07 & 0.14 & 0.14 & 0.72 & 0.14 & 0.13 & 0.45  \\
            \rowcolor{apricot!30} \cellcolor{white!0} & \multirow{-2}{*}{ResNet50} \cellcolor{white!0}& \checkmark & 1.00 & 1.00 & 0.96 & 1.00 & 0.92 & 1.00 & 0.96 & 0.99 & 0.10 & 0.50 & 0.29 & 1.00 & 0.43 & 0.32 & 0.75 \\ \cmidrule{2-18}

             &  & \ding{55} & 1.00 & 0.88 & 0.68 & 0.88 & 0.57 & 0.84 & 0.46 & 0.79 & 0.46 & 0.63 & 0.53 & 0.80 & 0.48 & 0.28 & 0.66 \\
             \rowcolor{apricot!30} \cellcolor{white!0} \multirow{-4}{*}{SSCD} & \multirow{-2}{*}{ResNext101} \cellcolor{white!0} & \checkmark & 1.00 & 1.00 & 0.96 & 1.00 & 0.90 & 0.99 & 0.77 & 0.97 & 0.53 & 0.85 & 0.64 & 1.00 & 0.74 & 0.37 & 0.84 \\ \midrule

             &  & \ding{55} & 1.00 & 0.66 & 0.65 & 0.65 & 0.52 & 0.71 & 0.52 & 0.82 & 0.07 & 0.20 & 0.51 & 0.84 & 0.62 & 0.18 & 0.57 \\
            \rowcolor{apricot!30} \cellcolor{white!0} & \multirow{-2}{*}{ResNet50} \cellcolor{white!0} & \checkmark & 1.00 & 0.99 & 0.88 & 0.99 & 0.75 & 0.93 & 0.72 & 0.94 & 0.08 & 0.25 & 0.57 & 0.99 & 0.82 & 0.23 & 0.72 \\ \cmidrule{2-18}

             &  & \ding{55} & 1.00 & 0.89 & 0.71 & 0.86 & 0.64 & 0.75 & 0.74 & 0.90 & 0.14 & 0.18 & 0.57 & 0.88 & 0.61 & 0.25 & 0.65 \\
             \rowcolor{apricot!30} \cellcolor{white!0} \multirow{-4}{*}{DINO} & \multirow{-2}{*}{ViT-s} \cellcolor{white!0} & \checkmark & 1.00 & 0.99 & 0.94 & 0.99 & 0.92 & 0.98 & 0.89 & 0.99 & 0.15 & 0.28 & 0.63 & 0.99 & 0.77 & 0.32 & 0.77 \\ \midrule

              &  & \ding{55} & 1.00 & 0.25 & 0.08 & 0.16 & 0.01 & 0.51 & 0.54 & 0.84 & 0.18 & 0.16 & 0.23 & 0.79 & 0.16 & 0.18 & 0.36 \\
              \rowcolor{apricot!30} \cellcolor{white!0} \multirow{-2}{*}{ISC-dt1} & \multirow{-2}{*}{EffNetv2} \cellcolor{white!0} & \checkmark & 1.00 & 0.57 & 0.16 & 0.33 & 0.01 & 0.88 & 0.79 & 0.97 & 0.20 & 0.24 & 0.29 & 0.97 & 0.26 & 0.26 & 0.49 \\
            \bottomrule
        \end{tabular}
    \endgroup
    }
\end{table}

\vspace*{-0.1cm}
\paragraph{Generalization to other indexes.}
The method easily generalizes to other types of indexing structures, the only difference being in the indexation loss $ \mathcal L _{\mathrm{f}} $~\eqref{eq:objective}.
We present some of them below:
\begin{itemize}[leftmargin=0.5cm,itemsep=0cm,topsep=-0.1cm]
    \item \textbf{PQ and OPQ}.\quad
        In PQ~\citep{jegou2010pq}, a vector $x \in \mathbb{R}^d$ is approximated by $q_\mathrm{f}(x)$.
        $ \mathcal L _{\mathrm{f}} $ reads $\norm{x-q_\mathrm{f}(x_o)}$.
        In OPQ~\citep{ge2013optimized}, vectors are rotated by matrix $R$ before codeword assignment, such that $RR^\top = I$.
        $ \mathcal L _{\mathrm{f}} $ becomes $\norm{x-R^\topq_\mathrm{f}(Rx_o)}$.
    \item \textbf{IVF.} \quad
        Here, we only do space partitioning.
        Employing $ \mathcal L _{\mathrm{f}}  = \norm{x- q_\mathrm{c} (x_o)}$ (``pushing towards the cluster centroid'') decreases the odds of $x$ falling in the wrong cell (see Sec.~\ref{sec:space_partitioning}).
        In this case, an issue can be that similar representations are all pushed together to a same centroid, which makes them less discriminate.
        Empirically, we found that this does not happen because perceptual constraint in the image domain prevents features from getting too close.
    \item \textbf{LSH.} \quad
        Locality Sensitive Hashing maps $x\in \mathbb{R}^d$ to a binary hash $b(x)\in \mathbb{R}^L$.
        It is commonly done with projections against a set of vectors, which give for $j \in [1,..,L]$, $b_j(x) = \mathrm{sign} (w_j^\top x)$.
        The objective $ \mathcal L _{\mathrm{f}}  = -1/L \sum_{j} \mathrm{sign}(b(x_o))\cdot w_j^\top x$,  allows to push $x$ along the LSH directions and to improve the robustness of the hash.
\end{itemize}\vspace*{0.2cm}
Table~\ref{tab:indexes} presents the $R$@$1$\@  and $\mu$AP\@  obtained on the 50k query set.
Again, results are always better in the active scenario.
We remark that active indexing has more impact on space partitioning techniques: the improvement for IVF is higher than with PQ and the LSH binary sketches. As to be expected, the impact is smaller when the indexing method is more accurate.

\begin{table}[h]
    \centering
    \resizebox{0.6\linewidth}{!}{
    \begingroup
        \setlength{\tabcolsep}{3pt}
        \begin{tabular}{ c|c |cc|cc}
                \toprule
                 \multirow{2}{*}{Index} & \multirow{2}{*}{Search time} & \multicolumn{2}{c|}{$R$@$1$\@  avg.} & \multicolumn{2}{c}{$\mu$AP\@ } \\
                & & Passive & Activated & Passive & Activated     \\ \midrule
                IVF 1024 & 0.32 ms & 0.47 & \textbf{0.83} & 0.16 & \textbf{0.43} \\
                OPQ 8x8   & 5.71 ms & 0.92 & \textbf{0.94} & 0.48 & \textbf{0.55} \\
                PCA64, LSH & 0.99 ms & 0.72 & \textbf{0.83} & 0.25 & \textbf{0.39} \\

                \bottomrule
        \end{tabular}
    \endgroup
    }
    \captionsetup{font=small}
    \caption{\makebox{$R$@$1$\@  averaged on transformations presented in Tab.~\ref{tab:act_vs_pas_retrieval} and $\mu$AP\@  for different indexing structures}}
    \label{tab:indexes}
\end{table}
\section{Analyses}\label{sec:analyses}

We provide insights on the method for IVF-PQ, considering the effects of quantization and space partitioning.
For an image $I$ whose representation is $x=f(I)\in\mathbb{R}^d$,
$\hat{x}$ denotes the representation of a transformed version: $\hat{x} = f(t(I))\in\mathbb{R}^d$, and $x^\star$ the representation of the activated image $I^\star$.
For details on the images and the implementation used in the experimental validations, see Sec. \ref{sec:experimental}.

\begin{figure}[b!]
   \begin{minipage}{0.45\textwidth}
        \centering
        \vspace{3pt}
        \includegraphics[width=1.05\linewidth, trim={0 0.8em 0 0em},clip]{figs/exp/prc.pdf}
        \captionsetup{font=small}
        \caption{Precision-Recall curve for ICD with 50k queries and 1M reference images (more details for the experimental setup in Sec. \ref{sec:experimental}). $p_\mathrm{f}^{\mathrm{ivf}}$ is the probability of failure of the IVF (Sec. \ref{sec:space_partitioning}).
        }
        \label{fig:prc}
   \end{minipage}\hfill
      \begin{minipage}{0.51\textwidth}
        \centering
        \includegraphics[width=0.8\textwidth, trim={0 0.15cm 0 0.23cm}, clip]{figs/exp/distances.pdf}
        \captionsetup{font=small}
        \caption{
            Distance estimates histograms (sec. \ref{sec:quantization}).
            With active indexing, $\|x- q(x)\|^2$ is reduced ({\color{blue}$\leftarrow$}), inducing a shift ({\color{orange}$\leftarrow$}) in the distribution of $\|\hat{x}- q(x)\|^2$, where $t(I)$ a hue-shifted version of $I$.
            $y$ is a random query.
        }
        \label{fig:dists}
   \end{minipage}
\end{figure}

\subsection{Product quantization: impact on distance estimate}\label{sec:quantization}

We start by analyzing the distance estimate considered by the index:
\begin{equation}
    \|\hat{x}-q(x)\|^2 = \|x-q(x)\|^2 + \|\hat{x}-x\|^2
    + 2 (x-q(x))^\top (\hat{x}-x).
    \label{eq:distance}
\end{equation}
The activation aims to reduce the first term, \textit{i.e}.\@  the quantization error $\|x- q(x)\|^2$, which in turn reduces $\|\hat{x}- q(x)\|^2$.
Figure~\ref{fig:dists} shows in blue the empirical distributions of $\|x- q(x)\|^2$ (passive) and $\|x^\star- q(x)\|^2$ (activated).
As expected the latter has a lower mean, but also a stronger variance.
The variation of the following factors may explain this: \emph{i)} the strength of the perturbation (due to the HVS modeled by $H_{\mathrm{JND}}$ in~\eqref{eq:scaling}), \emph{ii)} the sensitivity of the feature extractor $\|\nabla_x f(x)\|$ (some features are easier to push than others), \emph{iii)} the shapes and sizes of the Voronoï cells of PQ.

The second term of \eqref{eq:distance} models the impact of the image transformation in the feature space.
Comparing the orange and blue distributions in Fig.~\ref{fig:dists}, we see that it has a positive mean, but the shift is bigger for activated images.
We can assume that the third term has null expectation for two reasons: \emph{i)} the noise $\hat{x}-x$ is independent of $q(x)$ and centered around 0, \emph{ii)} in the high definition regime, quantification noise $x-q(x)$ is independent of $x$ and centered on 0.
Thus, this term only increases the variance.
Since $x^\star-q(x)$ has smaller norm, this increase is smaller for activated images.

All in all, $\|\hat{x}^\star-q(x)\|^2$ has a lower mean but a stronger variance than its passive counterpart $\|\hat{x}-q(x)\|^2$.
Nevertheless, the decrease of the mean is so large that it compensates the larger variance.
The orange distribution in active indexing is further away from the green distribution for negative pairs, \textit{i.e}.\@  the distance between an indexed vector $q(x)$ and an independent query $y$.

\subsection{Space partitioning: impact on the IVF probability of failure}\label{sec:space_partitioning}

We denote by $p_\mathrm{f} := \mathbb{P}(q_\mathrm{c}(x) \neq q_\mathrm{c} (\hat{x}) )$ the probability that $\hat{x}$ is assigned to a wrong bucket by IVF assignment $q_\mathrm{c}$.
In the single-probe search ($k'=1$), the recall (probability that a pair is detected when it is a true match, for a given threshold $\tau$ on the distance) is upper-bounded by $1 - p_\mathrm{f}$:
\begin{align}
   R_\tau
   = \mathbb{P} \left(\{ q_\mathrm{c}(\hat{x}) = q_\mathrm{c}(x) \} \cap \{ \| \hat{x}-q(x)\| < \tau \} \right)
   \leq \mathbb{P} \left(\{ q_\mathrm{c}(\hat{x}) = q_\mathrm{c}(x) \}  \right)
   =1 - p_\mathrm{f}.
\end{align}
In other terms, even with a high threshold $\tau \rightarrow \infty$ (and low precision), the detection misses representations that ought to be matched, with probability $p_\mathrm{f}$. It explains the sharp drop at recall $R=0.13$ in Fig.~\ref{fig:prc}.
This is why it is crucial to decrease $p_\mathrm{f}$.
The effect of active indexing is to reduce $\|\hat{x}-q_\mathrm{c}(x)\|$
therefore reducing $p_\mathrm{f}$ and increasing the upper-bound for $R$:
the drop shifts towards $R=0.32$.

This explanation suggests that pushing $x$ towards $q_\mathrm{c}(x)$ decreases even more efficiently $p_\mathrm{f}$.
This makes the IVF more robust to transformation but this may jeopardize the PQ search because features of activated images are packed altogether.
In a way, our strategy, which pushes $x$ towards $q(x)$, dispatches the improvement over the IVF and the PQ search.

\section{Conclusion \& Discussion}

We introduce a way to improve ICD in large-scale settings, when images can be changed before release.
It leverages an optimization scheme, similar to adversarial examples, that modifies images so that (1) their representations are better suited for indexing, (2) the perturbation is invisible to the human eye.
We provide grounded analyses on the method and show that it significantly improves retrieval performance of activated images, on a number of neural extractors and indexing structures.

Activating images takes time (in the order of 10~ms/image) but one advantage is that the database may contain both active and passive images: active indexing does not spoil the performance of passive indexing and vice-versa. This is good for legacy compliance and also opens the door to flexible digital asset management strategies (actively indexing images of particular importance).

The method has several limitations.
First, it is not agnostic to the indexing structure and extractor that are used by the similarity search.
Second, an adversary could break the indexing system in several ways.
In a black-box setting (no knowledge of the indexing structure and neural network extractor), adversarial purification~\citep{shi2021online} could get rid of the perturbation that activated the image.
In a semi-white-box setting (knowledge of the feature extractor), targeted mismatch attacks against passive indexing like ~\cite{tolias2019targeted} may also work.
Adversarial training ~\citep{madry2017towards} could be a defense.
For instance, it is interesting to know if adversarial training prevents active indexing, or if the perceptual perturbation that is used in our method is still able to push features in the latent space of a robust and defended neural network.

\newpage

\subsection*{Ethics Statement}
\paragraph{Societal impact statement.}
Content tracing is a double-edged sword.
On the one hand, it allows media platforms to more accurately track malicious content (pornographic, terrorist, violent images, \textit{e.g}.\@  Apple's NeuralHash and Microsoft's PhotoDNA) and to protect copyright (\textit{e.g}.\@  Youtube's Content ID).
On the other hand it can be used as a means of societal and political censorship, to restrict free speech of specific communities.
However, we still believe that research needs to be advanced to improve global moderation in the internet.
We also believe that advantages that a better copy detection could bring are more numerous than its drawbacks.

\paragraph{Environmental impact statement.}
We roughly estimated that the total GPU-days used for running all our experiments to $200$, or $\approx 5000$ GPU-hours.
Experiments were conducted using a private infrastructure and we estimate total emissions to be in the order of a ton CO$_2$eq.
Estimations were conducted using the \href{https://mlco2.github.io/impact#compute}{MachineLearning Impact calculator} presented in \cite{lacoste2019quantifying}.
We do not consider in this approximation: memory storage, CPU-hours, production cost of GPUs/ CPUs, etc. as well as the environmental cost of training the neural networks used as feature extractors.
Although the cost of the experiments and the method is high, it could possibly allow a reduction of the computations needed in large data-centers thanks to improved performance of indexing structures.

\subsection*{Reproducibility Statement}

\emph{The implementation will be made available.}
Models used for feature extraction (\href{https://github.com/facebookresearch/sscd-copy-detection/}{SSCD}, \href{https://github.com/facebookresearch/dino}{DINO}, \href{https://github.com/lyakaap/ISC21-Descriptor-Track-1st}{ISC-dt1}) can be downloaded in their respective repositories.
It builds upon the open-source Pytorch~\citep{paszke2019pytorch} and FAISS~\citep{johnson2019faiss} libraries.

The main dataset used in the experiments (DISC21) can be freely downloaded on its webpage \href{https://ai.facebook.com/datasets/disc21-dataset/}{https://ai.facebook.com/datasets/disc21-dataset/}.
Dataset processing is described in App. \ref{subsec:dataset}.

\section{Details on the Perceptual Attenuation Model}\label{sec:perceptual}

\subsection{Just Noticeable Difference map}

\begin{figure}[b]
    \centering
    \includegraphics[width= 0.23\textwidth]{figs/qualitative/ref.jpg}
    \hspace{0.1\textwidth}
    \captionsetup{font=small}
    \includegraphics[width= 0.23\textwidth]{figs/qualitative/heatmap.png}
    \caption[Caption]{A reference image $I$ from DISC21 (\href{http://www.flickr.com/photos/61368956@N00/5060849004/}{R002815.jpg}), and the associated perceptual heatmap $H_{\mathrm{JND}}(I)$.}
    \label{fig:heatmap}
    \vspace*{-0.5cm}
\end{figure}

The maximum change that the human visual system (HVS) cannot perceive is sometimes referred to as the just noticeable difference (JND)~\cite{krueger1989reconciling}.
It is used in many applications, such as image/video watemarking, compression, quality assessment (JND is also used in audio).

JND models in pixel domain directly calculate the JND at each pixel location (\textit{i.e}.\@  how much pixel difference is perceivable by the HVS). The JND map that we use is based on the work of \cite{chou1995perceptually}.
We use this model for its simplicity, its efficiency and its good qualitative results.
More complex HVS models could also be used if even higher imperceptibility is needed (\cite{watson1993dct, yang2005just, zhang2008just, jiang2022jnd} to cite a few).
The JND map takes into account two characteristics of the HVS, namely the luminance adaptation (LA) and the contrast masking (CM) phenomena. We follow the same notations as \cite{wu2017enhanced}.

The CM map $\mathcal{M}_C$ is a function of the image gradient magnitude $\mathcal{C}_l$ (the Sobel filter of the image):
\begin{equation}
    \mathcal{M}_C(x) = 0.115 \times
    \frac{\alpha \cdot \mathcal{C}_l(x)^{2.4}}
    { \mathcal{C}_l(x)^{2} + \beta^2}
    \textrm{\quad , with \,}
    \mathcal{C}_l = \sqrt{ \nabla_x I(x)^2 + \nabla_y I(x)^2},
\end{equation}
where $x$ is the pixel location, $I(x)$ the image intensity, $\alpha = 16$, and $\beta = 26$.
It is an increasing function of $\mathcal{C}_l$, meaning that the stronger the gradient is at $x$, the more the image is masking a local perturbation, and the higher the noticeable pixel difference is.

LA takes into account the fact that the HVS presents different sensitivity to background luminance (\textit{e.g}.\@  it is less sensible in dark backgrounds).
It is modeled as:
\begin{align}
    \mathcal{L}_A (x) =
    \begin{cases}
        \displaystyle 17 \times \left( 1-\sqrt{\frac{B(x)}{127}} \right) & \textrm{\quad if\,} B(x)<127 \\
        \displaystyle \frac{3 \times \left( B(x) - 127 \right)}{128} +3 & \textrm{\quad if\,}B(x)\geq 127,
    \end{cases}
\end{align}
where $B(x)$ is the background luminance, which is calculated as the mean luminance value of a local patch centered on $x$.

Finally, both effects are combined with a nonlinear additivity model:
\begin{equation}
    H_{\mathrm{JND}} = \mathcal{L}_A + \mathcal{M}_C - C \cdot \min \{ \mathcal{L}_A, \mathcal{M}_C \},
\end{equation}
where $C$ is set to $0.3$ and determines the overlapping effect.
For color images, the final RGB heatmap is $H_{\mathrm{JND}} = [\alpha_R H, \alpha_G H, \alpha_B H]$, where $(\alpha_{R}, \alpha_{G}, \alpha_{B})$ are inversely proportional to the mixing coefficients for the luminance: $(\alpha_{R}, \alpha_{G}, \alpha_{B}) = 0.072 / (0.299, 0.587, 0.114)$.

\subsection{Comparison with $\ell_\infty$ Constraint Embedding}\label{subsec:linf}
Figure~\ref{fig:linf_vs_perc} shows the same image activated using either the $\ell_\infty$ constraint (commonly used in the adversarial attack literature) or our perceptual constraint based on the JND model explained above.
Even with very small $\varepsilon$ ($4$ over 255 in the example bellow), the perturbation is visible  especially in the flat regions of the images, such as the sea or sky.

\cite{laidlaw2021perceptual} also show that the $\ell_\infty$ is not a good perceptual constraint.
They use the LPIPS loss~\citep{zhang2018unreasonable} as a surrogate for the HVS to develop more imperceptible adversarial attacks.
Although a similar approach could be used here, we found that at this small level of image distortion the LPIPS did not capture CM and LA as well as the handcrafted perceptual models present in the compression and watermarking literature.

\begin{figure}[h]
    \centering
    \begin{subfigure}[b]{0.49\textwidth}
        \centering
        \includegraphics[width= 0.48\textwidth]{figs/qualitative/linf4_psnr36,4.png}
        \includegraphics[width= 0.48\textwidth]{figs/qualitative/diff_linf.png}
        \captionsetup{font=small}
        \caption{$\ell_\infty=4$, $\mathrm{PSNR}=36.4$~dB, $\mathrm{SSIM}=0.91$}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.49\textwidth}
        \centering
        \includegraphics[width= 0.48\textwidth]{figs/qualitative/linf24_psnr34,4.png}
        \includegraphics[width= 0.48\textwidth]{figs/qualitative/diff_perc.png}
        \captionsetup{font=small}
        \caption{$\ell_\infty=23$, $\mathrm{PSNR}=34.4$~dB, $\mathrm{SSIM}=0.94$}
    \end{subfigure}
    \captionsetup{font=small}
    \caption{
        Activated images, either with (a) the $\ell_\infty \leq 4$ constraint or with (b) our perceptual model (best viewed on screen).
        We give the corresponding measures between the original and the protected image, as well as the pixel-wise difference.
        The perturbation on the right is much less perceptible thanks to the perceptual model, even though its $\ell_\infty$ distance with the original image is much higher.
    }
    \label{fig:linf_vs_perc}
\end{figure}

\section{More Experiments Details}

\subsection{Dataset}\label{subsec:dataset}
The dataset DISC 2021 was designed for the Image Similarity Challenge~\citep{douze2021disc} and can be downloaded in the dataset webpage: \href{https://ai.facebook.com/datasets/disc21-dataset/}{https://ai.facebook.com/datasets/disc21-dataset/}.

We want to test performance on edited versions of activated images but in DISC query set transformations are already applied to images. Therefore the query set cannot be used as it is.

We create a first test set ``Ref10k'' by selecting the 10k images from the reference set that were originally used to generate the queries (the ``dev queries'' from the downloadable version).
We also re-create a query set ``Query50k''.
To be as close as possible, we use the same images that were used for generating queries in DISC.
Edited images are generated using the AugLy library~\citep{papakipos2022augly}, following the guidelines given in the ``Automatic Transformations" section of the DISC paper.
Therefore, the main difference between the query set used in our experiments and the original one is that ours do not have manual augmentations.

\subsection{Transformations seen at test time}\label{subsec:transformations}

They cover both spatial transformations (crops, rotation, etc.), pixel-value transformations (contrast, hue, jpeg, etc.) and ``everyday life'' transformations with the AugLy augmentations.
All transformations are illustrated in Fig.~\ref{fig:all_transformations}.
The parameters for all transformations are the ones of the torchvision library~\citep{marcel2010torchvision}, except for the crop and resize that represent area ratios. For the Gaussian blur transformation we use alternatively $\sigma$, the scaling factor in the exponential, or the kernel size $k_b$ (in torchvision $k_b = (\sigma-0.35)/0.15$).
The ``Random'' transformation is the one used to develop the 50k query set.
A series of simple 1-4 AugLy transformations are picked at random, with skewed probability for a higher number.
Among the possible transformations, there are pixel-level, geometric ones, as well as embedding the image as a screenshot of a social network GUI.

\begin{table}[t]
    \centering
    \captionsetup{font=small}
    \caption{Illustration of all transformations evaluated in Tab.~\ref{tab:act_vs_pas_retrieval}.}
    \label{fig:all_transformations}
    \begin{tabular}{*{5}{l}}
         Identity & Contrast 0.5 & Contrast 2.0 & Brightness 0.5 & Brightness 2.0 \\
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/none.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/contrast1.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/contrast2.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/brightness1.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/brightness2.jpg}\end{minipage}
         \\ \\
          Hue 0.2 & Blur 2.0 & JPEG 50 & Rotation 25 & Rotation 90 \\
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/hue.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/blur.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/jpeg.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/rotation1.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/rotation2.jpg}\end{minipage}
         \\ \\
          Crop 0.5 & Resize 0.5 & Meme & Random \\
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/centercrop.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=\linewidth]{figs/attacks/resize.jpg}\end{minipage} &
         \begin{minipage}{.12\linewidth}\includegraphics[width=\linewidth]{figs/attacks/meme.jpg}\end{minipage} &
         \begin{minipage}{.16\linewidth}\includegraphics[width=1.7\linewidth]{figs/attacks/auto.jpg}\end{minipage} &
         \\

    \end{tabular}
\end{table}

\begin{figure}[b!]
    \centering
    \includegraphics[width=0.8\linewidth,trim={0 0.4cm 0 0.2cm}, clip]{figs/exp/transformations.pdf}
    \vspace*{-0.5cm}
    \captionsetup{font=small}
    \caption{
    Average $R$@1 over 10k images indexed with IVF-PQ.
   }
    \label{fig:tranformations}
\end{figure}

\section{More Experimental Results}

\subsection{Detailed metrics on different image transformations}

On Fig.~\ref{fig:tranformations}, we evaluate the average $R$@$1$\@  over the 10k images from the reference dataset.
The experimental setup is the same as for Tab.~\ref{tab:act_vs_pas_retrieval} but a higher number of transformation parameters are evaluated.
As expected, the higher the strength of the transformation, the lower the retrieval performance is.
The decrease in performance is significantly reduced with activated images.

\vspace*{-0.3cm}
\subsection{Additional ablations}

\paragraph{Speeding-up the optimization.}\label{sec:speedup}
In our experiments, the optimization is done using 10 iterations of gradient descent, which takes approximately 40ms/image.
If the indexation time is important (often, this is not the case and only the search time is), it can be reduced at the cost of some accuracy.

We activated 10k reference images, with the same IVF-PQ indexed presented in Sec.~\ref{sec:act_vs_passive} with only one step of gradient descent with a higher learning rate.
Activation times are computed on average.
The $R$@$1$\@  results in Tab.~\ref{tab:speedup} indicate that the speed-up in the image optimization has a small cost in retrieval accuracy.
Specifically, it reduces the $R$@$1$\@  for unedited images.
The reason is that the learning rate is too high: it can cause the representation to be pushed too far and to leave the indexing cell.
This is why a higher number number of steps and a lower learning rate are used in practice.
If activation time is a bottleneck, it can however be useful to use less optimization steps.

\begin{table}[h]
    \centering
    \captionsetup{font=small}
    \caption{$R$@$1$\@  for different transformations applied before search, with either 1 step at learning rate 10, or 10 steps at learning rate 1.}
    \label{tab:speedup}
    \vspace*{-0.5cm}
    \resizebox{0.95\linewidth}{!}{
    \begingroup
        \setlength{\tabcolsep}{4pt}
        \def1.1{1.1}
            \begin{tabular}{ l|l| *{15}{c}}
            \multicolumn{1}{c}{}           & \multicolumn{1}{l}{\rot{Activation}} & \rot{Identity} & \rot{Contr. 0.5} & \rot{Contr. 2.0} & \rot{Bright. 0.5} & \rot{Bright. 2.0} & \rot{Hue 0.2} & \rot{Blur 2.0}  & \rot{JPEG 50}  & \rot{Rot. 25} & \rot{Rot. 90} & \rot{Crop 0.5} & \rot{Resi. 0.5} & \rot{Meme} & \rot{Random} & \rot{Avg.} \\ \midrule
            Passive & - & 1.00 & 0.73 & 0.39 & 0.73 & 0.28 & 0.62 & 0.48 & 0.72 & 0.07 & 0.14 & 0.14 & 0.72 & 0.14 & 0.13 & 0.45 \\
            \rowcolor{apricot!30} lr=1 - 10 steps & 39.8 ms/img & 1.00 & 1.00 & 0.96 & 1.00 & 0.92 & 1.00 & 0.96 & 0.99 & 0.10 & 0.50 & 0.29 & 1.00 & 0.43 & 0.32 & 0.75 \\
            lr=10 - 1 step &  4.3 ms/img & 0.99 & 0.99 & 0.92 & 0.99 & 0.84 & 0.99 & 0.95 & 0.99 & 0.10 & 0.39 & 0.25 & 0.99 & 0.36 & 0.27 & 0.72  \\
            \bottomrule
    \end{tabular}
    \endgroup
    }
    \vspace*{-0.3cm}
\end{table}

\paragraph{Data augmentation at indexing time and EoT.}

Expectation over Transformations~\citep{athalye2018eot} was originally designed to create adversarial attacks robust to a set of image transformations.
We follow a similar approach to improve robustness of the marked image against a set of augmentations $\mathcal{T}$.
At each optimization step, we randomly sample $A$ augmentations $\{t_i\}_{i=1}^A$ in $\mathcal{T}$ and consider the average loss:
$ \mathcal L _{\mathrm{f}}  = \sum_{i=1}^{A} \mathcal{L}(I,t_i;I_o) /A $.
In our experiments, $\mathcal{T}$ encompasses rotations, Gaussian blurs, color jitters and a differentiable approximation of the JPEG compression~\cite{shin2017jpeg}.
$A$ is set to $8$ and we always take the un-augmented image in the chosen set of augmentations.

We activated 10k reference images, with the same IVF-PQ as Sec.~\ref{sec:act_vs_passive} with or without using EoT.
Table \ref{tab:eot} shows the average $R$@$1$\@  performance over the images submitted to different transformations before search.
EoT brings a small improvement, specifically on transformations where base performance is low (\textit{e.g}.\@  rotation or crops here).
However, it comes at a higher computational cost since each gradient descent iteration needs $A$ passes through the network, and since fewer images can be jointly activated due to GPU memory limitations (we need to store and back-propagate through $A$ transformations for every image).
If the time needed to index or activate an image is not a bottleneck, using EoT can therefore be useful.
Otherwise, it is not worth the computational cost.

\begin{table}[h]
    \centering
    \captionsetup{font=small}
    \caption{$R$@$1$\@  for different transformations applied before search, with or without EoT when activating the images.}
    \label{tab:eot}
    \vspace*{-0.5cm}
    \resizebox{0.95\linewidth}{!}{
    \begingroup
        \setlength{\tabcolsep}{4pt}
        \def1.1{1.1}
            \begin{tabular}{ l |l| *{15}{c}}
            \multicolumn{1}{c}{}          & \multicolumn{1}{l}{\rot{Activation}}  & \rot{Identity} & \rot{Contr. 0.5} & \rot{Contr. 2.0} & \rot{Bright. 0.5} & \rot{Bright. 2.0} & \rot{Hue 0.2} & \rot{Blur 2.0}  & \rot{JPEG 50}  & \rot{Rot. 25} & \rot{Rot. 90} & \rot{Crop 0.5} & \rot{Resi. 0.5} & \rot{Meme} & \rot{Random} & \rot{Avg.} \\ \midrule
                Without EOT & 40 ms & 1.00 & 1.00 & 0.96 & 1.00 & 0.92 & 1.00 & 0.96 & 0.99 & 0.10 & 0.50 & 0.29 & 1.00 & 0.43 & 0.32 & 0.75 \\
               \rowcolor{apricot!30}
                With EOT & 870 ms & 1.00 & 1.00 & 0.95 & 1.00 & 0.92 & 1.00 & 0.95 & 0.99 & 0.14 & 0.64 & 0.33 & 1.00 & 0.45 & 0.33 & 0.76 \\
            \bottomrule
    \end{tabular}
    \endgroup
    }
    \vspace*{-0.8cm}
\end{table}

\pagebreak

\section{Active Indexing vs. Watermarking}\label{sec:watermarking}

\paragraph{Discussion.}
Watermarking and active indexing both modify images for tracing and authentication, however there are significant differences between them.
Watermarking embeds arbitrary information into the image. The information can be a message, a copyright, a user ID, etc.
In contrast, active indexing modifies it to improve the efficiency of the search engine.
Watermarking also focuses on the control over the False Positive Rate of copyright detection, \textit{i.e}.\@  a bound on the probability that a random image has the same message as the watermarked one (up to a certain distance).

Although watermarking considers different settings than indexing methods, it could also be leveraged to facilitate the re-identification of near-duplicate images. In this supplemental section, we consider it to address a use-case similar to the one we address in this paper with our active indexing approach.
In this scenario, the watermark encoder embeds binary identifiers into database images.
The decoded identifier is then directly mapped to the image (as the index of a list of images).

\paragraph{Experimental setup.}
In the rest of the section, we compare active indexing against recent watermarking techniques based on deep learning.
\begin{itemize}[leftmargin=0.5cm,itemsep=0cm,topsep=-0.1cm]
    \item For indexing, we use the same setting as in Sec.~\ref{sec:experimental} (IVF-PQ index with 1M reference images).
    When searching for an image, we look up the closest neighbor with the help of the index.
    \item For watermarking, we encode $20$-bit messages into images, which allows to represent $2^{10}\approx 10^6$ images (the number of reference images).
    When searching for an image, we use the watermark decoder to get back an identifier and the corresponding image in the database.
\end{itemize}
Like before, we use $R$@$1$\@  as evaluation metric.
For indexing, it corresponds to the accuracy of the top-1 search result.
For watermarking, the $R$@$1$\@  also corresponds to the word accuracy of the decoding, that is the proportion of images where the message is perfectly decoded.
Indeed, with $20$-bit encoding almost all messages have an associated image in the reference set, so an error on a single bit causes a mis-identification (there is no error correction\footnote{In order to provide error correction capabilities, one needs longer messages. This makes it more difficult to insert bits: in our experiments, with 64 bits we observe a drastic increase of the watermarking bit error rate. }).

We use two state-of-the-art watermarking methods based on deep learning:
SSL Watermarking~\citep{fernandez2022sslwatermarking}, which also uses an adversarial-like optimization to embed messages, and HiDDeN~\citep{zhu2018hidden}, which encodes and decodes messages thanks to Conv-BN-ReLU networks.
The only difference with the original methods is that their perturbation $\delta$ is modulated by the handcrafted perceptual attenuation model presented in App.~\ref{sec:perceptual}.
This approximately gives the same image quality, thereby allowing for a direct comparison between active indexing and watermarking.

\paragraph{Results.}
Tab.~\ref{tab:watermarking} compares the $R$@$1$\@  when different transformations are applied before search or decoding.
Our active indexing method is overall the best by a large margin.
For some transformations, watermarking methods are not as effective as passive indexing, yet for some others, like crops for HiDDeN, the watermarks are more robust.

\begin{table}[h]
    \centering
    \captionsetup{font=small}
    \caption{$R$@$1$\@  for different transformations applied before search, when using either watermarking or active indexing. Results are averaged on 1k images. Best result is in \textbf{bold} and second best in \textit{italic}. }
    \label{tab:watermarking}
    \resizebox{0.99\linewidth}{!}{
    \begingroup
        \setlength{\tabcolsep}{4pt}
        \def1.1{1.1}
            \begin{tabular}{ p{4.0cm}| *{14}{c}c}
            \multicolumn{1}{c}{}          & \rot{Identity} & \rot{Contr. 0.5} & \rot{Contr. 2.0} & \rot{Bright. 0.5} & \rot{Bright. 2.0} & \rot{Hue 0.2} & \rot{Blur 2.0}  & \rot{JPEG 50}  & \rot{Rot. 25} & \rot{Rot. 90} & \rot{Crop 0.5} & \rot{Resi. 0.5} & \rot{Meme} & \rot{Random} & \rot{Avg.} \\ \midrule
    Passive indexing       & \bf 1.00 & 0.73 & 0.39 & 0.73 & 0.28 & 0.62 & 0.48 & \it 0.72 & \it 0.07 & 0.14 & 0.14 & \it  0.72 & 0.14 &  0.13 & 0.45  \\
        Active indexing (ours) & \bf 1.00 & \bf 1.00 & \bf 0.96 & \bf 1.00 & \bf 0.92 & \bf 1.00 & \bf 0.96 & \bf 0.99 & \bf 0.10 & \bf 0.50 & \it 0.29 & \bf 1.00 & 0.43 & \bf 0.32 & \bf 0.75 \\ \midrule
    \parbox{4.0cm}{SSL Watermarking \citep{fernandez2022sslwatermarking}} & \bf 1.00 & \it 0.98 & \it 0.53 & \it 0.98 & \it 0.63 & \it 0.85 & 0.13 & 0.00 & 0.00 & \it 0.15 & 0.11 & 0.00 & \it 0.46 & 0.07 & 0.42 \\ \midrule
             \parbox{3.5cm}{HiDDeN\footnote{} \citep{zhu2018hidden}} & \it 0.94 & 0.87 & 0.36 & 0.85 & 0.55 & 0.00 & \it 0.81 & 0.00 & 0.00 & 0.00 & \bf 0.92 & 0.44 & \bf 0.77 & \it 0.16 & \it 0.48 \\
            \bottomrule
    \end{tabular}
    \endgroup
    }
\end{table}

\footnotetext{Our implementation.
As reported in other papers from the literature, results of the original paper are hard to reproduce.
Therefore to make it work better, our model is trained on higher resolution images (224$\times$224), with a payload of $20$-bits, instead of 30 bits embedded into 128$\times$128.
Afterwards, the same network is used on images of arbitrary resolutions, to predict the image distortion which is later rescaled as in Eq.~\eqref{eq:scaling}.
In this setting the watermark can not always be inserted (6\% failure).}

\newpage
\section{More Qualitative Results}\label{sec:more_qualitative}

Figure~\ref{fig:more_qualitative2} gives more examples of activated images from the DISC dataset, using the same parameters as in Sec.~\ref{sec:act_vs_passive}.
The perturbation is very hard to notice (if not invisible), even in flat areas of the images because the perceptual model focuses on textures.
We also see that the perturbation forms a regular pattern.
This is due to the image (bilinear) resize that happens before feature extraction.

Figure~\ref{fig:more_qualitative} gives example of an image activated at several values of perturbation strength $\alpha$ of Eq.~\eqref{eq:scaling}
(for instance, for $\alpha=20$ the image has PSNR $27$dB and for $\alpha=1$ the image has PSNR $49$dB).
The higher the $\alpha$, the more visible the perturbation induced by the activation is.
Nevertheless, even with low PSNR values ($<35$dB), it is hard to notice if an image is activated or not.

\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\textwidth]{figs/qualitative/psnr/psnr.pdf}
    \captionsetup{font=small}
    \caption{Example of one activated image at different levels of $\alpha$.}
    \label{fig:more_qualitative}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth, trim={0 0.4cm 0 0.4cm}, clip]{figs/qualitative/diffs/qual_0.jpg}
    \includegraphics[width=0.9\textwidth, trim={0 0.4cm 0 0.4cm}, clip]{figs/qualitative/diffs/qual_2.jpg}

    \includegraphics[width=0.9\textwidth, trim={0 0.4cm 0 0.4cm}, clip]{figs/qualitative/diffs/qual_3.jpg}
    \includegraphics[width=0.9\textwidth, trim={0 0.4cm 0 0.4cm}, clip]{figs/qualitative/diffs/qual_4.jpg}
    \includegraphics[width=0.9\textwidth, trim={0 0.4cm 0 0.4cm}, clip]{figs/qualitative/diffs/qual_1.jpg}
    \captionsetup{font=small}
    \caption{Example of activated images for $\alpha=3.0$. (Left) original images, (Middle) activated images, (Right) pixel-wise difference. Images are
    \href{http://www.flickr.com/photos/87738888@N03/8039927199/}{R000005.jpg},
    \href{http://www.flickr.com/photos/79653482@N00/7766743380/}{R000045.jpg},
    \href{http://www.flickr.com/photos/12420018@N03/5452964454/}{R000076.jpg},
    \href{http://www.flickr.com/photos/56594044@N06/5933951717/}{R000172.jpg} and
    \href{http://www.flickr.com/photos/31369133@N04/5570867046/}{R000396.jpg}.
    }
    \label{fig:more_qualitative2}
\end{figure}

redpajama_set_name: RedPajamaArXiv


Out of Place | The zealot of fashion.
The very obvious yet questionable purpose of a fashion blogger comes to mind. When people ask me whether I'm a blogger after looking at my Instagram, I'd feel a sense of unease to admit it. Well, not because I'm not proud of my own blog, it's the invisible standards that people would expect from a blogger. I neither blog professionally nor do I have outstanding writing stills, I simply blog for the joy of it. Lately my blog content consists of personal styles followed by a short descriptions on my mood, my thoughts, and my daily activities. I guess for me a fashion blogger literally is a blogger who blogs about fashion. So I wouldn't consider myself one because I'm more narcissistic than that. There are many terms to describe one's blog contents so I'm pretty much all over the place. I'm trying to keep up the consistency of what I want to post about on this page but I still don't like the formalities of a blogger so I'm just gonna stick with what I'm doing now. I just hope I could at least give people some styling inspirations, that'd be great.


Port of Waterford has confirmed that the contract held by DFDS with Irish Rail to operate a freight train between Belview Port and Ballina, Co Mayo ceases today.
As RTE News reports, the Port of Waterford Chief Executive Frank Ronan said they were sorry to see the end of the service.
"It was a nice addition to our portfolio and it was great to be able to provide a port service to stakeholders in the northwest region.
"As the volumes involved were relatively small and likely to be replaced, we do not expect that our throughput of containers will be negatively impacted.
"However, it is a shame to see something that can take trucks off the road and reduce carbon emissions ceasing rather than growing and prospering.
More information on the story can be found here.


text:
Now the K-pop band will check off another milestone on its long list of achievements.
After conquering the American music market — and McDonald's — BTS' members will make a cameo on the highly anticipated "Friends" reunion special premiering early Thursday on HBO Max.
The cross-cultural pairing is not as random as you might think.
Band member RM, whose real name is Kim Nam-joon, says the show played a big role in teaching him English.
He's told interviewers that his mother bought all the seasons of the series on DVD when he was a teenager in South Korea. He said he began watching the show with Korean subtitles, then switched to English subtitles and eventually removed all subtitles completely.
"Ross, Chandler, Monica — they were my English teachers actually from the States," RM told television show ET this week. "I'm so excited about this (TV special)… I feel like I actually became friends with the Friends."
"Friends" debuted in 1994, the same year RM was born, and has remained popular worldwide through reruns. BTS fans went into a frenzy after news broke about their appearance on the "Friends" reunion, but Ben Winston, who directed the special, has tried to temper their enthusiasm.
"I want to manage your expectations on this — it's a really short moment from BTS," he tweeted. "They are great of course, but they are one of the interviews, talking briefly about why they like Friends."
HBO Max, which like CNN is part of WarnerMedia, declined to offer specifics about BTS' cameo.
"To ensure a spoiler-free viewing experience for fans, details about the special guest appearances are under embargo until premiere," the streaming service said in a statement.
BTS are increasingly part of big cultural moments
BTS' seven members, all in their 20s, debuted their first music together in 2013.
For the uninitiated BTS took their name from the Korean expression Bangtan Sonyeondan, which roughly translates to "bulletproof boy scouts."
But as they grew into an international movement the band announced their acronym would also stand for "Beyond the Scene" — a nod to the empowerment and individuality themes in their music (and their growing legions of English-speaking followers).
To their fans, it may as well mean Be Total Stans. BTS' fiercely loyal followers — known as ARMY — are many and unstoppable. With headlines comparing the band to the Beatles, the BTS effect has dominated international media coverage and thrust the band into big cultural moments.
Last week BTS released its latest English-language single, "Butter." Its video already has more than 200 million views. A record 10 million of those views came within the first 13 minutes of the video's posting.
BTS also has been part of the ongoing conversation around racial justice.
The band donated $1 million to the Black Lives Matter movement last year and tweeted its support for the cause. The members said they wanted to send a message after being subjected to prejudice and discrimination while on tour.
"We stand against racial discrimination. We condemn violence. You, I and we all have the right to be respected. We will stand together. #BlackLivesMatter," the band posted to its 30 million followers on Twitter.
Their fans interpreted it as a call to action and jumped in to match their donations.
Their fans have mastered the art of trolling
In a world where social media dictates trends, the prowess of K-pop fans may be unmatched.
BTS' ARMY works across time zones to make sure the stars remain trending topics. They've hijacked QAnon and MAGA hashtags and are among the online pranksters credited with helping derail President Trump's June 2020 campaign rally in Oklahoma.
Ahead of the rally, Trump bragged he'd received about 1 million RSVPs. But the arena didn't even reach its 19,000-seat capacity, and many people credited K-pop fans for the low turnout. One video had called on BTS fans to join the trolling campaign, in which people registered online for the rally but never showed up.
Democratic Rep. Alexandria Ocasio-Cortez credited them with flooding the Trump campaign with fake ticket reservations. "Kpop allies, we see and appreciate your contributions in the fight for justice too," she said.
A year later, Trump is gone. But BTS and their fans remain busy.
Today the band launched a celeb meal — specially packaged McNuggets, fries and a Coke, along with new dipping sauces — at McDonald's. Many of their super fans have been breathlessly awaiting the event.
"If the bts meal is not available in Hawaii i will simply riot McDonald's, do u hear me i will RIOT," one tweeted.
McDonald's has implored eager fans to remain calm. "Breathe bestie, BREATHE," the fast food chain tweeted.
A day after BTS' partnership with the Golden Arches, it's on to the "Friends" reunion special. BTS' members will be there for their new TV sitcom pals — and, as always, there for you, too.
Pentagon considering telling troops to report suspected 'Havana Syndrome' incidents

redpajama_set_name: RedPajamaCommonCrawl


The Storm Prediction Center just put out a Mesoscale Discussion for parts of southeast Missouri, southern Illinois, western Kentucky, and southwest Indiana. Basically, the MD lets the public know they are monitoring a specific area for a possible watch (severe thunderstorm or tornado).
A warm front is lifting northeast across the area. The concern is that the warm front will act as the mechanism to get thunderstorms to pop. I think the majority of thunderstorms will stay confined to near the front. There could be a few sporadic thunderstorms popping south of the warm front in the "warm sector".
We do have a pretty humid air mass in place. At noon (CT), temperatuers are in the middle to upper 70°'s and dew points are in the lower to middle 60°'s. That is plenty enough "juice" to get things going.
Here is a look at the instability in the atmosphere.
This thermodynamic field is called CAPE. CAPE stands for Convective Available Potential Energy. More specifically, this is SBCAPE (Surface Based Convective Available Potential Energy). This ultimately is the amount of instability in the atmosphere.


The school fee is invoiced monthly in 11 months.
(2816 NOK x 11 months).
85% of the school expenses is covered by the Norwegian Directorate for Education. The tuition fees are calculated based on this funding.
An administration fee of 1000 NOK is due at the time you submit your admission agreement for the new students. It is non-refundable if the application is withdrawn.
To complete the admission procedures, you must also pay the school fee of the first month of the school (August 2018) in addition to the application fee. If you start the school after August, the same procedure will be applied. You must pay both an administration fee of 1000 NOK and the fee of the starting month.
All invoices will be sent by e-mail and you will transfer the school fee to the school bank account number.
You will receive the invoice for the monthly tuition fees from this email address; autoinvoice@visma.com.
You are able to choose to pay with "eFaktura" (B2C).
With eFaktura you get the invoice filled out with KID number, account number and amount directly to your online bank.
The e-invoice must be approved by you before it is charged to the account every month.
Please note that you are not able to register for "Avtalegiro" as we do not offer this.
E-faktura makes the payment much easier and is recommended. If you have not already done this, you can easily register this in your online bank when paying the first invoice.
NLIS provides a family discount for siblings enrolled on 2018-2019 academic year. The school offers a 10% tuition discount for the second sibling, and a 15% discount for a third sibling. The discount is calculated on a monthly basis and will be visible on the monthly invoice.
The oldest student will be regarded as the first child at school, therefore, the family discount will be applied to the second and third students' fee.


text:
Dr. Trent Lovette
Email: trent.lovette@lvisd.org
As the first person in his family to go to college, Dr. Lovette believes that education can change lives. He holds a Doctorate in Educational Leadership from Texas Christian University, as well as a Master's degree in Educational Administration with Principal and Superintendent certifications from the University of Texas at Arlington. He also holds a B.S. in Biology with a minor in Chemistry from Palm Beach Atlantic University in West Palm Beach, Florida.
Dr. Lovette has been a Texas educator for over 23 years, serving as a teacher, coach, principal, and district administrator. Prior to being named superintendent of schools by the Board of Trustees for La Vernia ISD in April 2018, Dr. Lovette served Chief of Governance, Policy and Program Evaluation for Crowley ISD located near Fort Worth. After joining the Crowley ISD administration in 2003, Dr. Lovette helped CISD grow from 6,500 students to their current enrollment of over 15,000 students. While with Crowley ISD, Dr. Lovette served in a wide range of leadership roles, including Deputy Superintendent and Chief of School Leadership, Executive Director of School Operations, Chief School Administration Officer, and Principal for North Crowley High School. In 2008, he was named the State of Texas ATPE Administrator of the Year and was featured as the Keynote Speaker at Leadership Week for Yale University.
Dr. Lovette is married to Amanda, a pediatrician. They have two sons; Ethan attends the University of Alabama and Joshua attends the University of Arkansas. In his free time, Dr. Lovette enjoys outdoor activities, including fishing, golfing, hiking, tennis and traveling. He is also a die-hard college football fan.


The all-new women's RX 80 W combines Lange's innovative Dual Core shell technology with new women-specific fit and features. Our new Shin Control thermo-formable liner adapts perfectly to the lower leg for increased support while alleviating common pressure points and shin sensitivity for better, longer ski days.
Dual Core construction ensures unparalleled rebound and flex control for total control of your line.
Our unique women-specific SHIN CONTROL liner offers maximum comfort while maintaining the required precision and responsiveness for optimum ski control. The anatomical asymmetric tongue distributes shin pressure as you flex into the boot, dispersing it over a larger area for enhanced comfort and more control.
DUAL 3D LINERS are pre-shaped to match the inner shell for the most precise and accurate power transmission, and around the foot for our most comfortable anatomic fit. The DUAL 3D SPORT version is 60% thermo-formable (shin, ankle, instep, toe box) with low-density foam integrated in key fit zones for enhanced comfort and support.
Is the main difference between SX 80 W and RX 80 W the last. These are for my wife. She has narrow heel but medium, slightly wide forefoot.
Hi, The RX 80 W has a last of 100 mm (97 mm in the low volume model), while the SX 80 W has 102 mm last. The shell and cuff on the RX 80 W is made out of Polyurethane, while the SX 80 W has a Polyurethane shell and a Polyether cuff. They both have different liners and buckles. To figure out which model is right for your wife, we would recommend going into an authorized Lange dealer to have a Boot-Fitter properly size your wife's foot. You can find a dealer located near you by using our store finder at the link below: http://storefinder.lange-boots.com/ Enjoy Skiing! The Lange Team!


text:
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/security"
	xmlns:beans="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/security
http://www.springframework.org/schema/security/spring-security.xsd">
	<!-- 권한 이름 설정 -->
	<beans:bean id="roleVoter"
		class="org.springframework.security.access.vote.RoleVoter">
		<beans:property name="rolePrefix" value=""></beans:property>
    </beans:bean>

	<beans:bean id="accessDecisionManager"
		class="org.springframework.security.access.vote.AffirmativeBased">
		<beans:constructor-arg name="decisionVoters"  ref="roleVoter"/>
	</beans:bean>

	<!--
		authentication-manager-ref : 인증을 관리하는 매니저가 참조하는 설정
		accessDecisionManager : 기본 ACL을 사용자가 임의 정의로 변경하는 설정을 참조하는 속성
	 -->
	<!-- authentication-manager-ref 권한 정보를 가지는 객체 -->
	<http access-decision-manager-ref="accessDecisionManager"
		authentication-manager-ref="jdbc-auth">
		<intercept-url pattern="/admin/**" access="Admin" />
		<intercept-url pattern="/user/**" access="Admin,User" />
		<intercept-url pattern="/home" access="Admin,User,Anonymous" />
		<intercept-url pattern="/tvo/list" access="Admin,User,Anonymous" />
		<intercept-url pattern="/tvo/detail" access="Admin,User" />
		<intercept-url pattern="/tvo/form" access="Admin,User" />
		<form-login login-page="/login" authentication-failure-url="/denied"
			username-parameter="username"
			password-parameter="password"
		/>
		<logout invalidate-session="true"
			logout-url="/logout"
			logout-success-url="/logout-success"
			delete-cookies="true"
		/>
		<access-denied-handler error-page="/denied" />
		<anonymous enabled="true" granted-authority="Anonymous" username="anon_User"/>
		<session-management invalid-session-url="/login" >
			<concurrency-control max-sessions="1"
				expired-url="/login"/>
		</session-management>
	</http>

	<!--  authentication-manager 랑 연동할 dataSource-->
	<!-- dataSource -->
	<beans:bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
		<beans:property name="jndiName" value="java:comp/env/jdbc/myora" />
	</beans:bean>

	<!-- users-by-username-query 유저 테이블로부터 정보를 가져온다 -->
	<!-- authorities-by-username-query Roles 테이블로부터 권한 정보를 가져온다 -->
	<authentication-manager id="jdbc-auth">
		<authentication-provider>
			<jdbc-user-service data-source-ref="dataSource"
				users-by-username-query="select username,password,enabled from employees where username=?"
				authorities-by-username-query="select username,role from roles where username=?"
			/>
		</authentication-provider>
	</authentication-manager>

</beans:beans>

redpajama_set_name: RedPajamaGithub


text:
Category: Attractions
Vines Helen's Hill is heaven for the senses. Enjoy 180-degree views of the vineyard and rolling...
Four Pillars Gin Distillery started with the idea of making a kick-arse tonic many years ago but...
DOMAINE CHANDON
Domain Chandon flies the flag as the only Australian sparkling specialist with genuine French heritage. Located in the heart of the Yarra Valley.
COOMBE YARRA VALLEY
Once the home of world famous opera singer Dame Nellie Melba, Coombe Yarra Valley is a unique...
TARRAWARRA MUSEUM OF ART
TarraWarra Museum of Art is described as a 'privately funded, public art museum'. The TWMA...
LAKE MOUNTAIN – NEW MOUNTAIN BIKING TRAILS
The mountain bike trails off the Lake Mountain road have continued to grow over the last few years, with the latest track sculpted by Dirt Art in early 2019.
Take a walk in some of the most beautiful rainforest you will see anywhere in Australia. – The Warburton Rainforest Gallery.
Rising from the rustic, weather-beaten remnants of a barn left idle for generations, Stones of the...
HANRAHAN VINEYARD
Described as Gruyere's 'hidden gem', Hanrahan Vineyard is situated amongst...
CATHEDRAL RANGE STATE PARK
Providing spectacular views of Victoria's Yarra Valley, the Cathedral Range State Park...
WANDIN CUSTOM CAR AND BIKE SHOW
It's time to polish your hood ornaments and set the GPS for Wandin East Recreation Reserve...
GLOBAL BALLOONING
Over the last year we have been all over the Yarra Valley in all manner of vehicles, but there is...
GULF STATION – YARRA VALLEY'S WORKING FARM
Gulf Station in Yarra Glen is one of the only original timber farms left in Australia.The complex,...
Yarra Valley Chocolaterie and Ice Creamery
Set on sixteen hectares, Yarra Valley Chocolaterie and Ice Creamery combines striking architecture...
Splash About at Healesville Sanctuary
School holidays may be over but the water play continues at Healesville Sanctuary until the end of...
Yarra Farm Fresh
The Best of Local Seasonal Fruits and Vegetables Step into a colourful world of delicious fresh...
Top 3 Things To Do in the Yarra Valley
What are the Top 3 Attractions in the Yarra Valley, you ask? Well we would love to tell you, but...
YARRA RANGES REGIONAL MUSEUM
A hidden gem in the Yarra Valley. Having visited museums all around the world, it gives us great...
Mont De Lancey Historic Homestead – Wandin
Explore the unique history and heritage of the Yarra Valley. If you like country gardens with...
Yarra Glen Racecourse Market
Craft Markets Australia have been running markets since 1974.After so many years you'd...