query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : I am currently developing an application that can efficiently search for images relevant to given text descriptions by integrating both visual and textual data. Although current state-of-the-art pre-trained models are effective, they are limited by slow inference speeds and substantial computational costs. These limitations are primarily due to the attention mechanisms in the Transformer architecture, making them impractical for real-world applications where low latency and computational efficiency are crucial. Consequently, I am seeking a solution that can enhance inference speeds without sacrificing model accuracy. One potential approach is to represent texts and images in a structured, easily searchable embedding format. This format should remain a vectorized embedding but could incorporate certain structures that facilitate faster search and ranking. All these processes should be conducted offline, and during online inference, only highly efficient cosine similarity matching should be performed from their embeddings. Ultimately, my goal is for this solution to outperform existing pre-trained models in both speed and accuracy across various image and text retrieval benchmarks.

positive:
Conventional approaches to image-text retrieval mainly focus on indexing
visual objects appearing in pictures but ignore the interactions between these
objects. Such objects occurrences and interactions are equivalently useful and
important in this field as they are usually mentioned in the text. Scene graph
presentation is a suitable method for the image-text matching challenge and
obtained good results due to its ability to capture the inter-relationship
information. Both images and text are represented in scene graph levels and
formulate the retrieval challenge as a scene graph matching challenge. In this
paper, we introduce the Local and Global Scene Graph Matching (LGSGM) model
that enhances the state-of-the-art method by integrating an extra graph
convolution network to capture the general information of a graph.
Specifically, for a pair of scene graphs of an image and its caption, two
separate models are used to learn the features of each graph's nodes and edges.
Then a Siamese-structure graph convolution model is employed to embed graphs
into vector forms. We finally combine the graph-level and the vector-level to
calculate the similarity of this image-text pair. The empirical experiments
show that our enhancement with the combination of levels can improve the
performance of the baseline method by increasing the recall by more than 10% on
the Flickr30k dataset.

positive:
Text-to-image retrieval is an essential task in cross-modal information
retrieval, i.e., retrieving relevant images from a large and unlabelled dataset
given textual queries. In this paper, we propose VisualSparta, a novel
(Visual-text Sparse Transformer Matching) model that shows significant
improvement in terms of both accuracy and efficiency. VisualSparta is capable
of outperforming previous state-of-the-art scalable methods in MSCOCO and
Flickr30K. We also show that it achieves substantial retrieving speed
advantages, i.e., for a 1 million image index, VisualSparta using CPU gets
~391X speedup compared to CPU vector search and ~5.4X speedup compared to
vector search with GPU acceleration. Experiments show that this speed advantage
even gets bigger for larger datasets because VisualSparta can be efficiently
implemented as an inverted index. To the best of our knowledge, VisualSparta is
the first transformer-based text-to-image retrieval model that can achieve
real-time searching for large-scale datasets, with significant accuracy
improvement compared to previous state-of-the-art methods.

positive:
At Pinterest, we utilize image embeddings throughout our search and
recommendation systems to help our users navigate through visual content by
powering experiences like browsing of related content and searching for exact
products for shopping. In this work we describe a multi-task deep metric
learning system to learn a single unified image embedding which can be used to
power our multiple visual search products. The solution we present not only
allows us to train for multiple application objectives in a single deep neural
network architecture, but takes advantage of correlated information in the
combination of all training data from each application to generate a unified
embedding that outperforms all specialized embeddings previously deployed for
each product. We discuss the challenges of handling images from different
domains such as camera photos, high quality web images, and clean product
catalog images. We also detail how to jointly train for multiple product
objectives and how to leverage both engagement data and human labeled data. In
addition, our trained embeddings can also be binarized for efficient storage
and retrieval without compromising precision and recall. Through comprehensive
evaluations on offline metrics, user studies, and online A/B experiments, we
demonstrate that our proposed unified embedding improves both relevance and
engagement of our visual search products for both browsing and searching
purposes when compared to existing specialized embeddings. Finally, the
deployment of the unified embedding at Pinterest has drastically reduced the
operation and engineering cost of maintaining multiple embeddings while
improving quality.

positive:
Inspired by recent advances in multimodal learning and machine translation,
we introduce an encoder-decoder pipeline that learns (a): a multimodal joint
embedding space with images and text and (b): a novel language model for
decoding distributed representations from our space. Our pipeline effectively
unifies joint image-text embedding models with multimodal neural language
models. We introduce the structure-content neural language model that
disentangles the structure of a sentence to its content, conditioned on
representations produced by the encoder. The encoder allows one to rank images
and sentences while the decoder can generate novel descriptions from scratch.
Using LSTM to encode sentences, we match the state-of-the-art performance on
Flickr8K and Flickr30K without using object detections. We also set new best
results when using the 19-layer Oxford convolutional network. Furthermore we
show that with linear encoders, the learned embedding space captures multimodal
regularities in terms of vector space arithmetic e.g. *image of a blue car* -
"blue" + "red" is near images of red cars. Sample captions generated for 800
images are made available for comparison.

positive:
The ability to describe images with natural language sentences is the
hallmark for image and language understanding. Such a system has wide ranging
applications such as annotating images and using natural sentences to search
for images.In this project we focus on the task of bidirectional image
retrieval: such asystem is capable of retrieving an image based on a sentence
(image search) andretrieve sentence based on an image query (image annotation).
We present asystem based on a global ranking objective function which uses a
combinationof convolutional neural networks (CNN) and multi layer perceptrons
(MLP).It takes a pair of image and sentence and processes them in different
channels,finally embedding it into a common multimodal vector space. These
embeddingsencode abstract semantic information about the two inputs and can be
comparedusing traditional information retrieval approaches. For each such pair,
the modelreturns a score which is interpretted as a similarity metric. If this
score is high,the image and sentence are likely to convey similar meaning, and
if the score is low then they are likely not to.
  The visual input is modeled via deep convolutional neural network. On
theother hand we explore three models for the textual module. The first one
isbag of words with an MLP. The second one uses n-grams (bigram, trigrams,and a
combination of trigram & skip-grams) with an MLP. The third is morespecialized
deep network specific for modeling variable length sequences (SSE).We report
comparable performance to recent work in the field, even though ouroverall
model is simpler. We also show that the training time choice of how wecan
generate our negative samples has a significant impact on performance, and can
be used to specialize the bi-directional system in one particular task.

positive:
Current state-of-the-art approaches to cross-modal retrieval process text and
visual input jointly, relying on Transformer-based architectures with
cross-attention mechanisms that attend over all words and objects in an image.
While offering unmatched retrieval performance, such models: 1) are typically
pretrained from scratch and thus less scalable, 2) suffer from huge retrieval
latency and inefficiency issues, which makes them impractical in realistic
applications. To address these crucial gaps towards both improved and efficient
cross-modal retrieval, we propose a novel fine-tuning framework that turns any
pretrained text-image multi-modal model into an efficient retrieval model. The
framework is based on a cooperative retrieve-and-rerank approach which
combines: 1) twin networks (i.e., a bi-encoder) to separately encode all items
of a corpus, enabling efficient initial retrieval, and 2) a cross-encoder
component for a more nuanced (i.e., smarter) ranking of the retrieved small set
of items. We also propose to jointly fine-tune the two components with shared
weights, yielding a more parameter-efficient model. Our experiments on a series
of standard cross-modal retrieval benchmarks in monolingual, multilingual, and
zero-shot setups, demonstrate improved accuracy and huge efficiency benefits
over the state-of-the-art cross-encoders.

positive:
In the task of near similar image search, features from Deep Neural Network
is often used to compare images and measure similarity. In the past, we only
focused visual search in image dataset without text data. However, since deep
neural network emerged, the performance of visual search becomes high enough to
apply it in many industries from 3D data to multimodal data. Compared to the
needs of multimodal search, there has not been sufficient researches.
  In this paper, we present a method of near similar search with image and text
multimodal dataset. Earlier time, similar image search, especially when
searching shopping items, treated image and text separately to search similar
items and reorder the results. This regards two tasks of image search and text
matching as two different tasks. Our method, however, explore the vast data to
compute k-nearest neighbors using both image and text.
  In our experiment of similar item search, our system using multimodal data
shows better performance than single data while it only increases minute
computing time. For the experiment, we collected more than 15 million of
accessory and six million of digital product items from online shopping
websites, in which the product item comprises item images, titles, categories,
and descriptions. Then we compare the performance of multimodal searching to
single space searching in these datasets.

positive:
Image-text matching is an interesting and fascinating task in modern AI
research. Despite the evolution of deep-learning-based image and text
processing systems, multi-modal matching remains a challenging problem. In this
work, we consider the problem of accurate image-text matching for the task of
multi-modal large-scale information retrieval. State-of-the-art results in
image-text matching are achieved by inter-playing image and text features from
the two different processing pipelines, usually using mutual attention
mechanisms. However, this invalidates any chance to extract separate visual and
textual features needed for later indexing steps in large-scale retrieval
systems. In this regard, we introduce the Transformer Encoder Reasoning Network
(TERN), an architecture built upon one of the modern relationship-aware
self-attentive architectures, the Transformer Encoder (TE). This architecture
is able to separately reason on the two different modalities and to enforce a
final common abstract concept space by sharing the weights of the deeper
transformer layers. Thanks to this design, the implemented network is able to
produce compact and very rich visual and textual features available for the
successive indexing step. Experiments are conducted on the MS-COCO dataset, and
we evaluate the results using a discounted cumulative gain metric with
relevance computed exploiting caption similarities, in order to assess possibly
non-exact but relevant search results. We demonstrate that on this metric we
are able to achieve state-of-the-art results in the image retrieval task. Our
code is freely available at https://github.com/mesnico/TERN.

positive:
Cross-model retrieval has emerged as one of the most important upgrades for
text-only search engines (SE). Recently, with powerful representation for
pairwise text-image inputs via early interaction, the accuracy of
vision-language (VL) transformers has outperformed existing methods for
text-image retrieval. However, when the same paradigm is used for inference,
the efficiency of the VL transformers is still too low to be applied in a real
cross-modal SE. Inspired by the mechanism of human learning and using
cross-modal knowledge, this paper presents a novel Vision-Language Decomposed
Transformer (VLDeformer), which greatly increases the efficiency of VL
transformers while maintaining their outstanding accuracy. By the proposed
method, the cross-model retrieval is separated into two stages: the VL
transformer learning stage, and the VL decomposition stage. The latter stage
plays the role of single modal indexing, which is to some extent like the term
indexing of a text SE. The model learns cross-modal knowledge from
early-interaction pre-training and is then decomposed into an individual
encoder. The decomposition requires only small target datasets for supervision
and achieves both $1000+$ times acceleration and less than $0.6$\% average
recall drop. VLDeformer also outperforms state-of-the-art visual-semantic
embedding methods on COCO and Flickr30k.

positive:
Joint image-text embedding is the bedrock for most Vision-and-Language (V+L)
tasks, where multimodality inputs are simultaneously processed for joint visual
and textual understanding. In this paper, we introduce UNITER, a UNiversal
Image-TExt Representation, learned through large-scale pre-training over four
image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU
Captions), which can power heterogeneous downstream V+L tasks with joint
multimodal embeddings. We design four pre-training tasks: Masked Language
Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text
Matching (ITM), and Word-Region Alignment (WRA). Different from previous work
that applies joint random masking to both modalities, we use conditional
masking on pre-training tasks (i.e., masked language/region modeling is
conditioned on full observation of image/text). In addition to ITM for global
image-text alignment, we also propose WRA via the use of Optimal Transport (OT)
to explicitly encourage fine-grained alignment between words and image regions
during pre-training. Comprehensive analysis shows that both conditional masking
and OT-based WRA contribute to better pre-training. We also conduct a thorough
ablation study to find an optimal combination of pre-training tasks. Extensive
experiments show that UNITER achieves new state of the art across six V+L tasks
(over nine datasets), including Visual Question Answering, Image-Text
Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning,
Visual Entailment, and NLVR$^2$. Code is available at
https://github.com/ChenRocks/UNITER.

positive:
Our objective is language-based search of large-scale image and video
datasets. For this task, the approach that consists of independently mapping
text and vision to a joint embedding space, a.k.a. dual encoders, is attractive
as retrieval scales and is efficient for billions of images using approximate
nearest neighbour search. An alternative approach of using vision-text
transformers with cross-attention gives considerable improvements in accuracy
over the joint embeddings, but is often inapplicable in practice for
large-scale retrieval given the cost of the cross-attention mechanisms required
for each sample at test time. This work combines the best of both worlds. We
make the following three contributions. First, we equip transformer-based
models with a new fine-grained cross-attention architecture, providing
significant improvements in retrieval accuracy whilst preserving scalability.
Second, we introduce a generic approach for combining a Fast dual encoder model
with our Slow but accurate transformer-based model via distillation and
re-ranking. Finally, we validate our approach on the Flickr30K image dataset
where we show an increase in inference speed by several orders of magnitude
while having results competitive to the state of the art. We also extend our
method to the video domain, improving the state of the art on the VATEX
dataset.

positive:
In this paper we propose to learn a multimodal image and text embedding from
Web and Social Media data, aiming to leverage the semantic knowledge learnt in
the text domain and transfer it to a visual model for semantic image retrieval.
We demonstrate that the pipeline can learn from images with associated text
without supervision and perform a thourough analysis of five different text
embeddings in three different benchmarks. We show that the embeddings learnt
with Web and Social Media data have competitive performances over supervised
methods in the text based image retrieval task, and we clearly outperform state
of the art in the MIRFlickr dataset when training in the target data. Further
we demonstrate how semantic multimodal image retrieval can be performed using
the learnt embeddings, going beyond classical instance-level retrieval
problems. Finally, we present a new dataset, InstaCities1M, composed by
Instagram images and their associated texts that can be used for fair
comparison of image-text embeddings.

positive:
Self-Supervised learning from multimodal image and text data allows deep
neural networks to learn powerful features with no need of human annotated
data. Web and Social Media platforms provide a virtually unlimited amount of
this multimodal data. In this work we propose to exploit this free available
data to learn a multimodal image and text embedding, aiming to leverage the
semantic knowledge learnt in the text domain and transfer it to a visual model
for semantic image retrieval. We demonstrate that the proposed pipeline can
learn from images with associated textwithout supervision and analyze the
semantic structure of the learnt joint image and text embedding space. We
perform a thorough analysis and performance comparison of five different state
of the art text embeddings in three different benchmarks. We show that the
embeddings learnt with Web and Social Media data have competitive performances
over supervised methods in the text based image retrieval task, and we clearly
outperform state of the art in the MIRFlickr dataset when training in the
target data. Further, we demonstrate how semantic multimodal image retrieval
can be performed using the learnt embeddings, going beyond classical
instance-level retrieval problems. Finally, we present a new dataset,
InstaCities1M, composed by Instagram images and their associated texts that can
be used for fair comparison of image-text embeddings.

positive:
We propose Pixel-BERT to align image pixels with text by deep multi-modal
transformers that jointly learn visual and language embedding in a unified
end-to-end framework. We aim to build a more accurate and thorough connection
between image pixels and language semantics directly from image and sentence
pairs instead of using region-based image features as the most recent vision
and language tasks. Our Pixel-BERT which aligns semantic connection in pixel
and text level solves the limitation of task-specific visual representation for
vision and language tasks. It also relieves the cost of bounding box
annotations and overcomes the unbalance between semantic labels in visual task
and language semantic. To provide a better representation for down-stream
tasks, we pre-train a universal end-to-end model with image and sentence pairs
from Visual Genome dataset and MS-COCO dataset. We propose to use a random
pixel sampling mechanism to enhance the robustness of visual representation and
to apply the Masked Language Model and Image-Text Matching as pre-training
tasks. Extensive experiments on downstream tasks with our pre-trained model
show that our approach makes the most state-of-the-arts in downstream tasks,
including Visual Question Answering (VQA), image-text retrieval, Natural
Language for Visual Reasoning for Real (NLVR). Particularly, we boost the
performance of a single model in VQA task by 2.17 points compared with SOTA
under fair comparison.

positive:
A major challenge in matching images and text is that they have intrinsically
different data distributions and feature representations. Most existing
approaches are based either on embedding or classification, the first one
mapping image and text instances into a common embedding space for distance
measuring, and the second one regarding image-text matching as a binary
classification problem. Neither of these approaches can, however, balance the
matching accuracy and model complexity well. We propose a novel framework that
achieves remarkable matching performance with acceptable model complexity.
Specifically, in the training stage, we propose a novel Multi-modal Tensor
Fusion Network (MTFN) to explicitly learn an accurate image-text similarity
function with rank-based tensor fusion rather than seeking a common embedding
space for each image-text instance. Then, during testing, we deploy a generic
Cross-modal Re-ranking (RR) scheme for refinement without requiring additional
training procedure. Extensive experiments on two datasets demonstrate that our
MTFN-RR consistently achieves the state-of-the-art matching performance with
much less time complexity. The implementation code is available at
https://github.com/Wangt-CN/MTFN-RR-PyTorch-Code.

positive:
Textual-visual cross-modal retrieval has been a hot research topic in both
computer vision and natural language processing communities. Learning
appropriate representations for multi-modal data is crucial for the cross-modal
retrieval performance. Unlike existing image-text retrieval approaches that
embed image-text pairs as single feature vectors in a common representational
space, we propose to incorporate generative processes into the cross-modal
feature embedding, through which we are able to learn not only the global
abstract features but also the local grounded features. Extensive experiments
show that our framework can well match images and sentences with complex
content, and achieve the state-of-the-art cross-modal retrieval results on
MSCOCO dataset.

positive:
In this work, we propose an approach to index Deep Convolutional Neural
Network Features to support efficient content-based retrieval on large image
databases. To this aim, we have converted the these features into a textual
form, to index them into an inverted index by means of Lucene. In this way, we
were able to set up a robust retrieval system that combines full-text search
with content-based image retrieval capabilities. We evaluated different
strategies of textual representation in order to optimize the index occupation
and the query response time. In order to show that our approach is able to
handle large datasets, we have developed a web-based prototype that provides an
interface for combined textual and visual searching into a dataset of about 100
million of images.

positive:
In this paper, we describe our end-to-end content-based image retrieval
system built upon Elasticsearch, a well-known and popular textual search
engine. As far as we know, this is the first time such a system has been
implemented in eCommerce, and our efforts have turned out to be highly
worthwhile. We end up with a novel and exciting visual search solution that is
extremely easy to be deployed, distributed, scaled and monitored in a
cost-friendly manner. Moreover, our platform is intrinsically flexible in
supporting multimodal searches, where visual and textual information can be
jointly leveraged in retrieval.
  The core idea is to encode image feature vectors into a collection of string
tokens in a way such that closer vectors will share more string tokens in
common. By doing that, we can utilize Elasticsearch to efficiently retrieve
similar images based on similarities within encoded sting tokens. As part of
the development, we propose a novel vector to string encoding method, which is
shown to substantially outperform the previous ones in terms of both precision
and latency.
  First-hand experiences in implementing this Elasticsearch-based platform are
extensively addressed, which should be valuable to practitioners also
interested in building visual search engine on top of Elasticsearch.

positive:
This paper considers the task of matching images and sentences by learning a
visual-textual embedding space for cross-modal retrieval. Finding such a space
is a challenging task since the features and representations of text and image
are not comparable. In this work, we introduce an end-to-end deep multimodal
convolutional-recurrent network for learning both vision and language
representations simultaneously to infer image-text similarity. The model learns
which pairs are a match (positive) and which ones are a mismatch (negative)
using a hinge-based triplet ranking. To learn about the joint representations,
we leverage our newly extracted collection of tweets from Twitter. The main
characteristic of our dataset is that the images and tweets are not
standardized the same as the benchmarks. Furthermore, there can be a higher
semantic correlation between the pictures and tweets contrary to benchmarks in
which the descriptions are well-organized. Experimental results on MS-COCO
benchmark dataset show that our model outperforms certain methods presented
previously and has competitive performance compared to the state-of-the-art.
The code and dataset have been made available publicly.

positive:
Generative adversarial networks conditioned on textual image descriptions are
capable of generating realistic-looking images. However, current methods still
struggle to generate images based on complex image captions from a
heterogeneous domain. Furthermore, quantitatively evaluating these
text-to-image models is challenging, as most evaluation metrics only judge
image quality but not the conformity between the image and its caption. To
address these challenges we introduce a new model that explicitly models
individual objects within an image and a new evaluation metric called Semantic
Object Accuracy (SOA) that specifically evaluates images given an image
caption. The SOA uses a pre-trained object detector to evaluate if a generated
image contains objects that are mentioned in the image caption, e.g. whether an
image generated from "a car driving down the street" contains a car. We perform
a user study comparing several text-to-image models and show that our SOA
metric ranks the models the same way as humans, whereas other metrics such as
the Inception Score do not. Our evaluation also shows that models which
explicitly model objects outperform models which only model global image
characteristics.

positive:
This paper investigates the problem of modeling Internet images and
associated text or tags for tasks such as image-to-image search, tag-to-image
search, and image-to-tag search (image annotation). We start with canonical
correlation analysis (CCA), a popular and successful approach for mapping
visual and textual features to the same latent space, and incorporate a third
view capturing high-level image semantics, represented either by a single
category or multiple non-mutually-exclusive concepts. We present two ways to
train the three-view embedding: supervised, with the third view coming from
ground-truth labels or search keywords; and unsupervised, with semantic themes
automatically obtained by clustering the tags. To ensure high accuracy for
retrieval tasks while keeping the learning process scalable, we combine
multiple strong visual features and use explicit nonlinear kernel mappings to
efficiently approximate kernel CCA. To perform retrieval, we use a specially
designed similarity function in the embedded space, which substantially
outperforms the Euclidean distance. The resulting system produces compelling
qualitative results and outperforms a number of two-view baselines on retrieval
tasks on three large-scale Internet image datasets.

positive:
The use of deep learning models within scientific experimental facilities
frequently requires low-latency inference, so that, for example, quality
control operations can be performed while data are being collected. Edge
computing devices can be useful in this context, as their low cost and compact
form factor permit them to be co-located with the experimental apparatus. Can
such devices, with their limited resources, can perform neural network
feed-forward computations efficiently and effectively? We explore this question
by evaluating the performance and accuracy of a scientific image restoration
model, for which both model input and output are images, on edge computing
devices. Specifically, we evaluate deployments of TomoGAN, an image-denoising
model based on generative adversarial networks developed for low-dose x-ray
imaging, on the Google Edge TPU and NVIDIA Jetson. We adapt TomoGAN for edge
execution, evaluate model inference performance, and propose methods to address
the accuracy drop caused by model quantization. We show that these edge
computing devices can deliver accuracy comparable to that of a full-fledged CPU
or GPU model, at speeds that are more than adequate for use in the intended
deployments, denoising a 1024 x 1024 image in less than a second. Our
experiments also show that the Edge TPU models can provide 3x faster inference
response than a CPU-based model and 1.5x faster than an edge GPU-based model.
This combination of high speed and low cost permits image restoration anywhere.

positive:
Text-visual (or called semantic-visual) embedding is a central problem in
vision-language research. It typically involves mapping of an image and a text
description to a common feature space through a CNN image encoder and a RNN
language encoder. In this paper, we propose a new method for learning
text-visual embedding using both image titles and click-through data from an
image search engine. We also propose a new triplet loss function by modeling
positive awareness of the embedding, and introduce a novel mini-batch-based
hard negative sampling approach for better data efficiency in the learning
process. Experimental results show that our proposed method outperforms
existing methods, and is also effective for real-world text-to-visual
retrieval.

positive:
Zero-shot learning (ZSL) models rely on learning a joint embedding space
where both textual/semantic description of object classes and visual
representation of object images can be projected to for nearest neighbour
search. Despite the success of deep neural networks that learn an end-to-end
model between text and images in other vision problems such as image
captioning, very few deep ZSL model exists and they show little advantage over
ZSL models that utilise deep feature representations but do not learn an
end-to-end embedding. In this paper we argue that the key to make deep ZSL
models succeed is to choose the right embedding space. Instead of embedding
into a semantic space or an intermediate space, we propose to use the visual
space as the embedding space. This is because that in this space, the
subsequent nearest neighbour search would suffer much less from the hubness
problem and thus become more effective. This model design also provides a
natural mechanism for multiple semantic modalities (e.g., attributes and
sentence descriptions) to be fused and optimised jointly in an end-to-end
manner. Extensive experiments on four benchmarks show that our model
significantly outperforms the existing models. Code is available at
https://github.com/lzrobots/DeepEmbeddingModel_ZSL

positive:
Multilingual (or cross-lingual) embeddings represent several languages in a
unique vector space. Using a common embedding space enables for a shared
semantic between words from different languages. In this paper, we propose to
embed images and texts into a unique distributional vector space, enabling to
search images by using text queries expressing information needs related to the
(visual) content of images, as well as using image similarity. Our framework
forces the representation of an image to be similar to the representation of
the text that describes it. Moreover, by using multilingual embeddings we
ensure that words from two different languages have close descriptors and thus
are attached to similar images. We provide experimental evidence of the
efficiency of our approach by experimenting it on two datasets: Common Objects
in COntext (COCO) [19] and Multi30K [7].

positive:
Researchers often spend weeks sifting through decades of unlabeled satellite
imagery(on NASA Worldview) in order to develop datasets on which they can start
conducting research. We developed an interactive, scalable and fast image
similarity search engine (which can take one or more images as the query image)
that automatically sifts through the unlabeled dataset reducing dataset
generation time from weeks to minutes. In this work, we describe key components
of the end to end pipeline. Our similarity search system was created to be able
to identify similar images from a potentially petabyte scale database that are
similar to an input image, and for this we had to break down each query image
into its features, which were generated by a classification layer stripped CNN
trained in a supervised manner. To store and search these features efficiently,
we had to make several scalability improvements. To improve the speed, reduce
the storage, and shrink memory requirements for embedding search, we add a
fully connected layer to our CNN make all images into a 128 length vector
before entering the classification layers. This helped us compress the size of
our image features from 2048 (for ResNet, which was initially tried as our
featurizer) to 128 for our new custom model. Additionally, we utilize existing
approximate nearest neighbor search libraries to significantly speed up
embedding search. Our system currently searches over our entire database of
images at 5 seconds per query on a single virtual machine in the cloud. In the
future, we would like to incorporate a SimCLR based featurizing model which
could be trained without any labelling by a human (since the classification
aspect of the model is irrelevant to this use case).

negative:
Deep Learning is increasingly being adopted by industry for computer vision
applications running on embedded devices. While Convolutional Neural Networks'
accuracy has achieved a mature and remarkable state, inference latency and
throughput are a major concern especially when targeting low-cost and low-power
embedded platforms. CNNs' inference latency may become a bottleneck for Deep
Learning adoption by industry, as it is a crucial specification for many
real-time processes. Furthermore, deployment of CNNs across heterogeneous
platforms presents major compatibility issues due to vendor-specific technology
and acceleration libraries. In this work, we present QS-DNN, a fully automatic
search based on Reinforcement Learning which, combined with an inference engine
optimizer, efficiently explores through the design space and empirically finds
the optimal combinations of libraries and primitives to speed up the inference
of CNNs on heterogeneous embedded devices. We show that, an optimized
combination can achieve 45x speedup in inference latency on CPU compared to a
dependency-free baseline and 2x on average on GPGPU compared to the best vendor
library. Further, we demonstrate that, the quality of results and time
"to-solution" is much better than with Random Search and achieves up to 15x
better results for a short-time search.

negative:
Recently, Multi-modal Named Entity Recognition (MNER) has attracted a lot of
attention. Most of the work utilizes image information through region-level
visual representations obtained from a pretrained object detector and relies on
an attention mechanism to model the interactions between image and text
representations. However, it is difficult to model such interactions as image
and text representations are trained separately on the data of their respective
modality and are not aligned in the same space. As text representations take
the most important role in MNER, in this paper, we propose {\bf I}mage-{\bf
t}ext {\bf A}lignments (ITA) to align image features into the textual space, so
that the attention mechanism in transformer-based pretrained textual embeddings
can be better utilized. ITA first aligns the image into regional object tags,
image-level captions and optical characters as visual contexts, concatenates
them with the input texts as a new cross-modal input, and then feeds it into a
pretrained textual embedding model. This makes it easier for the attention
module of a pretrained textual embedding model to model the interaction between
the two modalities since they are both represented in the textual space. ITA
further aligns the output distributions predicted from the cross-modal input
and textual input views so that the MNER model can be more practical in dealing
with text-only inputs and robust to noises from images. In our experiments, we
show that ITA models can achieve state-of-the-art accuracy on multi-modal Named
Entity Recognition datasets, even without image information.

negative:
Text-based person search is a sub-task in the field of image retrieval, which
aims to retrieve target person images according to a given textual description.
The significant feature gap between two modalities makes this task very
challenging. Many existing methods attempt to utilize local alignment to
address this problem in the fine-grained level. However, most relevant methods
introduce additional models or complicated training and evaluation strategies,
which are hard to use in realistic scenarios. In order to facilitate the
practical application, we propose a simple but effective end-to-end learning
framework for text-based person search named TIPCB (i.e., Text-Image Part-based
Convolutional Baseline). Firstly, a novel dual-path local alignment network
structure is proposed to extract visual and textual local representations, in
which images are segmented horizontally and texts are aligned adaptively. Then,
we propose a multi-stage cross-modal matching strategy, which eliminates the
modality gap from three feature levels, including low level, local level and
global level. Extensive experiments are conducted on the widely-used benchmark
dataset (CUHK-PEDES) and verify that our method outperforms the
state-of-the-art methods by 3.69%, 2.95% and 2.31% in terms of Top-1, Top-5 and
Top-10. Our code has been released in https://github.com/OrangeYHChen/TIPCB.

negative:
In this paper, we study the task of image retrieval, where the input query is
specified in the form of an image plus some text that describes desired
modifications to the input image. For example, we may present an image of the
Eiffel tower, and ask the system to find images which are visually similar but
are modified in small ways, such as being taken at nighttime instead of during
the day. To tackle this task, we learn a similarity metric between a target
image and a source image plus source text, an embedding and composing function
such that target image feature is close to the source image plus text
composition feature. We propose a new way to combine image and text using such
function that is designed for the retrieval task. We show this outperforms
existing approaches on 3 different datasets, namely Fashion-200k, MIT-States
and a new synthetic dataset we create based on CLEVR. We also show that our
approach can be used to classify input queries, in addition to image retrieval.

negative:
With the prolification of multimodal interaction in various domains, recently
there has been much interest in text based image retrieval in the computer
vision community. However most of the state of the art techniques model this
problem in a purely neural way, which makes it difficult to incorporate
pragmatic strategies in searching a large scale catalog especially when the
search requirements are insufficient and the model needs to resort to an
interactive retrieval process through multiple iterations of
question-answering. Motivated by this, we propose a neural-symbolic approach
for a one-shot retrieval of images from a large scale catalog, given the
caption description. To facilitate this, we represent the catalog and caption
as scene-graphs and model the retrieval task as a learnable graph matching
problem, trained end-to-end with a REINFORCE algorithm. Further, we briefly
describe an extension of this pipeline to an iterative retrieval framework,
based on interactive questioning and answering.

negative:
Current Visual Question Answering (VQA) systems can answer intelligent
questions about `Known' visual content. However, their performance drops
significantly when questions about visually and linguistically `Unknown'
concepts are presented during inference (`Open-world' scenario). A practical
VQA system should be able to deal with novel concepts in real world settings.
To address this problem, we propose an exemplar-based approach that transfers
learning (i.e., knowledge) from previously `Known' concepts to answer questions
about the `Unknown'. We learn a highly discriminative joint embedding space,
where visual and semantic features are fused to give a unified representation.
Once novel concepts are presented to the model, it looks for the closest match
from an exemplar set in the joint embedding space. This auxiliary information
is used alongside the given Image-Question pair to refine visual attention in a
hierarchical fashion. Since handling the high dimensional exemplars on large
datasets can be a significant challenge, we introduce an efficient matching
scheme that uses a compact feature description for search and retrieval. To
evaluate our model, we propose a new split for VQA, separating Unknown visual
and semantic concepts from the training set. Our approach shows significant
improvements over state-of-the-art VQA models on the proposed Open-World VQA
dataset and standard VQA datasets.

negative:
Visual-semantic embedding enables various tasks such as image-text retrieval,
image captioning, and visual question answering. The key to successful
visual-semantic embedding is to express visual and textual data properly by
accounting for their intricate relationship. While previous studies have
achieved much advance by encoding the visual and textual data into a joint
space where similar concepts are closely located, they often represent data by
a single vector ignoring the presence of multiple important components in an
image or text. Thus, in addition to the joint embedding space, we propose a
novel multi-head self-attention network to capture various components of visual
and textual data by attending to important parts in data. Our approach achieves
the new state-of-the-art results in image-text retrieval tasks on MS-COCO and
Flicker30K datasets. Through the visualization of the attention maps that
capture distinct semantic components at multiple positions in the image and the
text, we demonstrate that our method achieves an effective and interpretable
visual-semantic joint space.

negative:
The convolutional neural network (CNN) features can give a good description
of image content, which usually represent images with unique global vectors.
Although they are compact compared to local descriptors, they still cannot
efficiently deal with large-scale image retrieval due to the cost of the linear
incremental computation and storage. To address this issue, we build a simple
but effective indexing framework based on inverted table, which significantly
decreases both the search time and memory usage. In addition, several
strategies are fully investigated under an indexing framework to adapt it to
CNN features and compensate for quantization errors. First, we use multiple
assignment for the query and database images to increase the probability of
relevant images' co-existing in the same Voronoi cells obtained via the
clustering algorithm. Then, we introduce embedding codes to further improve
precision by removing false matches during a search. We demonstrate that by
using hashing schemes to calculate the embedding codes and by changing the
ranking rule, indexing framework speeds can be greatly improved. Extensive
experiments conducted on several unsupervised and supervised benchmarks support
these results and the superiority of the proposed indexing framework. We also
provide a fair comparison between the popular CNN features.

negative:
The inclusion of semantic information in any similarity measures improves the
efficiency of the similarity measure and provides human interpretable results
for further analysis. The similarity calculation method that focuses on
features related to the text's words only, will give less accurate results.
This paper presents three different methods that not only focus on the text's
words but also incorporates semantic information of texts in their feature
vector and computes semantic similarities. These methods are based on
corpus-based and knowledge-based methods, which are: cosine similarity using
tf-idf vectors, cosine similarity using word embedding and soft cosine
similarity using word embedding. Among these three, cosine similarity using
tf-idf vectors performed best in finding similarities between short news texts.
The similar texts given by the method are easy to interpret and can be used
directly in other information retrieval applications.

negative:
Our objective is video retrieval based on natural language queries. In
addition, we consider the analogous problem of retrieving sentences or
generating descriptions given an input video. Recent work has addressed the
problem by embedding visual and textual inputs into a common space where
semantic similarities correlate to distances. We also adopt the embedding
approach, and make the following contributions: First, we utilize web image
search in sentence embedding process to disambiguate fine-grained visual
concepts. Second, we propose embedding models for sentence, image, and video
inputs whose parameters are learned simultaneously. Finally, we show how the
proposed model can be applied to description generation. Overall, we observe a
clear improvement over the state-of-the-art methods in the video and sentence
retrieval tasks. In description generation, the performance level is comparable
to the current state-of-the-art, although our embeddings were trained for the
retrieval tasks.

negative:
This paper describes Designovel's systems which are submitted to the Fashion
IQ Challenge 2019. Goal of the challenge is building an image retrieval system
where input query is a candidate image plus two text phrases describe user's
feedback about visual differences between the candidate image and the search
target. We built the systems by combining methods from recent work on deep
metric learning, multi-modal retrieval and natual language processing. First,
we encode both candidate and target images with CNNs into high-level
representations, and encode text descriptions to a single text vector using
Transformer-based encoder. Then we compose candidate image vector and text
representation into a single vector which is exptected to be biased toward
target image vector. Finally, we compute cosine similarities between composed
vector and encoded vectors of whole dataset, and rank them in desceding order
to get ranked list. We experimented with Fashion IQ 2019 dataset in various
settings of hyperparameters, achieved 39.12% average recall by a single model
and 43.67% average recall by an ensemble of 16 models on test dataset.

negative:
Text in natural images contains rich semantics that are often highly relevant
to objects or scene. In this paper, we focus on the problem of fully exploiting
scene text for visual understanding. The main idea is combining word
representations and deep visual features into a globally trainable deep
convolutional neural network. First, the recognized words are obtained by a
scene text reading system. Then, we combine the word embedding of the
recognized words and the deep visual features into a single representation,
which is optimized by a convolutional neural network for fine-grained image
classification. In our framework, the attention mechanism is adopted to reveal
the relevance between each recognized word and the given image, which further
enhances the recognition performance. We have performed experiments on two
datasets: Con-Text dataset and Drink Bottle dataset, that are proposed for
fine-grained classification of business places and drink bottles, respectively.
The experimental results consistently demonstrate that the proposed method
combining textual and visual cues significantly outperforms classification with
only visual representations. Moreover, we have shown that the learned
representation improves the retrieval performance on the drink bottle images by
a large margin, making it potentially useful in product search.

negative:
Conventional methods for the image-text generation tasks mainly tackle the
naturally bidirectional generation tasks separately, focusing on designing
task-specific frameworks to improve the quality and fidelity of the generated
samples. Recently, Vision-Language Pre-training models have greatly improved
the performance of the image-to-text generation tasks, but large-scale
pre-training models for text-to-image synthesis task are still under-developed.
In this paper, we propose ERNIE-ViLG, a unified generative pre-training
framework for bidirectional image-text generation with transformer model. Based
on the image quantization models, we formulate both image generation and text
generation as autoregressive generative tasks conditioned on the text/image
input. The bidirectional image-text generative modeling eases the semantic
alignments across vision and language. For the text-to-image generation
process, we further propose an end-to-end training method to jointly learn the
visual sequence generator and the image reconstructor. To explore the landscape
of large-scale pre-training for bidirectional text-image generation, we train a
10-billion parameter ERNIE-ViLG model on a large-scale dataset of 145 million
(Chinese) image-text pairs which achieves state-of-the-art performance for both
text-to-image and image-to-text tasks, obtaining an FID of 7.9 on MS-COCO for
text-to-image synthesis and best results on COCO-CN and AIC-ICC for image
captioning.

negative:
Internet search affects people's cognition of the world, so mitigating biases
in search results and learning fair models is imperative for social good. We
study a unique gender bias in image search in this work: the search images are
often gender-imbalanced for gender-neutral natural language queries. We
diagnose two typical image search models, the specialized model trained on
in-domain datasets and the generalized representation model pre-trained on
massive image and text data across the internet. Both models suffer from severe
gender bias. Therefore, we introduce two novel debiasing approaches: an
in-processing fair sampling method to address the gender imbalance issue for
training models, and a post-processing feature clipping method base on mutual
information to debias multimodal representations of pre-trained models.
Extensive experiments on MS-COCO and Flickr30K benchmarks show that our methods
significantly reduce the gender bias in image search models.

negative:
Achieving good speed and accuracy trade-off on a target platform is very
important in deploying deep neural networks in real world scenarios. However,
most existing automatic architecture search approaches only concentrate on high
performance. In this work, we propose an algorithm that can offer better
speed/accuracy trade-off of searched networks, which is termed "Partial Order
Pruning". It prunes the architecture search space with a partial order
assumption to automatically search for the architectures with the best speed
and accuracy trade-off. Our algorithm explicitly takes profile information
about the inference speed on the target platform into consideration. With the
proposed algorithm, we present several Dongfeng (DF) networks that provide high
accuracy and fast inference speed on various application GPU platforms. By
further searching decoder architectures, our DF-Seg real-time segmentation
networks yield state-of-the-art speed/accuracy trade-off on both the target
embedded device and the high-end GPU.

negative:
Representation learning of textual networks poses a significant challenge as
it involves capturing amalgamated information from two modalities: (i)
underlying network structure, and (ii) node textual attributes. For this, most
existing approaches learn embeddings of text and network structure by enforcing
embeddings of connected nodes to be similar. Then for achieving a modality
fusion they use the similarities between text embedding of a node with the
structure embedding of its connected node and vice versa. This implies that
these approaches require edge information for learning embeddings and they
cannot learn embeddings of unseen nodes. In this paper we propose an approach
that achieves both modality fusion and the capability to learn embeddings of
unseen nodes. The main feature of our model is that it uses an adversarial
mechanism between text embedding based discriminator, and structure embedding
based generator to learn efficient representations. Then for learning
embeddings of unseen nodes, we use the supervision provided by the text
embedding based discriminator. In addition this, we propose a novel
architecture for learning text embedding that can combine both mutual attention
and topological attention mechanism, which give more flexible text embeddings.
Through extensive experiments on real-world datasets, we demonstrate that our
model makes substantial gains over several state-of-the-art benchmarks. In
comparison with previous state-of-the-art, it gives up to 7% improvement in
performance in predicting links among nodes seen in the training and up to 12%
improvement in performance in predicting links involving nodes not seen in
training. Further, in the node classification task, it gives up to 2%
improvement in performance.

negative:
We propose a cross-modal attention distillation framework to train a
dual-encoder model for vision-language understanding tasks, such as visual
reasoning and visual question answering. Dual-encoder models have a faster
inference speed than fusion-encoder models and enable the pre-computation of
images and text during inference. However, the shallow interaction module used
in dual-encoder models is insufficient to handle complex vision-language
understanding tasks. In order to learn deep interactions of images and text, we
introduce cross-modal attention distillation, which uses the image-to-text and
text-to-image attention distributions of a fusion-encoder model to guide the
training of our dual-encoder model. In addition, we show that applying the
cross-modal attention distillation for both pre-training and fine-tuning stages
achieves further improvements. Experimental results demonstrate that the
distilled dual-encoder model achieves competitive performance for visual
reasoning, visual entailment and visual question answering tasks while enjoying
a much faster inference speed than fusion-encoder models. Our code and models
will be publicly available at https://github.com/kugwzk/Distilled-DualEncoder.

negative:
Text-image cross-modal retrieval is a challenging task in the field of
language and vision. Most previous approaches independently embed images and
sentences into a joint embedding space and compare their similarities. However,
previous approaches rarely explore the interactions between images and
sentences before calculating similarities in the joint space. Intuitively, when
matching between images and sentences, human beings would alternatively attend
to regions in images and words in sentences, and select the most salient
information considering the interaction between both modalities. In this paper,
we propose Cross-modal Adaptive Message Passing (CAMP), which adaptively
controls the information flow for message passing across modalities. Our
approach not only takes comprehensive and fine-grained cross-modal interactions
into account, but also properly handles negative pairs and irrelevant
information with an adaptive gating scheme. Moreover, instead of conventional
joint embedding approaches for text-image matching, we infer the matching score
based on the fused features, and propose a hardest negative binary
cross-entropy loss for training. Results on COCO and Flickr30k significantly
surpass state-of-the-art methods, demonstrating the effectiveness of our
approach.

negative:
Contextual embeddings derived from transformer-based neural language models
have shown state-of-the-art performance for various tasks such as question
answering, sentiment analysis, and textual similarity in recent years.
Extensive work shows how accurately such models can represent abstract,
semantic information present in text. In this expository work, we explore a
tangent direction and analyze such models' performance on tasks that require a
more granular level of representation. We focus on the problem of textual
similarity from two perspectives: matching documents on a granular level
(requiring embeddings to capture fine-grained attributes in the text), and an
abstract level (requiring embeddings to capture overall textual semantics). We
empirically demonstrate, across two datasets from different domains, that
despite high performance in abstract document matching as expected, contextual
embeddings are consistently (and at times, vastly) outperformed by simple
baselines like TF-IDF for more granular tasks. We then propose a simple but
effective method to incorporate TF-IDF into models that use contextual
embeddings, achieving relative improvements of up to 36% on granular tasks.

negative:
Large-scale pretraining of visual representations has led to state-of-the-art
performance on a range of benchmark computer vision tasks, yet the benefits of
these techniques at extreme scale in complex production systems has been
relatively unexplored. We consider the case of a popular visual discovery
product, where these representations are trained with multi-task learning, from
use-case specific visual understanding (e.g. skin tone classification) to
general representation learning for all visual content (e.g. embeddings for
retrieval). In this work, we describe how we (1) generate a dataset with over a
billion images via large weakly-supervised pretraining to improve the
performance of these visual representations, and (2) leverage Transformers to
replace the traditional convolutional backbone, with insights into both system
and performance improvements, especially at 1B+ image scale. To support this
backbone model, we detail a systematic approach to deriving weakly-supervised
image annotations from heterogenous text signals, demonstrating the benefits of
clustering techniques to handle the long-tail distribution of image labels.
Through a comprehensive study of offline and online evaluation, we show that
large-scale Transformer-based pretraining provides significant benefits to
industry computer vision applications. The model is deployed in a production
visual shopping system, with 36% improvement in top-1 relevance and 23%
improvement in click-through volume. We conduct extensive experiments to better
understand the empirical relationships between Transformer-based architectures,
dataset scale, and the performance of production vision systems.

negative:
We tackle the problem of visual search under resource constraints. Existing
systems use the same embedding model to compute representations (embeddings)
for the query and gallery images. Such systems inherently face a hard
accuracy-efficiency trade-off: the embedding model needs to be large enough to
ensure high accuracy, yet small enough to enable query-embedding computation on
resource-constrained platforms. This trade-off could be mitigated if gallery
embeddings are generated from a large model and query embeddings are extracted
using a compact model. The key to building such a system is to ensure
representation compatibility between the query and gallery models. In this
paper, we address two forms of compatibility: One enforced by modifying the
parameters of each model that computes the embeddings. The other by modifying
the architectures that compute the embeddings, leading to compatibility-aware
neural architecture search (CMP-NAS). We test CMP-NAS on challenging retrieval
tasks for fashion images (DeepFashion2), and face images (IJB-C). Compared to
ordinary (homogeneous) visual search using the largest embedding model
(paragon), CMP-NAS achieves 80-fold and 23-fold cost reduction while
maintaining accuracy within 0.3% and 1.6% of the paragon on DeepFashion2 and
IJB-C respectively.

negative:
We present a multi-modal dialog system to assist online shoppers in visually
browsing through large catalogs. Visual browsing is different from visual
search in that it allows the user to explore the wide range of products in a
catalog, beyond the exact search matches. We focus on a slightly asymmetric
version of the complete multi-modal dialog where the system can understand both
text and image queries but responds only in images. We formulate our problem of
"showing $k$ best images to a user" based on the dialog context so far, as
sampling from a Gaussian Mixture Model in a high dimensional joint multi-modal
embedding space, that embed both the text and the image queries. Our system
remembers the context of the dialog and uses an exploration-exploitation
paradigm to assist in visual browsing. We train and evaluate the system on a
multi-modal dialog dataset that we generate from large catalog data. Our
experiments are promising and show that the agent is capable of learning and
can display relevant results with an average cosine similarity of 0.85 to the
ground truth. Our preliminary human evaluation also corroborates the fact that
such a multi-modal dialog system for visual browsing is well-received and is
capable of engaging human users.

negative:
Point cloud architecture design has become a crucial problem for 3D deep
learning. Several efforts exist to manually design architectures with high
accuracy in point cloud tasks such as classification, segmentation, and
detection. Recent progress in automatic Neural Architecture Search (NAS)
minimizes the human effort in network design and optimizes high performing
architectures. However, these efforts fail to consider important factors such
as latency during inference. Latency is of high importance in time critical
applications like self-driving cars, robot navigation, and mobile applications,
that are generally bound by the available hardware. In this paper, we introduce
a new NAS framework, dubbed LC-NAS, where we search for point cloud
architectures that are constrained to a target latency. We implement a novel
latency constraint formulation to trade-off between accuracy and latency in our
architecture search. Contrary to previous works, our latency loss guarantees
that the final network achieves latency under a specified target value. This is
crucial when the end task is to be deployed in a limited hardware setting.
Extensive experiments show that LC-NAS is able to find state-of-the-art
architectures for point cloud classification in ModelNet40 with minimal
computational cost. We also show how our searched architectures achieve any
desired latency with a reasonably low drop in accuracy. Finally, we show how
our searched architectures easily transfer to a different task, part
segmentation on PartNet, where we achieve state-of-the-art results while
lowering latency by a factor of 10.

negative:
In this paper, we propose a single UniFied transfOrmer (UFO), which is
capable of processing either unimodal inputs (e.g., image or language) or
multimodal inputs (e.g., the concatenation of the image and the question), for
vision-language (VL) representation learning. Existing approaches typically
design an individual network for each modality and/or a specific fusion network
for multimodal tasks. To simplify the network architecture, we use a single
transformer network and enforce multi-task learning during VL pre-training,
which includes the image-text contrastive loss, image-text matching loss, and
masked language modeling loss based on the bidirectional and the seq2seq
attention mask. The same transformer network is used as the image encoder, the
text encoder, or the fusion network in different pre-training tasks.
Empirically, we observe less conflict among different tasks and achieve new
state of the arts on visual question answering, COCO image captioning
(cross-entropy optimization) and nocaps (in SPICE). On other downstream tasks,
e.g., image-text retrieval, we also achieve competitive performance.

negative:
In this paper, we study the problem of image-text matching. Inferring the
latent semantic alignment between objects or other salient stuff (e.g. snow,
sky, lawn) and the corresponding words in sentences allows to capture
fine-grained interplay between vision and language, and makes image-text
matching more interpretable. Prior work either simply aggregates the similarity
of all possible pairs of regions and words without attending differentially to
more and less important words or regions, or uses a multi-step attentional
process to capture limited number of semantic alignments which is less
interpretable. In this paper, we present Stacked Cross Attention to discover
the full latent alignments using both image regions and words in a sentence as
context and infer image-text similarity. Our approach achieves the
state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K,
our approach outperforms the current best methods by 22.1% relatively in text
retrieval from image query, and 18.2% relatively in image retrieval with text
query (based on Recall@1). On MS-COCO, our approach improves sentence retrieval
by 17.8% relatively and image retrieval by 16.6% relatively (based on Recall@1
using the 5K test set). Code has been made available at:
https://github.com/kuanghuei/SCAN.

negative:
Although transformers have become the neural architectures of choice for
natural language processing, they require orders of magnitude more training
data, GPU memory, and computations in order to compete with convolutional
neural networks for computer vision. The attention mechanism of transformers
scales quadratically with the length of the input sequence, and unrolled images
have long sequence lengths. Plus, transformers lack an inductive bias that is
appropriate for images. We tested three modifications to vision transformer
(ViT) architectures that address these shortcomings. Firstly, we alleviate the
quadratic bottleneck by using linear attention mechanisms, called X-formers
(such that, X in {Performer, Linformer, Nystr\"omformer}), thereby creating
Vision X-formers (ViXs). This resulted in up to a seven times reduction in the
GPU memory requirement. We also compared their performance with FNet and
multi-layer perceptron mixers, which further reduced the GPU memory
requirement. Secondly, we introduced an inductive bias for images by replacing
the initial linear embedding layer by convolutional layers in ViX, which
significantly increased classification accuracy without increasing the model
size. Thirdly, we replaced the learnable 1D position embeddings in ViT with
Rotary Position Embedding (RoPE), which increases the classification accuracy
for the same model size. We believe that incorporating such changes can
democratize transformers by making them accessible to those with limited data
and computing resources.

negative:
The existing methods for image search reranking suffer from the
unfaithfulness of the assumptions under which the text-based images search
result. The resulting images contain more irrelevant images. Hence the re
ranking concept arises to re rank the retrieved images based on the text around
the image and data of data of image and visual feature of image. A number of
methods are differentiated for this re-ranking. The high ranked images are used
as noisy data and a k means algorithm for classification is learned to rectify
the ranking further. We are study the affect ability of the cross validation
method to this training data. The pre eminent originality of the overall method
is in collecting text/metadata of image and visual features in order to achieve
an automatic ranking of the images. Supervision is initiated to learn the model
weights offline, previous to reranking process. While model learning needs
manual labeling of the results for a some limited queries, the resulting model
is query autonomous and therefore applicable to any other query .Examples are
given for a selection of other classes like vehicles, animals and other
classes.

negative:
The state of the art on many NLP tasks is currently achieved by large
pre-trained language models, which require a considerable amount of
computation. We explore a setting where many different predictions are made on
a single piece of text. In that case, some of the computational cost during
inference can be amortized over the different tasks using a shared text
encoder. We compare approaches for training such an encoder and show that
encoders pre-trained over multiple tasks generalize well to unseen tasks. We
also compare ways of extracting fixed- and limited-size representations from
this encoder, including different ways of pooling features extracted from
multiple layers or positions. Our best approach compares favorably to knowledge
distillation, achieving higher accuracy and lower computational cost once the
system is handling around 7 tasks. Further, we show that through binary
quantization, we can reduce the size of the extracted representations by a
factor of 16 making it feasible to store them for later use. The resulting
method offers a compelling solution for using large-scale pre-trained models at
a fraction of the computational cost when multiple tasks are performed on the
same text.

negative:
We study the problem of recognizing visual entities from the textual
descriptions of their classes. Specifically, given birds' images with free-text
descriptions of their species, we learn to classify images of previously-unseen
species based on specie descriptions. This setup has been studied in the vision
community under the name zero-shot learning from text, focusing on learning to
transfer knowledge about visual aspects of birds from seen classes to
previously-unseen ones. Here, we suggest focusing on the textual description
and distilling from the description the most relevant information to
effectively match visual features to the parts of the text that discuss them.
Specifically, (1) we propose to leverage the similarity between species,
reflected in the similarity between text descriptions of the species. (2) we
derive visual summaries of the texts, i.e., extractive summaries that focus on
the visual features that tend to be reflected in images. We propose a simple
attention-based model augmented with the similarity and visual summaries
components. Our empirical results consistently and significantly outperform the
state-of-the-art on the largest benchmarks for text-based zero-shot learning,
illustrating the critical importance of texts for zero-shot image-recognition.

negative:
Cross-modal similarity search is a problem about designing a search system
supporting querying across content modalities, e.g., using an image to search
for texts or using a text to search for images. This paper presents a compact
coding solution for efficient search, with a focus on the quantization approach
which has already shown the superior performance over the hashing solutions in
the single-modal similarity search. We propose a cross-modal quantization
approach, which is among the early attempts to introduce quantization into
cross-modal search. The major contribution lies in jointly learning the
quantizers for both modalities through aligning the quantized representations
for each pair of image and text belonging to a document. In addition, our
approach simultaneously learns the common space for both modalities in which
quantization is conducted to enable efficient and effective search using the
Euclidean distance computed in the common space with fast distance table
lookup. Experimental results compared with several competitive algorithms over
three benchmark datasets demonstrate that the proposed approach achieves the
state-of-the-art performance.

negative:
The accuracy of deep convolutional neural networks (CNNs) generally improves
when fueled with high resolution images. However, this often comes at a high
computational cost and high memory footprint. Inspired by the fact that not all
regions in an image are task-relevant, we propose a novel framework that
performs efficient image classification by processing a sequence of relatively
small inputs, which are strategically selected from the original image with
reinforcement learning. Such a dynamic decision process naturally facilitates
adaptive inference at test time, i.e., it can be terminated once the model is
sufficiently confident about its prediction and thus avoids further redundant
computation. Notably, our framework is general and flexible as it is compatible
with most of the state-of-the-art light-weighted CNNs (such as MobileNets,
EfficientNets and RegNets), which can be conveniently deployed as the backbone
feature extractor. Experiments on ImageNet show that our method consistently
improves the computational efficiency of a wide variety of deep models. For
example, it further reduces the average latency of the highly efficient
MobileNet-V3 on an iPhone XS Max by 20% without sacrificing accuracy. Code and
pre-trained models are available at
https://github.com/blackfeather-wang/GFNet-Pytorch.

negative:
Our objective in this work is video-text retrieval - in particular a joint
embedding that enables efficient text-to-video retrieval. The challenges in
this area include the design of the visual architecture and the nature of the
training data, in that the available large scale video-text training datasets,
such as HowTo100M, are noisy and hence competitive performance is achieved only
at scale through large amounts of compute. We address both these challenges in
this paper. We propose an end-to-end trainable model that is designed to take
advantage of both large-scale image and video captioning datasets. Our model is
an adaptation and extension of the recent ViT and Timesformer architectures,
and consists of attention in both space and time. The model is flexible and can
be trained on both image and video text datasets, either independently or in
conjunction. It is trained with a curriculum learning schedule that begins by
treating images as 'frozen' snapshots of video, and then gradually learns to
attend to increasing temporal context when trained on video datasets. We also
provide a new video-text pretraining dataset WebVid-2M, comprised of over two
million videos with weak captions scraped from the internet. Despite training
on datasets that are an order of magnitude smaller, we show that this approach
yields state-of-the-art results on standard downstream video-retrieval
benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.

negative:
Search engine has become a fundamental component in various web and mobile
applications. Retrieving relevant documents from the massive datasets is
challenging for a search engine system, especially when faced with verbose or
tail queries. In this paper, we explore a vector space search framework for
document retrieval. Specifically, we trained a deep semantic matching model so
that each query and document can be encoded as a low dimensional embedding. Our
model was trained based on BERT architecture. We deployed a fast
k-nearest-neighbor index service for online serving. Both offline and online
metrics demonstrate that our method improved retrieval performance and search
quality considerably, particularly for tail

negative:
LiveSketch is a novel algorithm for searching large image collections using
hand-sketched queries. LiveSketch tackles the inherent ambiguity of sketch
search by creating visual suggestions that augment the query as it is drawn,
making query specification an iterative rather than one-shot process that helps
disambiguate users' search intent. Our technical contributions are: a triplet
convnet architecture that incorporates an RNN based variational autoencoder to
search for images using vector (stroke-based) queries; real-time clustering to
identify likely search intents (and so, targets within the search embedding);
and the use of backpropagation from those targets to perturb the input stroke
sequence, so suggesting alterations to the query in order to guide the search.
We show improvements in accuracy and time-to-task over contemporary baselines
using a 67M image corpus.

negative:
Text-based person search aims to retrieve images of a certain pedestrian by a
textual description. The key challenge of this task is to eliminate the
inter-modality gap and achieve the feature alignment across modalities. In this
paper, we propose a semantic-aligned embedding method for text-based person
search, in which the feature alignment across modalities is achieved by
automatically learning the semantic-aligned visual features and textual
features. First, we introduce two Transformer-based backbones to encode robust
feature representations of the images and texts. Second, we design a
semantic-aligned feature aggregation network to adaptively select and aggregate
features with the same semantics into part-aware features, which is achieved by
a multi-head attention module constrained by a cross-modality part alignment
loss and a diversity loss. Experimental results on the CUHK-PEDES and Flickr30K
datasets show that our method achieves state-of-the-art performances.

negative:
Open-domain extractive question answering works well on textual data by first
retrieving candidate texts and then extracting the answer from those
candidates. However, some questions cannot be answered by text alone but
require information stored in tables. In this paper, we present an approach for
retrieving both texts and tables relevant to a question by jointly encoding
texts, tables and questions into a single vector space. To this end, we create
a new multi-modal dataset based on text and table datasets from related work
and compare the retrieval performance of different encoding schemata. We find
that dense vector embeddings of transformer models outperform sparse embeddings
on four out of six evaluation datasets. Comparing different dense embedding
models, tri-encoders with one encoder for each question, text and table,
increase retrieval performance compared to bi-encoders with one encoder for the
question and one for both text and tables. We release the newly created
multi-modal dataset to the community so that it can be used for training and
evaluation.

negative:
Predicting patient mortality is an important and challenging problem in the
healthcare domain, especially for intensive care unit (ICU) patients.
Electronic health notes serve as a rich source for learning patient
representations, that can facilitate effective risk assessment. However, a
large portion of clinical notes are unstructured and also contain domain
specific terminologies, from which we need to extract structured information.
In this paper, we introduce an embedding framework to learn
semantically-plausible distributed representations of clinical notes that
exploits the semantic correspondence between the unstructured texts and their
corresponding structured knowledge, known as semantic frame, in a hierarchical
fashion. Our approach integrates text modeling and semantic correspondence
learning into a single model that comprises 1) an unstructured embedding module
that makes use of self-similarity matrix representations in order to inject
structural regularities of different segments inherent in clinical texts to
promote local coherence, 2) a structured embedding module to embed the semantic
frames (e.g., UMLS semantic types) with deep ConvNet and 3) a hierarchical
semantic correspondence module that embeds by enhancing the interactions
between text-semantic frame embedding pairs at multiple levels (i.e., words,
sentence, note). Evaluations on multiple embedding benchmarks on post discharge
intensive care patient mortality prediction tasks demonstrate its effectiveness
compared to approaches that do not exploit the semantic interactions between
structured and unstructured information present in clinical notes.

negative:
Large-scale pre-training methods of learning cross-modal representations on
image-text pairs are becoming popular for vision-language tasks. While existing
methods simply concatenate image region features and text features as input to
the model to be pre-trained and use self-attention to learn image-text semantic
alignments in a brute force manner, in this paper, we propose a new learning
method Oscar (Object-Semantics Aligned Pre-training), which uses object tags
detected in images as anchor points to significantly ease the learning of
alignments. Our method is motivated by the observation that the salient objects
in an image can be accurately detected, and are often mentioned in the paired
text. We pre-train an Oscar model on the public corpus of 6.5 million
text-image pairs, and fine-tune it on downstream tasks, creating new
state-of-the-arts on six well-established vision-language understanding and
generation tasks.

negative:
Latent text representations exhibit geometric regularities, such as the
famous analogy: queen is to king what woman is to man. Such structured semantic
relations were not demonstrated on image representations. Recent works aiming
at bridging this semantic gap embed images and text into a multimodal space,
enabling the transfer of text-defined transformations to the image modality. We
introduce the SIMAT dataset to evaluate the task of Image Retrieval with
Multimodal queries. SIMAT contains 6k images and 18k textual transformation
queries that aim at either replacing scene elements or changing pairwise
relationships between scene elements. The goal is to retrieve an image
consistent with the (source image, text transformation) query. We use an
image/text matching oracle (OSCAR) to assess whether the image transformation
is successful. The SIMAT dataset will be publicly available. We use SIMAT to
evaluate the geometric properties of multimodal embedding spaces trained with
an image/text matching objective, like CLIP. We show that vanilla CLIP
embeddings are not very well suited to transform images with delta vectors, but
that a simple finetuning on the COCO dataset can bring dramatic improvements.
We also study whether it is beneficial to leverage pretrained universal
sentence encoders (FastText, LASER and LaBSE).

negative:
Web search engines focus on serving highly relevant results within hundreds
of milliseconds. Pre-trained language transformer models such as BERT are
therefore hard to use in this scenario due to their high computational demands.
We present our real-time approach to the document ranking problem leveraging a
BERT-based siamese architecture. The model is already deployed in a commercial
search engine and it improves production performance by more than 3%. For
further research and evaluation, we release DaReCzech, a unique data set of 1.6
million Czech user query-document pairs with manually assigned relevance
levels. We also release Small-E-Czech, an Electra-small language model
pre-trained on a large Czech corpus. We believe this data will support
endeavours both of search relevance and multilingual-focused research
communities.

negative:
This work addresses approximate nearest neighbor search applied in the domain
of large-scale image retrieval. Within the group testing framework we propose
an efficient off-line construction of the search structures. The linear-time
complexity orthogonal grouping increases the probability that at most one
element from each group is matching to a given query. Non-maxima suppression
with each group efficiently reduces the number of false positive results at no
extra cost. Unlike in other well-performing approaches, all processing is
local, fast, and suitable to process data in batches and in parallel. We
experimentally show that the proposed method achieves search accuracy of the
exhaustive search with significant reduction in the search complexity. The
method can be naturally combined with existing embedding methods.

negative:
Reasoning with knowledge expressed in natural language and Knowledge Bases
(KBs) is a major challenge for Artificial Intelligence, with applications in
machine reading, dialogue, and question answering. General neural architectures
that jointly learn representations and transformations of text are very
data-inefficient, and it is hard to analyse their reasoning process. These
issues are addressed by end-to-end differentiable reasoning systems such as
Neural Theorem Provers (NTPs), although they can only be used with small-scale
symbolic KBs. In this paper we first propose Greedy NTPs (GNTPs), an extension
to NTPs addressing their complexity and scalability limitations, thus making
them applicable to real-world datasets. This result is achieved by dynamically
constructing the computation graph of NTPs and including only the most
promising proof paths during inference, thus obtaining orders of magnitude more
efficient models. Then, we propose a novel approach for jointly reasoning over
KBs and textual mentions, by embedding logic facts and natural language
sentences in a shared embedding space. We show that GNTPs perform on par with
NTPs at a fraction of their cost while achieving competitive link prediction
results on large datasets, providing explanations for predictions, and inducing
interpretable models. Source code, datasets, and supplementary material are
available online at https://github.com/uclnlp/gntp.

negative:
In this paper, we introduce an attribute-based interactive image search which
can leverage human-in-the-loop feedback to iteratively refine image search
results. We study active image search where human feedback is solicited
exclusively in visual form, without using relative attribute annotations used
by prior work which are not typically found in many datasets. In order to
optimize the image selection strategy, a deep reinforcement model is trained to
learn what images are informative rather than rely on hand-crafted measures
typically leveraged in prior work. Additionally, we extend the recently
introduced Conditional Similarity Network to incorporate global similarity in
training visual embeddings, which results in more natural transitions as the
user explores the learned similarity embeddings. Our experiments demonstrate
the effectiveness of our approach, producing compelling results on both active
image search and image attribute representation tasks.

negative:
Person search targets to search the probe person from the unconstrainted
scene images, which can be treated as the combination of person detection and
person matching. However, the existing methods based on the Detection-Matching
framework ignore the person objectness and repulsion (OR) which are both
beneficial to reduce the effect of distractor images. In this paper, we propose
an OR similarity by jointly considering the objectness and repulsion
information. Besides the traditional visual similarity term, the OR similarity
also contains an objectness term and a repulsion term. The objectness term can
reduce the similarity of distractor images that not contain a person and boost
the performance of person search by improving the ranking of positive samples.
Because the probe person has a different person ID with its \emph{neighbors},
the gallery images having a higher similarity with the \emph{neighbors of
probe} should have a lower similarity with the probe person. Based on this
repulsion constraint, the repulsion term is proposed to reduce the similarity
of distractor images that are not most similar to the probe person. Treating
the Faster R-CNN as the person detector, the OR similarity is evaluated on PRW
and CUHK-SYSU datasets by the Detection-Matching framework with six description
models. The extensive experiments demonstrate that the proposed OR similarity
can effectively reduce the similarity of distractor samples and further boost
the performance of person search, e.g., improve the mAP from 92.32% to 93.23%
for CUHK-SYSY dataset, and from 50.91% to 52.30% for PRW datasets.

negative:
We present a universal framework to model contextualized sentence
representations with visual awareness that is motivated to overcome the
shortcomings of the multimodal parallel data with manual annotations. For each
sentence, we first retrieve a diversity of images from a shared cross-modal
embedding space, which is pre-trained on a large-scale of text-image pairs.
Then, the texts and images are respectively encoded by transformer encoder and
convolutional neural network. The two sequences of representations are further
fused by a simple and effective attention layer. The architecture can be easily
applied to text-only natural language processing tasks without manually
annotating multimodal parallel corpora. We apply the proposed method on three
tasks, including neural machine translation, natural language inference and
sequence labeling and experimental results verify the effectiveness.

negative:
The goal of text ranking is to generate an ordered list of texts retrieved
from a corpus in response to a query. Although the most common formulation of
text ranking is search, instances of the task can also be found in many natural
language processing applications. This survey provides an overview of text
ranking with neural network architectures known as transformers, of which BERT
is the best-known example. The combination of transformers and self-supervised
pretraining has been responsible for a paradigm shift in natural language
processing (NLP), information retrieval (IR), and beyond. In this survey, we
provide a synthesis of existing work as a single point of entry for
practitioners who wish to gain a better understanding of how to apply
transformers to text ranking problems and researchers who wish to pursue work
in this area. We cover a wide range of modern techniques, grouped into two
high-level categories: transformer models that perform reranking in multi-stage
architectures and dense retrieval techniques that perform ranking directly.
There are two themes that pervade our survey: techniques for handling long
documents, beyond typical sentence-by-sentence processing in NLP, and
techniques for addressing the tradeoff between effectiveness (i.e., result
quality) and efficiency (e.g., query latency, model and index size). Although
transformer architectures and pretraining techniques are recent innovations,
many aspects of how they are applied to text ranking are relatively well
understood and represent mature techniques. However, there remain many open
research questions, and thus in addition to laying out the foundations of
pretrained transformers for text ranking, this survey also attempts to
prognosticate where the field is heading.

negative:
Neural text matching models have been widely used in community question
answering, information retrieval, and dialogue. However, these models designed
for short texts cannot well address the long-form text matching problem,
because there are many contexts in long-form texts can not be directly aligned
with each other, and it is difficult for existing models to capture the key
matching signals from such noisy data. Besides, these models are
computationally expensive for simply use all textual data indiscriminately. To
tackle the effectiveness and efficiency problem, we propose a novel
hierarchical noise filtering model, namely Match-Ignition. The main idea is to
plug the well-known PageRank algorithm into the Transformer, to identify and
filter both sentence and word level noisy information in the matching process.
Noisy sentences are usually easy to detect because previous work has shown that
their similarity can be explicitly evaluated by the word overlapping, so we
directly use PageRank to filter such information based on a sentence similarity
graph. Unlike sentences, words rely on their contexts to express concrete
meanings, so we propose to jointly learn the filtering and matching process, to
well capture the critical word-level matching signals. Specifically, a word
graph is first built based on the attention scores in each self-attention block
of Transformer, and key words are then selected by applying PageRank on this
graph. In this way, noisy words will be filtered out layer by layer in the
matching process. Experimental results show that Match-Ignition outperforms
both SOTA short text matching models and recent long-form text matching models.
We also conduct detailed analysis to show that Match-Ignition efficiently
captures important sentences and words, to facilitate the long-form text
matching process.

negative:
Image captioning models generally lack the capability to take into account
user interest, and usually default to global descriptions that try to balance
readability, informativeness, and information overload. On the other hand, VQA
models generally lack the ability to provide long descriptive answers, while
expecting the textual question to be quite precise. We present a method to
control the concepts that an image caption should focus on, using an additional
input called the guiding text that refers to either groundable or ungroundable
concepts in the image. Our model consists of a Transformer-based multimodal
encoder that uses the guiding text together with global and object-level image
features to derive early-fusion representations used to generate the guided
caption. While models trained on Visual Genome data have an in-domain advantage
of fitting well when guided with automatic object labels, we find that guided
captioning models trained on Conceptual Captions generalize better on
out-of-domain images and guiding texts. Our human-evaluation results indicate
that attempting in-the-wild guided image captioning requires access to large,
unrestricted-domain training datasets, and that increased style diversity (even
without increasing the number of unique tokens) is a key factor for improved
performance.

negative:
Action recognition, early prediction, and online action detection are
complementary disciplines that are often studied independently. Most online
action detection networks use a pre-trained feature extractor, which might not
be optimal for its new task. We address the task-specific feature extraction
with a teacher-student framework between the aforementioned disciplines, and a
novel training strategy. Our network, Online Knowledge Distillation Action
Detection network (OKDAD), embeds online early prediction and online temporal
segment proposal subnetworks in parallel. Low interclass and high intraclass
similarity are encouraged during teacher training. Knowledge distillation to
the OKDAD network is ensured via layer reuse and cosine similarity between
teacher-student feature vectors. Layer reuse and similarity learning
significantly improve our baseline which uses a generic feature extractor. We
evaluate our framework on infrared videos from two popular datasets, NTU RGB+D
(action recognition, early prediction) and PKU MMD (action detection). Unlike
previous attempts on those datasets, our student networks perform without any
knowledge of the future. Even with this added difficulty, we achieve
state-of-the-art results on both datasets. Moreover, our networks use infrared
from RGB-D cameras, which we are the first to use for online action detection,
to our knowledge.

negative:
We consider the large-scale query-document retrieval problem: given a query
(e.g., a question), return the set of relevant documents (e.g., paragraphs
containing the answer) from a large document corpus. This problem is often
solved in two steps. The retrieval phase first reduces the solution space,
returning a subset of candidate documents. The scoring phase then re-ranks the
documents. Critically, the retrieval algorithm not only desires high recall but
also requires to be highly efficient, returning candidates in time sublinear to
the number of documents. Unlike the scoring phase witnessing significant
advances recently due to the BERT-style pre-training tasks on cross-attention
models, the retrieval phase remains less well studied. Most previous works rely
on classic Information Retrieval (IR) methods such as BM-25 (token matching +
TF-IDF weights). These models only accept sparse handcrafted features and can
not be optimized for different downstream tasks of interest. In this paper, we
conduct a comprehensive study on the embedding-based retrieval models. We show
that the key ingredient of learning a strong embedding-based Transformer model
is the set of pre-training tasks. With adequately designed paragraph-level
pre-training tasks, the Transformer models can remarkably improve over the
widely-used BM-25 as well as embedding models without Transformers. The
paragraph-level pre-training tasks we studied are Inverse Cloze Task (ICT),
Body First Selection (BFS), Wiki Link Prediction (WLP), and the combination of
all three.

negative:
The massive scale and growth of textual biomedical data have made its
indexing and classification increasingly important. However, existing research
on this topic mainly utilized convolutional and recurrent neural networks,
which generally achieve inferior performance than the novel transformers. On
the other hand, systems that apply transformers only focus on the target
documents, overlooking the rich semantic information that label descriptions
contain. To address this gap, we develop a transformer-based biomedical text
classifier that considers label information. The system achieves this with a
label attention module incorporated into the fine-tuning process of pretrained
language models (PTMs). Our results on two public medical datasets show that
the proposed fine-tuning scheme outperforms the vanilla PTMs and
state-of-the-art models.

negative:
In this paper we propose a framework for predicting kernelized classifiers in
the visual domain for categories with no training images where the knowledge
comes from textual description about these categories. Through our optimization
framework, the proposed approach is capable of embedding the class-level
knowledge from the text domain as kernel classifiers in the visual domain. We
also proposed a distributional semantic kernel between text descriptions which
is shown to be effective in our setting. The proposed framework is not
restricted to textual descriptions, and can also be applied to other forms
knowledge representations. Our approach was applied for the challenging task of
zero-shot learning of fine-grained categories from text descriptions of these
categories.

negative:
We propose an accurate and efficient scene text detection framework, termed
FAST (i.e., faster arbitrarily-shaped text detector). Different from recent
advanced text detectors that used complicated post-processing and hand-crafted
network architectures, resulting in low inference speed, FAST has two new
designs. (1) We design a minimalist kernel representation (only has 1-channel
output) to model text with arbitrary shape, as well as a GPU-parallel
post-processing to efficiently assemble text lines with a negligible time
overhead. (2) We search the network architecture tailored for text detection,
leading to more powerful features than most networks that are searched for
image classification. Benefiting from these two designs, FAST achieves an
excellent trade-off between accuracy and efficiency on several challenging
datasets, including Total Text, CTW1500, ICDAR 2015, and MSRA-TD500. For
example, FAST-T yields 81.6% F-measure at 152 FPS on Total-Text, outperforming
the previous fastest method by 1.7 points and 70 FPS in terms of accuracy and
speed. With TensorRT optimization, the inference speed can be further
accelerated to over 600 FPS. Code and models will be released at
https://github.com/czczup/FAST.

negative:
For some images, descriptions written by multiple people are consistent with
each other. But for other images, descriptions across people vary considerably.
In other words, some images are specific $-$ they elicit consistent
descriptions from different people $-$ while other images are ambiguous.
Applications involving images and text can benefit from an understanding of
which images are specific and which ones are ambiguous. For instance, consider
text-based image retrieval. If a query description is moderately similar to the
caption (or reference description) of an ambiguous image, that query may be
considered a decent match to the image. But if the image is very specific, a
moderate similarity between the query and the reference description may not be
sufficient to retrieve the image.
  In this paper, we introduce the notion of image specificity. We present two
mechanisms to measure specificity given multiple descriptions of an image: an
automated measure and a measure that relies on human judgement. We analyze
image specificity with respect to image content and properties to better
understand what makes an image specific. We then train models to automatically
predict the specificity of an image from image features alone without requiring
textual descriptions of the image. Finally, we show that modeling image
specificity leads to improvements in a text-based image retrieval application.

negative:
The goal of our industrial ticketing system is to retrieve a relevant
solution for an input query, by matching with historical tickets stored in
knowledge base. A query is comprised of subject and description, while a
historical ticket consists of subject, description and solution. To retrieve a
relevant solution, we use textual similarity paradigm to learn similarity in
the query and historical tickets. The task is challenging due to significant
term mismatch in the query and ticket pairs of asymmetric lengths, where
subject is a short text but description and solution are multi-sentence texts.
We present a novel Replicated Siamese LSTM model to learn similarity in
asymmetric text pairs, that gives 22% and 7% gain (Accuracy@10) for retrieval
task, respectively over unsupervised and supervised baselines. We also show
that the topic and distributed semantic features for short and long texts
improved both similarity learning and retrieval.

negative:
Modern convolutional neural networks apply the same operations on every pixel
in an image. However, not all image regions are equally important. To address
this inefficiency, we propose a method to dynamically apply convolutions
conditioned on the input image. We introduce a residual block where a small
gating branch learns which spatial positions should be evaluated. These
discrete gating decisions are trained end-to-end using the Gumbel-Softmax
trick, in combination with a sparsity criterion. Our experiments on CIFAR,
ImageNet and MPII show that our method has better focus on the region of
interest and better accuracy than existing methods, at a lower computational
complexity. Moreover, we provide an efficient CUDA implementation of our
dynamic convolutions using a gather-scatter approach, achieving a significant
improvement in inference speed with MobileNetV2 residual blocks. On human pose
estimation, a task that is inherently spatially sparse, the processing speed is
increased by 60% with no loss in accuracy.

negative:
Text-to-image person re-identification (ReID) aims to search for images
containing a person of interest using textual descriptions. However, due to the
significant modality gap and the large intra-class variance in textual
descriptions, text-to-image ReID remains a challenging problem. Accordingly, in
this paper, we propose a Semantically Self-Aligned Network (SSAN) to handle the
above problems. First, we propose a novel method that automatically extracts
semantically aligned part-level features from the two modalities. Second, we
design a multi-view non-local network that captures the relationships between
body parts, thereby establishing better correspondences between body parts and
noun phrases. Third, we introduce a Compound Ranking (CR) loss that makes use
of textual descriptions for other images of the same identity to provide extra
supervision, thereby effectively reducing the intra-class variance in textual
features. Finally, to expedite future research in text-to-image ReID, we build
a new database named ICFG-PEDES. Extensive experiments demonstrate that SSAN
outperforms state-of-the-art approaches by significant margins. Both the new
ICFG-PEDES database and the SSAN code are available at
https://github.com/zifyloo/SSAN.

negative:
The abundance of multimodal data (e.g. social media posts) has inspired
interest in cross-modal retrieval methods. Popular approaches rely on a variety
of metric learning losses, which prescribe what the proximity of image and text
should be, in the learned space. However, most prior methods have focused on
the case where image and text convey redundant information; in contrast,
real-world image-text pairs convey complementary information with little
overlap. Further, images in news articles and media portray topics in a
visually diverse fashion; thus, we need to take special care to ensure a
meaningful image representation. We propose novel within-modality losses which
encourage semantic coherency in both the text and image subspaces, which does
not necessarily align with visual coherency. Our method ensures that not only
are paired images and texts close, but the expected image-image and text-text
relationships are also observed. Our approach improves the results of
cross-modal retrieval on four datasets compared to five baselines.

negative:
Conducting text retrieval in a dense learned representation space has many
intriguing advantages over sparse retrieval. Yet the effectiveness of dense
retrieval (DR) often requires combination with sparse retrieval. In this paper,
we identify that the main bottleneck is in the training mechanisms, where the
negative instances used in training are not representative of the irrelevant
documents in testing. This paper presents Approximate nearest neighbor Negative
Contrastive Estimation (ANCE), a training mechanism that constructs negatives
from an Approximate Nearest Neighbor (ANN) index of the corpus, which is
parallelly updated with the learning process to select more realistic negative
training instances. This fundamentally resolves the discrepancy between the
data distribution used in the training and testing of DR. In our experiments,
ANCE boosts the BERT-Siamese DR model to outperform all competitive dense and
sparse retrieval baselines. It nearly matches the accuracy of
sparse-retrieval-and-BERT-reranking using dot-product in the ANCE-learned
representation space and provides almost 100x speed-up.

negative:
We propose Unicoder-VL, a universal encoder that aims to learn joint
representations of vision and language in a pre-training manner. Borrow ideas
from cross-lingual pre-trained models, such as XLM and Unicoder, both visual
and linguistic contents are fed into a multi-layer Transformer for the
cross-modal pre-training, where three pre-trained tasks are employed, including
Masked Language Modeling (MLM), Masked Object Classification (MOC) and
Visual-linguistic Matching (VLM). The first two tasks learn context-aware
representations for input tokens based on linguistic and visual contents
jointly. The last task tries to predict whether an image and a text describe
each other. After pretraining on large-scale image-caption pairs, we transfer
Unicoder-VL to caption-based image-text retrieval and visual commonsense
reasoning, with just one additional output layer. We achieve state-of-the-art
or comparable results on both two tasks and show the powerful ability of the
cross-modal pre-training.

negative:
This paper investigates whether the power of the models pre-trained on text
data, such as BERT, can be transferred to general token sequence classification
applications. To verify pre-trained models' transferability, we test the
pre-trained models on text classification tasks with meanings of tokens
mismatches, and real-world non-text token sequence classification data,
including amino acid, DNA, and music. We find that even on non-text data, the
models pre-trained on text converge faster, perform better than the randomly
initialized models, and only slightly worse than the models using task-specific
knowledge. We also find that the representations of the text and non-text
pre-trained models share non-trivial similarities.

negative:
Image to image translation aims to learn a mapping that transforms an image
from one visual domain to another. Recent works assume that images descriptors
can be disentangled into a domain-invariant content representation and a
domain-specific style representation. Thus, translation models seek to preserve
the content of source images while changing the style to a target visual
domain. However, synthesizing new images is extremely challenging especially in
multi-domain translations, as the network has to compose content and style to
generate reliable and diverse images in multiple domains. In this paper we
propose the use of an image retrieval system to assist the image-to-image
translation task. First, we train an image-to-image translation model to map
images to multiple domains. Then, we train an image retrieval model using real
and generated images to find images similar to a query one in content but in a
different domain. Finally, we exploit the image retrieval system to fine-tune
the image-to-image translation model and generate higher quality images. Our
experiments show the effectiveness of the proposed solution and highlight the
contribution of the retrieval network, which can benefit from additional
unlabeled data and help image-to-image translation models in the presence of
scarce data.

negative:
In this paper we propose the Structured Deep Neural Network (structured DNN)
as a structured and deep learning framework. This approach can learn to find
the best structured object (such as a label sequence) given a structured input
(such as a vector sequence) by globally considering the mapping relationships
between the structures rather than item by item.
  When automatic speech recognition is viewed as a special case of such a
structured learning problem, where we have the acoustic vector sequence as the
input and the phoneme label sequence as the output, it becomes possible to
comprehensively learn utterance by utterance as a whole, rather than frame by
frame.
  Structured Support Vector Machine (structured SVM) was proposed to perform
ASR with structured learning previously, but limited by the linear nature of
SVM. Here we propose structured DNN to use nonlinear transformations in
multi-layers as a structured and deep learning approach. This approach was
shown to beat structured SVM in preliminary experiments on TIMIT.

negative:
Lattices are an efficient and effective method to encode ambiguity of
upstream systems in natural language processing tasks, for example to compactly
capture multiple speech recognition hypotheses, or to represent multiple
linguistic analyses. Previous work has extended recurrent neural networks to
model lattice inputs and achieved improvements in various tasks, but these
models suffer from very slow computation speeds. This paper extends the
recently proposed paradigm of self-attention to handle lattice inputs.
Self-attention is a sequence modeling technique that relates inputs to one
another by computing pairwise similarities and has gained popularity for both
its strong results and its computational efficiency. To extend such models to
handle lattices, we introduce probabilistic reachability masks that incorporate
lattice structure into the model and support lattice scores if available. We
also propose a method for adapting positional embeddings to lattice structures.
We apply the proposed model to a speech translation task and find that it
outperforms all examined baselines while being much faster to compute than
previous neural lattice models during both training and inference.

negative:
Image manipulation can be considered a special case of image generation where
the image to be produced is a modification of an existing image. Image
generation and manipulation have been, for the most part, tasks that operate on
raw pixels. However, the remarkable progress in learning rich image and object
representations has opened the way for tasks such as text-to-image or
layout-to-image generation that are mainly driven by semantics. In our work, we
address the novel problem of image manipulation from scene graphs, in which a
user can edit images by merely applying changes in the nodes or edges of a
semantic graph that is generated from the image. Our goal is to encode image
information in a given constellation and from there on generate new
constellations, such as replacing objects or even changing relationships
between objects, while respecting the semantics and style from the original
image. We introduce a spatio-semantic scene graph network that does not require
direct supervision for constellation changes or image edits. This makes it
possible to train the system from existing real-world datasets with no
additional annotation effort.

negative:
Measuring visual similarity is critical for image understanding. But what
makes two images similar? Most existing work on visual similarity assumes that
images are similar because they contain the same object instance or category.
However, the reason why images are similar is much more complex. For example,
from the perspective of category, a black dog image is similar to a white dog
image. However, in terms of color, a black dog image is more similar to a black
horse image than the white dog image. This example serves to illustrate that
visual similarity is ambiguous but can be made precise when given an explicit
contextual perspective. Based on this observation, we propose the concept of
contextual visual similarity. To be concrete, we examine the concept of
contextual visual similarity in the application domain of image search. Instead
of providing only a single image for image similarity search (\eg, Google image
search), we require three images. Given a query image, a second positive image
and a third negative image, dissimilar to the first two images, we define a
contextualized similarity search criteria. In particular, we learn feature
weights over all the feature dimensions of each image such that the distance
between the query image and the positive image is small and their distances to
the negative image are large after reweighting their features. The learned
feature weights encode the contextualized visual similarity specified by the
user and can be used for attribute specific image search. We also show the
usefulness of our contextualized similarity weighting scheme for different
tasks, such as answering visual analogy questions and unsupervised attribute
discovery.

negative:
In an automated search system, similarity is a key concept in solving a human
task. Indeed, human process is usually a natural categorization that underlies
many natural abilities such as image recovery, language comprehension, decision
making, or pattern recognition. In the image search axis, there are several
ways to measure the similarity between images in an image database, to a query
image. Image search by content is based on the similarity of the visual
characteristics of the images. The distance function used to evaluate the
similarity between images depends on the criteria of the search but also on the
representation of the characteristics of the image; this is the main idea of
the near and fuzzy sets approaches. In this article, we introduce a new
category of beta type-2 fuzzy sets for the description of image characteristics
as well as the near sets approach for image recovery. Finally, we illustrate
our work with examples of image recovery problems used in the real world.

negative:
The growing deluge of scientific publications demands text analysis tools
that can help scientists and policy-makers navigate, forecast and beneficially
guide scientific research. Recent advances in natural language understanding
driven by deep transformer networks offer new possibilities for mapping
science. Because the same surface text can take on multiple and sometimes
contradictory specialized senses across distinct research communities,
sensitivity to context is critical for infometric applications. Transformer
embedding models such as BERT capture shades of association and connotation
that vary across the different linguistic contexts of any particular word or
span of text. Here we report a procedure for encoding scientific documents with
these tools, measuring their improvement over static word embeddings in a
nearest-neighbor retrieval task. We find discriminability of contextual
representations is strongly influenced by choice of pooling strategy for
summarizing the high-dimensional network activations. Importantly, we note that
fundamentals such as domain-matched training data are more important than
state-of-the-art NLP tools. Yet state-of-the-art models did offer significant
gains. The best approach we investigated combined domain-matched pretraining,
sound pooling, and state-of-the-art deep transformer network encoders. Finally,
with the goal of leveraging contextual representations from deep encoders, we
present a range of measurements for understanding and forecasting research
communities in science.

negative:
We present a new method for image feature-extraction which is based on
representing an image by a finite-dimensional vector of distances that measure
how different the image is from a set of image prototypes. We use the recently
introduced Universal Image Distance (UID) \cite{RatsabyChesterIEEE2012} to
compare the similarity between an image and a prototype image. The advantage in
using the UID is the fact that no domain knowledge nor any image analysis need
to be done. Each image is represented by a finite dimensional feature vector
whose components are the UID values between the image and a finite set of image
prototypes from each of the feature categories. The method is automatic since
once the user selects the prototype images, the feature vectors are
automatically calculated without the need to do any image analysis. The
prototype images can be of different size, in particular, different than the
image size. Based on a collection of such cases any supervised or unsupervised
learning algorithm can be used to train and produce an image classifier or
image cluster analysis. In this paper we present the image feature-extraction
method and use it on several supervised and unsupervised learning experiments
for satellite image data.

negative:
Matching model is essential for Image-Text Retrieval framework. Existing
research usually train the model with a triplet loss and explore various
strategy to retrieve hard negative sentences in the dataset. We argue that
current retrieval-based negative sample construction approach is limited in the
scale of the dataset thus fail to identify negative sample of high difficulty
for every image. We propose our TAiloring neGative Sentences with
Discrimination and Correction (TAGS-DC) to generate synthetic sentences
automatically as negative samples. TAGS-DC is composed of masking and refilling
to generate synthetic negative sentences with higher difficulty. To keep the
difficulty during training, we mutually improve the retrieval and generation
through parameter sharing. To further utilize fine-grained semantic of mismatch
in the negative sentence, we propose two auxiliary tasks, namely word
discrimination and word correction to improve the training. In experiments, we
verify the effectiveness of our model on MS-COCO and Flickr30K compared with
current state-of-the-art models and demonstrates its robustness and
faithfulness in the further analysis. Our code is available in
https://github.com/LibertFan/TAGS.

negative:
Recent progress in pretrained Transformer-based language models has shown
great success in learning contextual representation of text. However, due to
the quadratic self-attention complexity, most of the pretrained Transformers
models can only handle relatively short text. It is still a challenge when it
comes to modeling very long documents. In this work, we propose to use a graph
attention network on top of the available pretrained Transformers model to
learn document embeddings. This graph attention network allows us to leverage
the high-level semantic structure of the document. In addition, based on our
graph document model, we design a simple contrastive learning strategy to
pretrain our models on a large amount of unlabeled corpus. Empirically, we
demonstrate the effectiveness of our approaches in document classification and
document retrieval tasks.

negative:
Are we using the right potential functions in the Conditional Random Field
models that are popular in the Vision community? Semantic segmentation and
other pixel-level labelling tasks have made significant progress recently due
to the deep learning paradigm. However, most state-of-the-art structured
prediction methods also include a random field model with a hand-crafted
Gaussian potential to model spatial priors, label consistencies and
feature-based image conditioning.
  In this paper, we challenge this view by developing a new inference and
learning framework which can learn pairwise CRF potentials restricted only by
their dependence on the image pixel values and the size of the support. Both
standard spatial and high-dimensional bilateral kernels are considered. Our
framework is based on the observation that CRF inference can be achieved via
projected gradient descent and consequently, can easily be integrated in deep
neural networks to allow for end-to-end training. It is empirically
demonstrated that such learned potentials can improve segmentation accuracy and
that certain label class interactions are indeed better modelled by a
non-Gaussian potential. In addition, we compare our inference method to the
commonly used mean-field algorithm. Our framework is evaluated on several
public benchmarks for semantic segmentation with improved performance compared
to previous state-of-the-art CNN+CRF models.

negative:
Internal learning for single-image generation is a framework, where a
generator is trained to produce novel images based on a single image. Since
these models are trained on a single image, they are limited in their scale and
application. To overcome these issues, we propose a meta-learning approach that
enables training over a collection of images, in order to model the internal
statistics of the sample image more effectively. In the presented meta-learning
approach, a single-image GAN model is generated given an input image, via a
convolutional feedforward hypernetwork $f$. This network is trained over a
dataset of images, allowing for feature sharing among different models, and for
interpolation in the space of generative models. The generated single-image
model contains a hierarchy of multiple generators and discriminators. It is
therefore required to train the meta-learner in an adversarial manner, which
requires careful design choices that we justify by a theoretical analysis. Our
results show that the models obtained are as suitable as single-image GANs for
many common image applications, significantly reduce the training time per
image without loss in performance, and introduce novel capabilities, such as
interpolation and feedforward modeling of novel images.

negative:
Similarity searching of molecular structure has been an important application
in the Chemoinformatics, especially in drug discovery. Similarity searching is
a common method used for identification of molecular structure. It involve
three main principal component of similarity searching: structure
representation; weighting scheme; and similarity coefficient. In this paper, we
introduces Weighted Tanimoto Coefficient based on weighted Euclidean distance
in order to investigate the effect of weight function on the result for
similarity searching. The Tanimoto coefficient is one of the popular similarity
coefficients used to measure the similarity between pairs of the molecule. The
most of research area found that the similarity searching is based on binary or
fingerprint data. Meanwhile, we used non-binary data and was set amphetamine
structure as a reference or targeted structure and the rest of the dataset
becomes a database structure. Throughout this study, it showed that there is
definitely gives a different result between a similarity searching with and
without weight.

negative:
Understanding images without explicit supervision has become an important
problem in computer vision. In this paper, we address image captioning by
generating language descriptions of scenes without learning from annotated
pairs of images and their captions. The core component of our approach is a
shared latent space that is structured by visual concepts. In this space, the
two modalities should be indistinguishable. A language model is first trained
to encode sentences into semantically structured embeddings. Image features
that are translated into this embedding space can be decoded into descriptions
through the same language model, similarly to sentence embeddings. This
translation is learned from weakly paired images and text using a loss robust
to noisy assignments and a conditional adversarial component. Our approach
allows to exploit large text corpora outside the annotated distributions of
image/caption data. Our experiments show that the proposed domain alignment
learns a semantically meaningful representation which outperforms previous
work.

negative:
Deep learning models have significantly improved the visual quality and
accuracy on compressive sensing recovery. In this paper, we propose an
algorithm for signal reconstruction from compressed measurements with image
priors captured by a generative model. We search and constrain on latent
variable space to make the method stable when the number of compressed
measurements is extremely limited. We show that, by exploiting certain
structures of the latent variables, the proposed method produces improved
reconstruction accuracy and preserves realistic and non-smooth features in the
image. Our algorithm achieves high computation speed by projecting between the
original signal space and the latent variable space in an alternating fashion.

negative:
Additive Manufacturing (AM) is a crucial component of the smart industry. In
this paper, we propose an automated quality grading system for the AM process
using a deep convolutional neural network (CNN) model. The CNN model is trained
offline using the images of the internal and surface defects in the
layer-by-layer deposition of materials and tested online by studying the
performance of detecting and classifying the failure in AM process at different
extruder speeds and temperatures. The model demonstrates the accuracy of 94%
and specificity of 96%, as well as above 75% in three classifier measures of
the Fscore, the sensitivity, and precision for classifying the quality of the
printing process in five grades in real-time. The proposed online model adds an
automated, consistent, and non-contact quality control signal to the AM process
that eliminates the manual inspection of parts after they are entirely built.
The quality monitoring signal can also be used by the machine to suggest
remedial actions by adjusting the parameters in real-time. The proposed quality
predictive model serves as a proof-of-concept for any type of AM machines to
produce reliable parts with fewer quality hiccups while limiting the waste of
both time and materials.

negative:
Everyone knows that thousand of words are represented by a single image. As a
result image search has become a very popular mechanism for the Web searchers.
Image search means, the search results are produced by the search engine should
be a set of images along with their Web page Unified Resource Locator. Now Web
searcher can perform two types of image search, they are Text to Image and
Image to Image search. In Text to Image search, search query should be a text.
Based on the input text data system will generate a set of images along with
their Web page URL as an output. On the other hand, in Image to Image search,
search query should be an image and based on this image system will generate a
set of images along with their Web page URL as an output. According to the
current scenarios, Text to Image search mechanism always not returns perfect
result. It matches the text data and then displays the corresponding images as
an output, which is not always perfect. To resolve this problem, Web
researchers have introduced the Image to Image search mechanism. In this paper,
we have also proposed an alternate approach of Image to Image search mechanism
using Histogram.

negative:
The objective of deep metric learning (DML) is to learn embeddings that can
capture semantic similarity and dissimilarity information among data points.
Existing pairwise or tripletwise loss functions used in DML are known to suffer
from slow convergence due to a large proportion of trivial pairs or triplets as
the model improves. To improve this, ranking-motivated structured losses are
proposed recently to incorporate multiple examples and exploit the structured
information among them. They converge faster and achieve state-of-the-art
performance. In this work, we unveil two limitations of existing
ranking-motivated structured losses and propose a novel ranked list loss to
solve both of them. First, given a query, only a fraction of data points is
incorporated to build the similarity structure. Consequently, some useful
examples are ignored and the structure is less informative. To address this, we
propose to build a set-based similarity structure by exploiting all instances
in the gallery. The learning setting can be interpreted as few-shot retrieval:
given a mini-batch, every example is iteratively used as a query, and the rest
ones compose the gallery to search, i.e., the support set in few-shot setting.
The rest examples are split into a positive set and a negative set. For every
mini-batch, the learning objective of ranked list loss is to make the query
closer to the positive set than to the negative set by a margin. Second,
previous methods aim to pull positive pairs as close as possible in the
embedding space. As a result, the intraclass data distribution tends to be
extremely compressed. In contrast, we propose to learn a hypersphere for each
class in order to preserve useful similarity structure inside it, which
functions as regularisation. Extensive experiments demonstrate the superiority
of our proposal by comparing with the state-of-the-art methods.

negative:
The Flickr30k dataset has become a standard benchmark for sentence-based
image description. This paper presents Flickr30k Entities, which augments the
158k captions from Flickr30k with 244k coreference chains, linking mentions of
the same entities across different captions for the same image, and associating
them with 276k manually annotated bounding boxes. Such annotations are
essential for continued progress in automatic image description and grounded
language understanding. They enable us to define a new benchmark for
localization of textual entity mentions in an image. We present a strong
baseline for this task that combines an image-text embedding, detectors for
common objects, a color classifier, and a bias towards selecting larger
objects. While our baseline rivals in accuracy more complex state-of-the-art
models, we show that its gains cannot be easily parlayed into improvements on
such tasks as image-sentence retrieval, thus underlining the limitations of
current methods and the need for further research.

negative:
We introduce a new task, visual sense disambiguation for verbs: given an
image and a verb, assign the correct sense of the verb, i.e., the one that
describes the action depicted in the image. Just as textual word sense
disambiguation is useful for a wide range of NLP tasks, visual sense
disambiguation can be useful for multimodal tasks such as image retrieval,
image description, and text illustration. We introduce VerSe, a new dataset
that augments existing multimodal datasets (COCO and TUHOI) with sense labels.
We propose an unsupervised algorithm based on Lesk which performs visual sense
disambiguation using textual, visual, or multimodal embeddings. We find that
textual embeddings perform well when gold-standard textual annotations (object
labels and image descriptions) are available, while multimodal embeddings
perform well on unannotated images. We also verify our findings by using the
textual and multimodal embeddings as features in a supervised setting and
analyse the performance of visual sense disambiguation task. VerSe is made
publicly available and can be downloaded at:
https://github.com/spandanagella/verse.

negative:
Two competing file formats have become the de facto standards for
distributing pre-trained word embeddings. Both are named after the most popular
pre-trained embeddings that are distributed in that format. The GloVe format is
an entirely text based format that suffers from huge file sizes and slow reads,
and the word2vec format is a smaller binary format that mixes a textual
representation of words with a binary representation of the vectors themselves.
Both formats have problems that we solve with a new format we call the Leader
format. We include a word length prefix for faster reads while maintaining the
smaller file size a binary format offers. We also created a minimalist library
to facilitate the reading and writing of various word vector formats, as well
as tools for converting pre-trained embeddings to our new Leader format.

negative:
Image decomposition is a crucial subject in the field of image processing. It
can extract salient features from the source image. We propose a new image
decomposition method based on convolutional neural network. This method can be
applied to many image processing tasks. In this paper, we apply the image
decomposition network to the image fusion task. We input infrared image and
visible light image and decompose them into three high-frequency feature images
and a low-frequency feature image respectively. The two sets of feature images
are fused using a specific fusion strategy to obtain fusion feature images.
Finally, the feature images are reconstructed to obtain the fused image.
Compared with the state-of-the-art fusion methods, this method has achieved
better performance in both subjective and objective evaluation.

negative:
The multi-decoder (MD) end-to-end speech translation model has demonstrated
high translation quality by searching for better intermediate automatic speech
recognition (ASR) decoder states as hidden intermediates (HI). It is a two-pass
decoding model decomposing the overall task into ASR and machine translation
sub-tasks. However, the decoding speed is not fast enough for real-world
applications because it conducts beam search for both sub-tasks during
inference. We propose Fast-MD, a fast MD model that generates HI by
non-autoregressive (NAR) decoding based on connectionist temporal
classification (CTC) outputs followed by an ASR decoder. We investigated two
types of NAR HI: (1) parallel HI by using an autoregressive Transformer ASR
decoder and (2) masked HI by using Mask-CTC, which combines CTC and the
conditional masked language model. To reduce a mismatch in the ASR decoder
between teacher-forcing during training and conditioning on CTC outputs during
testing, we also propose sampling CTC outputs during training. Experimental
evaluations on three corpora show that Fast-MD achieved about 2x and 4x faster
decoding speed than that of the na\"ive MD model on GPU and CPU with comparable
translation quality. Adopting the Conformer encoder and intermediate CTC loss
further boosts its quality without sacrificing decoding speed.

negative:
Neural architecture search (NAS) has become an important approach to
automatically find effective architectures. To cover all possible good
architectures, we need to search in an extremely large search space with
billions of candidate architectures. More critically, given a large search
space, we may face a very challenging issue of space explosion. However, due to
the limitation of computational resources, we can only sample a very small
proportion of the architectures, which provides insufficient information for
the training. As a result, existing methods may often produce suboptimal
architectures. To alleviate this issue, we propose a curriculum search method
that starts from a small search space and gradually incorporates the learned
knowledge to guide the search in a large space. With the proposed search
strategy, our Curriculum Neural Architecture Search (CNAS) method significantly
improves the search efficiency and finds better architectures than existing NAS
methods. Extensive experiments on CIFAR-10 and ImageNet demonstrate the
effectiveness of the proposed method.

negative:
Recently, the vulnerability of deep image classification models to
adversarial attacks has been investigated. However, such an issue has not been
thoroughly studied for image-to-image tasks that take an input image and
generate an output image (e.g., colorization, denoising, deblurring, etc.) This
paper presents comprehensive investigations into the vulnerability of deep
image-to-image models to adversarial attacks. For five popular image-to-image
tasks, 16 deep models are analyzed from various standpoints such as output
quality degradation due to attacks, transferability of adversarial examples
across different tasks, and characteristics of perturbations. We show that
unlike image classification tasks, the performance degradation on
image-to-image tasks largely differs depending on various factors, e.g., attack
methods and task objectives. In addition, we analyze the effectiveness of
conventional defense methods used for classification models in improving the
robustness of the image-to-image models.

negative:
Edit-based approaches have recently shown promising results on multiple
monolingual sequence transduction tasks. In contrast to conventional
sequence-to-sequence (Seq2Seq) models, which learn to generate text from
scratch as they are trained on parallel corpora, these methods have proven to
be much more effective since they are able to learn to make fast and accurate
transformations while leveraging powerful pre-trained language models. Inspired
by these ideas, we present TST, a simple and efficient Text Simplification
system based on sequence Tagging, leveraging pre-trained Transformer-based
encoders. Our system makes simplistic data augmentations and tweaks in training
and inference on a pre-existing system, which makes it less reliant on large
amounts of parallel training data, provides more control over the outputs and
enables faster inference speeds. Our best model achieves near state-of-the-art
performance on benchmark test datasets for the task. Since it is fully
non-autoregressive, it achieves faster inference speeds by over 11 times than
the current state-of-the-art text simplification system.

negative:
Link prediction is the task of predicting missing connections between
entities in the knowledge graph (KG). While various forms of models are
proposed for the link prediction task, most of them are designed based on a few
known relation patterns in several well-known datasets. Due to the diversity
and complexity nature of the real-world KGs, it is inherently difficult to
design a model that fits all datasets well. To address this issue, previous
work has tried to use Automated Machine Learning (AutoML) to search for the
best model for a given dataset. However, their search space is limited only to
bilinear model families. In this paper, we propose a novel Neural Architecture
Search (NAS) framework for the link prediction task. First, the embeddings of
the input triplet are refined by the Representation Search Module. Then, the
prediction score is searched within the Score Function Search Module. This
framework entails a more general search space, which enables us to take
advantage of several mainstream model families, and thus it can potentially
achieve better performance. We relax the search space to be continuous so that
the architecture can be optimized efficiently using gradient-based search
strategies. Experimental results on several benchmark datasets demonstrate the
effectiveness of our method compared with several state-of-the-art approaches.

negative:
The requirement of large amounts of annotated images has become one grand
challenge while training deep neural network models for various visual
detection and recognition tasks. This paper presents a novel image synthesis
technique that aims to generate a large amount of annotated scene text images
for training accurate and robust scene text detection and recognition models.
The proposed technique consists of three innovative designs. First, it realizes
"semantic coherent" synthesis by embedding texts at semantically sensible
regions within the background image, where the semantic coherence is achieved
by leveraging the semantic annotations of objects and image regions that have
been created in the prior semantic segmentation research. Second, it exploits
visual saliency to determine the embedding locations within each semantic
sensible region, which coincides with the fact that texts are often placed
around homogeneous regions for better visibility in scenes. Third, it designs
an adaptive text appearance model that determines the color and brightness of
embedded texts by learning from the feature of real scene text images
adaptively. The proposed technique has been evaluated over five public datasets
and the experiments show its superior performance in training accurate and
robust scene text detection and recognition models.

negative:
Image-to-image translation is a long-established and a difficult problem in
computer vision. In this paper we propose an adversarial based model for
image-to-image translation. The regular deep neural-network based methods
perform the task of image-to-image translation by comparing gram matrices and
using image segmentation which requires human intervention. Our generative
adversarial network based model works on a conditional probability approach.
This approach makes the image translation independent of any local, global and
content or style features. In our approach we use a bidirectional
reconstruction model appended with the affine transform factor that helps in
conserving the content and photorealism as compared to other models. The
advantage of using such an approach is that the image-to-image translation is
semi-supervised, independant of image segmentation and inherits the properties
of generative adversarial networks tending to produce realistic. This method
has proven to produce better results than Multimodal Unsupervised
Image-to-image translation.

negative:
We study the behavior of several black-box search algorithms used for
generating adversarial examples for natural language processing (NLP) tasks. We
perform a fine-grained analysis of three elements relevant to search: search
algorithm, search space, and search budget. When new search algorithms are
proposed in past work, the attack search space is often modified alongside the
search algorithm. Without ablation studies benchmarking the search algorithm
change with the search space held constant, one cannot tell if an increase in
attack success rate is a result of an improved search algorithm or a less
restrictive search space. Additionally, many previous studies fail to properly
consider the search algorithms' run-time cost, which is essential for
downstream tasks like adversarial training. Our experiments provide a
reproducible benchmark of search algorithms across a variety of search spaces
and query budgets to guide future research in adversarial NLP. Based on our
experiments, we recommend greedy attacks with word importance ranking when
under a time constraint or attacking long inputs, and either beam search or
particle swarm optimization otherwise. Code implementation shared via
https://github.com/QData/TextAttack-Search-Benchmark

negative:
Learning text representation is crucial for text classification and other
language related tasks. There are a diverse set of text representation networks
in the literature, and how to find the optimal one is a non-trivial problem.
Recently, the emerging Neural Architecture Search (NAS) techniques have
demonstrated good potential to solve the problem. Nevertheless, most of the
existing works of NAS focus on the search algorithms and pay little attention
to the search space. In this paper, we argue that the search space is also an
important human prior to the success of NAS in different applications. Thus, we
propose a novel search space tailored for text representation. Through
automatic search, the discovered network architecture outperforms
state-of-the-art models on various public datasets on text classification and
natural language inference tasks. Furthermore, some of the design principles
found in the automatic network agree well with human intuition.

negative:
One of the major challenges in training text-to-image generation models is
the need of a large number of high-quality image-text pairs. While image
samples are often easily accessible, the associated text descriptions typically
require careful human captioning, which is particularly time- and
cost-consuming. In this paper, we propose the first work to train text-to-image
generation models without any text data. Our method leverages the well-aligned
multi-modal semantic space of the powerful pre-trained CLIP model: the
requirement of text-conditioning is seamlessly alleviated via generating text
features from image features. Extensive experiments are conducted to illustrate
the effectiveness of the proposed method. We obtain state-of-the-art results in
the standard text-to-image generation tasks. Importantly, the proposed
language-free model outperforms most existing models trained with full
image-text pairs. Furthermore, our method can be applied in fine-tuning
pre-trained models, which saves both training time and cost in training
text-to-image generation models. Our pre-trained model obtains competitive
results in zero-shot text-to-image generation on the MS-COCO dataset, yet with
around only 1% of the model size and training data size relative to the
recently proposed large DALL-E model.

negative:
A concept for a novel CMOS image sensor suited for analog image
pre-processing is presented in this paper. As an example, an image restoration
algorithm for reducing image noise is applied as image pre-processing in the
analog domain. To supply low-latency data input for analog image preprocessing,
the proposed concept for a CMOS image sensor offers a new sensor signal
acquisition method in 2D. In comparison to image pre-processing in the digital
domain, the proposed analog image pre-processing promises an improved image
quality. Furthermore, the image noise at the stage of analog sensor signal
acquisition can be used to select the most effective restoration algorithm
applied to the analog circuit due to image processing prior to the A/D
converter.

negative:
Language model pre-training has been shown to capture a surprising amount of
world knowledge, crucial for NLP tasks such as question answering. However,
this knowledge is stored implicitly in the parameters of a neural network,
requiring ever-larger networks to cover more facts.
  To capture knowledge in a more modular and interpretable way, we augment
language model pre-training with a latent knowledge retriever, which allows the
model to retrieve and attend over documents from a large corpus such as
Wikipedia, used during pre-training, fine-tuning and inference. For the first
time, we show how to pre-train such a knowledge retriever in an unsupervised
manner, using masked language modeling as the learning signal and
backpropagating through a retrieval step that considers millions of documents.
  We demonstrate the effectiveness of Retrieval-Augmented Language Model
pre-training (REALM) by fine-tuning on the challenging task of Open-domain
Question Answering (Open-QA). We compare against state-of-the-art models for
both explicit and implicit knowledge storage on three popular Open-QA
benchmarks, and find that we outperform all previous methods by a significant
margin (4-16% absolute accuracy), while also providing qualitative benefits
such as interpretability and modularity.