title: Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm

abstract: Transformer-based pre-trained language models have significantly improved the performance of various natural language processing (NLP) tasks in the recent years. While effective and prevalent, these models are usually prohibitively large for resource-limited deployment scenarios. A thread of research has thus been working on applying network pruning techniques under the pretrain-then-finetune paradigm widely adopted in NLP. However, the existing pruning results on benchmark transformers, such as BERT, are not as remarkable as the pruning results in the literature of convolutional neural networks (CNNs). In particular, common wisdom in pruning CNN states that sparse pruning technique compresses a model more than that obtained by reducing number of channels and layers (Elsen et al., 2020; Zhu and Gupta, 2017), while existing works on sparse pruning of BERT yields inferior results than its small-dense counterparts such as TinyBERT (Jiao et al., 2020). In this work, we aim to fill this gap by studying how knowledge are transferred and lost during the pre-train, fine-tune, and pruning process, and proposing a knowledge-aware sparse pruning process that achieves significantly superior results than existing literature. We show for the first time that sparse pruning compresses a BERT model significantly more than reducing its number of channels and layers. Experiments on multiple data sets of GLUE benchmark show that our method outperforms the leading competitors with a 20-times weight/FLOPs compression and neglectable loss in prediction accuracy.


title: Self-Supervised Pillar Motion Learning for Autonomous Driving

abstract: Autonomous driving can benefit from motion behavior comprehension when interacting with diverse traffic participants in highly dynamic environments. Recently, there has been a growing interest in estimating class-agnostic motion directly from point clouds. Current motion estimation methods usually require vast amount of annotated training data from self-driving scenes. However, manually labeling point clouds is notoriously difficult, error-prone and time-consuming. In this paper, we seek to answer the research question of whether the abundant unlabeled data collections can be utilized for accurate and efficient motion learning. To this end, we propose a learning framework that leverages free supervisory signals from point clouds and paired camera images to estimate motion purely via self-supervision. Our model involves a point cloud based structural consistency augmented with probabilistic motion masking as well as a cross-sensor motion regularization to realize the desired self-supervision. Experiments reveal that our approach performs competitively to supervised methods, and achieves the state-of-the-art result when combining our self-supervised model with supervised fine-tuning.


title: The GALAH Survey: No chemical evidence of an extragalactic origin for the Nyx stream

abstract: The results from the ESA Gaia astrometric mission and deep photometric surveys have revolutionized our knowledge of the Milky Way. There are many ongoing efforts to search these data for stellar substructure to find evidence of individual accretion events that built up the Milky Way and its halo. One of these newly identified features, called Nyx, was announced as an accreted stellar stream traveling in the plane of the disk. Using a combination of elemental abundances and stellar parameters from the GALAH and APOGEE surveys, we find that the abundances of the highest likelihood Nyx members are entirely consistent with membership of the thick disk, and inconsistent with a dwarf galaxy origin. We conclude that the postulated Nyx stream is most probably a high-velocity component of the Milky Way's thick disk. With the growing availability of large data sets including kinematics, stellar parameters, and detailed abundances, the probability of detecting chance associations increases, and hence new searches for substructure require confirmation across as many data dimensions as possible.


title: Linguistic Dependencies and Statistical Dependence

abstract: Are pairs of words that tend to occur together also likely to stand in a linguistic dependency? This empirical question is motivated by a long history of literature in cognitive science, psycholinguistics, and NLP. In this work we contribute an extensive analysis of the relationship between linguistic dependencies and statistical dependence between words. Improving on previous work, we introduce the use of large pretrained language models to compute contextualized estimates of the pointwise mutual information between words (CPMI). For multiple models and languages, we extract dependency trees which maximize CPMI, and compare to gold standard linguistic dependencies. Overall, we find that CPMI dependencies achieve an unlabelled undirected attachment score of at most $\approx 0.5$. While far above chance, and consistently above a non-contextualized PMI baseline, this score is generally comparable to a simple baseline formed by connecting adjacent words. We analyze which kinds of linguistic dependencies are best captured in CPMI dependencies, and also find marked differences between the estimates of the large pretrained language models, illustrating how their different training schemes affect the type of dependencies they capture.


title: A Black-Scholes user's guide to the Bachelier model

abstract: To cope with the negative oil futures price caused by the COVID-19 recession, global commodity futures exchanges temporarily switched the option model from Black--Scholes to Bachelier in 2020. This study reviews the literature on Bachelier's pioneering option pricing model and summarizes the practical results on volatility conversion, risk management, stochastic volatility, and barrier options pricing to facilitate the model transition. In particular, using the displaced Black-Scholes model as a model family with the Black-Scholes and Bachelier models as special cases, we not only connect the two models but also present a continuous spectrum of model choices.


title: A central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model

abstract: The Benjamini-Hochberg (BH) procedure remains widely popular despite having limited theoretical guarantees in the commonly encountered scenario of correlated test statistics. Of particular concern is the possibility that the method could exhibit bursty behavior, meaning that it might typically yield no false discoveries while occasionally yielding both a large number of false discoveries and a false discovery proportion (FDP) that far exceeds its own well controlled mean. In this paper, we investigate which test statistic correlation structures lead to bursty behavior and which ones lead to well controlled FDPs. To this end, we develop a central limit theorem for the FDP in a multiple testing setup where the test statistic correlations can be either short-range or long-range as well as either weak or strong. The theorem and our simulations from a data-driven factor model suggest that the BH procedure exhibits severe burstiness when the test statistics have many strong, long-range correlations, but does not otherwise.


title: Signal Processing Challenges and Examples for {\it in-situ} Transmission Electron Microscopy

abstract: Transmission Electron Microscopy (TEM) is a powerful tool for imaging material structure and characterizing material chemistry. Recent advances in data collection technology for TEM have enabled high-volume and high-resolution data collection at a microsecond frame rate. Taking advantage of these advances in data collection rates requires the development and application of data processing tools, including image analysis, feature extraction, and streaming data processing techniques. In this paper, we highlight a few areas in materials science that have benefited from combining signal processing and statistical analysis with data collection capabilities in TEM and present a future outlook on opportunities of integrating signal processing with automated TEM data analysis.


title: RPCL: A Framework for Improving Cross-Domain Detection with Auxiliary Tasks

abstract: Cross-Domain Detection (XDD) aims to train an object detector using labeled image from a source domain but have good performance in the target domain with only unlabeled images. Existing approaches achieve this either by aligning the feature maps or the region proposals from the two domains, or by transferring the style of source images to that of target image. Contrasted with prior work, this paper provides a complementary solution to align domains by learning the same auxiliary tasks in both domains simultaneously. These auxiliary tasks push image from both domains towards shared spaces, which bridges the domain gap. Specifically, this paper proposes Rotation Prediction and Consistency Learning (PRCL), a framework complementing existing XDD methods for domain alignment by leveraging the two auxiliary tasks. The first one encourages the model to extract region proposals from foreground regions by rotating an image and predicting the rotation angle from the extracted region proposals. The second task encourages the model to be robust to changes in the image space by optimizing the model to make consistent class predictions for region proposals regardless of image perturbations. Experiments show the detection performance can be consistently and significantly enhanced by applying the two proposed tasks to existing XDD methods.


title: Rethinking Image-Scaling Attacks: The Interplay Between Vulnerabilities in Machine Learning Systems

abstract: As real-world images come in varying sizes, the machine learning model is part of a larger system that includes an upstream image scaling algorithm. In this paper, we investigate the interplay between vulnerabilities of the image scaling procedure and machine learning models in the decision-based black-box setting. We propose a novel sampling strategy to make a black-box attack exploit vulnerabilities in scaling algorithms, scaling defenses, and the final machine learning model in an end-to-end manner. Based on this scaling-aware attack, we reveal that most existing scaling defenses are ineffective under threat from downstream models. Moreover, we empirically observe that standard black-box attacks can significantly improve their performance by exploiting the vulnerable scaling procedure. We further demonstrate this problem on a commercial Image Analysis API with decision-based black-box attacks.


title: The Power of Scale for Parameter-Efficient Prompt Tuning

abstract: In this work, we explore "prompt tuning", a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signal from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3's "few-shot" learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method "closes the gap" and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant in that large models are costly to share and serve, and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed "prefix tuning" of Li and Liang (2021), and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.


title: MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

abstract: Multilingual T5 (mT5) pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text-to-text transfer Transformer with translation pairs (mT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially non-autoregressive objective for text-to-text pre-training. We evaluate the methods on eight multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed mT6 improves cross-lingual transferability over mT5.


title: Quantum transitions from superfluid to insulating phases in disordered Bose systems

abstract: By the example of Heisenberg $d$-dimensional disordered non-frustrated antiferromagnets, we discuss quantum transitions at $d\ge2$ from magnetically ordered (superfluid) to various disorder-induced insulating phases (Bose-glass, Mott-glass, etc.) in Bose systems with quenched disorder. We perform a scaling consideration as well as a discussion based on the hydrodynamic description of long-wavelength excitations and on the assumption that the ordered part of the system shows fractal properties near the transition point. We propose that the scaling ansatz for the singular part of the free energy suggested before for the transition to the Bose-glass phase is applicable also for other transitions if the quenched disorder does not produce a local imbalance in sublattices magnetizations. We show using the scaling consideration that $\eta=2-z$ and $\beta=\nu d/2$, where $\eta$, $\beta$, and $\nu$, are critical exponents of the correlation function, the order parameter, and the correlation length, respectively, and $z$ is the dynamical critical index. These relations were missed in previous analytical discussions of Bose-glass and Mott-glass phases. They signify, in particular, that $z=d/2$ for the transition to the Mott-glass phase and that the density of states of localized excitations shows a superuniversal (i.e., independent of $d$) behavior near the transitions. Being derived solely from the scaling analysis, the above relations for $\eta$ and $\beta$ are valid also for the transition to the random-singlet phase.


title: Impact of particle size on the magnetic properties of highly crystalline Yb3+ substituted Ni-Zn nanoferrites

abstract: Yb-substituted Ni-Zn ferrites have been synthesized using sol-gel auto combustion method. The structural characterization of the compositions has been performed by X-ray diffraction analysis, field emission scanning electron microscopy (FESEM), quantum design physical properties measurement system (PPMS). That ensured the formation of single phase cubic spinel structure. Crystallite and average grain size are calculated and found to decrease with increasing Yb3+ contents. Saturation magnetization and Bohr magnetic moment decrease while the coercivity increases with the increase in Yb3+ contents successfully explained by the Neels collinear two sub-lattice model and critical size effect, respectively. Critical particle size has been estimated at 6.4 nm, the transition point between single domain regime (below the critical size) and multi-domain regime (beyond the critical size). Curie temperature reduces due to the weakening of A-O-B super exchange interaction and redistribution of cations, confirmed by the M-T graph. The compositions retain ferromagnetic ordered structured below Curie temperature and above Curie temperature, it becomes paramagnetic, making them plausible candidates for high temperature magnetic device applications. The relative quality factor peak is obtained at a very high frequency, indicating the compositions could also be applicable for high frequency magnetic device applications.


title: Model Error Propagation via Learned Contraction Metrics for Safe Feedback Motion Planning of Unknown Systems

abstract: We present a method for contraction-based feedback motion planning of locally incrementally exponentially stabilizable systems with unknown dynamics that provides probabilistic safety and reachability guarantees. Given a dynamics dataset, our method learns a deep control-affine approximation of the dynamics. To find a trusted domain where this model can be used for planning, we obtain an estimate of the Lipschitz constant of the model error, which is valid with a given probability, in a region around the training data, providing a local, spatially-varying model error bound. We derive a trajectory tracking error bound for a contraction-based controller that is subjected to this model error, and then learn a controller that optimizes this tracking bound. With a given probability, we verify the correctness of the controller and tracking error bound in the trusted domain. We then use the trajectory error bound together with the trusted domain to guide a sampling-based planner to return trajectories that can be robustly tracked in execution. We show results on a 4D car, a 6D quadrotor, and a 22D deformable object manipulation task, showing our method plans safely with learned models of high-dimensional underactuated systems, while baselines that plan without considering the tracking error bound or the trusted domain can fail to stabilize the system and become unsafe.


title: Knowledge Neurons in Pretrained Transformers

abstract: Large-scale pretrained language models are surprisingly good at recalling factual knowledge presented in the training corpus. In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons. Specifically, we examine the fill-in-the-blank cloze task for BERT. Given a relational fact, we propose a knowledge attribution method to identify the neurons that express the fact. We find that the activation of such knowledge neurons is positively correlated to the expression of their corresponding facts. In our case studies, we attempt to leverage knowledge neurons to edit (such as update, and erase) specific factual knowledge without fine-tuning. Our results shed light on understanding the storage of knowledge within pretrained Transformers. The code is available at https://github.com/Hunter-DDM/knowledge-neurons.


title: OSKDet: Towards Orientation-sensitive Keypoint Localization for Rotated Object Detection

abstract: Rotated object detection is a challenging issue of computer vision field. Loss of spatial information and confusion of parametric order have been the bottleneck for rotated detection accuracy. In this paper, we propose an orientation-sensitive keypoint based rotated detector OSKDet. We adopt a set of keypoints to characterize the target and predict the keypoint heatmap on ROI to form a rotated target. By proposing the orientation-sensitive heatmap, OSKDet could learn the shape and direction of rotated target implicitly and has stronger modeling capabilities for target representation, which improves the localization accuracy and acquires high quality detection results. To extract highly effective features at border areas, we design a rotation-aware deformable convolution module. Furthermore, we explore a new keypoint reorder algorithm and feature fusion module based on the angle distribution to eliminate the confusion of keypoint order. Experimental results on several public benchmarks show the state-of-the-art performance of OSKDet. Specifically, we achieve an AP of 77.81% on DOTA, 89.91% on HRSC2016, and 97.18% on UCAS-AOD, respectively.


title: A Simple and Effective Positional Encoding for Transformers

abstract: Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent works proposed variations of positional encodings with relative position encodings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to attention layer from the input. Motivated by this, we introduce Decoupled Positional Attention for Transformers (DIET), a simple yet effective mechanism to encode position and segment information into the Transformer models. The proposed method has faster training and inference time, while achieving competitive performance on GLUE, XTREME and WMT benchmarks. We further generalize our method to long-range transformers and show performance gain.


title: FOX: Hardware-Assisted File Auditing for Direct Access NVM-Hosted Filesystems

abstract: With emerging non-volatile memories entering the mainstream market, several operating systems start to incorporate new changes and optimizations. One major OS support is the direct-access for files, which enables efficient access for files hosted in byte-addressable NVM systems. With DAX-enabled filesystems, files can be accessed directly similar to memory with typical load/store operations. Despite its efficiency, the frequently used system call of direct access is troublesome for system auditing. File system auditing is mandatory and widely used because auditing logs can help detect anomalies, suspicious file accesses, or be used as an evidence in digital forensics. However, the frequent and long-time usage of direct access call blinds the operating system or file system from tracking process operations to shared files after the initial page faults. This might results in imprecise casualty analysis and leads to false conclusion for attack detection. To remedy the tension between enabling fine-grained file system auditing and leveraging the performance of NVM-hosted file systems, we propose a novel hardware-assisted auditing scheme, FOX. FOX enables file system auditing through lightweight hardware-software changes which can monitor every read or write event for mapped files on NVM. Additionally, we propose the optimized schemes, that enable auditing flexibility for selected files/memory range. By prototyping FOX on a full system simulator, Gem5, we observe a relatively small reduced throughput and an acceptable extra writes compared to our baseline. Compared to other instrumentation-based software schemes, our scheme is low-overhead and secure.


title: Lottery Jackpots Exist in Pre-trained Models

abstract: Network pruning is an effective approach to reduce network complexity with acceptable performance compromise. Existing studies achieve the sparsity of neural networks via time-consuming weight training or complex searching on networks with expanded width, which greatly limits the applications of network pruning. In this paper, we show that high-performing and sparse sub-networks without the involvement of weight training, termed "lottery jackpots", exist in pre-trained models with unexpanded width. Furthermore, we improve the efficiency for searching lottery jackpots from two perspectives. Firstly, we observe that the sparse masks derived from many existing pruning criteria have a high overlap with the searched mask of our lottery jackpot, among which, the magnitude-based pruning results in the most similar mask with ours. Consequently, our searched lottery jackpot removes 90% weights in ResNet-50, while it easily obtains more than 70% top-1 accuracy using only 5 searching epochs on ImageNet. In compliance with this insight, we initialize our sparse mask using the magnitude-based pruning, resulting in at least 3x cost reduction on the lottery jackpot searching while achieving comparable or even better performance. Secondly, we conduct an in-depth analysis of the searching process for lottery jackpots. Our theoretical result suggests that the decrease in training loss during weight searching can be disturbed by the dependency between weights in modern networks. To mitigate this, we propose a novel short restriction method to restrict change of masks that may have potential negative impacts on the training loss. Our code is available at https://github.com/zyxxmu/lottery-jackpots.


title: Intent Features for Rich Natural Language Understanding

abstract: Complex natural language understanding modules in dialog systems have a richer understanding of user utterances, and thus are critical in providing a better user experience. However, these models are often created from scratch, for specific clients and use cases, and require the annotation of large datasets. This encourages the sharing of annotated data across multiple clients. To facilitate this we introduce the idea of intent features: domain and topic agnostic properties of intents that can be learned from the syntactic cues only, and hence can be shared. We introduce a new neural network architecture, the Global-Local model, that shows significant improvement over strong baselines for identifying these features in a deployed, multi-intent natural language understanding module, and, more generally, in a classification setting where a part of an utterance has to be classified utilizing the whole context.


title: Reconsidering CO2 emissions from Computer Vision

abstract: Climate change is a pressing issue that is currently affecting and will affect every part of our lives. It's becoming incredibly vital we, as a society, address the climate crisis as a universal effort, including those in the Computer Vision (CV) community. In this work, we analyze the total cost of CO2 emissions by breaking it into (1) the architecture creation cost and (2) the life-time evaluation cost. We show that over time, these costs are non-negligible and are having a direct impact on our future. Importantly, we conduct an ethical analysis of how the CV-community is unintentionally overlooking its own ethical AI principles by emitting this level of CO2. To address these concerns, we propose adding "enforcement" as a pillar of ethical AI and provide some recommendations for how architecture designers and broader CV community can curb the climate crisis.


title: Photon Spheres and Spherical Accretion Image of a Hairy Black Hole

abstract: In this paper, we first consider null geodesics of a class of charged, spherical and asymptotically flat hairy black holes in an Einstein-Maxwell-scalar theory with a non-minimal coupling for the scalar and electromagnetic fields. Remarkably, we show that there are two unstable circular orbits for a photon in a certain parameter regime, corresponding to two unstable photon spheres of different sizes outside the event horizon. To illustrate the optical appearance of photon spheres, we then consider a simple spherical model of optically thin accretion on the hairy black hole, and obtain the accretion image seen by a distant observer. In the single photon sphere case, only one bright ring appears in the image, and is identified as the edge of the black hole shadow. Whereas in the case with two photon spheres, there can be two concentric bright rings of different radii in the image, and the smaller one serves as the boundary of the shadow, whose radius goes to zero at the critical charge.


title: A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation

abstract: Large pretrained generative models like GPT-3 often suffer from hallucinating non-existent or incorrect content, which undermines their potential merits in real applications. Existing work usually attempts to detect these hallucinations based on a corresponding oracle reference at a sentence or document level. However ground-truth references may not be readily available for many free-form text generation applications, and sentence- or document-level detection may fail to provide the fine-grained signals that would prevent fallacious content in real time. As a first step to addressing these issues, we propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDes (HAllucination DEtection dataSet). To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations. To mitigate label imbalance during annotation, we utilize an iterative model-in-loop strategy. We conduct comprehensive data analyses and create multiple baseline models.


title: A theory of integration for Ces\`aro limits

abstract: The Ces\`aro limit - the asymptotic average of a sequence of real numbers - is an operator of fundamental importance in probability, statistics and analysis. Surprisingly, spaces of sequences with Ces\`aro limits have not previously been studied. This paper introduces spaces of such sequences, denoted $K_p(\mathcal{A})$, with the Ces\`aro limit acting as a kind of integral. The space $\mathcal{F}$ comprised of all binary sequences with a Ces\`aro limit is studied first, along with the associated functional $\nu: \mathcal{F} \rightarrow [0,1]$ mapping each such sequence to its Ces\`aro limit. It is shown that $\mathcal{F}$ can be factored to produce a monotone class on which $\nu$ induces a countably additive set function. The space $K_p(\mathcal{A})$ is then defined, and a quotient denoted $\mathcal{K}_p(\mathcal{A})$ is shown to be isometrically isomorphic, under certain conditions, to the function space $\mathcal{L}_p(\mathbb{N},\mathcal{A},\nu)$, where $\mathcal{A}$ is a field of sets isomorphic to a subset of $\mathcal{F}$, and $\nu$ is a finitely additive measure induced by the functional mentioned above. The Ces\`aro limit of an element of $K_p(\mathcal{A})$ is shown to be equal to its integral. The complete $\mathcal{L}_p(\mathbb{N},\mathcal{A},\nu)$ spaces (and by implication, the $\mathcal{K}_p(\mathcal{A})$ spaces isomorphic to them) are characterised, and a sufficient condition for these spaces to be separable is identified.


title: Broadband Omni-resonant Coherent Perfect Absorption in Graphene

abstract: Coherent perfect absorption (CPA) refers to interferometrically induced complete absorption of incident light by a partial absorber independently of its intrinsic absorption (which may be vanishingly small) or its thickness. CPA is typically realized in a resonant device, and thus cannot be achieved over a broad continuous spectrum, which thwarts its applicability to photodetectors and solar cells, for example. Here, we demonstrate broadband omni-resonant CPA by placing a thin weak absorber in a planar cavity and pre-conditioning the incident optical field by introducing judicious angular dispersion. We make use of monolayer graphene embedded in silica as the absorber and boost its optical absorption from ~1.6% to ~60% over a bandwidth of ~70 nm in the visible. Crucially, an analytical model demonstrates that placement of the graphene monolayer at a peak in the cavity standing-wave field is not necessary to achieve CPA, contrary to conventional wisdom.


title: Contextualized Query Embeddings for Conversational Search

abstract: This paper describes a compact and effective model for low-latency passage retrieval in conversational search based on learned dense representations. Prior to our work, the state-of-the-art approach uses a multi-stage pipeline comprising conversational query reformulation and information retrieval modules. Despite its effectiveness, such a pipeline often includes multiple neural models that require long inference times. In addition, independently optimizing each module ignores dependencies among them. To address these shortcomings, we propose to integrate conversational query reformulation directly into a dense retrieval model. To aid in this goal, we create a dataset with pseudo-relevance labels for conversational search to overcome the lack of training data and to explore different training strategies. We demonstrate that our model effectively rewrites conversational queries as dense representations in conversational search and open-domain question answering datasets. Finally, after observing that our model learns to adjust the $L_2$ norm of query token embeddings, we leverage this property for hybrid retrieval and to support error analysis.


title: Complexity Lower Bounds for Nonconvex-Strongly-Concave Min-Max Optimization

abstract: We provide a first-order oracle complexity lower bound for finding stationary points of min-max optimization problems where the objective function is smooth, nonconvex in the minimization variable, and strongly concave in the maximization variable. We establish a lower bound of $\Omega\left(\sqrt{\kappa}\epsilon^{-2}\right)$ for deterministic oracles, where $\epsilon$ defines the level of approximate stationarity and $\kappa$ is the condition number. Our analysis shows that the upper bound achieved in (Lin et al., 2020b) is optimal in the $\epsilon$ and $\kappa$ dependence up to logarithmic factors. For stochastic oracles, we provide a lower bound of $\Omega\left(\sqrt{\kappa}\epsilon^{-2} + \kappa^{1/3}\epsilon^{-4}\right)$. It suggests that there is a significant gap between the upper bound $\mathcal{O}(\kappa^3 \epsilon^{-4})$ in (Lin et al., 2020a) and our lower bound in the condition number dependence.


title: Size and Shape Effects in the Orbital Magnetization of TMDs Monolayers

abstract: The intrinsic orbital magnetization of a TMD monolayer is usually calculated for a plane unbounded system without mentioning the geometrical shape of samples and boundary conditions (BCs) for electron wave functions. The method of calculations includes allowing for the Berry curvature contribution also in the case when the system is described by the two-band minimal model [9]. In the present paper, we show that the geometrical and topological properties of the specimen, as well as the BCs, play an important role in the problem of magnetization even for a macroscopic specimen.


title: Simple and Efficient ways to Improve REALM

abstract: Dense retrieval has been shown to be effective for retrieving relevant documents for Open Domain QA, surpassing popular sparse retrieval methods like BM25. REALM (Guu et al., 2020) is an end-to-end dense retrieval system that relies on MLM based pretraining for improved downstream QA efficiency across multiple datasets. We study the finetuning of REALM on various QA tasks and explore the limits of various hyperparameter and supervision choices. We find that REALM was significantly undertrained when finetuning and simple improvements in the training, supervision, and inference setups can significantly benefit QA results and exceed the performance of other models published post it. Our best model, REALM++, incorporates all the best working findings and achieves significant QA accuracy improvements over baselines (~5.5% absolute accuracy) without any model design changes. Additionally, REALM++ matches the performance of large Open Domain QA models which have 3x more parameters demonstrating the efficiency of the setup.


title: Comprehensive studies of $\Upsilon$ inclusive production in $Z$ boson decay

abstract: In this paper, we present a comprehensive study of $\Upsilon$ inclusive production in $Z$ boson decay, including the first complete next-to-leading-order calculations of the color-octet (CO) contributions. With the inclusion of the newly-calculated remarkable QCD corrections, the CO processes exhibit crucially phenomenological influence on the existing predictions built on the color-singlet mechanism. We also include the exhaustive evaluations of the feed-down contributions, which remained ignored in the literature, and find them to be considerable. Summing up all the contributions, the $\mathcal{B}_{Z \to \Upsilon(nS)+X}$ still notably undershoot the data released by the L3 Collaboration.


title: PaCo: Preconditions Attributed to Commonsense Knowledge

abstract: Humans can seamlessly reason with circumstantial preconditions of commonsense knowledge. We understand that a glass is used for drinking water, unless the glass is broken or the water is toxic. Despite state-of-the-art (SOTA) language models' (LMs) impressive performance on inferring commonsense knowledge, it is unclear whether they understand the circumstantial preconditions. To address this gap, we propose a novel challenge of reasoning with circumstantial preconditions. We collect a dataset, called PaCo, consisting of 12.4 thousand preconditions of commonsense statements expressed in natural language. Based on this dataset, we create three canonical evaluation tasks and use them to examine the capability of existing LMs to understand situational preconditions. Our results reveal a 10-30% gap between machine and human performance on our tasks, which shows that reasoning with preconditions is an open challenge.


title: Nonconvex, Fully Distributed Optimization based CAV Platooning Control under Nonlinear Vehicle Dynamics

abstract: CAV platooning technology has received considerable attention in the past few years, driven by the next generation smart transportation systems. Unlike most of the existing platooning methods that focus on linear vehicle dynamics of CAVs, this paper considers nonlinear vehicle dynamics and develops fully distributed optimization based CAV platooning control schemes via the model predictive control (MPC) approach for a possibly heterogeneous CAV platoon. The nonlinear vehicle dynamics leads to several major difficulties in distributed algorithm development and control analysis and design. Specifically, the underlying MPC optimization problem is nonconvex and densely coupled. Further, the closed loop dynamics becomes a time-varying nonlinear system subject to external perturbations, making closed loop stability analysis rather complicated. To overcome these difficulties, we formulate the underlying MPC optimization problem as a locally coupled, albeit nonconvex, optimization problem and develop a sequential convex programming based fully distributed scheme for a general MPC horizon. Such a scheme can be effectively implemented for real-time computing using operator splitting methods. To analyze the closed loop stability, we apply various tools from global implicit function theorems, stability of linear time-varying systems, and Lyapunov theory for input-to-state stability to show that the closed loop system is locally input-to-state stable uniformly in all small coefficients pertaining to the nonlinear dynamics. Numerical tests on homogeneous and heterogeneous CAV platoons demonstrate the effectiveness of the proposed fully distributed schemes and CAV platooning control.


title: Tensor product weight modules for the mirror-twisted Heisenberg-Virasoro algebra

abstract: In this paper, we study irreducible weight modules with infinite dimensional weight spaces over the mirror-twisted Heisenberg-Virasoro algebra $\mathcal{D}$. More precisely, the necessary and sufficient conditions for the tensor products of irreducible highest weight modules and irreducible modules of intermediates series over $\mathcal{D}$ to be irreducible are determined by using "shifting technique". This leads to a family of new irreducible weight modules over $\mathcal{D}$. Then we obtain that any two such tensor products are isomorphic if and only if the corresponding highest weight modules and modules of intermediate series are isomorphic respectively. Also we discuss submodules of the tensor product module when it is not irreducible.


title: Non-weight modules over the mirror Heisenberg-Virasoro algebra

abstract: In this paper, we study irreducible non-weight modules over the mirror Heisenberg-Virasoro algebra $\mathcal{D}$, including Whittaker modules, $\mathcal{U}(\mathbb{C} d_0)$-free modules, and their tensor products. More precisely, we give the necessary and sufficient conditions for the Whittaker modules to be irreducible. We determine all $\mathcal{D}$-module structures on $\mathcal{U}(\mathbb{C} d_0)$, and find the necessary and sufficient conditions for these modules to be irreducible. At last we determine the necessary and sufficient conditions for the tensor products of Whittaker modules and $\mathcal{U}(\mathbb{C} d_0)$-free modules to be irreducible, and obtain that any two such tensor products are isomorphic if and only if the corresponding Whittaker modules and $\mathcal{U}(\mathbb{C} d_0)$-free modules are isomorphic. These lead to many new irreducible non-weight modules over $\mathcal{D}$.


title: Deep Latent Emotion Network for Multi-Task Learning

abstract: Feed recommendation models are widely adopted by numerous feed platforms to encourage users to explore the contents they are interested in. However, most of the current research simply focus on targeting user's preference and lack in-depth study of avoiding objectionable contents to be frequently recommended, which is a common reason that let user detest. To address this issue, we propose a Deep Latent Emotion Network (DLEN) model to extract latent probability of a user preferring a feed by modeling multiple targets with semi-supervised learning. With this method, the conflicts of different targets are successfully reduced in the training phase, which improves the training accuracy of each target effectively. Besides, by adding this latent state of user emotion to multi-target fusion, the model is capable of decreasing the probability to recommend objectionable contents to improve user retention and stay time during online testing phase. DLEN is deployed on a real-world multi-task feed recommendation scenario of Tencent QQ-Small-World with a dataset containing over a billion samples, and it exhibits a significant performance advantage over the SOTA MTL model in offline evaluation, together with a considerable increase by 3.02% in view-count and 2.63% in user stay-time in production. Complementary offline experiments of DLEN model on a public dataset also repeat improvements in various scenarios. At present, DLEN model has been successfully deployed in Tencent's feed recommendation system.


title: Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses

abstract: Most segmentation losses are arguably variants of the Cross-Entropy (CE) or Dice losses. On the surface, these two categories of losses seem unrelated, and there is no clear consensus as to which category is a better choice, with varying performances for each across different benchmarks and applications. Furthermore, it is widely argued within the medical-imaging community that Dice and CE are complementary, which has motivated the use of compound CE-Dice losses. In this work, we provide a theoretical analysis, which shows that CE and Dice share a much deeper connection than previously thought. First, we show that, from a constrained-optimization perspective, they both decompose into two components, i.e., a similar ground-truth matching term, which pushes the predicted foreground regions towards the ground-truth, and a region-size penalty term imposing different biases on the size of the predicted regions. Then, we provide bound relationships and an information-theoretic analysis, which uncover hidden region-size biases: Dice has an intrinsic bias towards specific extremely imbalanced solutions, whereas CE implicitly encourages the ground-truth region proportions. Our theoretical results explain the wide experimental evidence in the medical-imaging literature, whereby Dice losses bring improvements for imbalanced segmentation. Based on our theoretical analysis, we propose a principled and simple solution, which enables to control explicitly the region-size bias. The proposed method integrates CE with explicit terms based on L1 or the KL divergence, which encourage segmenting region proportions to match target class proportions, thereby mitigating class imbalance but without losing generality. Comprehensive experiments and ablation studies over different losses and applications validate our theoretical analysis, as well as the effectiveness of explicit and simple region-size terms.


title: CLIPScore: A Reference-free Evaluation Metric for Image Captioning

abstract: Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality. In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE. Information gain experiments demonstrate that CLIPScore, with its tight focus on image-text compatibility, is complementary to existing reference-based metrics that emphasize text-text similarities. Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation. Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based metrics, e.g., news captions that require richer contextual knowledge.


title: On-Surface Synthesis of Graphene Nanoribbons on Two-Dimensional Rare Earth-Gold Intermetallic Compounds

abstract: Here, we demonstrate two reliable routes for the fabrication of armchair-edge graphene nanoribbons (GNRs) on TbAu2/Au(111), belonging to a class of two-dimensional ferromagnetic rare earth-gold intermetallic compounds. On-surface synthesis directly on TbAu2 leads to the formation of GNRs, which are short and interconnected with each other. In contrast, the intercalation approach - on-surface synthesis of GNRs directly on Au(111) followed by rare earth intercalation - yields GNRs on TbAu2/Au(111), where both the ribbons and TbAu2 are of high quality comparable with those directly grown on clean Au(111). Besides, the as-grown ribbons retain the same band gap while changing from p-doping to weak n-doping mainly due to a change in the work function of the substrate after the rare earth intercalation. The intercalation approach might also be employed to fabricate other types of GNRs on various rare earth intermetallic compounds, providing platforms to tailor the electronic and magnetic properties of GNRs on magnetic substrates.


title: The Hamilton-Jacobi analysis for higher order Maxwell-Chern-Simons gauge theory

abstract: By using the Hamilton-Jacobi [$HJ$] framework the higher-order Maxwell-Chern-Simons theory is analyzed. The complete set of $HJ$ Hamiltonians and a generalized $HJ$ differential are reported, from which all symmetries of the theory are identified. In addition, we complete our study by performing the higher order Gitman-Lyakhovich-Tyutin [$GLT$] framework and compare the results of both formalisms.


title: Embedding-Enhanced Giza++: Improving Alignment in Low- and High- Resource Scenarios Using Embedding Space Geometry

abstract: A popular natural language processing task decades ago, word alignment has been dominated until recently by GIZA++, a statistical method based on the 30-year-old IBM models. New methods that outperform GIZA++ primarily rely on large machine translation models, massively multilingual language models, or supervision from GIZA++ alignments itself. We introduce Embedding-Enhanced GIZA++, and outperform GIZA++ without any of the aforementioned factors. Taking advantage of monolingual embedding spaces of source and target language only, we exceed GIZA++'s performance in every tested scenario for three languages pairs. In the lowest-resource setting, we outperform GIZA++ by 8.5, 10.9, and 12 AER for Ro-En, De-En, and En-Fr, respectively. We release our code at https://github.com/kellymarchisio/ee-giza.


title: New canonical analysis for higher order topologically massive gravity

abstract: A detailed Gitman-Lyakhovich-Tyutin analysis for higher-order topologically massive gravity is performed. The full structure of the constraints, the counting of physical degrees of freedom, and the Dirac algebra among the constraints are reported. Moreover, our analysis presents a new structure of the constraints and we compare our results with those reported in the literature where a standard Ostrogradski framework was developed.


title: News Meets Microblog: Hashtag Annotation via Retriever-Generator

abstract: Hashtag annotation for microblog posts has been recently formulated as a sequence generation problem to handle emerging hashtags that are unseen in the training set. The state-of-the-art method leverages conversations initiated by posts to enrich contextual information for the short posts. However, it is unrealistic to assume the existence of conversations before the hashtag annotation itself. Therefore, we propose to leverage news articles published before the microblog post to generate hashtags following a Retriever-Generator framework. Extensive experiments on English Twitter datasets demonstrate superior performance and significant advantages of leveraging news articles to generate hashtags.


title: Extract, Denoise and Enforce: Evaluating and Improving Concept Preservation for Text-to-Text Generation

abstract: Prior studies on text-to-text generation typically assume that the model could figure out what to attend to in the input and what to include in the output via seq2seq learning, with only the parallel training data and no additional guidance. However, it remains unclear whether current models can preserve important concepts in the source input, as seq2seq learning does not have explicit focus on the concepts and commonly used evaluation metrics also treat concepts equally important as other tokens. In this paper, we present a systematic analysis that studies whether current seq2seq models, especially pre-trained language models, are good enough for preserving important input concepts and to what extent explicitly guiding generation with the concepts as lexical constraints is beneficial. We answer the above questions by conducting extensive analytical experiments on four representative text-to-text generation tasks. Based on the observations, we then propose a simple yet effective framework to automatically extract, denoise, and enforce important input concepts as lexical constraints. This new method performs comparably or better than its unconstrained counterpart on automatic metrics, demonstrates higher coverage for concept preservation, and receives better ratings in the human evaluation. Our code is available at https://github.com/morningmoni/EDE.


title: The Black hole Black string phase transition in Einstein-Gauss-Bonnet gravity

abstract: We investigate the presence of a black hole black string phase transition in Einstein Gauss Bonnet (EGB) gravity in the large dimension limit. The merger point is the static spacetime connecting the black string phase with the black hole phase. We consider several ranges of the Gauss-Bonnet parameter. We find that there is a range when the Gauss-Bonnet corrections are subordinate to the Einstein gravity terms in the large dimension limit, and yet the merger point geometry does not approach a black hole away from the neck. We cannot rule out a topology changing phase transition as argued by Kol. However as the merger point geometry does not approach the black hole geometry asymptotically it is not obvious that the transition is directly to a black hole phase. We also demonstrate that for another range of the Gauss-Bonnet parameter, the merger point geometry approaches the black hole geometry asymptotically when a certain parameter depending on the Gauss-Bonnet parameter $\alpha$ and on the parameters in the Einstein-Gauss-Bonnet black hole metric is small enough.


title: AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages

abstract: Pretrained multilingual models are able to perform cross-lingual transfer in a zero-shot setting, even for languages unseen during pretraining. However, prior work evaluating performance on unseen languages has largely been limited to low-level, syntactic tasks, and it remains unclear if zero-shot learning of high-level, semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, an extension of XNLI (Conneau et al., 2018) to 10 indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. Additionally, we explore model adaptation via continued pretraining and provide an analysis of the dataset by considering hypothesis-only models. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.62%. Continued pretraining offers improvements, with an average accuracy of 44.05%. Surprisingly, training on poorly translated data by far outperforms all other methods with an accuracy of 48.72%.


title: GooAQ: Open Question Answering with Diverse Answer Types

abstract: While day-to-day questions come with a variety of answer types, the current question-answering (QA) literature has failed to adequately address the answer diversity of questions. To this end, we present GooAQ, a large-scale dataset with a variety of answer types. This dataset contains over 5 million questions and 3 million answers collected from Google. GooAQ questions are collected semi-automatically from the Google search engine using its autocomplete feature. This results in naturalistic questions of practical interest that are nonetheless short and expressed using simple language. GooAQ answers are mined from Google's responses to our collected questions, specifically from the answer boxes in the search results. This yields a rich space of answer types, containing both textual answers (short and long) as well as more structured ones such as collections. We benchmarkT5 models on GooAQ and observe that: (a) in line with recent work, LM's strong performance on GooAQ's short-answer questions heavily benefit from annotated data; however, (b) their quality in generating coherent and accurate responses for questions requiring long responses (such as 'how' and 'why' questions) is less reliant on observing annotated data and mainly supported by their pre-training. We release GooAQ to facilitate further research on improving QA with diverse response types.


title: Revealing Persona Biases in Dialogue Systems

abstract: Dialogue systems in the form of chatbots and personal assistants are being increasingly integrated into people's lives. Modern dialogue systems may consider adopting anthropomorphic personas, mimicking societal demographic groups to appear more approachable and trustworthy to users. However, the adoption of a persona can result in the adoption of biases. In this paper, we present the first large-scale study on persona biases in dialogue systems and conduct analyses on personas of different social classes, sexual orientations, races, and genders. We define persona biases as harmful differences in responses (e.g., varying levels of offensiveness, agreement with harmful statements) generated from adopting different demographic personas. Furthermore, we introduce an open-source framework, UnitPersonaBias, to explore and aggregate persona biases in dialogue systems. By analyzing the Blender and DialoGPT dialogue systems, we observe that adopting personas can actually decrease harmful responses, compared to not using any personas. Additionally, we find that persona choices can affect the degree of harms in generated responses and thus should be systematically evaluated before deployment. We also analyze how personas can result in different amounts of harm towards specific demographics.


title: Unsupervised Deep Keyphrase Generation

abstract: Keyphrase generation aims to summarize long documents with a collection of salient phrases. Deep neural models have demonstrated a remarkable success in this task, capable of predicting keyphrases that are even absent from a document. However, such abstractiveness is acquired at the expense of a substantial amount of annotated data. In this paper, we present a novel method for keyphrase generation, AutoKeyGen, without the supervision of any human annotation. Motivated by the observation that an absent keyphrase in one document can appear in other places, in whole or in part, we first construct a phrase bank by pooling all phrases in a corpus. With this phrase bank, we then draw candidate absent keyphrases for each document through a partial matching process. To rank both types of candidates, we combine their lexical- and semantic-level similarities to the input document. Moreover, we utilize these top-ranked candidates as to train a deep generative model for more absent keyphrases. Extensive experiments demonstrate that AutoKeyGen outperforms all unsupervised baselines and can even beat strong supervised methods in certain cases.


title: Mpemba effect in an anisotropically driven granular gas

abstract: We demonstrate the existence, as well as determine the conditions, of a Mpemba effect - a counterintuitive phenomenon where a hotter system equilibrates faster than a cooler system when quenched to a cold temperature - in anisotropically driven granular gases. In contrast to earlier studies of Mpemba effect in granular systems, the initial states are stationary, making it a suitable system to experimentally study the effect. Our theoretical predictions for the regular, inverse and strong Mpemba effects agree well with results of event-driven molecular dynamics simulations of hard discs.


title: Can NLI Models Verify QA Systems' Predictions?

abstract: To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just "good enough" in the context of imperfect QA datasets. We explore the use of natural language inference (NLI) as a way to achieve this goal, as NLI inherently requires the premise (document context) to contain all necessary information to support the hypothesis (proposed answer to the question). We leverage large pre-trained models and recent prior datasets to construct powerful question converter and decontextualization modules, which can reformulate QA instances as premise-hypothesis pairs with very high reliability. Then, by combining standard NLI datasets with NLI examples automatically derived from QA training data, we can train NLI models to judge the correctness of QA models' proposed answers. We show that our NLI approach can generally improve the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. Careful manual analysis over the predictions of our NLI model shows that it can further identify cases where the QA model produces the right answer for the wrong reason, or where the answer cannot be verified as addressing all aspects of the question.


title: Application of Computer Vision and Machine Learning for Digitized Herbarium Specimens: A Systematic Literature Review

abstract: Herbarium contains treasures of millions of specimens which have been preserved for several years for scientific studies. To speed up more scientific discoveries, a digitization of these specimens is currently on going to facilitate easy access and sharing of its data to a wider scientific community. Online digital repositories such as IDigBio and GBIF have already accumulated millions of specimen images yet to be explored. This presents a perfect time to automate and speed up more novel discoveries using machine learning and computer vision. In this study, a thorough analysis and comparison of more than 50 peer-reviewed studies which focus on application of computer vision and machine learning techniques to digitized herbarium specimen have been examined. The study categorizes different techniques and applications which have been commonly used and it also highlights existing challenges together with their possible solutions. It is our hope that the outcome of this study will serve as a strong foundation for beginners of the relevant field and will also shed more light for both computer science and ecology experts.


title: Can Hubbard model resist electric current?

abstract: It is claimed by a recent quantum Monte Carlo simulation that the linear-in-temperature DC resistivity observed in the high-$T_{c}$ cuprate superconductors can be reproduced in the pure two dimensional Hubbard model\cite{Huang}. Here we show perturbatively that such a translational invariant electronic model can not support a steady state current in the presence of a uniform electric field at any finite temperature. Instead, the Hubbard model is perfectly conducting in the linear response regime and will undergo Bloch oscillation at finite electric field for any finite temperature. Nevertheless, the quantum Monte Carlo simulation can provide us the key information on the temperature dependence of the Drude weight, a quantity of central importance in the holographic description of the transport properties of the strange metal phase.


title: Barrier-Free Large-Scale Sparse Tensor Accelerator (BARISTA) For Convolutional Neural Networks

abstract: Convolutional neural networks (CNNs) are emerging as powerful tools for visual recognition. Recent architecture proposals for sparse CNNs exploit zeros in the feature maps and filters for performance and energy without losing accuracy. Sparse architectures that exploit two-sided sparsity in both feature maps and filters have been studied only at small scales (e.g., 1K multiply-accumulate(MAC) units). However, to realize their advantages in full, the sparse architectures have to be scaled up to levels of the dense architectures (e.g., 32K MACs in the TPU). Such scaling is challenging since achieving reuse through broadcasts incurs implicit barrier cost raises the inter-related issues of load imbalance, buffering, and on-chip bandwidth demand. SparTen, a previous scheme, addresses one aspect of load balancing but not other aspects, nor the other issues of buffering and bandwidth. To that end, we propose the barrier-free large-scale sparse tensor accelerator (BARISTA). BARISTA (1) is the first architecture for scaling up sparse CNN accelerators; (2) reduces on-chip bandwidth demand by telescoping request-combining the input map requests and snarfing the filter requests; (3) reduces buffering via basic buffer sharing and avoids the ensuing barriers between consecutive input maps by coloring the output buffers; (4) load balances intra-filter work via dynamic round-robin work assignment; and (5) employs hierarchical buffering which achieves high cache bandwidth via a few, wide, shared buffers and low buffering via narrower, private buffers at the compute. Our simulations show that, on average, barista performs 5.4x, 2.2x, 1.7x, 2.5x better than a dense, a one-sided, a naively-scaled two-sided, and an iso-area two-sided architecture, respectively. Using 45-nm technology, ASIC synthesis of our RTL design for four clusters of 8K MACs at 1 GHz clock speed, reports 213 mm$^2$ area and 170 W power.


title: Learning with Instance Bundles for Reading Comprehension

abstract: When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other. However, closely related questions and their corresponding answers are not independent, and leveraging these relationships could provide a strong supervision signal to a model. Drawing on ideas from contrastive estimation, we introduce several new supervision techniques that compare question-answer scores across multiple related instances. Specifically, we normalize these scores across various neighborhoods of closely contrasting questions and/or answers, adding another cross entropy loss term that is used in addition to traditional maximum likelihood estimation. Our techniques require bundles of related question-answer pairs, which we can either mine from within existing data or create using various automated heuristics. We empirically demonstrate the effectiveness of training with instance bundles on two datasets -- HotpotQA and ROPES -- showing up to 11% absolute gains in accuracy.


title: Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence

abstract: Areas under ROC (AUROC) and precision-recall curves (AUPRC) are common metrics for evaluating classification performance for imbalanced problems. Compared with AUROC, AUPRC is a more appropriate metric for highly imbalanced datasets. While stochastic optimization of AUROC has been studied extensively, principled stochastic optimization of AUPRC has been rarely explored. In this work, we propose a principled technical method to optimize AUPRC for deep learning. Our approach is based on maximizing the averaged precision (AP), which is an unbiased point estimator of AUPRC. We cast the objective into a sum of {\it dependent compositional functions} with inner functions dependent on random variables of the outer level. We propose efficient adaptive and non-adaptive stochastic algorithms named SOAP with {\it provable convergence guarantee under mild conditions} by leveraging recent advances in stochastic compositional optimization. Extensive experimental results on image and graph datasets demonstrate that our proposed method outperforms prior methods on imbalanced problems in terms of AUPRC. To the best of our knowledge, our work represents the first attempt to optimize AUPRC with provable convergence. The SOAP has been implemented in the libAUC library at~\url{https://libauc.org/}.


title: Low-Rank Subspaces for Unsupervised Entity Linking

abstract: Entity linking is an important problem with many applications. Most previous solutions were designed for settings where annotated training data is available, which is, however, not the case in numerous domains. We propose a light-weight and scalable entity linking method, Eigenthemes, that relies solely on the availability of entity names and a referent knowledge base. Eigenthemes exploits the fact that the entities that are truly mentioned in a document (the "gold entities") tend to form a semantically dense subset of the set of all candidate entities in the document. Geometrically speaking, when representing entities as vectors via some given embedding, the gold entities tend to lie in a low-rank subspace of the full embedding space. Eigenthemes identifies this subspace using the singular value decomposition and scores candidate entities according to their proximity to the subspace. On the empirical front, we introduce multiple strong baselines that compare favorably to (and sometimes even outperform) the existing state of the art. Extensive experiments on benchmark datasets from a variety of real-world domains showcase the effectiveness of our approach.


title: Well-posedness and singularity formation for inviscid Keller-Segel-fluid system of consumption type

abstract: We consider the Keller-Segel system of consumption type coupled with an incompressible fluid equation. The system describes the dynamics of oxygen and bacteria densities evolving within a fluid. We establish local well-posedness of the system in Sobolev spaces for partially inviscid and fully inviscid cases. In the latter, additional assumptions on the initial data are required when either the oxygen or bacteria density touches zero. Even though the oxygen density satisfies a maximum principle due to consumption, we prove finite time blow-up of its $C^{2}$--norm with certain initial data.


title: Continuity-Discrimination Convolutional Neural Network for Visual Object Tracking

abstract: This paper proposes a novel model, named Continuity-Discrimination Convolutional Neural Network (CD-CNN), for visual object tracking. Existing state-of-the-art tracking methods do not deal with temporal relationship in video sequences, which leads to imperfect feature representations. To address this problem, CD-CNN models temporal appearance continuity based on the idea of temporal slowness. Mathematically, we prove that, by introducing temporal appearance continuity into tracking, the upper bound of target appearance representation error can be sufficiently small with high probability. Further, in order to alleviate inaccurate target localization and drifting, we propose a novel notion, object-centroid, to characterize not only objectness but also the relative position of the target within a given patch. Both temporal appearance continuity and object-centroid are jointly learned during offline training and then transferred for online tracking. We evaluate our tracker through extensive experiments on two challenging benchmarks and show its competitive tracking performance compared with state-of-the-art trackers.


title: On the $\Phi$-Stability and Related Conjectures

abstract: Given a convex function $\Phi:[0,1]\to\mathbb{R}$ and the mean $\mathbb{E}f(\mathbf{X})=a\in[0,1]$, which Boolean function $f$ maximizes the $\Phi$-stability $\mathbb{E}[\Phi(T_{\rho}f(\mathbf{X}))]$ of $f$? Here $\mathbf{X}$ is a random vector uniformly distributed on the discrete cube $\{-1,1\}^{n}$ and $T_{\rho}$ is the Bonami-Beckner operator. Special cases of this problem include the (symmetric and asymmetric) $\alpha$-stability problems and the ``Most Informative Boolean Function'' problem. In this paper, we provide several upper bounds for the maximal $\Phi$-stability. When specializing $\Phi$ to some particular forms, by these upper bounds, we partially resolve Mossel and O'Donnell's conjecture on $\alpha$-stability with $\alpha>2$, Li and M\'edard's conjecture on $\alpha$-stability with $1<\alpha<2$, and Courtade and Kumar's conjecture on the ``Most Informative Boolean Function'' which corresponds to a conjecture on $\alpha$-stability with $\alpha=1$. Our proofs are based on discrete Fourier analysis, optimization theory, and improvements of the Friedgut--Kalai--Naor (FKN) theorem. Our improvements of the FKN theorem are sharp or asymptotically sharp for certain cases.


title: CEAR: Cross-Entity Aware Reranker for Knowledge Base Completion

abstract: Pre-trained language models (LMs) like BERT have shown to store factual knowledge about the world. This knowledge can be used to augment the information present in Knowledge Bases, which tend to be incomplete. However, prior attempts at using BERT for task of Knowledge Base Completion (KBC) resulted in performance worse than embedding based techniques that rely only on the graph structure. In this work we develop a novel model, Cross-Entity Aware Reranker (CEAR), that uses BERT to re-rank the output of existing KBC models with cross-entity attention. Unlike prior work that scores each entity independently, CEAR uses BERT to score the entities together, which is effective for exploiting its factual knowledge. CEAR achieves a new state of art for the OLPBench dataset.


title: Go Forth and Prosper: Language Modeling with Ancient Textual History

abstract: We introduce a technique for improving document-level language models (LM) by leveraging "ancient history": text that is outside the LM's current context window. We learn an auxiliary function to select spans from the ancient history which can help the LM to predict future text. The selected text spans are then copied directly into the LM's context window, replacing less predictive spans. This method can improve perplexity of pretrained LMs with no updates to the LM's own parameters. We further observe that an auxiliary function trained in a specific textual domain like Wikipedia will also work in a substantially different domain such as scientific publications. With this technique we see a 7 percent perplexity reduction on Wikipedia articles, and a 12 percent perplexity reduction on scientific texts.


title: Rough Sets in Graphs Using Similarity Relations

abstract: In this paper, we use theory of rough set to study graphs using the concept of orbits. We investigate the indiscernibility partitions and approximations of graphs induced by orbits of graphs. We also study rough membership functions, essential sets, discernibility matrix and their relationships for graphs.


title: Generative Context Pair Selection for Multi-hop Question Answering

abstract: Compositional reasoning tasks like multi-hop question answering, require making latent decisions to get the final answer, given a question. However, crowdsourced datasets often capture only a slice of the underlying task distribution, which can induce unanticipated biases in models performing compositional reasoning. Furthermore, discriminatively trained models exploit such biases to get a better held-out performance, without learning the right way to reason, as they do not necessitate paying attention to the question representation (conditioning variable) in its entirety, to estimate the answer likelihood. In this work, we propose a generative context selection model for multi-hop question answering that reasons about how the given question could have been generated given a context pair. While being comparable to the state-of-the-art answering performance, our proposed generative passage selection model has a better performance (4.9% higher than baseline) on adversarial held-out set which tests robustness of model's multi-hop reasoning capabilities.


title: Order Integrals

abstract: We define an integral of real-valued functions with respect to a measure that takes its values in the extended positive cone of a partially ordered vector space $E$. The monotone convergence theorem, Fatou's lemma, and the dominated convergence theorem are established; the analogues of the classical ${\mathcal L}^1$- and ${\mathrm L}^1$-spaces are investigated. The results extend earlier work by Wright and specialise to those for the Lebesgue integral when $E$ equals the real numbers. The hypothesis on $E$ that is needed for the definition of the integral and for the monotone convergence theorem to hold ($\sigma$-monotone completeness) is a rather mild one. It is satisfied, for example, by the space of regular operators between a directed partially ordered vector space and a $\sigma$-monotone complete partially ordered vector space, and by every JBW-algebra. Fatou's lemma and the dominated convergence theorem hold for every $\sigma$-Dedekind complete space. When $E$ consists of the regular operators on a Banach lattice with an order continuous norm, or when it consists of the self-adjoint elements of a strongly closed complex linear subspace of the bounded operators on a complex Hilbert space, then the finite measures as in the current paper are precisely the strongly $\sigma$-additive positive operator-valued measures. When $E$ is a partially ordered Banach space with a closed positive cone, then every positive vector measure is a measure in our sense, but not conversely. Even when a measure falls into both categories, the domain of the integral as defined in this paper can properly contain that of any reasonably defined integral with respect to the vector measure using Banach space methods.


title: Emergence of the Born rule in strongly-driven dissipative systems

abstract: To understand the dynamical origin of the measurement in quantum mechanics, several models have been put forward which have a quantum system coupled to an apparatus. The system and the apparatus evolve in time and the Born rule for the system to be in various eigenstates of the observable is naturally obtained. In this work, we show that the effect of the drive-induced dissipation in such a system can lead to the Born rule, even if there is no separate apparatus. The applied drive needs to be much stronger than the system-environment coupling. In this condition, we show that the dynamics of a driven-dissipative system could be reduced to a Milburn-like form, using a recently-proposed fluctuation-regulated quantum master equation [A. Chakrabarti and R. Bhattacharyya, Phys. Rev. A 97, 063837 (2018)]. The system evolves irreversibly under the action of the first-order effect of the drive and the drive-induced dissipation. The resulting mixed state is identical to that obtained by using the Born rule.


title: Multi-objective Feature Selection with Missing Data in Classification

abstract: Feature selection (FS) is an important research topic in machine learning. Usually, FS is modelled as a+ bi-objective optimization problem whose objectives are: 1) classification accuracy; 2) number of features. One of the main issues in real-world applications is missing data. Databases with missing data are likely to be unreliable. Thus, FS performed on a data set missing some data is also unreliable. In order to directly control this issue plaguing the field, we propose in this study a novel modelling of FS: we include reliability as the third objective of the problem. In order to address the modified problem, we propose the application of the non-dominated sorting genetic algorithm-III (NSGA-III). We selected six incomplete data sets from the University of California Irvine (UCI) machine learning repository. We used the mean imputation method to deal with the missing data. In the experiments, k-nearest neighbors (K-NN) is used as the classifier to evaluate the feature subsets. Experimental results show that the proposed three-objective model coupled with NSGA-III efficiently addresses the FS problem for the six data sets included in this study.


title: Submanifolds in Koszul-Vinberg geometry

abstract: A Koszul-Vinberg manifold is a manifold $M$ endowed with a pair $(\nabla,h)$ where $\nabla$ is a flat connection and $h$ is a symmetric bivector field satisfying a generalized Codazzi equation. The geometry of such manifolds could be seen as a type of bridge between Poisson geometry and pseudo-Riemannian geometry, as has been highlighted in our previous article [\textit{Contravariant Pseudo-Hessian manifolds and their associated Poisson structures}. \rm{Differential Geometry and its Applications} (2020)]. Our objective here will be to pursue our study by focusing in this setting on submanifolds by taking into account some developments in the theory of Poisson submanifolds.


title: Exact Matrix Elements of the Field Operator in the Thermodynamic Limit of the Lieb-Liniger Model

abstract: We study a matrix element of the field operator in the Lieb-Liniger model using the Bethe ansatz technique coupled with a functional approach to compute Slavnov determinants. We obtain the matrix element exactly in the thermodynamic limit for any coupling constant $c$, and compare our results to known semiclassics at the limit $c\to0.$

categories: cond-mat.stat-mech hep-th math-ph math.MP


title: Looking at the NANOGrav Signal Through the Anthropic Window of Axion-Like Particles

abstract: We explore the inflationary dynamics leading to formation of closed domain walls in course of evolution of an axion like particle (ALP) field whose Peccei-Quinn-like phase transition occurred well before inflationary epoch. Evolving after inflation, the domain walls may leave their imprint in stochastic gravitational waves background, in the frequency range accessible for the pulsar timing array measurements. We derive the characteristic strain power spectrum produced by the distribution of the closed domain walls and relate it with the recently reported NANOGrav signal excess. We found that the slope of the frequency dependence of the characteristic strain spectrum generated by the domain walls is very well centered inside the range of the slopes in the signal reported by the NANOGrav. Analyzing the inflationary dynamics of the ALP field, in consistency with the isocurvature constraint, we revealed those combinations of the parameters where the signal from the inflationary induced ALPs domain walls could saturate the amplitude of the NANOGrav excess. The evolution of big enough closed domain walls may incur in formation of wormholes with the walls escaping into baby universes. We studied the conditions, when closed walls escaped into baby universes could leave a detectable imprint in the stochastic gravitational waves background.


title: Load-Balancing Succinct B Trees

abstract: We propose a B tree representation storing $n$ keys, each of $k$ bits, in either (a) $nk + O(nk / \lg n)$ bits or (b) $nk + O(nk \lg \lg n/ \lg n)$ bits of space supporting all B tree operations in either (a) $O(\lg n )$ time or (b) $O(\lg n / \lg \lg n)$ time, respectively. We can augment each node with an aggregate value such as the minimum value within its subtree, and maintain these aggregate values within the same space and time complexities. Finally, we give the sparse suffix tree as an application, and present a linear-time algorithm computing the sparse longest common prefix array from the suffix AVL tree of Irving et al. [JDA'2003].


title: The Fulton-MacPherson compactification is not a Mori dream space

abstract: We show that the Fulton-MacPherson compactification of the configuration space of $n$ distinct labeled points in certain varieties of arbitrary dimension $d$, including projective space, is not a Mori dream space for $n$ larger than $d+8$.


title: One-shot quantum state redistribution and quantum Markov chains

abstract: We revisit the task of quantum state redistribution in the one-shot setting, and design a protocol for this task with communication cost in terms of a measure of distance from quantum Markov chains. More precisely, the distance is defined in terms of quantum max-relative entropy and quantum hypothesis testing entropy. Our result is the first to operationally connect quantum state redistribution and quantum Markov chains, and can be interpreted as an operational interpretation for a possible one-shot analogue of quantum conditional mutual information. The communication cost of our protocol is lower than all previously known ones and asymptotically achieves the well-known rate of quantum conditional mutual information. Thus, our work takes a step towards the important open question of near-optimal characterization of the one-shot quantum state redistribution.


title: Observation of a near-threshold enhancement in the $\Lambda\bar{\Lambda}$ mass spectrum from $e^+e^-\to\phi\Lambda\bar{\Lambda}$ at $\sqrt{s}$ from 3.51 to 4.60 GeV

abstract: The process $e^+ e^- \to \phi \Lambda \bar{\Lambda}$ is studied using data samples collected with the BESIII detector at the BEPCII collider at center-of-mass energies $\sqrt{s}$ ranging from $3.51$ to $4.60~{\rm GeV}$ . An intermediate resonance structure is observed near the threshold of $\Lambda \bar{\Lambda}$. It has a mass of $(2262 \pm 4 \pm 28)~{\rm{MeV}}/c^{2}$ and a width of $(72 \pm 5 \pm 43)~\rm{MeV}$, where the quoted uncertainties are statistical and systematic, respectively. The $J^{PC}$ quantum numbers of $0^{-+}$ and $0^{++}$ are rejected, while other $J^{PC}$ hypotheses are possible, according to the helicity angle study. The energy-dependent cross section of the $e^+ e^- \to \phi \Lambda \bar{\Lambda}$ process is measured for the first time in this energy region, and contributions from excited $\psi$ states and vector charmonium-like $Y$-states are investigated.


title: DCH-2: A Parallel Customer-Helpdesk Dialogue Corpus with Distributions of Annotators' Labels

abstract: We introduce a data set called DCH-2, which contains 4,390 real customer-helpdesk dialogues in Chinese and their English translations. DCH-2 also contains dialogue-level annotations and turn-level annotations obtained independently from either 19 or 20 annotators. The data set was built through our effort as organisers of the NTCIR-14 Short Text Conversation and NTCIR-15 Dialogue Evaluation tasks, to help researchers understand what constitutes an effective customer-helpdesk dialogue, and thereby build efficient and helpful helpdesk systems that are available to customers at all times. In addition, DCH-2 may be utilised for other purposes, for example, as a repository for retrieval-based dialogue systems, or as a parallel corpus for machine translation in the helpdesk domain.


title: Strict Arakelov inequality for a family of varieties of general type

abstract: Let $f:\, X\to Y$ be a semistable non-isotrivial family of $n$-folds over a smooth projective curve with discriminant locus $S \subseteq Y$ and with general fibre $F$ of general type. We show the strict Arakelov inequality \[\frac{\mathrm{deg}\, f_*\omega_{X/Y}^\nu}{\mathrm{rank}\, f_*\omega_{X/Y}^\nu} < {n\nu\over 2}\cdot\mathrm{deg}\,\Omega^1_Y(\log S),\] for all $\nu\in \mathbb N$ such that the $\nu$-th pluricanonical linear system $|\omega^\nu_F|$ is birational. This answers a question asked by M\"oller, Viehweg and the third named author.


title: Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders

abstract: Previous work mainly focuses on improving cross-lingual transfer for NLU tasks with a multilingual pretrained encoder (MPE), or improving the performance on supervised machine translation with BERT. However, it is under-explored that whether the MPE can help to facilitate the cross-lingual transferability of NMT model. In this paper, we focus on a zero-shot cross-lingual transfer task in NMT. In this task, the NMT model is trained with parallel dataset of only one language pair and an off-the-shelf MPE, then it is directly tested on zero-shot language pairs. We propose SixT, a simple yet effective model for this task. SixT leverages the MPE with a two-stage training schedule and gets further improvement with a position disentangled encoder and a capacity-enhanced decoder. Using this method, SixT significantly outperforms mBART, a pretrained multilingual encoder-decoder model explicitly designed for NMT, with an average improvement of 7.1 BLEU on zero-shot any-to-English test sets across 14 source languages. Furthermore, with much less training computation cost and training data, our model achieves better performance on 15 any-to-English test sets than CRISS and m2m-100, two strong multilingual NMT baselines.


title: Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

abstract: Large language models have led to remarkable progress on many NLP tasks, and researchers are turning to ever-larger text corpora to train them. Some of the largest corpora available are made by scraping significant portions of the internet, and are frequently introduced with only minimal documentation. In this work we provide some of the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin by investigating where the data came from, and find a significant amount of text from unexpected sources like patents and US military websites. Then we explore the content of the text itself, and find machine-generated text (e.g., from machine translation systems) and evaluation examples from other benchmark NLP datasets. To understand the impact of the filters applied to create this dataset, we evaluate the text that was removed, and show that blocklist filtering disproportionately removes text from and about minority individuals. Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.


title: Revisiting the Complexity Analysis of Conflict-Based Search: New Computational Techniques and Improved Bounds

abstract: The problem of Multi-Agent Path Finding (MAPF) calls for finding a set of conflict-free paths for a fleet of agents operating in a given environment. Arguably, the state-of-the-art approach to computing optimal solutions is Conflict-Based Search (CBS). In this work we revisit the complexity analysis of CBS to provide tighter bounds on the algorithm's run-time in the worst-case. Our analysis paves the way to better pinpoint the parameters that govern (in the worst case) the algorithm's computational complexity. Our analysis is based on two complementary approaches: In the first approach we bound the run-time using the size of a Multi-valued Decision Diagram (MDD) -- a layered graph which compactly contains all possible single-agent paths between two given vertices for a specific path length. In the second approach we express the running time by a novel recurrence relation which bounds the algorithm's complexity. We use generating functions-based analysis in order to tightly bound the recurrence. Using these technique we provide several new upper-bounds on CBS's complexity. The results allow us to improve the existing bound on the running time of CBS for many cases. For example, on a set of common benchmarks we improve the upper-bound by a factor of at least $2^{10^{7}}$.


title: Solving Inefficiency of Self-supervised Representation Learning

abstract: Self-supervised learning (especially contrastive learning) has attracted great interest due to its huge potential in learning discriminative representations in an unsupervised manner. Despite the acknowledged successes, existing contrastive learning methods suffer from very low learning efficiency, e.g., taking about ten times more training epochs than supervised learning for comparable recognition accuracy. In this paper, we reveal two contradictory phenomena in contrastive learning that we call under-clustering and over-clustering problems, which are major obstacles to learning efficiency. Under-clustering means that the model cannot efficiently learn to discover the dissimilarity between inter-class samples when the negative sample pairs for contrastive learning are insufficient to differentiate all the actual object classes. Over-clustering implies that the model cannot efficiently learn features from excessive negative sample pairs, forcing the model to over-cluster samples of the same actual classes into different clusters. To simultaneously overcome these two problems, we propose a novel self-supervised learning framework using a truncated triplet loss. Precisely, we employ a triplet loss tending to maximize the relative distance between the positive pair and negative pairs to address the under-clustering problem; and we construct the negative pair by selecting a negative sample deputy from all negative samples to avoid the over-clustering problem, guaranteed by the Bernoulli Distribution model. We extensively evaluate our framework in several large-scale benchmarks (e.g., ImageNet, SYSU-30k, and COCO). The results demonstrate our model's superiority (e.g., the learning efficiency) over the latest state-of-the-art methods by a clear margin. Codes available at: https://github.com/wanggrun/triplet .


title: An Anomaly Event Detection Method Based on GNN Algorithm for Multi-data Sources

abstract: Anomaly event detection is crucial for critical infrastructure security(transportation system, social-ecological sector, insurance service, government sector etc.) due to its ability to reveal and address the potential cyber-threats in advance by analysing the data(messages, microblogs, logs etc.) from digital systems and networks. However, the convenience and applicability of smart devices and the maturity of connected technology make the social anomaly events data multi-source and dynamic, which result in the inadaptability for multi-source data detection and thus affect the critical infrastructure security. To effectively address the proposed problems, in this paper, we design a novel anomaly detection method based on multi-source data. First, we leverage spectral clustering algorithm for feature extraction and fusion of multiple data sources. Second, by harnessing the power of deep graph neural network(Deep-GNN), we perform a fine-gained anomaly social event detection, revealing the threatening events and guarantee the critical infrastructure security. Experimental results demonstrate that our framework outperforms other baseline anomaly event detection methods and shows high tracking accuracy, strong robustness and stability.


title: Case-based Reasoning for Natural Language Queries over Knowledge Bases

abstract: It is often challenging to solve a complex problem from scratch, but much easier if we can access other similar problems with their solutions -- a paradigm known as case-based reasoning (CBR). We propose a neuro-symbolic CBR approach (CBR-KBQA) for question answering over large knowledge bases. CBR-KBQA consists of a nonparametric memory that stores cases (question and logical forms) and a parametric model that can generate a logical form for a new question by retrieving cases that are relevant to it. On several KBQA datasets that contain complex questions, CBR-KBQA achieves competitive performance. For example, on the ComplexWebQuestions dataset, CBR-KBQA outperforms the current state of the art by 11\% on accuracy. Furthermore, we show that CBR-KBQA is capable of using new cases \emph{without} any further training: by incorporating a few human-labeled examples in the case memory, CBR-KBQA is able to successfully generate logical forms containing unseen KB entities as well as relations.


title: Making Attention Mechanisms More Robust and Interpretable with Virtual Adversarial Training

abstract: Although attention mechanisms have become fundamental components of deep learning models, they are vulnerable to perturbations, which may degrade the prediction performance and model interpretability. Adversarial training (AT) for attention mechanisms has successfully reduced such drawbacks by considering adversarial perturbations. However, this technique requires label information, and thus, its use is limited to supervised settings. In this study, we explore the concept of incorporating virtual AT (VAT) into the attention mechanisms, by which adversarial perturbations can be computed even from unlabeled data. To realize this approach, we propose two general training techniques, namely VAT for attention mechanisms (Attention VAT) and "interpretable" VAT for attention mechanisms (Attention iVAT), which extend AT for attention mechanisms to a semi-supervised setting. In particular, Attention iVAT focuses on the differences in attention; thus, it can efficiently learn clearer attention and improve model interpretability, even with unlabeled data. Empirical experiments based on six public datasets revealed that our techniques provide better prediction performance than conventional AT-based as well as VAT-based techniques, and stronger agreement with evidence that is provided by humans in detecting important words in sentences. Moreover, our proposal offers these advantages without needing to add the careful selection of unlabeled data. That is, even if the model using our VAT-based technique is trained on unlabeled data from a source other than the target task, both the prediction performance and model interpretability can be improved.


title: Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods

abstract: Optimization is important in machine learning problems, and quasi-Newton methods have a reputation as the most efficient numerical schemes for smooth unconstrained optimization. In this paper, we consider the explicit superlinear convergence rates of quasi-Newton methods and address two open problems mentioned by Rodomanov and Nesterov. First, we extend Rodomanov and Nesterov's results to random quasi-Newton methods, which include common DFP, BFGS, SR1 methods. Such random methods adopt a random direction for updating the approximate Hessian matrix in each iteration. Second, we focus on the specific quasi-Newton methods: SR1 and BFGS methods. We provide improved versions of greedy and random methods with provable better explicit (local) superlinear convergence rates. Our analysis is closely related to the approximation of a given Hessian matrix, unconstrained quadratic objective, as well as the general strongly convex, smooth, and strongly self-concordant functions.


title: Improving Neural Model Performance through Natural Language Feedback on Their Explanations

abstract: A class of explainable NLP models for reasoning tasks support their decisions by generating free-form or structured explanations, but what happens when these supporting structures contain errors? Our goal is to allow users to interactively correct explanation structures through natural language feedback. We introduce MERCURIE - an interactive system that refines its explanations for a given reasoning task by getting human feedback in natural language. Our approach generates graphs that have 40% fewer inconsistencies as compared with the off-the-shelf system. Further, simply appending the corrected explanation structures to the output leads to a gain of 1.2 points on accuracy on defeasible reasoning across all three domains. We release a dataset of over 450k graphs for defeasible reasoning generated by our system at https://tinyurl.com/mercurie .


title: Observation of spin-space quantum transport induced by an atomic quantum point contact

abstract: Quantum transport is ubiquitous in physics. So far, quantum transport between terminals has been extensively studied in solid state systems from the fundamental point of views such as the quantized conductance to the applications to quantum devices. Recent works have demonstrated a cold-atom analog of a mesoscopic conductor by engineering a narrow conducting channel with optical potentials, which opens the door for a wealth of research of atomtronics emulating mesoscopic electronic devices and beyond. Here we realize an alternative scheme of the quantum transport experiment with ytterbium atoms in a two-orbital optical lattice system. Our system consists of a multi-component Fermi gas and a localized impurity, where the current can be created in the spin space by introducing the spin-dependent interaction with the impurity. We demonstrate a rich variety of localized-impurity-induced quantum transports, which paves the way for atomtronics exploiting spin degrees of freedom.