title: Designing Observables for Quantum Interference in Jets at Subleading Color

abstract: The large-$N_c$ or topologically planar limit of gauge theories can be considered as a classical limit because all gauge bosons are distinguishable particles and therefore cannot exhibit interference. Quantum effects due to the flow of color therefore arise starting at subleading in $1/N_c$. We introduce kinematic observables explicitly sensitive to effects at subleading color formed from the ratio of interfering to squared color-ordered amplitudes. Such observables are in general not infrared and collinear safe, so we introduce angular observables defined from appropriate multi-point energy correlators motivated by the form of color-ordered amplitudes. We demonstrate that color interference effects are manifest as sinusoidal oscillation in the simplest system, a collinear jet with three particles, and show the limitations of predicting this observable in all-purpose, leading-color parton shower Monte Carlos.


title: A Comparative Analysis of Ookla Speedtest and Measurement Labs Network Diagnostic Test (NDT7)

abstract: Consumers, regulators, and ISPs all use client-based "speed tests" to measure network performance, both in single-user settings and in aggregate. Two prevalent speed tests, Ookla's Speedtest and Measurement Lab's Network Diagnostic Test (NDT), are often used for similar purposes, despite having significant differences in both the test design and implementation, and in the infrastructure used to perform measurements. In this paper, we present the first-ever comparative evaluation of Ookla and NDT7 (the latest version of NDT), both in controlled and wide-area settings. Our goal is to characterize when and to what extent these two speed tests yield different results, as well as the factors that contribute to the differences. To study the effects of the test design, we conduct a series of controlled, in-lab experiments under a comprehensive set of network conditions and usage modes (e.g., TCP congestion control, native vs. browser client). Our results show that Ookla and NDT7 report similar speeds under most in-lab conditions, with the exception of networks that experience high latency, where Ookla consistently reports higher throughput. To characterize the behavior of these tools in wide-area deployment, we collect more than 80,000 pairs of Ookla and NDT7 measurements across nine months and 126 households, with a range of ISPs and speed tiers. This first-of-its-kind paired-test analysis reveals many previously unknown systemic issues, including high variability in NDT7 test results and systematically under-performing servers in the Ookla network.


title: Hardness of Maximum Likelihood Learning of DPPs

abstract: Determinantal Point Processes (DPPs) are a widely used probabilistic model for negatively correlated sets. DPPs have been successfully employed in Machine Learning applications to select a diverse, yet representative subset of data. In seminal work on DPPs in Machine Learning, Kulesza conjectured in his PhD Thesis (2011) that the problem of finding a maximum likelihood DPP model for a given data set is NP-complete. In this work we prove Kulesza's conjecture. In fact, we prove the following stronger hardness of approximation result: even computing a $\left(1-O(\frac{1}{\log^9{N}})\right)$-approximation to the maximum log-likelihood of a DPP on a ground set of $N$ elements is NP-complete. At the same time, we also obtain the first polynomial-time algorithm that achieves a nontrivial worst-case approximation to the optimal log-likelihood: the approximation factor is $\frac{1}{(1+o(1))\log{m}}$ unconditionally (for data sets that consist of $m$ subsets), and can be improved to $1-\frac{1+o(1)}{\log N}$ if all $N$ elements appear in a $O(1/N)$-fraction of the subsets. In terms of techniques, we reduce approximating the maximum log-likelihood of DPPs on a data set to solving a gap instance of a "vector coloring" problem on a hypergraph. Such a hypergraph is built on a bounded-degree graph construction of Bogdanov, Obata and Trevisan (FOCS 2002), and is further enhanced by the strong expanders of Alon and Capalbo (FOCS 2007) to serve our purposes.


title: Lyapunov based Stochastic Stability of a Quantum Decision System for Human-Machine Interaction

abstract: In mathematical psychology, decision makers are modeled using the Lindbladian equations from quantum mechanics to capture important human-centric features such as order effects and violation of the sure thing principle. We consider human-machine interaction involving a quantum decision maker (human) and a controller (machine). Given a sequence of human decisions over time, how can the controller dynamically provide input messages to adapt these decisions so as to converge to a specific decision? We show via novel stochastic Lyapunov arguments how the Lindbladian dynamics of the quantum decision maker can be controlled to converge to a specific decision asymptotically. Our methodology yields a useful mathematical framework for human-sensor decision making. The stochastic Lyapunov results are also of independent interest as they generalize recent results in the literature.


title: Gaussian Pre-Activations in Neural Networks: Myth or Reality?

abstract: The study of feature propagation at initialization in neural networks lies at the root of numerous initialization designs. An assumption very commonly made in the field states that the pre-activations are Gaussian. Although this convenient Gaussian hypothesis can be justified when the number of neurons per layer tends to infinity, it is challenged by both theoretical and experimental works for finite-width neural networks. Our major contribution is to construct a family of pairs of activation functions and initialization distributions that ensure that the pre-activations remain Gaussian throughout the network's depth, even in narrow neural networks. In the process, we discover a set of constraints that a neural network should fulfill to ensure Gaussian pre-activations. Additionally, we provide a critical review of the claims of the Edge of Chaos line of works and build an exact Edge of Chaos analysis. We also propose a unified view on pre-activations propagation, encompassing the framework of several well-known initialization procedures. Finally, our work provides a principled framework for answering the much-debated question: is it desirable to initialize the training of a neural network whose pre-activations are ensured to be Gaussian?


title: Metabolic Reactions Studied by Zero- and Low-Field Nuclear Magnetic Resonance

abstract: State-of-the-art magnetic resonance imaging uses hyperpolarized molecules to track metabolism in vivo, but large superconducting magnets are required, and the strong magnetic fields largely preclude measurement in the presence of conductive materials and magnify problems of magnetic susceptibility inhomogeneity. Operating at zero and low field circumvents these limitations, but until now has not been possible due to limited sensitivity. We show that zero- and low-field nuclear magnetic resonance can be used for probing two important metabolic reactions: the conversion of hyperpolarized fumarate to malate and pyruvate to lactate. This work paves the way to a heretofore unexplored class of biomedical imaging applications.


title: First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization

abstract: How can we train an assistive human-machine interface (e.g., an electromyography-based limb prosthesis) to translate a user's raw command signals into the actions of a robot or computer when there is no prior mapping, we cannot ask the user for supervision in the form of action labels or reward feedback, and we do not have prior knowledge of the tasks the user is trying to accomplish? The key idea in this paper is that, regardless of the task, when an interface is more intuitive, the user's commands are less noisy. We formalize this idea as a completely unsupervised objective for optimizing interfaces: the mutual information between the user's command signals and the induced state transitions in the environment. To evaluate whether this mutual information score can distinguish between effective and ineffective interfaces, we conduct an observational study on 540K examples of users operating various keyboard and eye gaze interfaces for typing, controlling simulated robots, and playing video games. The results show that our mutual information scores are predictive of the ground-truth task completion metrics in a variety of domains, with an average Spearman's rank correlation of 0.43. In addition to offline evaluation of existing interfaces, we use our unsupervised objective to learn an interface from scratch: we randomly initialize the interface, have the user attempt to perform their desired tasks using the interface, measure the mutual information score, and update the interface to maximize mutual information through reinforcement learning. We evaluate our method through a user study with 12 participants who perform a 2D cursor control task using a perturbed mouse, and an experiment with one user playing the Lunar Lander game using hand gestures. The results show that we can learn an interface from scratch, without any user supervision or prior knowledge of tasks, in under 30 minutes.


title: Challenges and Opportunities in Information Manipulation Detection: An Examination of Wartime Russian Media

abstract: NLP research on public opinion manipulation campaigns has primarily focused on detecting overt strategies such as fake news and disinformation. However, information manipulation in the ongoing Russia-Ukraine war exemplifies how governments and media also employ more nuanced strategies. We release a new dataset, VoynaSlov, containing 38M+ posts from Russian media outlets on Twitter and VKontakte, as well as public activity and responses, immediately preceding and during the 2022 Russia-Ukraine war. We apply standard and recently-developed NLP models on VoynaSlov to examine agenda setting, framing, and priming, several strategies underlying information manipulation, and reveal variation across media outlet control, social media platform, and time. Our examination of these media effects and extensive discussion of current approaches' limitations encourage further development of NLP models for understanding information manipulation in emerging crises, as well as other real-world and interdisciplinary tasks.


title: Existence and analyticity of the Lei-Lin solution of the Navier-Stokes equations on the torus

abstract: Lei and Lin have recently given a proof of a global mild solution of the three-dimensional Navier-Stokes equations in function spaces based on the Wiener algebra. An alternative proof of existence of these solutions was then developed by Bae, and this new proof allowed for an estimate of the radius of analyticity of the solutions at positive times. We adapt the Bae proof to prove existence of the Lei-Lin solution in the spatially periodic setting, finding an improved bound for the radius of analyticity in this case.


title: Hyperuniform Active Chiral Fluids with Tunable Internal Structure

abstract: Large density fluctuations observed in active systems and hyperuniformity are two seemingly incompatible phenomena. However, the formation of hyperuniform states has been recently predicted in non-equilibrium fluids formed by chiral particles performing circular motion with the same handedness. Here we report evidence of hyperuniformity realized in a chiral active fluid comprised of pear-shaped Quincke rollers of arbitrary handedness. We show that hyperuniformity and large density fluctuations, triggered by dynamic clustering, coexist in this system at different length scales. The system loses its hyperuniformity as the curvature of particles' motion increases transforming them into localized spinners. Our results experimentally demonstrate a novel hyperuniform active fluid and provide new insights into an interplay between chirality, activity and hyperuniformity.


title: Guiding self-assembly of active colloids by temporal modulation of activity

abstract: Self-organization phenomena in ensembles of self-propelled particles open pathways to the synthesis of new dynamic states not accessible by traditional equilibrium processes. The challenge is to develop a set of principles that facilitate the control and manipulation of emergent active states. Here, we report that dielectric rolling colloids energized by a pulsating electric field self-organize into alternating square lattices with a lattice constant controlled by the parameters of the field. We combine experiments and simulations to examine spatiotemporal properties of the emergent collective patterns, and investigate the underlying dynamics of the self-organization.We reveal the resistance of the dynamic lattices to compression/expansion stresses leading to a hysteretic behavior of the lattice constant. The general mechanism of pattern synthesis and control in active ensembles via temporal modulation of activity can be applied to other active colloidal systems.


title: PLAtE: A Large-scale Dataset for List Page Web Extraction

abstract: Recently, neural models have been leveraged to significantly improve the performance of information extraction from semi-structured websites. However, a barrier for continued progress is the small number of datasets large enough to train these models. In this work, we introduce the PLAtE (Pages of Lists Attribute Extraction) benchmark dataset as a challenging new web extraction task. PLAtE focuses on shopping data, specifically extractions from product review pages with multiple items encompassing the tasks of: (1) finding product-list segmentation boundaries and (2) extracting attributes for each product. PLAtE is composed of 52, 898 items collected from 6, 694 pages and 156, 014 attributes, making it the first largescale list page web extraction dataset. We use a multi-stage approach to collect and annotate the dataset and adapt three state-of-the-art web extraction models to the two tasks comparing their strengths and weaknesses both quantitatively and qualitatively.


title: The interacting pair of galaxies Arp 82: Integral field spectroscopy and numerical simulations

abstract: Spectral data cubes of the interacting pair of galaxies NGC 2535 and NGC 2536 (the Arp 82 system) targeting bright emission lines in the visible band, obtained with the imaging Fourier transform spectrometer (iFTS) SITELLE attached to the Canada-France-Hawaii Telescope (CFHT), are presented. Analysis of H$\upalpha$ velocity maps reveals a bar in $\rm NGC\,2536$. In $\rm NGC\,2535$, we find strong non-circular motions outside the ocular ring, in the elliptical arc and tidal tails of $\rm NGC\,2535$ and a misalignment between the kinematic and photometric position angles. We detect 155 HII region complexes in the interacting pair of galaxies and determine oxygen abundances for 66 of them using different calibrators. We find, regardless of the indicator used, that the oxygen abundance distribution in $\rm NGC\,2536$ is shallow whereas, in $\rm NGC\,2535$, it is best fitted by two slopes, the break occurring beyond the ocular ring. The inner slope is comparable to the one observed in isolated normal star-forming galaxies but the outer slope is shallow. We present a numerical simulation of the interaction that reproduces the observed tidal features, kinematics, and metallicity distribution, to investigate the effect of the interaction on the galaxies. The model indicates that the galaxies have undergone a close encounter, strongly prograde for the primary, and are half way in their course to a second close encounter.


title: DASP: A Framework for Driving the Adoption of Software Security Practices

abstract: Implementing software security practices is a critical concern in modern software development. Industry practitioners, security tool providers, and researchers have provided standard security guidelines and sophisticated security development tools to ensure a secure software development pipeline. But despite these efforts, there continues to be an increase in the number of vulnerabilities that can be exploited by malicious hackers. There is thus an urgent need to understand why developers still introduce security vulnerabilities into their applications and to understand what can be done to motivate them to write more secure code. To understand and address this problem further, we propose DASP, a framework for diagnosing and driving the adoption of software security practices among developers. DASP was conceived by combining behavioral science theories to shape a cross-sectional interview study with 28 software practitioners. Our interviews lead to a framework that consists of a comprehensive set of 33 drivers grouped into 7 higher-level categories that represent what needs to happen or change so that the adoption of software security practices occurs. Using the DASP framework, organizations can design interventions suitable for developers' specific development contexts that will motivate them to write more secure code.


title: Non-degenerate minimal submanifolds as energy concentration sets: a variational approach

abstract: We prove that every non-degenerate minimal submanifold of codimension two can be obtained as the energy concentration set of a family of critical maps for the (rescaled) Ginzburg-Landau functional. The proof is purely variational, and follows the strategy laid out by Jerrard and Sternberg, extending a recent result for geodesics by Colinet-Jerrard-Sternberg. The same proof applies also to the $U(1)$-Yang-Mills-Higgs and to the Allen-Cahn-Hilliard energies.


title: Toxicity Detection with Generative Prompt-based Inference

abstract: Due to the subtleness, implicity, and different possible interpretations perceived by different people, detecting undesirable content from text is a nuanced difficulty. It is a long-known risk that language models (LMs), once trained on corpus containing undesirable content, have the power to manifest biases and toxicity. However, recent studies imply that, as a remedy, LMs are also capable of identifying toxic content without additional fine-tuning. Prompt-methods have been shown to effectively harvest this surprising self-diagnosing capability. However, existing prompt-based methods usually specify an instruction to a language model in a discriminative way. In this work, we explore the generative variant of zero-shot prompt-based toxicity detection with comprehensive trials on prompt engineering. We evaluate on three datasets with toxicity labels annotated on social media posts. Our analysis highlights the strengths of our generative classification approach both quantitatively and qualitatively. Interesting aspects of self-diagnosis and its ethical implications are discussed.


title: Toward Understanding Bias Correlations for Mitigation in NLP

abstract: Natural Language Processing (NLP) models have been found discriminative against groups of different social identities such as gender and race. With the negative consequences of these undesired biases, researchers have responded with unprecedented effort and proposed promising approaches for bias mitigation. In spite of considerable practical importance, current algorithmic fairness literature lacks an in-depth understanding of the relations between different forms of biases. Social bias is complex by nature. Numerous studies in social psychology identify the "generalized prejudice", i.e., generalized devaluing sentiments across different groups. For example, people who devalue ethnic minorities are also likely to devalue women and gays. Therefore, this work aims to provide a first systematic study toward understanding bias correlations in mitigation. In particular, we examine bias mitigation in two common NLP tasks -- toxicity detection and word embeddings -- on three social identities, i.e., race, gender, and religion. Our findings suggest that biases are correlated and present scenarios in which independent debiasing approaches dominant in current literature may be insufficient. We further investigate whether jointly mitigating correlated biases is more desired than independent and individual debiasing. Lastly, we shed light on the inherent issue of debiasing-accuracy trade-off in bias mitigation. This study serves to motivate future research on joint bias mitigation that accounts for correlated biases.


title: Emergent Communication through Metropolis-Hastings Naming Game with Deep Generative Models

abstract: Constructive studies on symbol emergence systems seek to investigate computational models that can better explain human language evolution, the creation of symbol systems, and the construction of internal representations. This study provides a new model for emergent communication, which is based on a probabilistic generative model (PGM) instead of a discriminative model based on deep reinforcement learning. We define the Metropolis-Hastings (MH) naming game by generalizing previously proposed models. It is not a referential game with explicit feedback, as assumed by many emergent communication studies. Instead, it is a game based on joint attention without explicit feedback. Mathematically, the MH naming game is proved to be a type of MH algorithm for an integrative PGM that combines two agents that play the naming game. From this viewpoint, symbol emergence is regarded as decentralized Bayesian inference, and semiotic communication is regarded as inter-personal cross-modal inference. This notion leads to the collective predictive coding hypothesis} regarding language evolution and, in general, the emergence of symbols. We also propose the inter-Gaussian mixture model (GMM)+ variational autoencoder (VAE), a deep generative model for emergent communication based on the MH naming game. The model has been validated on MNIST and Fruits 360 datasets. Experimental findings demonstrate that categories are formed from real images observed by agents, and signs are correctly shared across agents by successfully utilizing both of the observations of agents via the MH naming game. Furthermore, scholars verified that visual images were recalled from signs uttered by agents. Notably, emergent communication without supervision and reward feedback improved the performance of the unsupervised representation learning of agents.


title: Fine-tuned Language Models are Continual Learners

abstract: Recent work on large language models relies on the intuition that most natural language processing tasks can be described via natural language instructions. Language models trained on these instructions show strong zero-shot performance on several standard datasets. However, these models even though impressive still perform poorly on a wide range of tasks outside of their respective training and evaluation sets. To address this limitation, we argue that a model should be able to keep extending its knowledge and abilities, without forgetting previous skills. In spite of the limited success of Continual Learning we show that Language Models can be continual learners. We empirically investigate the reason for this success and conclude that Continual Learning emerges from self-supervision pre-training. Our resulting model Continual-T0 (CT0) is able to learn diverse new tasks, while still maintaining good performance on previous tasks, spanning remarkably through 70 datasets in total. Finally, we show that CT0 is able to combine instructions in ways it was never trained for, demonstrating some compositionality.


title: MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification

abstract: In text summarization and simplification, system outputs must be evaluated along multiple dimensions such as relevance, factual consistency, fluency, and grammaticality, and a wide range of possible outputs could be of high quality. These properties make the development of an adaptable, reference-less evaluation metric both necessary and challenging. We introduce MaskEval, a reference-less metric for text summarization and simplification that operates by performing masked language modeling (MLM) on the concatenation of the candidate and the source texts. It features an attention-like weighting mechanism to modulate the relative importance of each MLM step, which crucially allows it to be adapted to evaluate different quality dimensions. We demonstrate its effectiveness on English summarization and simplification in terms of correlations with human judgments, and explore transfer scenarios between the two tasks.


title: Model of Charge Transfer Collisions Between $C_{60}$ and Slow Ions

abstract: A semi-classical model describing the charge transfer collisions of $C_{60}$ fullerene with different slow ions has been developed to explain available experimental data. This data reveals multiple Breit-Wigner like peaks in the cross sections, with subsequent peaks of reactive cross sections decreasing in magnitude. Calculations of the charge transfer probabilities and cross sections for quasi-resonant and reactive collisions have been performed using semi-empirical potentials of interaction between fullerenes and ion projectiles. All computations have been carried out with realistic wave functions for $C_{60}$'s valence electrons derived from the simplified jellium model. The quality of these electron wave functions have been successfully verified by comparing theoretical calculations and experimental data on the small angle cross sections of resonant $C_{60}+ C_{60}^+$ collisions. Using the semi-empirical potentials to describe resonant scattering phenomena in $C_{60}$ collisions with ions and Landau-Zener charge transfer theory, we calculated theoretical cross sections for various $C_{60}$ charge transfer and fragmentation reactions which agree with experiments.


title: Recipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks

abstract: Learning effective recipe representations is essential in food studies. Unlike what has been developed for image-based recipe retrieval or learning structural text embeddings, the combined effect of multi-modal information (i.e., recipe images, text, and relation data) receives less attention. In this paper, we formalize the problem of multi-modal recipe representation learning to integrate the visual, textual, and relational information into recipe embeddings. In particular, we first present Large-RG, a new recipe graph data with over half a million nodes, making it the largest recipe graph to date. We then propose Recipe2Vec, a novel graph neural network based recipe embedding model to capture multi-modal information. Additionally, we introduce an adversarial attack strategy to ensure stable learning and improve performance. Finally, we design a joint objective function of node classification and adversarial learning to optimize the model. Extensive experiments demonstrate that Recipe2Vec outperforms state-of-the-art baselines on two classic food study tasks, i.e., cuisine category classification and region prediction. Dataset and codes are available at https://github.com/meettyj/Recipe2Vec.


title: Predicting Post-Route Quality of Results Estimates for HLS Designs using Machine Learning

abstract: Machine learning (ML) has been widely used to improve the predictability of EDA tools. The use of CAD tools that express designs at higher levels of abstraction makes machine learning even more important to highlight the performance of various design steps. Behavioral descriptions used during the high-level synthesis (HLS) are completely technology independent making it hard for designers to interpret how changes in the synthesis options affect the resultant circuit. FPGA design flows are completely embracing HLS based methodologies so that software engineers with almost no hardware design skills can easily use their tools. HLS tools allow design space exploration by modifying synthesis options, however, they lack accuracy in the Quality of Results (QoR) reported right after HLS. This lack of correctness results in sub-optimal designs with problems in timing closure. This paper presents a robust ML based design flow that can accurately predict post-route QoR for a given behavioral description without the need to synthesize the design. The model is an important design exploration tool where a designer can quickly view the impact on overall design quality when local and global optimization directives are changed. The proposed methodology presents two strong advantages: (i) Accurate prediction of the design quality (QoR), and (ii) complete elimination of the need to execute high-level synthesis for each design option. We predict three post route parameters, (i). Area, (ii). Latency and (iii). Clock Period of a design just by analyzing the high level behavioral code and some intermediate representation codes. We have integrated the methodology with Xilinx HLS tools and have demonstrated accurate estimation on a variety of FPGA families. Our estimated results are within 10\% of actual computed values


title: Self-mixing in microtubule-kinesin active fluid from nonuniform to uniform distribution of activity

abstract: Active fluids have applications in micromixing, but little is known about the mixing kinematics of systems with spatiotemporally-varying activity. To investigate, UV-activated caged ATP was used to activate controlled regions of microtubule-kinesin active fluid and the mixing process was observed with fluorescent tracers and molecular dyes. At low P\'eclet numbers (diffusive transport), the active-inactive interface progressed toward the inactive area in a diffusion-like manner that was described by a simple model combining diffusion with Michaelis-Menten kinetics. At high P\'eclet numbers (convective transport), the active-inactive interface progressed in a superdiffusion-like manner that was qualitatively captured by an active-fluid hydrodynamic model coupled to ATP transport. Results showed that active fluid mixing involves complex coupling between distribution of active stress and active transport of ATP and reduces mixing time for suspended components with decreased impact of initial component distribution. This work will inform application of active fluids to promote micromixing in microfluidic devices.


title: Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT

abstract: We combine the capacity of sparsely gated Mixture-of-Experts (MoE) with the speed and stability of linear, mixing transformations to design the Sparse Mixer encoder model. Sparse Mixer slightly outperforms (<1%) BERT on GLUE and SuperGLUE, but more importantly trains 65% faster and runs inference 61% faster. We also present a faster variant, prosaically named Fast Sparse Mixer, that marginally underperforms BERT on SuperGLUE, but trains and runs nearly twice as fast. We justify the design of these two models by carefully ablating through various mixing mechanisms, MoE configurations and hyperparameters. Sparse Mixer overcomes many of the latency and stability concerns of MoE models and offers the prospect of serving sparse student models, without resorting to distilling them to dense variants.


title: Quantum-brachistochrone approach to the conversion from $W$ to Greenberger-Horne-Zeilinger states for Rydberg-atom qubits

abstract: Using the quantum-brachistochrone formalism, we address the problem of finding the fastest possible (time-optimal) deterministic conversion between $W$ and Greenberger-Horne-Zeilinger (GHZ) states in a system of three identical and equidistant neutral atoms that are acted upon by four external laser pulses. Assuming that all four pulses are close to being resonant with the same internal (atomic) transition -- the one between the atomic ground state and a high-lying Rydberg state -- each atom can be treated as an effective two-level system ($gr$-type qubit). Starting from an effective system Hamiltonian, which is valid in the Rydberg-blockade regime and defined on a four-state manifold, we derive the quantum-brachistochrone equations pertaining to the fastest possible $W$-to-GHZ state conversion. By numerically solving these equations, we determine the time-dependent Rabi frequencies of external laser pulses that correspond to the time-optimal state conversion. In particular, we show that the shortest possible $W$-to-GHZ state-conversion time is given by $T_{\textrm{QB}}= 6.8\:\hbar/E$, where $E$ is the total laser-pulse energy used, this last time being significantly shorter than the state-conversion times previously found using a dynamical-symmetry-based approach [$T_{\textrm{DS}}=(1.33-1.66)\:T_{\textrm{QB}}$].


title: Reward Uncertainty for Exploration in Preference-based Reinforcement Learning

abstract: Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL methods are able to learn a more flexible reward model based on human preferences by actively incorporating human feedback, i.e. teacher's preferences between two clips of behaviors. However, poor feedback-efficiency still remains a problem in current preference-based RL algorithms, as tailored human feedback is very expensive. To handle this issue, previous methods have mainly focused on improving query selection and policy initialization. At the same time, recent exploration methods have proven to be a recipe for improving sample-efficiency in RL. We present an exploration method specifically for preference-based RL algorithms. Our main idea is to design an intrinsic reward by measuring the novelty based on learned reward. Specifically, we utilize disagreement across ensemble of learned reward models. Our intuition is that disagreement in learned reward model reflects uncertainty in tailored human feedback and could be useful for exploration. Our experiments show that exploration bonus from uncertainty in learned reward improves both feedback- and sample-efficiency of preference-based RL algorithms on complex robot manipulation tasks from MetaWorld benchmarks, compared with other existing exploration methods that measure the novelty of state visitation.


title: Loop Closure Prioritization for Efficient and Scalable Multi-Robot SLAM

abstract: Multi-robot SLAM systems in GPS-denied environments require loop closures to maintain a drift-free centralized map. With an increasing number of robots and size of the environment, checking and computing the transformation for all the loop closure candidates becomes computationally infeasible. In this work, we describe a loop closure module that is able to prioritize which loop closures to compute based on the underlying pose graph, the proximity to known beacons, and the characteristics of the point clouds. We validate this system in the context of the DARPA Subterranean Challenge and on numerous challenging underground datasets and demonstrate the ability of this system to generate and maintain a map with low error. We find that our proposed techniques are able to select effective loop closures which results in 51% mean reduction in median error when compared to an odometric solution and 75% mean reduction in median error when compared to a baseline version of this system with no prioritization. We also find our proposed system is able to find a lower error in the mission time of one hour when compared to a system that processes every possible loop closure in four and a half hours. The code and dataset for this work can be found https://github.com/NeBula-Autonomy/LAMP


title: Jointly Optimizing Color Rendition and In-Camera Backgrounds in an RGB Virtual Production Stage

abstract: While the LED panels used in virtual production systems can display vibrant imagery with a wide color gamut, they produce problematic color shifts when used as lighting due to their peaky spectral output from narrow-band red, green, and blue LEDs. In this work, we present an improved color calibration process for virtual production stages which ameliorates this color rendition problem while also passing through accurate in-camera background colors. We do this by optimizing linear color correction transformations for 1) the LED panel pixels visible in the field of view of the camera, 2) the pixels outside the field of view of the camera illuminating the subjects, and, as a post-process, 3) the pixel values recorded by the camera. The result is that footage shot in an RGB LED panel virtual production stage can exhibit more accurate skin tones and costume colors while still reproducing the desired colors of the in-camera background.


title: FLUTE: Figurative Language Understanding through Textual Explanations

abstract: Figurative language understanding has been recently framed as a recognizing textual entailment (RTE) task (a.k.a. natural language inference, or NLI). However, similar to classical RTE/NLI datasets, the current benchmarks suffer from spurious correlations and annotation artifacts. To tackle this problem, work on NLI has built explanation-based datasets such as e-SNLI, allowing us to probe whether language models are right for the right reasons.Yet no such data exists for figurative language, making it harder to assess genuine understanding of such expressions. To address this issue, we release FLUTE, a dataset of 9,000 figurative NLI instances with explanations, spanning four categories: Sarcasm, Simile, Metaphor, and Idioms. We collect the data through a model-in-the-loop framework based on GPT-3, crowd workers, and expert annotators. We show how utilizing GPT-3 in conjunction with human annotators (novices and experts) can aid in scaling up the creation of datasets even for such complex linguistic phenomena as figurative language. The baseline performance of the T5 model fine-tuned on FLUTE shows that our dataset can bring us a step closer to developing models that understand figurative language through textual explanations.


title: Cosmological piecewise functions to treat the local Hubble tension

abstract: The current cosmic time evolution of the Universe is described by the General Relativity theory when a cosmological principle is considered under a flat space time landscape. The set of known as Friedmann equations, contain the principles that lead to the construction of the standard $\Lambda$CDM model. However, the current state-of-art regarding these equations, even if it is a fundamental method, is based in solving analytically the differential equations by considering several forms of matter/energy components or evaluating them in specific cosmic times where two or more components contribute at the same rate. This latter can be carry out through the approach of piecewice solutions, whose reduce the numerical integrals. In this paper we discuss new solutions through special analytical functions and constraint them with an updated compilation of observational Hubble observations in order to deal with the local $H_0$ tension reported.


title: Multi-Head Online Learning for Delayed Feedback Modeling

abstract: In online advertising, it is highly important to predict the probability and the value of a conversion (e.g., a purchase). It not only impacts user experience by showing relevant ads, but also affects ROI of advertisers and revenue of marketplaces. Unlike clicks, which often occur within minutes after impressions, conversions are expected to happen over a long period of time (e.g., 30 days for online shopping). It creates a challenge, as the true labels are only available after the long delays. Either inaccurate labels (partial conversions) are used, or models are trained on stale data (e.g., from 30 days ago). The problem is more eminent in online learning, which focuses on the live performance on the latest data. In this paper, a novel solution is presented to address this challenge using multi-head modeling. Unlike traditional methods, it directly quantizes conversions into multiple windows, such as day 1, day 2, day 3-7, and day 8-30. A sub-model is trained specifically on conversions within each window. Label freshness is maximally preserved in early models (e.g., day 1 and day 2), while late conversions are accurately utilized in models with longer delays (e.g., day 8-30). It is shown to greatly exceed the performance of known methods in online learning experiments for both conversion rate (CVR) and value per click (VPC) predictions. Lastly, as a general method for delayed feedback modeling, it can be combined with any advanced ML techniques to further improve the performance.


title: Convolutional Neural Processes for Inpainting Satellite Images

abstract: The widespread availability of satellite images has allowed researchers to model complex systems such as disease dynamics. However, many satellite images have missing values due to measurement defects, which render them unusable without data imputation. For example, the scanline corrector for the LANDSAT 7 satellite broke down in 2003, resulting in a loss of around 20\% of its data. Inpainting involves predicting what is missing based on the known pixels and is an old problem in image processing, classically based on PDEs or interpolation methods, but recent deep learning approaches have shown promise. However, many of these methods do not explicitly take into account the inherent spatiotemporal structure of satellite images. In this work, we cast satellite image inpainting as a natural meta-learning problem, and propose using convolutional neural processes (ConvNPs) where we frame each satellite image as its own task or 2D regression problem. We show ConvNPs can outperform classical methods and state-of-the-art deep learning inpainting models on a scanline inpainting problem for LANDSAT 7 satellite images, assessed on a variety of in and out-of-distribution images.


title: Using user's local context to support local news

abstract: American local newspapers have been experiencing a large loss of reader retention and business within the past 15 years due to the proliferation of online news sources. Local media companies are starting to shift from an advertising-supported business model to one based on subscriptions to mitigate this problem. With this subscription model, there is a need to increase user engagement and personalization, and recommender systems are one way for these news companies to accomplish this goal. However, using standard modeling approaches that focus on users' global preferences is not appropriate in this context because the local preferences of users exhibit some specific characteristics which do not necessarily match their long-term or global preferences in the news. Our research explores a localized session-based recommendation approach, using recommendations based on local news articles and articles pertaining to the different local news categories. Experiments performed on a news dataset from a local newspaper show that these local models, particularly certain categories of items, do indeed provide more accuracy and effectiveness for personalization which, in turn, may lead to more user engagement with local news content.


title: On the number of tilting modules over a class of Auslander algebras

abstract: Let $\Lambda$ be a radical square zero algebra of a Dynkin quiver and let $\Gamma$ be the Auslander algebra of $\Lambda$. Then the number of tilting right $\Gamma$-modules is $2^{m-1}$ if $\Lambda$ is of $A_{m}$ type for $m\geq 1$. Otherwise, the number of tilting right $\Gamma$-modules is $2^{m-3}\times14$ if $\Lambda$ is either of $D_{m}$ type for $m\geq 4$ or of $E_{m}$ type for $m=6,7,8$.


title: AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

abstract: Standard fine-tuning of large pre-trained language models (PLMs) for downstream tasks requires updating hundreds of millions to billions of parameters, and storing a large copy of the PLM weights for every task resulting in increased cost for storing, sharing and serving the models. To address this, parameter-efficient fine-tuning (PEFT) techniques were introduced where small trainable components are injected in the PLM and updated during fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of adaptation modules -- given the underlying PEFT method of choice -- introduced in each Transformer layer while keeping most of the PLM weights frozen. For instance, AdaMix can leverage a mixture of adapters like Houlsby or a mixture of low rank decomposition matrices like LoRA to improve downstream task performance over the corresponding PEFT methods for fully supervised and few-shot NLU and NLG tasks. Further, we design AdaMix such that it matches the same computational cost and the number of tunable parameters as the underlying PEFT method. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.


title: Linear Connectivity Reveals Generalization Strategies

abstract: It is widely accepted in the mode connectivity literature that when two neural networks are trained similarly on the same data, they are connected by a path through parameter space over which test set accuracy is maintained. Under some circumstances, including transfer learning from pretrained models, these paths are presumed to be linear. In contrast to existing results, we find that among text classifiers (trained on MNLI, QQP, and CoLA), some pairs of finetuned models have large barriers of increasing loss on the linear paths between them. On each task, we find distinct clusters of models which are linearly connected on the test loss surface, but are disconnected from models outside the cluster -- models that occupy separate basins on the surface. By measuring performance on specially-crafted diagnostic datasets, we find that these clusters correspond to different generalization strategies: one cluster behaves like a bag of words model under domain shift, while another cluster uses syntactic heuristics. Our work demonstrates how the geometry of the loss surface can guide models towards different heuristic functions.


title: Differentially Private AUC Computation in Vertical Federated Learning

abstract: Federated learning has gained great attention recently as a privacy-enhancing tool to jointly train a machine learning model by multiple parties. As a sub-category, vertical federated learning (vFL) focuses on the scenario where features and labels are split into different parties. The prior work on vFL has mostly studied how to protect label privacy during model training. However, model evaluation in vFL might also lead to potential leakage of private label information. One mitigation strategy is to apply label differential privacy (DP) but it gives bad estimations of the true (non-private) metrics. In this work, we propose two evaluation algorithms that can more accurately compute the widely used AUC (area under curve) metric when using label DP in vFL. Through extensive experiments, we show our algorithms can achieve more accurate AUCs compared to the baselines.


title: Looking forward to forward physics at the CERN's LHC

abstract: For decades, new physics searches in collider experiments have focused on the high-$p_T$ region. However, it has recently become evident that the LHC physics potential has not been fully exploited. To be specific, forward collisions, which produce particles along the beamline with enormous rates, have been almost completely ignored. For all practical purposes, these collisions are a treasure trove of physics, containing the highest-energy neutrinos ever produced by humans, as well as possible evidence for dark matter, light and weakly-coupled particles, and new forces. In the upcoming LHC Run 3 the ForwArd Search ExpeRiment (FASER) and its cousin FASER$\nu$ will extend the LHC's physics potential. A continuation of this forward physics program for the HL-LHC aims at the Forward Physics Facility (FPF), with larger scale experiments. In this report, I give an overview of the physics motivations for FASER, FASER$\nu$, and the FPF experiments, including both Standard Model and beyond Standard Model physics.


title: A COHERENT constraint on leptophobic dark matter using CsI data

abstract: We use data from the COHERENT CsI[Na] scintillation detector to constrain sub-GeV leptophobic dark matter models. This detector was built to observe low-energy nuclear recoils from coherent elastic neutrino-nucleus scattering. These capabilities enable searches for dark matter particles produced at the Spallation Neutron Source mediated by a vector portal particle with masses between 2 and 400 MeV/c$^2$. No evidence for dark matter is observed and a limit on the mediator coupling to quarks is placed. This constraint improves upon previous results by two orders of magnitude. This newly explored parameter space probes the region where the dark matter relic abundance is explained by leptophobic dark matter when the mediator mass is roughly twice the dark matter mass. COHERENT sets the best constraint on leptophobic dark matter at these masses.


title: Quasisymmetric Schubert calculus

abstract: The ring of symmetric functions occupies a central place in algebraic combinatorics, with a particularly notable role in Schubert calculus, where the standard cell decompositions of Grassmannians yield the celebrated family of Schur functions and the cohomology ring is governed by Littlewood-Richardson rules. The past 50 years have seen an analogous development of quasisymmetric function theory, with applications to enumerative combinatorics, Hopf algebras, graph theory, representation theory, and other areas. Despite such successes, this theory has lacked a quasisymmetric analogue of Schubert calculus. In particular, there has been much interest, since work of Lam and Pylyavskyy (2007), in developing "$K$-theoretic" analogues of quasisymmetric function theory, for which a major obstacle has been the lack of topological interpretations. Here, building on work of Baker and Richter (2008), we apply the philosophy of Schubert calculus to the loop space $\Omega(\Sigma(\mathbb{C}\mathbb{P}^\infty))$ through the homotopy model given by James reduced product $J(\mathbb{C}\mathbb{P}^\infty)$. We describe a canonical Schubert cell decomposition of $J(\mathbb{C}\mathbb{P}^\infty)$, yielding a canonical basis of its cohomology, which we explicitly identify with monomial quasisymmetric functions. Our constructions apply equally to James reduced products of generalized flag varieties $G/P$, and we show how Littlewood-Richardson rules for any $G/P$ lift to $H^*(J(G/P))$. If $J(\mathbb{C}\mathbb{P}^\infty)$ carried the structure of a normal projective algebraic variety, the structure sheaves of the cell closures would yield a "cellular $K$-theory" Schubert basis. We show this is impossible. Nonetheless, we introduce and study a more subtle $K$-theory Schubert basis. We characterize this $K$-theory ring and develop quasisymmetric representatives with an explicit combinatorial description.


title: Counterfactual Data Augmentation improves Factuality of Abstractive Summarization

abstract: Abstractive summarization systems based on pretrained language models often generate coherent but factually inconsistent sentences. In this paper, we present a counterfactual data augmentation approach where we augment data with perturbed summaries that increase the training data diversity. Specifically, we present three augmentation approaches based on replacing (i) entities from other and the same category and (ii) nouns with their corresponding WordNet hypernyms. We show that augmenting the training data with our approach improves the factual correctness of summaries without significantly affecting the ROUGE score. We show that in two commonly used summarization datasets (CNN/Dailymail and XSum), we improve the factual correctness by about 2.5 points on average


title: Electronic properties of the steps in bilayer Td-WTe2

abstract: Monolayer WTe2 stripes are quantum spin Hall (QSH) insulators. Density functional theory was used for investigating the electronic properties of the stripes and steps in bilayer Td-WTe2. For the stripes oriented along the dimer chains of W atoms (x direction), the hybridization between the two layers suppresses the QSH states. However, the QSH nature can be recovered by forming a step, depending on the atomic structure of the step. Conversely, the stripes and steps along the y direction maintain the QSH states. These findings can expand the application range of the QSH states in WTe2.


title: Tiered Reinforcement Learning: Pessimism in the Face of Uncertainty and Constant Regret

abstract: We propose a new learning framework that captures the tiered structure of many real-world user-interaction applications, where the users can be divided into two groups based on their different tolerance on exploration risks and should be treated separately. In this setting, we simultaneously maintain two policies $\pi^{\text{O}}$ and $\pi^{\text{E}}$: $\pi^{\text{O}}$ ("O" for "online") interacts with more risk-tolerant users from the first tier and minimizes regret by balancing exploration and exploitation as usual, while $\pi^{\text{E}}$ ("E" for "exploit") exclusively focuses on exploitation for risk-averse users from the second tier utilizing the data collected so far. An important question is whether such a separation yields advantages over the standard online setting (i.e., $\pi^{\text{E}}=\pi^{\text{O}}$) for the risk-averse users. We individually consider the gap-independent vs.~gap-dependent settings. For the former, we prove that the separation is indeed not beneficial from a minimax perspective. For the latter, we show that if choosing Pessimistic Value Iteration as the exploitation algorithm to produce $\pi^{\text{E}}$, we can achieve a constant regret for risk-averse users independent of the number of episodes $K$, which is in sharp contrast to the $\Omega(\log K)$ regret for any online RL algorithms in the same setting, while the regret of $\pi^{\text{O}}$ (almost) maintains its online regret optimality and does not need to compromise for the success of $\pi^{\text{E}}$.


title: Physics Guided Machine Learning for Variational Multiscale Reduced Order Modeling

abstract: We propose a new physics guided machine learning (PGML) paradigm that leverages the variational multiscale (VMS) framework and available data to dramatically increase the accuracy of reduced order models (ROMs) at a modest computational cost. The hierarchical structure of the ROM basis and the VMS framework enable a natural separation of the resolved and unresolved ROM spatial scales. Modern PGML algorithms are used to construct novel models for the interaction among the resolved and unresolved ROM scales. Specifically, the new framework builds ROM operators that are closest to the true interaction terms in the VMS framework. Finally, machine learning is used to reduce the projection error and further increase the ROM accuracy. Our numerical experiments for a two-dimensional vorticity transport problem show that the novel PGML-VMS-ROM paradigm maintains the low computational cost of current ROMs, while significantly increasing the ROM accuracy.


title: Learning Action Conditions from Instructional Manuals for Instruction Understanding

abstract: The ability to infer pre- and postconditions of an action is vital for comprehending complex instructions, and is essential for applications such as autonomous instruction-guided agents and assistive AI that supports humans to perform physical tasks. In this work, we propose a task dubbed action condition inference, and collecting a high-quality, human annotated dataset of preconditions and postconditions of actions in instructional manuals. We propose a weakly supervised approach to automatically construct large-scale training instances from online instructional manuals, and curate a densely human-annotated and validated dataset to study how well the current NLP models can infer action-condition dependencies in the instruction texts. We design two types of models differ by whether contextualized and global information is leveraged, as well as various combinations of heuristics to construct the weak supervisions. Our experimental results show a >20% F1-score improvement with considering the entire instruction contexts and a >6% F1-score benefit with the proposed heuristics.


title: Substring Complexities on Run-length Compressed Strings

abstract: Let $S_{T}(k)$ denote the set of distinct substrings of length $k$ in a string $T$, then the $k$-th substring complexity is defined by its cardinality $|S_{T}(k)|$. Recently, $\delta = \max \{ |S_{T}(k)| / k : k \ge 1 \}$ is shown to be a good compressibility measure of highly-repetitive strings. In this paper, given $T$ of length $n$ in the run-length compressed form of size $r$, we show that $\delta$ can be computed in $\mathit{C}_{\mathsf{sort}}(r, n)$ time and $O(r)$ space, where $\mathit{C}_{\mathsf{sort}}(r, n) = O(\min (r \lg\lg r, r \lg_{r} n))$ is the time complexity for sorting $r$ $O(\lg n)$-bit integers in $O(r)$ space in the Word-RAM model with word size $\Omega(\lg n)$.


title: Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL

abstract: Can non-programmers annotate natural language utterances with complex programs that represent their meaning? We introduce APEL, a framework in which non-programmers select among candidate programs generated by a seed semantic parser (e.g., Codex). Since they cannot understand the candidate programs, we ask them to select indirectly by examining the programs' input-ouput examples. For each utterance, APEL actively searches for a simple input on which the candidate programs tend to produce different outputs. It then asks the non-programmers only to choose the appropriate output, thus allowing us to infer which program is correct and could be used to fine-tune the parser. As a first case study, we recruited human non-programmers to use APEL to re-annotate SPIDER, a text-to-SQL dataset. Our approach achieved the same annotation accuracy as the original expert annotators (75%) and exposed many subtle errors in the original annotations.


title: Deletion and Insertion Tests in Regression Models

abstract: A basic task in explainable AI (XAI) is to identify the most important features behind a prediction made by a black box function $f$. The insertion and deletion tests of Petsiuk et al. (2018) can be used to judge the quality of algorithms that rank pixels from most to least important for a classification. Motivated by regression problems we establish a formula for their area under the curve (AUC) criteria in terms of certain main effects and interactions in an anchored decomposition of $f$. We find an expression for the expected value of the AUC under a random ordering of inputs to $f$ and propose an alternative area above a straight line for the regression setting. We use this criterion to compare feature importances computed by integrated gradients (IG) to those computed by Kernel SHAP (KS) as well as LIME, DeepLIFT, vanilla gradient and input$\times$gradient methods. KS has the best overall performance in two datasets we consider but it is very expensive to compute. We find that IG is nearly as good as KS while being much faster. Our comparison problems include some binary inputs that pose a challenge to IG because it must use values between the possible variable levels and so we consider ways to handle binary variables in IG. We show that sorting variables by their Shapley value does not necessarily give the optimal ordering for an insertion-deletion test. It will however do that for monotone functions of additive models, such as logistic regression.


title: VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection

abstract: This paper presents VulBERTa, a deep learning approach to detect security vulnerabilities in source code. Our approach pre-trains a RoBERTa model with a custom tokenisation pipeline on real-world code from open-source C/C++ projects. The model learns a deep knowledge representation of the code syntax and semantics, which we leverage to train vulnerability detection classifiers. We evaluate our approach on binary and multi-class vulnerability detection tasks across several datasets (Vuldeepecker, Draper, REVEAL and muVuldeepecker) and benchmarks (CodeXGLUE and D2A). The evaluation results show that VulBERTa achieves state-of-the-art performance and outperforms existing approaches across different datasets, despite its conceptual simplicity, and limited cost in terms of size of training data and number of model parameters.


title: Katara: Synthesizing CRDTs with Verified Lifting

abstract: Conflict-free replicated data types (CRDTs) are a promising tool for designing scalable, coordination-free distributed systems. However, constructing correct CRDTs is difficult, posing a challenge for even seasoned developers. As a result, CRDT development is still largely the domain of academics, with new designs often awaiting peer review and a manual proof of correctness. In this paper, we present Katara, a program synthesis-based system that takes sequential data type implementations and automatically synthesizes verified CRDT designs from them. Key to this process is a new formal definition of CRDT correctness that combines a reference sequential type with a lightweight ordering constraint that resolves conflicts between non-commutative operations. Our process follows the tradition of work in verified lifting, including an encoding of correctness into SMT logic using synthesized inductive invariants and hand-crafted grammars for the CRDT state and runtime. Katara is able to automatically synthesize CRDTs for a wide variety of scenarios, from reproducing classic CRDTs to synthesizing novel designs based on specifications in existing literature. Crucially, our synthesized CRDTs are fully, automatically verified, eliminating entire classes of common errors and reducing the process of producing a new CRDT from a painstaking paper proof of correctness to a lightweight specification.


title: Current-Controlled Topological Magnetic Transformations in a Nanostructured Kagome Magnet

abstract: Topological magnetic charge Q is a fundamental parameter that describes the magnetic domains and determines their intriguing electromagnetic properties. The ability to switch Q in a controlled way by electrical methods allows for flexible manipulation of electromagnetic behavior in future spintronic devices. Here we report the room-temperature current-controlled topological magnetic transformations between Q = -1 skyrmions and Q = 0 stripes or type-II bubbles in a kagome crystal Fe$_3$Sn$_2$. We show that the reproducible and reversible skyrmion-bubble and skyrmion-stripe transformations can be achieved by tuning the density of nanosecond pulsed current of the order of ~10$^{10}$ A$^{-2}$. Further numerical simulations suggest that spin-transfer torque combined with Joule thermal heating effects determine the current-induced topological magnetic transformations.


title: Non-stationary Bandits with Knapsacks

abstract: In this paper, we study the problem of bandits with knapsacks (BwK) in a non-stationary environment. The BwK problem generalizes the multi-arm bandit (MAB) problem to model the resource consumption associated with playing each arm. At each time, the decision maker/player chooses to play an arm, and s/he will receive a reward and consume certain amount of resource from each of the multiple resource types. The objective is to maximize the cumulative reward over a finite horizon subject to some knapsack constraints on the resources. Existing works study the BwK problem under either a stochastic or adversarial environment. Our paper considers a non-stationary environment which continuously interpolates between these two extremes. We first show that the traditional notion of variation budget is insufficient to characterize the non-stationarity of the BwK problem for a sublinear regret due to the presence of the constraints, and then we propose a new notion of global non-stationarity measure. We employ both non-stationarity measures to derive upper and lower bounds for the problem. Our results are based on a primal-dual analysis of the underlying linear programs and highlight the interplay between the constraints and the non-stationarity. Finally, we also extend the non-stationarity measure to the problem of online convex optimization with constraints and obtain new regret bounds accordingly.


title: Do we need Label Regularization to Fine-tune Pre-trained Language Models?

abstract: Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model. Considering the ever-growing size of pre-trained language models (PLMs), KD is often adopted in many NLP tasks involving PLMs. However, it is evident that in KD, deploying the teacher network during training adds to the memory and computational requirements of training. In the computer vision literature, the necessity of the teacher network is put under scrutiny by showing that KD is a label regularization technique that can be replaced with lighter teacher-free variants such as the label-smoothing technique. However, to the best of our knowledge, this issue is not investigated in NLP. Therefore, this work concerns studying different label regularization techniques and whether we actually need them to improve the fine-tuning of smaller PLM networks on downstream tasks. In this regard, we did a comprehensive set of experiments on different PLMs such as BERT, RoBERTa, and GPT with more than 600 distinct trials and ran each configuration five times. This investigation led to a surprising observation that KD and other label regularization techniques do not play any meaningful role over regular fine-tuning when the student model is pre-trained. We further explore this phenomenon in different settings of NLP and computer vision tasks and demonstrate that pre-training itself acts as a kind of regularization, and additional label regularization is unnecessary.


title: Interaction of a priori Anatomic Knowledge with Self-Supervised Contrastive Learning in Cardiac Magnetic Resonance Imaging

abstract: Training deep learning models on cardiac magnetic resonance imaging (CMR) can be a challenge due to the small amount of expert generated labels and inherent complexity of data source. Self-supervised contrastive learning (SSCL) has recently been shown to boost performance in several medical imaging tasks. However, it is unclear how much the pre-trained representation reflects the primary organ of interest compared to spurious surrounding tissue. In this work, we evaluate the optimal method of incorporating prior knowledge of anatomy into a SSCL training paradigm. Specifically, we evaluate using a segmentation network to explicitly local the heart in CMR images, followed by SSCL pretraining in multiple diagnostic tasks. We find that using a priori knowledge of anatomy can greatly improve the downstream diagnostic performance. Furthermore, SSCL pre-training with in-domain data generally improved downstream performance and more human-like saliency compared to end-to-end training and ImageNet pre-trained networks. However, introducing anatomic knowledge to pre-training generally does not have significant impact.


title: Additive Logistic Mechanism for Privacy-Preserving Self-Supervised Learning

abstract: We study the privacy risks that are associated with training a neural network's weights with self-supervised learning algorithms. Through empirical evidence, we show that the fine-tuning stage, in which the network weights are updated with an informative and often private dataset, is vulnerable to privacy attacks. To address the vulnerabilities, we design a post-training privacy-protection algorithm that adds noise to the fine-tuned weights and propose a novel differential privacy mechanism that samples noise from the logistic distribution. Compared to the two conventional additive noise mechanisms, namely the Laplace and the Gaussian mechanisms, the proposed mechanism uses a bell-shaped distribution that resembles the distribution of the Gaussian mechanism, and it satisfies pure $\epsilon$-differential privacy similar to the Laplace mechanism. We apply membership inference attacks on both unprotected and protected models to quantify the trade-off between the models' privacy and performance. We show that the proposed protection algorithm can effectively reduce the attack accuracy to roughly 50\%-equivalent to random guessing-while maintaining a performance loss below 5\%.


title: Detecting Abrupt Changes in Sequential Pairwise Comparison Data

abstract: The Bradley-Terry-Luce (BTL) model is a classic and very popular statistical approach for eliciting a global ranking among a collection of items using pairwise comparison data. In applications in which the comparison outcomes are observed as a time series, it is often the case that data are non-stationary, in the sense that the true underlying ranking changes over time. In this paper we are concerned with localizing the change points in a high-dimensional BTL model with piece-wise constant parameters. We propose novel and practicable algorithms based on dynamic programming that can consistently estimate the unknown locations of the change points. We provide consistency rates for our methodology that depend explicitly on the model parameters, the temporal spacing between two consecutive change points and the magnitude of the change. We corroborate our findings with extensive numerical experiments and a real-life example.


title: Secuer: ultrafast, scalable and accurate clustering of single-cell RNA-seq data

abstract: Identifying cell clusters is a critical step for single-cell transcriptomics study. Despite the numerous clustering tools developed recently, the rapid growth of scRNA-seq volumes prompts for a more (computationally) efficient clustering method. Here, we introduce Secuer, a Scalable and Efficient speCtral clUstERing algorithm for scRNA-seq data. By employing an anchor-based bipartite graph representation algorithm, Secuer enjoys reduced runtime and memory usage by orders of magnitude, especially for ultra-large datasets profiling over 1 or even 10 million cells. Meanwhile, Secuer also achieves better or comparable accuracy than competing methods in small and moderate benchmark datasets. Furthermore, we showcase that Secuer can also serve as a building block for a new consensus clustering method, Secuer-consensus, which again greatly improves the runtime and scalability of state-of-the-art consensus clustering methods while also maintaining the accuracy. Overall, Secuer is a versatile, accurate, and scalable clustering framework suitable for small to ultra-large single-cell clustering tasks.


title: Global Classical Solutions Near Vacuum to the Initial-Boundary Value Problem of Isentropic Supersonic Flows through Divergent Ducts

abstract: In this paper, we study the global existence and asymptotic behavior of classical solutions near vacuum for the initial-boundary value problem modeling isentropic supersonic flows through divergent ducts. The governing equations are the compressible Euler equations with a small parameter, which can be written as a hyperbolic system in terms of the Riemann invariants with a non-dissipative source. We provide a new result for the global existence of classical solutions to initial-boundary value problems of non-dissipative hyperbolic balance laws without the assumption of small data. The work is based on the local existence, the maximum principle and the uniform a priori estimates obtained by the generalized Lax transformations. The asymptotic behavior of classical solutions is also shown by studying the behavior of Riemann invariants along each characteristic curve and vertical line. The results can be applied to the spherically symmetric solutions to N-dimensional compressible Euler equations. Numerical simulations are provided to support our theoretical results.


title: A generic model for pandemics in networks of communities and the role of vaccination

abstract: The slogan "nobody is safe until everybody is safe" is a dictum to raise awareness that in an interconnected world, pandemics such as COVID-19, require a global approach. Motivated by the ongoing COVID-19 pandemic, we model here the spread of a virus in interconnected communities and explore different vaccination scenarios, assuming that the efficacy of the vaccination wanes over time. We start with susceptible populations and consider a susceptible-vaccinated-infected-recovered model with unvaccinated ("Bronze"), moderately vaccinated ("Silver") and very well vaccinated ("Gold") communities, connected through different types of networks via a diffusive linear coupling for local spreading. We show that when considering interactions in "Bronze"-"Gold" and "Bronze"-"Silver" communities, the "Bronze" community is driving an increase in infections in the "Silver" and "Gold" communities. This shows a detrimental, unidirectional effect of non-vaccinated to vaccinated communities. Regarding the interactions between "Gold", "Silver" and "Bronze" communities in a network, we find that two factors play central role: the coupling strength in the dynamics and network density. When considering the spread of a virus in Barab\'asi-Albert networks, infections in "Silver" and "Gold" communities are lower than in "Bronze" communities. We find that the "Gold" communities are the best in keeping their infection levels low. However, a small number of "Bronze" communities are enough to give rise to an increase in infections in moderately and well-vaccinated communities. When studying the spread of a virus in a dense Erd\H{o}s-R\'enyi, and sparse Watts-Strogatz and Barab\'asi-Albert networks, the communities reach the disease-free state in the dense Erd\H{o}s-R\'enyi networks, but not in the sparse Watts-Strogatz and Barab\'asi-Albert networks. However, we also find that if all these networks...


title: Tube classes over elementary vanishing cycles

abstract: Let $X$ be a closed Riemann surface. When $X$ is embedded into a projective space, the first rational cohomology group can be concretely obtained from the monodromy in the family of its smooth hyperplane sections by C. Schnell's tube mapping. We generalize this result to the first integral homology group by relating the tube mapping with the topological Abel--Jacobi mapping. By making use of the mapping class group action, we prove that all tube classes constructed from the elementary vanishing cycles form a cofinite subgroup of the first integral homology group of $X$.


title: An equivalence between vanishing cycles and microlocalization

abstract: We define the functors of specialization and microlocalization along a quasi-smooth closed immersion using the theory of quasi-smooth blow-ups developed in arXiv:1802.05702 [math.AG]. Our first main result describes the entire microlocalization functor along the inclusion of a derived zero fiber as the vanishing cycles with respect to a certain function. The proof is based on the observation that the entire Fourier-Sato transform may be written as a vanishing cycles. An equivalent result in a slightly different setting was recently obtained by Tasuki Kinjo in arXiv:2109.06468 [math.AG] using a different method of proof. Our second main result is an equivalence between the canonical twisted vanishing cycles sheaf introduced by Brav, Bussi, Dupont, Joyce, Szendr\H{o}i, and Sch\"urmann on the $-1$-shifted cotangent bundle of a quasi-smooth scheme and the quasi-smooth microlocalization of the perverse constant sheaf.


title: On the clique behavior and Hellyness of the complements of regular graphs

abstract: A collection of sets is intersecting, if any pair of sets in the collection has nonempty intersection. A collection of sets \(\mathcal{C}\) has the Helly property if any intersecting subcollection has nonempty intersection. A graph is /Helly/ if the collection of maximal complete subgraphs of \(G\) has the Helly property. We prove that if \(G\) is a \(k\)-regular graph with \(n\) vertices such that \(n>3k+\sqrt{2k^{2}-k}\), then the complement \(\bar{G}\) is not Helly. We also consider the problem of whether the properties of Hellyness and convergence under the clique graph operator are equivalent for the complement of \(k\)-regular graphs, for small values of \(k\).


title: Skin Cancer Diagnostics with an All-Inclusive Smartphone Application

abstract: Among the different types of skin cancer, melanoma is considered to be the deadliest and is difficult to treat at advanced stages. Detection of melanoma at earlier stages can lead to reduced mortality rates. Desktop-based computer-aided systems have been developed to assist dermatologists with early diagnosis. However, there is significant interest in developing portable, at-home melanoma diagnostic systems which can assess the risk of cancerous skin lesions. Here, we present a smartphone application that combines image capture capabilities with preprocessing and segmentation to extract the Asymmetry, Border irregularity, Color variegation, and Diameter (ABCD) features of a skin lesion. Using the feature sets, classification of malignancy is achieved through support vector machine classifiers. By using adaptive algorithms in the individual data-processing stages, our approach is made computationally light, user friendly, and reliable in discriminating melanoma cases from benign ones. Images of skin lesions are either captured with the smartphone camera or imported from public datasets. The entire process from image capture to classification runs on an Android smartphone equipped with a detachable 10x lens, and processes an image in less than a second. The overall performance metrics are evaluated on a public database of 200 images with Synthetic Minority Over-sampling Technique (SMOTE) (80% sensitivity, 90% specificity, 88% accuracy, and 0.85 area under curve (AUC)) and without SMOTE (55% sensitivity, 95% specificity, 90% accuracy, and 0.75 AUC). The evaluated performance metrics and computation times are comparable or better than previous methods. This all-inclusive smartphone application is designed to be easy-to-download and easy-to-navigate for the end user, which is imperative for the eventual democratization of such medical diagnostic systems.


title: Prime power order circulant determinants

abstract: Newman showed that for primes $p\geq 5$ an integral circulant determinant of prime power order $p^t$ cannot take the value $p^{t+1}$ once $t\geq 2.$ We show that many other values are also excluded. In particular, we show that $p^{2t}$ is the smallest power of $p$ attained for any $t\geq 3$, $p\geq 3.$ We demonstrate the complexity involved by giving a complete description of the $25\times 25$ and $27\times 27$ integral circulant determinants. The former case involves a partition of the primes that are $1\bmod5$ into two sets, Tanner's \textit{perissads} and \textit{artiads}, which were later characterized by E. Lehmer.


title: Transfer of angular momentum of guided light to an atom with an electric quadrupole transition near an optical nanofiber

abstract: We study the transfer of angular momentum of guided photons to a two-level atom with an electric quadrupole transition near an optical nanofiber. We show that the generation of the axial orbital torque of the driving guided field on the atom is governed by the internal-state selection rules for the quadrupole transition and by the angular momentum conservation law with the photon angular momentum given in the Minkowski formulation. We find that the torque depends on the photon angular momentum, the change in the angular momentum of the atomic internal state, and the quadrupole-transition Rabi frequency. We calculate numerically the torques for the quadrupole transitions between the sublevel $M=2$ of the hyperfine-structure level $5S_{1/2}F=2$ and the sublevels $M'=0$, 1, 2, 3, and 4 of the hyperfine-structure level $4D_{5/2}F'=4$ of a $^{87}$Rb atom. We show that the absolute value of the torque for the higher-order mode HE$_{21}$ is larger than that of the torque for the fundamental mode HE$_{11}$ except for the case $M'-M=2$, where the torque for the mode HE$_{21}$ is vanishing.


title: Design of an embedded system with on-demand image capturing and transmission for remote agricultural monitoring

abstract: The ability to visually verify some element of a remotely controlled agricultural automation system through a photograph is valuable in many cases, not only in the operational phase of the system, but especially in the design and implementation phases. Owing to the remote location of many of the application sites, cellular technology is one enabling medium through which wireless transmission of the photographs could be realized. As data usage is a concern for systems using cellular technology, MQTT chosen over other protocols due to its lower message-to-header overhead. This paper outlines the hardware and firmware design of the LTE Cat-M1 enabled embedded system and the backend web development of the cloud-based web application to facilitate the receiving of the photograph. A satisfactory degree of implementation success was achieved in this project, with deployment to a production environment possible after further refinements.


title: Lyapunov function approach for approximation algorithm design and analysis: with applications in submodular maximization

abstract: We propose a two-phase systematical framework for approximation algorithm design and analysis via Lyapunov function. The first phase consists of using Lyapunov function as an input and outputs a continuous-time approximation algorithm with a provable approximation ratio. The second phase then converts this continuous-time algorithm to a discrete-time algorithm with almost the same approximation ratio along with provable time complexity. One distinctive feature of our framework is that we only need to know the parametric form of the Lyapunov function whose complete specification will not be decided until the end of the first phase by maximizing the approximation ratio of the continuous-time algorithm. Some immediate benefits of the Lyapunov function approach include: (i) unifying many existing algorithms; (ii) providing a guideline to design and analyze new algorithms; and (iii) offering new perspectives to potentially improve existing algorithms. We use various submodular maximization problems as running examples to illustrate our framework.


title: Generating Natural Language Proofs with Verifier-Guided Search

abstract: Reasoning over natural language is a challenging problem in NLP. In this work, we focus on proof generation: Given a hypothesis and a set of supporting facts, the model generates a proof tree indicating how to derive the hypothesis from supporting facts. Compared to generating the entire proof in one shot, stepwise generation can better exploit the compositionality and generalize to longer proofs but has achieved limited success on real-world data. Existing stepwise methods struggle to generate proof steps that are both logically valid and relevant to the hypothesis. Instead, they tend to hallucinate invalid steps given the hypothesis. In this paper, we present a novel stepwise method, NLProofS (Natural Language Proof Search), which learns to generate relevant steps conditioning on the hypothesis. At the core of our approach, we train an independent verifier to check the validity of the proof steps to prevent hallucination. Instead of generating steps greedily, we search for proofs maximizing a global proof score judged by the verifier. NLProofS achieves state-of-the-art performance on EntailmentBank and RuleTaker. Specifically, it improves the correctness of predicted proofs from 27.7% to 33.3% in the distractor setting of EntailmentBank, demonstrating the effectiveness of NLProofS in generating challenging human-authored proofs.


title: Evolution of the cyclotron mass with doping in La$_{2-x}$Sr$_x$CuO$_4$

abstract: The recent observation of cyclotron resonance in optimally-doped La$_{2-x}$Sr$_x$CuO$_4$ using time-domain THz spectroscopy in high magnetic field has given new possibilities for the study of cuprate superconductors. One can measure the cyclotron mass in more disordered cuprates possesing short scattering times therefore expanding the study to materials and dopings in which quantum oscillations have not been observed. Here we present the measurement of the carrier mass of the hole doped cuprate La$_{2-x}$Sr$_x$CuO$_4$ across a a range of dopings spanning the slightly underdoped ($x=0.13$) to highly overdoped ($x=0.26$), near to the termination of the superconducting dome. These results reveal a systematic increase of $m_c$ with doping, up to values greater than thirteen times the bare electron mass. This is in contrast with those masses extracted from the heat capacity, which show a peak near the pseudogap critical point $p^*$ and/or Lifshitz transition. The cyclotron frequency is linear in field up to 31 T for all dopings giving no evidence for field induced Fermi surface reconstructions. The cyclotron mass is found to be positive for all dopings, but with a magnitude systematically below the heat capacity mass for under and optimally doped samples, while exceeding it for overdoped samples. Among other aspects, these results are surprising as photoemission reveals a Lifshitz transition in the middle of our doping range and the sign of the cyclotron mass determined from a finite frequency resonance is -- in conventional theories -- a {\it topological quantity} only sensitive to whether or not the Fermi surface is closed around holes or electrons. We see no sign of a divergence of the mass near $p^*$ nor near the Lifshitz transition, showing that any singularity -- if it exists -- is not strong enough to affect the cyclotron mass.


title: Over-the-Air Design of GAN Training for mmWave MIMO Channel Estimation

abstract: Future wireless systems are trending towards higher carrier frequencies that offer larger communication bandwidth but necessitate the use of large antenna arrays. Existing signal processing techniques for channel estimation do not scale well to this "high-dimensional" regime in terms of performance and pilot overhead. Meanwhile, training deep learning based approaches for channel estimation requires large labeled datasets mapping pilot measurements to clean channel realizations, which can only be generated offline using simulated channels. In this paper, we develop a novel unsupervised over-the-air (OTA) algorithm that utilizes noisy received pilot measurements to train a deep generative model to output beamspace MIMO channel realizations. Our approach leverages Generative Adversarial Networks (GAN), while using a conditional input to distinguish between Line-of-Sight (LOS) and Non-Line-of-Sight (NLOS) channel realizations. We also present a federated implementation of the OTA algorithm that distributes the GAN training over multiple users and greatly reduces the user side computation. We then formulate channel estimation from a limited number of pilot measurements as an inverse problem and reconstruct the channel by optimizing the input vector of the trained generative model. Our proposed approach significantly outperforms Orthogonal Matching Pursuit on both LOS and NLOS channel models, and EM-GM-AMP -- an Approximate Message Passing algorithm -- on LOS channel models, while achieving comparable performance on NLOS channel models in terms of the normalized channel reconstruction error. More importantly, our proposed framework has the potential to be trained online using real noisy pilot measurements, is not restricted to a specific channel model and can even be utilized for a federated OTA design of a dataset generator from noisy data.


title: FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech

abstract: We introduce FLEURS, the Few-shot Learning Evaluation of Universal Representations of Speech benchmark. FLEURS is an n-way parallel speech dataset in 102 languages built on top of the machine translation FLoRes-101 benchmark, with approximately 12 hours of speech supervision per language. FLEURS can be used for a variety of speech tasks, including Automatic Speech Recognition (ASR), Speech Language Identification (Speech LangID), Translation and Retrieval. In this paper, we provide baselines for the tasks based on multilingual pre-trained models like mSLAM. The goal of FLEURS is to enable speech technology in more languages and catalyze research in low-resource speech understanding.


title: Uniformly Bounded Regret in Dynamic Fair Allocation

abstract: We study a dynamic allocation problem in which $T$ sequentially arriving divisible resources are to be allocated to a number of agents with linear utilities. The marginal utilities of each resource to the agents are drawn stochastically from a known joint distribution, independently and identically across time, and the central planner makes immediate and irrevocable allocation decisions. Most works on dynamic resource allocation aim to maximize the utilitarian welfare, i.e., the efficiency of the allocation, which may result in unfair concentration of resources on certain high-utility agents while leaving others' demands under-fulfilled. In this paper, aiming at balancing efficiency and fairness, we instead consider a broad collection of welfare metrics, the H\"older means, which includes the Nash social welfare and the egalitarian welfare. To this end, we first study a fluid-based policy derived from a deterministic surrogate to the underlying problem and show that for all smooth H\"older mean welfare metrics it attains an $O(1)$ regret over the time horizon length $T$ against the hindsight optimum, i.e., the optimal welfare if all utilities were known in advance of deciding on allocations. However, when evaluated under the non-smooth egalitarian welfare, the fluid-based policy attains a regret of order $\Theta(\sqrt{T})$. We then propose a new policy built thereupon, called Backward Infrequent Re-solving with Thresholding ($\mathsf{BIRT}$), which consists of re-solving the deterministic surrogate problem at most $O(\log\log T)$ times. We prove the $\mathsf{BIRT}$ policy attains an $O(1)$ regret against the hindsight optimal egalitarian welfare, independently of the time horizon length $T$. We conclude by presenting numerical experiments to corroborate our theoretical claims and to illustrate the significant performance improvement against several benchmark policies.


title: Transportation-Inequalities, Lyapunov Stability and Sampling for Dynamical Systems on Continuous State Space

abstract: We study the concentration phenomenon for discrete-time random dynamical systems with an unbounded state space. We develop a heuristic approach towards obtaining exponential concentration inequalities for dynamical systems using an entirely functional analytic framework. We also show that existence of exponential-type Lyapunov function, compared to the purely deterministic setting, not only implies stability but also exponential concentration inequalities for sampling from the stationary distribution, via \emph{transport-entropy inequality} (T-E). These results have significant impact in \emph{reinforcement learning} (RL) and \emph{controls}, leading to exponential concentration inequalities even for unbounded observables, while neither assuming reversibility nor exact knowledge of random dynamical system (assumptions at heart of concentration inequalities in statistical mechanics and Markov diffusion processes).

categories: stat.ML cs.IT cs.LG cs.SY eess.SY math.IT


title: MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learning

abstract: Many recent breakthroughs in multi-agent reinforcement learning (MARL) require the use of deep neural networks, which are challenging for human experts to interpret and understand. On the other hand, existing work on interpretable reinforcement learning (RL) has shown promise in extracting more interpretable decision tree-based policies from neural networks, but only in the single-agent setting. To fill this gap, we propose the first set of algorithms that extract interpretable decision-tree policies from neural networks trained with MARL. The first algorithm, IVIPER, extends VIPER, a recent method for single-agent interpretable RL, to the multi-agent setting. We demonstrate that IVIPER learns high-quality decision-tree policies for each agent. To better capture coordination between agents, we propose a novel centralized decision-tree training algorithm, MAVIPER. MAVIPER jointly grows the trees of each agent by predicting the behavior of the other agents using their anticipated trees, and uses resampling to focus on states that are critical for its interactions with other agents. We show that both algorithms generally outperform the baselines and that MAVIPER-trained agents achieve better-coordinated performance than IVIPER-trained agents on three different multi-agent particle-world environments.


title: Cross-Domain Style Mixing for Face Cartoonization

abstract: Cartoon domain has recently gained increasing popularity. Previous studies have attempted quality portrait stylization into the cartoon domain; however, this poses a great challenge since they have not properly addressed the critical constraints, such as requiring a large number of training images or the lack of support for abstract cartoon faces. Recently, a layer swapping method has been used for stylization requiring only a limited number of training images; however, its use cases are still narrow as it inherits the remaining issues. In this paper, we propose a novel method called Cross-domain Style mixing, which combines two latent codes from two different domains. Our method effectively stylizes faces into multiple cartoon characters at various face abstraction levels using only a single generator without even using a large number of training images.


title: Region-aware Knowledge Distillation for Efficient Image-to-Image Translation

abstract: Recent progress in image-to-image translation has witnessed the success of generative adversarial networks (GANs). However, GANs usually contain a huge number of parameters, which lead to intolerant memory and computation consumption and limit their deployment on edge devices. To address this issue, knowledge distillation is proposed to transfer the knowledge from a cumbersome teacher model to an efficient student model. However, most previous knowledge distillation methods are designed for image classification and lead to limited performance in image-to-image translation. In this paper, we propose Region-aware Knowledge Distillation ReKo to compress image-to-image translation models. Firstly, ReKo adaptively finds the crucial regions in the images with an attention module. Then, patch-wise contrastive learning is adopted to maximize the mutual information between students and teachers in these crucial regions. Experiments with eight comparison methods on nine datasets demonstrate the substantial effectiveness of ReKo on both paired and unpaired image-to-image translation. For instance, our 7.08X compressed and 6.80X accelerated CycleGAN student outperforms its teacher by 1.33 and 1.04 FID scores on Horse to Zebra and Zebra to Horse, respectively. Codes will be released on GitHub.


title: Sparse*BERT: Sparse Models Generalize To New tasks and Domains

abstract: Large Language Models have become the core architecture upon which most modern natural language processing (NLP) systems build. These models can consistently deliver impressive accuracy and robustness across tasks and domains, but their high computational overhead can make inference difficult and expensive. To make using these models less costly, recent work has explored leveraging structured and unstructured pruning, quantization, and distillation to improve inference speed and decrease size. This paper studies how models pruned using Gradual Unstructured Magnitude Pruning can transfer between domains and tasks. Our experimentation shows that models that are pruned during pretraining using general domain masked language models can transfer to novel domains and tasks without extensive hyperparameter exploration or specialized approaches. We demonstrate that our general sparse model Sparse*BERT can become SparseBioBERT simply by pretraining the compressed architecture on unstructured biomedical text. Moreover, we show that SparseBioBERT can match the quality of BioBERT with only 10\% of the parameters.


title: Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning

abstract: A recent family of techniques, dubbed lightweight fine-tuning methods, facilitates parameter-efficient transfer learning by updating only a small set of additional parameters while keeping the parameters of the pretrained language model frozen. While proven to be an effective method, there are no existing studies on if and how such knowledge of the downstream fine-tuning approach should affect the pretraining stage. In this work, we show that taking the ultimate choice of fine-tuning method into consideration boosts the performance of parameter-efficient fine-tuning. By relying on optimization-based meta-learning using MAML with certain modifications for our distinct purpose, we prime the pretrained model specifically for parameter-efficient fine-tuning, resulting in gains of up to 1.7 points on cross-lingual NER fine-tuning. Our ablation settings and analyses further reveal that the tweaks we introduce in MAML are crucial for the attained gains.