title: Non-uniqueness up to the Onsager threshold for the forced SQG equation

abstract: We establish new non-uniqueness results for the forced inviscid surface quasi-geostrophic equation, via an alternating formulation of convex integration techniques. Our results imply non-uniquenesss in the class of weak solutions with $|\nabla|^{-1}\theta\in C_tC_x^\alpha$, for any $\alpha<1$.


title: The free energy of matrix models

abstract: In this paper we study multi-matrix models whose potentials are small perturbations of the quadratic potential associated with independent GUE random matrices. More precisely, we compute the free energy and the expectation of the trace of polynomials evaluated in those matrices. We prove an asymptotic expansion in the inverse of the matrix dimension to any order. Out of this result we deduce new formulas for map enumeration and the microstates free entropy. The approach that we take is based on the interpolation method between random matrices and free operators developed in [8,29].


title: The factorial function and generalizations, extended

abstract: This paper presents an extension of Bhargava's theory of factorials associated to any nonempty subset $S$ of $\mathbb{Z}$. Bhargava's factorials $k!_S$ are invariants, constructed using the notion of $p$-orderings of $S$ where $p$ is a prime. This paper defines $b$-orderings of any nonempty subset $S$ of $\mathbb{Z}$ for all integers $b\ge2$, as well as "extreme" cases $b=1$ and $b=0$. It defines generalized factorials $k !_{S,T}$ and generalized binomial coefficients $\binom{k+\ell}{k}_{S,T}$ as nonnegative integers, for all nonempty $S$ and allowing only $b$ in $T\subseteq\mathbb{N}$. It computes $b$-ordering invariants when $S$ is $\mathbb{Z}$ and when $S$ is the set of all primes.


title: Microphysics of Particle Reflection in Weibel-Mediated Shocks

abstract: Particle-in-cell (PIC) simulations have shown that relativistic collisionless shocks mediated by the Weibel instability accelerate about 1% of incoming particles, while the majority are transmitted through the shock and become thermalized. The microphysical processes that determine whether an incoming particle will be transmitted or reflected are poorly understood. We study the microphysics of particle reflection in Weibel-mediated shocks by tracking a shell of test particles in a PIC simulation of a shock in pair plasma. We find that electrons in positron-dominated filaments and positrons in electron-dominated filaments efficiently reflect off of strong magnetic structures at the shock. These reflected particles headed towards the upstream must then find filaments of the same sign of current as the current carried by the reflected particles in order to successfully move with the shock and participate in diffusive shock acceleration (DSA). The final injection efficiency on the order of about 1% thus results from the effectiveness of the initial reflection at the shock and the reflected particles' probability of survival in the upstream post-reflection. We develop a model that predicts the fraction of high-energy particles as a function of the properties of Weibel filamentation.


title: Spacetime geometry from canonical spherical gravity

abstract: We study covariant models for vacuum spherical gravity within a canonical setting. Starting from a general ansatz, we derive the most general family of Hamiltonian constraints that are quadratic in first-order and linear in second-order spatial derivatives of the triad variables, and obey certain specific covariance conditions. These conditions ensure that the dynamics generated by such family univocally defines a spacetime geometry, independently of gauge or coordinates choices. This analysis generalizes the Hamiltonian constraint of general relativity, though keeping intact the covariance of the theory, and leads to a rich variety of new geometries. We find that the resulting geometries depend on seven free functions of one scalar variable, and we study their generic features. By construction, there are no propagating degrees of freedom in the theory. However, we also show that it is possible to add matter to the system by simply following the usual minimal-coupling prescription, which leads to novel models to describe dynamical scenarios.


title: Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learning

abstract: Measuring diversity accurately is important for many scientific fields, including machine learning (ML), ecology, and chemistry. The Vendi Score was introduced as a generic similarity-based diversity metric that extends the Hill number of order q=1 by leveraging ideas from quantum statistical mechanics. Contrary to many diversity metrics in ecology, the Vendi Score accounts for similarity and does not require knowledge of the prevalence of the categories in the collection to be evaluated for diversity. However, the Vendi Score treats each item in a given collection with a level of sensitivity proportional to the item's prevalence. This is undesirable in settings where there is a significant imbalance in item prevalence. In this paper, we extend the other Hill numbers using similarity to provide flexibility in allocating sensitivity to rare or common items. This leads to a family of diversity metrics -- Vendi scores with different levels of sensitivity -- that can be used in a variety of applications. We study the properties of the scores in a synthetic controlled setting where the ground truth diversity is known. We then test their utility in improving molecular simulations via Vendi Sampling. Finally, we use the Vendi scores to better understand the behavior of image generative models in terms of memorization, duplication, diversity, and sample quality.


title: Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation

abstract: Thanks to their generative capabilities, large language models (LLMs) have become an invaluable tool for creative processes. These models have the capacity to produce hundreds and thousands of visual and textual outputs, offering abundant inspiration for creative endeavors. But are we harnessing their full potential? We argue that current interaction paradigms fall short, guiding users towards rapid convergence on a limited set of ideas, rather than empowering them to explore the vast latent design space in generative models. To address this limitation, we propose a framework that facilitates the structured generation of design space in which users can seamlessly explore, evaluate, and synthesize a multitude of responses. We demonstrate the feasibility and usefulness of this framework through the design and development of an interactive system, Luminate, and a user study with 14 professional writers. Our work advances how we interact with LLMs for creative tasks, introducing a way to harness the creative potential of LLMs.


title: Single-Mode Squeezed Light Generation and Tomography with an Integrated Optical Parametric Oscillator

abstract: Quantum optical technologies promise advances in sensing, computing, and communication. A key resource is squeezed light, where quantum noise is redistributed between optical quadratures. We introduce a monolithic, chip-scale platform that exploits the $\chi^{(2)}$ nonlinearity of a thin-film lithium niobate (TFLN) resonator device to efficiently generate squeezed states of light. Our system integrates all essential components -- except for the laser and two detectors -- on a single chip with an area of one square centimeter, significantly reducing the size, operational complexity, and power consumption associated with conventional setups. Our work addresses challenges that have limited previous integrated nonlinear photonic implementations that rely on either $\chi^{(3)}$ nonlinear resonators or on integrated waveguide $\chi^{(2)}$ parametric amplifiers. Using the balanced homodyne measurement subsystem that we implemented on the same chip, we measure a squeezing of 0.55 dB and an anti-squeezing of 1.55 dB. We use 20 mW of input power to generate the parametric oscillator pump field by employing second harmonic generation on the same chip. Our work represents a substantial step toward compact and efficient quantum optical systems posed to leverage the rapid advances in integrated nonlinear and quantum photonics.


title: Towards Robust Offline Reinforcement Learning under Diverse Data Corruption

abstract: Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collected by humans in real-world environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first investigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q functions under dynamics corruption. To tackle this challenge, we draw inspiration from robust statistics to employ the Huber loss to handle the heavy-tailedness and utilize quantile estimators to balance penalization for corrupted data and learning stability. By incorporating these simple yet effective modifications into IQL, we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive experiments demonstrate that RIQL exhibits highly robust performance when subjected to diverse data corruption scenarios.


title: Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems

abstract: In this work, we study rapid improvements of the training loss in transformers when being confronted with multi-step decision tasks. We found that transformers struggle to learn the intermediate task and both training and validation loss saturate for hundreds of epochs. When transformers finally learn the intermediate task, they do this rapidly and unexpectedly. We call these abrupt improvements Eureka-moments, since the transformer appears to suddenly learn a previously incomprehensible concept. We designed synthetic tasks to study the problem in detail, but the leaps in performance can be observed also for language modeling and in-context learning (ICL). We suspect that these abrupt transitions are caused by the multi-step nature of these tasks. Indeed, we find connections and show that ways to improve on the synthetic multi-step tasks can be used to improve the training of language modeling and ICL. Using the synthetic data we trace the problem back to the Softmax function in the self-attention block of transformers and show ways to alleviate the problem. These fixes reduce the required number of training steps, lead to higher likelihood to learn the intermediate task, to higher final accuracy and training becomes more robust to hyper-parameters.


title: Kink dynamics in a high-order field model

abstract: We study various properties of topological solitons (kinks) of a field-theoretic model with a polynomial potential of the twelfth degree. This model is remarkable in that it has several topological sectors, in which kinks have different masses. We obtain asymptotic estimates for the kink-antikink and antikink-kink interaction forces. We also study numerically kink-antikink and antikink-kink collisions and observe a number of interesting phenomena: annihilation of a kink-antikink pair in one topological sector and the production in its place of a pair in another sector; resonance phenomena -- escape windows, despite the absence of vibrational modes in the kink excitation spectra.


title: Local Non-Cooperative Games with Principled Player Selection for Scalable Motion Planning

abstract: Game-theoretic motion planners are a powerful tool for the control of interactive multi-agent robot systems. Indeed, contrary to predict-then-plan paradigms, game-theoretic planners do not ignore the interactive nature of the problem, and simultaneously predict the behaviour of other agents while considering change in one's policy. This, however, comes at the expense of computational complexity, especially as the number of agents considered grows. In fact, planning with more than a handful of agents can quickly become intractable, disqualifying game-theoretic planners as possible candidates for large scale planning. In this paper, we propose a planning algorithm enabling the use of game-theoretic planners in robot systems with a large number of agents. Our planner is based on the reality of locality of information and thus deploys local games with a selected subset of agents in a receding horizon fashion to plan collision avoiding trajectories. We propose five different principled schemes for selecting game participants and compare their collision avoidance performance. We observe that the use of Control Barrier Functions for priority ranking is a potent solution to the player selection problem for motion planning.


title: Consistently constraining $f_{\rm NL}$ with the squeezed lensing bispectrum using consistency relations

abstract: We introduce a non-perturbative method to constrain the amplitude of local-type primordial non-Gaussianity ($f_{\rm NL}$) using squeezed configurations of the CMB lensing convergence and cosmic shear bispectra. First, we use cosmological consistency relations to derive a model for the squeezed limit of angular auto- and cross-bispectra of lensing convergence fields in the presence of $f_{\rm NL}$. Using this model, we perform a Fisher forecast with specifications expected for upcoming CMB lensing measurements from the Simons Observatory and CMB-S4, as well as cosmic shear measurements from a Rubin LSST/Euclid-like experiment. Assuming a minimum multipole $\ell_{\rm min}=10$ and maximum multipole $\ell_{\rm max}=1400$, we forecast $\sigma_{f_{\rm NL}}=175$ ($95$) for Simons Observatory (CMB-S4). Our forecasts improve considerably for an LSST/Euclid-like cosmic shear experiment with three tomographic bins and $\ell_{\rm min}=10$ and $\ell_{\rm max}=1400$ ($5000$) with $\sigma_{f_{\rm NL}}=31$ ($16$). A joint analysis of CMB-S4 lensing and LSST/Euclid-like shear yields little gain over the shear-only forecasts; however, we show that a joint analysis could be useful if the CMB lensing convergence can be reliably reconstructed at larger angular scales than the shear field. The method presented in this work is a novel and robust technique to constrain local primordial non-Gaussianity from upcoming large-scale structure surveys that is completely independent of the galaxy field (and therefore any nuisance parameters such as $b_\phi$), thus complementing existing techniques to constrain $f_{\rm NL}$ using the scale-dependent halo bias.


title: SEGO: Sequential Subgoal Optimization for Mathematical Problem-Solving

abstract: Large Language Models (LLMs) have driven substantial progress in artificial intelligence in recent years, exhibiting impressive capabilities across a wide range of tasks, including mathematical problem-solving. Inspired by the success of subgoal-based methods, we propose a novel framework called \textbf{SE}quential sub\textbf{G}oal \textbf{O}ptimization (SEGO) to enhance LLMs' ability to solve mathematical problems. By establishing a connection between the subgoal breakdown process and the probability of solving problems, SEGO aims to identify better subgoals with theoretical guarantees. Addressing the challenge of identifying suitable subgoals in a large solution space, our framework generates problem-specific subgoals and adjusts them according to carefully designed criteria. Incorporating these optimized subgoals into the policy model training leads to significant improvements in problem-solving performance. We validate SEGO's efficacy through experiments on two benchmarks, GSM8K and MATH, where our approach outperforms existing methods, highlighting the potential of SEGO in AI-driven mathematical problem-solving. Data and code associated with this paper will be available at https://github.com/zhaoxlpku/SEGO


title: Multi-moir\'{e} trilayer graphene: lattice relaxation, electronic structure, and magic angles

abstract: We systematically explore the structural and electronic properties of twisted trilayer graphene systems. In general, these systems are characterized by two twist angles, which lead to two incommensurate moir\'{e} periods. We show that lattice relaxation results in the formation of domains of periodic single-moir\'{e} structures only for twist angles close to the simplest fractions. For the majority of other twist angles, the incommensurate moir\'{e} periods lead to a quasicrystalline structure. We identify experimentally relevant magic angles at which the electronic density of states is sharply peaked and strongly correlated physics is most likely to be realized.


title: An Emulator for Fine-Tuning Large Language Models using Small Language Models

abstract: Widely used language models (LMs) are typically built by scaling up a two-stage training pipeline: a pre-training stage that uses a very large, diverse dataset of text and a fine-tuning (sometimes, 'alignment') stage that uses targeted examples or other specifications of desired behaviors. While it has been hypothesized that knowledge and skills come from pre-training, and fine-tuning mostly filters this knowledge and skillset, this intuition has not been extensively tested. To aid in doing so, we introduce a novel technique for decoupling the knowledge and skills gained in these two stages, enabling a direct answer to the question, "What would happen if we combined the knowledge learned by a large model during pre-training with the knowledge learned by a small model during fine-tuning (or vice versa)?" Using an RL-based framework derived from recent developments in learning from human preferences, we introduce emulated fine-tuning (EFT), a principled and practical method for sampling from a distribution that approximates (or 'emulates') the result of pre-training and fine-tuning at different scales. Our experiments with EFT show that scaling up fine-tuning tends to improve helpfulness, while scaling up pre-training tends to improve factuality. Beyond decoupling scale, we show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Finally, a special case of emulated fine-tuning, which we call LM up-scaling, avoids resource-intensive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models, essentially emulating the result of fine-tuning the large pre-trained model. Up-scaling consistently improves helpfulness and factuality of instruction-following models in the Llama, Llama-2, and Falcon families, without additional hyperparameters or training.


title: AutoMix: Automatically Mixing Language Models

abstract: Large language models (LLMs) are now available from cloud API providers in various sizes and configurations. While this diversity offers a broad spectrum of choices, effectively leveraging the options to optimize computational cost and performance remains challenging. In this work, we present AutoMix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM. Central to AutoMix is a few-shot self-verification mechanism, which estimates the reliability of its own outputs without requiring training. Given that verifications can be noisy, we employ a meta-verifier in AutoMix to refine the accuracy of these assessments. Our experiments using LLAMA2-13B and GPT-4, on five context-grounded reasoning datasets demonstrate that AutoMix surpasses established baselines, improving the incremental benefit per cost by up to 86%. Our code and data are available at https://github.com/automix-llm/automix.


title: PAC Prediction Sets Under Label Shift

abstract: Prediction sets capture uncertainty by predicting sets of labels rather than individual labels, enabling downstream decisions to conservatively account for all plausible outcomes. Conformal inference algorithms construct prediction sets guaranteed to contain the true label with high probability. These guarantees fail to hold in the face of distribution shift, which is precisely when reliable uncertainty quantification can be most useful. We propose a novel algorithm for constructing prediction sets with PAC guarantees in the label shift setting. This method estimates the predicted probabilities of the classes in a target domain, as well as the confusion matrix, then propagates uncertainty in these estimates through a Gaussian elimination algorithm to compute confidence intervals for importance weights. Finally, it uses these intervals to construct prediction sets. We evaluate our approach on five datasets: the CIFAR-10, ChestX-Ray and Entity-13 image datasets, the tabular CDC Heart dataset, and the AGNews text dataset. Our algorithm satisfies the PAC guarantee while producing smaller, more informative, prediction sets compared to several baselines.


title: Pre-optimizing variational quantum eigensolvers with tensor networks

abstract: The variational quantum eigensolver (VQE) is a promising algorithm for demonstrating quantum advantage in the noisy intermediate-scale quantum (NISQ) era. However, optimizing VQE from random initial starting parameters is challenging due to a variety of issues including barren plateaus, optimization in the presence of noise, and slow convergence. While simulating quantum circuits classically is generically difficult, classical computing methods have been developed extensively, and powerful tools now exist to approximately simulate quantum circuits. This opens up various strategies that limit the amount of optimization that needs to be performed on quantum hardware. Here we present and benchmark an approach where we find good starting parameters for parameterized quantum circuits by classically simulating VQE by approximating the parameterized quantum circuit (PQC) as a matrix product state (MPS) with a limited bond dimension. Calling this approach the variational tensor network eigensolver (VTNE), we apply it to the 1D and 2D Fermi-Hubbard model with system sizes that use up to 32 qubits. We find that in 1D, VTNE can find parameters for PQC whose energy error is within 0.5% relative to the ground state. In 2D, the parameters that VTNE finds have significantly lower energy than their starting configurations, and we show that starting VQE from these parameters requires non-trivially fewer operations to come down to a given energy. The higher the bond dimension we use in VTNE, the less work needs to be done in VQE. By generating classically optimized parameters as the initialization for the quantum circuit one can alleviate many of the challenges that plague VQE on quantum computers.


title: Bialgebra structures on flat Lie algebras

abstract: In this paper, we give a complete characterization of Lie bialgebra structures on nondegenerate flat Lie algebras. Furthermore, we illustrate this classification with examples from various classes.


title: Does Your Model Think Like an Engineer? Explainable AI for Bearing Fault Detection with Deep Learning

abstract: Deep Learning has already been successfully applied to analyze industrial sensor data in a variety of relevant use cases. However, the opaque nature of many well-performing methods poses a major obstacle for real-world deployment. Explainable AI (XAI) and especially feature attribution techniques promise to enable insights about how such models form their decision. But the plain application of such methods often fails to provide truly informative and problem-specific insights to domain experts. In this work, we focus on the specific task of detecting faults in rolling element bearings from vibration signals. We propose a novel and domain-specific feature attribution framework that allows us to evaluate how well the underlying logic of a model corresponds with expert reasoning. Utilizing the framework we are able to validate the trustworthiness and to successfully anticipate the generalization ability of different well-performing deep learning models. Our methodology demonstrates how signal processing tools can effectively be used to enhance Explainable AI techniques and acts as a template for similar problems.


title: Late-time tails in nonlinear evolutions of merging black hole binaries

abstract: We study nonlinear evolutions of binary black hole mergers, uncovering power-law contributions generated by the long-range behaviour of the highly-curved dynamical spacetime. The result is achieved by exploiting the strong increase of the tail relevance due to binary eccentricity, recently observed in perturbative evolutions of a small-mass-ratio binary under accurate radiation reaction by Albanesi and collaborators [PhysRevD.108.084037]. We demonstrate the presence of this enhancement even in the nonlinear case of comparable-mass binary mergers in eccentric orbits, using the public RIT waveform catalog. The instantaneous frequency of the simulations displays large oscillations at intermediate to late-times, due to interference terms in the transition between a fast-decaying, constant-frequency quasinormal-mode driven regime, and a power-law, slowly-decaying one. The power-law exponent displays broad convergence with perturbative predictions, although longer and more accurate simulations will be needed to pinpoint the asymptotic value. Our results offer yet another confirmation of the highly predictive power of black hole perturbation theory in the presence of a source, even when applied to nonlinear solutions. The magnitude of the tail signal is within reach of gravitational-wave detectors measurements, unlocking the possibility of observationally investigating an additional set of general relativistic predictions on the long-range dynamics of relaxing compact objects.


title: Demystifying the Myths and Legends of Nonconvex Convergence of SGD

abstract: Stochastic gradient descent (SGD) and its variants are the main workhorses for solving large-scale optimization problems with nonconvex objective functions. Although the convergence of SGDs in the (strongly) convex case is well-understood, their convergence for nonconvex functions stands on weak mathematical foundations. Most existing studies on the nonconvex convergence of SGD show the complexity results based on either the minimum of the expected gradient norm or the functional sub-optimality gap (for functions with extra structural property) by searching the entire range of iterates. Hence the last iterations of SGDs do not necessarily maintain the same complexity guarantee. This paper shows that an $\epsilon$-stationary point exists in the final iterates of SGDs, given a large enough total iteration budget, $T$, not just anywhere in the entire range of iterates -- a much stronger result than the existing one. Additionally, our analyses allow us to measure the density of the $\epsilon$-stationary points in the final iterates of SGD, and we recover the classical $O(\frac{1}{\sqrt{T}})$ asymptotic rate under various existing assumptions on the objective function and the bounds on the stochastic gradient. As a result of our analyses, we addressed certain myths and legends related to the nonconvex convergence of SGD and posed some thought-provoking questions that could set new directions for research.


title: Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding

abstract: The real-world deployment of an autonomous driving system requires its components to run on-board and in real-time, including the motion prediction module that predicts the future trajectories of surrounding traffic participants. Existing agent-centric methods have demonstrated outstanding performance on public benchmarks. However, they suffer from high computational overhead and poor scalability as the number of agents to be predicted increases. To address this problem, we introduce the K-nearest neighbor attention with relative pose encoding (KNARPE), a novel attention mechanism allowing the pairwise-relative representation to be used by Transformers. Then, based on KNARPE we present the Heterogeneous Polyline Transformer with Relative pose encoding (HPTR), a hierarchical framework enabling asynchronous token update during the online inference. By sharing contexts among agents and reusing the unchanged contexts, our approach is as efficient as scene-centric methods, while performing on par with state-of-the-art agent-centric methods. Experiments on Waymo and Argoverse-2 datasets show that HPTR achieves superior performance among end-to-end methods that do not apply expensive post-processing or model ensembling. The code is available at https://github.com/zhejz/HPTR.


title: CLAIR: Evaluating Image Captions with Large Language Models

abstract: The evaluation of machine-generated image captions poses an interesting yet persistent challenge. Effective evaluation measures must consider numerous dimensions of similarity, including semantic relevance, visual structure, object interactions, caption diversity, and specificity. Existing highly-engineered measures attempt to capture specific aspects, but fall short in providing a holistic score that aligns closely with human judgments. Here, we propose CLAIR, a novel method that leverages the zero-shot language modeling capabilities of large language models (LLMs) to evaluate candidate captions. In our evaluations, CLAIR demonstrates a stronger correlation with human judgments of caption quality compared to existing measures. Notably, on Flickr8K-Expert, CLAIR achieves relative correlation improvements over SPICE of 39.6% and over image-augmented methods such as RefCLIP-S of 18.3%. Moreover, CLAIR provides noisily interpretable results by allowing the language model to identify the underlying reasoning behind its assigned score. Code is available at https://davidmchan.github.io/clair/


title: CCIL: Continuity-based Data Augmentation for Corrective Imitation Learning

abstract: We present a new technique to enhance the robustness of imitation learning methods by generating corrective data to account for compounding errors and disturbances. While existing methods rely on interactive expert labeling, additional offline datasets, or domain-specific invariances, our approach requires minimal additional assumptions beyond access to expert data. The key insight is to leverage local continuity in the environment dynamics to generate corrective labels. Our method first constructs a dynamics model from the expert demonstration, encouraging local Lipschitz continuity in the learned model. In locally continuous regions, this model allows us to generate corrective labels within the neighborhood of the demonstrations but beyond the actual set of states and actions in the dataset. Training on this augmented data enhances the agent's ability to recover from perturbations and deal with compounding errors. We demonstrate the effectiveness of our generated labels through experiments in a variety of robotics domains in simulation that have distinct forms of continuity and discontinuity, including classic control problems, drone flying, navigation with high-dimensional sensor observations, legged locomotion, and tabletop manipulation.


title: Frozen Transformers in Language Models Are Effective Visual Encoder Layers

abstract: This paper reveals that large language models (LLMs), despite being trained solely on textual data, are surprisingly strong encoders for purely visual tasks in the absence of language. Even more intriguingly, this can be achieved by a simple yet previously overlooked strategy -- employing a frozen transformer block from pre-trained LLMs as a constituent encoder layer to directly process visual tokens. Our work pushes the boundaries of leveraging LLMs for computer vision tasks, significantly departing from conventional practices that typically necessitate a multi-modal vision-language setup with associated language prompts, inputs, or outputs. We demonstrate that our approach consistently enhances performance across a diverse range of tasks, encompassing pure 2D and 3D visual recognition tasks (e.g., image and point cloud classification), temporal modeling tasks (e.g., action recognition), non-semantic tasks (e.g., motion forecasting), and multi-modal tasks (e.g., 2D/3D visual question answering and image-text retrieval). Such improvements are a general phenomenon, applicable to various types of LLMs (e.g., LLaMA and OPT) and different LLM transformer blocks. We additionally propose the information filtering hypothesis to explain the effectiveness of pre-trained LLMs in visual encoding -- the pre-trained LLM transformer blocks discern informative visual tokens and further amplify their effect. This hypothesis is empirically supported by the observation that the feature activation, after training with LLM transformer blocks, exhibits a stronger focus on relevant regions. We hope that our work inspires new perspectives on utilizing LLMs and deepening our understanding of their underlying mechanisms. Code is available at https://github.com/ziqipang/LM4VisualEncoding.


title: FSD: Fast Self-Supervised Single RGB-D to Categorical 3D Objects

abstract: In this work, we address the challenging task of 3D object recognition without the reliance on real-world 3D labeled data. Our goal is to predict the 3D shape, size, and 6D pose of objects within a single RGB-D image, operating at the category level and eliminating the need for CAD models during inference. While existing self-supervised methods have made strides in this field, they often suffer from inefficiencies arising from non-end-to-end processing, reliance on separate models for different object categories, and slow surface extraction during the training of implicit reconstruction models; thus hindering both the speed and real-world applicability of the 3D recognition process. Our proposed method leverages a multi-stage training pipeline, designed to efficiently transfer synthetic performance to the real-world domain. This approach is achieved through a combination of 2D and 3D supervised losses during the synthetic domain training, followed by the incorporation of 2D supervised and 3D self-supervised losses on real-world data in two additional learning stages. By adopting this comprehensive strategy, our method successfully overcomes the aforementioned limitations and outperforms existing self-supervised 6D pose and size estimation baselines on the NOCS test-set with a 16.4% absolute improvement in mAP for 6D pose estimation while running in near real-time at 5 Hz.


title: Variational Inference for SDEs Driven by Fractional Noise

abstract: We present a novel variational framework for performing inference in (neural) stochastic differential equations (SDEs) driven by Markov-approximate fractional Brownian motion (fBM). SDEs offer a versatile tool for modeling real-world continuous-time dynamic systems with inherent noise and randomness. Combining SDEs with the powerful inference capabilities of variational methods, enables the learning of representative function distributions through stochastic gradient descent. However, conventional SDEs typically assume the underlying noise to follow a Brownian motion (BM), which hinders their ability to capture long-term dependencies. In contrast, fractional Brownian motion (fBM) extends BM to encompass non-Markovian dynamics, but existing methods for inferring fBM parameters are either computationally demanding or statistically inefficient. In this paper, building upon the Markov approximation of fBM, we derive the evidence lower bound essential for efficient variational inference of posterior path measures, drawing from the well-established field of stochastic analysis. Additionally, we provide a closed-form expression to determine optimal approximation coefficients. Furthermore, we propose the use of neural networks to learn the drift, diffusion and control terms within our variational posterior, leading to the variational training of neural-SDEs. In this framework, we also optimize the Hurst index, governing the nature of our fractional noise. Beyond validation on synthetic data, we contribute a novel architecture for variational latent video prediction,-an approach that, to the best of our knowledge, enables the first variational neural-SDE application to video perception.


title: On the Hidden Waves of Image

abstract: In this paper, we introduce an intriguing phenomenon-the successful reconstruction of images using a set of one-way wave equations with hidden and learnable speeds. Each individual image corresponds to a solution with a unique initial condition, which can be computed from the original image using a visual encoder (e.g., a convolutional neural network). Furthermore, the solution for each image exhibits two noteworthy mathematical properties: (a) it can be decomposed into a collection of special solutions of the same one-way wave equations that are first-order autoregressive, with shared coefficient matrices for autoregression, and (b) the product of these coefficient matrices forms a diagonal matrix with the speeds of the wave equations as its diagonal elements. We term this phenomenon hidden waves, as it reveals that, although the speeds of the set of wave equations and autoregressive coefficient matrices are latent, they are both learnable and shared across images. This represents a mathematical invariance across images, providing a new mathematical perspective to understand images.


title: Training Dynamics of Deep Network Linear Regions

abstract: The study of Deep Network (DN) training dynamics has largely focused on the evolution of the loss function, evaluated on or around train and test set data points. In fact, many DN phenomenon were first introduced in literature with that respect, e.g., double descent, grokking. In this study, we look at the training dynamics of the input space partition or linear regions formed by continuous piecewise affine DNs, e.g., networks with (leaky)ReLU nonlinearities. First, we present a novel statistic that encompasses the local complexity (LC) of the DN based on the concentration of linear regions inside arbitrary dimensional neighborhoods around data points. We observe that during training, the LC around data points undergoes a number of phases, starting with a decreasing trend after initialization, followed by an ascent and ending with a final descending trend. Using exact visualization methods, we come across the perplexing observation that during the final LC descent phase of training, linear regions migrate away from training and test samples towards the decision boundary, making the DN input-output nearly linear everywhere else. We also observe that the different LC phases are closely related to the memorization and generalization performance of the DN, especially during grokking.


title: HumanTOMATO: Text-aligned Whole-body Motion Generation

abstract: This work targets a novel text-driven whole-body motion generation task, which takes a given textual description as input and aims at generating high-quality, diverse, and coherent facial expressions, hand gestures, and body motions simultaneously. Previous works on text-driven motion generation tasks mainly have two limitations: they ignore the key role of fine-grained hand and face controlling in vivid whole-body motion generation, and lack a good alignment between text and motion. To address such limitations, we propose a Text-aligned whOle-body Motion generATiOn framework, named HumanTOMATO, which is the first attempt to our knowledge towards applicable holistic motion generation in this research area. To tackle this challenging task, our solution includes two key designs: (1) a Holistic Hierarchical VQ-VAE (aka H$^2$VQ) and a Hierarchical-GPT for fine-grained body and hand motion reconstruction and generation with two structured codebooks; and (2) a pre-trained text-motion-alignment model to help generated motion align with the input textual description explicitly. Comprehensive experiments verify that our model has significant advantages in both the quality of generated motions and their alignment with text.


title: Predicting a Protein's Stability under a Million Mutations

abstract: Stabilizing proteins is a foundational step in protein engineering. However, the evolutionary pressure of all extant proteins makes identifying the scarce number of mutations that will improve thermodynamic stability challenging. Deep learning has recently emerged as a powerful tool for identifying promising mutations. Existing approaches, however, are computationally expensive, as the number of model inferences scales with the number of mutations queried. Our main contribution is a simple, parallel decoding algorithm. Our Mutate Everything is capable of predicting the effect of all single and double mutations in one forward pass. It is even versatile enough to predict higher-order mutations with minimal computational overhead. We build Mutate Everything on top of ESM2 and AlphaFold, neither of which were trained to predict thermodynamic stability. We trained on the Mega-Scale cDNA proteolysis dataset and achieved state-of-the-art performance on single and higher-order mutations on S669, ProTherm, and ProteinGym datasets. Code is available at https://github.com/jozhang97/MutateEverything


title: SymTrees and Multi-Sector QFTs

abstract: The global symmetries of a $D$-dimensional QFT can, in many cases, be captured in terms of a $(D+1)$-dimensional symmetry topological field theory (SymTFT). In this work we construct a $(D+1)$-dimensional theory which governs the symmetries of QFTs with multiple sectors which have connected correlators that admit a decoupling limit. The associated symmetry field theory decomposes into a SymTree, namely a treelike structure of SymTFTs fused along possibly non-topological junctions. In string-realized multi-sector QFTs, these junctions are smoothed out in the extra-dimensional geometry, as we demonstrate in examples. We further use this perspective to study the fate of higher-form symmetries in the context of holographic large $M$ averaging where the topological sectors of different large $M$ replicas become dressed by additional extended operators associated with the SymTree.


title: Improved Pairwise Measurement-Based Surface Code

abstract: We devise a new realization of the surface code on a rectangular lattice of qubits utilizing single-qubit and nearest-neighbor two-qubit Pauli measurements and three auxiliary qubits per plaquette. This realization gains substantial advantages over prior pairwise measurement-based realizations of the surface code. It has a short operation period of 4 steps and our performance analysis for a standard circuit noise model yields a high fault-tolerance threshold of approximately $0.66\% $. The syndrome extraction circuits avoid bidirectional hook errors, so we can achieve full code distance by choosing appropriate boundary conditions. We also construct variants of the syndrome extraction circuits that entirely prevent hook errors, at the cost of larger circuit depth. This achieves full distance regardless of boundary conditions, with only a modest decrease in the threshold. Furthermore, we propose an efficient strategy for dealing with dead components (qubits and measurements) in our surface code realization, which can be adopted more generally for other surface code realizations. This new surface code realization is highly optimized for Majorana-based hardware, accounting for constraints imposed by layouts and the implementation of measurements, making it competitive with the recently proposed Floquet codes.


title: Putting the Object Back into Video Object Segmentation

abstract: We present Cutie, a video object segmentation (VOS) network with object-level memory reading, which puts the object representation from memory back into the video object segmentation result. Recent works on VOS employ bottom-up pixel-level memory reading which struggles due to matching noise, especially in the presence of distractors, resulting in lower performance in more challenging data. In contrast, Cutie performs top-down object-level memory reading by adapting a small set of object queries. Via those, it interacts with the bottom-up pixel features iteratively with a query-based object transformer (qt, hence Cutie). The object queries act as a high-level summary of the target object, while high-resolution feature maps are retained for accurate segmentation. Together with foreground-background masked attention, Cutie cleanly separates the semantics of the foreground object from the background. On the challenging MOSE dataset, Cutie improves by 8.7 J&F over XMem with a similar running time and improves by 4.2 J&F over DeAOT while being three times faster. Code is available at: https://hkchengrex.github.io/Cutie


title: Majority rule as a unique voting method in elections with multiple candidates

abstract: May's classical theorem states that in a single-winner choose-one voting system with just two candidates, majority rule is the only social choice function satisfying anonimity, neutrality and positive responsiveness axiom. Anonimity and neutrality are usually regarded as very natural constraints on the social choice function. Positive responsiveness, on the other hand, is sometimes deemed too strong of an axiom, which stimulates further search for less stringent conditions. One viable substitute is Gerhard J. Woeginger's "reducibility to subsocieties". We demonstrate that the condition generalizes to more than two candidates and, consequently, characterizes majority rule for elections with multiple candidates.


title: On the Recurrence Formula for Fixed Points of the Josephus Function

abstract: In this paper, we provide a comprehensive solution to the open problem regarding the existence of a recurrence formula for computing fixed points of the Josephus function precisely when the reduction constant is three. Incorporating this formula into recursive algorithms significantly improves addressing the Josephus problem, particularly for large inputs.


title: Enabling energy-Efficient object detection with surrogate gradient descent in spiking neural networks

abstract: Spiking Neural Networks (SNNs) are a biologically plausible neural network model with significant advantages in both event-driven processing and spatio-temporal information processing, rendering SNNs an appealing choice for energyefficient object detection. However, the non-differentiability of the biological neuronal dynamics model presents a challenge during the training of SNNs. Furthermore, a suitable decoding strategy for object detection in SNNs is currently lacking. In this study, we introduce the Current Mean Decoding (CMD) method, which solves the regression problem to facilitate the training of deep SNNs for object detection tasks. Based on the gradient surrogate and CMD, we propose the SNN-YOLOv3 model for object detection. Our experiments demonstrate that SNN-YOLOv3 achieves a remarkable performance with an mAP of 61.87% on the PASCAL VOC dataset, requiring only 6 time steps. Compared to SpikingYOLO, we have managed to increase mAP by nearly 10% while reducing energy consumption by two orders of magnitude.


title: A survey of manifold learning and its applications for multimedia

abstract: Manifold learning is an emerging research domain of machine learning. In this work, we give an introduction into manifold learning and how it is employed for important application fields in multimedia.


title: Spec-NeRF: Multi-spectral Neural Radiance Fields

abstract: We propose Multi-spectral Neural Radiance Fields(Spec-NeRF) for jointly reconstructing a multispectral radiance field and spectral sensitivity functions(SSFs) of the camera from a set of color images filtered by different filters. The proposed method focuses on modeling the physical imaging process, and applies the estimated SSFs and radiance field to synthesize novel views of multispectral scenes. In this method, the data acquisition requires only a low-cost trichromatic camera and several off-the-shelf color filters, making it more practical than using specialized 3D scanning and spectral imaging equipment. Our experiments on both synthetic and real scenario datasets demonstrate that utilizing filtered RGB images with learnable NeRF and SSFs can achieve high fidelity and promising spectral reconstruction while retaining the inherent capability of NeRF to comprehend geometric structures. Code is available at https://github.com/CPREgroup/SpecNeRF-v2.


title: Monotonicity Failure in Ranked Choice Voting -- Necessary and Sufficient Conditions for 3-Candidate Elections

abstract: Ranked choice voting is vulnerable to monotonicity failure - a voting failure where a candidate is cost an election due to losing voter preference or granted an election due to gaining voter preference. Despite increasing use of ranked choice voting at the time of writing of this paper, the frequency of monotonicity failure is still a very open question. This paper builds on previous work to develop conditions which can be used to test if it's possible that monotonicity failure has happened in a 3-candidate ranked choice voting election.


title: Enhancing Health Data Interoperability with Large Language Models: A FHIR Study

abstract: In this study, we investigated the ability of the large language model (LLM) to enhance healthcare data interoperability. We leveraged the LLM to convert clinical texts into their corresponding FHIR resources. Our experiments, conducted on 3,671 snippets of clinical text, demonstrated that the LLM not only streamlines the multi-step natural language processing and human calibration processes but also achieves an exceptional accuracy rate of over 90% in exact matches when compared to human annotations.


title: Wave-informed dictionary learning for high-resolution imaging in complex media

abstract: We propose an approach for imaging in scattering media when large and diverse data sets are available. It has two steps. Using a dictionary learning algorithm the first step estimates the true Green's function vectors as columns in an unordered sensing matrix. The array data comes from many sparse sets of sources whose location and strength are not known to us. In the second step, the columns of the estimated sensing matrix are ordered for imaging using Multi-Dimensional Scaling with connectivity information derived from cross-correlations of its columns, as in time reversal. For these two steps to work together we need data from large arrays of receivers so the columns of the sensing matrix are incoherent for the first step, as well as from sub-arrays so that they are coherent enough to obtain the connectivity needed in the second step. Through simulation experiments, we show that the proposed approach is able to provide images in complex media whose resolution is that of a homogeneous medium.


title: DFT Studies of 2D Materials Inspired by Lie Algebras

abstract: Inspired by the root systems of Lie algebras of rank 2, we propose a mathematical method to engineer new 2D materials with double periodic structures tessellating the plane. Concretely, we investigate two geometries relaying on squares and hexagons exhibiting D4 D4 and D6 D6 dihedral group invariances, respectively. Due to lack of empirical verifications of such double configurations, we provide a numerical investigation by help of the open source quantum espresso. Motivated by hybrid structures of graphene, silicene, germanene, we investigate two models involving D4 D4 and D6 D6 dihedral symmetries which we refer to as Si4Ge4 and Si6C6 compounds, respectively. For simplicities, we study only the opto-electronic physical properties by applying an electromagnetic source propagating in linear and isotropic mediums. We believe that the Lie algebra inspiration of such 2D material studies, via density functional theory techniques, could open new roads to think about higher dimensional cases by implementing generalized Cartan matrices.


title: Mod\'elisation des performances des structures lasers \`a base des chalcog\'enures de m\'etaux de transition: MoS$_2$/WSe$_2$

abstract: Two-dimensional semiconductors, known as Transition Metal Dichalcogenides (TMDCs), are of great interest among many materials due to their unique 2D characteristics, including exceptional electronic and optical properties. These compounds belong to the family of lamellar materials having a general formula of type $MX_2$, where M represents a transition metal belonging to one of groups IV, V, or VI of Mendeleev's periodic table, and X is a chalcogenide. These materials exhibit fundamental properties that are qualitatively different from their three-dimensional (3D) counterparts, often relating to their electronic and phononic band structure. The in-depth study of these materials has revealed remarkable phenomena, such as two-dimensional quantum phase transitions and optoelectronic phenomena related to electronic "valleys", which exploit the wavevector selectivity of the emission and the absorption of photons. Unlike graphene, materials such as MoS$_2$, MoSe$_2$, WS$_2$, and WSe$_2$ are semiconductors that have a bandgap varying from 1.2 to 1.9 electron volts (eV), depending on the number of layers. This characteristic makes them suitable for use as active elements in optoelectronic devices. Additionally, their two-dimensional nature and mechanical strength enable the realization of quantum phase transitions and electronic components on flexible substrates. It is also possible to create quantum wells based on MoS$_2$, with materials such as MoSe$_2$, WS$_2$, and WSe$_2$, depending on the band shift between the components of the quantum well. This work aims to propose a theoretical and numerical model for these materials, which will make it possible to determine the electronic band structure as well as the optical gain in an active zone composed of two-dimensional semiconductors. This research is of great importance for evaluating the performance of active zones in laser devices based on these materials.


title: Redheffer: Trig to Quantum Error Bounds

abstract: In the existing literature, the Redheffer inequality is typically proven using mathematical induction. In this short paper, we present a straightforward proof of this inequality by leveraging trigonometric substitution. We then extend the Redheffer inequality by introducing an exponent factor, aiming for the sharpest possible refinement. Notably, when the exponent is 2, our findings have implications for quantum error correction in the context of quantum phase estimation.


title: Dimensions of Disagreement: Unpacking Divergence and Misalignment in Cognitive Science and Artificial Intelligence

abstract: The increasing prevalence of artificial agents creates a correspondingly increasing need to manage disagreements between humans and artificial agents, as well as between artificial agents themselves. Considering this larger space of possible agents exposes an opportunity for furthering our understanding of the nature of disagreement: past studies in psychology have often cast disagreement as two agents forming diverging evaluations of the same object, but disagreement can also arise from differences in how agents represent that object. AI research on human-machine alignment and recent work in computational cognitive science have focused on this latter kind of disagreement, and have developed tools that can be used to quantify the extent of representational overlap between agents. Understanding how divergence and misalignment interact to produce disagreement, and how resolution strategies depend on this interaction, is key to promoting effective collaboration between diverse types of agents.


title: Comprehensive Multimodal Segmentation in Medical Imaging: Combining YOLOv8 with SAM and HQ-SAM Models

abstract: This paper introduces a comprehensive approach for segmenting regions of interest (ROI) in diverse medical imaging datasets, encompassing ultrasound, CT scans, and X-ray images. The proposed method harnesses the capabilities of the YOLOv8 model for approximate boundary box detection across modalities, alongside the Segment Anything Model (SAM) and High Quality (HQ) SAM for fully automatic and precise segmentation. To generate boundary boxes, the YOLOv8 model was trained using a limited set of 100 images and masks from each modality. The results obtained from our approach are extensively computed and analyzed, demonstrating its effectiveness and potential in medical image analysis. Various evaluation metrics, including precision, recall, F1 score, and Dice Score, were employed to quantify the accuracy of the segmentation results. A comparative analysis was conducted to assess the individual and combined performance of the YOLOv8, YOLOv8+SAM, and YOLOv8+HQ-SAM models. The results indicate that the SAM model performs better than the other two models, exhibiting higher segmentation accuracy and overall performance. While HQ-SAM offers potential advantages, its incremental gains over the standard SAM model may not justify the additional computational cost. The YOLOv8+SAM model shows promise for enhancing medical image segmentation and its clinical implications.


title: Zero-shot Learning of Drug Response Prediction for Preclinical Drug Screening

abstract: Conventional deep learning methods typically employ supervised learning for drug response prediction (DRP). This entails dependence on labeled response data from drugs for model training. However, practical applications in the preclinical drug screening phase demand that DRP models predict responses for novel compounds, often with unknown drug responses. This presents a challenge, rendering supervised deep learning methods unsuitable for such scenarios. In this paper, we propose a zero-shot learning solution for the DRP task in preclinical drug screening. Specifically, we propose a Multi-branch Multi-Source Domain Adaptation Test Enhancement Plug-in, called MSDA. MSDA can be seamlessly integrated with conventional DRP methods, learning invariant features from the prior response data of similar drugs to enhance real-time predictions of unlabeled compounds. We conducted experiments using the GDSCv2 and CellMiner datasets. The results demonstrate that MSDA efficiently predicts drug responses for novel compounds, leading to a general performance improvement of 5-10\% in the preclinical drug screening phase. The significance of this solution resides in its potential to accelerate the drug discovery process, improve drug candidate assessment, and facilitate the success of drug discovery.


title: Parking Spot Classification based on surround view camera system

abstract: Surround-view fisheye cameras are commonly used for near-field sensing in automated driving scenarios, including urban driving and auto valet parking. Four fisheye cameras, one on each side, are sufficient to cover 360{\deg} around the vehicle capturing the entire near-field region. Based on surround view cameras, there has been much research on parking slot detection with main focus on the occupancy status in recent years, but little work on whether the free slot is compatible with the mission of the ego vehicle or not. For instance, some spots are handicap or electric vehicles accessible only. In this paper, we tackle parking spot classification based on the surround view camera system. We adapt the object detection neural network YOLOv4 with a novel polygon bounding box model that is well-suited for various shaped parking spaces, such as slanted parking slots. To the best of our knowledge, we present the first detailed study on parking spot detection and classification on fisheye cameras for auto valet parking scenarios. The results prove that our proposed classification approach is effective to distinguish between regular, electric vehicle, and handicap parking spots.


title: Combined Experimental and Theoretical Studies on Iodine Capture of Zr-based Metal-Organic Frameworks: Effect of N-functionalization and Adsorption Mechanism

abstract: The potential leakage of nuclear waste, especially radioiodine, is a major safety concerning issue around the world. To remove radioiodine from nuclear waste efficiently, there is an urgent demand for adsorbents that possess both high stability and strong adsorption affinity for environmental remediation. Herein, two Zr-based metal-organic frameworks (Zr-MOFs) and their N-functionalized analogues have been synthesized and researched for iodine adsorption in both vapours and solutions. It was found that Zr-MOFs with N-enriched ligands (e.g., pyridine and amino) exhibited the faster iodine adsorption rate and the higher iodine uptake amount (e.g., reaching adsorption equilibrium within 4 hours with the removal rate of above 85% for iodine solution adsorption) than their unfunctionalized counterparts (UiO-66 and UiO-67). The critical role played by N-enriched groups in enhancing iodine adsorption has been revealed through versatile model fittings, X-ray photoelectron spectroscopy (XPS) and Raman spectroscopy characterizations, as well as density functional theory (DFT) calculations. Compared to those in amino-group, the N-atoms in pyridine-groups showed a deeper affinity towards iodine molecules. Remarkably, the N-enriched UiOs adsorbents also exhibited good recyclability, especially UiO-66-PYDC and UiO-67-NH2 could maintain the removal efficiency of 89.05% and 85.49% after four adsorption-desorption recycling tests. With the strong iodine uptake affinity and outstanding regeneration performance, this work has systematically investigated the impact of N-functionalization on the enhanced performance for iodine capture by using the N-enriched UiO MOFs as promising adsorbents, providing an insightful guideline into the physical chemistry of adsorption mechanism behind the radioiodine capture.


title: Adaptive Dynamic Programming for Energy-Efficient Base Station Cell Switching

abstract: Energy saving in wireless networks is growing in importance due to increasing demand for evolving new-gen cellular networks, environmental and regulatory concerns, and potential energy crises arising from geopolitical tensions. In this work, we propose an approximate dynamic programming (ADP)-based method coupled with online optimization to switch on/off the cells of base stations to reduce network power consumption while maintaining adequate Quality of Service (QoS) metrics. We use a multilayer perceptron (MLP) given each state-action pair to predict the power consumption to approximate the value function in ADP for selecting the action with optimal expected power saved. To save the largest possible power consumption without deteriorating QoS, we include another MLP to predict QoS and a long short-term memory (LSTM) for predicting handovers, incorporated into an online optimization algorithm producing an adaptive QoS threshold for filtering cell switching actions based on the overall QoS history. The performance of the method is evaluated using a practical network simulator with various real-world scenarios with dynamic traffic patterns.


title: Document-Level Relation Extraction with Relation Correlation Enhancement

abstract: Document-level relation extraction (DocRE) is a task that focuses on identifying relations between entities within a document. However, existing DocRE models often overlook the correlation between relations and lack a quantitative analysis of relation correlations. To address this limitation and effectively capture relation correlations in DocRE, we propose a relation graph method, which aims to explicitly exploit the interdependency among relations. Firstly, we construct a relation graph that models relation correlations using statistical co-occurrence information derived from prior relation knowledge. Secondly, we employ a re-weighting scheme to create an effective relation correlation matrix to guide the propagation of relation information. Furthermore, we leverage graph attention networks to aggregate relation embeddings. Importantly, our method can be seamlessly integrated as a plug-and-play module into existing models. Experimental results demonstrate that our approach can enhance the performance of multi-relation extraction, highlighting the effectiveness of considering relation correlations in DocRE.


title: Conversational Financial Information Retrieval Model (ConFIRM)

abstract: With the exponential growth in large language models (LLMs), leveraging their emergent properties for specialized domains like finance merits exploration. However, regulated fields such as finance pose unique constraints, requiring domain-optimized frameworks. We present ConFIRM, an LLM-based conversational financial information retrieval model tailored for query intent classification and knowledge base labeling. ConFIRM comprises two modules: 1) a method to synthesize finance domain-specific question-answer pairs, and 2) evaluation of parameter efficient fine-tuning approaches for the query classification task. We generate a dataset of over 4000 samples, assessing accuracy on a separate test set. ConFIRM achieved over 90% accuracy, essential for regulatory compliance. ConFIRM provides a data-efficient solution to extract precise query intent for financial dialog systems.


title: Are Large Language Models Geospatially Knowledgeable?

abstract: Despite the impressive performance of Large Language Models (LLM) for various natural language processing tasks, little is known about their comprehension of geographic data and related ability to facilitate informed geospatial decision-making. This paper investigates the extent of geospatial knowledge, awareness, and reasoning abilities encoded within such pretrained LLMs. With a focus on autoregressive language models, we devise experimental approaches related to (i) probing LLMs for geo-coordinates to assess geospatial knowledge, (ii) using geospatial and non-geospatial prepositions to gauge their geospatial awareness, and (iii) utilizing a multidimensional scaling (MDS) experiment to assess the models' geospatial reasoning capabilities and to determine locations of cities based on prompting. Our results confirm that it does not only take larger, but also more sophisticated LLMs to synthesize geospatial knowledge from textual information. As such, this research contributes to understanding the potential and limitations of LLMs in dealing with geospatial information.


title: An Interactive Web-Based System for Creating Single Panel Cartoons with Visually Valid Compositions

abstract: The creation of cartoon-based stories (comics) requires a lot of creativity and hard work for naive users. We observe that single-panel cartoons are the building blocks of any comic story. To develop a strong comic story, it is critical to obtain visually valid single panels. In this work, we have developed a methodology to guarantee the placement of characters to obtain a valid cartoon frame based on the methods used by professional cartoonists. Using this methodology, we have developed a web-based system to create single-panel cartoons from a given set of character images. We have made this system available in GitHub as open-source so that this basic single-panel cartoon can be used as infrastructure to develop more complex structures. Our web-based system for single-panel cartoons can be viewed at http://storytelling.viz.tamu.edu.


title: Progressively Efficient Learning

abstract: Assistant AI agents should be capable of rapidly acquiring novel skills and adapting to new user preferences. Traditional frameworks like imitation learning and reinforcement learning do not facilitate this capability because they support only low-level, inefficient forms of communication. In contrast, humans communicate with progressive efficiency by defining and sharing abstract intentions. Reproducing similar capability in AI agents, we develop a novel learning framework named Communication-Efficient Interactive Learning (CEIL). By equipping a learning agent with an abstract, dynamic language and an intrinsic motivation to learn with minimal communication effort, CEIL leads to emergence of a human-like pattern where the learner and the teacher communicate progressively efficiently by exchanging increasingly more abstract intentions. CEIL demonstrates impressive performance and communication efficiency on a 2D MineCraft domain featuring long-horizon decision-making tasks. Agents trained with CEIL quickly master new tasks, outperforming non-hierarchical and hierarchical imitation learning by up to 50% and 20% in absolute success rate, respectively, given the same number of interactions with the teacher. Especially, the framework performs robustly with teachers modeled after human pragmatic communication behavior.


title: Metacognitive threshold: a computational account

abstract: This paper will explore ways of computationally accounting for the metacognitive threshold -- the minimum amount of stimulus needed for a mental state to be perceived -- and discuss potential cognitive mechanisms by which this threshold can be influenced through metacognitive training and meditation.


title: Software Metadata Classification based on Generative Artificial Intelligence

abstract: This paper presents a novel approach to enhance the performance of binary code comment quality classification models through the application of Generative Artificial Intelligence (AI). By leveraging the OpenAI API, a dataset comprising 1239 newly generated code-comment pairs, extracted from various GitHub repositories and open-source projects, has been labelled as "Useful" or "Not Useful", and integrated into the existing corpus of 9048 pairs in the C programming language. Employing a cutting-edge Large Language Model Architecture, the generated dataset demonstrates notable improvements in model accuracy. Specifically, when incorporated into the Support Vector Machine (SVM) model, a 6% increase in precision is observed, rising from 0.79 to 0.85. Additionally, the Artificial Neural Network (ANN) model exhibits a 1.5% increase in recall, climbing from 0.731 to 0.746. This paper sheds light on the potential of Generative AI in augmenting code comment quality classification models. The results affirm the effectiveness of this methodology, indicating its applicability in broader contexts within software development and quality assurance domains. The findings underscore the significance of integrating generative techniques to advance the accuracy and efficacy of machine learning models in practical software engineering scenarios.


title: A Critical Survey on Fairness Benefits of Explainable AI

abstract: In this critical survey, we analyze typical claims on the relationship between explainable AI (XAI) and fairness to disentangle the multidimensional relationship between these two concepts. Based on a systematic literature review and a subsequent qualitative content analysis, we identify seven archetypal claims from 175 scientific articles on the alleged fairness benefits of XAI. We present crucial caveats with respect to these claims and provide an entry point for future discussions around the potentials and limitations of XAI for specific fairness desiderata. Importantly, we notice that claims are often (i) vague and simplistic, (ii) lacking normative grounding, or (iii) poorly aligned with the actual capabilities of XAI. We suggest to conceive XAI not as an ethical panacea but as one of many tools to approach the multidimensional, sociotechnical challenge of algorithmic fairness. Moreover, when making a claim about XAI and fairness, we emphasize the need to be more specific about what kind of XAI method is used, which fairness desideratum it refers to, how exactly it enables fairness, and who is the stakeholder that benefits from XAI.


title: LoBaSS: Gauging Learnability in Supervised Fine-tuning Data

abstract: Supervised Fine-Tuning (SFT) serves as a crucial phase in aligning Large Language Models (LLMs) to specific task prerequisites. The selection of fine-tuning data profoundly influences the model's performance, whose principle is traditionally grounded in data quality and distribution. In this paper, we introduce a new dimension in SFT data selection: learnability. This new dimension is motivated by the intuition that SFT unlocks capabilities acquired by a LLM during the pretraining phase. Given that different pretrained models have disparate capabilities, the SFT data appropriate for one may not suit another. Thus, we introduce the term learnability to define the suitability of data for effective learning by the model. We present the Loss Based SFT Data Selection (LoBaSS) method, utilizing data learnability as the principal criterion for the selection SFT data. This method provides a nuanced approach, allowing the alignment of data selection with inherent model capabilities, ensuring optimal compatibility and learning efficiency. In experimental comparisons involving 7B and 13B models, our LoBaSS method is able to surpass full-data fine-tuning at merely 6% of the total training data. When employing 16.7% of the data, LoBaSS harmonizes the model's capabilities across conversational and mathematical domains, proving its efficacy and adaptability.


title: Application of Machine Learning Based Top Quark and W Jet Tagging to Hadronic Four-Top Final States Induced by SM as well as BSM Processes

abstract: We apply gradient boosting machine learning techniques to the problem of hadronic jet substructure recognition using classical subjettiness variables available within a common parameterized detector simulation package DELPHES. Per-jet tagging classification is being explored. Jets produced in simulated proton-proton collisions are identified as consistent with the hypothesis of coming from the decay of a top quark or a W boson and are used to reconstruct the mass of a hypothetical scalar resonance decaying to a pair of top quarks in events where in total four top quarks are produced. Results are compared to the case of a simple cut-based tagging technique for the stacked histograms of a mixture of a Standard Model as well as the new physics process.


title: Detecting Speech Abnormalities with a Perceiver-based Sequence Classifier that Leverages a Universal Speech Model

abstract: We propose a Perceiver-based sequence classifier to detect abnormalities in speech reflective of several neurological disorders. We combine this classifier with a Universal Speech Model (USM) that is trained (unsupervised) on 12 million hours of diverse audio recordings. Our model compresses long sequences into a small set of class-specific latent representations and a factorized projection is used to predict different attributes of the disordered input speech. The benefit of our approach is that it allows us to model different regions of the input for different classes and is at the same time data efficient. We evaluated the proposed model extensively on a curated corpus from the Mayo Clinic. Our model outperforms standard transformer (80.9%) and perceiver (81.8%) models and achieves an average accuracy of 83.1%. With limited task-specific data, we find that pretraining is important and surprisingly pretraining with the unrelated automatic speech recognition (ASR) task is also beneficial. Encodings from the middle layers provide a mix of both acoustic and phonetic information and achieve best prediction results compared to just using the final layer encodings (83.1% vs. 79.6%). The results are promising and with further refinements may help clinicians detect speech abnormalities without needing access to highly specialized speech-language pathologists.


title: Compositional preference models for aligning LMs

abstract: As language models (LMs) become more capable, it is increasingly important to align them with human preferences. However, the dominant paradigm for training Preference Models (PMs) for that purpose suffers from fundamental limitations, such as lack of transparency and scalability, along with susceptibility to overfitting the preference dataset. We propose Compositional Preference Models (CPMs), a novel PM framework that decomposes one global preference assessment into several interpretable features, obtains scalar scores for these features from a prompted LM, and aggregates these scores using a logistic regression classifier. Through these simple steps, CPMs allow to control which properties of the preference data are used to train the preference model and to build it based on features that are believed to underlie the human preference judgment. Our experiments show that CPMs not only improve generalization and are more robust to overoptimization than standard PMs, but also that best-of-n samples obtained using CPMs tend to be preferred over samples obtained using conventional PMs. Overall, our approach demonstrates the benefits of endowing PMs with priors about which features determine human preferences while relying on LM capabilities to extract those features in a scalable and robust way.


title: H2O Open Ecosystem for State-of-the-art Large Language Models

abstract: Large Language Models (LLMs) represent a revolution in AI. However, they also pose many significant risks, such as the presence of biased, private, copyrighted or harmful text. For this reason we need open, transparent and safe solutions. We introduce a complete open-source ecosystem for developing and testing LLMs. The goal of this project is to boost open alternatives to closed-source approaches. We release h2oGPT, a family of fine-tuned LLMs of diverse sizes. We also introduce H2O LLM Studio, a framework and no-code GUI designed for efficient fine-tuning, evaluation, and deployment of LLMs using the most recent state-of-the-art techniques. Our code and models are fully open-source. We believe this work helps to boost AI development and make it more accessible, efficient and trustworthy. The demo is available at: https://gpt.h2o.ai/


title: Generative error correction for code-switching speech recognition using large language models

abstract: Code-switching (CS) speech refers to the phenomenon of mixing two or more languages within the same sentence. Despite the recent advances in automatic speech recognition (ASR), CS-ASR is still a challenging task ought to the grammatical structure complexity of the phenomenon and the data scarcity of specific training corpus. In this work, we propose to leverage large language models (LLMs) and lists of hypotheses generated by an ASR to address the CS problem. Specifically, we first employ multiple well-trained ASR models for N-best hypotheses generation, with the aim of increasing the diverse and informative elements in the set of hypotheses. Next, we utilize the LLMs to learn the hypotheses-to-transcription (H2T) mapping by adding a trainable low-rank adapter. Such a generative error correction (GER) method directly predicts the accurate transcription according to its expert linguistic knowledge and N-best hypotheses, resulting in a paradigm shift from the traditional language model rescoring or error correction techniques. Experimental evidence demonstrates that GER significantly enhances CS-ASR accuracy, in terms of reduced mixed error rate (MER). Furthermore, LLMs show remarkable data efficiency for H2T learning, providing a potential solution to the data scarcity problem of CS-ASR in low-resource languages.


title: Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament

abstract: Accurately predicting the future would be an important milestone in the capabilities of artificial intelligence. However, research on the ability of large language models to provide probabilistic predictions about future events remains nascent. To empirically test this ability, we enrolled OpenAI's state-of-the-art large language model, GPT-4, in a three-month forecasting tournament hosted on the Metaculus platform. The tournament, running from July to October 2023, attracted 843 participants and covered diverse topics including Big Tech, U.S. politics, viral outbreaks, and the Ukraine conflict. Focusing on binary forecasts, we show that GPT-4's probabilistic forecasts are significantly less accurate than the median human-crowd forecasts. We find that GPT-4's forecasts did not significantly differ from the no-information forecasting strategy of assigning a 50% probability to every question. We explore a potential explanation, that GPT-4 might be predisposed to predict probabilities close to the midpoint of the scale, but our data do not support this hypothesis. Overall, we find that GPT-4 significantly underperforms in real-world predictive tasks compared to median human-crowd forecasts. A potential explanation for this underperformance is that in real-world forecasting tournaments, the true answers are genuinely unknown at the time of prediction; unlike in other benchmark tasks like professional exams or time series forecasting, where strong performance may at least partly be due to the answers being memorized from the training data. This makes real-world forecasting tournaments an ideal environment for testing the generalized reasoning and prediction capabilities of artificial intelligence going forward.


title: Audio-AdapterFusion: A Task-ID-free Approach for Efficient and Non-Destructive Multi-task Speech Recognition

abstract: Adapters are an efficient, composable alternative to full fine-tuning of pre-trained models and help scale the deployment of large ASR models to many tasks. In practice, a task ID is commonly prepended to the input during inference to route to single-task adapters for the specified task. However, one major limitation of this approach is that the task ID may not be known during inference, rendering it unsuitable for most multi-task settings. To address this, we propose three novel task-ID-free methods to combine single-task adapters in multi-task ASR and investigate two learning algorithms for training. We evaluate our methods on 10 test sets from 4 diverse ASR tasks and show that our methods are non-destructive and parameter-efficient. While only updating 17% of the model parameters, our methods can achieve an 8% mean WER improvement relative to full fine-tuning and are on-par with task-ID adapter routing.


title: Solving the multiplication problem of a large language model system using a graph-based method

abstract: The generative pre-trained transformer (GPT)-based chatbot software ChatGPT possesses excellent natural language processing capabilities but is inadequate for solving arithmetic problems, especially multiplication. Its GPT structure uses a computational graph for multiplication, which has limited accuracy beyond simple multiplication operations. We developed a graph-based multiplication algorithm that emulated human-like numerical operations by incorporating a 10k operator, where k represents the maximum power to base 10 of the larger of two input numbers. Our proposed algorithm attained 100% accuracy for 1,000,000 large number multiplication tasks, effectively solving the multiplication challenge of GPT-based and other large language models. Our work highlights the importance of blending simple human insights into the design of artificial intelligence algorithms. Keywords: Graph-based multiplication; ChatGPT; Multiplication problem


title: Position Interpolation Improves ALiBi Extrapolation

abstract: Linear position interpolation helps pre-trained models using rotary position embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using linear position interpolation to extend the extrapolation range of models using Attention with Linear Biases (ALiBi). We find position interpolation significantly improves extrapolation capability on upstream language modelling and downstream summarization and retrieval tasks.


title: Getting aligned on representational alignment

abstract: Biological and artificial information processing systems form representations that they can use to categorize, reason, plan, navigate, and make decisions. How can we measure the extent to which the representations formed by these diverse systems agree? Do similarities in representations then translate into similar behavior? How can a system's representations be modified to better match those of another system? These questions pertaining to the study of representational alignment are at the heart of some of the most active research areas in cognitive science, neuroscience, and machine learning. For example, cognitive scientists measure the representational alignment of multiple individuals to identify shared cognitive priors, neuroscientists align fMRI responses from multiple individuals into a shared representational space for group-level analyses, and ML researchers distill knowledge from teacher models into student models by increasing their alignment. Unfortunately, there is limited knowledge transfer between research communities interested in representational alignment, so progress in one field often ends up being rediscovered independently in another. Thus, greater cross-field communication would be advantageous. To improve communication between these fields, we propose a unifying framework that can serve as a common language between researchers studying representational alignment. We survey the literature from all three fields and demonstrate how prior work fits into this framework. Finally, we lay out open problems in representational alignment where progress can benefit all three of these fields. We hope that our work can catalyze cross-disciplinary collaboration and accelerate progress for all communities studying and developing information processing systems. We note that this is a working paper and encourage readers to reach out with their suggestions for future revisions.


title: Tailoring Adversarial Attacks on Deep Neural Networks for Targeted Class Manipulation Using DeepFool Algorithm

abstract: Deep neural networks (DNNs) have significantly advanced various domains, but their vulnerability to adversarial attacks poses serious concerns. Understanding these vulnerabilities and developing effective defense mechanisms is crucial. DeepFool, an algorithm proposed by Moosavi-Dezfooli et al. (2016), finds minimal perturbations to misclassify input images. However, DeepFool lacks a targeted approach, making it less effective in specific attack scenarios. Also, in previous related works, researchers primarily focus on success, not considering how much an image is getting distorted; the integrity of the image quality, and the confidence level to misclassifying. So, in this paper, we propose Enhanced Targeted DeepFool, an augmented version of DeepFool that allows targeting specific classes for misclassification and also introduce a minimum confidence score requirement hyperparameter to enhance flexibility. Our experiments demonstrate the effectiveness and efficiency of the proposed method across different deep neural network architectures while preserving image integrity as much and perturbation rate as less as possible. By using our approach, the behavior of models can be manipulated arbitrarily using the perturbed images, as we can specify both the target class and the associated confidence score, unlike other DeepFool-derivative works, such as Targeted DeepFool by Gajjar et al. (2022). Results show that one of the deep convolutional neural network architectures, AlexNet, and one of the state-of-the-art model Vision Transformer exhibit high robustness to getting fooled. This approach can have larger implication, as our tuning of confidence level can expose the robustness of image recognition models. Our code will be made public upon acceptance of the paper.


title: Quantum batteries -- The future of energy storage?

abstract: According to the International Energy Agency, each human uses more than 80 GJ of energy per year; this is equivalent to leaving a washing machine continuously running for one year for every person on Earth. This consumption is expected to increase by 28% by 2040 (from 2015 levels). The majority (86%) of this energy comes from fossil fuels. This dependence on fossil fuels comes with major environmental costs, with climate change arguably being the greatest challenge facing our era. Renewable energy offers a possible solution. However, renewable energy sources, like solar and wind are not continuous sources, and therefore energy storage technology or batteries, remain an urgent challenge for further worldwide adoption of renewable energy. Alongside the need for efficient batteries to store renewable energy, the portability of batteries makes them an essential component in mobile technologies, including electric vehicles. Current batteries operate on the basis of well-understood electrochemical principles which were developed two centuries ago. While there is an ongoing intense effort aimed at improving their performance through optimization of the materials and the device architecture, it is worth exploring completely novel and disruptive approaches towards energy storage. Quantum batteries are energy storage devices that utilise quantum mechanics to enhance performance or functionality. While they are still in their infancy with only proof-of-principle demonstrations achieved, their radically innovative design principles offer a potential solution to future energy challenges.


title: AI for Mathematics: A Cognitive Science Perspective

abstract: Mathematics is one of the most powerful conceptual systems developed and used by the human species. Dreams of automated mathematicians have a storied history in artificial intelligence (AI). Rapid progress in AI, particularly propelled by advances in large language models (LLMs), has sparked renewed, widespread interest in building such systems. In this work, we reflect on these goals from a \textit{cognitive science} perspective. We call attention to several classical and ongoing research directions from cognitive science, which we believe are valuable for AI practitioners to consider when seeking to build truly human (or superhuman)-level mathematical systems. We close with open discussions and questions that we believe necessitate a multi-disciplinary perspective -- cognitive scientists working in tandem with AI researchers and mathematicians -- as we move toward better mathematical AI systems which not only help us push the frontier of the mathematics, but also offer glimpses into how we as humans are even capable of such great cognitive feats.


title: Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding

abstract: The recent success of large pre-trained language models (PLMs) heavily hinges on massive labeled data, which typically produces inferior performance in low-resource scenarios. To remedy this dilemma, we study self-training as one of the predominant semi-supervised learning (SSL) approaches, which utilizes large-scale unlabeled data to generate synthetic examples. However, too many noisy labels will hurt the model performance, and the self-training procedure requires multiple training iterations making it more expensive if all the model parameters of the PLM are updated. This paper presents UPET, a novel Uncertainty-aware Parameter-Efficient self-Training framework to effectively and efficiently address the labeled data scarcity issue. Specifically, we incorporate Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation for the teacher model and then judiciously select reliable pseudo-labeled examples based on confidence and certainty. During the student training, we introduce multiple parameter-efficient learning (PEL) paradigms that allow the optimization of only a small percentage of parameters. We also propose a novel Easy-Hard Contrastive Tuning to enhance the robustness and generalization. Extensive experiments over multiple downstream tasks demonstrate that UPET achieves a substantial improvement in terms of performance and efficiency. Our codes and data are released at https: //github.com/wjn1996/UPET.


title: GraphGPT: Graph Instruction Tuning for Large Language Models

abstract: Graph Neural Networks (GNNs) have evolved to understand graph structures through recursive exchanges and aggregations among nodes. To enhance robustness, self-supervised learning (SSL) has become a vital tool for data augmentation. Traditional methods often depend on fine-tuning with task-specific labels, limiting their effectiveness when labeled data is scarce. Our research tackles this by advancing graph model generalization in zero-shot learning environments. Inspired by the success of large language models (LLMs), we aim to create a graph-oriented LLM capable of exceptional generalization across various datasets and tasks without relying on downstream graph data. We introduce the GraphGPT framework, which integrates LLMs with graph structural knowledge through graph instruction tuning. This framework includes a text-graph grounding component to link textual and graph structures and a dual-stage instruction tuning approach with a lightweight graph-text alignment projector. These innovations allow LLMs to comprehend complex graph structures and enhance adaptability across diverse datasets and tasks. Our framework demonstrates superior generalization in both supervised and zero-shot graph learning tasks, surpassing existing benchmarks. The open-sourced model implementation of our GraphGPT is available at https://github.com/HKUDS/GraphGPT.


title: Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompt

abstract: Continual pre-training has been urgent for adapting a pre-trained model to a multitude of domains and tasks in the fast-evolving world. In practice, a continually pre-trained model is expected to demonstrate not only greater capacity when fine-tuned on pre-trained domains but also a non-decreasing performance on unseen ones. In this work, we first investigate such anytime fine-tuning effectiveness of existing continual pre-training approaches, concluding with unanimously decreased performance on unseen domains. To this end, we propose a prompt-guided continual pre-training method, where we train a hypernetwork to generate domain-specific prompts by both agreement and disagreement losses. The agreement loss maximally preserves the generalization of a pre-trained model to new domains, and the disagreement one guards the exclusiveness of the generated hidden states for each domain. Remarkably, prompts by the hypernetwork alleviate the domain identity when fine-tuning and promote knowledge transfer across domains. Our method achieved improvements of 3.57% and 3.4% on two real-world datasets (including domain shift and temporal shift), respectively, demonstrating its efficacy.


title: Powerset multi-class cross entropy loss for neural speaker diarization

abstract: Since its introduction in 2019, the whole end-to-end neural diarization (EEND) line of work has been addressing speaker diarization as a frame-wise multi-label classification problem with permutation-invariant training. Despite EEND showing great promise, a few recent works took a step back and studied the possible combination of (local) supervised EEND diarization with (global) unsupervised clustering. Yet, these hybrid contributions did not question the original multi-label formulation. We propose to switch from multi-label (where any two speakers can be active at the same time) to powerset multi-class classification (where dedicated classes are assigned to pairs of overlapping speakers). Through extensive experiments on 9 different benchmarks, we show that this formulation leads to significantly better performance (mostly on overlapping speech) and robustness to domain mismatch, while eliminating the detection threshold hyperparameter, critical for the multi-label formulation.


title: Weakly-Supervised Semantic Segmentation with Image-Level Labels: from Traditional Models to Foundation Models

abstract: The rapid development of deep learning has driven significant progress in the field of image semantic segmentation - a fundamental task in computer vision. Semantic segmentation algorithms often depend on the availability of pixel-level labels (i.e., masks of objects), which are expensive, time-consuming, and labor-intensive. Weakly-supervised semantic segmentation (WSSS) is an effective solution to avoid such labeling. It utilizes only partial or incomplete annotations and provides a cost-effective alternative to fully-supervised semantic segmentation. In this paper, we focus on the WSSS with image-level labels, which is the most challenging form of WSSS. Our work has two parts. First, we conduct a comprehensive survey on traditional methods, primarily focusing on those presented at premier research conferences. We categorize them into four groups based on where their methods operate: pixel-wise, image-wise, cross-image, and external data. Second, we investigate the applicability of visual foundation models, such as the Segment Anything Model (SAM), in the context of WSSS. We scrutinize SAM in two intriguing scenarios: text prompting and zero-shot learning. We provide insights into the potential and challenges associated with deploying visual foundational models for WSSS, facilitating future developments in this exciting research area.


title: Be Bayesian by Attachments to Catch More Uncertainty

abstract: Bayesian Neural Networks (BNNs) have become one of the promising approaches for uncertainty estimation due to the solid theorical foundations. However, the performance of BNNs is affected by the ability of catching uncertainty. Instead of only seeking the distribution of neural network weights by in-distribution (ID) data, in this paper, we propose a new Bayesian Neural Network with an Attached structure (ABNN) to catch more uncertainty from out-of-distribution (OOD) data. We first construct a mathematical description for the uncertainty of OOD data according to the prior distribution, and then develop an attached Bayesian structure to integrate the uncertainty of OOD data into the backbone network. ABNN is composed of an expectation module and several distribution modules. The expectation module is a backbone deep network which focuses on the original task, and the distribution modules are mini Bayesian structures which serve as attachments of the backbone. In particular, the distribution modules aim at extracting the uncertainty from both ID and OOD data. We further provide theoretical analysis for the convergence of ABNN, and experimentally validate its superiority by comparing with some state-of-the-art uncertainty estimation methods Code will be made available.