title: Zero-Shot Learning Over Large Output Spaces : Utilizing Indirect Knowledge Extraction from Large Language Models

abstract: Extreme Multi-label Learning (XMC) is a task that allocates the most relevant labels for an instance from a predefined label set. Extreme Zero-shot XMC (EZ-XMC) is a special setting of XMC wherein no supervision is provided; only the instances (raw text of the document) and the predetermined label set are given. The scenario is designed to address cold-start problems in categorization and recommendation. Traditional state-of-the-art methods extract pseudo labels from the document title or segments. These labels from the document are used to train a zero-shot bi-encoder model. The main issue with these generated labels is their misalignment with the tagging task. In this work, we propose a framework to train a small bi-encoder model via the feedback from the large language model (LLM), the bi-encoder model encodes the document and labels into embeddings for retrieval. Our approach leverages the zero-shot ability of LLM to assess the correlation between labels and the document instead of using the low-quality labels extracted from the document itself. Our method also guarantees fast inference without the involvement of LLM. The performance of our approach outperforms the SOTA methods on various datasets while retaining a similar training time for large datasets.


title: Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models

abstract: Conversational Large Language Models are trained to refuse to answer harmful questions. However, emergent jailbreaking techniques can still elicit unsafe outputs, presenting an ongoing challenge for model alignment. To better understand how different jailbreak types circumvent safeguards, this paper analyses model activations on different jailbreak inputs. We find that it is possible to extract a jailbreak vector from a single class of jailbreaks that works to mitigate jailbreak effectiveness from other classes. This may indicate that different kinds of effective jailbreaks operate via similar internal mechanisms. We investigate a potential common mechanism of harmfulness feature suppression, and provide evidence for its existence by looking at the harmfulness vector component. These findings offer actionable insights for developing more robust jailbreak countermeasures and lay the groundwork for a deeper, mechanistic understanding of jailbreak dynamics in language models.


title: Exploring Spoken Language Identification Strategies for Automatic Transcription of Multilingual Broadcast and Institutional Speech

abstract: This paper addresses spoken language identification (SLI) and speech recognition of multilingual broadcast and institutional speech, real application scenarios that have been rarely addressed in the SLI literature. Observing that in these domains language changes are mostly associated with speaker changes, we propose a cascaded system consisting of speaker diarization and language identification and compare it with more traditional language identification and language diarization systems. Results show that the proposed system often achieves lower language classification and language diarization error rates (up to 10% relative language diarization error reduction and 60% relative language confusion reduction) and leads to lower WERs on multilingual test sets (more than 8% relative WER reduction), while at the same time does not negatively affect speech recognition on monolingual audio (with an absolute WER increase between 0.1% and 0.7% w.r.t. monolingual ASR).


title: A Flexible, Equivariant Framework for Subgraph GNNs via Graph Products and Graph Coarsening

abstract: Subgraph Graph Neural Networks (Subgraph GNNs) enhance the expressivity of message-passing GNNs by representing graphs as sets of subgraphs. They have shown impressive performance on several tasks, but their complexity limits applications to larger graphs. Previous approaches suggested processing only subsets of subgraphs, selected either randomly or via learnable sampling. However, they make suboptimal subgraph selections or can only cope with very small subset sizes, inevitably incurring performance degradation. This paper introduces a new Subgraph GNNs framework to address these issues. We employ a graph coarsening function to cluster nodes into super-nodes with induced connectivity. The product between the coarsened and the original graph reveals an implicit structure whereby subgraphs are associated with specific sets of nodes. By running generalized message-passing on such graph product, our method effectively implements an efficient, yet powerful Subgraph GNN. Controlling the coarsening function enables meaningful selection of any number of subgraphs while, contrary to previous methods, being fully compatible with standard training techniques. Notably, we discover that the resulting node feature tensor exhibits new, unexplored permutation symmetries. We leverage this structure, characterize the associated linear equivariant layers and incorporate them into the layers of our Subgraph GNN architecture. Extensive experiments on multiple graph learning benchmarks demonstrate that our method is significantly more flexible than previous approaches, as it can seamlessly handle any number of subgraphs, while consistently outperforming baseline approaches.


title: Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models

abstract: We address the problem of multi-object 3D pose control in image diffusion models. Instead of conditioning on a sequence of text tokens, we propose to use a set of per-object representations, Neural Assets, to control the 3D pose of individual objects in a scene. Neural Assets are obtained by pooling visual representations of objects from a reference image, such as a frame in a video, and are trained to reconstruct the respective objects in a different image, e.g., a later frame in the video. Importantly, we encode object visuals from the reference image while conditioning on object poses from the target frame. This enables learning disentangled appearance and pose features. Combining visual and 3D pose representations in a sequence-of-tokens format allows us to keep the text-to-image architecture of existing models, with Neural Assets in place of text tokens. By fine-tuning a pre-trained text-to-image diffusion model with this information, our approach enables fine-grained 3D pose and placement control of individual objects in a scene. We further demonstrate that Neural Assets can be transferred and recomposed across different scenes. Our model achieves state-of-the-art multi-object editing results on both synthetic 3D scene datasets, as well as two real-world video datasets (Objectron, Waymo Open).


title: StableMaterials: Enhancing Diversity in Material Generation via Semi-Supervised Learning

abstract: We introduce StableMaterials, a novel approach for generating photorealistic physical-based rendering (PBR) materials that integrate semi-supervised learning with Latent Diffusion Models (LDMs). Our method employs adversarial training to distill knowledge from existing large-scale image generation models, minimizing the reliance on annotated data and enhancing the diversity in generation. This distillation approach aligns the distribution of the generated materials with that of image textures from an SDXL model, enabling the generation of novel materials that are not present in the initial training dataset. Furthermore, we employ a diffusion-based refiner model to improve the visual quality of the samples and achieve high-resolution generation. Finally, we distill a latent consistency model for fast generation in just four steps and propose a new tileability technique that removes visual artifacts typically associated with fewer diffusion steps. We detail the architecture and training process of StableMaterials, the integration of semi-supervised training within existing LDM frameworks and show the advantages of our approach. Comparative evaluations with state-of-the-art methods show the effectiveness of StableMaterials, highlighting its potential applications in computer graphics and beyond. StableMaterials is publicly available at https://gvecchio.com/stablematerials.


title: You Don't Need Data-Augmentation in Self-Supervised Learning

abstract: Self-Supervised learning (SSL) with Joint-Embedding Architectures (JEA) has led to outstanding performances. All instantiations of this paradigm were trained using strong and well-established hand-crafted data augmentations, leading to the general belief that they are required for the proper training and performance of such models. On the other hand, generative reconstruction-based models such as BEIT and MAE or Joint-Embedding Predictive Architectures such as I-JEPA have shown strong performance without using data augmentations except masking. In this work, we challenge the importance of invariance and data-augmentation in JEAs at scale. By running a case-study on a recent SSL foundation model - DINOv2 - we show that strong image representations can be obtained with JEAs and only cropping without resizing provided the training data is large enough, reaching state-of-the-art results and using the least amount of augmentation in the literature. Through this study, we also discuss the impact of compute constraints on the outcomes of experimental deep learning research, showing that they can lead to very different conclusions.


title: AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models

abstract: Evaluating the alignment capabilities of large Vision-Language Models (VLMs) is essential for determining their effectiveness as helpful assistants. However, existing benchmarks primarily focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. In this paper, we address this gap by introducing AlignMMBench, a comprehensive alignment benchmark specifically designed for emerging Chinese VLMs. This benchmark is meticulously curated from real-world scenarios and Chinese Internet sources, encompassing thirteen specific tasks across three categories, and includes both single-turn and multi-turn dialogue scenarios. Incorporating a prompt rewrite strategy, AlignMMBench encompasses 1,054 images and 4,978 question-answer pairs. To facilitate the evaluation pipeline, we propose CritiqueVLM, a rule-calibrated evaluator that exceeds GPT-4's evaluation ability. Finally, we report the performance of representative VLMs on AlignMMBench, offering insights into the capabilities and limitations of different VLM architectures. All evaluation codes and data are available on https://alignmmbench.github.io.


title: Parameter-Efficient Active Learning for Foundational models

abstract: Foundational vision transformer models have shown impressive few shot performance on many vision tasks. This research presents a novel investigation into the application of parameter efficient fine-tuning methods within an active learning (AL) framework, to advance the sampling selection process in extremely budget constrained classification tasks. The focus on image datasets, known for their out-of-distribution characteristics, adds a layer of complexity and relevance to our study. Through a detailed evaluation, we illustrate the improved AL performance on these challenging datasets, highlighting the strategic advantage of merging parameter efficient fine tuning methods with foundation models. This contributes to the broader discourse on optimizing AL strategies, presenting a promising avenue for future exploration in leveraging foundation models for efficient and effective data annotation in specialized domains.


title: MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding

abstract: Auto-regressive inference of transformers benefit greatly from Key-Value (KV) caching, but can lead to major memory bottlenecks as model size, batch size, and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV) sharing, a novel approach extending KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention (MQA) and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and inference metrics using uptrained Pythia-160M variants demonstrate that MLKV significantly reduces memory usage with minimal performance loss, reducing KV cache size down to a factor of 6x compared to MQA. These results highlight MLKV's potential for efficient deployment of transformer models at scale. We provide code at https://github.com/zaydzuhri/pythia-mlkv


title: Magnon spin transport through atomic ferrimagnetic domain walls

abstract: It is a well-established notion that the spin of a magnon should be flipped when it passes through a $180^{\circ}$ domain wall (DW) in both ferromagnets and antiferromagnets, while the magnon spin transport through ferrimagnetic DW is still elusive. In this work, we report that the magnon preserves its spin after the transmission through an atomically sharp DW in ferrimagnets, due to the intriguing interband magnon scattering at the domain interface. This finding may provide significant insight to resolve the puzzling insensitivity of magnon spin diffusion to the $180^{\circ}$ ferrimagnetic DWs observed by recent experiments. Our results reveal the unique role of ferrimagnetic DWs in manipulating the magnon spin and may facilitate the design of novel magnonic devices based on ferrimagnets.


title: Pauli Noise Learning for Mid-Circuit Measurements

abstract: Current benchmarks for mid-circuit measurements (MCMs) are limited in scalability or the types of error they can quantify, necessitating new techniques for quantifying their performance. Here, we introduce a theory for learning Pauli noise in MCMs and use it to create MCM cycle benchmarking, a scalable method for benchmarking MCMs. MCM cycle benchmarking extracts detailed information about the rates of errors in randomly compiled layers of MCMs and Clifford gates, and we demonstrate how its results can be used to quantify correlated errors during MCMs on current quantum hardware. Our method can be integrated into existing Pauli noise learning techniques to scalably characterize and benchmark wide classes of circuits containing MCMs.


title: Nested Sequents for Quasi-transitive Modal Logics

abstract: Previous works by Gor\'e, Postniece and Tiu have provided sound and cut-free complete proof systems for modal logics extended with path axioms using the formalism of nested sequent. Our aim is to provide (i) a constructive cut-elimination procedure and (ii) alternative modular formulations for these systems. We present our methodology to achieve these two goals on a subclass of path axioms, namely quasi-transitivity axioms.


title: Hands-free teleoperation of a nearby manipulator through a virtual body-to-robot link

abstract: This paper introduces an innovative control approach for teleoperating a robot in close proximity to a human operator, which could be useful to control robots embedded on wheelchairs. The method entails establishing a virtual connection between a specific body part and the robot's end-effector, visually displayed through an Augmented Reality (AR) headset. This linkage enables the transformation of body rotations into amplified effector translations, extending the robot's workspace beyond the capabilities of direct one-to-one mapping. Moreover, the linkage can be reconfigured using a joystick, resulting in a hybrid position/velocity control mode using the body/joystick motions respectively. After providing a comprehensive overview of the control methodology, we present the results of an experimental campaign designed to elucidate the advantages and drawbacks of our approach compared to the conventional joystick-based teleoperation method. The body-link control demonstrates slightly faster task completion and is naturally preferred over joystick velocity control, albeit being more physically demanding for tasks with a large range. The hybrid mode, where participants could simultaneously utilize both modes, emerges as a compromise, combining the intuitiveness of the body mode with the extensive task range of the velocity mode. Finally, we provide preliminary observations on potential assistive applications using head motions, especially for operators with limited range of motion in their bodies.


title: The reflection complexity of sequences over finite alphabets

abstract: In combinatorics on words, the well-studied factor complexity function $\rho_{\bf x}$ of a sequence ${\bf x}$ over a finite alphabet counts, for any nonnegative integer $n$, the number of distinct length-$n$ factors of ${\bf x}$. In this paper, we introduce the \emph{reflection complexity} function $r_{\bf x}$ to enumerate the factors occurring in a sequence ${\bf x}$, up to reversing the order of symbols in a word. We introduce and prove results on $r_{\bf x}$ regarding its growth properties and relationship with other complexity functions. We prove that if ${\bf x}$ is $k$-automatic, then $r_{\bf x}$ is computably $k$-regular, and we use the software {\tt Walnut} to evaluate the reflection complexity of automatic sequences, such as the Thue--Morse sequence. We prove a Morse--Hedlund-type result characterizing eventually periodic sequences in terms of their reflection complexity, and we deduce a characterization of Sturmian sequences. Furthermore, we investigate the reflection complexity of episturmian, $(s+1)$-dimensional billiard, and Rote sequences. There are still many unanswered questions about this measure.


title: All-optically tunable enantio-selectivity and chirality transfer

abstract: Detecting and controlling the chirality of materials play an essential role in exploring nature, providing new avenues for material creation, discrimination, and manipulation. In such tasks, chiral reagents are essential in defining or enhancing the chiral dichroism response. However, ignoring their influences on the symmetry of the medium hamper the ability to control and induce asymmetric synthesis. Here, we propose a simple but versatile chirality transfer method for synthesizing and manipulating the chirality of medium. The proposed method induces the dispersion of light in a neutral atomic system, allowing to deterministically and tunably control the chirality transfer using a helical field. First, we theoretically analyze the mechanism for this optically induced chirality transfer. Afterwards, we experimentally study the enantio-sensitive feature of the medium exposed to the auxiliary chiral field. This result can be suppressed or enhanced in a deterministic enantio-selection, opening up an efficient way to manipulate asymmetric synthesis.


title: Self-reconfigurable Multifunctional Memristive Nociceptor for Intelligent Robotics

abstract: Artificial nociceptors, mimicking human-like stimuli perception, are of significance for intelligent robotics to work in hazardous and dynamic scenarios. One of the most essential characteristics of the human nociceptor is its self-adjustable attribute, which indicates that the threshold of determination of a potentially hazardous stimulus relies on environmental knowledge. This critical attribute has been currently omitted, but it is highly desired for artificial nociceptors. Inspired by these shortcomings, this article presents, for the first time, a Self-Directed Channel (SDC) memristor-based self-reconfigurable nociceptor, capable of perceiving hazardous pressure stimuli under different temperatures and demonstrates key features of tactile nociceptors, including 'threshold,' 'no-adaptation,' and 'sensitization.' The maximum amplification of hazardous external stimuli is 1000%, and its response characteristics dynamically adapt to current temperature conditions by automatically altering the generated modulation schemes for the memristor. The maximum difference ratio of the response of memristors at different temperatures is 500%, and this adaptability closely mimics the functions of biological tactile nociceptors, resulting in accurate danger perception in various conditions. Beyond temperature adaptation, this memristor-based nociceptor has the potential to integrate different sensory modalities by applying various sensors, thereby achieving human-like perception capabilities in real-world environments.


title: Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation

abstract: In subject-driven text-to-image generation, recent works have achieved superior performance by training the model on synthetic datasets containing numerous image pairs. Trained on these datasets, generative models can produce text-aligned images for specific subject from arbitrary testing image in a zero-shot manner. They even outperform methods which require additional fine-tuning on testing images. However, the cost of creating such datasets is prohibitive for most researchers. To generate a single training pair, current methods fine-tune a pre-trained text-to-image model on the subject image to capture fine-grained details, then use the fine-tuned model to create images for the same subject based on creative text prompts. Consequently, constructing a large-scale dataset with millions of subjects can require hundreds of thousands of GPU hours. To tackle this problem, we propose Toffee, an efficient method to construct datasets for subject-driven editing and generation. Specifically, our dataset construction does not need any subject-level fine-tuning. After pre-training two generative models, we are able to generate infinite number of high-quality samples. We construct the first large-scale dataset for subject-driven image editing and generation, which contains 5 million image pairs, text prompts, and masks. Our dataset is 5 times the size of previous largest dataset, yet our cost is tens of thousands of GPU hours lower. To test the proposed dataset, we also propose a model which is capable of both subject-driven image editing and generation. By simply training the model on our proposed dataset, it obtains competitive results, illustrating the effectiveness of the proposed dataset construction framework.


title: Single-photon and multi-photon Fano lines for helium and neon using tRecX-haCC

abstract: Single-photon and multi-photon ionization of helium and neon atoms by ultrashort extreme ultraviolet radiation is studied using the tRecX-haCC package developed by the authors. The tRecX-haCC package solves the time dependent Schrodinger equation in the presence of a laser pulse and computes the photoelectron spectra using the iSurf technique to efficiently capture the slow decay of the resonant states. Wavelength of the radiation is chosen to expose doubly excited states embedded in the continuum. The lineshapes in the spectra that arise as a result of these doubly excited states are analyzed. The computed peak positions and linewidths are in good agreement with the literature. In addition, the dependence of the Fano q parameter on the order of the multi-photon process is demonstrated.


title: A tutorial on fairness in machine learning in healthcare

abstract: $\textbf{OBJECTIVE}$: Ensuring that machine learning (ML) algorithms are safe and effective within all patient groups, and do not disadvantage particular patients, is essential to clinical decision making and preventing the reinforcement of existing healthcare inequities. The objective of this tutorial is to introduce the medical informatics community to the common notions of fairness within ML, focusing on clinical applications and implementation in practice. $\textbf{TARGET AUDIENCE}$: As gaps in fairness arise in a variety of healthcare applications, this tutorial is designed to provide an understanding of fairness, without assuming prior knowledge, to researchers and clinicians who make use of modern clinical data. $\textbf{SCOPE}$: We describe the fundamental concepts and methods used to define fairness in ML, including an overview of why models in healthcare may be unfair, a summary and comparison of the metrics used to quantify fairness, and a discussion of some ongoing research. We illustrate some of the fairness methods introduced through a case study of mortality prediction in a publicly available electronic health record dataset. Finally, we provide a user-friendly R package for comprehensive group fairness evaluation, enabling researchers and clinicians to assess fairness in their own ML work.


title: Transformers meet Neural Algorithmic Reasoners

abstract: Transformers have revolutionized machine learning with their simple yet effective architecture. Pre-training Transformers on massive text datasets from the Internet has led to unmatched generalization for natural language understanding (NLU) tasks. However, such language models remain fragile when tasked with algorithmic forms of reasoning, where computations must be precise and robust. To address this limitation, we propose a novel approach that combines the Transformer's language understanding with the robustness of graph neural network (GNN)-based neural algorithmic reasoners (NARs). Such NARs proved effective as generic solvers for algorithmic tasks, when specified in graph form. To make their embeddings accessible to a Transformer, we propose a hybrid architecture with a two-phase training procedure, allowing the tokens in the language model to cross-attend to the node embeddings from the NAR. We evaluate our resulting TransNAR model on CLRS-Text, the text-based version of the CLRS-30 benchmark, and demonstrate significant gains over Transformer-only models for algorithmic reasoning, both in and out of distribution.


title: Teleoperation of a robotic manipulator in peri-personal space: a virtual wand approach

abstract: The paper deals with the well-known problem of teleoperating a robotic arm along six degrees of freedom. The prevailing and most effective approach to this problem involves a direct position-to-position mapping, imposing robotic end-effector movements that mirrors those of the user. In the particular case where the robot stands near the operator, there are alternatives to this approach. Drawing inspiration from head pointers utilized in the 1980s, originally designed to enable drawing with limited head motions for tetraplegic individuals, we propose a "virtual wand" mapping. It employs a virtual rigid linkage between the hand and the robot's end-effector. With this approach, rotations produce amplified translations through a lever arm, creating a "rotation-to-position" coupling. This approach expands the translation workspace at the expense of a reduced rotation space. We compare the virtual wand approach to the one-to-one position mapping through the realization of 6-DoF reaching tasks. Results indicate that the two different mappings perform comparably well, are equally well-received by users, and exhibit similar motor control behaviors. Nevertheless, the virtual wand mapping is anticipated to outperform in tasks characterized by large translations and minimal effector rotations, whereas direct mapping is expected to demonstrate advantages in large rotations with minimal translations. These results pave the way for new interactions and interfaces, particularly in disability assistance utilizing head movements (instead of hands). Leveraging body parts with substantial rotations could enable the accomplishment of tasks previously deemed infeasible with standard direct coupling interfaces.


title: Neural networks in non-metric spaces

abstract: Leveraging the infinite dimensional neural network architecture we proposed in arXiv:2109.13512v4 and which can process inputs from Fr\'echet spaces, and using the universal approximation property shown therein, we now largely extend the scope of this architecture by proving several universal approximation theorems for a vast class of input and output spaces. More precisely, the input space $\mathfrak X$ is allowed to be a general topological space satisfying only a mild condition ("quasi-Polish"), and the output space can be either another quasi-Polish space $\mathfrak Y$ or a topological vector space $E$. Similarly to arXiv:2109.13512v4, we show furthermore that our neural network architectures can be projected down to "finite dimensional" subspaces with any desirable accuracy, thus obtaining approximating networks that are easy to implement and allow for fast computation and fitting. The resulting neural network architecture is therefore applicable for prediction tasks based on functional data. To the best of our knowledge, this is the first result which deals with such a wide class of input/output spaces and simultaneously guarantees the numerical feasibility of the ensuing architectures. Finally, we prove an obstruction result which indicates that the category of quasi-Polish spaces is in a certain sense the correct category to work with if one aims at constructing approximating architectures on infinite-dimensional spaces $\mathfrak X$ which, at the same time, have sufficient expressive power to approximate continuous functions on $\mathfrak X$, are specified by a finite number of parameters only and are "stable" with respect to these parameters.


title: Learning High-dimensional Latent Variable Models via Doubly Stochastic Optimisation by Unadjusted Langevin

abstract: Latent variable models are widely used in social and behavioural sciences, such as education, psychology, and political science. In recent years, high-dimensional latent variable models have become increasingly common for analysing large and complex data. Estimating high-dimensional latent variable models using marginal maximum likelihood is computationally demanding due to the complexity of integrals involved. To address this challenge, stochastic optimisation, which combines stochastic approximation and sampling techniques, has been shown to be effective. This method iterates between two steps -- (1) sampling the latent variables from their posterior distribution based on the current parameter estimate, and (2) updating the fixed parameters using an approximate stochastic gradient constructed from the latent variable samples. In this paper, we propose a computationally more efficient stochastic optimisation algorithm. This improvement is achieved through the use of a minibatch of observations when sampling latent variables and constructing stochastic gradients, and an unadjusted Langevin sampler that utilises the gradient of the negative complete-data log-likelihood to sample latent variables. Theoretical results are established for the proposed algorithm, showing that the iterative parameter update converges to the marginal maximum likelihood estimate as the number of iterations goes to infinity. Furthermore, the proposed algorithm is shown to scale well to high-dimensional settings through simulation studies and a personality test application with 30,000 respondents, 300 items, and 30 latent dimensions.


abstract: In this article, we will explore the fundamental concepts, including various basic concepts on $d$-complex manifolds, along with several differential operators and examine the relationships between them. A $d$-K\"ahler manifold is a $d$-complex manifold equipped with a metric that satisfies a specific condition. We prove the Hodge decomposition theorem on compact $d$-K\"ahler manifolds, which establishes a crucial relationship between certain de-Rham cohomology groups and Dolbeault cohomology groups on a compact $d$-K\"ahler manifold .


title: Less Cybersickness, Please: Demystifying and Detecting Stereoscopic Visual Inconsistencies in VR Apps

abstract: The quality of Virtual Reality (VR) apps is vital, particularly the rendering quality of the VR Graphical User Interface (GUI). Different from traditional 2D apps, VR apps create a 3D digital scene for users, by rendering two distinct 2D images for the user's left and right eyes, respectively. Stereoscopic visual inconsistency (denoted as "SVI") issues, however, undermine the rendering process of the user's brain, leading to user discomfort and even adverse health effects. Such issues commonly exist but remain underexplored. We conduct an empirical analysis on 282 SVI bug reports from 15 VR platforms, summarizing 15 types of manifestations. The empirical analysis reveals that automatically detecting SVI issues is challenging, mainly because: (1) lack of training data; (2) the manifestations of SVI issues are diverse, complicated, and often application-specific; (3) most accessible VR apps are closed-source commercial software. Existing pattern-based supervised classification approaches may be inapplicable or ineffective in detecting the SVI issues. To counter these challenges, we propose an unsupervised black-box testing framework named StereoID to identify the stereoscopic visual inconsistencies, based only on the rendered GUI states. StereoID generates a synthetic right-eye image based on the actual left-eye image and computes distances between the synthetic right-eye image and the actual right-eye image to detect SVI issues. We propose a depth-aware conditional stereo image translator to power the image generation process, which captures the expected perspective shifts between left-eye and right-eye images. We build a large-scale unlabeled VR stereo screenshot dataset with larger than 171K images from 288 real-world VR apps for experiments. After substantial experiments, StereoID demonstrates superior performance for detecting SVI issues in both user reports and wild VR apps.


title: Ringdown signatures of Kerr black holes immersed in a magnetic field

abstract: We analyze the quasinormal mode spectrum for Kerr black holes surrounded by an asymptotically uniform magnetic field, modeled with the Ernst-Wild geometry. A perturbative expansion in both the rotation parameter $a$ and the magnetic field $B$ allows separation of the perturbation equations, and we obtain the spectrum for a variety of scalar quasinormal modes over a range of parameters using the continued fraction method. We then interpolate the low-lying mode spectrum to construct an Ernst-Wild template for the ringdown, and use the LIGO-Virgo-KAGRA analysis tool pyRing to assess the impact of the magnetosphere on the extraction of ringdown signatures from several observed binary black hole mergers.


title: Vertical LoRA: Dense Expectation-Maximization Interpretation of Transformers

abstract: In this paper, we show how Transformers can be interpreted as dense Expectation-Maximization algorithms performed on Bayesian Nets. Based on the above interpretation, we propose a new model design paradigm, namely Vertical LoRA (VLoRA), which reduces the parameter count dramatically while preserving performance. In VLoRA, a model consists of layers, each of which recursively learns an increment based on the previous layer. We then apply LoRA decomposition to the increments. VLoRA works on the base model, which is orthogonal to LoRA, meaning they can be used together. We do experiments on various tasks and models. The results show that 1) with VLoRA, the Transformer model parameter count can be reduced dramatically and 2) the performance of the original model is preserved. The source code is available at \url{https://github.com/neverUseThisName/vlora}


title: Bose-Hubbard model with a single qubit

abstract: The use of a single-qubit parametrized circuit as an Ansatz for the variational wave function in the calculation of the ground state energy of a quantum many-body system is demonstrated using the one-dimensional Bose-Hubbard model. Comparison is made to calculations where a classic neural network is used to generate the variational wave function. Computations carried out on IBM Quantum hardware are also presented.


title: Common and Rare Fundus Diseases Identification Using Vision-Language Foundation Model with Knowledge of Over 400 Diseases

abstract: The current retinal artificial intelligence models were trained using data with a limited category of diseases and limited knowledge. In this paper, we present a retinal vision-language foundation model (RetiZero) with knowledge of over 400 fundus diseases. Specifically, we collected 341,896 fundus images paired with text descriptions from 29 publicly available datasets, 180 ophthalmic books, and online resources, encompassing over 400 fundus diseases across multiple countries and ethnicities. RetiZero achieved outstanding performance across various downstream tasks, including zero-shot retinal disease recognition, image-to-image retrieval, internal domain and cross-domain retinal disease classification, and few-shot fine-tuning. Specially, in the zero-shot scenario, RetiZero achieved a Top5 score of 0.8430 and 0.7561 on 15 and 52 fundus diseases respectively. In the image-retrieval task, RetiZero achieved a Top5 score of 0.9500 and 0.8860 on 15 and 52 retinal diseases respectively. Furthermore, clinical evaluations by ophthalmology experts from different countries demonstrate that RetiZero can achieve performance comparable to experienced ophthalmologists using zero-shot and image retrieval methods without requiring model retraining. These capabilities of retinal disease identification strengthen our RetiZero foundation model in clinical implementation.


title: Characterising Interventions in Causal Games

abstract: Causal games are probabilistic graphical models that enable causal queries to be answered in multi-agent settings. They extend causal Bayesian networks by specifying decision and utility variables to represent the agents' degrees of freedom and objectives. In multi-agent settings, whether each agent decides on their policy before or after knowing the causal intervention is important as this affects whether they can respond to the intervention by adapting their policy. Consequently, previous work in causal games imposed chronological constraints on permissible interventions. We relax this by outlining a sound and complete set of primitive causal interventions so the effect of any arbitrarily complex interventional query can be studied in multi-agent settings. We also demonstrate applications to the design of safe AI systems by considering causal mechanism design and commitment.


title: Autohomeomorphisms of pre-images of $\mathbb N^*$

abstract: In the study of the Stone-\u{C}ech remainder of the real line a detailed study of the Stone-\u{C}ech remainder of the space $\mathbb N\times [0,1]$, which we denote as $\mathbb M$, has often been utilized. Of course the real line can be covered by two closed sets that are each homeomorphic to $\mathbb M$. It is known that an autohomeomorphism of $\mathbb M^*$ induces an autohomeomorphism of $\mathbb N^*$. We prove that it is consistent with there being non-trivial autohomeomorphism of $\mathbb N^*$ that those induced by autohomeomorphisms of $\mathbb M^*$ are trivial.


title: Khmer Semantic Search Engine (KSE): Digital Information Access and Document Retrieval

abstract: The search engine process is crucial for document content retrieval. For Khmer documents, an effective tool is needed to extract essential keywords and facilitate accurate searches. Despite the daily generation of significant Khmer content, Cambodians struggle to find necessary documents due to the lack of an effective semantic searching tool. Even Google does not deliver high accuracy for Khmer content. Semantic search engines improve search results by employing advanced algorithms to understand various content types. With the rise in Khmer digital content such as reports, articles, and social media feedback enhanced search capabilities are essential. This research proposes the first Khmer Semantic Search Engine (KSE), designed to enhance traditional Khmer search methods. Utilizing semantic matching techniques and formally annotated semantic content, our tool extracts meaningful keywords from user queries, performs precise matching, and provides the best matching offline documents and online URLs. We propose three semantic search frameworks: semantic search based on a keyword dictionary, semantic search based on ontology, and semantic search based on ranking. Additionally, we developed tools for data preparation, including document addition and manual keyword extraction. To evaluate performance, we created a ground truth dataset and addressed issues related to searching and semantic search. Our findings demonstrate that understanding search term semantics can lead to significantly more accurate results.


title: JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models

abstract: Jailbreak attacks aim to induce Large Language Models (LLMs) to generate harmful responses for forbidden instructions, presenting severe misuse threats to LLMs. Up to now, research into jailbreak attacks and defenses is emerging, however, there is (surprisingly) no consensus on how to evaluate whether a jailbreak attempt is successful. In other words, the methods to assess the harmfulness of an LLM's response are varied, such as manual annotation or prompting GPT-4 in specific ways. Each approach has its own set of strengths and weaknesses, impacting their alignment with human values, as well as the time and financial cost. This diversity in evaluation presents challenges for researchers in choosing suitable evaluation methods and conducting fair comparisons across different jailbreak attacks and defenses. In this paper, we conduct a comprehensive analysis of jailbreak evaluation methodologies, drawing from nearly ninety jailbreak research released between May 2023 and April 2024. Our study introduces a systematic taxonomy of jailbreak evaluators, offering in-depth insights into their strengths and weaknesses, along with the current status of their adaptation. Moreover, to facilitate subsequent research, we propose JailbreakEval, a user-friendly toolkit focusing on the evaluation of jailbreak attempts. It includes various well-known evaluators out-of-the-box, so that users can obtain evaluation results with only a single command. JailbreakEval also allows users to customize their own evaluation workflow in a unified framework with the ease of development and comparison. In summary, we regard JailbreakEval to be a catalyst that simplifies the evaluation process in jailbreak research and fosters an inclusive standard for jailbreak evaluation within the community.


title: Active Inference Meeting Energy-Efficient Control of Parallel and Identical Machines

abstract: We investigate the application of active inference in developing energy-efficient control agents for manufacturing systems. Active inference, rooted in neuroscience, provides a unified probabilistic framework integrating perception, learning, and action, with inherent uncertainty quantification elements. Our study explores deep active inference, an emerging field that combines deep learning with the active inference decision-making framework. Leveraging a deep active inference agent, we focus on controlling parallel and identical machine workstations to enhance energy efficiency. We address challenges posed by the problem's stochastic nature and delayed policy response by introducing tailored enhancements to existing agent architectures. Specifically, we introduce multi-step transition and hybrid horizon methods to mitigate the need for complex planning. Our experimental results demonstrate the effectiveness of these enhancements and highlight the potential of the active inference-based approach.


title: Master of Disaster: A Disaster-Related Event Monitoring System From News Streams

abstract: The need for a disaster-related event monitoring system has arisen due to the societal and economic impact caused by the increasing number of severe disaster events. An event monitoring system should be able to extract event-related information from texts, and discriminates event instances. We demonstrate our open-source event monitoring system, namely, Master of Disaster (MoD), which receives news streams, extracts event information, links extracted information to a knowledge graph (KG), in this case Wikidata, and discriminates event instances visually. The goal of event visualization is to group event mentions referring to the same real-world event instance so that event instance discrimination can be achieved by visual screening.


title: Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs

abstract: Although Large Language Models (LLMs) have demonstrated significant capabilities in executing complex tasks in a zero-shot manner, they are susceptible to jailbreak attacks and can be manipulated to produce harmful outputs. Recently, a growing body of research has categorized jailbreak attacks into token-level and prompt-level attacks. However, previous work primarily overlooks the diverse key factors of jailbreak attacks, with most studies concentrating on LLM vulnerabilities and lacking exploration of defense-enhanced LLMs. To address these issues, we evaluate the impact of various attack settings on LLM performance and provide a baseline benchmark for jailbreak attacks, encouraging the adoption of a standardized evaluation framework. Specifically, we evaluate the eight key factors of implementing jailbreak attacks on LLMs from both target-level and attack-level perspectives. We further conduct seven representative jailbreak attacks on six defense methods across two widely used datasets, encompassing approximately 320 experiments with about 50,000 GPU hours on A800-80G. Our experimental results highlight the need for standardized benchmarking to evaluate these attacks on defense-enhanced LLMs. Our code is available at https://github.com/usail-hkust/Bag_of_Tricks_for_LLM_Jailbreaking.


title: REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Space

abstract: Large language models (LLMs) risk inadvertently memorizing and divulging sensitive or personally identifiable information (PII) seen in training data, causing privacy concerns. Current approaches to address this issue involve costly dataset scrubbing, or model filtering through unlearning and model editing, which can be bypassed through extraction attacks. We propose REVS, a novel model editing method for unlearning sensitive information from LLMs. REVS identifies and modifies a small subset of neurons relevant for each piece of sensitive information. By projecting these neurons to the vocabulary space (unembedding), we pinpoint the components driving its generation. We then compute a model edit based on the pseudo-inverse of the unembedding matrix, and apply it to de-promote generation of the targeted sensitive data. To adequately evaluate our method on truly sensitive information, we curate two datasets: an email dataset inherently memorized by GPT-J, and a synthetic social security number dataset that we tune the model to memorize. Compared to other state-of-the-art model editing methods, REVS demonstrates superior performance in both eliminating sensitive information and robustness to extraction attacks, while retaining integrity of the underlying model. The code and a demo notebook are available at https://technion-cs-nlp.github.io/REVS.


title: PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance

abstract: Recently, artificial intelligence techniques for education have been received increasing attentions, while it still remains an open problem to design the effective music instrument instructing systems. Although key presses can be directly derived from sheet music, the transitional movements among key presses require more extensive guidance in piano performance. In this work, we construct a piano-hand motion generation benchmark to guide hand movements and fingerings for piano playing. To this end, we collect an annotated dataset, PianoMotion10M, consisting of 116 hours of piano playing videos from a bird's-eye view with 10 million annotated hand poses. We also introduce a powerful baseline model that generates hand motions from piano audios through a position predictor and a position-guided gesture generator. Furthermore, a series of evaluation metrics are designed to assess the performance of the baseline model, including motion similarity, smoothness, positional accuracy of left and right hands, and overall fidelity of movement distribution. Despite that piano key presses with respect to music scores or audios are already accessible, PianoMotion10M aims to provide guidance on piano fingering for instruction purposes. The dataset and source code can be accessed at https://agnjason.github.io/PianoMotion-page.


title: Towards AI Lesion Tracking in PET/CT Imaging: A Siamese-based CNN Pipeline applied on PSMA PET/CT Scans

abstract: Assessing tumor response to systemic therapies is one of the main applications of PET/CT. Routinely, only a small subset of index lesions out of multiple lesions is analyzed. However, this operator dependent selection may bias the results due to possible significant inter-metastatic heterogeneity of response to therapy. Automated, AI based approaches for lesion tracking hold promise in enabling the analysis of many more lesions and thus providing a better assessment of tumor response. This work introduces a Siamese CNN approach for lesion tracking between PET/CT scans. Our approach is applied on the laborious task of tracking a high number of bone lesions in full-body baseline and follow-up [68Ga]Ga- or [18F]F-PSMA PET/CT scans after two cycles of [177Lu]Lu-PSMA therapy of metastatic castration resistant prostate cancer patients. Data preparation includes lesion segmentation and affine registration. Our algorithm extracts suitable lesion patches and forwards them into a Siamese CNN trained to classify the lesion patch pairs as corresponding or non-corresponding lesions. Experiments have been performed with different input patch types and a Siamese network in 2D and 3D. The CNN model successfully learned to classify lesion assignments, reaching a lesion tracking accuracy of 83 % in its best configuration with an AUC = 0.91. For remaining lesions the pipeline accomplished a re-identification rate of 89 %. We proved that a CNN may facilitate the tracking of multiple lesions in PSMA PET/CT scans. Future clinical studies are necessary if this improves the prediction of the outcome of therapies.


title: Learnable Fractal Flames

abstract: This work presents a differentiable rendering approach that allows latent fractal flame parameters to be learned from image supervision. The approach extends the state-of-the-art in differentiable fractal rendering through support for color images, non-linear generator functions, and multi-fractal compositions. With these additions, differentiable rendering is now a viable tool for the generation of fractal artwork.


title: Is Value Learning Really the Main Bottleneck in Offline RL?

abstract: While imitation learning requires access to high-quality data, offline reinforcement learning (RL) should, in principle, perform similarly or better with substantially lower data quality by using a value function. However, current results indicate that offline RL often performs worse than imitation learning, and it is often unclear what holds back the performance of offline RL. Motivated by this observation, we aim to understand the bottlenecks in current offline RL algorithms. While poor performance of offline RL is typically attributed to an imperfect value function, we ask: is the main bottleneck of offline RL indeed in learning the value function, or something else? To answer this question, we perform a systematic empirical study of (1) value learning, (2) policy extraction, and (3) policy generalization in offline RL problems, analyzing how these components affect performance. We make two surprising observations. First, we find that the choice of a policy extraction algorithm significantly affects the performance and scalability of offline RL, often more so than the value learning objective. For instance, we show that common value-weighted behavioral cloning objectives (e.g., AWR) do not fully leverage the learned value function, and switching to behavior-constrained policy gradient objectives (e.g., DDPG+BC) often leads to substantial improvements in performance and scalability. Second, we find that a big barrier to improving offline RL performance is often imperfect policy generalization on test-time states out of the support of the training data, rather than policy learning on in-distribution states. We then show that the use of suboptimal but high-coverage data or test-time policy training techniques can address this generalization issue in practice. Specifically, we propose two simple test-time policy improvement methods and show that these methods lead to better performance.


title: Learning from Natural Language Explanations for Generalizable Entity Matching

abstract: Entity matching is the task of linking records from different sources that refer to the same real-world entity. Past work has primarily treated entity linking as a standard supervised learning problem. However, supervised entity matching models often do not generalize well to new data, and collecting exhaustive labeled training data is often cost prohibitive. Further, recent efforts have adopted LLMs for this task in few/zero-shot settings, exploiting their general knowledge. But LLMs are prohibitively expensive for performing inference at scale for real-world entity matching tasks. As an efficient alternative, we re-cast entity matching as a conditional generation task as opposed to binary classification. This enables us to "distill" LLM reasoning into smaller entity matching models via natural language explanations. This approach achieves strong performance, especially on out-of-domain generalization tests (10.85% F-1) where standalone generative methods struggle. We perform ablations that highlight the importance of explanations, both for performance and model robustness.


title: Topological isotopy and finite type invariants

abstract: In 1974, D. Rolfsen asked: If two PL links in $S^3$ are isotopic (=homotopic through embeddings), then are they PL isotopic? We prove that they are PL isotopic to another pair of links which are indistinguishable from each other by finite type invariants. Thus if finite type invariants separate PL links in $S^3$, then Rolfsen's problem has an affirmative solution. In fact, we show that finite type invariants separate PL links in $S^3$ if and only if certain 6 rather diverse conjectures hold simultaneously. We also show that if $v$ is a finite type invariant (or more generally a colored finite type invariant) of PL links, and $v$ is invariant under PL isotopy, then $v$ assumes the same value on all sufficiently close $C^0$-approximations of any given topological link; moreover, the extension of $v$ by continuity to topological links is an invariant of isotopy. Some specific invariants of this kind are discussed.


title: RoTipBot: Robotic Handling of Thin and Flexible Objects using Rotatable Tactile Sensors

abstract: This paper introduces RoTipBot, a novel robotic system for handling thin, flexible objects. Different from previous works that are limited to singulating them using suction cups or soft grippers, RoTipBot can grasp and count multiple layers simultaneously, emulating human handling in various environments. Specifically, we develop a novel vision-based tactile sensor named RoTip that can rotate and sense contact information around its tip. Equipped with two RoTip sensors, RoTipBot feeds multiple layers of thin, flexible objects into the centre between its fingers, enabling effective grasping and counting. RoTip's tactile sensing ensures both fingers maintain good contact with the object, and an adjustment approach is designed to allow the gripper to adapt to changes in the object. Extensive experiments demonstrate the efficacy of the RoTip sensor and the RoTipBot approach. The results show that RoTipBot not only achieves a higher success rate but also grasps and counts multiple layers simultaneously -- capabilities not possible with previous methods. Furthermore, RoTipBot operates up to three times faster than state-of-the-art methods. The success of RoTipBot paves the way for future research in object manipulation using mobilised tactile sensors. All the materials used in this paper are available at \url{https://sites.google.com/view/rotipbot}.


title: Memory-Efficient Sparse Pyramid Attention Networks for Whole Slide Image Analysis

abstract: Whole Slide Images (WSIs) are crucial for modern pathological diagnosis, yet their gigapixel-scale resolutions and sparse informative regions pose significant computational challenges. Traditional dense attention mechanisms, widely used in computer vision and natural language processing, are impractical for WSI analysis due to the substantial data scale and the redundant processing of uninformative areas. To address these challenges, we propose Memory-Efficient Sparse Pyramid Attention Networks with Shifted Windows (SPAN), drawing inspiration from state-of-the-art sparse attention techniques in other domains. SPAN introduces a sparse pyramid attention architecture that hierarchically focuses on informative regions within the WSI, aiming to reduce memory overhead while preserving critical features. Additionally, the incorporation of shifted windows enables the model to capture long-range contextual dependencies essential for accurate classification. We evaluated SPAN on multiple public WSI datasets, observing its competitive performance. Unlike existing methods that often struggle to model spatial and contextual information due to memory constraints, our approach enables the accurate modeling of these crucial features. Our study also highlights the importance of key design elements in attention mechanisms, such as the shifted-window scheme and the hierarchical structure, which contribute substantially to the effectiveness of SPAN in WSI analysis. The potential of SPAN for memory-efficient and effective analysis of WSI data is thus demonstrated, and the code will be made publicly available following the publication of this work.


title: ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models

abstract: Performance prediction is a method to estimate the performance of Language Models (LMs) on various Natural Language Processing (NLP) tasks, mitigating computational costs associated with model capacity and data for fine-tuning. Our paper introduces ProxyLM, a scalable framework for predicting LM performance using proxy models in multilingual tasks. These proxy models act as surrogates, approximating the performance of the LM of interest. By leveraging proxy models, ProxyLM significantly reduces computational overhead on task evaluations, achieving up to a 37.08x speedup compared to traditional methods, even with our smallest proxy models. Additionally, our methodology showcases adaptability to previously unseen languages in pre-trained LMs, outperforming the state-of-the-art performance by 1.89x as measured by root-mean-square error (RMSE). This framework streamlines model selection, enabling efficient deployment and iterative LM enhancements without extensive computational resources.


title: Instance-level quantitative saliency in multiple sclerosis lesion segmentation

abstract: In recent years, explainable methods for artificial intelligence (XAI) have tried to reveal and describe models' decision mechanisms in the case of classification tasks. However, XAI for semantic segmentation and in particular for single instances has been little studied to date. Understanding the process underlying automatic segmentation of single instances is crucial to reveal what information was used to detect and segment a given object of interest. In this study, we proposed two instance-level explanation maps for semantic segmentation based on SmoothGrad and Grad-CAM++ methods. Then, we investigated their relevance for the detection and segmentation of white matter lesions (WML), a magnetic resonance imaging (MRI) biomarker in multiple sclerosis (MS). 687 patients diagnosed with MS for a total of 4043 FLAIR and MPRAGE MRI scans were collected at the University Hospital of Basel, Switzerland. Data were randomly split into training, validation and test sets to train a 3D U-Net for MS lesion segmentation. We observed 3050 true positive (TP), 1818 false positive (FP), and 789 false negative (FN) cases. We generated instance-level explanation maps for semantic segmentation, by developing two XAI methods based on SmoothGrad and Grad-CAM++. We investigated: 1) the distribution of gradients in saliency maps with respect to both input MRI sequences; 2) the model's response in the case of synthetic lesions; 3) the amount of perilesional tissue needed by the model to segment a lesion. Saliency maps (based on SmoothGrad) in FLAIR showed positive values inside a lesion and negative in its neighborhood. Peak values of saliency maps generated for these four groups of volumes presented distributions that differ significantly from one another, suggesting a quantitative nature of the proposed saliency. Contextual information of 7mm around the lesion border was required for their segmentation.


title: Perfectly hidden order and Z2 confinement transition in a fully packed monopole liquid

abstract: We investigate a simple variant of spin ice whose degenerate ground states are densely packed monopole configurations. An applied field drives this model through a Z2 confinement transition, in absence of any local order parameter. Instead, this hidden order turns out to be diagnosed by a string order invisible to any local probe. We describe the transition in terms of a bosonic field theory with a pairing term, as well as a Kramers-Wannier duality into a 3D Ising model, which establishes its Z2 nature. This topological transition can be thought of as a variant of the celebrated U(1) Kasteleyn transition; but instead of the traditional '3/2'-order kink, the system shows critical scaling expected near a 3D Ising transition. Remarkably, however, the magnetic response scales with the critical exponent not of the susceptibility, but of the specific heat.


title: Different Planetary Eccentricity-Period (PEP) Distributions of Small- and Giant-Planets

abstract: We used the database of $1040$ short-period ($1 \leq P < 200$ days) exoplanets radial-velocity (RV) orbits to study the planetary eccentricity-period (PEP) distribution. We first divided the sample into low- and high-mass exoplanet sub-samples based on the distribution of the (minimum) planetary masses, which displays a clear two-Gaussian distribution, separated at $0.165M_J$. We then selected $216$ orbits, low- and high-mass alike, with eccentricities significantly distinct from circular orbits. The $131$ giant-planet eccentric orbits display a clear upper envelope, which we model quantitatively, rises monotonically from zero eccentricity and reaches an eccentricity of $0.8$ at $P \sim 100$ days. Conversely, the $85$ low-mass planetary orbits display a flat eccentricity distribution between $0.1$ and $0.5$, with almost no dependence on the orbital period. We show that the striking difference between the two PEP distributions is not a result of the detection technique used. The upper envelope of the high-mass planets, also seen in short-period binary stars, is a clear signature of tidal circularization, which probably took place inside the planets, while the small-planet PEP distribution suggests that the circularization was not effective, probably due to dynamical interactions with neighboring planets.


title: Learning the Influence Graph of a High-Dimensional Markov Process with Memory

abstract: Motivated by multiple applications in social networks, nervous systems, and financial risk analysis, we consider the problem of learning the underlying (directed) influence graph or causal graph of a high-dimensional multivariate discrete-time Markov process with memory. At any discrete time instant, each observed variable of the multivariate process is a binary string of random length, which is parameterized by an unobservable or hidden [0,1]-valued scalar. The hidden scalars corresponding to the variables evolve according to discrete-time linear stochastic dynamics dictated by the underlying influence graph whose nodes are the variables. We extend an existing algorithm for learning i.i.d. graphical models to this Markovian setting with memory and prove that it can learn the influence graph based on the binary observations using logarithmic (in number of variables or nodes) samples when the degree of the influence graph is bounded. The crucial analytical contribution of this work is the derivation of the sample complexity result by upper and lower bounding the rate of convergence of the observed Markov process with memory to its stationary distribution in terms of the parameters of the influence graph.


title: Wigner function method for the Gibbons-Hawking and the Unruh effect

abstract: An observer at rest with the expanding universe experiences some extra noise in the quantum vacuum, and so does an accelerated observer in a vacuum at rest (in Minkowski space). The literature mainly focuses on the ideal cases of exponential expansion (de-Sitter space) or uniform acceleration (Rindler trajectories) or both, but the real cosmic expansion is non-exponential and real accelerations are non-uniform. Here we use the frequency-time Wigner function of vacuum correlations to define time-dependent spectra. We found excellent Planck spectra for a class of realistic cosmological models, but also strongly non-Planckian, negative Wigner functions for a standard scenario testable with laboratory analogues.


title: Prospects for NMR Spectral Prediction on Fault-Tolerant Quantum Computers

abstract: Nuclear magnetic resonance spectroscopy is a prominent analytical tool, with applications throughout chemistry, medicine and solid-state physics. While conventional NMR spectrometers require large magnetic fields to interrogate a sample, recent advances in atomic magnetometry have enabled this spectroscopy far below geomagnetic field strengths. This zero-to-ultralow (ZULF) field regime can be advantageous since it mitigates relaxation and reveals spin couplings that are otherwise obscured, all while using compact and lower-overhead instrumentation. The resulting spectra are nonetheless difficult to interpret without computation, which can be taxing due to the presence of vector couplings and long-range spin networks. Following recent proposals, we demonstrate how fault-tolerant quantum computation could be used to simulate these spectra. Our analysis spans from input selection to the construction of explicit circuits based on qubitized quantum dynamics. By maintaining parity with experimental requirements, we demonstrate how NMR spectral prediction might be an early application for fault-tolerant quantum computers.


title: Investigate the Performance of Distribution Loading with Conditional Quantum Generative Adversarial Network Algorithm on Quantum Hardware with Error Suppression

abstract: The study examines the efficacy of the Fire Opal error suppression and AI circuit optimization system integrated with IBM's quantum computing platform for a multi-modal distribution loading algorithm. Using Kullback-Leibler (KL) divergence as a quantitative error analysis, the results indicate that Fire Opal can improve on the time-dependent distributions generated by our Conditional Quantum Generative Adversarial algorithm by 30-40\% in comparison with the results on the simulator. In addition, Fire Opal's performance remains consistent for complex circuits despite the needs to run more trials. The research concludes that Fire Opal's error suppression and circuit optimization significantly enhanced quantum computing processes, highlighting its potential for practical applications. In addition, the study also reviews leading error mitigation strategies, including zero noise extrapolation (ZNE), probabilistic error cancellation (PEC), Pauli twirling, measurement error mitigation, and machine learning methods, assessing their advantages and disadvantages in terms of technical implementation, quantum resources, and scalability.


title: Wavefront shaping simulations with augmented partial factorization

abstract: Wavefront shaping can tailor multipath interference to control multiple scattering of waves in complex optical systems. However, full-wave simulations that capture multiple scattering are computationally demanding given the large system size and the large number of input channels. Recently, an "augmented partial factorization" (APF) method was proposed to significantly speed-up such full-wave simulations. In this tutorial, we illustrate how to perform wavefront shaping simulations with the APF method using the open-source frequency-domain electromagnetic scattering solver MESTI. We present the foundational concepts and then walk through four examples: computing the scattering matrix of a slab with random permittivities, open high-transmission channels through disorder, focusing inside disorder with phase conjugation, and reflection matrix computation in a spatial focused-beam basis. The goal is to lower the barrier for researchers to use simulations to explore the rich phenomena enabled by wavefront shaping.


title: Frameworks, Modeling and Simulations of Misinformation and Disinformation: A Systematic Literature Review

abstract: The prevalence of misinformation and disinformation poses a significant challenge in today's digital landscape. That is why several methods and tools are proposed to analyze and understand these phenomena from a scientific perspective. To assess how the mis/disinformation is being conceptualized and evaluated in the literature, this paper surveys the existing frameworks, models and simulations of mis/disinformation dynamics by performing a systematic literature review up to 2023. After applying the PRISMA methodology, 57 research papers are inspected to determine (1) the terminology and definitions of mis/disinformation, (2) the methods used to represent mis/disinformation, (3) the primary purpose beyond modeling and simulating mis/disinformation, (4) the context where the mis/disinformation is studied, and (5) the validation of the proposed methods for understanding mis/disinformation. The main findings reveal a consistent essence definition of misinformation and disinformation across studies, with intent as the key distinguishing factor. Research predominantly uses social frameworks, epidemiological models, and belief updating simulations. These studies aim to estimate the effectiveness of mis/disinformation, primarily in health and politics. The preferred validation strategy is to compare methods with real-world data and statistics. Finally, this paper identifies current trends and open challenges in the mis/disinformation research field, providing recommendations for future work agenda.


title: Hamiltonian stationary maps with infinitely many singularities

abstract: For any $k\in \mathbb{N}$ we construct an Hamiltonian stationary Lagrangian map from a disc to $\mathbb{C}^2$ with infinitely many Schoen-Wolfson singularities which is of class $C^k$ up to the boundary and has smooth trace.


title: DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding

abstract: The integration of pre-trained text-based large language models (LLM) with speech input has enabled instruction-following capabilities for diverse speech tasks. This integration requires the use of a speech encoder, a speech adapter, and an LLM, trained on diverse tasks. We propose the use of discrete speech units (DSU), rather than continuous-valued speech encoder outputs, that are converted to the LLM token embedding space using the speech adapter. We generate DSU using a self-supervised speech encoder followed by k-means clustering. The proposed model shows robust performance on speech inputs from seen/unseen domains and instruction-following capability in spoken question answering. We also explore various types of DSU extracted from different layers of the self-supervised speech encoder, as well as Mel frequency Cepstral Coefficients (MFCC). Our findings suggest that the ASR task and datasets are not crucial in instruction-tuning for spoken question answering tasks.


title: Scoreformer: A Surrogate Model For Large-Scale Prediction of Docking Scores

abstract: In this study, we present ScoreFormer, a novel graph transformer model designed to accurately predict molecular docking scores, thereby optimizing high-throughput virtual screening (HTVS) in drug discovery. The architecture integrates Principal Neighborhood Aggregation (PNA) and Learnable Random Walk Positional Encodings (LRWPE), enhancing the model's ability to understand complex molecular structures and their relationship with their respective docking scores. This approach significantly surpasses traditional HTVS methods and recent Graph Neural Network (GNN) models in both recovery and efficiency due to a wider coverage of the chemical space and enhanced performance. Our results demonstrate that ScoreFormer achieves competitive performance in docking score prediction and offers a substantial 1.65-fold reduction in inference time compared to existing models. We evaluated ScoreFormer across multiple datasets under various conditions, confirming its robustness and reliability in identifying potential drug candidates rapidly.


title: Separations in the Representational Capabilities of Transformers and Recurrent Architectures

abstract: Transformer architectures have been widely adopted in foundation models. Due to their high inference costs, there is renewed interest in exploring the potential of efficient recurrent architectures (RNNs). In this paper, we analyze the differences in the representational capabilities of Transformers and RNNs across several tasks of practical relevance, including index lookup, nearest neighbor, recognizing bounded Dyck languages, and string equality. For the tasks considered, our results show separations based on the size of the model required for different architectures. For example, we show that a one-layer Transformer of logarithmic width can perform index lookup, whereas an RNN requires a hidden state of linear size. Conversely, while constant-size RNNs can recognize bounded Dyck languages, we show that one-layer Transformers require a linear size for this task. Furthermore, we show that two-layer Transformers of logarithmic size can perform decision tasks such as string equality or disjointness, whereas both one-layer Transformers and recurrent models require linear size for these tasks. We also show that a log-size two-layer Transformer can implement the nearest neighbor algorithm in its forward pass; on the other hand recurrent models require linear size. Our constructions are based on the existence of $N$ nearly orthogonal vectors in $O(\log N)$ dimensional space and our lower bounds are based on reductions from communication complexity problems. We supplement our theoretical results with experiments that highlight the differences in the performance of these architectures on practical-size sequences.


title: Emergence of Fluctuation Relations in UNO

abstract: Fluctuation theorems are generalisations of the second law that describe the relations between work, temperature, and free energy in thermodynamic processes and are used extensively in studies of irreversibility and entropy. Many experiments have verified these relations for different physical systems in the setting of thermodynamics. In this study, we observe the same behavior away from physical thermodynamics, namely for the card game UNO, by performing numerical simulations of the game. As the analog of work, we choose the number of steps one player needs to effect a transition in her deck; the other players and the remaining cards play the role of a finite, non-Markovian bath. We also compare our observation with is expected for a Markovian random walk.


title: Measuring the CMB spectral distortions with COSMO: the multi-mode antenna system

abstract: In this work, we present the design and manufacturing of the two multi-mode antenna arrays of the COSMO experiment and the preliminary beam pattern measurements of their fundamental mode compared with simulations. COSMO is a cryogenic Martin-Puplett Fourier Transform Spectrometer that aims at measuring the isotropic y-type spectral distortion of the Cosmic Microwave Background from Antarctica, by performing differential measurements between the sky and an internal, cryogenic reference blackbody. To reduce the atmospheric contribution, a spinning wedge mirror performs fast sky-dips at varying elevations while fast, low-noise Kinetic Inductance detectors scan the interferogram. Two arrays of antennas couple the radiation to the detectors. Each array consists of nine smooth-walled multi-mode feed-horns, operating in the $120-180$ GHz and $210-300$ GHz range, respectively. The multi-mode propagation helps increase the instrumental sensitivity without employing large focal planes with hundreds of detectors. The two arrays have a step-linear and a linear profile, respectively, and are obtained by superimposing aluminum plates made with CNC milling. The simulated multi-mode beam pattern has a $\sim 20^{\circ} - 26^{\circ}$ FWHM for the low-frequency array and $\sim 16^{\circ}$ FWHM for the high-frequency one. The side lobes are below $-15$ dB. To characterize the antenna response, we measured the beam pattern of the fundamental mode using a Vector Network Analyzer, in far-field conditions inside an anechoic chamber at room temperature. We completed the measurements of the low-frequency array and found a good agreement with the simulations. We also identified a few non-idealities that we attribute to the measuring setup and will further investigate. A comprehensive multi-mode measurement will be feasible at cryogenic temperature once the full receiver is integrated.


title: Quantum statistics in the minimal scenario

abstract: In any given experimental scenario, the rules of quantum theory provide statistical distributions that the observed outcomes are expected to follow. The set formed by all these distributions contains the imprint of quantum theory, capturing some of its core properties. So far, only partial descriptions have been known for this set, even in the simplest scenarios. Here, we obtain the analytical description of a complete set of quantum statistics in terms of extremal points. This is made possible by finding all bipartite quantum states and pairs of binary measurements which can be self-tested, i.e. identified from statistics only. Our description provides a direct insight into the properties and limitations of quantum theory. These are not expressed in terms of Hilbert spaces, but rather directly in terms of measurement observation statistics.


title: On the Expressibility of the Reconstructional Color Refinement

abstract: One of the most basic facts related to the famous Ulam reconstruction conjecture is that the connectedness of a graph can be determined by the deck of its vertex-deleted subgraphs, which are considered up to isomorphism. We strengthen this result by proving that connectedness can still be determined when the subgraphs in the deck are given up to equivalence under the color refinement isomorphism test. Consequently, this implies that connectedness is recognizable by Reconstruction Graph Neural Networks, a recently introduced GNN architecture inspired by the reconstruction conjecture (Cotta, Morris, Ribeiro 2021).


title: An optical atomic clock using $4D_J$ states of rubidium

abstract: We analyze an optical atomic clock using two-photon $5S_{1/2} \rightarrow 4D_J$ transitions in rubidium. Four one- and two-color excitation schemes to probe the fine-structure states $4D_{3/2}$ and $4D_{5/2}$ are considered in detail. We compare key characteristics of Rb $4D_J$ and $5D_{5/2}$ two-photon clocks. The $4D_J$ clock features a high signal-to-noise ratio due to two-photon decay at favorable wavelengths, low dc electric and magnetic susceptibilities, and minimal black-body shifts. Ac Stark shifts from the clock interrogation lasers are compensated by two-color Rabi-frequency matching. We identify a "magic" wavelength near 1060~nm, which allows for in-trap, Doppler-free clock-transition interrogation with lattice-trapped cold atoms. From our analysis of clock statistics and systematics, we project a quantum-noise-limited relative clock stability at the $10^{-13}/\sqrt{\tau(s)}$-level, with integration time $\tau$ in seconds, and a relative accuracy of $\sim 10^{-13}$. We describe a potential architecture for implementing the proposed clock using a single telecom clock laser at 1550~nm, which is conducive to optical communication and long-distance clock comparisons. Our work could be of interest in efforts to realize small and portable Rb clocks and in high-precision measurements of atomic properties of Rb $4D_J$-states.


title: Enhancing Domain Adaptation through Prompt Gradient Alignment

abstract: Prior Unsupervised Domain Adaptation (UDA) methods often aim to train a domain-invariant feature extractor, which may hinder the model from learning sufficiently discriminative features. To tackle this, a line of works based on prompt learning leverages the power of large-scale pre-trained vision-language models to learn both domain-invariant and specific features through a set of domain-agnostic and domain-specific learnable prompts. Those studies typically enforce invariant constraints on representation, output, or prompt space to learn such prompts. Differently, we cast UDA as a multiple-objective optimization problem in which each objective is represented by a domain loss. Under this new framework, we propose aligning per-objective gradients to foster consensus between them. Additionally, to prevent potential overfitting when fine-tuning this deep learning architecture, we penalize the norm of these gradients. To achieve these goals, we devise a practical gradient update procedure that can work under both single-source and multi-source UDA. Empirically, our method consistently surpasses other prompt-based baselines by a large margin on different UDA benchmarks


title: Foundations for reconstructing early microbial life

abstract: For more than 3.5 billion years, life experienced dramatic environmental extremes on Earth. These include shifts from oxygen-less to over-oxygenated atmospheres and cycling between hothouse conditions and global glaciations. Meanwhile, an ecological revolution took place. The planet evolved from one dominated by microbial life to one containing the plants and animals that are most familiar today. The activities of many key cellular inventions evolved early in the history of life, collectively defining the nature of our biosphere and underpinning human survival. There is a critical need for a new disciplinary synthesis to reveal how microbes and their molecular systems survived ever changing global conditions over deep time. This review critically examines our current understanding of early microbial life and describes the foundations of an emerging area in microbiology and evolutionary synthetic biology to reconstruct the earliest microbial innovations.


title: Can't Hide Behind the API: Stealing Black-Box Commercial Embedding Models

abstract: Embedding models that generate representation vectors from natural language text are widely used, reflect substantial investments, and carry significant commercial value. Companies such as OpenAI and Cohere have developed competing embedding models accessed through APIs that require users to pay for usage. In this architecture, the models are "hidden" behind APIs, but this does not mean that they are "well guarded". We present, to our knowledge, the first effort to "steal" these models for retrieval by training local models on text-embedding pairs obtained from the commercial APIs. Our experiments show using standard benchmarks that it is possible to efficiently replicate the retrieval effectiveness of the commercial embedding models using an attack that costs only around $200 to train (presumably) smaller models with fewer dimensions. Our findings raise important considerations for deploying commercial embedding models and suggest measures to mitigate the risk of model theft.


title: CMC-Bench: Towards a New Paradigm of Visual Signal Compression

abstract: Ultra-low bitrate image compression is a challenging and demanding topic. With the development of Large Multimodal Models (LMMs), a Cross Modality Compression (CMC) paradigm of Image-Text-Image has emerged. Compared with traditional codecs, this semantic-level compression can reduce image data size to 0.1\% or even lower, which has strong potential applications. However, CMC has certain defects in consistency with the original image and perceptual quality. To address this problem, we introduce CMC-Bench, a benchmark of the cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models for image compression. This benchmark covers 18,000 and 40,000 images respectively to verify 6 mainstream I2T and 12 T2I models, including 160,000 subjective preference scores annotated by human experts. At ultra-low bitrates, this paper proves that the combination of some I2T and T2I models has surpassed the most advanced visual signal codecs; meanwhile, it highlights where LMMs can be further optimized toward the compression task. We encourage LMM developers to participate in this test to promote the evolution of visual signal codec protocols.


title: Advancing Graph Generation through Beta Diffusion

abstract: Diffusion models have demonstrated effectiveness in generating natural images and have been extended to generate diverse data types, including graphs. This new generation of diffusion-based graph generative models has demonstrated significant performance improvements over methods that rely on variational autoencoders or generative adversarial networks. It's important to recognize, however, that most of these models employ Gaussian or categorical diffusion processes, which can struggle with sparse and long-tailed data distributions. In our work, we introduce Graph Beta Diffusion (GBD), a diffusion-based generative model particularly adept at capturing diverse graph structures. GBD utilizes a beta diffusion process, tailored for the sparse and range-bounded characteristics of graph adjacency matrices. Furthermore, we have developed a modulation technique that enhances the realism of the generated graphs by stabilizing the generation of critical graph structures, while preserving flexibility elsewhere. The outstanding performance of GBD across three general graph benchmarks and two biochemical graph benchmarks highlights its capability to effectively capture the complexities of real-world graph data. The code will be made available at https://github.com/YH-UtMSB/Graph_Beta_Diffusion


title: Understanding Hallucinations in Diffusion Models through Mode Interpolation

abstract: Colloquially speaking, image generation models based upon diffusion processes are frequently said to exhibit "hallucinations," samples that could never occur in the training data. But where do such hallucinations come from? In this paper, we study a particular failure mode in diffusion models, which we term mode interpolation. Specifically, we find that diffusion models smoothly "interpolate" between nearby data modes in the training set, to generate samples that are completely outside the support of the original training distribution; this phenomenon leads diffusion models to generate artifacts that never existed in real data (i.e., hallucinations). We systematically study the reasons for, and the manifestation of this phenomenon. Through experiments on 1D and 2D Gaussians, we show how a discontinuous loss landscape in the diffusion model's decoder leads to a region where any smooth approximation will cause such hallucinations. Through experiments on artificial datasets with various shapes, we show how hallucination leads to the generation of combinations of shapes that never existed. Finally, we show that diffusion models in fact know when they go out of support and hallucinate. This is captured by the high variance in the trajectory of the generated sample towards the final few backward sampling process. Using a simple metric to capture this variance, we can remove over 95% of hallucinations at generation time while retaining 96% of in-support samples. We conclude our exploration by showing the implications of such hallucination (and its removal) on the collapse (and stabilization) of recursive training on synthetic data with experiments on MNIST and 2D Gaussians dataset. We release our code at https://github.com/locuslab/diffusion-model-hallucination.


title: More Likely Than You Think: Inclination-Driving Secular Resonances are Common in Known Exoplanet Systems

abstract: Multi-planet systems face significant challenges to detection. For example, further orbiting planets have reduced signal-to-noise ratio in radial velocity detection methods, and small mutual inclinations between planets can prevent them from all transiting. One mechanism to excite mutual inclination between planets is secular resonance, where the nodal precession frequencies of the planets align such as to greatly increase the efficiency of angular momentum transport between planets. These resonances can significantly misalign planets from one another, hindering detection, and typically can only occur when there are three or more planets in the system. Naively, systems can only be in resonance for particular combinations of planet semimajor axes and masses; however, effects that alter the nodal precession frequencies of the planets, such as the decay of stellar oblateness, can significantly expand the region of parameter space where resonances occur. In this work, we explore known three-planet systems, determine whether they are in (or were in) secular resonance due to evolving stellar oblateness, and demonstrate the implications of resonance on their detectability and stability. We show that about 20% of a sample of three planet transiting systems seem to undergo these resonances early in their lives.


title: On Arratia's coupling and the Dirichlet law for the factors of a random integer

abstract: Let $x \ge 2$, let $N_x$ be an integer chosen uniformly at random from the set $\mathbb Z \cap [1, x]$, and let $(V_1, V_2, \ldots)$ be a Poisson--Dirichlet process of parameter $1$. We prove that there exists a coupling of these two random objects such that $$ \mathbb E \, \sum_{i \ge 1} |\log P_i- V_i\log x| \ll 1, $$ where the implied constant is absolute and $N_x = P_1P_2 \cdots$ is the unique factorization of $N_x$ into primes or ones with the $P_i$'s being non-increasing. This establishes a conjecture of Arratia (2002) arXiv:1305.0941 who proved that the right-hand side in the above estimate can be made $\ll \log\!\log x$. We also use this coupling to give a probabilistic proof of the Dirichlet law for the average distribution of the integer factorization into $k$ parts proved in 2023 by Leung arXiv:2206.14728 and we improve on its error term.


title: On the independence number of sparser random Cayley graphs

abstract: The Cayley sum graph $\Gamma_A$ of a set $A \subseteq \mathbb{Z}_n$ is defined to have vertex set $\mathbb{Z}_n$ and an edge between two distinct vertices $x, y \in \mathbb{Z}_n$ if $x + y \in A$. Green and Morris proved that if the set $A$ is a $p$-random subset of $\mathbb{Z}_n$ with $p = 1/2$, then the independence number of $\Gamma_A$ is asymptotically equal to $\alpha(G(n, 1/2))$ with high probability. Our main theorem is the first extension of their result to $p = o(1)$: we show that, with high probability, $$\alpha(\Gamma_A) = (1 + o(1)) \alpha(G(n, p))$$ as long as $p \ge (\log n)^{-1/80}$. One of the tools in our proof is a geometric-flavoured theorem that generalises Fre\u{i}man's lemma, the classical lower bound on the size of high dimensional sumsets. We also give a short proof of this result up to a constant factor; this version yields a much simpler proof of our main theorem at the expense of a worse constant.


title: L\'evy measures on Banach spaces

abstract: We establish an explicit characterisation of L\'evy measures on both $L^p$-spaces and UMD Banach spaces. In the case of $L^p$-spaces, L\'evy measures are characterised by an integrability condition, which directly generalises the known description of L\'evy measures on sequence spaces. The latter has been the only known description of L\'evy measures on infinite dimensional Banach spaces that are not Hilbert. L\'evy measures on UMD Banach spaces are characterised by the finiteness of the expectation of a random {\gamma}-radonifying norm. Although this description is more abstract, it reduces to simple integrability conditions in the case of $L^p$-spaces.


title: ElicitationGPT: Text Elicitation Mechanisms via Language Models

abstract: Scoring rules evaluate probabilistic forecasts of an unknown state against the realized state and are a fundamental building block in the incentivized elicitation of information and the training of machine learning models. This paper develops mechanisms for scoring elicited text against ground truth text using domain-knowledge-free queries to a large language model (specifically ChatGPT) and empirically evaluates their alignment with human preferences. The empirical evaluation is conducted on peer reviews from a peer-grading dataset and in comparison to manual instructor scores for the peer reviews.


title: On the existence of magic squares of powers

abstract: For any $d \geq 2$, we prove that there exists an integer $n_0(d)$ such that there exists an $n \times n$ magic square of $d^\text{th}$ powers for all $n \geq n_0(d)$. In particular, we establish the existence of an $n \times n$ magic square of squares for all $n \geq 4$, which settles a conjecture of V\'{a}rilly-Alvarado. All previous approaches had been based on constructive methods and the existence of $n \times n$ magic squares of $d^\text{th}$ powers had only been known for sparse values of $n$. We prove our result by the Hardy-Littlewood circle method, which in this setting essentially reduces the problem to finding a sufficient number of disjoint linearly independent subsets of the columns of the coefficient matrix of the equations defining magic squares. We prove an optimal (up to a constant) lower bound for this quantity.


title: Is every knot isotopic to the unknot?

abstract: In 1974, D. Rolfsen asked: Is every knot in $S^3$ isotopic (=homotopic through embeddings) to a PL knot or, equivalently, to the unknot? In particular, is the Bing sling isotopic to a PL knot? We show that the Bing sling is not isotopic to any PL knot: (1) by an isotopy which extends to an isotopy of $2$-component links with $lk=1$; (2) through knots that are intersections of nested sequences of solid tori. There are also stronger versions of these results. In (1), the additional component may be allowed to self-intersect, and even to get replaced by a new one as long as it represents the same conjugacy class in $G/[G',G'']$, where $G$ is the fundamental group of the complement to the original component. In (2), the "solid tori" can be replaced by "boundary-link-like handlebodies", where a handlebody $V\subset S^3$ of genus $g$ is called boundary-link-like if $\pi_1(\overline{S^3-V})$ admits a homomorphism to the free group $F_g$ such that the composition $\pi_1(\partial V)\to\pi_1(\overline{S^3-V})\to F_g$ is surjective.


title: Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations

abstract: Maximum Manifold Capacity Representations (MMCR) is a recent multi-view self-supervised learning (MVSSL) method that matches or surpasses other leading MVSSL methods. MMCR is intriguing because it does not fit neatly into any of the commonplace MVSSL lineages, instead originating from a statistical mechanical perspective on the linear separability of data manifolds. In this paper, we seek to improve our understanding and our utilization of MMCR. To better understand MMCR, we leverage tools from high dimensional probability to demonstrate that MMCR incentivizes alignment and uniformity of learned embeddings. We then leverage tools from information theory to show that such embeddings maximize a well-known lower bound on mutual information between views, thereby connecting the geometric perspective of MMCR to the information-theoretic perspective commonly discussed in MVSSL. To better utilize MMCR, we mathematically predict and experimentally confirm non-monotonic changes in the pretraining loss akin to double descent but with respect to atypical hyperparameters. We also discover compute scaling laws that enable predicting the pretraining loss as a function of gradients steps, batch size, embedding dimension and number of views. We then show that MMCR, originally applied to image data, is performant on multimodal image-text data. By more deeply understanding the theoretical and empirical behavior of MMCR, our work reveals insights on improving MVSSL methods.


title: Needle In A Video Haystack: A Scalable Synthetic Framework for Benchmarking Video MLLMs

abstract: Video understanding is a crucial next step for multimodal large language models (MLLMs). To probe specific aspects of video understanding ability, existing video benchmarks typically require careful video selection based on the target capability, along with laborious annotation of query-response pairs to match the specific video content. This process is both challenging and resource-intensive. In this paper, we propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation. VideoNIAH decouples test video content from their query-responses by inserting unrelated image/text 'needles' into original videos. It generates annotations solely from these needles, ensuring diversity in video sources and a variety of query-responses. Additionally, by inserting multiple needles, VideoNIAH rigorously evaluates the temporal understanding capabilities of models. We utilized VideoNIAH to compile a video benchmark VNBench, including tasks such as retrieval, ordering, and counting. VNBench can efficiently evaluate the fine-grained understanding ability and spatio-temporal modeling ability of a video model, while also supporting the long-context evaluation. Additionally, we evaluated recent video-centric multimodal large language models (MLLMs), both open-source and proprietary, providing a comprehensive analysis. We found that although proprietary models have significant advantages over open-source models, all existing video models still perform poorly on long-distance dependency tasks. VideoNIAH is a simple yet highly scalable benchmark construction framework, and we believe it will inspire future video benchmark works. The code and data are available at https://github.com/joez17/VideoNIAH.