title: Optimizing topology for quantum probing with discrete-time quantum walks

abstract: Discrete-time quantum walk (DTQW) represents a convenient mathematical framework for describing the motion of a particle on a discrete set of positions when this motion is conditioned by the values of certain internal degrees of freedom, which are usually referred to as the {\em coin} of the particle. As such, and owing to the inherent dependence of the position distribution on the coin degrees of freedom, DTQWs naturally emerge as promising candidates for quantum metrology. In this paper, we explore the use of DTQWs as quantum probes in scenarios where the parameter of interest is encoded in the internal degree of freedom of the walker, and investigate the role of the topology of the walker's space on the attainable precision. In particular, we start considering the encoding of the parameter by rotations for a walker on the line, and evaluate the quantum Fisher information (QFI) and the position Fisher information (FI), explicitly determining the optimal initial state in position space that maximizes the QFI across all encoding schemes. This allows us to understand the role of interference in the position space and to introduce an optimal topology, which maximizes the QFI of the coin parameter and makes the position FI equal to the QFI.


title: Large-scale time-multiplexed nanophotonic parametric oscillators

abstract: Arrays of nonlinear resonators offer a fertile ground for a wide range of complex phenomena and opportunities for advanced photonic sensing and computing. Recently, significant attention has focused on studying coupled resonators in special-purpose configurations either on chips or in table-top experiments. However, a path to realizing a large-scale programmable network of nonlinear photonic resonators remains elusive because of the challenges associated with simultaneously achieving strong nonlinearity, independent operation of the resonators, and programmability of the couplings. In this work, we break these barriers by realizing large-scale, time-multiplexed optical parametric oscillators (OPOs) on a single lithium niobate nanophotonic chip. We show independent operation of 70 identical OPOs in an ultrafast nanophotonic circuit. The OPOs exhibit an ultra-low threshold of a few picojoules, substantially surpassing the strength of nonlinearity of other platforms. Using our ultrafast nanophotonic circuit, a network of N OPOs with programmable all-to-all couplings requires only a few additional components. The time-multiplexed nanophotonic OPOs can enable myriad applications, including ultrafast classical and quantum information processing.


title: Physical Implementability for Reversible Magic State Manipulation

abstract: Magic states are essential for achieving universal quantum computation. This study introduces a reversible framework for the manipulation of magic states in odd dimensions, delineating a necessary and sufficient condition for the exact transformations between magic states under maps that preserve the trace of states and positivity of discrete Wigner representation. Utilizing the stochastic formalism, we demonstrate that magic mana emerges as the unique measure for such reversible magic state transformations. We propose the concept of physical implementability for characterizing the hardness and cost of maintaining reversibility. Our findings show that, analogous to the entanglement theory, going beyond the positivity constraint enables an exact reversible theory of magic manipulation, thereby hinting at a potential incongruity between the reversibility of quantum resources and the fundamental principles of quantum mechanics. Physical implementability for reversible manipulation provides a new perspective for understanding and quantifying quantum resources, contributing to an operational framework for understanding the cost of reversible quantum resource manipulation.


title: DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution

abstract: Fine-tuning large-scale pre-trained models is inherently a resource-intensive task. While it can enhance the capabilities of the model, it also incurs substantial computational costs, posing challenges to the practical application of downstream tasks. Existing parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) rely on a bypass framework that ignores the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-tuning outcomes. To address this issue, we introduce the Dynamic Low-Rank Adaptation (DoRA) method. DoRA decomposes high-rank LoRA layers into structured single-rank components, allowing for dynamic pruning of parameter budget based on their importance to specific tasks during training, which makes the most of the limited parameter budget. Experimental results demonstrate that DoRA can achieve competitive performance compared with LoRA and full model fine-tuning, and outperform various strong baselines with the same storage parameter budget. Our code is available at https://github.com/MIkumikumi0116/DoRA


title: Rethinking Transformers in Solving POMDPs

abstract: Sequential decision-making algorithms such as reinforcement learning (RL) in real-world scenarios inevitably face environments with partial observability. This paper scrutinizes the effectiveness of a popular architecture, namely Transformers, in Partially Observable Markov Decision Processes (POMDPs) and reveals its theoretical limitations. We establish that regular languages, which Transformers struggle to model, are reducible to POMDPs. This poses a significant challenge for Transformers in learning POMDP-specific inductive biases, due to their lack of inherent recurrence found in other models like RNNs. This paper casts doubt on the prevalent belief in Transformers as sequence models for RL and proposes to introduce a point-wise recurrent structure. The Deep Linear Recurrent Unit (LRU) emerges as a well-suited alternative for Partially Observable RL, with empirical results highlighting the sub-optimal performance of the Transformer and considerable strength of LRU.


title: EPOCHS Paper X: Environmental effects on Galaxy Formation and Protocluster Galaxy candidates at $4.5<z<10$ from JWST observations

abstract: In this paper we describe our search for galaxy protocluster candidates at $4.5< z < 10$ and explore the environmental and physical properties of their member galaxies identified through JWST wide-field surveys within the CEERS, JADES, and PEARLS NEP-TDF fields. Combining with HST data, we identify 2948 robust $z>4.5$ candidates within an area of 185.4 arcmin$^2$. We determine nearest neighbour statistics and galaxy environments. We find that high-$z$ galaxies in overdense environments exhibit higher star formation activity compared to those in underdense regions. Galaxies in dense environments have a slightly increased SFR at a given mass compared with galaxies in the lower density environments. At the high mass end we also find a gradual flattening of the $M_{\star}$-SFR slope. We find that galaxies in high-density regions often have redder UV slopes than those in low-density regions, suggesting more dust extinction, weaker Lyman-alpha emission and / or a higher damped Lyman-alpha absorption. We also find that the mass-size relation remains consistent and statistically similar across all environments. Furthermore, we quantitatively assess the probability of a galaxy belonging to a protocluster candidate. In total, we identified 26 overdensities at $z=5-7$ and estimate their dark matter halo masses. We find that all protocluster candidates could evolve into clusters with $M_{\rm halo} > 10^{14}M_{\odot}$ at $z = 0$, thereby supporting the theoretical and simulation predictions of cluster formation. Notably, this marks an early search for protocluster candidates in JWST wide field based on photometric data, providing valuable candidates to study cosmic structure formation at the early stages.


title: Asymptotics of rational representations for algebraic groups

abstract: We study the asymptotic behaviour of the cohomology of subgroups $\Gamma$ of an algebraic group $G$ with coefficients in the various irreducible rational representations of $G$ and raise a conjecture about it. Namely, we expect that the dimensions of these cohomology groups approximate the $\ell^2$-Betti numbers of $\Gamma$ with a controlled error term. We provide positive answers when $G$ is a product of copies of $SL_2$. As an application, we obtain new proofs of J. Lott's and W. L\"uck's computation of the $\ell^2$-Betti numbers of hyperbolic $3$-manifolds and W. Fu's upper bound on the growth of cusp forms for non totally real fields, which is sharp in the imaginary quadratic case.


title: A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness

abstract: This paper reveals a key insight that a one-layer decoder-only Transformer is equivalent to a two-layer Recurrent Neural Network (RNN). Building on this insight, we propose ARC-Tran, a novel approach for verifying the robustness of decoder-only Transformers against arbitrary perturbation spaces. Compared to ARC-Tran, current robustness verification techniques are limited either to specific and length-preserving perturbations like word substitutions or to recursive models like LSTMs. ARC-Tran addresses these limitations by meticulously managing position encoding to prevent mismatches and by utilizing our key insight to achieve precise and scalable verification. Our evaluation shows that ARC-Tran (1) trains models more robust to arbitrary perturbation spaces than those produced by existing techniques and (2) shows high certification accuracy of the resulting models.


title: Controllable Deformations in Compressible Isotropic Implicit Elasticity

abstract: For a given material, \emph{controllable deformations} are those deformations that can be maintained in the absence of body forces and by applying only boundary tractions. For a given class of materials, \emph{universal deformations} are those deformations that are controllable for any material within the class. In this paper, we characterize the universal deformations in compressible isotropic implicit elasticity defined by solids whose constitutive equations, in terms of the Cauchy stress $\boldsymbol{\sigma}$ and the left Cauchy-Green strain $\mathbf{b}$, have the implicit form $\boldsymbol{\mathsf{f}}(\boldsymbol{\sigma},\mathbf{b})=\mathbf{0}$. We prove that universal deformations are homogeneous. However, an important observation is that, unlike Cauchy (and Green) elasticity, not every homogeneous deformation is permissible for a given implicit-elastic solid. In other words, the set of universal deformations is material-dependent, yet it remains a subset of homogeneous deformations.


title: Optimized thread-block arrangement in a GPU implementation of a linear solver for atmospheric chemistry mechanisms

abstract: Earth system models (ESM) demand significant hardware resources and energy consumption to solve atmospheric chemistry processes. Recent studies have shown improved performance from running these models on GPU accelerators. Nonetheless, there is room for improvement in exploiting even more GPU resources. This study proposes an optimized distribution of the chemical solver's computational load on the GPU, named Block-cells. Additionally, we evaluate different configurations for distributing the computational load in an NVIDIA GPU. We use the linear solver from the Chemistry Across Multiple Phases (CAMP) framework as our test bed. An intermediate-complexity chemical mechanism under typical atmospheric conditions is used. Results demonstrate a 35x speedup compared to the single-CPU thread reference case. Even using the full resources of the node (40 physical cores) on the reference case, the Block-cells version outperforms them by 50%. The Block-cells approach shows promise in alleviating the computational burden of chemical solvers on GPU architectures.


title: Speech Loudness in Broadcasting and Streaming

abstract: The introduction and regulation of loudness in broadcasting and streaming brought clear benefits to the audience, e.g., a level of uniformity across programs and channels. Yet, speech loudness is frequently reported as being too low in certain passages, which can hinder the full understanding and enjoyment of movies and TV programs. This paper proposes expanding the set of loudness-based measures typically used in the industry. We focus on speech loudness, and we show that, when clean speech is not available, Deep Neural Networks (DNNs) can be used to isolate the speech signal and so to accurately estimate speech loudness, providing a more precise estimate compared to speech-gated loudness. Moreover, we define critical passages, i.e., passages in which speech is likely to be hard to understand. Critical passages are defined based on the local Speech Loudness Deviation (SLD) and the local Speech-to-Background Loudness Difference (SBLD), as SLD and SBLD significantly contribute to intelligibility and listening effort. In contrast to other more comprehensive measures of intelligibility and listening effort, SLD and SBLD can be straightforwardly measured, are intuitive, and, most importantly, can be easily controlled by adjusting the speech level in the mix or by enabling personalization at the user's end. Finally, examples are provided that show how the detection of critical passages can support the evaluation and control of the speech signal during and after content production.


title: DR-CGRA: Supporting Loop-Carried Dependencies in CGRAs Without Spilling Intermediate Values

abstract: Coarse-grain reconfigurable architectures (CGRAs) are gaining traction thanks to their performance and power efficiency. Utilizing CGRAs to accelerate the execution of tight loops holds great potential for achieving significant overall performance gains, as a substantial portion of program execution time is dedicated to tight loops. But loop parallelization using CGRAs is challenging because of loop-carried data dependencies. Traditionally, loop-carried dependencies are handled by spilling dependent values out of the reconfigurable array to a memory medium and then feeding them back to the grid. Spilling the values and feeding them back into the grid imposes additional latencies and logic that impede performance and limit parallelism. In this paper, we present the Dependency Resolved CGRA (DR-CGRA) architecture that is designed to accelerate the execution of tight loops. DR-CGRA, which is based on a massively-multithreaded CGRA, runs each iteration as a separate CGRA thread and maps loop-carried data dependencies to inter-thread communication inside the grid. This design ensures the passage of data-dependent values across loop iterations without spilling them out of the grid. The proposed DR-CGRA architecture was evaluated on various SPEC CPU 2017 benchmarks. The results demonstrated significant performance improvements, with an average speedup ranging from 2.1 to 4.5 and an overall average of 3.1 when compared to state-of-the-art CGRA architecture.


title: EM-GANSim: Real-time and Accurate EM Simulation Using Conditional GANs for 3D Indoor Scenes

abstract: We present a novel machine-learning (ML) approach (EM-GANSim) for real-time electromagnetic (EM) propagation that is used for wireless communication simulation in 3D indoor environments. Our approach uses a modified conditional Generative Adversarial Network (GAN) that incorporates encoded geometry and transmitter location while adhering to the electromagnetic propagation theory. The overall physically-inspired learning is able to predict the power distribution in 3D scenes, which is represented using heatmaps. Our overall accuracy is comparable to ray tracing-based EM simulation, as evidenced by lower mean squared error values. Furthermore, our GAN-based method drastically reduces the computation time, achieving a 5X speedup on complex benchmarks. In practice, it can compute the signal strength in a few milliseconds on any location in 3D indoor environments. We also present a large dataset of 3D models and EM ray tracing-simulated heatmaps. To the best of our knowledge, EM-GANSim is the first real-time algorithm for EM simulation in complex 3D indoor environments. We plan to release the code and the dataset.


title: Finite Fractal Dimension of uniform attractors for non-autonomous dynamical systems with infinite dimensional symbol space

abstract: The aim of this paper is to find an upper bound for the box-counting dimension of uniform attractors for non-autonomous dynamical systems. Contrary to the results in literature, we do not ask the symbol space to have finite box-counting dimension. Instead, we ask a condition on the semi-continuity of pullback attractors of the system as time goes to infinity. This semi-continuity can be achieved if we suppose the existence of finite-dimensional exponential uniform attractors for the limit symbols. After showing these new results, we apply them to study the box-counting dimension of the uniform attractor for a reaction-diffusion equation, and we find a specific forcing term such that the symbol space has infinite box-counting dimension but the uniform attractor has finite box-counting dimension anyway.


title: Fusing uncalibrated IMUs and handheld smartphone video to reconstruct knee kinematics

abstract: Video and wearable sensor data provide complementary information about human movement. Video provides a holistic understanding of the entire body in the world while wearable sensors provide high-resolution measurements of specific body segments. A robust method to fuse these modalities and obtain biomechanically accurate kinematics would have substantial utility for clinical assessment and monitoring. While multiple video-sensor fusion methods exist, most assume that a time-intensive, and often brittle, sensor-body calibration process has already been performed. In this work, we present a method to combine handheld smartphone video and uncalibrated wearable sensor data at their full temporal resolution. Our monocular, video-only, biomechanical reconstruction already performs well, with only several degrees of error at the knee during walking compared to markerless motion capture. Reconstructing from a fusion of video and wearable sensor data further reduces this error. We validate this in a mixture of people with no gait impairments, lower limb prosthesis users, and individuals with a history of stroke. We also show that sensor data allows tracking through periods of visual occlusion.


title: Predict joint angle of body parts based on sequence pattern recognition

abstract: The way organs are positioned and moved in the workplace can cause pain and physical harm. Therefore, ergonomists use ergonomic risk assessments based on visual observation of the workplace, or review pictures and videos taken in the workplace. Sometimes the workers in the photos are not in perfect condition. Some parts of the workers' bodies may not be in the camera's field of view, could be obscured by objects, or by self-occlusion, this is the main problem in 2D human posture recognition. It is difficult to predict the position of body parts when they are not visible in the image, and geometric mathematical methods are not entirely suitable for this purpose. Therefore, we created a dataset with artificial images of a 3D human model, specifically for painful postures, and real human photos from different viewpoints. Each image we captured was based on a predefined joint angle for each 3D model or human model. We created various images, including images where some body parts are not visible. Nevertheless, the joint angle is estimated beforehand, so we could study the case by converting the input images into the sequence of joint connections between predefined body parts and extracting the desired joint angle with a convolutional neural network. In the end, we obtained root mean square error (RMSE) of 12.89 and mean absolute error (MAE) of 4.7 on the test dataset.


title: Model-Agnostic Zeroth-Order Policy Optimization for Meta-Learning of Ergodic Linear Quadratic Regulators

abstract: Meta-learning has been proposed as a promising machine learning topic in recent years, with important applications to image classification, robotics, computer games, and control systems. In this paper, we study the problem of using meta-learning to deal with uncertainty and heterogeneity in ergodic linear quadratic regulators. We integrate the zeroth-order optimization technique with a typical meta-learning method, proposing an algorithm that omits the estimation of policy Hessian, which applies to tasks of learning a set of heterogeneous but similar linear dynamic systems. The induced meta-objective function inherits important properties of the original cost function when the set of linear dynamic systems are meta-learnable, allowing the algorithm to optimize over a learnable landscape without projection onto the feasible set. We provide a convergence result for the exact gradient descent process by analyzing the boundedness and smoothness of the gradient for the meta-objective, which justify the proposed algorithm with gradient estimation error being small. We also provide a numerical example to corroborate this perspective.


title: Muon spin relaxation in mixed perovskite (LaAlO$_3$)$_{x}$(SrAl$_{0.5}$Ta$_{0.5}$O$_3$)$_{1-x}$ with $x\simeq 0.3$

abstract: We report on muon spin relaxation ($\mu^+$SR) measurements in a mixed perovskite compound, (LaAlO$_3$)$_{x}$(SrAl$_{0.5}$Ta$_{0.5}$O$_3$)$_{1-x}$ with $x\simeq 0.3$ (LSAT), which is widely used as a single-crystalline substrate for thin film deposition. In zero applied field (ZF), muon depolarization due to the distribution of nuclear dipole fields was observed in the temperature range from 4 K to 270 K. Interestingly, $\mu^+$SR time spectra in ZF maintained a Gaussian-like feature over the entire range, while the depolarization rate exhibited a monotonic decrease with increasing temperature. This behavior may be attributed to the thermally activated diffusion of muons between a few adjacent sites within a confined space of the angstrom scale, where the motionally averaged local field that each muon experiences can remain non-zero and result in maintaining the Gaussian-like line shape. The spatial distribution of electrostatic potential at lattice interstices evaluated via density functional theory calculations suggests that such a restriction of muon diffusion paths can be caused by the random distribution of cations with different nominal valences in the mixed perovskite lattice.


title: BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction

abstract: Simulating realistic interactions among traffic agents is crucial for efficiently validating the safety of autonomous driving systems. Existing leading simulators primarily use an encoder-decoder structure to encode the historical trajectories for future simulation. However, such a paradigm complicates the model architecture, and the manual separation of history and future trajectories leads to low data utilization. To address these challenges, we propose Behavior Generative Pre-trained Transformers (BehaviorGPT), a decoder-only, autoregressive architecture designed to simulate the sequential motion of multiple agents. Crucially, our approach discards the traditional separation between "history" and "future," treating each time step as the "current" one, resulting in a simpler, more parameter- and data-efficient design that scales seamlessly with data and computation. Additionally, we introduce the Next-Patch Prediction Paradigm (NP3), which enables models to reason at the patch level of trajectories and capture long-range spatial-temporal interactions. BehaviorGPT ranks first across several metrics on the Waymo Sim Agents Benchmark, demonstrating its exceptional performance in multi-agent and agent-map interactions. We outperformed state-of-the-art models with a realism score of 0.741 and improved the minADE metric to 1.540, with an approximately 91.6% reduction in model parameters.


title: Probing the Relationship between Defects and Enhanced Mobility in MoS2 Monolayers Grown by Mo Foil

abstract: Atomic vacancies, such as chalcogen vacancies in 2D TMDs, are important in changing the host material's electronic structure and transport properties. We present a straightforward one-step method for growing monolayer MoS2 utilizing oxidized Molybdenum (Mo) foil using CVD and delve into the transport properties of as-grown samples. Devices fabricated from these MoS2 sheets exhibit excellent electrical responses, with the standout device achieving mobility exceeding 100 cm2V-1s-1. Structural analysis and optical signatures unveiled the presence of chalcogen defects within these samples. To decipher the influence of inherent defects on the electronic transport properties, we measured low-temperature transport on two distinct sets of devices exhibiting relatively high or low mobilities. Combining the thermally activated transport model with quantum capacitance calculations, we have shown the existence of shallow states near the conduction band, likely attributed to sulfur vacancies within MoS2. These vacancies are responsible for the hopping conduction of electrons in the device channel. Furthermore, our claims were substantiated through low-temperature scanning tunnelling microscopy measurements, which revealed an abundance of isolated and lateral double sulfur vacancies in Mo foil-grown samples. We found that these vacancies increase the density of states near the conduction band, inducing intrinsic n-type doping in the MoS2 channel. Consequently, this elevated conductivity enhances the field-effect mobility of MoS2 transistors. Our study offers insights into chalcogen vacancies in CVD-grown monolayer MoS2 and highlights their beneficial impact on electronic transport properties.


title: Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models

abstract: Safety alignment is the key to guiding the behaviors of large language models (LLMs) that are in line with human preferences and restrict harmful behaviors at inference time, but recent studies show that it can be easily compromised by finetuning with only a few adversarially designed training examples. We aim to measure the risks in finetuning LLMs through navigating the LLM safety landscape. We discover a new phenomenon observed universally in the model parameter space of popular open-source LLMs, termed as "safety basin": randomly perturbing model weights maintains the safety level of the original aligned model in its local neighborhood. Our discovery inspires us to propose the new VISAGE safety metric that measures the safety in LLM finetuning by probing its safety landscape. Visualizing the safety landscape of the aligned model enables us to understand how finetuning compromises safety by dragging the model away from the safety basin. LLM safety landscape also highlights the system prompt's critical role in protecting a model, and that such protection transfers to its perturbed variants within the safety basin. These observations from our safety landscape research provide new insights for future work on LLM safety community.


title: Global long root $A$-packets for $\mathsf{G}_2$: the dihedral case

abstract: Cuspidal automorphic representations $\tau$ of $\mathrm{PGL}_2$ correspond to global long root $A$-parameters for $\mathsf{G}_2$. Using an exceptional theta lift between $\mathrm{PU}_3$ and $\mathsf{G}_2$, we construct the associated global $A$-packet and prove the Arthur multiplicity formula for these representations when $\tau$ is dihedral and satisfies some technical hypotheses. We also prove that this subspace of the discrete automorphic spectrum forms a full near equivalence class. Our construction yields new examples of quaternionic modular forms on $\mathsf{G}_2$.


title: Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clients

abstract: Automatic speech recognition models require large amounts of speech recordings for training. However, the collection of such data often is cumbersome and leads to privacy concerns. Federated learning has been widely used as an effective decentralized technique that collaboratively learns a shared prediction model while keeping the data local on different clients. Unfortunately, client devices often feature limited computation and communication resources leading to practical difficulties for large models. In addition, the heterogeneity that characterizes edge devices makes it sub-optimal to generate a single model that fits all of them. Differently from the recent literature, where multiple models with different architectures are used, in this work, we propose using dynamical architectures which, employing early-exit solutions, can adapt their processing (i.e. traversed layers) depending on the input and on the operation conditions. This solution falls in the realm of partial training methods and brings two benefits: a single model is used on a variety of devices; federating the models after local training is straightforward. Experiments on public datasets show that our proposed approach is effective and can be combined with basic federated learning strategies.


title: How Does Perfect Fitting Affect Representation Learning? On the Training Dynamics of Representations in Deep Neural Networks

abstract: In this paper, we elucidate how representations in deep neural networks (DNNs) evolve during training. We focus on overparameterized learning settings where the training continues much after the trained DNN starts to perfectly fit its training data. We examine the evolution of learned representations along the entire training process, including its perfect fitting regime, and with respect to the epoch-wise double descent phenomenon. We explore the representational similarity of DNN layers, each layer with respect to its own representations throughout the training process. For this, we use two similarity metrics: (1) The centered kernel alignment (CKA) similarity; (2) Similarity of decision regions of linear classifier probes that we train for the DNN layers. Our extensive experiments discover training dynamics patterns that can emerge in layers depending on the relative layer-depth, DNN width, and architecture. We show that representations at the deeper layers evolve much more in the training when an epoch-wise double descent occurs. For Vision Transformer, we show that the perfect fitting threshold creates a transition in the evolution of representations across all the encoder blocks.


title: RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projects

abstract: Large Language Models (LLMs) have demonstrated potential in assisting with Register Transfer Level (RTL) design tasks. Nevertheless, there remains to be a significant gap in benchmarks that accurately reflect the complexity of real-world RTL projects. To address this, this paper presents RTL-Repo, a benchmark specifically designed to evaluate LLMs on large-scale RTL design projects. RTL-Repo includes a comprehensive dataset of more than 4000 Verilog code samples extracted from public GitHub repositories, with each sample providing the full context of the corresponding repository. We evaluate several state-of-the-art models on the RTL-Repo benchmark, including GPT-4, GPT-3.5, Starcoder2, alongside Verilog-specific models like VeriGen and RTLCoder, and compare their performance in generating Verilog code for complex projects. The RTL-Repo benchmark provides a valuable resource for the hardware design community to assess and compare LLMs' performance in real-world RTL design scenarios and train LLMs specifically for Verilog code generation in complex, multi-file RTL projects. RTL-Repo is open-source and publicly available on Github.


title: Classifying 2D topological phases: mapping ground states to string-nets

abstract: We prove the conjectured classification of topological phases in two spatial dimensions with gappable boundary, in a simplified setting. Two gapped ground states of lattice Hamiltonians are in the same quantum phase of matter, or topological phase, if they can be connected by a constant-depth quantum circuit. It is conjectured that the Levin-Wen string-net models exhaust all possible gapped phases with gappable boundary, and these phases are labeled by unitary modular tensor categories. We prove this under the assumption that every phase has a representative state with zero correlation length satisfying the entanglement bootstrap axioms, or a strict form of area law. Our main technical development is to transform these states into string-net states using constant-depth quantum circuits.

categories: quant-ph cond-mat.str-el hep-th math-ph math.MP


title: Rokhlin Dimension and Inductive Limit Actions on AF-algebras

abstract: Given a separable, AF-algebra A and an inductive limit action on A of a finitely generated abelian group with finite Rokhlin dimension with commuting towers, we give a local description of the associated crossed product C*-algebra. In particular, when A is unital and $\alpha \in Aut(A)$ is approximately inner and has the Rokhlin property, we conclude that $A \rtimes_{\alpha} \mathbb{Z}$ is an A$\mathbb{T}$-algebra.


title: Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention

abstract: We present Lightning Attention, the first linear attention implementation that maintains a constant training speed for various sequence lengths under fixed memory consumption. Due to the issue with cumulative summation operations (cumsum), previous linear attention implementations cannot achieve their theoretical advantage in a casual setting. However, this issue can be effectively solved by utilizing different attention calculation strategies to compute the different parts of attention. Specifically, we split the attention calculation into intra-blocks and inter-blocks and use conventional attention computation for intra-blocks and linear attention kernel tricks for inter-blocks. This eliminates the need for cumsum in the linear attention calculation. Furthermore, a tiling technique is adopted through both forward and backward procedures to take full advantage of the GPU hardware. To enhance accuracy while preserving efficacy, we introduce TransNormerLLM (TNL), a new architecture that is tailored to our lightning attention. We conduct rigorous testing on standard and self-collected datasets with varying model sizes and sequence lengths. TNL is notably more efficient than other language models. In addition, benchmark results indicate that TNL performs on par with state-of-the-art LLMs utilizing conventional transformer structures. The source code is released at github.com/OpenNLPLab/TransnormerLLM.


title: ReMoDetect: Reward Models Recognize Aligned LLM's Generations

abstract: The remarkable capabilities and easy accessibility of large language models (LLMs) have significantly increased societal risks (e.g., fake news generation), necessitating the development of LLM-generated text (LGT) detection methods for safe usage. However, detecting LGTs is challenging due to the vast number of LLMs, making it impractical to account for each LLM individually; hence, it is crucial to identify the common characteristics shared by these models. In this paper, we draw attention to a common feature of recent powerful LLMs, namely the alignment training, i.e., training LLMs to generate human-preferable texts. Our key finding is that as these aligned LLMs are trained to maximize the human preferences, they generate texts with higher estimated preferences even than human-written texts; thus, such texts are easily detected by using the reward model (i.e., an LLM trained to model human preference distribution). Based on this finding, we propose two training schemes to further improve the detection ability of the reward model, namely (i) continual preference fine-tuning to make the reward model prefer aligned LGTs even further and (ii) reward modeling of Human/LLM mixed texts (a rephrased texts from human-written texts using aligned LLMs), which serves as a median preference text corpus between LGTs and human-written texts to learn the decision boundary better. We provide an extensive evaluation by considering six text domains across twelve aligned LLMs, where our method demonstrates state-of-the-art results. Code is available at https://github.com/hyunseoklee-ai/reward_llm_detect.


title: Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective

abstract: We present the Linear Complexity Sequence Model (LCSM), a comprehensive solution that unites various sequence modeling techniques with linear complexity, including linear attention, state space model, long convolution, and linear RNN, within a single framework. The goal is to enhance comprehension of these models by analyzing the impact of each component from a cohesive and streamlined viewpoint. Specifically, we segment the modeling processes of these models into three distinct stages: Expand, Oscillation, and Shrink (EOS), with each model having its own specific settings. The Expand stage involves projecting the input signal onto a high-dimensional memory state. This is followed by recursive operations performed on the memory state in the Oscillation stage. Finally, the memory state is projected back to a low-dimensional space in the Shrink stage. We perform comprehensive experiments to analyze the impact of different stage settings on language modeling and retrieval tasks. Our results show that data-driven methods are crucial for the effectiveness of the three stages in language modeling, whereas hand-crafted methods yield better performance in retrieval tasks.


title: Supernova Remnants in Gamma Rays

abstract: In the 1960s, the remnants of supernova explosions (SNRs) were indicated as a possible source of galactic cosmic rays through the Diffusive Shock Acceleration (DSA) mechanism. Since then, the observation of gamma-ray emission from relativistic ions in these objects has been one of the main goals of high-energy astrophysics. A few dozen SNRs have been detected at GeV and TeV photon energies in the last two decades. However, these observations have shown a complex phenomenology that is not easy to reduce to the standard paradigm based on DSA acceleration. Although the understanding of these objects has greatly increased, and their nature as efficient electron and proton accelerators has been observed, it remains to be clarified whether these objects are the main contributors to galactic cosmic rays. Here, we review the observations of {\gamma}-ray emission from SNRs and the perspectives for the future.


title: Thermalization and Criticality on an Analog-Digital Quantum Simulator

abstract: Understanding how interacting particles approach thermal equilibrium is a major challenge of quantum simulators. Unlocking the full potential of such systems toward this goal requires flexible initial state preparation, precise time evolution, and extensive probes for final state characterization. We present a quantum simulator comprising 69 superconducting qubits which supports both universal quantum gates and high-fidelity analog evolution, with performance beyond the reach of classical simulation in cross-entropy benchmarking experiments. Emulating a two-dimensional (2D) XY quantum magnet, we leverage a wide range of measurement techniques to study quantum states after ramps from an antiferromagnetic initial state. We observe signatures of the classical Kosterlitz-Thouless phase transition, as well as strong deviations from Kibble-Zurek scaling predictions attributed to the interplay between quantum and classical coarsening of the correlated domains. This interpretation is corroborated by injecting variable energy density into the initial state, which enables studying the effects of the eigenstate thermalization hypothesis (ETH) in targeted parts of the eigenspectrum. Finally, we digitally prepare the system in pairwise-entangled dimer states and image the transport of energy and vorticity during thermalization. These results establish the efficacy of superconducting analog-digital quantum processors for preparing states across many-body spectra and unveiling their thermalization dynamics.


title: MindMerger: Efficient Boosting LLM Reasoning in non-English Languages

abstract: Reasoning capabilities are crucial for Large Language Models (LLMs), yet a notable gap exists between English and non-English languages. To bridge this disparity, some works fine-tune LLMs to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with an external model's outputs such as English translation text to circumvent the challenge of LLM understanding non-English. Unfortunately, these methods often underutilize the built-in skilled reasoning and useful language understanding capabilities of LLMs. In order to better utilize the minds of reasoning and language understanding in LLMs, we propose a new method, namely MindMerger, which merges LLMs with the external language understanding capabilities from multilingual models to boost the multilingual reasoning performance. Furthermore, a two-step training scheme is introduced to first train to embeded the external capabilities into LLMs and then train the collaborative utilization of the external capabilities and the built-in capabilities in LLMs. Experiments on three multilingual reasoning datasets and a language understanding dataset demonstrate that MindMerger consistently outperforms all baselines, especially in low-resource languages. Without updating the parameters of LLMs, the average accuracy improved by 6.7% and 8.0% across all languages and low-resource languages on the MGSM dataset, respectively.


title: Batteryless BLE and Light-based IoT Sensor Nodes for Reliable Environmental Sensing

abstract: The sustainable design of Internet of Things (IoT) networks encompasses considerations related to energy efficiency and autonomy as well as considerations related to reliable communications, ensuring no energy is wasted on undelivered data. Under these considerations, this work proposes the design and implementation of energy-efficient Bluetooth Low Energy (BLE) and Light-based IoT (LIoT) batteryless IoT sensor nodes powered by an indoor light Energy Harvesting Unit (EHU). Our design intends to integrate these nodes into a sensing network to improve its reliability by combining both technologies and taking advantage of their features. The nodes incorporate state-of-the-art components, such as low-power sensors and efficient System-on-Chips (SoCs). Moreover, we design a strategy for adaptive switching between active and sleep cycles as a function of the available energy, allowing the IoT nodes to continuously operate without batteries. Our results show that by adapting the duty cycle of the BLE and LIoT nodes depending on the environment's light intensity, we can ensure a continuous and reliable node operation. In particular, measurements show that our proposed BLE and LIoT node designs are able to communicate with an IoT gateway in a bidirectional way, every 19.3 and 624.6 seconds, respectively, in an energy-autonomous and reliable manner.


title: Non-Unitary Quantum Machine Learning

abstract: We introduce several novel probabilistic quantum algorithms that overcome the normal unitary restrictions in quantum machine learning by leveraging the Linear Combination of Unitaries (LCU) method. Among our contributions are quantum native implementations of Residual Networks (ResNet); demonstrating a path to avoiding barren plateaus while maintaining the complexity of models that are hard to simulate classically. Furthermore, by generalising to allow control of the strength of residual connections, we show that the lower bound of the LCU success probability can be set to any arbitrary desired value. We also implement a quantum analogue of average pooling layers from convolutional networks. Our empirical analysis demonstrates that the LCU success probability remains stable for the MNIST database, unlocking a potential quadratic advantage in terms of image size compared to classical techniques. Finally, we propose a general framework for irreducible subspace projections for quantum encoded data. Using this, we demonstrate a novel rotationally invariant encoding for point cloud data via Schur-Weyl duality. We also show how this framework can be used to parameterise and control the amount of symmetry in an encoding; demonstrating improved classification performance for partially permutation invariant encoded point cloud data when compared to non-invariant or fully permutation invariant encodings. These new general algorithmic frameworks are all constructed under the same LCU method, suggesting that even more novel algorithms could be achieved by utilising the LCU technique.


title: Turbulent circumnuclear disc and cold gas outflow in the newborn radio source 4C 31.04

abstract: We present deep kpc- and pc-scale neutral atomic hydrogen (HI) absorption observations of a very young radio source (< 5000 yrs), 4C 31.04, using the WSRT and the Global VLBI array. Using $z=0.0598$, we detect a broad absorption feature centred at the systemic velocity, and narrow absorption redshifted by 220 km/s both previously observed. Additionally, we detect a new blueshifted, broad, shallow absorption wing. At pc scales, the broad absorption at the systemic velocity is detected across the entire radio source while the shallow wing is only seen against part of the eastern lobe. The velocity dispersion of the gas is overall high ($\geq$40 km/s), and is highest (>60 km/s) in the region including the outflow and the radio hot spot. While we detect a velocity gradient along the western lobe and parts of the eastern lobe, most of the gas along the rest of the eastern lobe exhibits no signs of rotation. We therefore conclude that the radio lobes of 4C 31.04 are expanding into a circumnuclear disc, partially disrupting it and making the gas highly turbulent. The distribution of gas is predominantly smooth at the spatial resolution of ~4 pc studied here. However, clumps of gas are also present, particularly along the eastern lobe. This lobe is strongly interacting with the clouds and driving an outflow ~35 pc from the radio core, with a mass-outflow rate of $0.3 \leq \dot{M} \leq 1.4$ M$_\odot$/yr. We compare our observations with a model on the survival of atomic gas clouds in radio-jet-driven outflows and find that the existence of a sub-kpc outflow implies high gas density and inefficient mixing of the cold gas with the hot medium, leading to shorter cooling times. Overall, this provides further evidence of the strong impact of young radio jets on cold ISM and supports the predictions of simulations regarding jet$-$ISM interactions and the nature of the gas into which the jets expand.


title: KSW: Khmer Stop Word based Dictionary for Keyword Extraction

abstract: This paper introduces KSW, a Khmer-specific approach to keyword extraction that leverages a specialized stop word dictionary. Due to the limited availability of natural language processing resources for the Khmer language, effective keyword extraction has been a significant challenge. KSW addresses this by developing a tailored stop word dictionary and implementing a preprocessing methodology to remove stop words, thereby enhancing the extraction of meaningful keywords. Our experiments demonstrate that KSW achieves substantial improvements in accuracy and relevance compared to previous methods, highlighting its potential to advance Khmer text processing and information retrieval. The KSW resources, including the stop word dictionary, are available at the following GitHub repository: (https://github.com/back-kh/KSWv2-Khmer-Stop-Word-based-Dictionary-for-Keyword-Extraction.git).


title: Dataset-learning duality and emergent criticality

abstract: In artificial neural networks, the activation dynamics of non-trainable variables is strongly coupled to the learning dynamics of trainable variables. During the activation pass, the boundary neurons (e.g., input neurons) are mapped to the bulk neurons (e.g., hidden neurons), and during the learning pass, both bulk and boundary neurons are mapped to changes in trainable variables (e.g., weights and biases). For example, in feed-forward neural networks, forward propagation is the activation pass and backward propagation is the learning pass. We show that a composition of the two maps establishes a duality map between a subspace of non-trainable boundary variables (e.g., dataset) and a tangent subspace of trainable variables (i.e., learning). In general, the dataset-learning duality is a complex non-linear map between high-dimensional spaces, but in a learning equilibrium, the problem can be linearized and reduced to many weakly coupled one-dimensional problems. We use the duality to study the emergence of criticality, or the power-law distributions of fluctuations of the trainable variables. In particular, we show that criticality can emerge in the learning system even from the dataset in a non-critical state, and that the power-law distribution can be modified by changing either the activation function or the loss function.

categories: cs.LG cond-mat.dis-nn cond-mat.stat-mech cs.NE


title: Global existence, fast signal diffusion limit, and $L^\infty$-in-time convergence rates in a competitive chemotaxis system

abstract: We study a chemotaxis system that includes two competitive prey and one predator species in a two-dimensional domain, where the movement of prey (resp. predators) is driven by chemicals secreted by predators (resp. prey), called mutually repulsive (resp. mutually attractive) chemotaxis effect. The kinetics for all species are chosen according to the competitive Lotka-Volterra equations for prey and to a Holling type functional response for the predator. Under the biologically relevant scenario that the chemicals diffuse much faster than the individual diffusion of all species and a suitable re-scaling, equations for chemical concentrations are parabolic with slow evolution of coefficient $0<\varepsilon\ll 1$. In the first main result, we show the global existence of a unique classical solution to the system for each $\varepsilon$. Secondly, we study rigorously the so-called fast signal diffusion limit, passing from the system including parabolic equations with the slow evolution to the one with all elliptic equations for chemical concentrations, i.e. the limit as $\varepsilon \to 0$. This explains why elliptic equations can be proposed for chemical concentration instead of parabolic ones with slow evolution. Thirdly, the $L^\infty$-in-time convergence rates for the fast signal diffusion limit are estimated, where the effect of the initial layer is carefully treated. Finally, the differences between the systems with and without the slow evolution, and between the systems with one or two preys are discussed due to numerical simulations.


title: EASI-Tex: Edge-Aware Mesh Texturing from Single Image

abstract: We present a novel approach for single-image mesh texturing, which employs a diffusion model with judicious conditioning to seamlessly transfer an object's texture from a single RGB image to a given 3D mesh object. We do not assume that the two objects belong to the same category, and even if they do, there can be significant discrepancies in their geometry and part proportions. Our method aims to rectify the discrepancies by conditioning a pre-trained Stable Diffusion generator with edges describing the mesh through ControlNet, and features extracted from the input image using IP-Adapter to generate textures that respect the underlying geometry of the mesh and the input texture without any optimization or training. We also introduce Image Inversion, a novel technique to quickly personalize the diffusion model for a single concept using a single image, for cases where the pre-trained IP-Adapter falls short in capturing all the details from the input image faithfully. Experimental results demonstrate the efficiency and effectiveness of our edge-aware single-image mesh texturing approach, coined EASI-Tex, in preserving the details of the input texture on diverse 3D objects, while respecting their geometry.


title: The Expressive Capacity of State Space Models: A Formal Language Perspective

abstract: Recently, recurrent models based on linear state space models (SSMs) have shown promising performance in language modeling (LM), competititve with transformers. However, there is little understanding of the in-principle abilities of such models, which could provide useful guidance to the search for better LM architectures. We present a comprehensive theoretical study of the capacity of such SSMs as it compares to that of transformers and traditional RNNs. We find that SSMs and transformers have overlapping but distinct strengths. In star-free state tracking, SSMs implement straightforward and exact solutions to problems that transformers struggle to represent exactly. They can also model bounded hierarchical structure with optimal memory even without simulating a stack. On the other hand, we identify a design choice in current SSMs that limits their expressive power. We discuss implications for SSM and LM research, and verify results empirically on a recent SSM, Mamba.


title: CrEIMBO: Cross Ensemble Interactions in Multi-view Brain Observations

abstract: Modern recordings of neural activity provide diverse observations of neurons across brain areas, behavioral conditions, and subjects -- thus presenting an exciting opportunity to reveal the fundamentals of brain-wide dynamics underlying cognitive function. Current methods, however, often fail to fully harness the richness of such data as they either provide an uninterpretable representation (e.g., via "black box" deep networks) or over-simplify the model (e.g., assume stationary dynamics or analyze each session independently). Here, instead of regarding asynchronous recordings that lack alignment in neural identity or brain areas as a limitation, we exploit these diverse views of the same brain system to learn a unified model of brain dynamics. We assume that brain observations stem from the joint activity of a set of functional neural ensembles (groups of co-active neurons) that are similar in functionality across recordings, and propose to discover the ensemble and their non-stationary dynamical interactions in a new model we term CrEIMBO (Cross-Ensemble Interactions in Multi-view Brain Observations). CrEIMBO identifies the composition of the per-session neural ensembles through graph-driven dictionary learning and models the ensemble dynamics as a latent sparse time-varying decomposition of global sub-circuits, thereby capturing non-stationary dynamics. CrEIMBO identifies multiple co-active sub-circuits while maintaining representation interpretability due to sharing sub-circuits across sessions. CrEIMBO distinguishes session-specific from global (session-invariant) computations by exploring when distinct sub-circuits are active. We demonstrate CrEIMBO's ability to recover ground truth components in synthetic data and uncover meaningful brain dynamics, capturing cross-subject and inter- and intra-area variability, in high-density electrode recordings of humans performing a memory task.


title: Cosmological constraints on curved quintessence

abstract: Dynamical dark energy has gained renewed interest due to recent theoretical and observational developments. In the present paper, we focus on a string-motivated dark energy set-up, and perform a detailed cosmological analysis of exponential quintessence with potential $V=V_0 e^{-\lambda\phi}$, allowing for non-zero spatial curvature. We first gain some physical intuition into the full evolution of such a scenario by analysing the corresponding dynamical system. Then, we test the model using a combination of Planck CMB data, DESI BAO data, as well as recent supernovae datasets. For the model parameter $\lambda$, we obtain a preference for nonzero values: $\lambda = 0.48^{+0.28}_{-0.21},\; 0.68^{+0.31}_{-0.20},\; 0.77^{+0.18}_{-0.15}$ at 68% C.L. when combining CMB+DESI with Pantheon+, Union3 and DES-Y5 supernovae datasets respectively. We find no significant hint for spatial curvature. We discuss the implications of current cosmological results for the exponential quintessence model, and more generally for dark energy in string theory.


title: Occlusion Handling in 3D Human Pose Estimation with Perturbed Positional Encoding

abstract: Understanding human behavior fundamentally relies on accurate 3D human pose estimation. Graph Convolutional Networks (GCNs) have recently shown promising advancements, delivering state-of-the-art performance with rather lightweight architectures. In the context of graph-structured data, leveraging the eigenvectors of the graph Laplacian matrix for positional encoding is effective. Yet, the approach does not specify how to handle scenarios where edges in the input graph are missing. To this end, we propose a novel positional encoding technique, PerturbPE, that extracts consistent and regular components from the eigenbasis. Our method involves applying multiple perturbations and taking their average to extract the consistent and regular component from the eigenbasis. PerturbPE leverages the Rayleigh-Schrodinger Perturbation Theorem (RSPT) for calculating the perturbed eigenvectors. Employing this labeling technique enhances the robustness and generalizability of the model. Our results support our theoretical findings, e.g. our experimental analysis observed a performance enhancement of up to $12\%$ on the Human3.6M dataset in instances where occlusion resulted in the absence of one edge. Furthermore, our novel approach significantly enhances performance in scenarios where two edges are missing, setting a new benchmark for state-of-the-art.


title: Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability

abstract: World models can foresee the outcomes of different actions, which is of paramount importance for autonomous driving. Nevertheless, existing driving world models still have limitations in generalization to unseen environments, prediction fidelity of critical details, and action controllability for flexible application. In this paper, we present Vista, a generalizable driving world model with high fidelity and versatile controllability. Based on a systematic diagnosis of existing methods, we introduce several key ingredients to address these limitations. To accurately predict real-world dynamics at high resolution, we propose two novel losses to promote the learning of moving instances and structural information. We also devise an effective latent replacement approach to inject historical frames as priors for coherent long-horizon rollouts. For action controllability, we incorporate a versatile set of controls from high-level intentions (command, goal point) to low-level maneuvers (trajectory, angle, and speed) through an efficient learning strategy. After large-scale training, the capabilities of Vista can seamlessly generalize to different scenarios. Extensive experiments on multiple datasets show that Vista outperforms the most advanced general-purpose video generator in over 70% of comparisons and surpasses the best-performing driving world model by 55% in FID and 27% in FVD. Moreover, for the first time, we utilize the capacity of Vista itself to establish a generalizable reward for real-world action evaluation without accessing the ground truth actions.


title: Transformers Can Do Arithmetic with the Right Embeddings

abstract: The poor performance of transformers on arithmetic tasks seems to stem in large part from their inability to keep track of the exact position of each digit inside of a large span of digits. We mend this problem by adding an embedding to each digit that encodes its position relative to the start of the number. In addition to the boost these embeddings provide on their own, we show that this fix enables architectural modifications such as input injection and recurrent layers to improve performance even further. With positions resolved, we can study the logical extrapolation ability of transformers. Can they solve arithmetic problems that are larger and more complex than those in their training data? We find that training on only 20 digit numbers with a single GPU for one day, we can reach state-of-the-art performance, achieving up to 99% accuracy on 100 digit addition problems. Finally, we show that these gains in numeracy also unlock improvements on other multi-step reasoning tasks including sorting and multiplication.


title: A new search pipeline for gravitational waves with higher-order modes using mode-by-mode filtering

abstract: Nearly all template-based gravitational wave (GW) searches only include the quasi-circular quadrupolar modes of the signals in their templates. Including additional degrees of freedom in the GW templates corresponding to higher-order harmonics, orbital precession, or eccentricity is challenging because: ($i$) the size of template banks and the matched-filtering cost increases significantly with the number of degrees of freedom, $(ii)$ if these additional degrees are not included properly, the search can lose sensitivity overall (due to an increase in the rate of background triggers). Here, we focus on including aligned-spin higher harmonics in GW search templates. We use a new mode-by-mode filtering approach, where we separately filter GW strain data with three harmonics [namely $(\ell, |m|)=(2,2)$, $(3,3)$ and $(4,4)$]. This results in an increase in the matched-filtering cost by only a factor of $3$ compared to that of a $(2,2)$-only search. We develop computationally cheap trigger-ranking statistics to optimally combine the different signal-to-noise ratios (SNR) timeseries from different harmonics, which ensure only physically-allowed combinations of the different harmonics are triggered on. We use an empirical template-dependent background model in our ranking statistic to account for non-Gaussian transients. In addition, we develop a tool called band eraser which specifically excises narrow time-varying noisy bands in time-frequency space (without having to excise entire time chunks in the data). New GW candidate events that we detect using our search pipeline and the details of our template banks are discussed in accompanying papers: Wadekar et al. [1] and [2] respectively. Apart from higher harmonics, we expect our methodology to also be useful for cheap and optimal searches including orbital precession and eccentricity in GW waveforms.

categories: gr-qc astro-ph.CO astro-ph.HE astro-ph.IM


title: RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control

abstract: We propose Reference-Based Modulation (RB-Modulation), a new plug-and-play solution for training-free personalization of diffusion models. Existing training-free approaches exhibit difficulties in (a) style extraction from reference images in the absence of additional style or content text descriptions, (b) unwanted content leakage from reference style images, and (c) effective composition of style and content. RB-Modulation is built on a novel stochastic optimal controller where a style descriptor encodes the desired attributes through a terminal cost. The resulting drift not only overcomes the difficulties above, but also ensures high fidelity to the reference style and adheres to the given text prompt. We also introduce a cross-attention-based feature aggregation scheme that allows RB-Modulation to decouple content and style from the reference image. With theoretical justification and empirical evidence, our framework demonstrates precise extraction and control of content and style in a training-free manner. Further, our method allows a seamless composition of content and style, which marks a departure from the dependency on external adapters or ControlNets.


title: THREAD: Thinking Deeper with Recursive Spawning

abstract: Large language models (LLMs) have shown impressive capabilities across diverse settings, but still struggle as the length and complexity of the context increases. To address this challenge, we propose Thinking Recursively and Dynamically (ThReaD). THREAD frames model generation as a thread of execution that, based on the context, can run to completion or dynamically spawn new threads. By spawning, threads can offload work (e.g., thinking, retrieving information) to child threads, which only return tokens needed for the parent thread to do its work. In effect, this enables the model to adapt, as needed, the amount of intermediate work used to produce tokens. We apply THREAD in the settings of LLM task solving and question answering, where the dynamic threading allows the model to recursively decompose the given task or question into progressively simpler sub-problems that can be solved by separate child threads. We test THREAD, implemented using a few-shot learning approach, on diverse benchmarks for agent tasks and data-grounded question answering. THREAD achieves state-of-the-art performance with GPT-4 and GPT-3.5 on these benchmarks, including ALFWorld, TextCraft, and WebShop, along with two new benchmarks, DataCommons QA and MIMIC-III ICU QA. In addition, THREAD outperforms existing frameworks by 10% to 50% absolute points with smaller models, including Llama-3-8b and CodeLlama-7b.


title: A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training

abstract: Training diffusion models is always a computation-intensive task. In this paper, we introduce a novel speed-up method for diffusion model training, called, which is based on a closer look at time steps. Our key findings are: i) Time steps can be empirically divided into acceleration, deceleration, and convergence areas based on the process increment. ii) These time steps are imbalanced, with many concentrated in the convergence area. iii) The concentrated steps provide limited benefits for diffusion training. To address this, we design an asymmetric sampling strategy that reduces the frequency of steps from the convergence area while increasing the sampling probability for steps from other areas. Additionally, we propose a weighting strategy to emphasize the importance of time steps with rapid-change process increments. As a plug-and-play and architecture-agnostic approach, SpeeD consistently achieves 3-times acceleration across various diffusion architectures, datasets, and tasks. Notably, due to its simple design, our approach significantly reduces the cost of diffusion model training with minimal overhead. Our research enables more researchers to train diffusion models at a lower cost.


title: Spectral Greedy Coresets for Graph Neural Networks

abstract: The ubiquity of large-scale graphs in node-classification tasks significantly hinders the real-world applications of Graph Neural Networks (GNNs). Node sampling, graph coarsening, and dataset condensation are effective strategies for enhancing data efficiency. However, owing to the interdependence of graph nodes, coreset selection, which selects subsets of the data examples, has not been successfully applied to speed up GNN training on large graphs, warranting special treatment. This paper studies graph coresets for GNNs and avoids the interdependence issue by selecting ego-graphs (i.e., neighborhood subgraphs around a node) based on their spectral embeddings. We decompose the coreset selection problem for GNNs into two phases: a coarse selection of widely spread ego graphs and a refined selection to diversify their topologies. We design a greedy algorithm that approximately optimizes both objectives. Our spectral greedy graph coreset (SGGC) scales to graphs with millions of nodes, obviates the need for model pre-training, and applies to low-homophily graphs. Extensive experiments on ten datasets demonstrate that SGGC outperforms other coreset methods by a wide margin, generalizes well across GNN architectures, and is much faster than graph condensation.


title: Human4DiT: Free-view Human Video Generation with 4D Diffusion Transformer

abstract: We present a novel approach for generating high-quality, spatio-temporally coherent human videos from a single image under arbitrary viewpoints. Our framework combines the strengths of U-Nets for accurate condition injection and diffusion transformers for capturing global correlations across viewpoints and time. The core is a cascaded 4D transformer architecture that factorizes attention across views, time, and spatial dimensions, enabling efficient modeling of the 4D space. Precise conditioning is achieved by injecting human identity, camera parameters, and temporal signals into the respective transformers. To train this model, we curate a multi-dimensional dataset spanning images, videos, multi-view data and 3D/4D scans, along with a multi-dimensional training strategy. Our approach overcomes the limitations of previous methods based on GAN or UNet-based diffusion models, which struggle with complex motions and viewpoint changes. Through extensive experiments, we demonstrate our method's ability to synthesize realistic, coherent and free-view human videos, paving the way for advanced multimedia applications in areas such as virtual reality and animation. Our project website is https://human4dit.github.io.


title: Deep Learning Calabi-Yau four folds with hybrid and recurrent neural network architectures

abstract: In this work, we report the results of applying deep learning based on hybrid convolutional-recurrent and purely recurrent neural network architectures to the dataset of almost one million complete intersection Calabi-Yau four-folds (CICY4) to machine-learn their four Hodge numbers $h^{1,1}, h^{2,1}, h^{3,1}, h^{2,2}$. In particular, we explored and experimented with twelve different neural network models, nine of which are convolutional-recurrent (CNN-RNN) hybrids with the RNN unit being either GRU (Gated Recurrent Unit) or Long Short Term Memory (LSTM). The remaining four models are purely recurrent neural networks based on LSTM. In terms of the $h^{1,1}, h^{2,1}, h^{3,1}, h^{2,2}$ prediction accuracies, at 72% training ratio, our best performing individual model is CNN-LSTM-400, a hybrid CNN-LSTM with the LSTM hidden size of 400, which obtained 99.74%, 98.07%, 95.19%, 81.01%, our second best performing individual model is LSTM-448, an LSTM-based model with the hidden size of 448, which obtained 99.74%, 97.51%, 94.24%, and 78.63%. These results were improved by forming ensembles of the top two, three or even four models. Our best ensemble, consisting of the top four models, achieved the accuracies of 99.84%, 98.71%, 96.26%, 85.03%. At 80% training ratio, the top two performing models LSTM-448 and LSTM-424 are both LSTM-based with the hidden sizes of 448 and 424. Compared with the 72% training ratio, there is a significant improvement of accuracies, which reached 99.85%, 98.66%, 96.26%, 84.77% for the best individual model and 99.90%, 99.03%, 97.97%, 87.34% for the best ensemble.


title: On anti-tempered local Arthur packets and a lemma of Arthur

abstract: In this note, following Arthur's ideas, we rework the process of constructing the anti-tempered local Arthur packets for quasi-split classical groups and their pure inner forms. In particular, we present explicit examples illustrating certain gap in a lemma of Arthur and provide modifications, based on the work of Moeglin, Waldspurger, and Xu.


title: Revisiting flares in Sagittarius A* based on general relativistic magnetohydrodynamic numerical simulations of black hole accretion

abstract: High-resolution observations with GRAVITY-VLTI instrument have provided abundant information about the flares in Sgr A*, the supermassive black hole in our Galactic center, including the time-dependent location of the centroid (a "hot spot"), the light curve, and polarization. Yuan et al. (2009) proposed a "coronal mass ejection" model to explain the flares and their association with the plasma ejection. The key idea is that magnetic reconnection in the accretion flow produces the flares and results in the formation and ejection of flux ropes. The dynamical process proposed in the model has been confirmed by three-dimensional GRMHD simulations in a later work. Based on this scenario, in our previous works the radiation of the flux rope has been calculated analytically and compared to the observations. In the present paper, we develop the model by directly using numerical simulation data to interpret observations. We first identify flux ropes formed due to reconnection from the data. By assuming that electrons are accelerated in the reconnection current sheet and flow into the flux rope and emit their radiation there, we have calculated the time-dependent energy distribution of electrons after phenomenologically considering their injection due to reconnection acceleration, radiative and adiabatic cooling. The radiation of these electrons is calculated using the ray-tracing approach. The trajectory of the hot spot, the radiation light curve during the flare, and the polarization are calculated. These results are compared with the GRAVITY observations and good consistencies are found.


title: Ultraheavy Ultrahigh-Energy Cosmic Rays

abstract: We investigate the propagation of ultraheavy (UH) nuclei as ultrahigh-energy cosmic rays (UHECRs). We show that their energy loss lengths at $\lesssim300$~EeV are significantly longer than those of protons and intermediate-mass nuclei, and that the highest-energy cosmic rays with energies beyond $\sim100$ EeV, including the Amaterasu particle, may originate from such UH-UHECRs. We derive constraints on the contribution of UH-UHECR sources, and find that they are consistent with energy generation rate densities of UHECRs from collapsars and neutron star mergers.


title: The Peripatetic Hater: Predicting Movement Among Hate Subreddits

abstract: Many online hate groups exist to disparage others based on race, gender identity, sex, or other characteristics. The accessibility of these communities allows users to join multiple types of hate groups (e.g., a racist community and misogynistic community), which calls into question whether these peripatetic users could be further radicalized compared to users that stay in one type of hate group. However, little is known about the dynamics of joining multiple types of hate groups, nor the effect of these groups on peripatetic users. In this paper, we develop a new method to classify hate subreddits, and the identities they disparage, which we use to better understand how users become peripatetic (join different types of hate subreddits). The hate classification technique utilizes human-validated LLMs to extract the protected identities attacked, if any, across 168 subreddits. We then cluster identity-attacking subreddits to discover three broad categories of hate: racist, anti-LGBTQ, and misogynistic. We show that becoming active in a user's first hate subreddit can cause them to become active in additional hate subreddits of a different category. We also find that users who join additional hate subreddits, especially of a different category, become more active in hate subreddits as a whole and develop a wider hate group lexicon. We are therefore motivated to train an AI model that we find usefully predicts the hate categories users will become active in based on post text read and written. The accuracy of this model may be partly driven by peripatetic users often using the language of hate subreddits they eventually join. Overall, these results highlight the unique risks associated with hate communities on a social media platform, as discussion of alternative targets of hate may lead users to target more protected identities.


title: A universal constraint on axion non-Abelian dynamics during inflation

abstract: Inflationary models equipped with Chern-Simons coupling between their axion and gauge sectors exhibit an array of interesting signals including a testable chiral gravitational wave spectrum. The energy injection in the gauge sector triggered by the rolling axion leads to a well-studied enhancement of gauge field fluctuations. These may in turn affect observables such as the scalar and tensor spectra and also account for non-linear corrections to field propagators. In this work, we focus on non-Abelian gauge sectors. We show that gauge field self-interactions and axion-gauge field non-linear couplings significantly renormalize the gauge field mode function. Operating within the regime of validity of the perturbative treatment places strong constraints on the accessible parameter space of this class of models. We calculate corrections to the gauge field propagator that are universally present in these scenarios. Enforcing perturbativity on such propagators leads to bounds that are competitive with those stemming from analytical estimates on the onset of the strong backreaction regime.


title: Towards One Model for Classical Dimensionality Reduction: A Probabilistic Perspective on UMAP and t-SNE

abstract: This paper shows that the dimensionality reduction methods, UMAP and t-SNE, can be approximately recast as MAP inference methods corresponding to a generalized Wishart-based model introduced in ProbDR. This interpretation offers deeper theoretical insights into these algorithms, while introducing tools with which similar dimensionality reduction methods can be studied.


title: Enhancing Music Genre Classification through Multi-Algorithm Analysis and User-Friendly Visualization

abstract: The aim of this study is to teach an algorithm how to recognize different types of music. Users will submit songs for analysis. Since the algorithm hasn't heard these songs before, it needs to figure out what makes each song unique. It does this by breaking down the songs into different parts and studying things like rhythm, melody, and tone via supervised learning because the program learns from examples that are already labelled. One important thing to consider when classifying music is its genre, which can be quite complex. To ensure accuracy, we use five different algorithms, each working independently, to analyze the songs. This helps us get a more complete understanding of each song's characteristics. Therefore, our goal is to correctly identify the genre of each submitted song. Once the analysis is done, the results are presented using a graphing tool, making it easy for users to understand and provide feedback.


title: Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control

abstract: Research on video generation has recently made tremendous progress, enabling high-quality videos to be generated from text prompts or images. Adding control to the video generation process is an important goal moving forward and recent approaches that condition video generation models on camera trajectories make strides towards it. Yet, it remains challenging to generate a video of the same scene from multiple different camera trajectories. Solutions to this multi-video generation problem could enable large-scale 3D scene generation with editable camera trajectories, among other applications. We introduce collaborative video diffusion (CVD) as an important step towards this vision. The CVD framework includes a novel cross-video synchronization module that promotes consistency between corresponding frames of the same video rendered from different camera poses using an epipolar attention mechanism. Trained on top of a state-of-the-art camera-control module for video generation, CVD generates multiple videos rendered from different camera trajectories with significantly better consistency than baselines, as shown in extensive experiments. Project page: https://collaborativevideodiffusion.github.io/.


title: Splitting aspects of holomorphic distributions with locally free tangent sheaf

abstract: In this work, we mainly deal with a two-dimensional singular holomorphic distribution $\mathcal{D}$ defined on $M$, in the two situations $M=\mathbb{P}^n$ or $M=(\mathbb{C}^n,0)$, tangent to a one-dimensional foliation $\mathcal{G}$ on $M$, and whose tangent sheaf $T_{\mathcal{D}}$ is locally free. We provide sufficient conditions on $\mathcal{G}$ so that there is another one-dimensional foliation $\mathcal{H}$ on $M$ tangent to $\mathcal{D}$, such that their respective tangent sheaves satisfy the splitting relation $T_{\mathcal{D}}=T_{\mathcal{G}} \oplus T_{\mathcal{H}}$. As an application, we show that if $\mathcal{F}$ is a codimension one holomorphic foliation on $\mathbb{P}^3$ with locally free tangent sheaf and tangent to a nontrivial holomorphic vector field on $\mathbb{P}^3$, then $T_{\mathcal{F}}$ splits. Some division results for vector fields and differential forms are also obtained.


title: A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learning

abstract: $Q$-learning algorithms are appealing for real-world applications due to their data-efficiency, but they are very prone to overfitting and training instabilities when trained from visual observations. Prior work, namely SVEA, finds that selective application of data augmentation can improve the visual generalization of RL agents without destabilizing training. We revisit its recipe for data augmentation, and find an assumption that limits its effectiveness to augmentations of a photometric nature. Addressing these limitations, we propose a generalized recipe, SADA, that works with wider varieties of augmentations. We benchmark its effectiveness on DMC-GB2 -- our proposed extension of the popular DMControl Generalization Benchmark -- as well as tasks from Meta-World and the Distracting Control Suite, and find that our method, SADA, greatly improves training stability and generalization of RL agents across a diverse set of augmentations. Visualizations, code, and benchmark: see https://aalmuzairee.github.io/SADA/


title: Critical one-arm probability for the metric Gaussian free field in low dimensions

abstract: We investigate the bond percolation model on transient weighted graphs ${G}$ induced by the excursion sets of the Gaussian free field on the corresponding metric graph. Under the sole assumption that its sign clusters do not percolate, we derive an extension of Lupu's formula for the two-point function at criticality. We then focus on the low-dimensional case $0< \nu < \frac{\alpha}{2}$, where $\alpha$ governs the polynomial volume growth of $G$ and $\nu$ the decay rate of the Green's function on $G$. In particular, this includes the benchmark case ${G}=\mathbb{Z}^3$, for which $\alpha=3$ and $\nu= \alpha-2=1$. We prove under these assumptions that the critical one-arm probability decays with distance $R$ like $R^{-\frac{\nu}{2}}$, up to multiplicative constants.


title: Self-Corrected Multimodal Large Language Model for End-to-End Robot Manipulation

abstract: Robot manipulation policies have shown unsatisfactory action performance when confronted with novel task or object instances. Hence, the capability to automatically detect and self-correct failure action is essential for a practical robotic system. Recently, Multimodal Large Language Models (MLLMs) have shown promise in visual instruction following and demonstrated strong reasoning abilities in various tasks. To unleash general MLLMs as an end-to-end robotic agent, we introduce a Self-Corrected (SC)-MLLM, equipping our model not only to predict end-effector poses but also to autonomously recognize and correct failure actions. Specifically, we first conduct parameter-efficient fine-tuning to empower MLLM with pose prediction ability, which is reframed as a language modeling problem. When facing execution failures, our model learns to identify low-level action error causes (i.e., position and rotation errors) and adaptively seeks prompt feedback from experts. Based on the feedback, SC-MLLM rethinks the current failure scene and generates the corrected actions. Furthermore, we design a continuous policy learning method for successfully corrected samples, enhancing the model's adaptability to the current scene configuration and reducing the frequency of expert intervention. To evaluate our SC-MLLM, we conduct extensive experiments in both simulation and real-world settings. SC-MLLM agent significantly improve manipulation accuracy compared to previous state-of-the-art robotic MLLM (ManipLLM), increasing from 57\% to 79\% on seen object categories and from 47\% to 69\% on unseen novel categories.


title: MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

abstract: Detecting out-of-distribution (OOD) samples is important for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused on unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential to leverage information from multiple modalities to enhance the efficacy of OOD detection. To establish a foundation for more realistic Multimodal OOD Detection, we introduce the first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms on MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD detection. Based on the observation of Modality Prediction Discrepancy between in-distribution (ID) and OOD data, and its strong correlation with OOD performance, we propose the Agree-to-Disagree (A2D) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix, which explores broader feature spaces by leveraging the information from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by a large margin. Our source code and MultiOOD benchmark are available at https://github.com/donghao51/MultiOOD.


title: Survival of the Fittest Representation: A Case Study with Modular Addition

abstract: When a neural network can learn multiple distinct algorithms to solve a task, how does it "choose" between them during training? To approach this question, we take inspiration from ecology: when multiple species coexist, they eventually reach an equilibrium where some survive while others die out. Analogously, we suggest that a neural network at initialization contains many solutions (representations and algorithms), which compete with each other under pressure from resource constraints, with the "fittest" ultimately prevailing. To investigate this Survival of the Fittest hypothesis, we conduct a case study on neural networks performing modular addition, and find that these networks' multiple circular representations at different Fourier frequencies undergo such competitive dynamics, with only a few circles surviving at the end. We find that the frequencies with high initial signals and gradients, the "fittest," are more likely to survive. By increasing the embedding dimension, we also observe more surviving frequencies. Inspired by the Lotka-Volterra equations describing the dynamics between species, we find that the dynamics of the circles can be nicely characterized by a set of linear differential equations. Our results with modular addition show that it is possible to decompose complicated representations into simpler components, along with their basic interactions, to offer insight on the training dynamics of representations.


title: MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds

abstract: We introduce 4D Motion Scaffolds (MoSca), a neural information processing system designed to reconstruct and synthesize novel views of dynamic scenes from monocular videos captured casually in the wild. To address such a challenging and ill-posed inverse problem, we leverage prior knowledge from foundational vision models, lift the video data to a novel Motion Scaffold (MoSca) representation, which compactly and smoothly encodes the underlying motions / deformations. The scene geometry and appearance are then disentangled from the deformation field, and are encoded by globally fusing the Gaussians anchored onto the MoSca and optimized via Gaussian Splatting. Additionally, camera poses can be seamlessly initialized and refined during the dynamic rendering process, without the need for other pose estimation tools. Experiments demonstrate state-of-the-art performance on dynamic rendering benchmarks.


title: Hardness-Aware Scene Synthesis for Semi-Supervised 3D Object Detection

abstract: 3D object detection aims to recover the 3D information of concerning objects and serves as the fundamental task of autonomous driving perception. Its performance greatly depends on the scale of labeled training data, yet it is costly to obtain high-quality annotations for point cloud data. While conventional methods focus on generating pseudo-labels for unlabeled samples as supplements for training, the structural nature of 3D point cloud data facilitates the composition of objects and backgrounds to synthesize realistic scenes. Motivated by this, we propose a hardness-aware scene synthesis (HASS) method to generate adaptive synthetic scenes to improve the generalization of the detection models. We obtain pseudo-labels for unlabeled objects and generate diverse scenes with different compositions of objects and backgrounds. As the scene synthesis is sensitive to the quality of pseudo-labels, we further propose a hardness-aware strategy to reduce the effect of low-quality pseudo-labels and maintain a dynamic pseudo-database to ensure the diversity and quality of synthetic scenes. Extensive experimental results on the widely used KITTI and Waymo datasets demonstrate the superiority of the proposed HASS method, which outperforms existing semi-supervised learning methods on 3D object detection. Code: https://github.com/wzzheng/HASS.


title: Privacy-Aware Visual Language Models

abstract: This paper aims to advance our understanding of how Visual Language Models (VLMs) handle privacy-sensitive information, a crucial concern as these technologies become integral to everyday life. To this end, we introduce a new benchmark PrivBench, which contains images from 8 sensitive categories such as passports, or fingerprints. We evaluate 10 state-of-the-art VLMs on this benchmark and observe a generally limited understanding of privacy, highlighting a significant area for model improvement. Based on this we introduce PrivTune, a new instruction-tuning dataset aimed at equipping VLMs with knowledge about visual privacy. By tuning two pretrained VLMs, TinyLLaVa and MiniGPT-v2, on this small dataset, we achieve strong gains in their ability to recognize sensitive content, outperforming even GPT4-V. At the same time, we show that privacy-tuning only minimally affects the VLMs performance on standard benchmarks such as VQA. Overall, this paper lays out a crucial challenge for making VLMs effective in handling real-world data safely and provides a simple recipe that takes the first step towards building privacy-aware VLMs.


title: LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence

abstract: Due to the need to interact with the real world, embodied agents are required to possess comprehensive prior knowledge, long-horizon planning capability, and a swift response speed. Despite recent large language model (LLM) based agents achieving promising performance, they still exhibit several limitations. For instance, the output of LLMs is a descriptive sentence, which is ambiguous when determining specific actions. To address these limitations, we introduce the large auto-regressive model (LARM). LARM leverages both text and multi-view images as input and predicts subsequent actions in an auto-regressive manner. To train LARM, we develop a novel data format named auto-regressive node transmission structure and assemble a corresponding dataset. Adopting a two-phase training regimen, LARM successfully harvests enchanted equipment in Minecraft, which demands significantly more complex decision-making chains than the highest achievements of prior best methods. Besides, the speed of LARM is 6.8x faster.


title: From Neurons to Neutrons: A Case Study in Interpretability

abstract: Mechanistic Interpretability (MI) promises a path toward fully understanding how neural networks make their predictions. Prior work demonstrates that even when trained to perform simple arithmetic, models can implement a variety of algorithms (sometimes concurrently) depending on initialization and hyperparameters. Does this mean neuron-level interpretability techniques have limited applicability? We argue that high-dimensional neural networks can learn low-dimensional representations of their training data that are useful beyond simply making good predictions. Such representations can be understood through the mechanistic interpretability lens and provide insights that are surprisingly faithful to human-derived domain knowledge. This indicates that such approaches to interpretability can be useful for deriving a new understanding of a problem from models trained to solve it. As a case study, we extract nuclear physics concepts by studying models trained to reproduce nuclear data.


title: Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving

abstract: Recent advancements in bird's eye view (BEV) representations have shown remarkable promise for in-vehicle 3D perception. However, while these methods have achieved impressive results on standard benchmarks, their robustness in varied conditions remains insufficiently assessed. In this study, we present RoboBEV, an extensive benchmark suite designed to evaluate the resilience of BEV algorithms. This suite incorporates a diverse set of camera corruption types, each examined over three severity levels. Our benchmarks also consider the impact of complete sensor failures that occur when using multi-modal models. Through RoboBEV, we assess 33 state-of-the-art BEV-based perception models spanning tasks like detection, map segmentation, depth estimation, and occupancy prediction. Our analyses reveal a noticeable correlation between the model's performance on in-distribution datasets and its resilience to out-of-distribution challenges. Our experimental results also underline the efficacy of strategies like pre-training and depth-free BEV transformations in enhancing robustness against out-of-distribution data. Furthermore, we observe that leveraging extensive temporal information significantly improves the model's robustness. Based on our observations, we design an effective robustness enhancement strategy based on the CLIP model. The insights from this study pave the way for the development of future BEV models that seamlessly combine accuracy with real-world robustness.


title: Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model

abstract: Recent advancements in multimodal large language models (LLMs) have shown their potential in various domains, especially concept reasoning. Despite these developments, applications in understanding 3D environments remain limited. This paper introduces Reason3D, a novel LLM designed for comprehensive 3D understanding. Reason3D takes point cloud data and text prompts as input to produce textual responses and segmentation masks, facilitating advanced tasks like 3D reasoning segmentation, hierarchical searching, express referring, and question answering with detailed mask outputs. Specifically, we propose a hierarchical mask decoder to locate small objects within expansive scenes. This decoder initially generates a coarse location estimate covering the object's general area. This foundational estimation facilitates a detailed, coarse-to-fine segmentation strategy that significantly enhances the precision of object identification and segmentation. Experiments validate that Reason3D achieves remarkable results on large-scale ScanNet and Matterport3D datasets for 3D express referring, 3D question answering, and 3D reasoning segmentation tasks. Code and models are available at: https://github.com/KuanchihHuang/Reason3D.


title: NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models

abstract: Decoder-only large language model (LLM)-based embedding models are beginning to outperform BERT or T5-based embedding models in general-purpose text embedding tasks, including dense vector-based retrieval. In this work, we introduce the NV-Embed model with a variety of architectural designs and training procedures to significantly enhance the performance of LLM as a versatile embedding model, while maintaining its simplicity and reproducibility. For model architecture, we propose a latent attention layer to obtain pooled embeddings, which consistently improves retrieval and downstream task accuracy compared to mean pooling or using the last <EOS> token embedding from LLMs. To enhance representation learning, we remove the causal attention mask of LLMs during contrastive training. For model training, we introduce a two-stage contrastive instruction-tuning method. It first applies contrastive training with instructions on retrieval datasets, utilizing in-batch negatives and curated hard negative examples. At stage-2, it blends various non-retrieval datasets into instruction tuning, which not only enhances non-retrieval task accuracy but also improves retrieval performance. Combining these techniques, our NV-Embed model, using only publicly available data, has achieved a record-high score of 69.32, ranking No. 1 on the Massive Text Embedding Benchmark (MTEB) (as of May 24, 2024), with 56 tasks, encompassing retrieval, reranking, classification, clustering, and semantic textual similarity tasks. Notably, our model also attains the highest score of 59.36 on 15 retrieval tasks in the MTEB benchmark (also known as BEIR). We will open-source the model at: https://huggingface.co/nvidia/NV-Embed-v1.