title: Quantum nondemolition measurement operator with spontaneous emission

abstract: We present a theory for quantum nondemolition (QND) measurements of an atomic ensemble in the presence of spontaneous emission. We derive the master equation that governs the evolution of the ground state of the atoms and the quantum state of light. Solving the master equation exactly without invoking the Holstein-Primakoff approximation and projecting out the quantum state of light, we derive a positive operator-valued measure that describes the QND measurement. We show that at high spontaneous emission conditions, the QND measurement has a unique dominant state to which the measurement collapses. We additionally investigate the behavior of the QND measurement in the limiting case of strong atom-light interactions, where we show that the positive operator valued measure becomes a projection operator. We further analyze the effect of spontaneous emission noise on atomic state preparation. We find that it limits the width of the eigenvalue spectrum available to a quantum state in a linear superposition. This effect leads to state collapse on the dominant state. We generate various non-classical states of the atom by tuning the atom-light interaction strength. We find that non-classical states such as the Schr\"odinger-cat state, whose coherence spans the entire eigenvalue spectrum of the total spin operator $J_z$ for a given spin eigenvalue $J$, lose their coherence because spontaneous emission limits the accessibility of states farther away from the dominant state.


title: Sums: Sniffing Unknown Multiband Signals under Low Sampling Rates

abstract: Due to sophisticated deployments of all kinds of wireless networks (e.g., 5G, Wi-Fi, Bluetooth, LEO satellite, etc.), multiband signals distribute in a large bandwidth (e.g., from 70 MHz to 8 GHz). Consequently, for network monitoring and spectrum sharing applications, a sniffer for extracting physical layer information, such as structure of packet, with low sampling rate (especially, sub-Nyquist sampling) can significantly improve their cost- and energy-efficiency. However, to achieve a multiband signals sniffer is really a challenge. To this end, we propose Sums, a system that can sniff and analyze multiband signals in a blind manner. Our Sums takes advantage of hardware and algorithm co-design, multi-coset sub-Nyquist sampling hardware, and a multi-task deep learning framework. The hardware component breaks the Nyquist rule to sample GHz bandwidth, but only pays for a 50 MSPS sampling rate. Our multi-task learning framework directly tackles the sampling data to perform spectrum sensing, physical layer protocol recognition, and demodulation for deep inspection from multiband signals. Extensive experiments demonstrate that Sums achieves higher accuracy than the state-of-theart baselines in spectrum sensing, modulation classification, and demodulation. As a result, our Sums can help researchers and end-users to diagnose or troubleshoot their problems of wireless infrastructures deployments in practice.


title: The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

abstract: Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical success is incomplete and remains an active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of a model's learned representations as a fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the few-shot setting.


title: Digitized Counterdiabatic Quantum Algorithms for Logistics Scheduling

abstract: We study a job shop scheduling problem for an automatized robot in a high-throughput laboratory and a travelling salesperson problem with recently proposed digitized counterdiabatic quantum optimization (DCQO) algorithms. In DCQO, we find the solution of an optimization problem via an adiabatic quantum dynamics, which is accelerated with counterdiabatic protocols. Thereafter, we digitize the global unitary to encode it in a digital quantum computer. For the job-shop scheduling problem, we aim at finding the optimal schedule for a robot executing a number of tasks under specific constraints, such that the total execution time of the process is minimized. For the traveling salesperson problem, the goal is to find the path that covers all cities and is associated with the shortest traveling distance. We consider both hybrid and pure versions of DCQO algorithms and benchmark the performance against digitized quantum annealing and the quantum approximate optimization algorithm (QAOA). In comparison to QAOA, the DCQO solution is improved by several orders of magnitude in success probability using the same number of two-qubit gates. Moreover, we experimentally implement our algorithms on superconducting and trapped-ion quantum processors. Our results demonstrate that circuit compression using counterdiabatic protocols is amenable to current NISQ hardware and can solve logistics scheduling problems, where other digital quantum algorithms show insufficient performance.


title: EmpathicStories++: A Multimodal Dataset for Empathy towards Personal Experiences

abstract: Modeling empathy is a complex endeavor that is rooted in interpersonal and experiential dimensions of human interaction, and remains an open problem within AI. Existing empathy datasets fall short in capturing the richness of empathy responses, often being confined to in-lab or acted scenarios, lacking longitudinal data, and missing self-reported labels. We introduce a new multimodal dataset for empathy during personal experience sharing: the EmpathicStories++ dataset (https://mitmedialab.github.io/empathic-stories-multimodal/) containing 53 hours of video, audio, and text data of 41 participants sharing vulnerable experiences and reading empathically resonant stories with an AI agent. EmpathicStories++ is the first longitudinal dataset on empathy, collected over a month-long deployment of social robots in participants' homes, as participants engage in natural, empathic storytelling interactions with AI agents. We then introduce a novel task of predicting individuals' empathy toward others' stories based on their personal experiences, evaluated in two contexts: participants' own personal shared story context and their reflections on stories they read. We benchmark this task using state-of-the-art models to pave the way for future improvements in contextualized and longitudinal empathy modeling. Our work provides a valuable resource for further research in developing empathetic AI systems and understanding the intricacies of human empathy within genuine, real-world settings.


title: Information-theoretic Generalization Analysis for Expected Calibration Error

abstract: While the expected calibration error (ECE), which employs binning, is widely adopted to evaluate the calibration performance of machine learning models, theoretical understanding of its estimation bias is limited. In this paper, we present the first comprehensive analysis of the estimation bias in the two common binning strategies, uniform mass and uniform width binning. Our analysis establishes upper bounds on the bias, achieving an improved convergence rate. Moreover, our bounds reveal, for the first time, the optimal number of bins to minimize the estimation bias. We further extend our bias analysis to generalization error analysis based on the information-theoretic approach, deriving upper bounds that enable the numerical evaluation of how small the ECE is for unknown data. Experiments using deep learning models show that our bounds are nonvacuous thanks to this information-theoretic generalization analysis approach.


title: On the theory of earthquakes: Paradoxical contradiction of Omori's law to the law of energy conservation

abstract: After the main shock of an earthquake the aftershocks are observed. According to Omori's law, the frequency of aftershocks decreases hyperbolically over time. We noticed that, strictly speaking, Omori's law paradoxically contradicts the law of energy conservation. The contradiction is that the excitation of each aftershock consumes a finite portion of the source's energy, so that the total energy released by the source tends to infinity over time. The paradox is formally theoretical, but its analysis has proved useful. Eliminating the contradiction between Omori's law and the fundamental law of conservation of energy allowed us to further understand the nature of the phenomenological theory of aftershocks. We used the concept of deactivation of a source after the formation of a main rupture in it. We have based the theory on the original aftershock evolution equation, which has the form of a first-order linear differential equation. Two ways to eliminate the paradoxical situation are indicated. Key words: earthquake source, aftershock, evolution equation, deactivation coefficient, inverse problem, Omori epoch, source bifurcation, logistic equation, Hirano-Utsu formula.


title: An Adaptive Framework for Manipulator Skill Reproduction in Dynamic Environments

abstract: Robot skill learning and execution in uncertain and dynamic environments is a challenging task. This paper proposes an adaptive framework that combines Learning from Demonstration (LfD), environment state prediction, and high-level decision making. Proactive adaptation prevents the need for reactive adaptation, which lags behind changes in the environment rather than anticipating them. We propose a novel LfD representation, Elastic-Laplacian Trajectory Editing (ELTE), which continuously adapts the trajectory shape to predictions of future states. Then, a high-level reactive system using an Unscented Kalman Filter (UKF) and Hidden Markov Model (HMM) prevents unsafe execution in the current state of the dynamic environment based on a discrete set of decisions. We first validate our LfD representation in simulation, then experimentally assess the entire framework using a legged mobile manipulator in 36 real-world scenarios. We show the effectiveness of the proposed framework under different dynamic changes in the environment. Our results show that the proposed framework produces robust and stable adaptive behaviors.


title: Infinite Limits of Multi-head Transformer Dynamics

abstract: In this work, we analyze various scaling limits of the training dynamics of transformer models in the feature learning regime. We identify the set of parameterizations that admit well-defined infinite width and depth limits, allowing the attention layers to update throughout training--a relevant notion of feature learning in these models. We then use tools from dynamical mean field theory (DMFT) to analyze various infinite limits (infinite key/query dimension, infinite heads, and infinite depth) which have different statistical descriptions depending on which infinite limit is taken and how attention layers are scaled. We provide numerical evidence of convergence to the limits and discuss how the parameterization qualitatively influences learned features.


title: Understanding the Star Formation Efficiency in Dense Gas: Initial Results from the CAFFEINE Survey with ArT\'eMiS

abstract: Despite recent progress, the question of what regulates the star formation efficiency in galaxies remains one of the most debated problems in astrophysics. According to the dominant picture, star formation (SF) is regulated by turbulence and feedback, and the SFE is 1-2% per local free-fall time. In an alternate scenario, the SF rate in galactic disks is linearly proportional to the mass of dense gas above a critical density threshold. We aim to discriminate between these two pictures thanks to high-resolution observations tracing dense gas and young stellar objects (YSOs) for a comprehensive sample of 49 nearby massive SF complexes out to d < 3 kpc in the Galactic disk. We use data from CAFFEINE, a 350/450 $\mu$m survey with APEX/ArT\'eMiS of the densest portions of all southern molecular clouds, in combination with Herschel data to produce column density maps at 8" resolution. Our maps are free of saturation and resolve the structure of dense gas and the typical 0.1 pc width of molecular filaments at 3 kpc, which is impossible with Herschel data alone. Coupled with SFR estimates derived from Spitzer observations of the YSO content of the same clouds, this allows us to study the dependence of the SFE with density in the CAFFEINE clouds. We also combine our findings with existing SFE measurements in nearby clouds to extend our analysis down to lower column densities. Our results suggest that the SFE does not increase with density above the critical threshold and support a scenario in which the SFE in dense gas is approximately constant. However, the SFE measurements traced by Class I YSOs in nearby clouds are more inconclusive, since they are consistent with both the presence of a density threshold and a dependence on density above the threshold. Overall, we suggest that the SFE in dense gas is primarily governed by the physics of filament fragmentation into protostellar cores.


title: Mean Field Limit for Congestion Dynamics in One Dimension

abstract: This paper addresses congested transport, which can be described, at macroscopic scales, by a continuity equation with a pressure variable generated from the hard-congestion constraint (maximum value of the density). The main goal of the paper is to show that, in one spatial dimension, this continuum PDE can be derived as the mean-field limit of a system of ordinary differential equations that describes the motion of a large number of particles constrained to stay at some finite distance from each others. To show that these two models describe the same dynamics at different scale, we will rely on both the Eulerian and Lagrangian points of view and use two different approximations for the density and pressure variables in the continuum limit.


title: Hyperbolic Shear Metasurfaces

abstract: Polar dielectrics with low crystal symmetry and sharp phonon resonances can support hyperbolic shear polaritons - highly confined surface modes with frequency-dependent optical axes and asymmetric dissipation features. So far, these modes have been observed only in bulk natural materials at mid-infrared frequencies, with properties limited by available crystal geometries and phonon resonance strength. Here we introduce hyperbolic shear metasurfaces: ultrathin engineered surfaces supporting hyperbolic surface modes with symmetry-tailored axial dispersion and loss redistribution that can maximally enhance light-matter interactions. By engineering effective shear phenomena in these engineered surfaces, we demonstrate geometry-controlled, ultra-confined, low-loss hyperbolic surface waves with broadband Purcell enhancements, applicable across a broad range of the electromagnetic spectrum.


title: Empirical Crypto Asset Pricing

abstract: We motivate the study of the crypto asset class with eleven empirical facts, and study the drivers of crypto asset returns through the lens of univariate factors. We argue crypto assets are a new, attractive, and independent asset class. In a novel and rigorously built panel of crypto assets, we examine pricing ability of sixty three asset characteristics to find rich signal content across the characteristics and at several future horizons. Only univariate financial factors (i.e., functions of previous returns) were associated with statistically significant long-short strategies, suggestive of speculatively driven returns as opposed to more fundamental pricing factors.


title: Integrated Design for Wave Energy Converter Farms: Assessing Plant, Control, Layout, and Site Selection Coupling in the Presence of Irregular Waves

abstract: A promising direction towards reducing the levelized cost of energy for wave energy converter (WEC) farms is to improve their performance. WEC design studies generally focus on a single design domain (e.g., geometry, control, or layout) to improve the farm's performance under simplifying assumptions, such as regular waves. This strategy, however, has resulted in design recommendations that are impractical or limited in scope because WEC farms are complex systems that exhibit strong coupling among geometry, control, and layout domains. In addition, the location of the candidate site, which has a large impact on the performance of the farm, is often overlooked. Motivated by some of the limitations observed in WEC literature, this study uses an integrated design framework, based on simultaneous control co-design (CCD) principles, to discuss the impact of site selection and wave type on WEC farm design. Interactions among plant, control, and layout are also investigated and discussed using a wide range of simulations and optimization studies. All of the studies were conducted using frequency-domain heaving cylinder WEC devices within a farm with a linear reactive controller in the presence of irregular probabilistic waves. The results provide high-level guidelines to help the WEC design community move toward an integrated design perspective.


title: The electromagnetic symmetry sphere: a framework for energy, momentum, spin and other electromagnetic quantities

abstract: Electromagnetic quantities such as energy density, momentum, spin, and helicity bring meaning and intuition to electromagnetism and possess intricate interrelations, particularly prominent in complex non-paraxial near-fields. These quantities are conventionally expressed using electric and magnetic field vectors, yet the electric-magnetic basis is one among other often overlooked alternatives, including parallel-antiparallel and right-left-handed helicity bases, related to the parity and duality symmetries of electromagnetism. Projecting time-harmonic electromagnetic fields into a variety of bases allows re-interpreting established quantities and reveals underlying mathematical structures: a Bloch sphere which describes asymmetries in electromagnetic energy, a systematic path to unify and uncover relations between electromagnetic quantities, and the unlocking of symmetry-driven equations in light-matter interaction.


title: Hierarchical Uncertainty Exploration via Feedforward Posterior Trees

abstract: When solving ill-posed inverse problems, one often desires to explore the space of potential solutions rather than be presented with a single plausible reconstruction. Valuable insights into these feasible solutions and their associated probabilities are embedded in the posterior distribution. However, when confronted with data of high dimensionality (such as images), visualizing this distribution becomes a formidable challenge, necessitating the application of effective summarization techniques before user examination. In this work, we introduce a new approach for visualizing posteriors across multiple levels of granularity using tree-valued predictions. Our method predicts a tree-valued hierarchical summarization of the posterior distribution for any input measurement, in a single forward pass of a neural network. We showcase the efficacy of our approach across diverse datasets and image restoration challenges, highlighting its prowess in uncertainty quantification and visualization. Our findings reveal that our method performs comparably to a baseline that hierarchically clusters samples from a diffusion-based posterior sampler, yet achieves this with orders of magnitude greater speed.


title: Study of triaxial loading of segregated granular assemblies through experiments and DEM simulations

abstract: A simple position-dependent body force-based confinement for simulating triaxial tests using the Discrete Element Method is presented. The said method is used to perform triaxial simulations on mono-disperse and segregated assemblies of glass spheres. The macroscopic load response obtained in simulations is validated with experimental load response. A mesh construction algorithm is presented to check whether the confinement applied in the triaxial simulations is accurate. The particle displacement data obtained from triaxial simulations are used to obtain a particle-wise average strain tensor. This is further used to compare the strain localisation between the mono-disperse and segregated assemblies. It is observed that, in the segregated assembly, the interface between the two particle phases acts as a barrier for strain localisation, and the smaller particles preferentially undergo a higher degree of shear strain on average.


title: Dynamic Latent-Factor Model with High-Dimensional Asset Characteristics

abstract: We develop novel estimation procedures with supporting econometric theory for a dynamic latent-factor model with high-dimensional asset characteristics, that is, the number of characteristics is on the order of the sample size. Utilizing the Double Selection Lasso estimator, our procedure employs regularization to eliminate characteristics with low signal-to-noise ratios yet maintains asymptotically valid inference for asset pricing tests. The crypto asset class is well-suited for applying this model given the limited number of tradable assets and years of data as well as the rich set of available asset characteristics. The empirical results present out-of-sample pricing abilities and risk-adjusted returns for our novel estimator as compared to benchmark methods. We provide an inference procedure for measuring the risk premium of an observable nontradable factor, and employ this to find that the inflation-mimicking portfolio in the crypto asset class has positive risk compensation.


title: Models That Prove Their Own Correctness

abstract: How can we trust the correctness of a learned model on a particular input of interest? Model accuracy is typically measured *on average* over a distribution of inputs, giving no guarantee for any fixed input. This paper proposes a theoretically-founded solution to this problem: to train *Self-Proving models* that prove the correctness of their output to a verification algorithm $V$ via an Interactive Proof. Self-Proving models satisfy that, with high probability over a random input, the model generates a correct output *and* successfully proves its correctness to $V\!$. The *soundness* property of $V$ guarantees that, for *every* input, no model can convince $V$ of the correctness of an incorrect output. Thus, a Self-Proving model proves correctness of most of its outputs, while *all* incorrect outputs (of any model) are detected by $V$. We devise a generic method for learning Self-Proving models, and we prove convergence bounds under certain assumptions. The theoretical framework and results are complemented by experiments on an arithmetic capability: computing the greatest common divisor (GCD) of two integers. Our learning method is used to train a Self-Proving transformer that computes the GCD *and* proves the correctness of its answer.


title: Bisimulation Learning

abstract: We introduce a data-driven approach to computing finite bisimulations for state transition systems with very large, possibly infinite state space. Our novel technique computes stutter-insensitive bisimulations of deterministic systems, which we characterize as the problem of learning a state classifier together with a ranking function for each class. Our procedure learns a candidate state classifier and candidate ranking functions from a finite dataset of sample states; then, it checks whether these generalise to the entire state space using satisfiability modulo theory solving. Upon the affirmative answer, the procedure concludes that the classifier constitutes a valid stutter-insensitive bisimulation of the system. Upon a negative answer, the solver produces a counterexample state for which the classifier violates the claim, adds it to the dataset, and repeats learning and checking in a counterexample-guided inductive synthesis loop until a valid bisimulation is found. We demonstrate on a range of benchmarks from reactive verification and software model checking that our method yields faster verification results than alternative state-of-the-art tools in practice. Our method produces succinct abstractions that enable an effective verification of linear temporal logic without next operator, and are interpretable for system diagnostics.


title: Reconfiguration Algorithms for Cubic Modular Robots with Realistic Movement Constraints

abstract: We introduce and analyze a model for self-reconfigurable robots made up of unit-cube modules. Compared to past models, our model aims to newly capture two important practical aspects of real-world robots. First, modules often do not occupy an exact unit cube, but rather have features like bumps extending outside the allotted space so that modules can interlock. Thus, for example, our model forbids modules from squeezing in between two other modules that are one unit distance apart. Second, our model captures the practical scenario of many passive modules assembled by a single robot, instead of requiring all modules to be able to move on their own. We prove two universality results. First, with a supply of auxiliary modules, we show that any connected polycube structure can be constructed by a carefully aligned plane sweep. Second, without additional modules, we show how to construct any structure for which a natural notion of external feature size is at least a constant; this property largely consolidates forbidden-pattern properties used in previous works on reconfigurable modular robots.


title: Murray-von Neumann dimension for strictly semifinite weights

abstract: Given a von Neumann algebra $M$ equipped with a faithful normal strictly semifinite weight $\varphi$, we develop a notion of Murray-von Neumann dimension over $(M,\varphi)$ that is defined for modules over the basic construction associated to the inclusion $M^\varphi \subset M$. For $\varphi=\tau$ a faithful normal tracial state, this recovers the usual Murray-von Neumann dimension for finite von Neumann algebras. If $M$ is either a type $\mathrm{III}_\lambda$ factor with $0<\lambda <1$ or a full type $\mathrm{III}_1$ factor with $\text{Sd}(M)\neq \mathbb{R}$, then amongst extremal almost periodic weights the dimension function depends on $\varphi$ only up to scaling. As an application, we show that if an inclusion of diffuse factors with separable preduals $N\subset M$ is with expectation $\mathcal{E}$ and admits a compatible extremal almost periodic state $\varphi$, then this dimension quantity bounds the index $\text{Ind}{\mathcal{E}}$, and in fact equals it when the modular operators $\Delta_\varphi$ and $\Delta_{\varphi|_N}$ have the same point spectrum. In the pursuit of this result, we also show such inclusions always admit Pimsner-Popa orthogonal bases.


title: Interaction in the dark sector: a phenomenological approach

abstract: The non-gravitational interaction between the dark components of the Universe could lead to the variation of dark matter energy density standard evolution law. When we assume this scenario, the dark matter energy density follows $\rho_{{dm}}\sim(1+z)^{3 + \epsilon(z)}$ (where $\epsilon(z)=0$ the standard law is recovered). In this paper, we perform a Bayesian analysis to test three parameterizations for $\epsilon(z)$, namely: $\epsilon(z)=\epsilon_0$, $\epsilon(z)=\epsilon_0 + \epsilon_1\frac{z}{1+z}$ and $\epsilon(z)=\epsilon_0 + \epsilon_1\frac{z(1+z)}{1+z^2}$, where the first one is motivated through the fundamental grounds and the others are on the phenomenological ones. Through the Gaussian process regression, our method uses galaxy cluster gas mass fraction measurements, SNe Ia observations, Cosmic Chronometers, and BAO data. No specific cosmological model is considered. In all possibilities analyzed, the standard evolution law ($\epsilon(z)=0$) is within $2\sigma$ c.l. The investigated cases generally indicated scenarios of inconclusive or weak evidence toward the simplest model from the Bayesian standpoint.


title: Anomalous Change Point Detection Using Probabilistic Predictive Coding

abstract: Change point detection (CPD) and anomaly detection (AD) are essential techniques in various fields to identify abrupt changes or abnormal data instances. However, existing methods are often constrained to univariate data, face scalability challenges with large datasets due to computational demands, and experience reduced performance with high-dimensional or intricate data, as well as hidden anomalies. Furthermore, they often lack interpretability and adaptability to domain-specific knowledge, which limits their versatility across different fields. In this work, we propose a deep learning-based CPD/AD method called Probabilistic Predictive Coding (PPC) that jointly learns to encode sequential data to low dimensional latent space representations and to predict the subsequent data representations as well as the corresponding prediction uncertainties. The model parameters are optimized with maximum likelihood estimation by comparing these predictions with the true encodings. At the time of application, the true and predicted encodings are used to determine the probability of conformity, an interpretable and meaningful anomaly score. Furthermore, our approach has linear time complexity, scalability issues are prevented, and the method can easily be adjusted to a wide range of data types and intricate applications. We demonstrate the effectiveness and adaptability of our proposed method across synthetic time series experiments, image data, and real-world magnetic resonance spectroscopic imaging data.


title: Disease-informed Adaptation of Vision-Language Models

abstract: In medical image analysis, the expertise scarcity and the high cost of data annotation limits the development of large artificial intelligence models. This paper investigates the potential of transfer learning with pre-trained vision-language models (VLMs) in this domain. Currently, VLMs still struggle to transfer to the underrepresented diseases with minimal presence and new diseases entirely absent from the pretraining dataset. We argue that effective adaptation of VLMs hinges on the nuanced representation learning of disease concepts. By capitalizing on the joint visual-linguistic capabilities of VLMs, we introduce disease-informed contextual prompting in a novel disease prototype learning framework. This approach enables VLMs to grasp the concepts of new disease effectively and efficiently, even with limited data. Extensive experiments across multiple image modalities showcase notable enhancements in performance compared to existing techniques.


title: Optimizing Large Language Models for OpenAPI Code Completion

abstract: Recent advancements in Large Language Models (LLMs) and their utilization in code generation tasks have significantly reshaped the field of software development. Despite the remarkable efficacy of code completion solutions in mainstream programming languages, their performance lags when applied to less ubiquitous formats such as OpenAPI definitions. This study evaluates the OpenAPI completion performance of GitHub Copilot, a prevalent commercial code completion tool, and proposes a set of task-specific optimizations leveraging Meta's open-source model Code Llama. A semantics-aware OpenAPI completion benchmark proposed in this research is used to perform a series of experiments through which the impact of various prompt-engineering and fine-tuning techniques on the Code Llama model's performance is analyzed. The fine-tuned Code Llama model reaches a peak correctness improvement of 55.2% over GitHub Copilot despite utilizing 25 times fewer parameters than the commercial solution's underlying Codex model. Additionally, this research proposes an enhancement to a widely used code infilling training technique, addressing the issue of underperformance when the model is prompted with context sizes smaller than those used during training. The dataset, the benchmark, and the model fine-tuning code are made publicly available.


title: Multi-objective control for stochastic parabolic equations with dynamic boundary conditions

abstract: This paper deals with a hierarchical multi-objective control problem for forward stochastic parabolic equations with dynamic boundary conditions. The controls are divided into two classes: leaders and followers. The goal of the leaders is of null controllability type while the followers are in charge of letting the state close to prescribed targets in fixed observation regions. To solve the problem, Nash and Stackelberg strategies are used. To implement these strategies, we combine some appropriate Carleman estimates and the well-known control duality approach.


title: Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks

abstract: Softmax attention is the principle backbone of foundation models for various artificial intelligence applications, yet its quadratic complexity in sequence length can limit its inference throughput in long-context settings. To address this challenge, alternative architectures such as linear attention, State Space Models (SSMs), and Recurrent Neural Networks (RNNs) have been considered as more efficient alternatives. While connections between these approaches exist, such models are commonly developed in isolation and there is a lack of theoretical understanding of the shared principles underpinning these architectures and their subtle differences, greatly influencing performance and scalability. In this paper, we introduce the Dynamical Systems Framework (DSF), which allows a principled investigation of all these architectures in a common representation. Our framework facilitates rigorous comparisons, providing new insights on the distinctive characteristics of each model class. For instance, we compare linear attention and selective SSMs, detailing their differences and conditions under which both are equivalent. We also provide principled comparisons between softmax attention and other model classes, discussing the theoretical conditions under which softmax attention can be approximated. Additionally, we substantiate these new insights with empirical validations and mathematical arguments. This shows the DSF's potential to guide the systematic development of future more efficient and scalable foundation models.


title: Neural Persistence Dynamics

abstract: We consider the problem of learning the dynamics in the topology of time-evolving point clouds, the prevalent spatiotemporal model for systems exhibiting collective behavior, such as swarms of insects and birds or particles in physics. In such systems, patterns emerge from (local) interactions among self-propelled entities. While several well-understood governing equations for motion and interaction exist, they are difficult to fit to data due to the often large number of entities and missing correspondences between the observation times, which may also not be equidistant. To evade such confounding factors, we investigate collective behavior from a \textit{topological perspective}, but instead of summarizing entire observation sequences (as in prior work), we propose learning a latent dynamical model from topological features \textit{per time point}. The latter is then used to formulate a downstream regression task to predict the parametrization of some a priori specified governing equation. We implement this idea based on a latent ODE learned from vectorized (static) persistence diagrams and show that this modeling choice is justified by a combination of recent stability results for persistent homology. Various (ablation) experiments not only demonstrate the relevance of each individual model component, but provide compelling empirical evidence that our proposed model -- \textit{neural persistence dynamics} -- substantially outperforms the state-of-the-art across a diverse set of parameter regression tasks.


title: Embedding Nearly Spanning Trees

abstract: The Erd\H{o}s-S\'os Conjecture states that every graph with average degree exceeding $k-1$ contains every tree with $k$ edges as a subgraph. We prove that there are $\delta>0$ and $k_0\in\mathbb N$ such that the conjecture holds for every tree $T$ with $k \ge k_0$ edges and every graph $G$ with $|V(G)| \le (1+\delta)|V(T)|$.


title: LM4LV: A Frozen Large Language Model for Low-level Vision Tasks

abstract: The success of large language models (LLMs) has fostered a new research trend of multi-modality large language models (MLLMs), which changes the paradigm of various fields in computer vision. Though MLLMs have shown promising results in numerous high-level vision and vision-language tasks such as VQA and text-to-image, no works have demonstrated how low-level vision tasks can benefit from MLLMs. We find that most current MLLMs are blind to low-level features due to their design of vision modules, thus are inherently incapable for solving low-level vision tasks. In this work, we purpose $\textbf{LM4LV}$, a framework that enables a FROZEN LLM to solve a range of low-level vision tasks without any multi-modal data or prior. This showcases the LLM's strong potential in low-level vision and bridges the gap between MLLMs and low-level vision tasks. We hope this work can inspire new perspectives on LLMs and deeper understanding of their mechanisms. Code is available at https://github.com/bytetriper/LM4LV.


title: A Higher Order Local Mesh Method for Approximating Laplacians on Unknown Manifolds

abstract: In this work, we introduce a numerical method for approximating arbitrary differential operators on vector fields in the weak form given point cloud data sampled randomly from a $d$ dimensional manifold embedded in $\mathbb{R}^n$. This method generalizes the local linear mesh method to the local curved mesh method, thus, allowing for the estimation of differential operators with nontrivial Christoffer symbols, such as the Bochner or Hodge Laplacians. In particular, we leverage the potentially small intrinsic dimension of the manifold $(d \ll n)$ to construct local parameterizations that incorporate both local meshes and higher-order curvature information. The former is constructed using low dimensional meshes obtained from local data projected to the tangent spaces, while the latter is obtained by fitting local polynomials with the generalized moving least squares. Theoretically, we prove the weak and spectral convergence rates for the proposed method for the estimation of the Bochner Laplacian. We provide numerical results supporting the theoretical convergence rates for the Bochner and Hodge Laplacians on simple manifolds.


title: Single-Round Proofs of Quantumness from Knowledge Assumptions

abstract: A proof of quantumness is an efficiently verifiable interactive test that an efficient quantum computer can pass, but all efficient classical computers cannot (under some cryptographic assumption). Such protocols play a crucial role in the certification of quantum devices. Existing single-round protocols (like asking the quantum computer to factor a large number) require large quantum circuits, whereas multi-round ones use smaller circuits but require experimentally challenging mid-circuit measurements. As such, current proofs of quantumness are out of reach for near-term devices. In this work, we construct efficient single-round proofs of quantumness based on existing knowledge assumptions. While knowledge assumptions have not been previously considered in this context, we show that they provide a natural basis for separating classical and quantum computation. Specifically, we show that multi-round protocols based on Decisional Diffie-Hellman (DDH) or Learning With Errors (LWE) can be "compiled" into single-round protocols using a knowledge-of-exponent assumption or knowledge-of-lattice-point assumption, respectively. We also prove an adaptive hardcore-bit statement for a family of claw-free functions based on DDH, which might be of independent interest. Previous approaches to constructing single-round protocols relied on the random oracle model and thus incurred the overhead associated with instantiating the oracle with a cryptographic hash function. In contrast, our protocols have the same resource requirements as their multi-round counterparts without necessitating mid-circuit measurements, making them, arguably, the most efficient single-round proofs of quantumness to date. Our work also helps in understanding the interplay between black-box/white-box reductions and cryptographic assumptions in the design of proofs of quantumness.


title: More Insight from Being More Focused: Analysis of Clustered Market Apps

abstract: The increasing attraction of mobile apps has inspired researchers to analyze apps from different perspectives. As with any software product, apps have different attributes such as size, content maturity, rating, category, or number of downloads. Current research studies mostly consider sampling across all apps. This often results in comparisons of apps being quite different in nature and category (games compared with weather and calendar apps), also being different in size and complexity. Similar to proprietary software and web-based services, more specific results can be expected from looking at more homogeneous samples as they can be received as a result of applying clustering. In this paper, we target homogeneous samples of apps to increase the degree of insight gained from analytics. As a proof-of-concept, we applied the clustering technique DBSCAN and subsequent correlation analysis between app attributes for a set of 940 open-source mobile apps from F-Droid. We showed that (i) clusters of apps with similar characteristics provided more insight compared to applying the same to the whole data and (ii) defining the similarity of apps based on the similarity of topics as created from the topic modeling technique Latent Dirichlet Allocation does not significantly improve clustering results.


title: ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models

abstract: High-resolution Large Multimodal Models (LMMs) encounter the challenges of excessive visual tokens and quadratic visual complexity. Current high-resolution LMMs address the quadratic complexity while still generating excessive visual tokens. However, the redundancy in visual tokens is the key problem as it leads to more substantial compute. To mitigate this issue, we propose ConvLLaVA, which employs ConvNeXt, a hierarchical backbone, as the visual encoder of LMM to replace Vision Transformer (ViT). ConvLLaVA compresses high-resolution images into information-rich visual features, effectively preventing the generation of excessive visual tokens. To enhance the capabilities of ConvLLaVA, we propose two critical optimizations. Since the low-resolution pretrained ConvNeXt underperforms when directly applied on high resolution, we update it to bridge the gap. Moreover, since ConvNeXt's original compression ratio is inadequate for much higher resolution inputs, we train a successive stage to further compress the visual tokens, thereby reducing redundancy. These optimizations enable ConvLLaVA to support inputs of 1536x1536 resolution generating only 576 visual tokens, capable of handling images of arbitrary aspect ratios. Experimental results demonstrate that our method achieves competitive performance with state-of-the-art models on mainstream benchmarks. The ConvLLaVA model series are publicly available at https://github.com/alibaba/conv-llava.


title: Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias

abstract: Citation practices are crucial in shaping the structure of scientific knowledge, yet they are often influenced by contemporary norms and biases. The emergence of Large Language Models (LLMs) like GPT-4 introduces a new dynamic to these practices. Interestingly, the characteristics and potential biases of references recommended by LLMs that entirely rely on their parametric knowledge, and not on search or retrieval-augmented generation, remain unexplored. Here, we analyze these characteristics in an experiment using a dataset of 166 papers from AAAI, NeurIPS, ICML, and ICLR, published after GPT-4's knowledge cut-off date, encompassing 3,066 references in total. In our experiment, GPT-4 was tasked with suggesting scholarly references for the anonymized in-text citations within these papers. Our findings reveal a remarkable similarity between human and LLM citation patterns, but with a more pronounced high citation bias in GPT-4, which persists even after controlling for publication year, title length, number of authors, and venue. Additionally, we observe a large consistency between the characteristics of GPT-4's existing and non-existent generated references, indicating the model's internalization of citation patterns. By analyzing citation graphs, we show that the references recommended by GPT-4 are embedded in the relevant citation context, suggesting an even deeper conceptual internalization of the citation networks. While LLMs can aid in citation generation, they may also amplify existing biases and introduce new ones, potentially skewing scientific knowledge dissemination. Our results underscore the need for identifying the model's biases and for developing balanced methods to interact with LLMs in general.


title: On Flexible Inverse Probability of Treatment and Intensity Weighting: Informative Censoring, Variable Inclusion, and Weight Trimming

abstract: Many observational studies feature irregular longitudinal data, where the observation times are not common across individuals in the study. Further, the observation times may be related to the longitudinal outcome. In this setting, failing to account for the informative observation process may result in biased causal estimates. This can be coupled with other sources of bias, including non-randomized treatment assignments and informative censoring. This paper provides an overview of a flexible weighting method used to adjust for informative observation processes and non-randomized treatment assignments. We investigate the sensitivity of the flexible weighting method to violations of the noninformative censoring assumption, examine variable selection for the observation process weighting model, known as inverse intensity weighting, and look at the impacts of weight trimming for the flexible weighting model. We show that the flexible weighting method is sensitive to violations of the noninformative censoring assumption and show that a previously proposed extension fails under such violations. We also show that variables confounding the observation and outcome processes should always be included in the observation intensity model. Finally, we show that weight trimming should be applied in the flexible weighting model when the treatment assignment process is highly informative and driving the extreme weights. We conclude with an application of the methodology to a real data set to examine the impacts of household water sources on malaria diagnoses.


title: Near-threshold states in coupled $DD^{\ast}-D^{\ast}D^{\ast}$ scattering from lattice QCD

abstract: The first determination of doubly-charmed isospin-0 coupled-channel $DD^\ast-D^\ast D^\ast$ scattering amplitudes from lattice QCD is presented. The finite-volume spectrum is computed for three lattice volumes with a light-quark mass corresponding to $m_\pi\approx 391$ MeV and is used to extract the scattering amplitudes in $J^P = 1^+$ via the L\"{u}scher quantization condition. By analytically continuing the scattering amplitudes to complex energies, a $T_{cc}$ pole corresponding to a virtual bound state is found below $DD^\ast$ threshold. We also find a second pole, $T_{cc}^\prime$, corresponding to a resonance pole below the kinematically closed $D^\ast D^\ast$ channel, to which it has a strong coupling. A non-zero coupling is robustly found between the $S$-wave $D D^\ast$ and $D^\ast D^\ast$ channels producing a clear cusp in the $D D^\ast$ amplitude at the $D^\ast D^\ast$ threshold energy. This suggests that the experimental $T_{cc}^\prime$ should be observable in $D D^\ast$ and $D^\ast D^\ast$ final states at ongoing experiments.


title: Correlated Charge Density Wave Insulators in Chirally Twisted Triple Bilayer Graphene

abstract: Electrons residing in flat-band system can play a vital role in triggering spectacular phenomenology due to relatively large interactions and spontaneous breaking of different degeneracies. In this work we demonstrate chirally twisted triple bilayer graphene, a new moir\'e structure formed by three pieces of helically stacked Bernal bilayer graphene, as a highly tunable flat-band system. In addition to the correlated insulators showing at integer moir\'e fillings, commonly attributed to interaction induced symmetry broken isospin flavors in graphene, we observe abundant insulating states at half-integer moir\'e fillings, suggesting a longer-range interaction and the formation of charge density wave insulators which spontaneously break the moir\'e translation symmetry. With weak out-of-plane magnetic field applied, as observed half-integer filling states are enhanced and more quarter-integer filling states appear, pointing towards further quadrupling moir\'e unit cells. The insulating states at fractional fillings combined with Hartree-Fock calculations demonstrate the observation of a new type of correlated charge density wave insulators in graphene and points to a new accessible twist manner engineering correlated moir\'e electronics.


title: Sparse maximal update parameterization: A holistic approach to sparse training dynamics

abstract: Several challenges make it difficult for sparse neural networks to compete with dense models. First, setting a large fraction of weights to zero impairs forward and gradient signal propagation. Second, sparse studies often need to test multiple sparsity levels, while also introducing new hyperparameters (HPs), leading to prohibitive tuning costs. Indeed, the standard practice is to re-use the learning HPs originally crafted for dense models. Unfortunately, we show sparse and dense networks do not share the same optimal HPs. Without stable dynamics and effective training recipes, it is costly to test sparsity at scale, which is key to surpassing dense networks and making the business case for sparsity acceleration in hardware. A holistic approach is needed to tackle these challenges and we propose S$\mu$Par as one such approach. S$\mu$Par ensures activations, gradients, and weight updates all scale independently of sparsity level. Further, by reparameterizing the HPs, S$\mu$Par enables the same HP values to be optimal as we vary both sparsity level and model width. HPs can be tuned on small dense networks and transferred to large sparse models, greatly reducing tuning costs. On large-scale language modeling, S$\mu$Par training improves loss by up to 8.2% over the common approach of using the dense model standard parameterization.


title: CAFe: Cost and Age aware Federated Learning

abstract: In many federated learning (FL) models, a common strategy employed to ensure the progress in the training process, is to wait for at least $M$ clients out of the total $N$ clients to send back their local gradients based on a reporting deadline $T$, once the parameter server (PS) has broadcasted the global model. If enough clients do not report back within the deadline, the particular round is considered to be a failed round and the training round is restarted from scratch. If enough clients have responded back, the round is deemed successful and the local gradients of all the clients that responded back are used to update the global model. In either case, the clients that failed to report back an update within the deadline would have wasted their computational resources. Having a tighter deadline (small $T$) and waiting for a larger number of participating clients (large $M$) leads to a large number of failed rounds and therefore greater communication cost and computation resource wastage. However, having a larger $T$ leads to longer round durations whereas smaller $M$ may lead to noisy gradients. Therefore, there is a need to optimize the parameters $M$ and $T$ such that communication cost and the resource wastage is minimized while having an acceptable convergence rate. In this regard, we show that the average age of a client at the PS appears explicitly in the theoretical convergence bound, and therefore, can be used as a metric to quantify the convergence of the global model. We provide an analytical scheme to select the parameters $M$ and $T$ in this setting.


title: Magnetic effects in the Hadron Resonance Gas

abstract: We discuss the modeling of the hadronic phase of QCD at finite magnetic field in the framework of hadron resonance gas (HRG). We focus on the statistical description of particle yields that include contribution from resonance decays. We demonstrate that the swift increase in the number of protons with magnetic field predicted in the HRG is due to the ill-defined description of higher-spin states. We discuss fluctuations of conserved charges and show that at present the qualitative comparison of the model predictions with the Lattice QCD data should be treated with care. We also discuss the principle of detailed balance which allows to study the magnetic field dependence of neutral resonances.


title: Characterizing Discourse Group Roles in Inquiry-based University Science Labs

abstract: Group work is commonly adopted in university science laboratories. In this study, we characterize student discourse group roles in inquiry-based science labs. The instructional context for the study was a summer program hosted at a private research university in the eastern United States. The program was designed as a bridge program for matriculating students who were first generation and/or deaf or hard-of-hearing (DHH). Accommodations such as interpreters and technology were provided for DHH students. We analyzed 15 students' discourse moves in five lab activities from the video recordings, resulting in a total of 40 student-lab units. We developed codes to describe student discourse moves: asking a question, proposing an idea, participating in discussion, chatting off-task, and talking to instructor. Through a cluster analysis using those 40 student-lab units on our quantified discourse moves, we identified four discourse styles, high on-task high social, high on-task low social, low on-task low social, and low on-task high social. The results show that individual students tend to demonstrate varying discourse styles in different lab activities; students' discourse styles within the same groups tend to be aligned with their group members. Moreover, no difference was observed in discourse styles between genders, but DHH students were observed to participate significantly less in group discourse. The findings suggest that group-level interventions that specifically target discourse are needed to promote productive and equitable small-group discourse in university science labs.


title: Over-the-Air Runtime Wi-Fi MAC Address Re-randomization

abstract: Medium Access Control (MAC) address randomization is a key component for privacy protection in Wi-Fi networks. Current proposals periodically change the mobile device MAC addresses when it disconnects from the Access Point (AP). This way frames cannot be linked across changes, but the mobile device presence is exposed as long as it remains connected: all its communication is trivially linkable by observing the randomized yet same MAC address throughout the connection. Our runtime MAC re-randomization scheme addresses this issue, reducing or eliminating Wi-Fi frames linkability without awaiting for or requiring a disconnection. Our MAC re-randomization is practically 'over-the-air': MAC addresses are re-randomized just before transmission, while the protocol stacks (at the mobile and the AP) maintain locally the original connection MAC addresses - making our MAC layer scheme transparent to upper layers. With an implementation and a set of small-scale experiments with off-the-shelf devices, we show the feasibility of our scheme and the potential towards future deployment.


title: Cohomology of p-adic fields and Local class field theory

abstract: In this expository article, we outline a basic theory of group (co)homology and prove a cohomological formulation of the Local Reciprocity Law: $${\rm Gal}(L/K)^{\rm ab} \cong H_T^{-2}({\rm Gal}(L/K),\mathbb{Z}) \cong H_T^{0}({\rm Gal}(L/K),L^\times) \cong \frac{K^\times}{{\rm Nm}_{L/K}(L^\times)}$$ We first recall basic facts about local fields and homological algebra. Then we define group (co)homology, Tate cohomology, and furnish a toolbox. The Local Reciprocity Law is proven in an abstract cohomological setting, then applied to the case of local fields.


title: Collaborative Access Control for IoT -- A Blockchain Approach

abstract: The Internet of Things (IoT) necessitates robust access control mechanisms to secure a vast array of interconnected devices. Most of the existing IoT systems in practice use centralized solutions. We identify the problems in such solutions and adopt the blockchain based decentralized access control approach. Though there are works in the literature that use blockchain for access control, there are some gaps in these works. We develop a blockchain embedded access control (BEAC) framework to bridge the gaps. First, blockchain based solutions for access control require an enabling P2P network while existing P2P overlays do not support some required features. We develop a novel P2P infrastructure to seamlessly support our BEAC framework. Second, most of the works consider blockchain based access control for a single access control model, and we develop a generic blockchain mechanism and show that it can support the embedding of various access control models. Finally, existing works adopt existing blockchain mechanisms which may incur a high communication overhead. We develop a shortcut approach to improve the number of message rounds in the access protocol. Our experiments demonstrate the efficacy of our system, showing that the shortcut mechanism can reduces access time by approximately 43%.


title: Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidence

abstract: This paper introduces Filtered Corpus Training, a method that trains language models (LMs) on corpora with certain linguistic constructions filtered out from the training data, and uses it to measure the ability of LMs to perform linguistic generalization on the basis of indirect evidence. We apply the method to both LSTM and Transformer LMs (of roughly comparable size), developing filtered corpora that target a wide range of linguistic phenomena. Our results show that while transformers are better qua LMs (as measured by perplexity), both models perform equally and surprisingly well on linguistic generalization measures, suggesting that they are capable of generalizing from indirect evidence.


title: A review of active matter reviews

abstract: In the past years, the amount of research on active matter has grown extremely rapidly, a fact that is reflected in particular by the existence of more than 600 review articles on this topic. Moreover, the field has become very diverse, ranging from theoretical studies of the statistical mechanics of active particles to applied work on medical applications of microrobots and from biological systems to artificial swimmers. This makes it very difficult to get an overview over the field as a whole. Here, we provide such an overview in the form of a metareview article that surveys the existing review articles on active matter. Thereby, this article provides an introduction to the various subdisciplines of active matter science and constitutes a useful starting point for finding literature about a specific topic.

categories: cond-mat.soft cond-mat.stat-mech physics.bio-ph physics.flu-dyn physics.med-ph


title: Analysis of Marketed versus Not-marketed Mobile App Releases

abstract: Market and user characteristics of mobile apps make their release management different from proprietary software products and web services. Despite the wealth of information regarding users' feedback on an app, an in-depth analysis of app releases is difficult due to the inconsistency and uncertainty of the information. To better understand and potentially improve app release processes, we analyze major, minor, and patch releases for releases following semantic versioning. In particular, we were interested in finding out the difference between marketed and not-marketed releases. Our results show that, in general, major, minor, and patch releases have significant differences in the release cycle duration, nature, and change velocity. We also observed that there is a significant difference between marketed and non-marketed mobile app releases in terms of cycle duration, nature and the extent of changes, and the number of opened and closed issues.


title: Data Reconstruction: When You See It and When You Don't

abstract: We revisit the fundamental question of formally defining what constitutes a reconstruction attack. While often clear from the context, our exploration reveals that a precise definition is much more nuanced than it appears, to the extent that a single all-encompassing definition may not exist. Thus, we employ a different strategy and aim to "sandwich" the concept of reconstruction attacks by addressing two complementing questions: (i) What conditions guarantee that a given system is protected against such attacks? (ii) Under what circumstances does a given attack clearly indicate that a system is not protected? More specifically, * We introduce a new definitional paradigm -- Narcissus Resiliency -- to formulate a security definition for protection against reconstruction attacks. This paradigm has a self-referential nature that enables it to circumvent shortcomings of previously studied notions of security. Furthermore, as a side-effect, we demonstrate that Narcissus resiliency captures as special cases multiple well-studied concepts including differential privacy and other security notions of one-way functions and encryption schemes. * We formulate a link between reconstruction attacks and Kolmogorov complexity. This allows us to put forward a criterion for evaluating when such attacks are convincingly successful.


title: Score-based generative models are provably robust: an uncertainty quantification perspective

abstract: Through an uncertainty quantification (UQ) perspective, we show that score-based generative models (SGMs) are provably robust to the multiple sources of error in practical implementation. Our primary tool is the Wasserstein uncertainty propagation (WUP) theorem, a model-form UQ bound that describes how the $L^2$ error from learning the score function propagates to a Wasserstein-1 ($\mathbf{d}_1$) ball around the true data distribution under the evolution of the Fokker-Planck equation. We show how errors due to (a) finite sample approximation, (b) early stopping, (c) score-matching objective choice, (d) score function parametrization expressiveness, and (e) reference distribution choice, impact the quality of the generative model in terms of a $\mathbf{d}_1$ bound of computable quantities. The WUP theorem relies on Bernstein estimates for Hamilton-Jacobi-Bellman partial differential equations (PDE) and the regularizing properties of diffusion processes. Specifically, PDE regularity theory shows that stochasticity is the key mechanism ensuring SGM algorithms are provably robust. The WUP theorem applies to integral probability metrics beyond $\mathbf{d}_1$, such as the total variation distance and the maximum mean discrepancy. Sample complexity and generalization bounds in $\mathbf{d}_1$ follow directly from the WUP theorem. Our approach requires minimal assumptions, is agnostic to the manifold hypothesis and avoids absolute continuity assumptions for the target distribution. Additionally, our results clarify the trade-offs among multiple error sources in SGMs.


title: ETTrack: Enhanced Temporal Motion Predictor for Multi-Object Tracking

abstract: Many Multi-Object Tracking (MOT) approaches exploit motion information to associate all the detected objects across frames. However, many methods that rely on filtering-based algorithms, such as the Kalman Filter, often work well in linear motion scenarios but struggle to accurately predict the locations of objects undergoing complex and non-linear movements. To tackle these scenarios, we propose a motion-based MOT approach with an enhanced temporal motion predictor, ETTrack. Specifically, the motion predictor integrates a transformer model and a Temporal Convolutional Network (TCN) to capture short-term and long-term motion patterns, and it predicts the future motion of individual objects based on the historical motion information. Additionally, we propose a novel Momentum Correction Loss function that provides additional information regarding the motion direction of objects during training. This allows the motion predictor rapidly adapt to motion variations and more accurately predict future motion. Our experimental results demonstrate that ETTrack achieves a competitive performance compared with state-of-the-art trackers on DanceTrack and SportsMOT, scoring 56.4% and 74.4% in HOTA metrics, respectively.


title: Sparse Expansion and Neuronal Disentanglement

abstract: We show how to improve the inference efficiency of an LLM by expanding it into a mixture of sparse experts, where each expert is a copy of the original weights, one-shot pruned for a specific cluster of input values. We call this approach $\textit{Sparse Expansion}$. We show that, for models such as Llama 2 70B, as we increase the number of sparse experts, Sparse Expansion outperforms all other one-shot sparsification approaches for the same inference FLOP budget per token, and that this gap grows as sparsity increases, leading to inference speedups. But why? To answer this, we provide strong evidence that the mixture of sparse experts is effectively $\textit{disentangling}$ the input-output relationship of every individual neuron across clusters of inputs. Specifically, sparse experts approximate the dense neuron output distribution with fewer weights by decomposing the distribution into a collection of simpler ones, each with a separate sparse dot product covering it. Interestingly, we show that the Wasserstein distance between a neuron's output distribution and a Gaussian distribution is an indicator of its entanglement level and contribution to the accuracy of the model. Every layer of an LLM has a fraction of highly entangled Wasserstein neurons, and model performance suffers more when these are sparsified as opposed to others. The code for Sparse Expansion is available at: https://github.com/Shavit-Lab/Sparse-Expansion .


title: Looking Backward: Streaming Video-to-Video Translation with Feature Banks

abstract: This paper introduces StreamV2V, a diffusion model that achieves real-time streaming video-to-video (V2V) translation with user prompts. Unlike prior V2V methods using batches to process limited frames, we opt to process frames in a streaming fashion, to support unlimited frames. At the heart of StreamV2V lies a backward-looking principle that relates the present to the past. This is realized by maintaining a feature bank, which archives information from past frames. For incoming frames, StreamV2V extends self-attention to include banked keys and values and directly fuses similar past features into the output. The feature bank is continually updated by merging stored and new features, making it compact but informative. StreamV2V stands out for its adaptability and efficiency, seamlessly integrating with image diffusion models without fine-tuning. It can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. Quantitative metrics and user studies confirm StreamV2V's exceptional ability to maintain temporal consistency.


title: InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation

abstract: Recent talking avatar generation models have made strides in achieving realistic and accurate lip synchronization with the audio, but often fall short in controlling and conveying detailed expressions and emotions of the avatar, making the generated video less vivid and controllable. In this paper, we propose a novel text-guided approach for generating emotionally expressive 2D avatars, offering fine-grained control, improved interactivity, and generalizability to the resulting video. Our framework, named InstructAvatar, leverages a natural language interface to control the emotion as well as the facial motion of avatars. Technically, we design an automatic annotation pipeline to construct an instruction-video paired training dataset, equipped with a novel two-branch diffusion-based generator to predict avatars with audio and text instructions at the same time. Experimental results demonstrate that InstructAvatar produces results that align well with both conditions, and outperforms existing methods in fine-grained emotion control, lip-sync quality, and naturalness. Our project page is https://wangyuchi369.github.io/InstructAvatar/.


title: A skew Specht perspective of RoCK blocks and cuspidal systems for KLR algebras in affine type A

abstract: Cuspidal systems parameterize KLR algebra representations via root partitions $\pi$, where simple modules $L(\pi)$ arise as heads of proper standard modules. Working in affine type A with an arbitrary convex preorder, we construct explicit skew diagrams $\zeta(\pi)$ such that the skew Specht module $S^{\zeta(\pi)}$ has simple head $L(\pi)$ and a filtration by proper standard modules. This result arises from an in-depth study of the combinatorial interplay between cuspidal systems and RoCK cyclotomic KLR algebras.


title: GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction

abstract: Social biases in LLMs are usually measured via bias benchmark datasets. Current benchmarks have limitations in scope, grounding, quality, and human effort required. Previous work has shown success with a community-sourced, rather than crowd-sourced, approach to benchmark development. However, this work still required considerable effort from annotators with relevant lived experience. This paper explores whether an LLM (specifically, GPT-3.5-Turbo) can assist with the task of developing a bias benchmark dataset from responses to an open-ended community survey. We also extend the previous work to a new community and set of biases: the Jewish community and antisemitism. Our analysis shows that GPT-3.5-Turbo has poor performance on this annotation task and produces unacceptable quality issues in its output. Thus, we conclude that GPT-3.5-Turbo is not an appropriate substitute for human annotation in sensitive tasks related to social biases, and that its use actually negates many of the benefits of community-sourcing bias benchmarks.


title: Multi-mode delta Sct stars from the Zwicky Transient Facility Survey

abstract: We obtain the largest catalog of multi-mode $\delta$ Sct stars in the northern sky to date using the Zwicky Transient Facility (ZTF) Data Release 20 (DR20). The catalog includes 2254 objects, of which 2181 are new to our study. Among these multi-mode $\delta$ Sct stars, 2142 objects are double-mode $\delta$ Sct, while 109 objects are triple-mode $\delta$ Sct and 3 are quadruple-mode $\delta$ Sct. By analyzing the light curves in the $r$ and $g$ bands of the ZTF, we determine the basic parameters of multi-mode $\delta$ Sct stars, including the periods and amplitudes. Periods are checked by comparison with the OGLE catalog of double-mode $\delta$ Sct stars. On the Petersen diagram, multi-mode $\delta$ Sct stars form six sequences. We find that in Galactic coordinates, the periods of 1O/F double-mode $\delta$ Sct stars at high latitudes are shorter than those of 1O/F double-mode $\delta$ Sct stars in the disk, due to metallicity variations. In the future, our catalog can be used to establish the period--luminosity relation and the period--metallicity relation of double-mode $\delta$ Sct stars, and to study the Galactic structure.


title: Sliding-Mode Nash Equilibrium Seeking for a Quadratic Duopoly Game

abstract: This paper introduces a new method to achieve stable convergence to Nash equilibrium in duopoly noncooperative games. Inspired by the recent fixed-time Nash Equilibrium seeking (NES) as well as prescribed-time extremum seeking (ES) and source seeking schemes, our approach employs a distributed sliding mode control (SMC) scheme, integrating extremum seeking with sinusoidal perturbation signals to estimate the pseudogradients of quadratic payoff functions. Notably, this is the first attempt to address noncooperative games without relying on models, combining classical extremum seeking with relay components instead of proportional control laws. We prove finite-time convergence of the closed-loop average system to Nash equilibrium using stability analysis techniques such as time-scaling, Lyapunov's direct method, and averaging theory for discontinuous systems. Additionally, we quantify the size of residual sets around the Nash equilibrium and validate our theoretical results through simulations.


title: FreeMotion: A Unified Framework for Number-free Text-to-Motion Synthesis

abstract: Text-to-motion synthesis is a crucial task in computer vision. Existing methods are limited in their universality, as they are tailored for single-person or two-person scenarios and can not be applied to generate motions for more individuals. To achieve the number-free motion synthesis, this paper reconsiders motion generation and proposes to unify the single and multi-person motion by the conditional motion distribution. Furthermore, a generation module and an interaction module are designed for our FreeMotion framework to decouple the process of conditional motion generation and finally support the number-free motion synthesis. Besides, based on our framework, the current single-person motion spatial control method could be seamlessly integrated, achieving precise control of multi-person motion. Extensive experiments demonstrate the superior performance of our method and our capability to infer single and multi-human motions simultaneously.


title: Self-sensing with hollow cylindrical transducers for histotripsy enhanced aspiration mechanical thrombectomy applications

abstract: To address existing challenges with intravascular mechanical thrombectomy devices, a novel ultrasound-enhanced aspiration approach is being developed to mechanically degrade clots using cavitation. This method employs standing waves within a mm-scale hollow cylindrical transducer to generate high pressures sufficient to perform histotripsy on clots situated within the transducer lumen and generate substantial lesions. The objective of this study is to assess the feasibility of self-sensing cavitation detection by analyzing voltage signals across the transducer during treatment pulses. External ultrasound imaging of the transducer lumen validated cavitation detection. Impedance was also altered by the presence of clot material within the lumen. Experiments varying the driving voltage in water-filled lumens demonstrated changes in the relative amplitudes of the envelopes of the pulse body and ringdown portions of the voltage signals above the cavitation threshold, as well as changes in the spectral domain. In particular both broadband and ultraharmonic signals showed an increase in amplitude above the cavitation threshold. Similar temporal and spectral voltage signal changes in the presence of cavitation were also observed when treating clots within the lumen. This work demonstrates a highly sensitive method for detecting cavitation within the lumen, enabling monitoring with readily acquired signals without additional sensors in the catheter configuration. These findings hold significant potential for improving the efficacy of ultrasound-enhanced aspiration thrombectomy procedures.


title: Scaling Laws for Discriminative Classification in Large Language Models

abstract: Modern large language models (LLMs) represent a paradigm shift in what can plausibly be expected of machine learning models. The fact that LLMs can effectively generate sensible answers to a diverse range of queries suggests that they would be useful in customer support applications. While powerful, LLMs have been observed to be prone to hallucination which unfortunately makes their near term use in customer support applications challenging. To address this issue we present a system that allows us to use an LLM to augment our customer support advocates by re-framing the language modeling task as a discriminative classification task. In this framing, we seek to present the top-K best template responses for a customer support advocate to use when responding to a customer. We present the result of both offline and online experiments where we observed offline gains and statistically significant online lifts for our experimental system. Along the way, we present observed scaling curves for validation loss and top-K accuracy, resulted from model parameter ablation studies. We close by discussing the space of trade-offs with respect to model size, latency, and accuracy as well as and suggesting future applications to explore.


title: Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development

abstract: The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance, enhancing patient safety by identifying potential risks associated with medications, facilitating early detection of adverse events, and guiding regulatory decision-making. Traditional ADE detection methods are reliable but slow, not easily adaptable to large-scale operations, and offer limited information. With the exponential increase in data sources like social media content, biomedical literature, and Electronic Medical Records (EMR), extracting relevant ADE-related information from these unstructured texts is imperative. Previous ADE mining studies have focused on text-based methodologies, overlooking visual cues, limiting contextual comprehension, and hindering accurate interpretation. To address this gap, we present a MultiModal Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual information with visual aids. Additionally, we introduce a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events. Using our MMADE dataset, we showcase the significance of integrating visual cues from images to enhance overall performance. This approach holds promise for patient safety, ADE awareness, and healthcare accessibility, paving the way for further exploration in personalized healthcare.


title: Improved Particle Approximation Error for Mean Field Neural Networks

abstract: Mean-field Langevin dynamics (MFLD) minimizes an entropy-regularized nonlinear convex functional defined over the space of probability distributions. MFLD has gained attention due to its connection with noisy gradient descent for mean-field two-layer neural networks. Unlike standard Langevin dynamics, the nonlinearity of the objective functional induces particle interactions, necessitating multiple particles to approximate the dynamics in a finite-particle setting. Recent works (Chen et al., 2022; Suzuki et al., 2023b) have demonstrated the uniform-in-time propagation of chaos for MFLD, showing that the gap between the particle system and its mean-field limit uniformly shrinks over time as the number of particles increases. In this work, we improve the dependence on logarithmic Sobolev inequality (LSI) constants in their particle approximation errors, which can exponentially deteriorate with the regularization coefficient. Specifically, we establish an LSI-constant-free particle approximation error concerning the objective gap by leveraging the problem structure in risk minimization. As the application, we demonstrate improved convergence of MFLD, sampling guarantee for the mean-field stationary distribution, and uniform-in-time Wasserstein propagation of chaos in terms of particle complexity.


title: Canonical Variates in Wasserstein Metric Space

abstract: In this paper, we address the classification of instances each characterized not by a singular point, but by a distribution on a vector space. We employ the Wasserstein metric to measure distances between distributions, which are then used by distance-based classification algorithms such as k-nearest neighbors, k-means, and pseudo-mixture modeling. Central to our investigation is dimension reduction within the Wasserstein metric space to enhance classification accuracy. We introduce a novel approach grounded in the principle of maximizing Fisher's ratio, defined as the quotient of between-class variation to within-class variation. The directions in which this ratio is maximized are termed discriminant coordinates or canonical variates axes. In practice, we define both between-class and within-class variations as the average squared distances between pairs of instances, with the pairs either belonging to the same class or to different classes. This ratio optimization is achieved through an iterative algorithm, which alternates between optimal transport and maximization steps within the vector space. We conduct empirical studies to assess the algorithm's convergence and, through experimental validation, demonstrate that our dimension reduction technique substantially enhances classification performance. Moreover, our method outperforms well-established algorithms that operate on vector representations derived from distributional data. It also exhibits robustness against variations in the distributional representations of data clouds.


title: FastDrag: Manipulate Anything in One Step

abstract: Drag-based image editing using generative models provides precise control over image contents, enabling users to manipulate anything in an image with a few clicks. However, prevailing methods typically adopt $n$-step iterations for latent semantic optimization to achieve drag-based image editing, which is time-consuming and limits practical applications. In this paper, we introduce a novel one-step drag-based image editing method, i.e., FastDrag, to accelerate the editing process. Central to our approach is a latent warpage function (LWF), which simulates the behavior of a stretched material to adjust the location of individual pixels within the latent space. This innovation achieves one-step latent semantic optimization and hence significantly promotes editing speeds. Meanwhile, null regions emerging after applying LWF are addressed by our proposed bilateral nearest neighbor interpolation (BNNI) strategy. This strategy interpolates these regions using similar features from neighboring areas, thus enhancing semantic integrity. Additionally, a consistency-preserving strategy is introduced to maintain the consistency between the edited and original images by adopting semantic information from the original image, saved as key and value pairs in self-attention module during diffusion inversion, to guide the diffusion sampling. Our FastDrag is validated on the DragBench dataset, demonstrating substantial improvements in processing time over existing methods, while achieving enhanced editing performance. Project page: https://fastdrag-site.github.io/ .


title: Self-consistent evaluation of proximity and inverse proximity effects with pair-breaking in diffusive SN junctions

abstract: We consider a planar superconducting-normal-metal (SN) junction with both inelastic and spin-flip scattering processes present. In the diffusive limit, we use a one-dimensional formulation of the Usadel equation to compute the self-consistent energy dependence of the single-particle density of states as a function of distance from the interface on both the superconducting and metallic sides for various spatial profiles of a pair-breaking spin-flip term. The pair-breaking processes fill in the superconducting gap at zero energy, which is reflected in the zero-bias tunneling conductance in scanning tunneling microscopy/spectroscopy experiments, in the vicinity of the junction. We also investigate the impact of having a partially transparent interface at the junction. We compare our findings with the observed exponential rise in the zero-bias conductance at the 1H step edge in recent experiments on 4Hb-TaS$_2$ [A. K. Nayak et al., Nat. Phys. 17, 1413 (2021)].


title: Adaptive Splitting of Reusable Temporal Monitors for Rare Traffic Violations

abstract: Autonomous Vehicles (AVs) are often tested in simulation to estimate the probability they will violate safety specifications. Two common issues arise when using existing techniques to produce this estimation: If violations occur rarely, simple Monte-Carlo sampling techniques can fail to produce efficient estimates; if simulation horizons are too long, importance sampling techniques (which learn proposal distributions from past simulations) can fail to converge. This paper addresses both issues by interleaving rare-event sampling techniques with online specification monitoring algorithms. We use adaptive multi-level splitting to decompose simulations into partial trajectories, then calculate the distance of those partial trajectories to failure by leveraging robustness metrics from Signal Temporal Logic (STL). By caching those partial robustness metric values, we can efficiently re-use computations across multiple sampling stages. Our experiments on an interstate lane-change scenario show our method is viable for testing simulated AV-pipelines, efficiently estimating failure probabilities for STL specifications based on real traffic rules. We produce better estimates than Monte-Carlo and importance sampling in fewer simulations.


title: Scenario Engineering for Autonomous Transportation: A New Stage in Open-Pit Mines

abstract: In recent years, open-pit mining has seen significant advancement, the cooperative operation of various specialized machinery substantially enhancing the efficiency of mineral extraction. However, the harsh environment and complex conditions in open-pit mines present substantial challenges for the implementation of autonomous transportation systems. This research introduces a novel paradigm that integrates Scenario Engineering (SE) with autonomous transportation systems to significantly improve the trustworthiness, robustness, and efficiency in open-pit mines by incorporating the four key components of SE, including Scenario Feature Extractor, Intelligence and Index (I&I), Calibration and Certification (C&C), and Verification and Validation (V&V). This paradigm has been validated in two famous open-pit mines, the experiment results demonstrate marked improvements in robustness, trustworthiness, and efficiency. By enhancing the capacity, scalability, and diversity of autonomous transportation, this paradigm fosters the integration of SE and parallel driving and finally propels the achievement of the '6S' objectives.


title: Feature Aggregation with Latent Generative Replay for Federated Continual Learning of Socially Appropriate Robot Behaviours

abstract: For widespread real-world applications, it is beneficial for robots to explore Federated Learning (FL) settings where several robots, deployed in parallel, can learn independently while also sharing their learning with each other. This work explores a simulated living room environment where robots need to learn the social appropriateness of their actions. We propose Federated Root (FedRoot), a novel weight aggregation strategy which disentangles feature learning across clients from individual task-based learning. Adapting popular FL strategies to use FedRoot instead, we present a novel FL benchmark for learning the social appropriateness of different robot actions in diverse social configurations. FedRoot-based methods offer competitive performance compared to others while offering sizeable (up to 86% for CPU usage and up to 72% for GPU usage) reduction in resource consumption. Furthermore, real-world interactions require social robots to dynamically adapt to changing environmental and task settings. To facilitate this, we propose Federated Latent Generative Replay (FedLGR), a novel Federated Continual Learning (FCL) strategy that uses FedRoot-based weight aggregation and embeds each client with a generator model for pseudo-rehearsal of learnt feature embeddings to mitigate forgetting in a resource-efficient manner. Our benchmark results demonstrate that FedRoot-based FCL methods outperform other methods while also offering sizeable (up to 84% for CPU usage and up to 92% for GPU usage) reduction in resource consumption, with FedLGR providing the best results across evaluations.


title: Comprehensive Autonomous Vehicle Optimal Routing With Dynamic Heuristics

abstract: Auto manufacturers and research groups are working on autonomous driving for long period and achieved significant progress. Autonomous vehicles (AV) are expected to transform road traffic reduction from current conditions, avoiding accidents and congestion. As the implementation of an autonomous vehicle ecosystem includes complex automotive technology, ethics, passenger behaviour, traffic management policies and liability etc., the maturity of AV solutions are still evolving. The proposed model to improve AV user experience, uses a hybrid AV Network of multiple connected autonomous vehicles which communicate with each other in an environment shared by human driven vehicles. The proposed Optimal AV Network (OAVN) solution provides better coordination and optimization of autonomous vehicles, improved Transportation efficiency, improved passenger comfort and safety, real-time dynamic adaption of traffic & road conditions along with improved in-cabin assistance with inputs from various sensors. The true optimal solution for this problem, is to devise an automated guidance system for vehicles in an AV network, to reach destinations in best possible routes along with passenger comfort and safety. A custom informed search model is proposed along with other heuristic goals for better user experience. The results are analysed and compared to evaluate the effectiveness of the solution and identify gaps and future enhancements.


title: AI Robots and Humanoid AI: Review, Perspectives and Directions

abstract: In the approximately century-long journey of robotics, humanoid robots made their debut around six decades ago. The rapid advancements in generative AI, large language models (LLMs), and large multimodal models (LMMs) have reignited interest in humanoids, steering them towards real-time, interactive, and multimodal designs and applications. This resurgence unveils boundless opportunities for AI robotics and novel applications, paving the way for automated, real-time and humane interactions with humanoid advisers, educators, medical professionals, caregivers, and receptionists. However, while current humanoid robots boast human-like appearances, they have yet to embody true humaneness, remaining distant from achieving human-like intelligence. In our comprehensive review, we delve into the intricate landscape of AI robotics and AI humanoid robots in particular, exploring the challenges, perspectives and directions in transitioning from human-looking to humane humanoids and fostering human-like robotics. This endeavour synergizes the advancements in LLMs, LMMs, generative AI, and human-level AI with humanoid robotics, omniverse, and decentralized AI, ushering in the era of AI humanoids and humanoid AI.


title: CalliRewrite: Recovering Handwriting Behaviors from Calligraphy Images without Supervision

abstract: Human-like planning skills and dexterous manipulation have long posed challenges in the fields of robotics and artificial intelligence (AI). The task of reinterpreting calligraphy presents a formidable challenge, as it involves the decomposition of strokes and dexterous utensil control. Previous efforts have primarily focused on supervised learning of a single instrument, limiting the performance of robots in the realm of cross-domain text replication. To address these challenges, we propose CalliRewrite: a coarse-to-fine approach for robot arms to discover and recover plausible writing orders from diverse calligraphy images without requiring labeled demonstrations. Our model achieves fine-grained control of various writing utensils. Specifically, an unsupervised image-to-sequence model decomposes a given calligraphy glyph to obtain a coarse stroke sequence. Using an RL algorithm, a simulated brush is fine-tuned to generate stylized trajectories for robotic arm control. Evaluation in simulation and physical robot scenarios reveals that our method successfully replicates unseen fonts and styles while achieving integrity in unknown characters.


title: Multi-agent Collaborative Perception for Robotic Fleet: A Systematic Review

abstract: Collaborative perception in multi-robot fleets is a way to incorporate the power of unity in robotic fleets. Collaborative perception refers to the collective ability of multiple entities or agents to share and integrate their sensory information for a more comprehensive understanding of their environment. In other words, it involves the collaboration and fusion of data from various sensors or sources to enhance perception and decision-making capabilities. By combining data from diverse sources, such as cameras, lidar, radar, or other sensors, the system can create a more accurate and robust representation of the environment. In this review paper, we have summarized findings from 20+ research papers on collaborative perception. Moreover, we discuss testing and evaluation frameworks commonly accepted in academia and industry for autonomous vehicles and autonomous mobile robots. Our experiments with the trivial perception module show an improvement of over 200% with collaborative perception compared to individual robot perception. Here's our GitHub repository that shows the benefits of collaborative perception: https://github.com/synapsemobility/synapseBEV


title: Investigation of Energy-efficient AI Model Architectures and Compression Techniques for "Green" Fetal Brain Segmentation

abstract: Artificial intelligence have contributed to advancements across various industries. However, the rapid growth of artificial intelligence technologies also raises concerns about their environmental impact, due to associated carbon footprints to train computational models. Fetal brain segmentation in medical imaging is challenging due to the small size of the fetal brain and the limited image quality of fast 2D sequences. Deep neural networks are a promising method to overcome this challenge. In this context, the construction of larger models requires extensive data and computing power, leading to high energy consumption. Our study aims to explore model architectures and compression techniques that promote energy efficiency by optimizing the trade-off between accuracy and energy consumption through various strategies such as lightweight network design, architecture search, and optimized distributed training tools. We have identified several effective strategies including optimization of data loading, modern optimizers, distributed training strategy implementation, and reduced floating point operations precision usage with light model architectures while tuning parameters according to available computer resources. Our findings demonstrate that these methods lead to satisfactory model performance with low energy consumption during deep neural network training for medical image segmentation.


title: LiteNeXt: A Novel Lightweight ConvMixer-based Model with Self-embedding Representation Parallel for Medical Image Segmentation

abstract: The emergence of deep learning techniques has advanced the image segmentation task, especially for medical images. Many neural network models have been introduced in the last decade bringing the automated segmentation accuracy close to manual segmentation. However, cutting-edge models like Transformer-based architectures rely on large scale annotated training data, and are generally designed with densely consecutive layers in the encoder, decoder, and skip connections resulting in large number of parameters. Additionally, for better performance, they often be pretrained on a larger data, thus requiring large memory size and increasing resource expenses. In this study, we propose a new lightweight but efficient model, namely LiteNeXt, based on convolutions and mixing modules with simplified decoder, for medical image segmentation. The model is trained from scratch with small amount of parameters (0.71M) and Giga Floating Point Operations Per Second (0.42). To handle boundary fuzzy as well as occlusion or clutter in objects especially in medical image regions, we propose the Marginal Weight Loss that can help effectively determine the marginal boundary between object and background. Furthermore, we propose the Self-embedding Representation Parallel technique, that can help augment the data in a self-learning manner. Experiments on public datasets including Data Science Bowls, GlaS, ISIC2018, PH2, and Sunnybrook data show promising results compared to other state-of-the-art CNN-based and Transformer-based architectures. Our code will be published at: https://github.com/tranngocduvnvp/LiteNeXt.