title: Ultrafast Excitation Exchange in a Maxwell-Fish-Eye Lens

abstract: The strong coupling of quantum emitters to a cavity mode has been of paramount importance in the development of quantum optics. Recently, also the strong coupling to more than a single mode of an electromagnetic resonator has drawn considerable interest. We investigate how this multimode strong coupling regime can be harnessed to coherently control quantum systems. Specifically, we demonstrate that a Maxwell-Fish-Eye lens can be used to implement a pulsed excitation-exchange between two distant quantum emitters. This periodic exchange is mediated by single-photon pulses and can be extended to a photon-exchange between two atomic ensembles, for which the coupling strength is enhanced collectively.


title: Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web

abstract: Language model agents (LMA) recently emerged as a promising paradigm on muti-step decision making tasks, often outperforming humans and other reinforcement learning agents. Despite the promise, their performance on real-world applications that often involve combinations of tasks is still underexplored. In this work, we introduce a new benchmark, called CompWoB -- 50 new compositional web automation tasks reflecting more realistic assumptions. We show that while existing prompted LMAs (gpt-3.5-turbo or gpt-4) achieve 94.0% average success rate on base tasks, their performance degrades to 24.9% success rate on compositional tasks. On the other hand, transferred LMAs (finetuned only on base tasks) show less generalization gap, dropping from 85.4% to 54.8%. By balancing data distribution across tasks, we train a new model, HTML-T5++, that surpasses human-level performance (95.2%) on MiniWoB, and achieves the best zero-shot performance on CompWoB (61.5%). While these highlight the promise of small-scale finetuned and transferred models for task compositionality, their performance further degrades under different instruction compositions changing combinational order. In contrast to the recent remarkable success of LMA, our benchmark and detailed analysis emphasize the necessity of building LMAs that are robust and generalizable to task compositionality for real-world deployment.


title: Double-Virtual NNLO QCD Corrections for Five-Parton Scattering: The Quark Channels

abstract: We complete the computation of two-loop helicity amplitudes required to obtain next-to-next-to-leading order QCD corrections for three-jet production at hadron colliders, including all contributions beyond the leading-color approximation. The analytic expressions are reconstructed from finite-field samples obtained with the numerical unitarity method. We find that the reconstruction is significantly facilitated by exploiting the overlaps between rational coefficient functions of quark and gluon processes, and we display their compact generating sets in the appendix of the paper. We implement our results in a public code, and demonstrate its suitability for phenomenological applications.


title: A note on extremal constructions for the Erd\H{o}s--Rademacher problem

abstract: For given positive integers $r\ge 3$, $n$ and $e\le \binom{n}{2}$, the famous Erd\H{o}s--Rademacher problem asks for the minimum number of $r$-cliques in a graph with $n$ vertices and $e$ edges. A conjecture of Lov\'{a}sz and Simonovits from the 1970s states that, for every $r\ge 3$, if $n$ is sufficiently large then, for every $e\le \binom{n}{2}$, at least one extremal graph can be obtained from a complete partite graph by adding a triangle-free graph into one part. In this note, we explicitly write the minimum number of $r$-cliques predicted by the above conjecture. Also, we describe what we believe to be the set of extremal graphs for any $r\ge 4$ and all large $n$, amending the previous conjecture of Pikhurko and Razborov.


title: Projectively induced K\"ahler cones over regular Sasakian manifolds

abstract: Motivated by a conjecture in [9] we prove that the K\"ahler cone over a regular complete Sasakian manifold is Ricci-flat and projectively induced if and only if it is flat. We also obtain that, up to $\mathcal D_a$-homothetic transformations, K\"ahler cones over homogeneous compact Sasakian manifolds are projectively induced. As main tool we provide a relation between the K\"ahler potentials of the transverse K\"ahler metric and of the cone metric.


title: Effect of heating or cooling in a suspension of phototactic algae with no-slip boundary conditions

abstract: In this study, we investigate the impact of heating or cooling in a suspension experiencing phototactic bioconvection. The suspension is illuminated by collimated irradiation from the top and subjected to heating or cooling from the bottom. The governing equations include the Navier Stokes equations with the Boussinesq approximation, the diffusion equation for motile microorganisms, and the energy equation for temperature. Employing linear perturbation theory, we analyse the linear stability of the suspension. The findings predict that the suspension undergoes destabilization when heated from below and stabilization when cooled from below. This suggests a sensitive dependence of the system's stability on the thermal conditions, providing valuable insights into the behavior of phototactic bioconvection under different heating or cooling scenarios.


title: Stress control in non-ideal topological Maxwell lattices via geometry

abstract: Topological mechanical metamaterials have demonstrated exotic and robust mechanical properties which led to promising engineering applications. One of such properties is the focusing of stress at the interface connecting domains of topological Maxwell lattices of opposite topological polarizations, which protects the bulk of the material against fracturing. Here we generalize this theory to non-ideal Maxwell lattices, incorporate real material features that leads to interactions beyond previous ideal models. By quantitative analysis of stress distributions of topological Maxwell lattices with self-stress interfaces theoretically and computationally, we propose a design rule that minimizes stress in the bulk of the material. This design rule can guide the realization of stress focusing and fracturing protection in real materials.


title: Analytic Besov functional calculus for several commuting operators

abstract: This paper investigates when analytic Besov functions of $n$ variables act on the generators of $n$ commuting $C_0$-semigroups on a Banach space. The theory for $n=1$ has already been published, and the present paper uses a different approach to that case as well as extending to the cases when $n\ge2$. It also clarifies some spectral mapping properties and provides some operator norm estimates.


title: Semi-supervised Semantic Segmentation via Boosting Uncertainty on Unlabeled Data

abstract: We bring a new perspective to semi-supervised semantic segmentation by providing an analysis on the labeled and unlabeled distributions in training datasets. We first figure out that the distribution gap between labeled and unlabeled datasets cannot be ignored, even though the two datasets are sampled from the same distribution. To address this issue, we theoretically analyze and experimentally prove that appropriately boosting uncertainty on unlabeled data can help minimize the distribution gap, which benefits the generalization of the model. We propose two strategies and design an uncertainty booster algorithm, specially for semi-supervised semantic segmentation. Extensive experiments are carried out based on these theories, and the results confirm the efficacy of the algorithm and strategies. Our plug-and-play uncertainty booster is tiny, efficient, and robust to hyperparameters but can significantly promote performance. Our approach achieves state-of-the-art performance in our experiments compared to the current semi-supervised semantic segmentation methods on the popular benchmarks: Cityscapes and PASCAL VOC 2012 with different train settings.


title: Bootstrap Inference on Partially Linear Binary Choice Model

abstract: The partially linear binary choice model can be used for estimating structural equations where nonlinearity may appear due to diminishing marginal returns, different life cycle regimes, or hectic physical phenomena. The inference procedure for this model based on the analytic asymptotic approximation could be unreliable in finite samples if the sample size is not sufficiently large. This paper proposes a bootstrap inference approach for the model. Monte Carlo simulations show that the proposed inference method performs well in finite samples compared to the procedure based on the asymptotic approximation.


title: TaskBench: Benchmarking Large Language Models for Task Automation

abstract: Recently, the incredible progress of large language models (LLMs) has ignited the spark of task automation, which decomposes the complex tasks described by user instructions into sub-tasks, and invokes external tools to execute them, and plays a central role in autonomous agents. However, there lacks a systematic and standardized benchmark to foster the development of LLMs in task automation. To this end, we introduce TaskBench to evaluate the capability of LLMs in task automation. Specifically, task automation can be formulated into three critical stages: task decomposition, tool invocation, and parameter prediction to fulfill user intent. This complexity makes data collection and evaluation more challenging compared to common NLP tasks. To generate high-quality evaluation datasets, we introduce the concept of Tool Graph to represent the decomposed tasks in user intent, and adopt a back-instruct method to simulate user instruction and annotations. Furthermore, we propose TaskEval to evaluate the capability of LLMs from different aspects, including task decomposition, tool invocation, and parameter prediction. Experimental results demonstrate that TaskBench can effectively reflects the capability of LLMs in task automation. Benefiting from the mixture of automated data construction and human verification, TaskBench achieves a high consistency compared to the human evaluation, which can be utilized as a comprehensive and faithful benchmark for LLM-based autonomous agents.


title: Can training neural language models on a curriculum with developmentally plausible data improve alignment with human reading behavior?

abstract: The use of neural language models to model human behavior has met with mixed success. While some work has found that the surprisal estimates from these models can be used to predict a wide range of human neural and behavioral responses, other work studying more complex syntactic phenomena has found that these surprisal estimates generate incorrect behavioral predictions. This paper explores the extent to which the misalignment between empirical and model-predicted behavior can be minimized by training models on more developmentally plausible data, such as in the BabyLM Challenge. We trained teacher language models on the BabyLM "strict-small" dataset and used sentence level surprisal estimates from these teacher models to create a curriculum. We found tentative evidence that our curriculum made it easier for models to acquire linguistic knowledge from the training data: on the subset of tasks in the BabyLM challenge suite evaluating models' grammatical knowledge of English, models first trained on the BabyLM data curriculum and then on a few randomly ordered training epochs performed slightly better than models trained on randomly ordered epochs alone. This improved linguistic knowledge acquisition did not result in better alignment with human reading behavior, however: models trained on the BabyLM dataset (with or without a curriculum) generated predictions that were as misaligned with human behavior as models trained on larger less curated datasets. This suggests that training on developmentally plausible datasets alone is likely insufficient to generate language models capable of accurately predicting human language processing.


title: Performance Analysis of Integrated Sensing and Communications Under Gain-Phase Imperfections

abstract: This paper evaluates the performance of uplink integrated sensing and communication systems in the presence of gain and phase imperfections. Specifically, we consider multiple unmanned aerial vehicles (UAVs) transmitting data to a multiple-input-multiple-output base-station (BS) that is responsible for estimating the transmitted information in addition to localising the transmitting UAVs. The signal processing at the BS is divided into two consecutive stages: localisation and communication. A maximum likelihood (ML) algorithm is introduced for the localisation stage to jointly estimate the azimuth-elevation angles and Doppler frequency of the UAVs under gain-phase defects, which are then compared to the estimation of signal parameters via rotational invariance techniques (ESPRIT) and multiple signal classification (MUSIC). Furthermore, the Cramer-Rao lower bound (CRLB) is derived to evaluate the asymptotic performance and quantify the influence of the gain-phase imperfections which are modelled using Rician and von Mises distributions, respectively. Thereafter, in the communication stage, the location parameters estimated in the first stage are employed to estimate the communication channels which are fed into a maximum ratio combiner to preprocess the received communication signal. An accurate closed-form approximation of the achievable average sum data rate (SDR) for all UAVs is derived. The obtained results show that gain-phase imperfections have a significant influence on both localisation and communication, however, the proposed ML is less sensitive when compared to other algorithms. The derived analysis is concurred with simulations.


title: Continual Diffusion with STAMINA: STack-And-Mask INcremental Adapters

abstract: Recent work has demonstrated a remarkable ability to customize text-to-image diffusion models to multiple, fine-grained concepts in a sequential (i.e., continual) manner while only providing a few example images for each concept. This setting is known as continual diffusion. Here, we ask the question: Can we scale these methods to longer concept sequences without forgetting? Although prior work mitigates the forgetting of previously learned concepts, we show that its capacity to learn new tasks reaches saturation over longer sequences. We address this challenge by introducing a novel method, STack-And-Mask INcremental Adapters (STAMINA), which is composed of low-ranked attention-masked adapters and customized MLP tokens. STAMINA is designed to enhance the robust fine-tuning properties of LoRA for sequential concept learning via learnable hard-attention masks parameterized with low rank MLPs, enabling precise, scalable learning via sparse adaptation. Notably, all introduced trainable parameters can be folded back into the model after training, inducing no additional inference parameter costs. We show that STAMINA outperforms the prior SOTA for the setting of text-to-image continual customization on a 50-concept benchmark composed of landmarks and human faces, with no stored replay data. Additionally, we extended our method to the setting of continual learning for image classification, demonstrating that our gains also translate to state-of-the-art performance in this standard benchmark.


title: Rigidity of Kantorovich solutions in discrete Optimal Transport

abstract: We study optimal transport plans from $m$ equally weighted points (with weights $1/m$) to $n$ equally weighted points (with weights $1/n$). The Birkhoff-von Neumann Theorem implies that if $m=n$, then the optimal transport plan can be realized by a bijective map: the mass from each $x_i$ is sent to a unique $y_j$. This is impossible when $m \neq n$, however, a certain degree of rigidity prevails. We prove, assuming w.l.o.g. $m < n$, that for generic transport costs the optimal transport plan sends mass from each source $x_i$ to $n/m \leq \mbox{different targets} \leq n/m + m-1$. Moreover, the average target receives mass from $\leq 1 + m/\sqrt{n}$ sources. Stronger results might be true: in experiments, one observes that each source tends to distribute its mass over roughly $n/m +c$ different targets where $c$ appears to be rather small.


title: MLLMs-Augmented Visual-Language Representation Learning

abstract: Visual-language pre-training has achieved remarkable success in many multi-modal tasks, largely attributed to the availability of large-scale image-text datasets. In this work, we demonstrate that Multi-modal Large Language Models (MLLMs) can enhance visual-language representation learning by establishing richer image-text associations for image-text datasets. Our approach is simple, utilizing MLLMs to extend multiple diverse captions for each image. To prevent the bias introduced by MLLMs' hallucinations and monotonous language styles, we propose "text shearing" to maintain the quality and availability of extended captions. In image-text retrieval, without introducing additional training cost, our method consistently obtains 5.6 ~ 35.0 and 16.8 ~ 46.1 improvement on Recall@1 under the fine-tuning and zero-shot settings, respectively. Notably, we obtain zero-shot results that are comparable to fine-tuning on target datasets, which encourages more exploration of the versatile use of MLLMs.


title: Algebro-geometric proof of Christoph's Theorem

abstract: In this paper an algebraic proof of Christoph's theorem is provided. This theorem from algebraic-geometry is about the existence of a finite automaton for computing coefficient of a series for an algebraic function.


title: Universal Liouville action as a renormalized volume and its gradient flow

abstract: The universal Liouville action (also known as the Loewner energy for Jordan curves) is a K\"ahler potential on the Weil-Petersson universal Teichm\"uller space which is identified with the family of Weil-Petersson quasicircles via conformal welding. Our main result shows that, under regularity assumptions, the universal Liouville action equals the renormalized volume of the hyperbolic 3-manifold bounded by the two Epstein-Poincar\'e surfaces associated with the quasicircle. We also study the gradient descent flow of the universal Liouville action with respect to the Weil-Petersson metric and show that the flow always converges to the origin (the circle). This provides a bound of the Weil-Petersson distance to the origin by the universal Liouville action.


title: Evaluating the Impact of Flaky Simulators on Testing Autonomous Driving Systems

abstract: Simulators are widely used to test Autonomous Driving Systems (ADS), but their potential flakiness can lead to inconsistent test results. We investigate test flakiness in simulation-based testing of ADS by addressing two key questions: (1) How do flaky ADS simulations impact automated testing that relies on randomized algorithms? and (2) Can machine learning (ML) effectively identify flaky ADS tests while decreasing the required number of test reruns? Our empirical results, obtained from two widely-used open-source ADS simulators and five diverse ADS test setups, show that test flakiness in ADS is a common occurrence and can significantly impact the test results obtained by randomized algorithms. Further, our ML classifiers effectively identify flaky ADS tests using only a single test run, achieving F1-scores of $85$%, $82$% and $96$% for three different ADS test setups. Our classifiers significantly outperform our non-ML baseline, which requires executing tests at least twice, by $31$%, $21$%, and $13$% in F1-score performance, respectively. We conclude with a discussion on the scope, implications and limitations of our study. We provide our complete replication package in a Github repository.


title: Online Change Points Detection for Linear Dynamical Systems with Finite Sample Guarantees

abstract: The problem of online change point detection is to detect abrupt changes in properties of time series, ideally as soon as possible after those changes occur. Existing work on online change point detection either assumes i.i.d data, focuses on asymptotic analysis, does not present theoretical guarantees on the trade-off between detection accuracy and detection delay, or is only suitable for detecting single change points. In this work, we study the online change point detection problem for linear dynamical systems with unknown dynamics, where the data exhibits temporal correlations and the system could have multiple change points. We develop a data-dependent threshold that can be used in our test that allows one to achieve a pre-specified upper bound on the probability of making a false alarm. We further provide a finite-sample-based bound for the probability of detecting a change point. Our bound demonstrates how parameters used in our algorithm affect the detection probability and delay, and provides guidance on the minimum required time between changes to guarantee detection.


title: Effects of anisotropic pressure on interacting quark star structure

abstract: Perturbative Quantum Chromodynamics (pQCD) corrections and color superconductivity predict that strongly interacting matter can reveal new physical phenomena under extreme conditions. Taking into account these interaction effects, we investigate the role of anisotropic pressure in quark stars composed of interacting quark matter. Adopting two physically well-motivated anisotropy profiles, we numerically solve the stellar structure equations in order to explore the consequences of anisotropic pressure on various macroscopic properties such as radius, gravitational mass, surface redshift, moment of inertia, tidal Love number and oscillation spectrum. Remarkably, for both anisotropy models, negative anisotropies increase the radial stability of interacting quark stars, while the opposite occurs for positive anisotropies. However, for the Bowers-Liang profile, the central density corresponding to the maximum-mass point does not coincide with the central density where the squared oscillation frequency vanishes, indicating that the existence of stable anisotropic interacting quark stars is possible beyond the maximum mass for negative anisotropies. Additionally, we compare our theoretical predictions with several observational mass-radius measurements and tidal deformability constraints, which suggest that both strong interaction effects and anisotropy effects play a crucial role in describing compact stars observed in the Universe.


title: A dichotomy for central sequence algebras

abstract: We prove that the central sequence algebra of a separable C*-algebra is either subhomogeneous or non-exact, confirming a conjecture of Enders and Shulman. We also prove analogous dichotomy for other massive C*-algebras.


title: Experimental Study of the Game Exact Nim(5, 2)

abstract: We compare to different extensions of the ancient game of nim: Moore's nim$(n, \leq k)$ and exact nim$(n, = k)$. Given integers $n$ and $k$ such that $0 < k \leq n$, we consider $n$ piles of stones. Two players alternate turns. By one move it is allowed to choose and reduce any (i) at most $k$ or (ii) exactly $k$ piles of stones in games nim$(n, \leq k)$ and nim$(n, = k)$, respectively. The player who has to move but cannot is the loser. Both games coincide with nim when $k=1$. Game nim$(n, \leq k)$ was introduced by Moore (1910) who characterized its Sprague-Grundy (SG) values 0 (that is, P-positions) and 1. The first open case is SG values 2 for nim$(4, \leq 2)$. Game nim$(n, = k)$, was introduced in 2018. An explicit formula for its SG function was computed for $2k \geq n$. In contrast, case $2k < n$ seems difficult: even the P-positions are not known already for nim$(5,=2)$. Yet, it seems that the P-position of games nim$(n+1,=2)$ and nim$(n+1,\leq 2)$ are closely related. (Note that P-positions of the latter are known.) Here we provide some theoretical and computational evidence of such a relation for $n=5$.


title: Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding

abstract: Learning from videos is an emerging research area that enables robots to acquire skills from human demonstrations, such as procedural videos. To do this, video-language models must be able to obtain structured understandings, such as the temporal segmentation of a demonstration into sequences of actions and skills, and to generalize the understandings to novel domains. In pursuit of this goal, we introduce Spacewalk-18, a benchmark containing two tasks: (1) step recognition and (2) intra-video retrieval over a dataset of temporally segmented and labeled tasks in International Space Station spacewalk recordings. In tandem, the two tasks quantify a model's ability to make use of: (1) out-of-domain visual information; (2) a high temporal context window; and (3) multimodal (e.g. visual and speech) domains. This departs from existing benchmarks for procedural video understanding, which typically deal with short context lengths and can be solved with a single modality. Spacewalk-18, with its inherent multimodal and long-form complexity, exposes the high difficulty of task recognition and segmentation. We find that state-of-the-art methods perform poorly on our benchmark, but improvements can be obtained by incorporating information from longer-range temporal context across different modalities. Our experiments underscore the need to develop new approaches to these tasks. Data, model, and code will be released at https://brown-palm.github.io/Spacewalk-18/.


title: An Aliasing-Free Hybrid Digital-Analog Polyphonic Synthesizer

abstract: Analog subtractive synthesizers are generally considered to provide superior sound quality compared to digital emulations. However, analog circuitry requires calibration and suffers from aging, temperature instability, and limited flexibility in generating a wide variety of waveforms. Digital synthesis can mitigate many of these drawbacks, but generating arbitrary aliasing-free waveforms remains challenging. In this paper, we present the +-synth, a hybrid digital-analog eight-voice polyphonic synthesizer prototype that combines the best of both worlds. At the heart of the synthesizer is the big Fourier oscillator (BFO), a novel digital very-large scale integration (VLSI) design that utilizes additive synthesis to generate a wide variety of aliasing-free waveforms. Each BFO produces two voices, using four oscillators per voice. A single oscillator can generate up to 1024 freely configurable partials (harmonic or inharmonic), which are calculated using coordinate rotation digital computers (CORDICs). The BFOs were fabricated as 65nm CMOS custom application-specific integrated circuits (ASICs), which are integrated in the +-synth to simultaneously generate up to 32768 partials. Four 24-bit 96kHz stereo DACs then convert the eight voices into the analog domain, followed by digitally controlled analog low-pass filtering and amplification. Measurement results of the +-synth prototype demonstrate high fidelity and low latency.


title: CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation

abstract: We present CoDi-2, a versatile and interactive Multimodal Large Language Model (MLLM) that can follow complex multimodal interleaved instructions, conduct in-context learning (ICL), reason, chat, edit, etc., in an any-to-any input-output modality paradigm. By aligning modalities with language for both encoding and generation, CoDi-2 empowers Large Language Models (LLMs) to not only understand complex modality-interleaved instructions and in-context examples, but also autoregressively generate grounded and coherent multimodal outputs in the continuous feature space. To train CoDi-2, we build a large-scale generation dataset encompassing in-context multimodal instructions across text, vision, and audio. CoDi-2 demonstrates a wide range of zero-shot capabilities for multimodal generation, such as in-context learning, reasoning, and compositionality of any-to-any modality generation through multi-round interactive conversation. CoDi-2 surpasses previous domain-specific models on tasks such as subject-driven image generation, vision transformation, and audio editing. CoDi-2 signifies a substantial breakthrough in developing a comprehensive multimodal foundation model adept at interpreting in-context language-vision-audio interleaved instructions and producing multimodal outputs.


title: Powers of the operator $u(z)\frac{d}{dz}$ and their connection with the combinatorial numbers

abstract: In this paper the operator $A = u(z)\frac{d}{dz}$ is considered, where $u$ is an arbitrary entire function. The expansion of $A^{k}$ ($k\geq1$) with the help of powers of differential operators is obtained, and it is shown that this expansion depends on special numbers. A connection between these numbers and known combinatorial numbers is given.


title: Strict BV relaxed area of Sobolev maps into the circle: the high dimension case

abstract: We deal with the relaxed area functional in the strict $BV$-convergence of non-smooth maps defined in domains of generic dimension and taking values into the unit circle. In case of Sobolev maps, a complete explicit formula is obtained. Our proof is based on tools from Geometric Measure Theory and Cartesian currents. We then discuss the possible extension to the wider class of maps with bounded variation. Finally, we show a counterexample to the locality property in case of both dimension and codimension larger than two.


title: Mavericks at BLP-2023 Task 1: Ensemble-based Approach Using Language Models for Violence Inciting Text Detection

abstract: This paper presents our work for the Violence Inciting Text Detection shared task in the First Workshop on Bangla Language Processing. Social media has accelerated the propagation of hate and violence-inciting speech in society. It is essential to develop efficient mechanisms to detect and curb the propagation of such texts. The problem of detecting violence-inciting texts is further exacerbated in low-resource settings due to sparse research and less data. The data provided in the shared task consists of texts in the Bangla language, where each example is classified into one of the three categories defined based on the types of violence-inciting texts. We try and evaluate several BERT-based models, and then use an ensemble of the models as our final submission. Our submission is ranked 10th in the final leaderboard of the shared task with a macro F1 score of 0.737.


title: Compact K\"ahler manifolds with quasi-positive holomorphic sectional curvature

abstract: In this paper, we prove that if a compact complex manifold $M$ admits a K\"{a}hler metric $\omega$ with nonnegative holomorphic sectional curvature and $(M,\omega)$ has no nonzero truly flat tangent vector at some point (which is satisfied when the holomorphic sectional curvature is quasi-positive), then it holds that $H_{\overline{\partial} }^{p,0}(M)=0$ for every $1\leq p \leq\dim M$. As a corollary, $M$ is projective and rationally connected. This answers a problem raised up by Matsumura and Yang. Furthermore, we show that a compact K\"{a}hler manifold admits a Hermitian metric with positive real bisectional curvature is a projective and rationally connected manifold.


title: MultiResFormer: Transformer with Adaptive Multi-Resolution Modeling for General Time Series Forecasting

abstract: Transformer-based models have greatly pushed the boundaries of time series forecasting recently. Existing methods typically encode time series data into $\textit{patches}$ using one or a fixed set of patch lengths. This, however, could result in a lack of ability to capture the variety of intricate temporal dependencies present in real-world multi-periodic time series. In this paper, we propose MultiResFormer, which dynamically models temporal variations by adaptively choosing optimal patch lengths. Concretely, at the beginning of each layer, time series data is encoded into several parallel branches, each using a detected periodicity, before going through the transformer encoder block. We conduct extensive evaluations on long- and short-term forecasting datasets comparing MultiResFormer with state-of-the-art baselines. MultiResFormer outperforms patch-based Transformer baselines on long-term forecasting tasks and also consistently outperforms CNN baselines by a large margin, while using much fewer parameters than these baselines.


title: Displayed Type Theory and Semi-Simplicial Types

abstract: We introduce Displayed Type Theory (dTT), a multi-modal homotopy type theory with discrete and simplicial modes. In the intended semantics, the discrete mode is interpreted by a model for an arbitrary $\infty$-topos, while the simplicial mode is interpreted by Reedy fibrant augmented semi-simplicial diagrams in that model. This simplicial structure is represented inside the theory by a primitive notion of display or dependency, guarded by modalities, yielding a partially-internal form of unary parametricity. Using the display primitive, we then give a coinductive definition, at the simplicial mode, of a type $\mathsf{SST}$ of semi-simplicial types. Roughly speaking, a semi-simplicial type $X$ consists of a type $X_0$ together with, for each $x:X_0$, a displayed semi-simplicial type over $X$. This mimics how simplices can be generated geometrically through repeated cones, and is made possible by the display primitive at the simplicial mode. The discrete part of $\mathsf{SST}$ then yields the usual infinite indexed definition of semi-simplicial types, both semantically and syntactically. Thus, dTT enables working with semi-simplicial types in full semantic generality.


title: Duality and Stacking of Bosonic and Fermionic SPT Phases

abstract: We study the interplay of duality and stacking of bosonic and fermionic symmetry-protected topological phases in one spatial dimension. In general the classifications of bosonic and fermionic phases have different group structures under the operation of stacking, but we argue that they are often isomorphic and give an explicit isomorphism when it exists. This occurs for all unitary symmetry groups and many groups with antiunitary symmetries, which we characterize. We find that this isomorphism is typically not implemented by the Jordan-Wigner transformation, nor is it a consequence of any other duality transformation that falls within the framework of topological holography. Along the way to this conclusion, we recover the fermionic stacking rule in terms of G-pin partition functions, give a gauge-invariant characterization of the twisted group cohomology invariant, and state a procedure for stacking gapped phases in the formalism of symmetry topological field theory.


title: A robust and adaptive GenEO-type domain decomposition preconditioner for $\mathbf{H}(\mathbf{curl})$ problems in general non-convex three-dimensional geometries

abstract: In this paper we develop and analyse domain decomposition methods for linear systems of equations arising from conforming finite element discretisations of positive Maxwell-type equations, namely for $\mathbf{H}(\mathbf{curl})$ problems. It is well known that convergence of domain decomposition methods rely heavily on the efficiency of the coarse space used in the second level. We design adaptive coarse spaces that complement a near-kernel space made from the gradient of scalar functions. The new class of preconditioner is inspired by the idea of subspace decomposition, but based on spectral coarse spaces, and is specially designed for curl-conforming discretisations of Maxwell's equations in heterogeneous media on general domains which may have holes. Our approach has wider applicability and theoretical justification than the well-known Hiptmair-Xu auxiliary space preconditioner, with results extending to the variable coefficient case and non-convex domains at the expense of a larger coarse space.


title: Non-equilibrium dynamics of electron emission from cold and hot graphene under proton irradiation

abstract: Characteristic properties of secondary electrons emitted from irradiated two-dimensional materials arise from multi-length and time-scale relaxation processes that connect the initial non-equilibrium excited electron distribution with their eventual emission. To understand these processes, which are critical for using secondary electrons as high-resolution thermalization probes, we combine first-principles real-time electron dynamics with modern experiments. Our data for cold and hot proton-irradiated graphene shows signatures of kinetic and potential emission and generally good agreement for electron yields between experiment and theory. The duration of the emission pulse is about 1.5 femtoseconds, indicating high time resolution when used as a probe. Our newly developed method to predict kinetic energy spectra shows good agreement with electron and ion irradiation experiments and prior models. We find that lattice temperature significantly increases secondary electron emission, whereas electron temperature has a negligible effect.


title: Tensor network renormalization: application to dynamic correlation functions and non-hermitian systems

abstract: In recent years, tensor network renormalization (TNR) has emerged as an efficient and accurate method for studying (1+1)D quantum systems or 2D classical systems using real-space renormalization group (RG) techniques. One notable application of TNR is its ability to extract central charge and conformal scaling dimensions for critical systems. In this paper, we present the implementation of the Loop-TNR algorithm, which allows for the computation of dynamical correlation functions. Our algorithm goes beyond traditional approaches by not only calculating correlations in the spatial direction, where the separation is an integer, but also in the temporal direction, where the time difference can contain decimal values. Our algorithm is designed to handle both imaginary-time and real-time correlations, utilizing a tensor network representation constructed from a path-integral formalism. Additionally, we highlight that the Loop-TNR algorithm can also be applied to investigate critical properties of non-Hermitian systems, an area that was previously inaccessible using density matrix renormalization group(DMRG) and matrix product state(MPS) based algorithms.


title: Slow graph bootstrap percolation II: Accelerating properties

abstract: For a graph $H$ and an $n$-vertex graph $G$, the $H$-bootstrap process on $G$ is the process which starts with $G$ and, at every time step, adds any missing edges on the vertices of $G$ that complete a copy of $H$. This process eventually stabilises and we are interested in the extremal question raised by Bollob\'as, of determining the maximum running time (number of time steps before stabilising) of this process, over all possible choices of $n$-vertex graph $G$. In this paper, we initiate a systematic study of the asymptotics of this parameter, denoted $M_H(n)$, and its dependence on properties of the graph $H$. Our focus is on $H$ which define relatively fast bootstrap processes, that is, with $M_H(n)$ being at most linear in $n$. We study the graph class of trees, showing that one can bound $M_T(n)$ by a quadratic function in $v(T)$ for all trees $T$ and all $n$. We then go on to explore the relationship between the running time of the $H$-process and the minimum vertex degree and connectivity of $H$.


title: Communication-Efficient Federated Optimization over Semi-Decentralized Networks

abstract: In large-scale federated and decentralized learning, communication efficiency is one of the most challenging bottlenecks. While gossip communication -- where agents can exchange information with their connected neighbors -- is more cost-effective than communicating with the remote server, it often requires a greater number of communication rounds, especially for large and sparse networks. To tackle the trade-off, we examine the communication efficiency under a semi-decentralized communication protocol, in which agents can perform both agent-to-agent and agent-to-server communication in a probabilistic manner. We design a tailored communication-efficient algorithm over semi-decentralized networks, referred to as PISCO, which inherits the robustness to data heterogeneity thanks to gradient tracking and allows multiple local updates for saving communication. We establish the convergence rate of PISCO for nonconvex problems and show that PISCO enjoys a linear speedup in terms of the number of agents and local updates. Our numerical results highlight the superior communication efficiency of PISCO and its resilience to data heterogeneity and various network topologies.


title: Automated interpretation of congenital heart disease from multi-view echocardiograms

abstract: Congenital heart disease (CHD) is the most common birth defect and the leading cause of neonate death in China. Clinical diagnosis can be based on the selected 2D key-frames from five views. Limited by the availability of multi-view data, most methods have to rely on the insufficient single view analysis. This study proposes to automatically analyze the multi-view echocardiograms with a practical end-to-end framework. We collect the five-view echocardiograms video records of 1308 subjects (including normal controls, ventricular septal defect (VSD) patients and atrial septal defect (ASD) patients) with both disease labels and standard-view key-frame labels. Depthwise separable convolution-based multi-channel networks are adopted to largely reduce the network parameters. We also approach the imbalanced class problem by augmenting the positive training samples. Our 2D key-frame model can diagnose CHD or negative samples with an accuracy of 95.4\%, and in negative, VSD or ASD classification with an accuracy of 92.3\%. To further alleviate the work of key-frame selection in real-world implementation, we propose an adaptive soft attention scheme to directly explore the raw video data. Four kinds of neural aggregation methods are systematically investigated to fuse the information of an arbitrary number of frames in a video. Moreover, with a view detection module, the system can work without the view records. Our video-based model can diagnose with an accuracy of 93.9\% (binary classification), and 92.1\% (3-class classification) in a collected 2D video testing set, which does not need key-frame selection and view annotation in testing. The detailed ablation study and the interpretability analysis are provided.

categories: eess.IV cs.AI cs.CV cs.MM physics.med-ph


title: Unsupervised learning architecture based on neural Darwinism and Hopfield networks recognizes symbols with high accuracy

abstract: This paper introduces a novel unsupervised learning paradigm inspired by Gerald Edelman's theory of neuronal group selection ("Neural Darwinism"). The presented automaton learns to recognize arbitrary symbols (e.g., letters of an alphabet) when they are presented repeatedly, as they are when children learn to read. On a second hierarchical level, the model creates abstract categories representing the learnt symbols. The fundamental computational unit are simple McCulloch-Pitts neurons arranged into fully-connected groups (Hopfield networks with randomly initialized weights), which are "selected", in an evolutionary sense, through symbol presentation. The learning process is fully tractable and easily interpretable for humans, in contrast to most neural network architectures. Computational properties of Hopfield networks enabling pattern recognition are discussed. In simulations, the model achieves high accuracy in learning the letters of the Latin alphabet, presented as binary patterns on a grid. This paper is a proof of concept with no claims to state-of-the-art performance in letter recognition, but hopefully inspires new thinking in bio-inspired machine learning.


title: Composition operators on the algebra of Dirichlet series

abstract: The algebra of Dirichlet series $\mathcal{A}(\mathcal{C}_{+})$ consists on those Dirichlet series convergent in the right half-plane $\mathcal{C}_{+}$ and which are also uniformly continuous there. This algebra was recently introduced by Aron, Bayart, Gauthier, Maestre, and Nestoridis. We describe the symbols $\Phi:\mathcal{C}_{+}\to\mathcal{C}_{+}$ giving rise to bounded composition operators $\mathit{C}_{\Phi}$ in $\mathcal{A}(\mathcal{C}_{+})$ and denote this class by $\mathcal{G}_{\mathcal{A}}$. We also characterise when the operator $\mathit{C}_{\Phi}$ is compact in $\mathcal{A}(\mathit{C}_{+})$. As a byproduct, we show that the weak compactness is equivalent to the compactness for $\mathit{C}_{\Phi}$. Next, the closure under the local uniform convergence of several classes of symbols of composition operators in Banach spaces of Dirichlet series is discussed. We also establish a one-to-one correspondence between continuous semigroups of analytic functions $\{\Phi_{t}\}$ in the class $\mathcal{G}_{\mathcal{A}}$ and strongly continuous semigroups of composition operators $\{T_{t}\}$, $T_{t}f=f\circ\Phi_{t}$, $f\in\mathcal{A}(\mathcal{C}_{+})$. We conclude providing examples showing the differences between the symbols of bounded composition operators in $\mathcal{A}(\mathcal{C}_{+})$ and the Hardy spaces of Dirichlet series $\mathcal{H}^{p}$ and $\mathcal{H}^{\infty}$.


title: Minimizing Weighted Sum Age of Information with Open-Loop Cyclic Scheduling

abstract: We study the scheduling problem in a status update system composed of an arbitrary number of information sources with different service time distributions and weights for the purpose of minimizing the weighted sum age of information (AoI). In particular, we study open-loop schedulers which rely only on the statistics (specifically, only on the first two moments) of the source service times, in contrast to closed-loop schedulers that also make use of the actual realizations of the service times and the AoI processes in making scheduling decisions. Open-loop scheduling policies can be constructed off-line and are simpler to implement compared to their closed-loop counterparts. We consider the generate-at-will (GAW) model, and develop an analytical method to calculate the exact AoI for the probabilistic and cyclic open-loop schedulers. In both cases, the server initiates the sampling of a source and the ensuing transmission of the update packet from the source to the server in an open-loop manner; either based on a certain probability (probabilistic scheme) or according to a deterministic cyclic pattern (cyclic scheme). We derive the optimum open-loop cyclic scheduling policy in closed form for the specific case of N=2 sources and propose well-performing heuristic cyclic schedulers for general number of sources, i.e., N>2. We study the proposed cyclic schedulers against probabilistic schedulers and several existing methods in the literature to validate their effectiveness.


title: Resource Sharing in Energy Communities: A Cooperative Game Approach

abstract: We analyze the overall benefits of an energy community cooperative game under which distributed energy resources (DER) are shared behind a regulated distribution utility meter under a general net energy metering (NEM) tariff. Two community DER scheduling algorithms are examined. The first is a community with centrally controlled DER, whereas the second is decentralized letting its members schedule their own DER locally. For both communities, we prove that the cooperative game's value function is superadditive, hence the grand coalition achieves the highest welfare. We also prove the balancedness of the cooperative game under the two DER scheduling algorithms, which means that there is a welfare re-distribution scheme that de-incentivizes players from leaving the grand coalition to form smaller ones. Lastly, we present five ex-post and an ex-ante welfare re-distribution mechanisms and evaluate them in simulation, in addition to investigating the performance of various community sizes under the two DER scheduling algorithms.


title: Finite-volume scattering on the left-hand cut

abstract: The two-particle finite-volume scattering formalism derived by L\"uscher and generalized in many subsequent works does not hold for energies far enough below the two-particle threshold to reach the nearest left-hand cut. The breakdown of the formalism is signaled by the fact that a real scattering amplitude is predicted in a regime where it should be complex. In this work, we address this limitation by deriving an extended formalism that includes the nearest branch cut, arising from single particle exchange. We focus on two-nucleon ($NN \to NN$) scattering, for which the cut arises from pion exchange, but give expressions for any system with a single channel of identical particles. The new result takes the form of a modified quantization condition that can be used to constrain an intermediate K-matrix in which the cut is removed. In a second step, integral equations, also derived in this work, must be used to convert the K-matrix to the physical scattering amplitude. We also show how the new formalism reduces to the standard approach when the $N \to N \pi$ coupling is set to zero.


title: Electrobaryonic axion: hair of neutron stars

abstract: Axion-like particles are predicted in many physics scenarios beyond the Standard Model (SM). Their interactions with SM particles may arise from the triangle anomaly of the associated global symmetry, along with other SM global and gauge symmetries, including anomalies with the global baryon number and electromagnetic gauge symmetries. We initiate the phenomenological study of the corresponding ``electrobaryonic axion", a particle that couples with both the baryon chemical potential and the electromagnetic field. Neutron stars, particularly magnetars, possessing high baryon density and strong magnetic fields, can naturally develop a thin axion hair around their surface. In this study, we calculate this phenomenon, considering the effects of neutron star rotation and general relativity. For axion particles lighter than the neutron star rotation frequency, the anomalous interaction can also induce the emission of axion particles from the neutron star. This emission, in the light axion regime, can have a significant contribution to the neutron star cooling rate.


title: Supersonic Gravitational Collapse for Non-Isentropic Gaseous Stars

abstract: We show the existence of a new class of initially smooth spherically symmetric self-similar solutions to the non-isentropic Euler-Poisson system. These solutions exhibit supersonic gravitational implosion in the sense that the density blows-up in finite time while the fluid velocity remains supersonic. In particular, they occupy a portion of the phase space that is far from the recently constructed isentropic self-similar implosion. At the heart of our proof is the presence of a two-parameter scaling invariance and the reduction of the problem to a non-autonomous system of ordinary differential equations. We use the requirement of smoothness of the flow as a selection principle that constrains the choice of scaling indices. An important consequence of our analysis is that for all the solutions we construct, the polytropic index $\gamma$ is strictly bigger than $\frac{4}{3}$, which is in sharp contrast to the known results in the isentropic case.


title: On product Schur triples in the integers

abstract: Schur's theorem states that in any $k$-colouring of the set of integers $[n]$ there is a monochromatic solution to $a+b=c$, provided $n$ is sufficiently large. Abbott and Wang studied the size of the largest subset of $[n]$ such that there is a $k$-colouring avoiding a monochromatic $a+b=c$. In other directions, the minimum number of $a+b=c$ in $k$-colourings of $[n]$ and the probability threshold in random subsets of $[n]$ for the property of having a monochromatic $a+b=c$ in any $k$-colouring were investigated. In this paper, we study natural generalisations of these streams to products $ab=c$, in a deterministic, random, and randomly perturbed environments.


title: $\epsilon$-Uniform Mixing in Discrete Quantum Walks

abstract: We study whether a discrete quantum walk can get arbitrarily close to a state whose entries have the same absolute value over all the arcs, given that the walk starts with a uniform superposition of the outgoing arcs of some vertex. We characterize this phenomenon on non-bipartite graphs using the adjacency spectrum of the graph; in particular, if this happens in some association scheme and the state we get arbitrarily close to ``respects the neighborhood", then it happens regardless of the initial vertex, and the adjacency algebra of the graph contains a real (regular) Hadamard matrix. We then find infinite families of primitive strongly regular graphs that admit this phenomenon. We also derive some results on a strengthening of this phenomenon called simultaneous $\epsilon$-uniform mixing, which enables local $\epsilon$-uniform mixing at every vertex.


title: A lower bound for the discrepancy in a Sato-Tate type measure

abstract: Let $S_k(N)$ denote the space of cusp forms of even integer weight $k$ and level $N$. We prove an asymptotic for the Petersson trace formula for $S_k(N)$ under an appropriate condition. Using the non-vanishing of a Kloosterman sum involved in the asymptotic, we give a lower bound for discrepancy in the Sato-Tate distribution for levels not divisible by $8$. This generalizes a result of Jung and Sardari for squarefree levels. An analogue of the Sato-Tate distribution was obtained by Omar and Mazhouda for the distribution of eigenvalues $\lambda_{p^2}(f)$ where $f$ is a Hecke eigenform and $p$ is a prime number. As an application of the above-mentioned asymptotic, we obtain a sequence of weights $k_n$ such that discrepancy in the analogue distribution obtained by Omar and Mazhouda has a lower bound.


title: X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning

abstract: Vision-language pre-training and instruction tuning have demonstrated general-purpose capabilities in 2D visual reasoning tasks by aligning visual encoders with state-of-the-art large language models (LLMs). In this paper, we introduce a simple, yet effective, cross-modality framework built atop frozen LLMs that allows the integration of various modalities without extensive modality-specific customization. To facilitate instruction-modality fine-tuning, we collect high-quality instruction tuning data in an automatic and scalable manner, composed of 24K QA samples for audio and 250K QA samples for 3D. Leveraging instruction-aware representations, our model performs comparably with leading-edge counterparts without the need of extensive modality-specific pre-training or customization. Furthermore, our approach demonstrates cross-modal reasoning abilities across two or more input modalities, despite each modality projection being trained individually. To study the model's cross-modal abilities, we contribute a novel Discriminative Cross-modal Reasoning (DisCRn) evaluation task, comprising 9K audio-video QA samples and 28K image-3D QA samples that require the model to reason discriminatively across disparate input modalities.


title: Differential Rotation of the Solar Chromosphere: A Century-long Perspective from Kodaikanal Solar Observatory Ca II K Data

abstract: Chromospheric differential rotation is a key component in comprehending the atmospheric coupling between the chromosphere and the photosphere at different phases of the solar cycle. In this study, we therefore utilize the newly calibrated multidecadal Ca II K spectroheliograms (1907-2007) from the Kodaikanal Solar Observatory (KoSO) to investigate the differential rotation of the solar chromosphere using the technique of image cross-correlation. Our analysis yields the chromospheric differential rotation rate $\Omega (\theta) = (14.61\pm 0.04 - 2.18\pm 0.37\sin^2{\theta} - 1.10 \pm 0.61\sin^4{\theta})^\circ{\rm /day}$. These results suggest the chromospheric plages exhibit an equatorial rotation rate 1.59% faster than the photosphere when compared with the differential rotation rate measured using sunspots and also a smaller latitudinal gradient compared to the same. To compare our results to those from other observatories, we have applied our method on a small sample of Ca II K data from Rome, Meudon, and Mt. Wilson observatories, which support our findings from KoSO data. Additionally, we have not found any significant north-south asymmetry or any systematic variation in chromospheric differential rotation over the last century.


title: Distributed Global Structure-from-Motion with a Deep Front-End

abstract: While initial approaches to Structure-from-Motion (SfM) revolved around both global and incremental methods, most recent applications rely on incremental systems to estimate camera poses due to their superior robustness. Though there has been tremendous progress in SfM `front-ends' powered by deep models learned from data, the state-of-the-art (incremental) SfM pipelines still rely on classical SIFT features, developed in 2004. In this work, we investigate whether leveraging the developments in feature extraction and matching helps global SfM perform on par with the SOTA incremental SfM approach (COLMAP). To do so, we design a modular SfM framework that allows us to easily combine developments in different stages of the SfM pipeline. Our experiments show that while developments in deep-learning based two-view correspondence estimation do translate to improvements in point density for scenes reconstructed with global SfM, none of them outperform SIFT when comparing with incremental SfM results on a range of datasets. Our SfM system is designed from the ground up to leverage distributed computation, enabling us to parallelize computation on multiple machines and scale to large scenes.


title: GYOTO 2.0: a polarized relativistic ray-tracing code

abstract: Polarized general-relativistic radiative transfer in the vicinity of black holes and other compact objects has become a crucial tool for probing the properties of relativistic astrophysics plasmas. Instruments like GRAVITY, the Event Horizon telescope, ALMA, or IXPE make it very timely to develop such numerical frameworks. In this article, we present the polarized extension of the public ray-tracing code Gyoto, and offer a python notebook allowing to easily perform a first realistic computation. The code is very modular and allows to conveniently add extensions for the specific needs of the user. It is agnostic about the spacetime and can be used for arbitrary compact objects. We demonstrate the validity of the code by providing tests, and show in particular a perfect agreement with the ipole code. Our article also aims at pedagogically introducing all the relevant formalism in a self-contained manner.


title: BioCLIP: A Vision Foundation Model for the Tree of Life

abstract: Images of the natural world, collected by a variety of cameras, from drones to individual phones, are increasingly abundant sources of biological information. There is an explosion of computational methods and tools, particularly computer vision, for extracting biologically relevant information from images for science and conservation. Yet most of these are bespoke approaches designed for a specific task and are not easily adaptable or extendable to new questions, contexts, and datasets. A vision model for general organismal biology questions on images is of timely need. To approach this, we curate and release TreeOfLife-10M, the largest and most diverse ML-ready dataset of biology images. We then develop BioCLIP, a foundation model for the tree of life, leveraging the unique properties of biology captured by TreeOfLife-10M, namely the abundance and variety of images of plants, animals, and fungi, together with the availability of rich structured biological knowledge. We rigorously benchmark our approach on diverse fine-grained biology classification tasks and find that BioCLIP consistently and substantially outperforms existing baselines (by 16% to 17% absolute). Intrinsic evaluation reveals that BioCLIP has learned a hierarchical representation conforming to the tree of life, shedding light on its strong generalizability. https://imageomics.github.io/bioclip has models, data and code.


title: Holographic complexity of the Klebanov-Strassler background

abstract: We study the complexity of the gravity dual to the confining $SU(N)\times SU(N+M)$ Klebanov-Strassler gauge theory, which is an important test case for holographic complexity in higher-dimensional and nonconformal gauge/gravity dualities. We emphasize the dependence of the complexity on parameters of the gauge theory, finding a common behavior with confinement scale for several complexity functionals. We also analyze how the complexity diverges with the UV cut off, which is more complicated than in AdS backgrounds because the theory is nonconformal. Our results may provide new perspectives on questions in the holographic complexity program as well as a starting point for further studies of complexity in general gauge/gravity dualities.


title: Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Text

abstract: While Large Language Models (LLMs) have achieved remarkable performance in many tasks, much about their inner workings remains unclear. In this study, we present novel experimental insights into the resilience of LLMs, particularly GPT-4, when subjected to extensive character-level permutations. To investigate this, we first propose the Scrambled Bench, a suite designed to measure the capacity of LLMs to handle scrambled input, in terms of both recovering scrambled sentences and answering questions given scrambled context. The experimental results indicate that most powerful LLMs demonstrate the capability akin to typoglycemia, a phenomenon where humans can understand the meaning of words even when the letters within those words are scrambled, as long as the first and last letters remain in place. More surprisingly, we found that only GPT-4 nearly flawlessly processes inputs with unnatural errors, even under the extreme condition, a task that poses significant challenges for other LLMs and often even for humans. Specifically, GPT-4 can almost perfectly reconstruct the original sentences from scrambled ones, decreasing the edit distance by 95%, even when all letters within each word are entirely scrambled. It is counter-intuitive that LLMs can exhibit such resilience despite severe disruption to input tokenization caused by scrambled text.


title: Efficient Baseline for Quantitative Precipitation Forecasting in Weather4cast 2023

abstract: Accurate precipitation forecasting is indispensable for informed decision-making across various industries. However, the computational demands of current models raise environmental concerns. We address the critical need for accurate precipitation forecasting while considering the environmental impact of computational resources and propose a minimalist U-Net architecture to be used as a baseline for future weather forecasting initiatives.


title: Pre-registration for Predictive Modeling

abstract: Amid rising concerns of reproducibility and generalizability in predictive modeling, we explore the possibility and potential benefits of introducing pre-registration to the field. Despite notable advancements in predictive modeling, spanning core machine learning tasks to various scientific applications, challenges such as overlooked contextual factors, data-dependent decision-making, and unintentional re-use of test data have raised questions about the integrity of results. To address these issues, we propose adapting pre-registration practices from explanatory modeling to predictive modeling. We discuss current best practices in predictive modeling and their limitations, introduce a lightweight pre-registration template, and present a qualitative study with machine learning researchers to gain insight into the effectiveness of pre-registration in preventing biased estimates and promoting more reliable research outcomes. We conclude by exploring the scope of problems that pre-registration can address in predictive modeling and acknowledging its limitations within this context.


title: Prismatic decompositions and rational $G$-spectra

abstract: We study the tensor-triangular geometry of the category of rational $G$-spectra for a compact Lie group $G$. In particular, we prove that this category can be naturally decomposed into local factors supported on individual subgroups, each of which admits an algebraic model. This is an important step and strong evidence towards the third author's conjecture that the category of rational $G$-spectra admits an algebraic model for all compact Lie groups. To facilitate these results, we relate topological properties of the associated Balmer spectrum to structural features of the group $G$ and the category of rational $G$-spectra. A key ingredient is our presentation of the spectrum as a Priestley space, separating the Hausdorff topology on conjugacy classes of closed subgroups of $G$ from the cotoral ordering. We use this to prove that the telescope conjecture holds in general for rational $G$-spectra, and we determine exactly when the Balmer spectrum is Noetherian. In order to construct the desired decomposition of the category, we develop a general theory of `prismatic decompositions' of rigidly-compactly generated tensor-triangulated categories, which in favourable cases gives a series of recollements for reconstructing the category from local factors over individual points of the spectrum.


title: FoundPose: Unseen Object Pose Estimation with Foundation Features

abstract: We propose FoundPose, a method for 6D pose estimation of unseen rigid objects from a single RGB image. The method assumes that 3D models of the objects are available but does not require any object-specific training. This is achieved by building upon DINOv2, a recent vision foundation model with impressive generalization capabilities. An online pose estimation stage is supported by a minimal object representation that is built during a short onboarding stage from DINOv2 patch features extracted from rendered object templates. Given a query image with an object segmentation mask, FoundPose first rapidly retrieves a handful of similarly looking templates by a DINOv2-based bag-of-words approach. Pose hypotheses are then generated from 2D-3D correspondences established by matching DINOv2 patch features between the query image and a retrieved template, and finally optimized by featuremetric refinement. The method can handle diverse objects, including challenging ones with symmetries and without any texture, and noticeably outperforms existing RGB methods for coarse pose estimation in both accuracy and speed on the standard BOP benchmark. With the featuremetric and additional MegaPose refinement, which are demonstrated complementary, the method outperforms all RGB competitors. Source code is at: evinpinar.github.io/foundpose.


title: Convergence of Nonconvex PnP-ADMM with MMSE Denoisers

abstract: Plug-and-Play Alternating Direction Method of Multipliers (PnP-ADMM) is a widely-used algorithm for solving inverse problems by integrating physical measurement models and convolutional neural network (CNN) priors. PnP-ADMM has been theoretically proven to converge for convex data-fidelity terms and nonexpansive CNNs. It has however been observed that PnP-ADMM often empirically converges even for expansive CNNs. This paper presents a theoretical explanation for the observed stability of PnP-ADMM based on the interpretation of the CNN prior as a minimum mean-squared error (MMSE) denoiser. Our explanation parallels a similar argument recently made for the iterative shrinkage/thresholding algorithm variant of PnP (PnP-ISTA) and relies on the connection between MMSE denoisers and proximal operators. We also numerically evaluate the performance gap between PnP-ADMM using a nonexpansive DnCNN denoiser and expansive DRUNet denoiser, thus motivating the use of expansive CNNs.


title: Glauber Phases in Non-Global LHC Observables: Resummation for Gluon-Initiated Processes

abstract: The resummation of the ''Glauber series'' in non-global LHC observables is extended to processes with gluons in the initial state. This series simultaneously incorporates large double-logarithmic corrections, the so-called ''super-leading logarithms'', together with higher-order exchanges of pairs of Glauber gluons associated with the large numerical factor $(i\pi)^2$. On a technical level, the main part of this work is devoted to the systematic reduction of the appearing color traces and construction of basis structures, which consist of thirteen elements for $gg$ and eleven elements for $qg$ scattering. Numerical estimates for wide-angle gap-between-jet cross sections at the parton level show that, in particular for $gg$ scattering at relatively small vetoes $Q_0$, the contribution involving four Glauber exchanges gives a sizeable correction and should not be neglected.


title: What Do Llamas Really Think? Revealing Preference Biases in Language Model Representations

abstract: Do large language models (LLMs) exhibit sociodemographic biases, even when they decline to respond? To bypass their refusal to "speak," we study this research question by probing contextualized embeddings and exploring whether this bias is encoded in its latent representations. We propose a logistic Bradley-Terry probe which predicts word pair preferences of LLMs from the words' hidden vectors. We first validate our probe on three pair preference tasks and thirteen LLMs, where we outperform the word embedding association test (WEAT), a standard approach in testing for implicit association, by a relative 27% in error rate. We also find that word pair preferences are best represented in the middle layers. Next, we transfer probes trained on harmless tasks (e.g., pick the larger number) to controversial ones (compare ethnicities) to examine biases in nationality, politics, religion, and gender. We observe substantial bias for all target classes: for instance, the Mistral model implicitly prefers Europe to Africa, Christianity to Judaism, and left-wing to right-wing politics, despite declining to answer. This suggests that instruction fine-tuning does not necessarily debias contextualized embeddings. Our codebase is at https://github.com/castorini/biasprobe.


title: Optimal switching strategies for navigation in stochastic settings

abstract: Inspired by the intermittent reorientation strategy seen in the behavior of the dung beetle, we consider the problem of the navigation strategy of an active Brownian particle moving in two dimensions. We assume that the heading of the particle can be reoriented to the preferred direction by paying a fixed cost as it tries to maximize its total displacement in a fixed direction. Using optimal control theory, we derive analytically and confirm numerically the strategy that maximizes the particle speed, and show that the average time between reorientations scales inversely with the magnitude of the environmental noise. We then extend our framework to describe execution errors and sensory acquisition noise. Our approach may be amenable to other navigation problems involving multiple sensory modalities that require switching between egocentric and geocentric strategies.


title: Is Underwater Image Enhancement All Object Detectors Need?

abstract: Underwater object detection is a crucial and challenging problem in marine engineering and aquatic robot. The difficulty is partly because of the degradation of underwater images caused by light selective absorption and scattering. Intuitively, enhancing underwater images can benefit high-level applications like underwater object detection. However, it is still unclear whether all object detectors need underwater image enhancement as pre-processing. We therefore pose the questions "Does underwater image enhancement really improve underwater object detection?" and "How does underwater image enhancement contribute to underwater object detection?". With these two questions, we conduct extensive studies. Specifically, we use 18 state-of-the-art underwater image enhancement algorithms, covering traditional, CNN-based, and GAN-based algorithms, to pre-process underwater object detection data. Then, we retrain 7 popular deep learning-based object detectors using the corresponding results enhanced by different algorithms, obtaining 126 underwater object detection models. Coupled with 7 object detection models retrained using raw underwater images, we employ these 133 models to comprehensively analyze the effect of underwater image enhancement on underwater object detection. We expect this study can provide sufficient exploration to answer the aforementioned questions and draw more attention of the community to the joint problem of underwater image enhancement and underwater object detection. The pre-trained models and results are publicly available and will be regularly updated. Project page: https://github.com/BIGWangYuDong/lqit/tree/main/configs/detection/uw_enhancement_affect_detection.


title: IMMA: Immunizing text-to-image Models against Malicious Adaptation

abstract: Advancements in text-to-image models and fine-tuning methods have led to the increasing risk of malicious adaptation, i.e., fine-tuning to generate harmful unauthorized content. Recent works, e.g., Glaze or MIST, have developed data-poisoning techniques which protect the data against adaptation methods. In this work, we consider an alternative paradigm for protection. We propose to ``immunize'' the model by learning model parameters that are difficult for the adaptation methods when fine-tuning malicious content; in short IMMA. Empirical results show IMMA's effectiveness against malicious adaptations, including mimicking the artistic style and learning of inappropriate/unauthorized content, over three adaptation methods: LoRA, Textual-Inversion, and DreamBooth.


title: Family of Approximations for Dirichlet $L$-functions

abstract: We introduce an infinite family of approximations for a Dirichlet $L$-function $L(s, \chi)$ arising from truncated Euler products. These approximations are entire functions and satisfy the same functional equation as $L(s, \chi)$. We provide numerical evidence of the accuracy of estimating values of $L(s, \chi)$ in the critical strip using these approximations. We then provide a precise expression for the error of the approximations and show that the error has exponential decay in largest prime considered in the truncated Euler product.


title: Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking

abstract: Recent work by Power et al. (2022) highlighted a surprising "grokking" phenomenon in learning arithmetic tasks: a neural net first "memorizes" the training set, resulting in perfect training accuracy but near-random test accuracy, and after training for sufficiently longer, it suddenly transitions to perfect test accuracy. This paper studies the grokking phenomenon in theoretical setups and shows that it can be induced by a dichotomy of early and late phase implicit biases. Specifically, when training homogeneous neural nets with large initialization and small weight decay on both classification and regression tasks, we prove that the training process gets trapped at a solution corresponding to a kernel predictor for a long time, and then a very sharp transition to min-norm/max-margin predictors occurs, leading to a dramatic change in test accuracy.


title: Orthosymplectic Yangians

abstract: We study the RTT orthosymplectic super Yangians and present their Drinfeld realizations for any parity sequence, generalizing the results of for non-super types BCD, a standard parity sequence, and super A-type.


title: Structure in the speed of sound: from neutron stars to heavy-ion collisions

abstract: From the observation of both heavy neutron stars and light ones with small radii, one anticipates a steep rise in the speed of sound of nuclear matter as a function of baryon density up to values close to the causal limit. A question follows whether such behavior of the speed of sound in neutron-rich matter is compatible with the equation of state extracted from low-energy heavy-ion collisions. In this work, we consider a family of neutron star equations of state characterized by a steep rise in the speed of sound, and use the symmetry energy expansion to obtain equations of state applicable to the almost-symmetric nuclear matter created in heavy-ion collisions. We then compare collective flow data from low-energy heavy-ion experiments with results of simulations obtained using the hadronic transport code SMASH with the mean-field potential reproducing the density-dependence of the speed of sound. We show that equations of state featuring a peak in the speed of sound squared occurring at densities between 2-3 times the saturation density of normal nuclear matter, producing neutron stars of nearly M_max~2.5 M_Sun, are consistent with heavy-ion collision data.


title: Adversarial Attacks and Defenses for Wireless Signal Classifiers using CDI-aware GANs

abstract: We introduce a Channel Distribution Information (CDI)-aware Generative Adversarial Network (GAN), designed to address the unique challenges of adversarial attacks in wireless communication systems. The generator in this CDI-aware GAN maps random input noise to the feature space, generating perturbations intended to deceive a target modulation classifier. Its discriminators play a dual role: one enforces that the perturbations follow a Gaussian distribution, making them indistinguishable from Gaussian noise, while the other ensures these perturbations account for realistic channel effects and resemble no-channel perturbations. Our proposed CDI-aware GAN can be used as an attacker and a defender. In attack scenarios, the CDI-aware GAN demonstrates its prowess by generating robust adversarial perturbations that effectively deceive the target classifier, outperforming known methods. Furthermore, CDI-aware GAN as a defender significantly improves the target classifier's resilience against adversarial attacks.


title: Tunneling Potentials to Nothing

abstract: The catastrophic decay of a spacetime with compact dimensions, via bubbles of nothing (BoNs), is probably a generic phenomenon. BoNs admit a 4-dimensional description as singular Coleman-de Luccia bounces of the size modulus field, stabilized by some potential $V(\phi)$. We apply the tunneling potential approach to this 4d description to provide a very simple picture of BoNs. Using it we identify four different types of BoN, corresponding to different classes of higher dimensional theories. We study the quenching of BoN decays and their interplay with standard vacuum decays.


title: ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separation

abstract: Diffusion models have revolutionized image generation in recent years, yet they are still limited to a few sizes and aspect ratios. We propose ElasticDiffusion, a novel training-free decoding method that enables pretrained text-to-image diffusion models to generate images with various sizes. ElasticDiffusion attempts to decouple the generation trajectory of a pretrained model into local and global signals. The local signal controls low-level pixel information and can be estimated on local patches, while the global signal is used to maintain overall structural consistency and is estimated with a reference image. We test our method on CelebA-HQ (faces) and LAION-COCO (objects/indoor/outdoor scenes). Our experiments and qualitative results show superior image coherence quality across aspect ratios compared to MultiDiffusion and the standard decoding strategy of Stable Diffusion. Project page: https://elasticdiffusion.github.io/


title: Initializing Models with Larger Ones

abstract: Weight initialization plays an important role in neural network training. Widely used initialization methods are proposed and evaluated for networks that are trained from scratch. However, the growing number of pretrained models now offers new opportunities for tackling this classical problem of weight initialization. In this work, we introduce weight selection, a method for initializing smaller models by selecting a subset of weights from a pretrained larger model. This enables the transfer of knowledge from pretrained weights to smaller models. Our experiments demonstrate that weight selection can significantly enhance the performance of small models and reduce their training time. Notably, it can also be used together with knowledge distillation. Weight selection offers a new approach to leverage the power of pretrained models in resource-constrained settings, and we hope it can be a useful tool for training small models in the large-model era. Code is available at https://github.com/OscarXZQ/weight-selection.


title: An Adaptive Framework for Generalizing Network Traffic Prediction towards Uncertain Environments

abstract: We have developed a new framework using time-series analysis for dynamically assigning mobile network traffic prediction models in previously unseen wireless environments. Our framework selectively employs learned behaviors, outperforming any single model with over a 50% improvement relative to current studies. More importantly, it surpasses traditional approaches without needing prior knowledge of a cell. While this paper focuses on network traffic prediction using our adaptive forecasting framework, this framework can also be applied to other machine learning applications in uncertain environments. The framework begins with unsupervised clustering of time-series data to identify unique trends and seasonal patterns. Subsequently, we apply supervised learning for traffic volume prediction within each cluster. This specialization towards specific traffic behaviors occurs without penalties from spatial and temporal variations. Finally, the framework adaptively assigns trained models to new, previously unseen cells. By analyzing real-time measurements of a cell, our framework intelligently selects the most suitable cluster for that cell at any given time, with cluster assignment dynamically adjusting to spatio-temporal fluctuations.


title: CAST: Cross-Attention in Space and Time for Video Action Recognition

abstract: Recognizing human actions in videos requires spatial and temporal understanding. Most existing action recognition models lack a balanced spatio-temporal understanding of videos. In this work, we propose a novel two-stream architecture, called Cross-Attention in Space and Time (CAST), that achieves a balanced spatio-temporal understanding of videos using only RGB input. Our proposed bottleneck cross-attention mechanism enables the spatial and temporal expert models to exchange information and make synergistic predictions, leading to improved performance. We validate the proposed method with extensive experiments on public benchmarks with different characteristics: EPIC-KITCHENS-100, Something-Something-V2, and Kinetics-400. Our method consistently shows favorable performance across these datasets, while the performance of existing methods fluctuates depending on the dataset characteristics.


title: Geometry-Aware Normalizing Wasserstein Flows for Optimal Causal Inference

abstract: This paper presents a groundbreaking approach to causal inference by integrating continuous normalizing flows (CNFs) with parametric submodels, enhancing their geometric sensitivity and improving upon traditional Targeted Maximum Likelihood Estimation (TMLE). Our method employs CNFs to refine TMLE, optimizing the Cram\'er-Rao bound and transitioning from a predefined distribution $p_0$ to a data-driven distribution $p_1$. We innovate further by embedding Wasserstein gradient flows within Fokker-Planck equations, thus imposing geometric structures that boost the robustness of CNFs, particularly in optimal transport theory. Our approach addresses the disparity between sample and population distributions, a critical factor in parameter estimation bias. We leverage optimal transport and Wasserstein gradient flows to develop causal inference methodologies with minimal variance in finite-sample settings, outperforming traditional methods like TMLE and AIPW. This novel framework, centered on Wasserstein gradient flows, minimizes variance in efficient influence functions under distribution $p_t$. Preliminary experiments showcase our method's superiority, yielding lower mean-squared errors compared to standard flows, thereby demonstrating the potential of geometry-aware normalizing Wasserstein flows in advancing statistical modeling and inference.


title: Motion-Conditioned Image Animation for Video Editing

abstract: We introduce MoCA, a Motion-Conditioned Image Animation approach for video editing. It leverages a simple decomposition of the video editing problem into image editing followed by motion-conditioned image animation. Furthermore, given the lack of robust evaluation datasets for video editing, we introduce a new benchmark that measures edit capability across a wide variety of tasks, such as object replacement, background changes, style changes, and motion edits. We present a comprehensive human evaluation of the latest video editing methods along with MoCA, on our proposed benchmark. MoCA establishes a new state-of-the-art, demonstrating greater human preference win-rate, and outperforming notable recent approaches including Dreamix (63%), MasaCtrl (75%), and Tune-A-Video (72%), with especially significant improvements for motion edits.


title: One-step Diffusion with Distribution Matching Distillation

abstract: Diffusion models generate high-quality images but require dozens of forward passes. We introduce Distribution Matching Distillation (DMD), a procedure to transform a diffusion model into a one-step image generator with minimal impact on image quality. We enforce the one-step image generator match the diffusion model at distribution level, by minimizing an approximate KL divergence whose gradient can be expressed as the difference between 2 score functions, one of the target distribution and the other of the synthetic distribution being produced by our one-step generator. The score functions are parameterized as two diffusion models trained separately on each distribution. Combined with a simple regression loss matching the large-scale structure of the multi-step diffusion outputs, our method outperforms all published few-step diffusion approaches, reaching 2.62 FID on ImageNet 64x64 and 11.49 FID on zero-shot COCO-30k, comparable to Stable Diffusion but orders of magnitude faster. Utilizing FP16 inference, our model generates images at 20 FPS on modern hardware.


title: MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generation

abstract: We present MicroCinema, a straightforward yet effective framework for high-quality and coherent text-to-video generation. Unlike existing approaches that align text prompts with video directly, MicroCinema introduces a Divide-and-Conquer strategy which divides the text-to-video into a two-stage process: text-to-image generation and image\&text-to-video generation. This strategy offers two significant advantages. a) It allows us to take full advantage of the recent advances in text-to-image models, such as Stable Diffusion, Midjourney, and DALLE, to generate photorealistic and highly detailed images. b) Leveraging the generated image, the model can allocate less focus to fine-grained appearance details, prioritizing the efficient learning of motion dynamics. To implement this strategy effectively, we introduce two core designs. First, we propose the Appearance Injection Network, enhancing the preservation of the appearance of the given image. Second, we introduce the Appearance Noise Prior, a novel mechanism aimed at maintaining the capabilities of pre-trained 2D diffusion models. These design elements empower MicroCinema to generate high-quality videos with precise motion, guided by the provided text prompts. Extensive experiments demonstrate the superiority of the proposed framework. Concretely, MicroCinema achieves SOTA zero-shot FVD of 342.86 on UCF-101 and 377.40 on MSR-VTT. See https://wangyanhui666.github.io/MicroCinema.github.io/ for video samples.


title: MotionEditor: Editing Video Motion via Content-Aware Diffusion

abstract: Existing diffusion-based video editing models have made gorgeous advances for editing attributes of a source video over time but struggle to manipulate the motion information while preserving the original protagonist's appearance and background. To address this, we propose MotionEditor, a diffusion model for video motion editing. MotionEditor incorporates a novel content-aware motion adapter into ControlNet to capture temporal motion correspondence. While ControlNet enables direct generation based on skeleton poses, it encounters challenges when modifying the source motion in the inverted noise due to contradictory signals between the noise (source) and the condition (reference). Our adapter complements ControlNet by involving source content to transfer adapted control signals seamlessly. Further, we build up a two-branch architecture (a reconstruction branch and an editing branch) with a high-fidelity attention injection mechanism facilitating branch interaction. This mechanism enables the editing branch to query the key and value from the reconstruction branch in a decoupled manner, making the editing branch retain the original background and protagonist appearance. We also propose a skeleton alignment algorithm to address the discrepancies in pose size and position. Experiments demonstrate the promising motion editing ability of MotionEditor, both qualitatively and quantitatively.


title: Schwinger-Dyson equation on the complex plane -- A four-fermion interaction model at finite temperature --

abstract: We extend the Schwinger-Dyson equation (SDE) on the complex plane, which was treated in our previous research, to finite temperature. As a simple example, we solve the SDE for a model with four-fermion interactions in the (1+1) space-time dimensions at strong coupling region. We investigate the properties of the effective mass and energy for the fermions, especially near the phase transition temperature.


title: Exploiting Diffusion Prior for Generalizable Dense Prediction

abstract: Contents generated by recent advanced Text-to-Image (T2I) diffusion models are sometimes too imaginative for existing off-the-shelf dense predictors to estimate due to the immitigable domain gap. We introduce DMP, a pipeline utilizing pre-trained T2I models as a prior for dense prediction tasks. To address the misalignment between deterministic prediction tasks and stochastic T2I models, we reformulate the diffusion process through a sequence of interpolations, establishing a deterministic mapping between input RGB images and output prediction distributions. To preserve generalizability, we use low-rank adaptation to fine-tune pre-trained models. Extensive experiments across five tasks, including 3D property estimation, semantic segmentation, and intrinsic image decomposition, showcase the efficacy of the proposed method. Despite limited-domain training data, the approach yields faithful estimations for arbitrary images, surpassing existing state-of-the-art algorithms.


title: VLA FRAMEx. I. Wideband Radio Properties of the AGN in NGC 4388

abstract: We present the first results from Karl G. Jansky Very Large Array (VLA) observations as a part of the Fundamental Reference Active Galactic Nucleus (AGN) Monitoring Experiment (FRAMEx), a program to understand the relationship between AGN accretion physics and wavelength-dependent position as a function of time. With this VLA survey, we investigate the radio properties from a volume-complete sample of 25 hard X-ray-selected AGNs using the VLA in its wideband mode. We observed the targets in the A-array configuration at $4-12$ GHz with all polarization products. In this work, we introduce our calibration and imaging methods for this survey, and we present our results and analysis for the radio quiet AGN NGC 4388. We calibrated and imaged these data using the multi-term, multi-frequency synthesis imaging algorithm to determine its spatial, spectral and polarization structure across a continuous $4-12$ GHz band. In the AGN, we measure a broken power law spectrum with $\alpha=-0.06$ below a break frequency of 7.3 GHz and $\alpha=-0.34$ above. We detect polarization at sub-arcsecond resolution across both the AGN and a secondary radio knot. We compare our results to ancillary data and find that the VLA radio continuum is likely due to AGN winds interacting with the local interstellar medium that gets resolved away at sub-parsec spatial scales as probed by the Very Long Baseline Array. A well-known ionization cone to the southwest of the AGN appears likely to be projected material onto the underside of the disk of the host galaxy.


title: ART$\boldsymbol{\cdot}$V: Auto-Regressive Text-to-Video Generation with Diffusion Models

abstract: We present ART$\boldsymbol{\cdot}$V, an efficient framework for auto-regressive video generation with diffusion models. Unlike existing methods that generate entire videos in one-shot, ART$\boldsymbol{\cdot}$V generates a single frame at a time, conditioned on the previous ones. The framework offers three distinct advantages. First, it only learns simple continual motions between adjacent frames, therefore avoiding modeling complex long-range motions that require huge training data. Second, it preserves the high-fidelity generation ability of the pre-trained image diffusion models by making only minimal network modifications. Third, it can generate arbitrarily long videos conditioned on a variety of prompts such as text, image or their combinations, making it highly versatile and flexible. To combat the common drifting issue in AR models, we propose masked diffusion model which implicitly learns which information can be drawn from reference images rather than network predictions, in order to reduce the risk of generating inconsistent appearances that cause drifting. Moreover, we further enhance generation coherence by conditioning it on the initial frame, which typically contains minimal noise. This is particularly useful for long video generation. When trained for only two weeks on four GPUs, ART$\boldsymbol{\cdot}$V already can generate videos with natural motions, rich details and a high level of aesthetic quality. Besides, it enables various appealing applications, e.g., composing a long video from multiple text prompts.


title: InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation

abstract: Empowering models to dynamically accomplish tasks specified through natural language instructions represents a promising path toward more capable and general artificial intelligence. In this work, we introduce InstructSeq, an instruction-conditioned multi-modal modeling framework that unifies diverse vision tasks through flexible natural language control and handling of both visual and textual data. InstructSeq employs a multimodal transformer architecture encompassing visual, language, and sequential modeling. We utilize a visual encoder to extract image features and a text encoder to encode instructions. An autoregressive transformer fuses the representations and generates sequential task outputs. By training with LLM-generated natural language instructions, InstructSeq acquires a strong comprehension of free-form instructions for specifying visual tasks. This provides an intuitive interface for directing capabilities using flexible natural instructions. Without any task-specific tuning, InstructSeq achieves compelling performance on semantic segmentation, referring expression segmentation/comprehension, and image captioning. The flexible control and multi-task unification empower the model with more human-like versatility and generalizability for computer vision. The code will be released soon at https://github.com/rongyaofang/InstructSeq.


title: ChatPose: Chatting about 3D Human Pose

abstract: We introduce ChatPose, a framework employing Large Language Models (LLMs) to understand and reason about 3D human poses from images or textual descriptions. Our work is motivated by the human ability to intuitively understand postures from a single image or a brief description, a process that intertwines image interpretation, world knowledge, and an understanding of body language. Traditional human pose estimation and generation methods often operate in isolation, lacking semantic understanding and reasoning abilities. ChatPose addresses these limitations by embedding SMPL poses as distinct signal tokens within a multimodal LLM, enabling the direct generation of 3D body poses from both textual and visual inputs. Leveraging the powerful capabilities of multimodal LLMs, ChatPose unifies classical 3D human pose and generation tasks while offering user interactions. Additionally, ChatPose empowers LLMs to apply their extensive world knowledge in reasoning about human poses, leading to two advanced tasks: speculative pose generation and reasoning about pose estimation. These tasks involve reasoning about humans to generate 3D poses from subtle text queries, possibly accompanied by images. We establish benchmarks for these tasks, moving beyond traditional 3D pose generation and estimation methods. Our results show that ChatPose outperforms existing multimodal LLMs and task-specific methods on these newly proposed tasks. Furthermore, ChatPose's ability to understand and generate 3D human poses based on complex reasoning opens new directions in human pose analysis.