title: Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection

abstract: The primary bottleneck towards obtaining good recognition performance in IR images is the lack of sufficient labeled training data, owing to the cost of acquiring such data. Realizing that object detection methods for the RGB modality are quite robust (at least for some commonplace classes, like person, car, etc.), thanks to the giant training sets that exist, in this work we seek to leverage cues from the RGB modality to scale object detectors to the IR modality, while preserving model performance in the RGB modality. At the core of our method, is a novel tensor decomposition method called TensorFact which splits the convolution kernels of a layer of a Convolutional Neural Network (CNN) into low-rank factor matrices, with fewer parameters than the original CNN. We first pretrain these factor matrices on the RGB modality, for which plenty of training data are assumed to exist and then augment only a few trainable parameters for training on the IR modality to avoid over-fitting, while encouraging them to capture complementary cues from those trained only on the RGB modality. We validate our approach empirically by first assessing how well our TensorFact decomposed network performs at the task of detecting objects in RGB images vis-a-vis the original network and then look at how well it adapts to IR images of the FLIR ADAS v1 dataset. For the latter, we train models under scenarios that pose challenges stemming from data paucity. From the experiments, we observe that: (i) TensorFact shows performance gains on RGB images; (ii) further, this pre-trained model, when fine-tuned, outperforms a standard state-of-the-art object detector on the FLIR ADAS v1 dataset by about 4% in terms of mAP 50 score.


title: Navigating Healthcare Insights: A Birds Eye View of Explainability with Knowledge Graphs

abstract: Knowledge graphs (KGs) are gaining prominence in Healthcare AI, especially in drug discovery and pharmaceutical research as they provide a structured way to integrate diverse information sources, enhancing AI system interpretability. This interpretability is crucial in healthcare, where trust and transparency matter, and eXplainable AI (XAI) supports decision making for healthcare professionals. This overview summarizes recent literature on the impact of KGs in healthcare and their role in developing explainable AI models. We cover KG workflow, including construction, relationship extraction, reasoning, and their applications in areas like Drug-Drug Interactions (DDI), Drug Target Interactions (DTI), Drug Development (DD), Adverse Drug Reactions (ADR), and bioinformatics. We emphasize the importance of making KGs more interpretable through knowledge-infused learning in healthcare. Finally, we highlight research challenges and provide insights for future directions.


title: Deterministic Fully Dynamic SSSP and More

abstract: We present the first non-trivial fully dynamic algorithm maintaining exact single-source distances in unweighted graphs. This resolves an open problem stated by Sankowski [COCOON 2005] and van den Brand and Nanongkai [FOCS 2019]. Previous fully dynamic single-source distances data structures were all approximate, but so far, non-trivial dynamic algorithms for the exact setting could only be ruled out for polynomially weighted graphs (Abboud and Vassilevska Williams, [FOCS 2014]). The exact unweighted case remained the main case for which neither a subquadratic dynamic algorithm nor a quadratic lower bound was known. Our dynamic algorithm works on directed graphs, is deterministic, and can report a single-source shortest paths tree in subquadratic time as well. Thus we also obtain the first deterministic fully dynamic data structure for reachability (transitive closure) with subquadratic update and query time. This answers an open problem of van den Brand, Nanongkai, and Saranurak [FOCS 2019]. Finally, using the same framework we obtain the first fully dynamic data structure maintaining all-pairs $(1+\epsilon)$-approximate distances within non-trivial sub-$n^\omega$ worst-case update time while supporting optimal-time approximate shortest path reporting at the same time. This data structure is also deterministic and therefore implies the first known non-trivial deterministic worst-case bound for recomputing the transitive closure of a digraph.


title: Can LLMs Effectively Leverage Graph Structural Information through Prompts, and Why?

abstract: Large language models (LLMs) are gaining increasing attention for their capability to process graphs with rich text attributes, especially in a zero-shot fashion. Recent studies demonstrate that LLMs obtain decent text classification performance on common text-rich graph benchmarks, and the performance can be improved by appending encoded structural information as natural languages into prompts. We aim to understand why the incorporation of structural information inherent in graph data can improve the prediction performance of LLMs. First, we rule out the concern of data leakage by curating a novel leakage-free dataset and conducting a comparative analysis alongside a previously widely-used dataset. Second, as past work usually encodes the ego-graph by describing the graph structure in natural language, we ask the question: do LLMs understand the graph structure in accordance with the intent of the prompt designers? Third, we investigate why LLMs can improve their performance after incorporating structural information. Our exploration of these questions reveals that (i) there is no substantial evidence that the performance of LLMs is significantly attributed to data leakage; (ii) instead of understanding prompts as graph structures as intended by the prompt designers, LLMs tend to process prompts more as contextual paragraphs and (iii) the most efficient elements of the local neighborhood included in the prompt are phrases that are pertinent to the node label, rather than the graph structure.


title: Local minima in quantum systems

abstract: Finding ground states of quantum many-body systems is known to be hard for both classical and quantum computers. As a result, when Nature cools a quantum system in a low-temperature thermal bath, the ground state cannot always be found efficiently. Instead, Nature finds a local minimum of the energy. In this work, we study the problem of finding local minima in quantum systems under thermal perturbations. While local minima are much easier to find than ground states, we show that finding a local minimum is computationally hard for classical computers, even when the task is to output a single-qubit observable at any local minimum. In contrast, we prove that a quantum computer can always find a local minimum efficiently using a thermal gradient descent algorithm that mimics the cooling process in Nature. To establish the classical hardness of finding local minima, we consider a family of two-dimensional Hamiltonians such that any problem solvable by polynomial-time quantum algorithms can be reduced to finding ground states of these Hamiltonians. We prove that for such Hamiltonians, all local minima are global minima. Therefore, assuming quantum computation is more powerful than classical computation, finding local minima is classically hard and quantumly easy.

categories: quant-ph cond-mat.dis-nn cs.CC math-ph math.MP math.OC


title: Transfer Learning for Bayesian Optimization on Heterogeneous Search Spaces

abstract: Bayesian optimization (BO) is a popular black-box function optimization method, which makes sequential decisions based on a Bayesian model, typically a Gaussian process (GP), of the function. To ensure the quality of the model, transfer learning approaches have been developed to automatically design GP priors by learning from observations on "training" functions. These training functions are typically required to have the same domain as the "test" function (black-box function to be optimized). In this paper, we introduce MPHD, a model pre-training method on heterogeneous domains, which uses a neural net mapping from domain-specific contexts to specifications of hierarchical GPs. MPHD can be seamlessly integrated with BO to transfer knowledge across heterogeneous search spaces. Our theoretical and empirical results demonstrate the validity of MPHD and its superior performance on challenging black-box function optimization tasks.


title: Cross-Prediction-Powered Inference

abstract: While reliable data-driven decision-making hinges on high-quality labeled data, the acquisition of quality labels often involves laborious human annotations or slow and expensive scientific measurements. Machine learning is becoming an appealing alternative as sophisticated predictive techniques are being used to quickly and cheaply produce large amounts of predicted labels; e.g., predicted protein structures are used to supplement experimentally derived structures, predictions of socioeconomic indicators from satellite imagery are used to supplement accurate survey data, and so on. Since predictions are imperfect and potentially biased, this practice brings into question the validity of downstream inferences. We introduce cross-prediction: a method for valid inference powered by machine learning. With a small labeled dataset and a large unlabeled dataset, cross-prediction imputes the missing labels via machine learning and applies a form of debiasing to remedy the prediction inaccuracies. The resulting inferences achieve the desired error probability and are more powerful than those that only leverage the labeled data. Closely related is the recent proposal of prediction-powered inference, which assumes that a good pre-trained model is already available. We show that cross-prediction is consistently more powerful than an adaptation of prediction-powered inference in which a fraction of the labeled data is split off and used to train the model. Finally, we observe that cross-prediction gives more stable conclusions than its competitors; its confidence intervals typically have significantly lower variability.


title: Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation

abstract: Zero-shot translation (ZST), which is generally based on a multilingual neural machine translation model, aims to translate between unseen language pairs in training data. The common practice to guide the zero-shot language mapping during inference is to deliberately insert the source and target language IDs, e.g., <EN> for English and <DE> for German. Recent studies have shown that language IDs sometimes fail to navigate the ZST task, making them suffer from the off-target problem (non-target language words exist in the generated translation) and, therefore, difficult to apply the current multilingual translation model to a broad range of zero-shot language scenarios. To understand when and why the navigation capabilities of language IDs are weakened, we compare two extreme decoder input cases in the ZST directions: Off-Target (OFF) and On-Target (ON) cases. By contrastively visualizing the contextual word representations (CWRs) of these cases with teacher forcing, we show that 1) the CWRs of different languages are effectively distributed in separate regions when the sentence and ID are matched (ON setting), and 2) if the sentence and ID are unmatched (OFF setting), the CWRs of different languages are chaotically distributed. Our analyses suggest that although they work well in ideal ON settings, language IDs become fragile and lose their navigation ability when faced with off-target tokens, which commonly exist during inference but are rare in training scenarios. In response, we employ unlikelihood tuning on the negative (OFF) samples to minimize their probability such that the language IDs can discriminate between the on- and off-target tokens during training. Experiments spanning 40 ZST directions show that our method reduces the off-target ratio by -48.0% on average, leading to a +9.1 BLEU improvement with only an extra +0.3% tuning cost.


title: Constraining Ultralight Dark Matter through an Accelerated Resonant Search

abstract: Experiments aimed at detecting ultralight dark matter typically rely on resonant effects, which are sensitive to the dark matter mass that matches the resonance frequency. In this study, we investigate the nucleon couplings of ultralight axion dark matter using a magnetometer operating in a nuclear magnetic resonance (NMR) mode. Our approach involves the use of a $^{21}$Ne spin-based sensor, which features the lowest nuclear magnetic moment among noble-gas spins. This configuration allows us to achieve an ultrahigh sensitivity of 0.73 fT/Hz$^{1/2}$ at around 5 Hz, corresponding to energy resolution of approximately 1.5$\times 10^{-23}\,\rm{eV/Hz^{1/2}}$. Our analysis reveals that under certain conditions it is beneficial to scan the frequency with steps significantly larger than the resonance width. The analytical results are in agreement with experimental data and the scan strategy is potentially applicable to other resonant searches. Further, our study establishes stringent constraints on axion-like particles (ALP) in the 4.5--15.5 Hz Compton-frequency range coupling to neutrons and protons, improving on prior work by several-fold. Within a band around 4.6--6.6 Hz and around 7.5 Hz, our laboratory findings surpass astrophysical limits derived from neutron-star cooling. Hence, we demonstrate an accelerated resonance search for ultralight dark matter, achieving an approximately 30-fold increase in scanning step while maintaining competitive sensitivity.

categories: hep-ph astro-ph.CO hep-ex physics.atom-ph quant-ph


title: Field-control of symmetry-broken and quantum disordered phases in frustrated moir\'e bilayers with population imbalance

abstract: We determine the ground states and excitation spectra of the paradigmatic four-flavour Heisenberg model with nearest- and next-nearest-neighbor exchange couplings on the triangular lattice in a field controlling the population imbalance of flavor pairs. Such a system arises in the strongly correlated limit of moir\'e bilayers of transition metal dichalcogenides in an electric displacement field or in-plane magnetic field, and can be simulated via ultracold alkaline-earth atoms. We argue that the field tunes between effective SU(4) and SU(2) symmetries in the balanced and fully polarised limits and employ a combination of mean-field calculations, flavour-wave theory, and exact diagonalisation to analyse the intermediate, imbalanced regime. We find different symmetry-broken phases with simultaneous spin and excitonic order depending on the field and next-nearest-neighbor coupling. Furthermore, we demonstrate that there is a strongly fluctuating regime without long-range order that connects candidate spin liquids of the SU(2) and SU(4) limit. The strong fluctuations are facilitated by an extensive classical degeneracy of the model, and we argue that they are also responsible for a strong polarisability at 1/3 polarisation that survives from the mean-field level to the exact spectrum.


title: Multicollinear Singularities in Celestial CFT

abstract: The purpose of this paper is to study the holomorphic multicollinear limit of (celestial) amplitudes and use it to further investigate the double residue condition for (hard celestial) amplitudes and the celestial operator product expansion. We first set up the notion of holomorphic multicollinear limits of amplitudes and derive the 3-collinear splitting functions for Yang-Mills theory, Einstein gravity, and massless $\phi^3$ theory. In particular, we find that in $\phi^3$ theory the celestial 3-OPE contains a term with a branch cut. This explicit example confirms that branch cuts can obstruct the double residue condition for hard celestial amplitudes, which is the underlying cause of the celestial Jacobi identities not holding for certain theories. This addresses an ongoing debate in the literature about associativity of the celestial OPEs and concretely demonstrates a new (multi-particle) term in the celestial OPE coming from the multi-particle channel in the amplitudes.


title: Deep Learning Based Uplink Multi-User SIMO Beamforming Design

abstract: The advancement of fifth generation (5G) wireless communication networks has created a greater demand for wireless resource management solutions that offer high data rates, extensive coverage, minimal latency and energy-efficient performance. Nonetheless, traditional approaches have shortcomings when it comes to computational complexity and their ability to adapt to dynamic conditions, creating a gap between theoretical analysis and the practical execution of algorithmic solutions for managing wireless resources. Deep learning-based techniques offer promising solutions for bridging this gap with their substantial representation capabilities. We propose a novel unsupervised deep learning framework, which is called NNBF, for the design of uplink receive multi-user single input multiple output (MU-SIMO) beamforming. The primary objective is to enhance the throughput by focusing on maximizing the sum-rate while also offering computationally efficient solution, in contrast to established conventional methods. We conduct experiments for several antenna configurations. Our experimental results demonstrate that NNBF exhibits superior performance compared to our baseline methods, namely, zero-forcing beamforming (ZFBF) and minimum mean square error (MMSE) equalizer. Additionally, NNBF is scalable to the number of single-antenna user equipments (UEs) while baseline methods have significant computational burden due to matrix pseudo-inverse operation.


title: Exploiting Edge Features in Graphs with Fused Network Gromov-Wasserstein Distance

abstract: Pairwise comparison of graphs is key to many applications in Machine learning ranging from clustering, kernel-based classification/regression and more recently supervised graph prediction. Distances between graphs usually rely on informative representations of these structured objects such as bag of substructures or other graph embeddings. A recently popular solution consists in representing graphs as metric measure spaces, allowing to successfully leverage Optimal Transport, which provides meaningful distances allowing to compare them: the Gromov-Wasserstein distances. However, this family of distances overlooks edge attributes, which are essential for many structured objects. In this work, we introduce an extension of Gromov-Wasserstein distance for comparing graphs whose both nodes and edges have features. We propose novel algorithms for distance and barycenter computation. We empirically show the effectiveness of the novel distance in learning tasks where graphs occur in either input space or output space, such as classification and graph prediction.


title: One point compactifications of configuration spaces and the self duality of the little disks operad

abstract: Using configuration space level Pontryagin--Thom constructions, we construct a simple Koszul self duality map for the little disks operad $\Sigma^\infty_+ E_n$. For a framed $n$-manifold $M$, we show that a compatible self duality map exists for $\Sigma^\infty_+ E_M$.


title: "AI enhances our performance, I have no doubt this one will do the same": The Placebo effect is robust to negative descriptions of AI

abstract: Heightened AI expectations facilitate performance in human-AI interactions through placebo effects. While lowering expectations to control for placebo effects is advisable, overly negative expectations could induce nocebo effects. In a letter discrimination task, we informed participants that an AI would either increase or decrease their performance by adapting the interface, but in reality, no AI was present in any condition. A Bayesian analysis showed that participants had high expectations and performed descriptively better irrespective of the AI description when a sham-AI was present. Using cognitive modeling, we could trace this advantage back to participants gathering more information. A replication study verified that negative AI descriptions do not alter expectations, suggesting that performance expectations with AI are biased and robust to negative verbal descriptions. We discuss the impact of user expectations on AI interactions and evaluation and provide a behavioral placebo marker for human-AI interaction


title: Subspace Profiles over Finite Fields and $q$-Whittaker Expansions of Symmetric Functions

abstract: Bender, Coley, Robbins and Rumsey posed the problem of counting the number of subspaces which have a given profile with respect to a linear endomorphism defined on a finite vector space. Several special cases of this problem have been solved in the literature. We settle this problem in full generality by giving an explicit counting formula in terms of symmetric functions. This formula can be expressed compactly in terms a Hall scalar product involving dual $q$-Whittaker functions and another symmetric function that is determined by conjugacy class invariants of the linear endomorphism. As corollaries, we obtain new combinatorial interpretations for the coefficients in the $q$-Whittaker expansions of several symmetric functions. These include the power sum, complete homogeneous, products of modified Hall-Littlewood polynomials and certain products of $q$-Whittaker functions. These results are used to derive a formula for the number of anti-invariant subspaces (as defined by Barr\'ia and Halmos) with respect to an arbitrary operator. We also give an application to an open problem in Krylov subspace theory.


title: KV Inversion: KV Embeddings Learning for Text-Conditioned Real Image Action Editing

abstract: Text-conditioned image editing is a recently emerged and highly practical task, and its potential is immeasurable. However, most of the concurrent methods are unable to perform action editing, i.e. they can not produce results that conform to the action semantics of the editing prompt and preserve the content of the original image. To solve the problem of action editing, we propose KV Inversion, a method that can achieve satisfactory reconstruction performance and action editing, which can solve two major problems: 1) the edited result can match the corresponding action, and 2) the edited object can retain the texture and identity of the original real image. In addition, our method does not require training the Stable Diffusion model itself, nor does it require scanning a large-scale dataset to perform time-consuming training.


title: Qwen Technical Report

abstract: Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans. In this work, we introduce Qwen, the first installment of our large language model series. Qwen is a comprehensive language model series that encompasses distinct models with varying parameter counts. It includes Qwen, the base pretrained language models, and Qwen-Chat, the chat models finetuned with human alignment techniques. The base language models consistently demonstrate superior performance across a multitude of downstream tasks, and the chat models, particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are highly competitive. The chat models possess advanced tool-use and planning capabilities for creating agent applications, showcasing impressive performance even when compared to bigger models on complex tasks like utilizing a code interpreter. Furthermore, we have developed coding-specialized models, Code-Qwen and Code-Qwen-Chat, as well as mathematics-focused models, Math-Qwen-Chat, which are built upon base language models. These models demonstrate significantly improved performance in comparison with open-source models, and slightly fall behind the proprietary models.


title: $\mathcal{H}$-Killing Spinors and Spinorial Duality for Homogeneous 3-$(\alpha,\delta)$-Sasaki Manifolds

abstract: We show that $3$-$(\alpha,\delta)$-Sasaki manifolds admit solutions of a certain new spinorial field equation (the $\mathcal{H}$-Killing equation) generalizing the well-known Killing spinors on $3$-Sasakian manifolds. These $\mathcal{H}$-Killing spinors have more desirable geometric properties than the spinors obtained by simply deforming a $3$-Sasakian metric; in particular we obtain a one-to-one correspondence between $\mathcal{H}$-Killing spinors on dual pairs of homogeneous $3$-$(\alpha,\delta)$-Sasaki spaces. Finally, we show that $\mathcal{H}$-Killing spinors generalize certain special spinors in dimension $7$ previously constructed by Agricola-Friedrich and Agricola-Dileo using $\text{G}_2$-geometry.


title: Positivity restrictions on the mixing of dimension-eight SMEFT operators

abstract: We discuss the structure of the mixing among dimension-eight operators in the SMEFT relying on the positivity of two-to-two forward scattering amplitudes. We uncover tens of new non-trivial zeros as well as hundreds of terms with definite sign in (a particular basis of) the corresponding anomalous dimension matrix. We highlight that our results are not immediately apparent from the Feynman diagrammatic perspective, nor from on-shell amplitude methods. As a byproduct of this work, we provide positivity bounds not previously derived in the literature, as well as explicit values of certain elements of the anomalous dimension matrix that serve for cross-check of our results.


title: Curvature of positive relative line modules over the quantum projective spaces

abstract: We show that the curvature of a positive relative line module over quantum projective space is given by $q$-integer deformation of its classical curvature. This generalises a result of Majid for the Podle\'s sphere.


title: Conjugacy in Rearrangement Groups of Fractals

abstract: We describe a method for solving the conjugacy problem in a vast class of rearrangement groups of fractals, a family of Thompson-like groups introduced in 2019 by Belk and Forrest. We generalize the methods of Belk and Matucci for the solution of the conjugacy problem in Thompson groups $F$, $T$ and $V$ via strand diagrams. In particular, we solve the conjugacy problem for the Basilica, the Airplane, the Vicsek and the Bubble Bath rearrangement groups and for the groups $QV$ (also known as $QAut(\mathcal{T}_{2,c})$), $\tilde{Q}V$, $QT$, $\tilde{Q}T$ and $QF$, and we provide a new solution to the conjugacy problem for the Houghton groups and for the Higman-Thompson groups, where conjugacy was already known to be solvable. Our methods involve two distinct rewriting systems, one of which is an instance of a graph rewriting system, whose confluence in general is of interest in computer science.


title: The twisting index in semitoric systems

abstract: Semitoric integrable systems were symplectically classified by Pelayo and Vu Ngoc in 2009-2011 in terms of five invariants. Four of these invariants were already well-understood prior to the classification, but the fifth invariant, the so-called twisting index invariant, came as a surprise. Intuitively, the twisting index encodes how the structure in a neighborhood of a focus-focus fiber compares to the large-scale structure of the semitoric system and it was originally defined by comparing certain momentum maps. In the first half of the present paper, we produce several new formulations of the twisting index which give rise to dynamical, geometric, and topological interpretations. More specifically, we describe it in terms of differences of action variables, Taylor series, and homology cycles. In the second half of the paper, we compute the twisting index invariant of a specific family of systems with two focus-focus singular points (the so-called generalized coupled angular momenta), which is the first time that the twisting index has been computed for a system with more than one focus-focus point. Moreover, we also compute the terms of the Taylor series invariant up to second order. Since the other invariants of this family were already computed, this becomes the third family of semitoric systems for which all invariants are known, after the coupled spin oscillators and the coupled angular momenta.


title: Anatomy of real intermediate state-subtraction scheme

abstract: We study the origin of the real intermediate state subtraction problem and compare its different solutions. We show that the ambiguity in subtraction schemes arises from the on-shell approximation for the 2-point functions that reduces the Schwinger-Dyson equations to the Boltzmann limit. We also suggest a new subtraction scheme which, unlike the earlier definitions, never leads to negative scattering rates. This scheme also quantifies the validity of the on-shell limit in terms of an effective one-particle weight function $R(\Delta )$, where $\Delta$ measures the region around the resonance associated with the real state.


abstract: We give Kaplansky/Nagata-type theorems for the half factorial domains inside the class of atomic domains.


title: Adaptive Output-Feedback Model Predictive Control of Hammerstein Systems with Unknown Linear Dynamics

abstract: This paper considers model predictive control of Hammerstein systems, where the linear dynamics are a priori unknown and the input nonlinearity is known. Predictive cost adaptive control (PCAC) is applied to this system using recursive least squares for online, closed-loop system identification with optimization over a receding horizon performed by quadratic programming (QP). In order to account for the input nonlinearity, the input matrix is defined to be control dependent, and the optimization is performed iteratively. This technique is applied to output stabilization of a chain of integrators with unknown dynamics under control saturation and deadzone input nonlinearity.


title: Revisiting Neural Program Smoothing for Fuzzing

abstract: Testing with randomly generated inputs (fuzzing) has gained significant traction due to its capacity to expose program vulnerabilities automatically. Fuzz testing campaigns generate large amounts of data, making them ideal for the application of machine learning (ML). Neural program smoothing (NPS), a specific family of ML-guided fuzzers, aims to use a neural network as a smooth approximation of the program target for new test case generation. In this paper, we conduct the most extensive evaluation of NPS fuzzers against standard gray-box fuzzers (>11 CPU years and >5.5 GPU years), and make the following contributions: (1) We find that the original performance claims for NPS fuzzers do not hold; a gap we relate to fundamental, implementation, and experimental limitations of prior works. (2) We contribute the first in-depth analysis of the contribution of machine learning and gradient-based mutations in NPS. (3) We implement Neuzz++, which shows that addressing the practical limitations of NPS fuzzers improves performance, but that standard gray-box fuzzers almost always surpass NPS-based fuzzers. (4) As a consequence, we propose new guidelines targeted at benchmarking fuzzing based on machine learning, and present MLFuzz, a platform with GPU access for easy and reproducible evaluation of ML-based fuzzers. Neuzz++, MLFuzz, and all our data are public.


title: Cubical Approximation for Directed Topology II

abstract: The paper establishes an equivalence between localizations of (diagrams of) cubical sets and (diagrams of) directed topological spaces by those maps defining (natural) cubical homotopy equivalences after application of the directed singular functor and a directed analogue of fibrant replacement. This equivalence both lifts and extends an equivalence between classical homotopy categories of cubical sets and topological spaces. Some simple applications include combinatorial descriptions and subsequent calculations of directed homotopy monoids and directed singular 1-cohomology monoids. Another application is a characterization of isomorphisms between small categories up to zig-zags of natural transformations as directed homotopy equivalences between directed classifying spaces. Cubical sets throughout the paper are taken to mean presheaves over the minimal symmetric monoidal variant of the cube category. Along the way, the paper characterizes morphisms in this variant as the interval-preserving lattice homomorphisms between finite Boolean lattice and describes some of the test model structure on presheaves over this variant.


title: Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit

abstract: The cost of hyperparameter tuning in deep learning has been rising with model sizes, prompting practitioners to find new tuning methods using a proxy of smaller networks. One such proposal uses $\mu$P parameterized networks, where the optimal hyperparameters for small width networks transfer to networks with arbitrarily large width. However, in this scheme, hyperparameters do not transfer across depths. As a remedy, we study residual networks with a residual branch scale of $1/\sqrt{\text{depth}}$ in combination with the $\mu$P parameterization. We provide experiments demonstrating that residual architectures including convolutional ResNets and Vision Transformers trained with this parameterization exhibit transfer of optimal hyperparameters across width and depth on CIFAR-10 and ImageNet. Furthermore, our empirical findings are supported and motivated by theory. Using recent developments in the dynamical mean field theory (DMFT) description of neural network learning dynamics, we show that this parameterization of ResNets admits a well-defined feature learning joint infinite-width and infinite-depth limit and show convergence of finite-size network dynamics towards this limit.


title: Stress Testing Chain-of-Thought Prompting for Large Language Models

abstract: This report examines the effectiveness of Chain-of-Thought (CoT) prompting in improving the multi-step reasoning abilities of large language models (LLMs). Inspired by previous studies \cite{Min2022RethinkingWork}, we analyze the impact of three types of CoT prompt perturbations, namely CoT order, CoT values, and CoT operators on the performance of GPT-3 on various tasks. Our findings show that incorrect CoT prompting leads to poor performance on accuracy metrics. Correct values in the CoT is crucial for predicting correct answers. Moreover, incorrect demonstrations, where the CoT operators or the CoT order are wrong, do not affect the performance as drastically when compared to the value based perturbations. This research deepens our understanding of CoT prompting and opens some new questions regarding the capability of LLMs to learn reasoning in context.


title: Results on Elastic Cross Sections in Proton-Proton Collisions at $\sqrt{s} = 510$ GeV with the STAR Detector at RHIC

abstract: We report results on an elastic cross section measurement in proton-proton collisions at a center-of-mass energy $\sqrt{s}=510$ GeV, obtained with the Roman Pot setup of the STAR experiment at the Relativistic Heavy Ion Collider (RHIC). The elastic differential cross section is measured in the four-momentum transfer squared range $0.23 \leq -t \leq 0.67$ GeV$^2$. We find that a constant slope $B$ does not fit the data in the aforementioned $t$ range, and we obtain a much better fit using a second-order polynomial for $B(t)$. The $t$ dependence of $B$ is determined using six subintervals of $t$ in the STAR measured $t$ range, and is in good agreement with the phenomenological models. The measured elastic differential cross section $\mathrm{d}\sigma/\mathrm{dt}$ agrees well with the results obtained at $\sqrt{s} = 546$ GeV for proton--antiproton collisions by the UA4 experiment. We also determine that the integrated elastic cross section within the STAR $t$-range is $\sigma^\mathrm{fid}_\mathrm{el} = 462.1 \pm 0.9 (\mathrm{stat.}) \pm 1.1 (\mathrm {syst.}) \pm 11.6 (\mathrm {scale})$~$\mu\mathrm{b}$.


title: $p$-typical curves on $p$-adic Tate twists and de Rham-Witt forms

abstract: We show that de Rham--Witt forms are naturally isomorphic to $p$-typical curves on $p$-adic Tate twists, which revisits a question of Artin--Mazur pursued in the earlier work of Bloch and Kato. We show this by more generally equipping a related result of Hesselholt on topological cyclic homology with the motivic filtrations introduced by Bhatt--Morrow--Scholze.


title: On generalised majority edge-colourings of graphs

abstract: A $\frac{1}{k}$-majority $l$-edge-colouring of a graph $G$ is a colouring of its edges with $l$ colours such that for every colour $i$ and each vertex $v$ of $G$, at most $\frac{1}{k}$'th of the edges incident with $v$ have colour $i$. We conjecture that for every integer $k\geq 2$, each graph with minimum degree $\delta\geq k^2$ is $\frac{1}{k}$-majority $(k+1)$-edge-colourable and observe that such result would be best possible. This was already known to hold for $k=2$. We support the conjecture by proving it with $2k^2$ instead of $k^2$, which confirms the right order of magnitude of the conjectured optimal lower bound for $\delta$. We at the same time improve the previously known bound of order $k^3\log k$, based on a straightforward probabilistic approach. As this technique seems not applicable towards any further improvement, we use a more direct non-random approach. We also strengthen our result, in particular substituting $2k^2$ by $(\frac{7}{4}+o(1))k^2$. Finally, we provide the proof of the conjecture itself for $k\leq 4$ and completely solve an analogous problem for the family of bipartite graphs.


title: Higgs-Portal Dark Matter in Brane-World Cosmology

abstract: The Higgs-portal scalar dark matter (DM) model is a simple extension of the Standard Model (SM) to incorporate a DM particle to the SM, where a $Z_2$-odd real scalar field is introduced as a DM candidate. We consider this DM model in the context of 5-dimensional brane-world cosmology, where our 3-dimensional space is realized as a hyper-surface embedded in 4-dimensional space. In the setup, all the SM and DM fields reside on the hyper-surface while graviton lives in the bulk. We consider two well-known brane-world cosmologies, namely, the Randall-Sundrum (RS) and the Gauss-Bonnet (GB) brane-world cosmologies, in which the standard Big Bang cosmology is reproduced at low temperatures below the so-called ``transition temperature" while at high temperatures the expansion law of the universe is significantly modified. Such a non-standard expansion law directly impacts the prediction for the relic density of the Higgs-portal DM. We investigate the brane-world cosmological effects and identify the allowed model parameter region by combining the constraints from the observed DM relic density, and the direct and indirect DM detection experiments. It is well-known that only DM masses in the vicinity of half the Higgs boson mass are allowed in the Higgs-portal scalar DM model. We find that the allowed parameter region becomes more severely constrained and even disappears in the RS cosmology, while the GB cosmological effect significantly enlarges the allowed region. Upon discovering Higgs-portal DM, we can determine transition temperature in the GB brane-world cosmology.


title: Chemical evolution of local post-starburst galaxies: Implications for the mass-metallicity relation

abstract: We use the stellar fossil record to constrain the stellar metallicity evolution and star-formation histories of the post-starburst (PSB) regions within 45 local post-starburst galaxies from the MaNGA survey. The direct measurement of the regions' stellar metallicity evolution is achieved by a new two-step metallicity model that allows for stellar metallicity to change at the peak of the starburst. We also employ a Gaussian process noise model that accounts for correlated errors introduced by the observational data reduction or inaccuracies in the models. We find that a majority of PSB regions (69% at $>1\sigma$ significance) increased in stellar metallicity during the recent starburst, with an average increase of 0.8 dex and a standard deviation of 0.4 dex. A much smaller fraction of PSBs are found to have remained constant (22%) or declined in metallicity (9%, average decrease 0.4 dex, standard deviation 0.3 dex). The pre-burst metallicities of the PSB galaxies are in good agreement with the mass-metallicity relation of local star-forming galaxies. These results are consistent with hydrodynamic simulations, which suggest that mergers between gas-rich galaxies are the primary formation mechanism of local PSBs, and rapid metal recycling during the starburst outweighs the impact of dilution by any gas inflows. The final mass-weighted metallicities of the PSB galaxies are consistent with the mass-metallicity relation of local passive galaxies. Our results suggest that rapid quenching following a merger-driven starburst is entirely consistent with the observed gap between the stellar mass-metallicity relations of local star-forming and passive galaxies.


title: Class Activation Map-based Weakly supervised Hemorrhage Segmentation using Resnet-LSTM in Non-Contrast Computed Tomography images

abstract: In clinical settings, intracranial hemorrhages (ICH) are routinely diagnosed using non-contrast CT (NCCT) for severity assessment. Accurate automated segmentation of ICH lesions is the initial and essential step, immensely useful for such assessment. However, compared to other structural imaging modalities such as MRI, in NCCT images ICH appears with very low contrast and poor SNR. Over recent years, deep learning (DL)-based methods have shown great potential, however, training them requires a huge amount of manually annotated lesion-level labels, with sufficient diversity to capture the characteristics of ICH. In this work, we propose a novel weakly supervised DL method for ICH segmentation on NCCT scans, using image-level binary classification labels, which are less time-consuming and labor-efficient when compared to the manual labeling of individual ICH lesions. Our method initially determines the approximate location of ICH using class activation maps from a classification network, which is trained to learn dependencies across contiguous slices. We further refine the ICH segmentation using pseudo-ICH masks obtained in an unsupervised manner. The method is flexible and uses a computationally light architecture during testing. On evaluating our method on the validation data of the MICCAI 2022 INSTANCE challenge, our method achieves a Dice value of 0.55, comparable with those of existing weakly supervised method (Dice value of 0.47), despite training on a much smaller training data.


title: On the Role of 5G and Beyond Sidelink Communication in Multi-Hop Tactical Networks

abstract: This work investigates the potential of 5G and beyond sidelink (SL) communication to support multi-hop tactical networks. We first provide a technical and historical overview of 3GPP SL standardization activities, and then consider applications to current problems of interest in tactical networking. We consider a number of multi-hop routing techniques which are expected to be of interest for SL-enabled multi-hop tactical networking and examine open-source tools useful for network emulation. Finally, we discuss relevant research directions which may be of interest for 5G SL-enabled tactical communications, namely the integration of RF sensing and positioning, as well as emerging machine learning tools such as federated and decentralized learning, which may be of great interest for resource allocation and routing problems that arise in tactical applications. We conclude by summarizing recent developments in the 5G SL literature and provide guidelines for future research.


title: A Deterministic Almost-Linear Time Algorithm for Minimum-Cost Flow

abstract: We give a deterministic $m^{1+o(1)}$ time algorithm that computes exact maximum flows and minimum-cost flows on directed graphs with $m$ edges and polynomially bounded integral demands, costs, and capacities. As a consequence, we obtain the first running time improvement for deterministic algorithms that compute maximum-flow in graphs with polynomial bounded capacities since the work of Goldberg-Rao [J.ACM '98]. Our algorithm builds on the framework of Chen-Kyng-Liu-Peng-Gutenberg-Sachdeva [FOCS '22] that computes an optimal flow by computing a sequence of $m^{1+o(1)}$-approximate undirected minimum-ratio cycles. We develop a deterministic dynamic graph data-structure to compute such a sequence of minimum-ratio cycles in an amortized $m^{o(1)}$ time per edge update. Our key technical contributions are deterministic analogues of the vertex sparsification and edge sparsification components of the data-structure from Chen et al. For the vertex sparsification component, we give a method to avoid the randomness in Chen et al. which involved sampling random trees to recurse on. For the edge sparsification component, we design a deterministic algorithm that maintains an embedding of a dynamic graph into a sparse spanner. We also show how our dynamic spanner can be applied to give a deterministic data structure that maintains a fully dynamic low-stretch spanning tree on graphs with polynomially bounded edge lengths, with subpolynomial average stretch and subpolynomial amortized time per edge update.


title: On Learning with LAD

abstract: The logical analysis of data, LAD, is a technique that yields two-class classifiers based on Boolean functions having disjunctive normal form (DNF) representation. Although LAD algorithms employ optimization techniques, the resulting binary classifiers or binary rules do not lead to overfitting. We propose a theoretical justification for the absence of overfitting by estimating the Vapnik-Chervonenkis dimension (VC dimension) for LAD models where hypothesis sets consist of DNFs with a small number of cubic monomials. We illustrate and confirm our observations empirically.


title: Robust Offline Reinforcement Learning -- Certify the Confidence Interval

abstract: Currently, reinforcement learning (RL), especially deep RL, has received more and more attention in the research area. However, the security of RL has been an obvious problem due to the attack manners becoming mature. In order to defend against such adversarial attacks, several practical approaches are developed, such as adversarial training, data filtering, etc. However, these methods are mostly based on empirical algorithms and experiments, without rigorous theoretical analysis of the robustness of the algorithms. In this paper, we develop an algorithm to certify the robustness of a given policy offline with random smoothing, which could be proven and conducted as efficiently as ones without random smoothing. Experiments on different environments confirm the correctness of our algorithm.


title: Sparse Submodular Function Minimization

abstract: In this paper we study the problem of minimizing a submodular function $f : 2^V \rightarrow \mathbb{R}$ that is guaranteed to have a $k$-sparse minimizer. We give a deterministic algorithm that computes an additive $\epsilon$-approximate minimizer of such $f$ in $\widetilde{O}(\mathsf{poly}(k) \log(|f|/\epsilon))$ parallel depth using a polynomial number of queries to an evaluation oracle of $f$, where $|f| = \max_{S \subseteq V} |f(S)|$. Further, we give a randomized algorithm that computes an exact minimizer of $f$ with high probability using $\widetilde{O}(|V| \cdot \mathsf{poly}(k))$ queries and polynomial time. When $k = \widetilde{O}(1)$, our algorithms use either nearly-constant parallel depth or a nearly-linear number of evaluation oracle queries. All previous algorithms for this problem either use $\Omega(|V|)$ parallel depth or $\Omega(|V|^2)$ queries. In contrast to state-of-the-art weakly-polynomial and strongly-polynomial time algorithms for SFM, our algorithms use first-order optimization methods, e.g., mirror descent and follow the regularized leader. We introduce what we call {\em sparse dual certificates}, which encode information on the structure of sparse minimizers, and both our parallel and sequential algorithms provide new algorithmic tools for allowing first-order optimization methods to efficiently compute them. Correspondingly, our algorithm does not invoke fast matrix multiplication or general linear system solvers and in this sense is more combinatorial than previous state-of-the-art methods.


title: Mixup Your Own Pairs

abstract: In representation learning, regression has traditionally received less attention than classification. Directly applying representation learning techniques designed for classification to regression often results in fragmented representations in the latent space, yielding sub-optimal performance. In this paper, we argue that the potential of contrastive learning for regression has been overshadowed due to the neglect of two crucial aspects: ordinality-awareness and hardness. To address these challenges, we advocate "mixup your own contrastive pairs for supervised contrastive regression", instead of relying solely on real/augmented samples. Specifically, we propose Supervised Contrastive Learning for Regression with Mixup (SupReMix). It takes anchor-inclusive mixtures (mixup of the anchor and a distinct negative sample) as hard negative pairs and anchor-exclusive mixtures (mixup of two distinct negative samples) as hard positive pairs at the embedding level. This strategy formulates harder contrastive pairs by integrating richer ordinal information. Through extensive experiments on six regression datasets including 2D images, volumetric images, text, tabular data, and time-series signals, coupled with theoretical analysis, we demonstrate that SupReMix pre-training fosters continuous ordered representations of regression data, resulting in significant improvement in regression performance. Furthermore, SupReMix is superior to other approaches in a range of regression challenges including transfer learning, imbalanced training data, and scenarios with fewer training samples.


title: End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon

abstract: Most recent work in goal oriented visual navigation resorts to large-scale machine learning in simulated environments. The main challenge lies in learning compact representations generalizable to unseen environments and in learning high-capacity perception modules capable of reasoning on high-dimensional input. The latter is particularly difficult when the goal is not given as a category ("ObjectNav") but as an exemplar image ("ImageNav"), as the perception module needs to learn a comparison strategy requiring to solve an underlying visual correspondence problem. This has been shown to be difficult from reward alone or with standard auxiliary tasks. We address this problem through a sequence of two pretext tasks, which serve as a prior for what we argue is one of the main bottleneck in perception, extremely wide-baseline relative pose estimation and visibility prediction in complex scenes. The first pretext task, cross-view completion is a proxy for the underlying visual correspondence problem, while the second task addresses goal detection and finding directly. We propose a new dual encoder with a large-capacity binocular ViT model and show that correspondence solutions naturally emerge from the training signals. Experiments show significant improvements and SOTA performance on the two benchmarks, ImageNav and the Instance-ImageNav variant, where camera intrinsics and height differ between observation and goal.


title: Analysis of the Usability of Automatically Enriched Cultural Heritage Data

abstract: This chapter presents the potential of interoperability and standardised data publication for cultural heritage resources, with a focus on community-driven approaches and web standards for usability. The Linked Open Usable Data (LOUD) design principles, which rely on JSON-LD as lingua franca, serve as the foundation. We begin by exploring the significant advances made by the International Image Interoperability Framework (IIIF) in promoting interoperability for image-based resources. The principles and practices of IIIF have paved the way for Linked Art, which expands the use of linked data by demonstrating how it can easily facilitate the integration and sharing of semantic cultural heritage data across portals and institutions. To provide a practical demonstration of the concepts discussed, the chapter highlights the implementation of LUX, the Yale Collections Discovery platform. LUX serves as a compelling case study for the use of linked data at scale, demonstrating the real-world application of automated enrichment in the cultural heritage domain. Rooted in empirical study, the analysis presented in this chapter delves into the broader context of community practices and semantic interoperability. By examining the collaborative efforts and integration of diverse cultural heritage resources, the research sheds light on the potential benefits and challenges associated with LOUD.


title: The logarithmic Dirichlet Laplacian on Ahlfors regular spaces

abstract: We introduce the logarithmic analogue of the Laplace-Beltrami operator on Ahlfors regular metric-measure spaces. This operator is intrinsically defined with spectral properties analogous to those of elliptic pseudo-differential operators on Riemannian manifolds. Specifically, its heat semigroup consists of compact operators which are trace-class after some critical point in time. Moreover, its domain is a Banach module over the Dini continuous functions and every H\"older continuous function is a $C^{\infty}$-smooth vector. Finally, the operator is compatible, in the sense of noncommutative geometry, with the action of a large class of non-isometric homeomorphisms.


title: The $p$-Adic Valuation Trees for Quadratic Polynomials for Odd Primes

abstract: We examine the behavior of the sequences of $p$-adic valuations of quadratic polynomials with integer coefficients for an odd prime $p$ through tree representations. Under this representation, a finite tree corresponds to a periodic sequence, and an infinite tree corresponds to an unbounded sequence. We use the polynomial coefficients to determine whether the $p$-adic valuation trees are finite or infinite, the number of infinite branches, the number of levels, the valuations at terminating nodes, and their relationship to the corresponding sequences.


title: Blind spots and biases: the dangers of ignoring eccentricity in gravitational-wave signals from binary black holes

abstract: Most gravitational wave (GW) events observed by the LIGO and Virgo detectors are consistent with mergers of binary black holes (BBHs) on quasi-circular orbits. However, some events are also consistent with non-zero orbital eccentricity, which can indicate that the binary formed via dynamical interactions. Active GW search pipelines using quasi-circular waveform templates are inefficient for detecting eccentric mergers. Also, analysing eccentric GW signals with waveform models neglecting eccentricity can lead to biases in the recovered parameters. We explore the detectability and characterisation of eccentric signals when searches and analyses rely on quasi-circular waveform models. We find that for a reference eccentric population, the fraction of events having fitting factor (FF) $< 0.95$ can be up to $\approx 2.2\%$ compared to $\approx 0.4\%$ for the baseline population. This leads to the loss in signal recovery fraction for up to $6\%$ for parameter space with non-negligible eccentricity ($e_{10} > 0.01$) and high mass ratio ($q > 3$). We perform parameter estimation (PE) for non-spinning and aligned-spin eccentric GW injections from BBHs with a total mass $M=35 M_\odot$, based on numerical relativity simulations and an EOB based inspiral-merger-ringdown model (TEOBResumS). We recover these injections using both quasi-circular and eccentric waveform models. For cases with $e_{20} \sim 0.1$, quasi-circular models fail to estimate chirp mass within the 90% credible interval accurately. Further, for these low-mass injections, spin-induced precession does not mimic eccentricity. For injections of $e_{20}\sim 0.1$, PE conducted with an inspiral-only eccentric waveform model correctly characterises the injected signal to within 90% confidence, and recovers the injected eccentricities, suggesting that such models are sufficient for characterisation of low-mass eccentric BBH. (abridged)


title: MindShift: Leveraging Large Language Models for Mental-States-Based Problematic Smartphone Use Intervention

abstract: Problematic smartphone use negatively affects physical and mental health. Despite the wide range of prior research, existing persuasive techniques are not flexible enough to provide dynamic persuasion content based on users' physical contexts and mental states. We first conducted a Wizard-of-Oz study (N=12) and an interview study (N=10) to summarize the mental states behind problematic smartphone use: boredom, stress, and inertia. This informs our design of four persuasion strategies: understanding, comforting, evoking, and scaffolding habits. We leveraged large language models (LLMs) to enable the automatic and dynamic generation of effective persuasion content. We developed MindShift, a novel LLM-powered problematic smartphone use intervention technique. MindShift takes users' in-the-moment app usage behaviors, physical contexts, mental states, goals \& habits as input, and generates personalized and dynamic persuasive content with appropriate persuasion strategies. We conducted a 5-week field experiment (N=25) to compare MindShift with its simplified version (remove mental states) and baseline techniques (fixed reminder). The results show that MindShift improves intervention acceptance rates by 4.7-22.5% and reduces smartphone usage duration by 7.4-9.8%. Moreover, users have a significant drop in smartphone addiction scale scores and a rise in self-efficacy scale scores. Our study sheds light on the potential of leveraging LLMs for context-aware persuasion in other behavior change domains.


title: Brick wall diagrams as a completely integrable system

abstract: We study the free energy of an integrable, planar, chiral and non-unitary four-dimensional Yukawa theory, the bi-fermion fishnet theory discovered by Pittelli and Preti. The typical Feynman-diagrams of this model are of regular "brick-wall"-type, replacing the regular square lattices of standard fishnet theory. We adapt A. B. Zamolodchikov's powerful classic computation of the thermodynamic free energy of fishnet graphs to the brick-wall case in a transparent fashion, and find the result in closed form. Finally, we briefly discuss two further candidate integrable models in three and six dimensions related to the brick wall model.


title: Anomalous Purcell decay of strongly driven inhomogeneous emitters coupled to a cavity

abstract: We perform resonant fluorescence lifetime measurements on a nanocavity-coupled erbium ensemble as a function of cavity-laser detuning and pump power. Our measurements reveal an anomalous three-fold suppression of the ensemble Purcell factor at zero cavity detuning and high pump fluence. We capture qualitative aspects of this decay rate suppression using a Tavis-Cummings model of non-interacting spins coupled to a common cavity.


title: The steady states of positive reaction-diffusion equations with Dirichlet conditions

abstract: We study the uniqueness of reaction-diffusion steady states in general domains with Dirichlet boundary data. Here we consider "positive" (monostable) reactions. We describe geometric conditions on the domain that ensure uniqueness and we provide complementary examples of nonuniqueness. Along the way, we formulate a number of open problems and conjectures. To derive our results, we develop a general framework, the stable-compact method, to study qualitative properties of nonlinear elliptic equations.


title: Deep Geometrized Cartoon Line Inbetweening

abstract: We aim to address a significant but understudied problem in the anime industry, namely the inbetweening of cartoon line drawings. Inbetweening involves generating intermediate frames between two black-and-white line drawings and is a time-consuming and expensive process that can benefit from automation. However, existing frame interpolation methods that rely on matching and warping whole raster images are unsuitable for line inbetweening and often produce blurring artifacts that damage the intricate line structures. To preserve the precision and detail of the line drawings, we propose a new approach, AnimeInbet, which geometrizes raster line drawings into graphs of endpoints and reframes the inbetweening task as a graph fusion problem with vertex repositioning. Our method can effectively capture the sparsity and unique structure of line drawings while preserving the details during inbetweening. This is made possible via our novel modules, i.e., vertex geometric embedding, a vertex correspondence Transformer, an effective mechanism for vertex repositioning and a visibility predictor. To train our method, we introduce MixamoLine240, a new dataset of line drawings with ground truth vectorization and matching labels. Our experiments demonstrate that AnimeInbet synthesizes high-quality, clean, and complete intermediate line drawings, outperforming existing methods quantitatively and qualitatively, especially in cases with large motions. Data and code are available at https://github.com/lisiyao21/AnimeInbet.


title: Optical and microstructural characterization of Er$^{3+}$ doped epitaxial cerium oxide on silicon

abstract: Rare-earth ion dopants in solid-state hosts are ideal candidates for quantum communication technologies such as quantum memory, due to the intrinsic spin-photon interface of the rare-earth ion combined with the integration methods available in the solid-state. Erbium-doped cerium oxide (Er:CeO$_2$) is a particularly promising platform for such a quantum memory, as it combines the telecom-wavelength (~1.5 $\mu$m) 4f-4f transition of erbium, a predicted long electron spin coherence time supported by CeO$_2$, and is also near lattice-matched to silicon for heteroepitaxial growth. In this work, we report on the epitaxial growth of Er:CeO$_2$ thin films on silicon using molecular beam epitaxy (MBE), with controlled erbium concentration down to 2 parts per million (ppm). We carry out a detailed microstructural study to verify the CeO$_2$ host structure, and characterize the spin and optical properties of the embedded Er$^{3+}$ ions. In the 2-3 ppm Er regime, we identify EPR linewidths of 245(1) MHz, optical inhomogeneous linewidths of 9.5(2) GHz, optical excited state lifetimes of 3.5(1) ms, and spectral diffusion-limited homogenoeus linewidths as narrow as 4.8(3) MHz in the as-grown material. We test annealing of the Er:CeO$_2$ films up to 900 deg C, which yields modest narrowing of the inhomogeneous linewidth by 20% and extension of the excited state lifetime by 40%. We have also studied the variation of the optical properties as a function of Er doping and find that the results are consistent with the trends expected from inter-dopant charge interactions.


title: Reusability report: Prostate cancer stratification with diverse biologically-informed neural architectures

abstract: In Elmarakeby et al., "Biologically informed deep neural network for prostate cancer discovery", a feedforward neural network with biologically informed, sparse connections (P-NET) was presented to model the state of prostate cancer. We verified the reproducibility of the study conducted by Elmarakeby et al., using both their original codebase, and our own re-implementation using more up-to-date libraries. We quantified the contribution of network sparsification by Reactome biological pathways, and confirmed its importance to P-NET's superior performance. Furthermore, we explored alternative neural architectures and approaches to incorporating biological information into the networks. We experimented with three types of graph neural networks on the same training data, and investigated the clinical prediction agreement between different models. Our analyses demonstrated that deep neural networks with distinct architectures make incorrect predictions for individual patient that are persistent across different initializations of a specific neural architecture. This suggests that different neural architectures are sensitive to different aspects of the data, an important yet under-explored challenge for clinical prediction tasks.


title: Improving Equivariance in State-of-the-Art Supervised Depth and Normal Predictors

abstract: Dense depth and surface normal predictors should possess the equivariant property to cropping-and-resizing -- cropping the input image should result in cropping the same output image. However, we find that state-of-the-art depth and normal predictors, despite having strong performances, surprisingly do not respect equivariance. The problem exists even when crop-and-resize data augmentation is employed during training. To remedy this, we propose an equivariant regularization technique, consisting of an averaging procedure and a self-consistency loss, to explicitly promote cropping-and-resizing equivariance in depth and normal networks. Our approach can be applied to both CNN and Transformer architectures, does not incur extra cost during testing, and notably improves the supervised and semi-supervised learning performance of dense predictors on Taskonomy tasks. Finally, finetuning with our loss on unlabeled images improves not only equivariance but also accuracy of state-of-the-art depth and normal predictors when evaluated on NYU-v2. GitHub link: https://github.com/mikuhatsune/equivariance


title: A generalization of immanants based on partition algebra characters

abstract: We introduce a generalization of immanants of matrices, using partition algebra characters in place of symmetric group characters. We prove that our immanant-like function on square matrices, which we refer to as the recombinant, agrees with the usual definition for immanants for the special case whereby the vacillating tableaux associated with the irreducible characters correspond, according to the Bratteli diagram for partition algebra representations, to the integer partition shapes for symmetric group characters. In contrast to previously studied variants and generalizations of immanants, as in Temperley-Lieb immanants and $f$-immanants, the sum that we use to define recombinants is indexed by a full set of partition diagrams, as opposed to permutations.


title: Hausdorff vs Gromov-Hausdorff distances

abstract: Let $M$ be a closed Riemannian manifold and let $X\subseteq M$. If the sample $X$ is sufficiently dense relative to the curvature of $M$, then the Gromov--Hausdorff distance between $X$ and $M$ is bounded from below by half their Hausdorff distance, namely $d_{GH}(X,M) \ge \frac{1}{2} d_H(X,M)$. The constant $\frac{1}{2}$ can be improved depending on the dimension and curvature of the manifold $M$, and obtains the optimal value $1$ in the case of the unit circle, meaning that if $X\subseteq S^1$ satisfies $d_{GH}(X,S^1)<\tfrac{\pi}{6}$, then $d_{GH}(X,S^1)=d_H(X,S^1)$. We also provide versions lower bounding the Gromov--Hausdorff distance $d_{GH}(X,Y)$ between two subsets $X,Y\subseteq M$. Our proofs convert discontinuous functions between metric spaces into simplicial maps between \v{C}ech or Vietoris--Rips complexes. We then produce topological obstructions to the existence of certain maps using the nerve lemma and the fundamental class of the manifold.


title: FLIP: Cross-domain Face Anti-spoofing with Language Guidance

abstract: Face anti-spoofing (FAS) or presentation attack detection is an essential component of face recognition systems deployed in security-critical applications. Existing FAS methods have poor generalizability to unseen spoof types, camera sensors, and environmental conditions. Recently, vision transformer (ViT) models have been shown to be effective for the FAS task due to their ability to capture long-range dependencies among image patches. However, adaptive modules or auxiliary loss functions are often required to adapt pre-trained ViT weights learned on large-scale datasets such as ImageNet. In this work, we first show that initializing ViTs with multimodal (e.g., CLIP) pre-trained weights improves generalizability for the FAS task, which is in line with the zero-shot transfer capabilities of vision-language pre-trained (VLP) models. We then propose a novel approach for robust cross-domain FAS by grounding visual representations with the help of natural language. Specifically, we show that aligning the image representation with an ensemble of class descriptions (based on natural language semantics) improves FAS generalizability in low-data regimes. Finally, we propose a multimodal contrastive learning strategy to boost feature generalization further and bridge the gap between source and target domains. Extensive experiments on three standard protocols demonstrate that our method significantly outperforms the state-of-the-art methods, achieving better zero-shot transfer performance than five-shot transfer of adaptive ViTs. Code: https://github.com/koushiksrivats/FLIP


title: ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

abstract: For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )


title: Diffusion coefficients preserving long-time correlations: Consequences on the Einstein relation and on entanglement in a bosonic Bogoliubov system

abstract: We analytically derive the diffusion coefficients that drive a system of $N$ coupled harmonic oscillators to an equilibrium state exhibiting persistent correlations. It is shown that the main effect of the latter consists in a renormalization of the natural frequencies and the friction coefficients of the oscillators. We find that the Einstein relation may be satisfied at low temperatures with frequency-dependent effective friction coefficients, provided that the physical constraints are fulfilled. We also investigate the entanglement evolution in a bipartite bosonic Bogoliubov system initially prepared in a thermal squeezed state. It is found that, in contrast to what one may expect, strong coupling slows down the entanglement sudden death, and for initially separable states, entanglement generation may occur.


title: Perceiving Extrinsic Contacts from Touch Improves Learning Insertion Policies

abstract: Robotic manipulation tasks such as object insertion typically involve interactions between object and environment, namely extrinsic contacts. Prior work on Neural Contact Fields (NCF) use intrinsic tactile sensing between gripper and object to estimate extrinsic contacts in simulation. However, its effectiveness and utility in real-world tasks remains unknown. In this work, we improve NCF to enable sim-to-real transfer and use it to train policies for mug-in-cupholder and bowl-in-dishrack insertion tasks. We find our model NCF-v2, is capable of estimating extrinsic contacts in the real-world. Furthermore, our insertion policy with NCF-v2 outperforms policies without it, achieving 33% higher success and 1.36x faster execution on mug-in-cupholder, and 13% higher success and 1.27x faster execution on bowl-in-dishrack.


title: DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation

abstract: Recent advances in 3D content creation mostly leverage optimization-based 3D generation via score distillation sampling (SDS). Though promising results have been exhibited, these methods often suffer from slow per-sample optimization, limiting their practical usage. In this paper, we propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks. To further enhance the texture quality and facilitate downstream applications, we introduce an efficient algorithm to convert 3D Gaussians into textured meshes and apply a fine-tuning stage to refine the details. Extensive experiments demonstrate the superior efficiency and competitive generation quality of our proposed approach. Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes from a single-view image, achieving approximately 10 times acceleration compared to existing methods.


title: Novel Deep Learning Pipeline for Automatic Weapon Detection

abstract: Weapon and gun violence have recently become a pressing issue today. The degree of these crimes and activities has risen to the point of being termed as an epidemic. This prevalent misuse of weapons calls for an automatic system that detects weapons in real-time. Real-time surveillance video is captured and recorded in almost all public forums and places. These videos contain abundant raw data which can be extracted and processed into meaningful information. This paper proposes a novel pipeline consisting of an ensemble of convolutional neural networks with distinct architectures. Each neural network is trained with a unique mini-batch with little to no overlap in the training samples. This paper will present several promising results using multiple datasets associated with comparing the proposed architecture and state-of-the-art (SoA) models. The proposed pipeline produced an average increase of 5% in accuracy, specificity, and recall compared to the SoA systems.


title: Monogamy of highly symmetric states

abstract: We study the question of how highly entangled two particles can be when also entangled in a similar way with other particles on the complete graph for the case of Werner, isotropic and Brauer states. In order to do so we solve optimization problems motivated by many-body physics, computational complexity and quantum cryptography. We formalize our question as a semi-definite program and then solve this optimization problem analytically, using tools from representation theory. In particular, we determine the exact maximum values of the projection to the maximally entangled state and antisymmetric Werner state possible, solving long-standing open problems. We find these optimal values by use of SDP duality and representation theory of the symmetric and orthogonal groups, and the Brauer algebra.


title: Visual In-Context Learning for Few-Shot Eczema Segmentation

abstract: Automated diagnosis of eczema from digital camera images is crucial for developing applications that allow patients to self-monitor their recovery. An important component of this is the segmentation of eczema region from such images. Current methods for eczema segmentation rely on deep neural networks such as convolutional (CNN)-based U-Net or transformer-based Swin U-Net. While effective, these methods require high volume of annotated data, which can be difficult to obtain. Here, we investigate the capabilities of visual in-context learning that can perform few-shot eczema segmentation with just a handful of examples and without any need for retraining models. Specifically, we propose a strategy for applying in-context learning for eczema segmentation with a generalist vision model called SegGPT. When benchmarked on a dataset of annotated eczema images, we show that SegGPT with just 2 representative example images from the training dataset performs better (mIoU: 36.69) than a CNN U-Net trained on 428 images (mIoU: 32.60). We also discover that using more number of examples for SegGPT may in fact be harmful to its performance. Our result highlights the importance of visual in-context learning in developing faster and better solutions to skin imaging tasks. Our result also paves the way for developing inclusive solutions that can cater to minorities in the demographics who are typically heavily under-represented in the training data.


title: Asymptotically Optimal Sequential Multiple Testing Procedures for Correlated Normal

abstract: Simultaneous statistical inference has been a cornerstone in the statistics methodology literature because of its fundamental theory and paramount applications. The mainstream multiple testing literature has traditionally considered two frameworks: the sample size is deterministic, and the test statistics corresponding to different tests are independent. However, in many modern scientific avenues, these assumptions are often violated. There is little study that explores the multiple testing problem in a sequential framework where the test statistics corresponding to the various streams are dependent. This work fills this gap in a unified way by considering the classical means-testing problem in an equicorrelated Gaussian and sequential framework. We focus on sequential test procedures that control the type I and type II familywise error probabilities at pre-specified levels. We establish that our proposed test procedures achieve the optimal expected sample sizes under every possible signal configuration asymptotically, as the two error probabilities vanish at arbitrary rates. Towards this, we elucidate that the ratio of the expected sample size of our proposed rule and that of the classical SPRT goes to one asymptotically, thus illustrating their connection. Generalizing this, we show that our proposed procedures, with appropriately adjusted critical values, are asymptotically optimal for controlling any multiple testing error metric lying between multiples of FWER in a certain sense. This class of metrics includes FDR/FNR, pFDR/pFNR, the per-comparison and per-family error rates, and the false positive rate.


title: Aspects of $T\bar{T}+J\bar{T }$ deformed 2D topological gravity : from partition function to late-time SFF

abstract: In this paper, we investigate different thermodynamic properties of $T\bar{T}+J\bar{T }$ deformed 2D-gravity. First, we compute the partition function of $U(1)$ coupled 2D-gravity with fixed chemical potential, obtained from the dimensional reduction of the four-dimensional Einstein-Maxwell theory. Then, we compute the partition function of the deformed theory and study the genus expansion of the one and two-point correlation function of the partition function of the theory. Subsequently, we use the one-point function to compute the ``Annealed'' and ``Quenched'' free energy in low-temperature limits and make a qualitative comparison with the undeformed theory. Then, using the two-point function, we compute the Spectral Form Factor of the deformed theory in early and late time. We find a dip and ramp structure in early and late time, respectively. We also get a plateau structure in the $\tau$-scaling limit. Last but not least, we comment on the late-time topology change to give a physical interpretation of the ramp of the Spectral Form Factor for our theory.


title: The Eccentric Kozai-Lidov Mechanism as the Cause of Exocomet Transits of KIC 8462852

abstract: KIC 8462852 is a star in the Kepler field that exhibits almost unique behaviour. The deep, irregular and aperiodic dips in its light curve have been interpreted as the breakup of a large exocomet on a highly eccentric orbit whose post-disruption material obscures the star. It is hypothesised that a nearby M-dwarf, recently confirmed to be bound to the system, could be exciting planetesimals in a source belt to high eccentricities if its orbit is highly misaligned with the belt: an effect known as the 'Eccentric Kozai-Lidov Mechanism'. To quantify how often this effect is expected to occur, this paper presents a Monte Carlo model of wide binary stars with embedded, misaligned planetesimal belts. These belts collisionally erode over time until they are excited to high eccentricities on secular timescales by a companion star if its orbit is sufficiently misaligned. The large planetesimals then produce an observable dimming signature in the light curve for a set period of time which may or may not overlap with similar events. The model finds that, for dimming events that persist for 100 yr, the most likely companion stars are located at $10^2 - 10^4$ au, the most likely belts are at $10^2-10^3$ au and the system age is most likely to be $10^2 - 10^3$ Myr. However, the probability of observing one or more stars exhibiting this phenomenon in the Kepler field is $1.3 \times 10^{-3}$, such that it is unlikely this mechanism is driving the observations of KIC 8462852.


title: Folding QQ-relations and transfer matrix eigenvalues: towards a unified approach to Bethe ansatz for super spin chains

abstract: Extending the method proposed in [arXiv:1109.5524], we derive QQ-relations (functional relations among Baxter Q-functions) and T-functions (eigenvalues of transfer matrices) for fusion vertex models associated with the twisted quantum affine superalgebras $U_{q}(gl(2r+1|2s)^{(2)})$, $U_{q}(gl(2r|2s+1)^{(2)})$, $U_{q}(gl(2r|2s)^{(2)})$, $U_{q}(osp(2r|2s)^{(2)})$ and the non-twisted quantum affine orthosymplectic superalgebras $U_{q}(osp(2r+1|2s)^{(1)})$ and $U_{q}(osp(2r|2s)^{(1)})$ (and their Yangian counterparts, $Y(osp(2r+1|2s))$ and $Y(osp(2r|2s))$) as reductions (a kind of folding) of those for associated with $U_{q}(gl(M|N)^{(1)})$. In particular, we reproduce previously proposed generating functions (difference operators) of the T-functions for the symmetric or anti-symmetric representations, and tableau sum expressions for more general representations for orthosymplectic superalgebras [arXiv:0911.5393,arXiv:0911.5390], and obtain Wronskian-type expressions (analogues of Weyl-type character formulas) for them. T-functions for spinorial representations are related to reductions of those for asymptotic limits of typical representations of $U_{q}(gl(M|N)^{(1)})$.


title: SA2-Net: Scale-aware Attention Network for Microscopic Image Segmentation

abstract: Microscopic image segmentation is a challenging task, wherein the objective is to assign semantic labels to each pixel in a given microscopic image. While convolutional neural networks (CNNs) form the foundation of many existing frameworks, they often struggle to explicitly capture long-range dependencies. Although transformers were initially devised to address this issue using self-attention, it has been proven that both local and global features are crucial for addressing diverse challenges in microscopic images, including variations in shape, size, appearance, and target region density. In this paper, we introduce SA2-Net, an attention-guided method that leverages multi-scale feature learning to effectively handle diverse structures within microscopic images. Specifically, we propose scale-aware attention (SA2) module designed to capture inherent variations in scales and shapes of microscopic regions, such as cells, for accurate segmentation. This module incorporates local attention at each level of multi-stage features, as well as global attention across multiple resolutions. Furthermore, we address the issue of blurred region boundaries (e.g., cell boundaries) by introducing a novel upsampling strategy called the Adaptive Up-Attention (AuA) module. This module enhances the discriminative ability for improved localization of microscopic regions using an explicit attention mechanism. Extensive experiments on five challenging datasets demonstrate the benefits of our SA2-Net model. Our source code is publicly available at \url{https://github.com/mustansarfiaz/SA2-Net}.


title: Geodesic Regression Characterizes 3D Shape Changes in the Female Brain During Menstruation

abstract: Women are at higher risk of Alzheimer's and other neurological diseases after menopause, and yet research connecting female brain health to sex hormone fluctuations is limited. We seek to investigate this connection by developing tools that quantify 3D shape changes that occur in the brain during sex hormone fluctuations. Geodesic regression on the space of 3D discrete surfaces offers a principled way to characterize the evolution of a brain's shape. However, in its current form, this approach is too computationally expensive for practical use. In this paper, we propose approximation schemes that accelerate geodesic regression on shape spaces of 3D discrete surfaces. We also provide rules of thumb for when each approximation can be used. We test our approach on synthetic data to quantify the speed-accuracy trade-off of these approximations and show that practitioners can expect very significant speed-up while only sacrificing little accuracy. Finally, we apply the method to real brain shape data and produce the first characterization of how the female hippocampus changes shape during the menstrual cycle as a function of progesterone: a characterization made (practically) possible by our approximation schemes. Our work paves the way for comprehensive, practical shape analyses in the fields of bio-medicine and computer vision. Our implementation is publicly available on GitHub: https://github.com/bioshape-lab/my28brains.


title: HyperPPO: A scalable method for finding small policies for robotic control

abstract: Models with fewer parameters are necessary for the neural control of memory-limited, performant robots. Finding these smaller neural network architectures can be time-consuming. We propose HyperPPO, an on-policy reinforcement learning algorithm that utilizes graph hypernetworks to estimate the weights of multiple neural architectures simultaneously. Our method estimates weights for networks that are much smaller than those in common-use networks yet encode highly performant policies. We obtain multiple trained policies at the same time while maintaining sample efficiency and provide the user the choice of picking a network architecture that satisfies their computational constraints. We show that our method scales well - more training resources produce faster convergence to higher-performing architectures. We demonstrate that the neural policies estimated by HyperPPO are capable of decentralized control of a Crazyflie2.1 quadrotor. Website: https://sites.google.com/usc.edu/hyperppo


title: Associativity certificates for Kontsevich's star-product $\star$ mod $\bar{o}(\hbar^k)$: $k\leqslant 6$ unlike $k\geqslant7$

abstract: The formula $\star$ mod $\bar{o}(\hbar^k)$ of Kontsevich's star-product with harmonic propagators was known in full at $\hbar^{k\leqslant 6}$ since 2018 for generic Poisson brackets, and since 2022 also at $k=7$ for affine brackets. We discover that the mechanism of associativity for the star-product up to $\bar{o}(\hbar^6)$ is different from the mechanism at order $7$ for both the full star-product and the affine star-product. Namely, at lower orders the needed consequences of the Jacobi identity are immediately obtained from the associator mod $\bar{o}(\hbar^6)$, whereas at order $\hbar^7$ and higher, some of the necessary differential consequences are reached from the Kontsevich graphs in the associator in strictly more than one step.


title: Modular quantum signal processing in many variables

abstract: Despite significant advances in quantum algorithms, quantum programs in practice are often expressed at the circuit level, forgoing helpful structural abstractions common to their classical counterparts. Consequently, as many quantum algorithms have been unified with the advent of quantum signal processing (QSP) and quantum singular value transformation (QSVT), an opportunity has appeared to cast these algorithms as modules that can be combined to constitute complex programs. Complicating this, however, is that while QSP/QSVT are often described by the polynomial transforms they apply to the singular values of large linear operators, and the algebraic manipulation of polynomials is simple, the QSP/QSVT protocols realizing analogous manipulations of their embedded polynomials are non-obvious. Here we provide a theory of modular multi-input-output QSP-based superoperators, the basic unit of which we call a gadget, and show they can be snapped together with LEGO-like ease at the level of the functions they apply. To demonstrate this ease, we also provide a Python package for assembling gadgets and compiling them to circuits. Viewed alternately, gadgets both enable the efficient block encoding of large families of useful multivariable functions, and substantiate a functional-programming approach to quantum algorithm design in recasting QSP and QSVT as monadic types.


title: Probing Parity Violation in the Stochastic Gravitational Wave Background with Astrometry

abstract: Astrometry holds the potential for testing fundamental physics through the effects of the Stochastic Gravitational Wave Background (SGWB) in the $\sim 1-100$ nHz frequency band on precision measurements of stellar positions. Such measurements are complementary to tests made possible by the detection of the SGWB using Pulsar Timing Arrays. Here, the feasibility of using astrometry for the identification of parity-violating signals within the SGWB is investigated. This is achieved by defining and quantifying a non-vanishing $EB$ correlation function within astrometric correlation functions, and investigating how one might estimate the detectability of such signals.


title: Subconvexity for $L$-functions on ${\rm U}(n) \times {\rm U}(n+1)$ in the depth aspect

abstract: Let $E/F$ be a CM extension of number fields, and let $H < G$ be a unitary Gan--Gross--Prasad pair defined with respect to $E/F$ that is compact at infinity. We consider a family $\mathcal{F}$ of automorphic representations of $G \times H$ that is varying at a finite place $w$ that splits in $E/F$. We assume that the representations in $\mathcal{F}$ satisfy certain conditions, including being tempered and distinguished by the GGP period. For a representation $\pi \times \pi_H \in \mathcal{F}$ with base change $\Pi \times \Pi_H$ to ${\rm GL}_{n+1}(E) \times {\rm GL}_n(E)$, we prove a subconvex bound \[ L(1/2, \Pi \times \Pi_H^\vee) \ll C(\Pi \times \Pi_H^\vee)^{1/4 - \delta} \] for any $\delta < \tfrac{1}{4n(n+1)(2n^2 + 3n + 3)}$. Our proof uses the unitary Ichino--Ikeda period formula to relate the central $L$-value to an automorphic period, before bounding that period using the amplification method of Iwaniec--Sarnak.


title: RealFill: Reference-Driven Generation for Authentic Image Completion

abstract: Recent advances in generative imagery have brought forth outpainting and inpainting models that can produce high-quality, plausible image content in unknown regions. However, the content these models hallucinate is necessarily inauthentic, since they are unaware of the true scene. In this work, we propose RealFill, a novel generative approach for image completion that fills in missing regions of an image with the content that should have been there. RealFill is a generative inpainting model that is personalized using only a few reference images of a scene. These reference images do not have to be aligned with the target image, and can be taken with drastically varying viewpoints, lighting conditions, camera apertures, or image styles. Once personalized, RealFill is able to complete a target image with visually compelling contents that are faithful to the original scene. We evaluate RealFill on a new image completion benchmark that covers a set of diverse and challenging scenarios, and find that it outperforms existing approaches by a large margin. Project page: https://realfill.github.io


title: Training a Large Video Model on a Single Machine in a Day

abstract: Videos are big, complex to pre-process, and slow to train on. State-of-the-art large-scale video models are trained on clusters of 32 or more GPUs for several days. As a consequence, academia largely ceded the training of large video models to industry. In this paper, we show how to still train a state-of-the-art video model on a single machine with eight consumer-grade GPUs in a day. We identify three bottlenecks, IO, CPU, and GPU computation, and optimize each. The result is a highly efficient video training pipeline. For comparable architectures, our pipeline achieves higher accuracies with $\frac{1}{8}$ of the computation compared to prior work. Code is available at https://github.com/zhaoyue-zephyrus/AVION.


title: Decaf: Monocular Deformation Capture for Face and Hand Interactions

abstract: Existing methods for 3D tracking from monocular RGB videos predominantly consider articulated and rigid objects. Modelling dense non-rigid object deformations in this setting remained largely unaddressed so far, although such effects can improve the realism of the downstream applications such as AR/VR and avatar communications. This is due to the severe ill-posedness of the monocular view setting and the associated challenges. While it is possible to naively track multiple non-rigid objects independently using 3D templates or parametric 3D models, such an approach would suffer from multiple artefacts in the resulting 3D estimates such as depth ambiguity, unnatural intra-object collisions and missing or implausible deformations. Hence, this paper introduces the first method that addresses the fundamental challenges depicted above and that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system. As a pivotal step in its creation, we process the reconstructed raw 3D shapes with position-based dynamics and an approach for non-uniform stiffness estimation of the head tissues, which results in plausible annotations of the surface deformations, hand-face contact regions and head-hand positions. At the core of our neural approach are a variational auto-encoder supplying the hand-face depth prior and modules that guide the 3D tracking by estimating the contacts and the deformations. Our final 3D hand and face reconstructions are realistic and more plausible compared to several baselines applicable in our setting, both quantitatively and qualitatively. https://vcai.mpi-inf.mpg.de/projects/Decaf


title: Demystifying CLIP Data

abstract: Contrastive Language-Image Pre-training (CLIP) is an approach that has advanced research and applications in computer vision, fueling modern recognition systems and generative models. We believe that the main ingredient to the success of CLIP is its data and not the model architecture or pre-training objective. However, CLIP only provides very limited information about its data and how it has been collected, leading to works that aim to reproduce CLIP's data by filtering with its model parameters. In this work, we intend to reveal CLIP's data curation approach and in our pursuit of making it open to the community introduce Metadata-Curated Language-Image Pre-training (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution. Our experimental study rigorously isolates the model and training settings, concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M image-text data pairs outperforms CLIP's data on multiple standard benchmarks. In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy, surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining the same training budget, attains 72.4%. Our observations hold across various model sizes, exemplified by ViT-H achieving 80.5%, without any bells-and-whistles. Curation code and training data distribution on metadata is made available at https://github.com/facebookresearch/MetaCLIP.


title: Learning to Transform for Generalizable Instance-wise Invariance

abstract: Computer vision research has long aimed to build systems that are robust to spatial transformations found in natural data. Traditionally, this is done using data augmentation or hard-coding invariances into the architecture. However, too much or too little invariance can hurt, and the correct amount is unknown a priori and dependent on the instance. Ideally, the appropriate invariance would be learned from data and inferred at test-time. We treat invariance as a prediction problem. Given any image, we use a normalizing flow to predict a distribution over transformations and average the predictions over them. Since this distribution only depends on the instance, we can align instances before classifying them and generalize invariance across classes. The same distribution can also be used to adapt to out-of-distribution poses. This normalizing flow is trained end-to-end and can learn a much larger range of transformations than Augerino and InstaAug. When used as data augmentation, our method shows accuracy and robustness gains on CIFAR 10, CIFAR10-LT, and TinyImageNet.