title: Whittaker vectors at finite energy scale, topological recursion and Hurwitz numbers

abstract: We upgrade the results of Borot--Bouchard--Chidambaram--Creutzig to show that the Gaiotto vector in $\mathcal{N} = 2$ pure supersymmetric gauge theory admits an analytic continuation with respect to the energy scale (which can therefore be taken to be finite, instead of infinitesimal), and is computed by topological recursion on the (ramified) UV or Gaiotto spectral curve. This has a number of interesting consequences for the Gaiotto vector: relations to intersection theory on $\overline{\mathcal{M}}_{g,n}$ in at least two different ways, Hurwitz numbers, quantum curves, and (almost complete) description of the correlators as analytic functions of $\hbar$ (instead of formal series). The same method is used to establish analogous results for the more general Whittaker vector constructed in the recent work of Chidambaram--Dolega--Osuga.


title: $p$-Adic hypergeometric functions and certain weight three newforms

abstract: For an odd prime $p$ and a positive integer $n$, let ${_n}G_n[\cdots]_p$ denote McCarthy's $p$-adic hypergeometric function. In this article, we prove $p$-adic analogue of certain classical hypergeometric identities and using these identities we express the $p$-th Fourier coefficient of certain weight three newforms in terms of special values of ${_3}G_3[\cdots]_p$. Rodriguez-Villegas conjectured certain supercongruences between values of truncated hypergeometric series and the $p$-th Fourier coefficients of these newforms. As a consequence of our main results, we obtain another proof of these supercongruences which were earlier proved by Mortenson and Sun.


title: Dynamics of Affective Polarization: From Consensus to Partisan Divides

abstract: Politically divided societies are also often divided emotionally: people like and trust those with similar political views (in-group favoritism) while disliking and distrusting those with different views (out-group animosity). This phenomenon, called affective polarization, influences individual decisions, including seemingly apolitical choices such as whether to wear a mask or what car to buy. We present a dynamical model of decision-making in an affectively polarized society, identifying three potential global outcomes separated by a sharp boundary in the parameter space: consensus, partisan polarization, and non-partisan polarization. Analysis reveals that larger out-group animosity compared to in-group favoritism, i.e. more hate than love, is sufficient for polarization, while larger in-group favoritism compared to out-group animosity, i.e., more love than hate, is necessary for consensus. We also show that, counter-intuitively, increasing cross-party connections facilitates polarization, and that by emphasizing partisan differences, mass media creates self-fulfilling prophecies that lead to polarization. Affective polarization also creates tipping points in the opinion landscape where one group suddenly reverses their trends. Our findings aid in understanding and addressing the cascading effects of affective polarization, offering insights for strategies to mitigate polarization.


title: SPACE-IDEAS: A Dataset for Salient Information Detection in Space Innovation

abstract: Detecting salient parts in text using natural language processing has been widely used to mitigate the effects of information overflow. Nevertheless, most of the datasets available for this task are derived mainly from academic publications. We introduce SPACE-IDEAS, a dataset for salient information detection from innovation ideas related to the Space domain. The text in SPACE-IDEAS varies greatly and includes informal, technical, academic and business-oriented writing styles. In addition to a manually annotated dataset we release an extended version that is annotated using a large generative language model. We train different sentence and sequential sentence classifiers, and show that the automatically annotated dataset can be leveraged using multitask learning to train better classifiers.


title: Generic gravito-magnetic clock effects

abstract: General relativity predicts that two counter-orbiting clocks around a spinning mass differ in the time required to complete the same orbit. The difference in these two values for the orbital period is generally referred to as the gravito-magnetic (GM) clock effect. It has been proposed to measure the GM clock effect using atomic clocks carried by satellites in prograde and retrograde orbits around the Earth. The precision and stability required for satellites to accurately perform this measurement remains a challenge for current instrumentation. One of the most accurate clocks in the Universe is a millisecond pulsar, which emits periodic radio pulses with high stability. Timing of the pulsed signals from millisecond pulsars has proven to be very successful in testing predictions of general relativity and the GM clock effect is potentially measurable in binary systems. In this work we derive the generic GM clock effect by considering a slowly-spinning binary system on an elliptical orbit, with both arbitrary mass ratio and arbitrary spin orientations. The spin-orbit interaction introduces a perturbation to the orbit, causing the orbital plane to precess and nutate. We identify several different contributions to the clock effects: the choice of spin supplementary condition and the observer-dependent definition of a full revolution and "nearly-identical" orbits. We discuss the impact of these subtle definitions on the formula for GM clock effects and show that most of the existing formulae in the literature can be recovered under appropriate assumptions.


title: Raman spectroscopy of Na$_3$Co$_2$SbO$_6$

abstract: Raman spectroscopy together with density functional calculations were used to study lattice dynamics in a layered honeycomb cobaltite Na$_3$Co$_2$SbO$_6$, which can host a field-induced phase related with the Kitaev physics. We show that there develops an additional mode well above Neel temperature (at $\approx 200$K) at 525 cm$^{-1}$, which origin can be related to an electronic excitation to one of $j_{3/2}$ doublets. Moreover, our theoretical calculations demonstrate that the highest frequency intensive mode related to the oxygen vibrations is very sensitive to type of the magnetic order. Thus, we propose to use the softening of this mode as a hallmark of the transition to a fully polarized regime, which is stabilized in Kitaev materials in strong magnetic fields.


title: The Tarantula -- Revealed by X-rays (T-ReX)

abstract: The Tarantula Nebula (30 Doradus) is the most important star-forming complex in the Local Group, offering a microscope on starburst astrophysics. At its heart lies the exceptionally rich young stellar cluster R136, containing the most massive stars known. Stellar winds and supernovae have carved 30 Dor into an amazing display of arcs, pillars, and bubbles. We present first results and advanced data processing products from the 2-Ms Chandra X-ray Visionary Project, "The Tarantula - Revealed by X-rays" (T-ReX). The 3615 point sources in the T-ReX catalog include massive stars, compact objects, binaries, bright pre-main-sequence stars and compact young stellar (sub)clusters in 30 Dor. After removing point sources and excluding the exceptionally bright supernova remnant N157B (30 Dor B), the global diffuse X-ray maps reveal hot plasma structures resolved at 1-10 pc scales, with an absorption-corrected total-band (0.5-7 keV) X-ray luminosity of $2.110\times 10^{37}$ erg s$^{-1}$. Spatially-resolved spectral modeling provides evidence for emission lines enhanced by charge-exchange processes at the interfaces. We identify a candidate for the oldest X-ray pulsar detected to date in 30 Dor, PSR J0538-6902, inside a newly-resolved arctuate X-ray wind nebula, the Manta Ray. The long time baseline of T-ReX monitored dozens of massive stars, several showing periodic variability tied to binary orbital periods, and captured strong flares from at least three low-mass Galactic foreground stars.


title: New evaluations of inverse binomial series via cyclotomic multiple zeta values

abstract: Through the application of an evaluation technique based on cyclotomic multiple zeta values recently due to Au, we solve open problems on inverse binomial series that were included in a 2010 analysis textbook by Chen.


title: Some Findings from the Longitudinal Migration of Starspots

abstract: We present results regarding the longitudinal migrations of cool stellar spots that exhibit remarkable oscillations and explore their possible causes. We conducted analyses using high-quality data from nine target systems of various spectral types, spanning from F to M, which were observed by the Kepler Satellite. The systems in which the behaviour of the spots was examined are as follows: KIC 4357272, KIC 6025466, KIC 6058875, KIC 6962018, KIC 7798259, KIC 9210828, KIC 11706658, KIC 12599700, and KIC 8669092. Basic stellar parameters were calculated from light curve analysis using the PHOEBE V.0.32 software, and light curves were modelled to obtain sinusoidal variations occurring out-of-eclipses phases, induced by rotational modulation. Subsequently, we calculated the minimum times of the obtained sinusoidal variations using the Fourier transform. The distributions of {\theta}min corresponding to these minimum times over time were computed using linear fits to determine the longitudinal migrations of the spotted areas. We then compared the longitudinal migration periods with the stellar parameters found in the literature. In addition, we also found a secondary variation in the spot migrations apart from the linear models. Our results revealed that the longitudinal migration periods vary in relation to the B - V colour index of the stars.


title: $M$-ideals in $H^\infty(\mathbb{D})$

abstract: This article intends to initiate an investigation into the structure of $M$-ideals in $H^\infty(\mathbb{D})$, where $H^\infty(\mathbb{D})$ denotes the Banach algebra of all bounded analytic functions on the open unit disc $\mathbb{D}$ in $\mathbb{C}$. We introduce the notion of analytic primes and prove that $M$-ideals in $H^\infty(\mathbb{D})$ are analytic primes. From Hilbert function space perspective, we additionally prove that $M$-ideals in $H^\infty(\mathbb{D})$ are dense in the Hardy space. We show that outer functions play a key role in representing singly generated closed ideals in $H^\infty(\mathbb{D})$ that are $M$-ideals. This is also relevant to $M$-ideals in $H^\infty(\mathbb{D})$ that are finitely generated closed ideals in $H^\infty(\mathbb{D})$. We analyze $p$-sets of $H^\infty(\mathbb{D})$ and their connection to the \v{S}ilov boundary of the maximal ideal space of $H^\infty(\mathbb{D})$. Some of our results apply to the polydisc. In addition to addressing questions regarding $M$-ideals, the results presented in this paper offer some new perspectives on bounded analytic functions.


title: Reinforcement Learning-based Recommender Systems with Large Language Models for State Reward and Action Modeling

abstract: Reinforcement Learning (RL)-based recommender systems have demonstrated promising performance in meeting user expectations by learning to make accurate next-item recommendations from historical user-item interactions. However, existing offline RL-based sequential recommendation methods face the challenge of obtaining effective user feedback from the environment. Effectively modeling the user state and shaping an appropriate reward for recommendation remains a challenge. In this paper, we leverage language understanding capabilities and adapt large language models (LLMs) as an environment (LE) to enhance RL-based recommenders. The LE is learned from a subset of user-item interaction data, thus reducing the need for large training data, and can synthesise user feedback for offline data by: (i) acting as a state model that produces high quality states that enrich the user representation, and (ii) functioning as a reward model to accurately capture nuanced user preferences on actions. Moreover, the LE allows to generate positive actions that augment the limited offline training data. We propose a LE Augmentation (LEA) method to further improve recommendation performance by optimising jointly the supervised component and the RL policy, using the augmented actions and historical user signals. We use LEA, the state and reward models in conjunction with state-of-the-art RL recommenders and report experimental results on two publicly available datasets.


title: Beyond-$\Lambda$CDM Cosmologies with Bayesian Neural Networks II: Massive Neutrinos, Baryonic Feedback, and the Theoretical Error

abstract: We study the capacity of Bayesian Neural Networks (BNNs) to detect new physics in the dark matter power spectrum. As in previous studies, the Bayesian Cosmological Network (BaCoN) classifies spectra into one of 5 classes: $\Lambda$CDM, $f(R)$, $w$CDM, Dvali-Gabadaze-Porrati (DGP) gravity and a 'random' class, with this work extending it to include the effects of massive neutrinos and baryonic feedback. We further develop the treatment of theoretical errors in BaCoN-II, investigating several approaches and identifying the one that best allows the trained network to generalise to other power spectrum modelling prescriptions. In particular, we compare power spectra data produced by EuclidEmulator2, HMcode and halofit, all supplemented with the halo model reaction to model beyond-$\Lambda$CDM physics. We investigate BNN-classifiers trained on these sets of spectra, adding in Stage-IV survey noise and various theoretical error models. Using our optimal theoretical error model, our fiducial classifier achieves a total classification accuracy of $\sim$ 95% when it is trained on EuclidEmulator2-based spectra with modification parameters drawn from a Gaussian distribution centred around $\Lambda$CDM ($f(R)$: $\sigma_{fR0} = 10^{-5.5}$, DGP: $\sigma_{r\mathrm{c}} = 0.173$, $w$CDM: $\sigma_{w0} = 0.097$, $\sigma_{wa}=0.32$). This strengthens the promise of this method to glean the maximal amount of unbiased gravitational and cosmological information from forthcoming Stage-IV galaxy surveys.


title: Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

abstract: Large Language Models (LLMs) have demonstrated promising capabilities as automatic evaluators in assessing the quality of generated natural language. However, LLMs still exhibit biases in evaluation and often struggle to generate coherent evaluations that align with human assessments. In this work, we first conduct a systematic study of the misalignment between LLM evaluators and human judgement, revealing that existing calibration methods aimed at mitigating biases are insufficient for effectively aligning LLM evaluators. Inspired by the use of preference data in RLHF, we formulate the evaluation as a ranking problem and introduce Pairwise-preference Search (PairS), an uncertainty-guided search method that employs LLMs to conduct pairwise comparisons and efficiently ranks candidate texts. PairS achieves state-of-the-art performance on representative evaluation tasks and demonstrates significant improvements over direct scoring. Furthermore, we provide insights into the role of pairwise preference in quantifying the transitivity of LLMs and demonstrate how PairS benefits from calibration.


title: Network-Assisted Delivery of Adaptive Video Streaming Services through CDN, SDN, and MEC

abstract: Multimedia applications, mainly video streaming services, are currently the dominant source of network load worldwide. In recent Video-on-Demand (VoD) and live video streaming services, traditional streaming delivery techniques have been replaced by adaptive solutions based on the HTTP protocol. Current trends toward high-resolution (e.g., 8K) and/or low-latency VoD and live video streaming pose new challenges to end-to-end (E2E) bandwidth demand and have stringent delay requirements. To do this, video providers typically rely on Content Delivery Networks (CDNs) to ensure that they provide scalable video streaming services. To support future streaming scenarios involving millions of users, it is necessary to increase the CDNs' efficiency. It is widely agreed that these requirements may be satisfied by adopting emerging networking techniques to present Network-Assisted Video Streaming (NAVS) methods. Motivated by this, this thesis goes one step beyond traditional pure client-based HAS algorithms by incorporating (an) in-network component(s) with a broader view of the network to present completely transparent NAVS solutions for HAS clients.


title: Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

abstract: Pretraining data of large language models composes multiple domains (e.g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome models. While existing endeavors rely on heuristics or qualitative strategies to tune the proportions, we discover the quantitative predictability of model performance regarding the mixture proportions in function forms, which we refer to as the data mixing laws. Fitting such functions on sample mixtures unveils model performance on unseen mixtures before actual runs, thus guiding the selection of an ideal data mixture. Furthermore, we propose nested use of the scaling laws of training steps, model sizes, and our data mixing law to enable predicting the performance of large models trained on massive data under various mixtures with only small-scale training. Moreover, experimental results verify that our method effectively optimizes the training mixture of a 1B model trained for 100B tokens in RedPajama, reaching a performance comparable to the one trained for 48% more steps on the default mixture. Extending the application of data mixing laws to continual training accurately predicts the critical mixture proportion that avoids catastrophic forgetting and outlooks the potential for dynamic data schedules


title: Learning Symbolic and Subsymbolic Temporal Task Constraints from Bimanual Human Demonstrations

abstract: Learning task models of bimanual manipulation from human demonstration and their execution on a robot should take temporal constraints between actions into account. This includes constraints on (i) the symbolic level such as precedence relations or temporal overlap in the execution, and (ii) the subsymbolic level such as the duration of different actions, or their starting and end points in time. Such temporal constraints are crucial for temporal planning, reasoning, and the exact timing for the execution of bimanual actions on a bimanual robot. In our previous work, we addressed the learning of temporal task constraints on the symbolic level and demonstrated how a robot can leverage this knowledge to respond to failures during execution. In this work, we propose a novel model-driven approach for the combined learning of symbolic and subsymbolic temporal task constraints from multiple bimanual human demonstrations. Our main contributions are a subsymbolic foundation of a temporal task model that describes temporal nexuses of actions in the task based on distributions of temporal differences between semantic action keypoints, as well as a method based on fuzzy logic to derive symbolic temporal task constraints from this representation. This complements our previous work on learning comprehensive temporal task models by integrating symbolic and subsymbolic information based on a subsymbolic foundation, while still maintaining the symbolic expressiveness of our previous approach. We compare our proposed approach with our previous pure-symbolic approach and show that we can reproduce and even outperform it. Additionally, we show how the subsymbolic temporal task constraints can synchronize otherwise unimanual movement primitives for bimanual behavior on a humanoid robot.


title: Isolated Diffusion: Optimizing Multi-Concept Text-to-Image Generation Training-Freely with Isolated Diffusion Guidance

abstract: Large-scale text-to-image diffusion models have achieved great success in synthesizing high-quality and diverse images given target text prompts. Despite the revolutionary image generation ability, current state-of-the-art models still struggle to deal with multi-concept generation accurately in many cases. This phenomenon is known as ``concept bleeding" and displays as the unexpected overlapping or merging of various concepts. This paper presents a general approach for text-to-image diffusion models to address the mutual interference between different subjects and their attachments in complex scenes, pursuing better text-image consistency. The core idea is to isolate the synthesizing processes of different concepts. We propose to bind each attachment to corresponding subjects separately with split text prompts. Besides, we introduce a revision method to fix the concept bleeding problem in multi-subject synthesis. We first depend on pre-trained object detection and segmentation models to obtain the layouts of subjects. Then we isolate and resynthesize each subject individually with corresponding text prompts to avoid mutual interference. Overall, we achieve a training-free strategy, named Isolated Diffusion, to optimize multi-concept text-to-image synthesis. It is compatible with the latest Stable Diffusion XL (SDXL) and prior Stable Diffusion (SD) models. We compare our approach with alternative methods using a variety of multi-concept text prompts and demonstrate its effectiveness with clear advantages in text-image consistency and user study.


title: Modes of the Dark Ages 21cm field accessible to a lunar radio interferometer

abstract: At redshifts beyond $z \gtrsim 30$, the 21cm line from neutral hydrogen is expected to be essentially the only viable probe of the 3D matter distribution. The lunar far-side is an extremely appealing site for future radio arrays that target this signal, as it is protected from terrestrial radio frequency interference, and has no ionosphere to attenuate and absorb radio emission at low frequencies (tens of MHz and below). We forecast the sensitivity of low-frequency lunar radio arrays to the bispectrum of the 21cm brightness temperature field, which can in turn be used to probe primordial non-Gaussianity generated by particular early universe models. We account for the loss of particular regions of Fourier space due to instrumental limitations and systematic effects, and predict the sensitivity of different representative array designs to local-type non-Gaussianity in the bispectrum, parametrised by $f_{\rm NL}$. Under the most optimistic assumption of sample variance-limited observations, we find that $\sigma(f_{\rm NL}) \lesssim 0.01$ could be achieved for several broad redshift bins at $z \gtrsim 30$ if foregrounds can be removed effectively. These values degrade to between $\sigma(f_{\rm NL}) \sim 0.03$ and $0.7$ for $z=30$ to $z=170$ respectively when a large foreground wedge region is excluded.


title: Bayesian Methods for Trust in Collaborative Multi-Agent Autonomy

abstract: Multi-agent, collaborative sensor fusion is a vital component of a multi-national intelligence toolkit. In safety-critical and/or contested environments, adversaries may infiltrate and compromise a number of agents. We analyze state of the art multi-target tracking algorithms under this compromised agent threat model. We prove that the track existence probability test ("track score") is significantly vulnerable to even small numbers of adversaries. To add security awareness, we design a trust estimation framework using hierarchical Bayesian updating. Our framework builds beliefs of trust on tracks and agents by mapping sensor measurements to trust pseudomeasurements (PSMs) and incorporating prior trust beliefs in a Bayesian context. In case studies, our trust estimation algorithm accurately estimates the trustworthiness of tracks/agents, subject to observability limitations.


title: Hydrodynamic fluctuations and topological susceptibility in chiral magnetohydrodynamics

abstract: Chiral magnetohydrodynamics is devoted to understanding the late-time and long-distance behavior of a system with an Adler-Bell-Jackiw anomaly at finite temperatures. The non-conservation of the axial charge is determined by the topological density $\vec{E} \cdot \vec{B}$; in a classical hydrodynamic description this decay rate can be suppressed by tuning the background magnetic field to zero. However it is in principle possible for thermal fluctuations of $\vec{E} \cdot \vec{B}$ to result in a non-conservation of the charge even at vanishing $B$-field; this would invalidate the classical hydrodynamic effective theory. We investigate this by computing the real-time susceptibility of the topological density at one-loop level in magnetohydrodynamic fluctuations, relating its low-frequency limit to the decay rate of the axial charge. We find that the frequency-dependence of this susceptibility is sufficiently soft as to leave the axial decay rate unaffected, validating the classical hydrodynamic description. We show that the susceptibility contains non-analytic frequency-dependence which is universally determined by hydrodynamic data. We comment briefly on possible connections to the recent formulation of the ABJ anomaly in terms of non-invertible symmetry.


title: TwinLiteNetPlus: A Stronger Model for Real-time Drivable Area and Lane Segmentation

abstract: Semantic segmentation is crucial for autonomous driving, particularly for Drivable Area and Lane Segmentation, ensuring safety and navigation. To address the high computational costs of current state-of-the-art (SOTA) models, this paper introduces TwinLiteNetPlus (TwinLiteNet$^+$), a model adept at balancing efficiency and accuracy. TwinLiteNet$^+$ incorporates standard and depth-wise separable dilated convolutions, reducing complexity while maintaining high accuracy. It is available in four configurations, from the robust 1.94 million-parameter TwinLiteNet$^+_{\text{Large}}$ to the ultra-compact 34K-parameter TwinLiteNet$^+_{\text{Nano}}$. Notably, TwinLiteNet$^+_{\text{Large}}$ attains a 92.9\% mIoU for Drivable Area Segmentation and a 34.2\% IoU for Lane Segmentation. These results notably outperform those of current SOTA models while requiring a computational cost that is approximately 11 times lower in terms of Floating Point Operations (FLOPs) compared to the existing SOTA model. Extensively tested on various embedded devices, TwinLiteNet$^+$ demonstrates promising latency and power efficiency, underscoring its suitability for real-world autonomous vehicle applications.


title: The thermodynamics of the quantum Mpemba effect

abstract: We investigate the quantum Mpemba effect from the perspective of non-equilibrium quantum thermodynamics by studying relaxation dynamics described by Davies maps. Starting from a state with coherences in the energy eigenbasis, we demonstrate that an exponential speedup to equilibrium will always occur if the state is transformed to a diagonal state in the energy eigenbasis, provided that the spectral gap of the generator is defined by a complex eigenvalue. When the transformed state has a higher non-equilibrium free energy, we argue using thermodynamic reasoning that this is a \textit{genuine} quantum Mpemba effect. Furthermore, we show how a unitary transformation on an initial state can always be constructed to yield the effect and demonstrate our findings by studying the dynamics of both the non-equilibrium free energy and the irreversible entropy production in single and multi-qubit examples.


title: A Mathematical Description of the Quasi-Periodically Developed Heat Transfer Regime in Channels with Arrays of Periodic Solid Structures

abstract: In this article, we present the governing equations for the temperature field upstream and downstream of the periodically developed flow region in channels with arrays of periodic solid structures. From the ansatz that the temperature field in this region is determined by exponentially decaying modes, just like the quasi-developed flow field, we arrive at an eigenvalue problem which governs the allowed temperature modes. This eigenvalue problem is virtually identical to the one for periodically developed heat transfer in isothermal solids, except that the conjugate heat transfer in the solid is effectively taken into account.


title: Biased ensembles of pulsating active matter

abstract: We discover unexpected connections between packing configurations and rare fluctuations in dense systems of active particles subject to pulsation of size. Using large deviation theory, we examine biased ensembles which select atypical realizations of the dynamics exhibiting high synchronization in particle size. We show that the order emerging at high bias can manifest as distinct dynamical states with either a finite or a vanishing size current. Remarkably, transitions between these states arise from changing the system geometry at fixed bias and constant density. We rationalize such transitions as stemming from a change in the packing configurations favored by the geometry. Specifically, we reveal that a master curve in the unbiased dynamics, correlating polydispersity and current, helps predict the dynamical state emerging in the biased dynamics. Finally, we demonstrate that deformation waves can propagate under suitable geometries when biasing with local order.


title: An $\alpha$-potential game framework for $N$-player games

abstract: This paper proposes and studies a general form of dynamic $N$-player non-cooperative games called $\alpha$-potential games, where the change of a player's value function upon her unilateral deviation from her strategy is equal to the change of an $\alpha$-potential function up to an error $\alpha$. Analogous to the static potential game (which corresponds to $\alpha=0$), the $\alpha$-potential game framework is shown to reduce the challenging task of finding approximate Nash equilibria for a dynamic game to minimizing the $\alpha$-potential function. Moreover, an analytical characterization of $\alpha$-potential functions is established, with $\alpha$ represented in terms of the magnitude of the asymmetry of value functions' second-order derivatives. For stochastic differential games in which the state dynamic is a controlled diffusion, $\alpha$ is explicitly identified in terms of the number of players, the choice of admissible strategies, and the intensity of interactions and the level of heterogeneity among players. This is achieved by introducing a suitable linear derivative of the value functions with respect to unilateral deviations of strategies and via analyzing the sensitivity processes of state dynamics with respect to controls. For games with mean-field type interactions, $\alpha$ is shown to decay to zero as the number of players goes to infinity, even with heterogeneity in state dynamics, cost functions, and admissible strategy classes. For distributed games, if a static potential function can be derived from the cost functions, then $\alpha=0$. For crowd aversion games, $\alpha$ is capable of capturing the subtle difference between the choice of admissible strategies.


title: Constraining the history of reheating with the NANOGrav 15-year data

abstract: Over the last few years, primordial black holes (PBHs) have emerged as a strong candidate for cold dark matter. A significant number of PBHs are produced when the strength of the primordial scalar power spectrum is enhanced on small scales (compared to the COBE normalized values on large scales). Such primordial spectra also inevitably lead to strong amplification of the scalar-induced, secondary gravitational waves (GWs) at higher frequencies. The recent detection of the stochastic gravitational wave background (SGWB) by the pulsar timing arrays (PTAs) has opened up the possibility of directly probing the very early universe. Different studies have shown that, when PBHs are assumed to have been formed during the epoch of radiation domination, the mechanism for the amplification of the scalar-induced GWs that is required to explain the PTA data can overproduce the PBHs over some ranges of masses. In this work, we assume a specific functional form for the primordial scalar power spectrum and examine the production of PBHs and the scalar-induced secondary GWs during the phase of reheating, which precedes the standard epoch of radiation domination. Specifically, we account for the uncertainties in the conditions for the formation of PBHs and ensure that the extent of PBHs produced remains within the observational bounds. We find that the scalar-induced SGWB generated during a phase of reheating with a steeper equation of state (than that of radiation) fit the NANOGrav 15-year data with a stronger Bayesian evidence than the astrophysical scenario involving GWs produced by merging supermassive binary black holes.


title: GSDF: 3DGS Meets SDF for Improved Rendering and Reconstruction

abstract: Presenting a 3D scene from multiview images remains a core and long-standing challenge in computer vision and computer graphics. Two main requirements lie in rendering and reconstruction. Notably, SOTA rendering quality is usually achieved with neural volumetric rendering techniques, which rely on aggregated point/primitive-wise color and neglect the underlying scene geometry. Learning of neural implicit surfaces is sparked from the success of neural rendering. Current works either constrain the distribution of density fields or the shape of primitives, resulting in degraded rendering quality and flaws on the learned scene surfaces. The efficacy of such methods is limited by the inherent constraints of the chosen neural representation, which struggles to capture fine surface details, especially for larger, more intricate scenes. To address these issues, we introduce GSDF, a novel dual-branch architecture that combines the benefits of a flexible and efficient 3D Gaussian Splatting (3DGS) representation with neural Signed Distance Fields (SDF). The core idea is to leverage and enhance the strengths of each branch while alleviating their limitation through mutual guidance and joint supervision. We show on diverse scenes that our design unlocks the potential for more accurate and detailed surface reconstructions, and at the meantime benefits 3DGS rendering with structures that are more aligned with the underlying geometry.


title: The first cut is the cheapest: optimizing Athena/X-IFU-like TES detectors resolution by filter truncation

abstract: The X-ray Integral Field Unit (X-IFU) instrument on the future ESA mission Athena X-ray Observatory is a cryogenic micro-calorimeter array of Transition Edge Sensor (TES) detectors designed to provide spatially-resolved high-resolution spectroscopy. The onboard reconstruction software provides energy, spatial location and arrival time of incoming X-ray photons hitting the detector. A new processing algorithm based on a truncation of the classical optimal filter and called 0-padding, has been recently proposed aiming to reduce the computational cost without compromising energy resolution. Initial tests with simple synthetic data displayed promising results. This study explores the slightly better performance of the 0-padding filter and assess its final application to real data. The goal is to examine the larger sensitivity to instrumental conditions that was previously observed during the analysis of the simulations. This 0-padding technique is thoroughly tested using more realistic simulations and real data acquired from NASA and NIST laboratories employing X-IFU-like TES detectors. Different fitting methods are applied to the data, and a comparative analysis is performed to assess the energy resolution values obtained from these fittings. The 0-padding filter achieves energy resolutions as good as those obtained with standard filters, even with those of larger lengths, across different line complexes and instrumental conditions. This method proves to be useful for energy reconstruction of X-ray photons detected by the TES detectors provided proper corrections for baseline drift and jitter effects are applied. The finding is highly promising especially for onboard processing, offering efficiency in computational resources and facilitating the analysis of sources with higher count rates at high resolution.


title: Unsupervised Feature Selection via Nonnegative Orthogonal Constrained Regularized Minimization

abstract: Unsupervised feature selection has drawn wide attention in the era of big data since it is a primary technique for dimensionality reduction. However, many existing unsupervised feature selection models and solution methods were presented for the purpose of application, and lack of theoretical support, e.g., without convergence analysis. In this paper, we first establish a novel unsupervised feature selection model based on regularized minimization with nonnegative orthogonal constraints, which has advantages of embedding feature selection into the nonnegative spectral clustering and preventing overfitting. An effective inexact augmented Lagrangian multiplier method is proposed to solve our model, which adopts the proximal alternating minimization method to solve subproblem at each iteration. We show that the sequence generated by our method globally converges to a Karush-Kuhn-Tucker point of our model. Extensive numerical experiments on popular datasets demonstrate the stability and robustness of our method. Moreover, comparison results of algorithm performance show that our method outperforms some existing state-of-the-art methods.


title: Visual Whole-Body Control for Legged Loco-Manipulation

abstract: We study the problem of mobile manipulation using legged robots equipped with an arm, namely legged loco-manipulation. The robot legs, while usually utilized for mobility, offer an opportunity to amplify the manipulation capabilities by conducting whole-body control. That is, the robot can control the legs and the arm at the same time to extend its workspace. We propose a framework that can conduct the whole-body control autonomously with visual observations. Our approach, namely Visual Whole-Body Control(VBC), is composed of a low-level policy using all degrees of freedom to track the body velocities along with the end-effector position, and a high-level policy proposing the velocities and end-effector position based on visual inputs. We train both levels of policies in simulation and perform Sim2Real transfer for real robot deployment. We perform extensive experiments and show significant improvements over baselines in picking up diverse objects in different configurations (heights, locations, orientations) and environments.


title: Evaluating Shortest Edit Script Methods for Contextual Lemmatization

abstract: Modern contextual lemmatizers often rely on automatically induced Shortest Edit Scripts (SES), namely, the number of edit operations to transform a word form into its lemma. In fact, different methods of computing SES have been proposed as an integral component in the architecture of several state-of-the-art contextual lemmatizers currently available. However, previous work has not investigated the direct impact of SES in the final lemmatization performance. In this paper we address this issue by focusing on lemmatization as a token classification task where the only input that the model receives is the word-label pairs in context, where the labels correspond to previously induced SES. Thus, by modifying in our lemmatization system only the SES labels that the model needs to learn, we may then objectively conclude which SES representation produces the best lemmatization results. We experiment with seven languages of different morphological complexity, namely, English, Spanish, Basque, Russian, Czech, Turkish and Polish, using multilingual and language-specific pre-trained masked language encoder-only models as a backbone to build our lemmatizers. Comprehensive experimental results, both in- and out-of-domain, indicate that computing the casing and edit operations separately is beneficial overall, but much more clearly for languages with high-inflected morphology. Notably, multilingual pre-trained language models consistently outperform their language-specific counterparts in every evaluation setting.


title: Signatures of canted antiferromagnetism in infinite-layer nickelates studied by x-ray magnetic dichroism

abstract: We report an experimental study of the magnetic properties of infinite-layer Nd1-xSrxNiO2 thin films by x-ray magnetic circular dichroism (XMCD) at Ni L3,2 and Nd M5,4 edges. We show that at low temperatures the out-of-plane component of the Ni1+ spin-moment is characterized by a rapid increase for magnetic fields below 1T , followed by a slower linear increase reaching a spin-moment value of 0.25uB/Ni at 9T in the case of superconducting Nd0.8Sr0.2NiO2. On the other hand, the Nd M5,4 XMCD shows a clear paramagnetic behaviour, which make both Ni- and Nd-spin-sublattices fully uncorrelated.The magnetic field and temperature dependencies of the Ni L3,2 XMCD data can be explained by assuming an out-of-plane canting of the strongly in-plane anti-ferromagnetic ordered Ni1+ spins. A symmetry lowering of the NiO2 planes observed via four-dimensional scanning transmission electron microscopy, triggering a DMI, can be responsible of the proposed Ni1+ spin-canting at zero-field. The resulting out-of-plane weak-ferromagnetic coupling under magnetic field explains the relatively large spin-moment and its magnetic field and temperature dependence.


title: Joint chest X-ray diagnosis and clinical visual attention prediction with multi-stage cooperative learning: enhancing interpretability

abstract: As deep learning has become the state-of-the-art for computer-assisted diagnosis, interpretability of the automatic decisions is crucial for clinical deployment. While various methods were proposed in this domain, visual attention maps of clinicians during radiological screening offer a unique asset to provide important insights and can potentially enhance the quality of computer-assisted diagnosis. With this paper, we introduce a novel deep-learning framework for joint disease diagnosis and prediction of corresponding visual saliency maps for chest X-ray scans. Specifically, we designed a novel dual-encoder multi-task UNet, which leverages both a DenseNet201 backbone and a Residual and Squeeze-and-Excitation block-based encoder to extract diverse features for saliency map prediction, and a multi-scale feature-fusion classifier to perform disease classification. To tackle the issue of asynchronous training schedules of individual tasks in multi-task learning, we proposed a multi-stage cooperative learning strategy, with contrastive learning for feature encoder pretraining to boost performance. Experiments show that our proposed method outperformed existing techniques for chest X-ray diagnosis and the quality of visual saliency map prediction.


title: AIOS: LLM Agent Operating System

abstract: The integration and deployment of large language model (LLM)-based intelligent agents have been fraught with challenges that compromise their efficiency and efficacy. Among these issues are sub-optimal scheduling and resource allocation of agent requests over the LLM, the difficulties in maintaining context during interactions between agent and LLM, and the complexities inherent in integrating heterogeneous agents with different capabilities and specializations. The rapid increase of agent quantity and complexity further exacerbates these issues, often leading to bottlenecks and sub-optimal utilization of resources. Inspired by these challenges, this paper presents AIOS, an LLM agent operating system, which embeds large language model into operating systems (OS) as the brain of the OS, enabling an operating system "with soul" -- an important step towards AGI. Specifically, AIOS is designed to optimize resource allocation, facilitate context switch across agents, enable concurrent execution of agents, provide tool service for agents, and maintain access control for agents. We present the architecture of such an operating system, outline the core challenges it aims to resolve, and provide the basic design and implementation of the AIOS. Our experiments on concurrent execution of multiple agents demonstrate the reliability and efficiency of our AIOS modules. Through this, we aim to not only improve the performance and efficiency of LLM agents but also to pioneer for better development and deployment of the AIOS ecosystem in the future. The project is open-source at https://github.com/agiresearch/AIOS.


title: Role of Fragility of the Glass Formers in the Yielding Transition under Oscillatory Shear

abstract: We study the effect of the fragility of glass formers on the yielding transition under oscillatory shear via extensive computer simulations. Employing sphere assemblies interacting with a harmonic potential as our model glass former, we tune fragility by changing the system's density - higher density corresponds to large fragility. Our study reveals significant differences in the yielding transition between strong and fragile glass formers. While both glass formers exhibit similar behaviour under poorly annealed initial conditions, the yielding transition shifts to larger values with increased annealing for fragile glasses while remaining relatively constant for strong glasses. We rationalize our results by introducing a new elastoplastic model, which qualitatively reproduces the simulation results and offers valuable insight into the physics of yielding transition under oscillatory shear deformation.


title: VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

abstract: We introduce VoiceCraft, a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts. VoiceCraft employs a Transformer decoder architecture and introduces a token rearrangement procedure that combines causal masking and delayed stacking to enable generation within an existing sequence. On speech editing tasks, VoiceCraft produces edited speech that is nearly indistinguishable from unedited recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS, our model outperforms prior SotA models including VALLE and the popular commercial model XTTS-v2. Crucially, the models are evaluated on challenging and realistic datasets, that consist of diverse accents, speaking styles, recording conditions, and background noise and music, and our model performs consistently well compared to other models and real recordings. In particular, for speech editing evaluation, we introduce a high quality, challenging, and realistic dataset named RealEdit. We encourage readers to listen to the demos at https://jasonppy.github.io/VoiceCraft_web.


title: Self-STORM: Deep Unrolled Self-Supervised Learning for Super-Resolution Microscopy

abstract: The use of fluorescent molecules to create long sequences of low-density, diffraction-limited images enables highly-precise molecule localization. However, this methodology requires lengthy imaging times, which limits the ability to view dynamic interactions of live cells on short time scales. Many techniques have been developed to reduce the number of frames needed for localization, from classic iterative optimization to deep neural networks. Particularly, deep algorithm unrolling utilizes both the structure of iterative sparse recovery algorithms and the performance gains of supervised deep learning. However, the robustness of this approach is highly dependant on having sufficient training data. In this paper we introduce deep unrolled self-supervised learning, which alleviates the need for such data by training a sequence-specific, model-based autoencoder that learns only from given measurements. Our proposed method exceeds the performance of its supervised counterparts, thus allowing for robust, dynamic imaging well below the diffraction limit without any labeled training samples. Furthermore, the suggested model-based autoencoder scheme can be utilized to enhance generalization in any sparse recovery framework, without the need for external training data.


title: Unconditionally positivity-preserving approximations of the Ait-Sahalia type model: Explicit Milstein-type schemes

abstract: The present article aims to design and analyze efficient first-order strong schemes for a generalized A\"{i}t-Sahalia type model arising in mathematical finance and evolving in a positive domain $(0, \infty)$, which possesses a diffusion term with superlinear growth and a highly nonlinear drift that blows up at the origin. Such a complicated structure of the model unavoidably causes essential difficulties in the construction and convergence analysis of time discretizations. By incorporating implicitness in the term $\alpha_{-1} x^{-1}$ and a corrective mapping $\Phi_h$ in the recursion, we develop a novel class of explicit and unconditionally positivity-preserving (i.e., for any step-size $h>0$) Milstein-type schemes for the underlying model. In both non-critical and general critical cases, we introduce a novel approach to analyze mean-square error bounds of the novel schemes, without relying on a priori high-order moment bounds of the numerical approximations. The expected order-one mean-square convergence is attained for the proposed scheme. The above theoretical guarantee can be used to justify the optimal complexity of the Multilevel Monte Carlo method. Numerical experiments are finally provided to verify the theoretical findings.


title: The Massive and Distant Clusters of WISE Survey 2: Equatorial First Data Release

abstract: The Massive and Distant Clusters of WISE Survey 2 (MaDCoWS2) is a new survey designed as the successor of the original MaDCoWS survey. MaDCoWS2 improves upon its predecessor by using deeper optical and infrared data and a more powerful detection algorithm (PZWav). As input to the search, we use grz photometry from DECaLS in combination with W1 and W2 photometry from the CatWISE2020 catalog to derive the photometric redshifts with full redshift probability distribution functions for WISE-selected galaxies. Cluster candidates are then detected using the PZWav algorithm to find three-dimensional galaxy overdensities from the sky positions and photometric redshifts. This paper provides the first MaDCoWS2 data release, covering 1461 (1838 without masking) deg^2 centered on the Hyper-SuprimeCam Subaru Strategic Program equatorial fields. Within this region, we derive a catalog of 22,970 galaxy cluster candidates detected at S/N>5. These clusters span the redshift range 0.1<z<2, including 1312 candidates at z>1.5. We compare MaDCoWS2 to six existing catalogs in the area. We rediscover 60%-92% of the clusters in these surveys at S/N>5. The medians of the absolute redshift offset are <0.02 relative to these surveys, while the standard deviations are less than 0.06. The median offsets between the detection position from MaDCoWS2 and other surveys are less than 0.25 Mpc. We quantify the relation between S/N and gas mass, total mass, luminosity, and richness from other surveys using a redshift-dependent power law relation. We find that the S/N-richness relation exhibits the lowest scatter.


title: A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course

abstract: This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, both with and without prompt engineering, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Comparing 50 student submissions to 50 AI-generated submissions across different categories, and marked blindly by three independent markers, we amassed $n = 300$ data points. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8) - a statistically significant difference (p = $2.482 \times 10^{-10}$). Prompt engineering significantly improved scores for both GPT-4 (p = $1.661 \times 10^{-4}$) and GPT-3.5 (p = $4.967 \times 10^{-9}$). Additionally, the blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from `Definitely AI' to `Definitely Human'. They accurately identified the authorship, with 92.1% of the work categorized as 'Definitely Human' being human-authored. Simplifying this to a binary `AI' or `Human' categorization resulted in an average accuracy rate of 85.3%. These findings suggest that while AI-generated work closely approaches the quality of university students' work, it often remains detectable by human evaluators.


title: Locally Purified Density Operators for Symmetry-Protected Topological Phases in Mixed States

abstract: We propose a tensor network approach known as the locally purified density operator (LPDO) to investigate the classification and characterization of symmetry-protected topological (SPT) phases in open quantum systems. We extend the concept of injectivity, originally associated with matrix product states and projected entangled pair states, to LPDOs in $(1+1)D$ and $(2+1)D$ systems, unveiling two distinct types of injectivity conditions inherent in short-range entangled density matrices. Within the LPDO framework, we outline a classification scheme for decohered average symmetry-protected topological (ASPT) phases, consistent with earlier results obtained through spectrum sequence techniques. We first illustrate our framework with ASPT phases protected by fermion parity symmetry, then extend the classification of ASPT phases to a general group extension. We demonstrate examples of explicit construction of fixed-point LPDOs for ASPT phases including intrinsic ASPTs in both $(1+1)D$ and $(2+1)D$ systems.


title: An Optimal Solution to Infinite Horizon Nonlinear Control Problems: Part II

abstract: This paper considers the infinite horizon optimal control problem for nonlinear systems. Under the condition of nonlinear controllability of the system to any terminal set containing the origin and forward invariance of the terminal set, we establish a regularized solution approach consisting of a ``finite free final time" optimal transfer problem to the terminal set which renders the set globally asymptotically stable. Further, we show that the approximations converge to the optimal infinite horizon cost as the size of the terminal set decreases to zero. We also perform the analysis for the discounted problem and show that the terminal set is asymptotically stable only for a subset of the state space and not globally. The theory is empirically evaluated on various nonholonomic robotic systems to show that the cost of our approximate problem converges and the transfer time into the terminal set is dependent on the initial state of the system, necessitating the free final time formulation.


title: Economic DAO Governance: A Contestable Control Approach

abstract: In this article, we propose a new form of DAO governance that uses a sequential auction mechanism to overcome entrenched control issues that have emerged for DAOs by creating a regime of temporary contestable control. The mechanism avoids potential public choice problems inherent in voting approaches but at the same time provides a vehicle that can enhance and secure value that inheres to DAO voting and other DAO non-market governance procedures. It is robust to empty voting and is code feasible. It facilitates the ability of DAOs to meet their normative and operational goals in the face of diverse regulatory approaches. Designed to shift control to the party with the most promising business plan, at the same time it distributes surplus in a way that tends to promote investment by other parties.


title: The Sample Complexity of Simple Binary Hypothesis Testing

abstract: The sample complexity of simple binary hypothesis testing is the smallest number of i.i.d. samples required to distinguish between two distributions $p$ and $q$ in either: (i) the prior-free setting, with type-I error at most $\alpha$ and type-II error at most $\beta$; or (ii) the Bayesian setting, with Bayes error at most $\delta$ and prior distribution $(\alpha, 1-\alpha)$. This problem has only been studied when $\alpha = \beta$ (prior-free) or $\alpha = 1/2$ (Bayesian), and the sample complexity is known to be characterized by the Hellinger divergence between $p$ and $q$, up to multiplicative constants. In this paper, we derive a formula that characterizes the sample complexity (up to multiplicative constants that are independent of $p$, $q$, and all error parameters) for: (i) all $0 \le \alpha, \beta \le 1/8$ in the prior-free setting; and (ii) all $\delta \le \alpha/4$ in the Bayesian setting. In particular, the formula admits equivalent expressions in terms of certain divergences from the Jensen--Shannon and Hellinger families. The main technical result concerns an $f$-divergence inequality between members of the Jensen--Shannon and Hellinger families, which is proved by a combination of information-theoretic tools and case-by-case analyses. We explore applications of our results to robust and distributed (locally-private and communication-constrained) hypothesis testing.


title: State-Augmented Linear Games with Antagonistic Error for High-Dimensional, Nonlinear Hamilton-Jacobi Reachability

abstract: Hamilton-Jacobi Reachability (HJR) is a popular method for analyzing the liveness and safety of a dynamical system with bounded control and disturbance. The corresponding HJ value function offers a robust controller and characterizes the reachable sets, but is traditionally solved with Dynamic Programming (DP) and limited to systems of dimension less than six. Recently, the space-parallelizeable, generalized Hopf formula has been shown to also solve the HJ value with a nearly three-log increase in dimension limit, but is limited to linear systems. To extend this potential, we demonstrate how state-augmented (SA) spaces, which are well-known for their improved linearization accuracy, may be used to solve tighter, conservative approximations of the value function with any linear model in this SA space. Namely, we show that with a representation of the true dynamics in the SA space, a series of inequalities confirms that the value of a SA linear game with antagonistic error is a conservative envelope of the true value function. It follows that if the optimal controller for the HJ SA linear game with error may succeed, it will also succeed in the true system. Unlike previous methods, this result offers the ability to safely approximate reachable sets and their corresponding controllers with the Hopf formula in a non-convex manner. Finally, we demonstrate this in the slow manifold system for clarity, and in the controlled Van der Pol system with different lifting functions.


title: Robust Filter Design for Graph Signals

abstract: Our goal in this paper is the robust design of filters acting on signals observed over graphs subject to small perturbations of their edges. The focus is on developing a method to identify spectral and polynomial graph filters that can adapt to the perturbations in the underlying graph structure while ensuring the filters adhere to the desired spectral mask. To address this, we propose a novel approach that leverages approximate closed-form expressions for the perturbed eigendecomposition of the Laplacian matrix associated with the nominal topology. Furthermore, when dealing with noisy input signals for graph filters, we propose a strategy for designing FIR filters that jointly minimize the approximation error with respect to the ideal filter and the estimation error of the output, ensuring robustness against both graph perturbations and noise. Numerical results validate the effectiveness of our proposed strategies, highlighting their capability to efficiently manage perturbations and noise.


title: Modelling Commonsense Commonalities with Multi-Facet Concept Embeddings

abstract: Concept embeddings offer a practical and efficient mechanism for injecting commonsense knowledge into downstream tasks. Their core purpose is often not to predict the commonsense properties of concepts themselves, but rather to identify commonalities, i.e.\ sets of concepts which share some property of interest. Such commonalities are the basis for inductive generalisation, hence high-quality concept embeddings can make learning easier and more robust. Unfortunately, standard embeddings primarily reflect basic taxonomic categories, making them unsuitable for finding commonalities that refer to more specific aspects (e.g.\ the colour of objects or the materials they are made of). In this paper, we address this limitation by explicitly modelling the different facets of interest when learning concept embeddings. We show that this leads to embeddings which capture a more diverse range of commonsense properties, and consistently improves results in downstream tasks such as ultra-fine entity typing and ontology completion.


title: Towards Low-Latency and Energy-Efficient Hybrid P2P-CDN Live Video Streaming

abstract: Streaming segmented videos over the Hypertext Transfer Protocol (HTTP) is an increasingly popular approach in both live and video-on-demand (VoD) applications. However, designing a scalable and adaptable framework that reduces servers energy consumption and supports low latency and high quality services, particularly for live video streaming scenarios, is still challenging for Over-The-Top (OTT) service providers. To address such challenges, this paper introduces a new hybrid P2P-CDN framework that leverages new networking and computing paradigms, i.e., Network Function Virtualization (NFV) and edge computing for live video streaming. The proposed framework introduces a multi-layer architecture and a tree of possible actions therein (an action tree), taking into account all available resources from peers, edge, and CDN servers to efficiently distribute video fetching and transcoding tasks across a hybrid P2P-CDN network, consequently enhancing the users latency and video quality. We also discuss our testbed designed to validate the framework and compare it with baseline methods. The experimental results indicate that the proposed framework improves user Quality of Experience (QoE), reduces client serving latency, and improves edge server energy consumption compared to baseline approaches.


title: Dynamic Relative Representations for Goal-Oriented Semantic Communications

abstract: In future 6G wireless networks, semantic and effectiveness aspects of communications will play a fundamental role, incorporating meaning and relevance into transmissions. However, obstacles arise when devices employ diverse languages, logic, or internal representations, leading to semantic mismatches that might jeopardize understanding. In latent space communication, this challenge manifests as misalignment within high-dimensional representations where deep neural networks encode data. This paper presents a novel framework for goal-oriented semantic communication, leveraging relative representations to mitigate semantic mismatches via latent space alignment. We propose a dynamic optimization strategy that adapts relative representations, communication parameters, and computation resources for energy-efficient, low-latency, goal-oriented semantic communications. Numerical results demonstrate our methodology's effectiveness in mitigating mismatches among devices, while optimizing energy consumption, delay, and effectiveness.


title: Multiple normalized solutions to a system of nonlinear Schr\"{o}dinger equations

abstract: We find a normalized solution $u=(u_1,\ldots,u_K)$ to the system of $K$ coupled nonlinear Schr\"odinger equations \begin{equation*} \left\{ \begin{array}{l} -\Delta u_i+ \lambda_i u_i = \sum_{j=1}^K\beta_{i,j}u_i|u_i|^{p/2-2}|u_j|^{p/2} \quad \mathrm{in} \, \mathbb{R}^3,\newline u_i \in H^1_{rad}(\mathbb{R}^3),\newline \int_{\mathbb{R}^3} |u_i|^2 \, dx = \rho_i^2 \quad \text{for }i=1,\ldots, K, \end{array} \right. \end{equation*} where $\rho=(\rho_1,\ldots,\rho_K)\in(0,\infty)^K$ is prescribed, $(\lambda,u) \in \mathbb{R}^K\times H^1(\mathbb{R}^3)^K$ are the unknown and $4\leq p<6$. In the case of two equations we show the existence of multiple solutions provided that the coupling is sufficiently large. We also show that for negative coupling there are no ground state solutions. The main novelty in our approach is that we use the Cwikel-Lieb-Rozenblum theorem in order to estimate the Morse index of a solution as well as a Liouville-type result in an exterior domain.


title: Multimodal operando microscopy reveals that interfacial chemistry and nanoscale performance disorder dictate perovskite solar cell stability

abstract: Next-generation low-cost semiconductors such as halide perovskites exhibit optoelectronic properties dominated by nanoscale variations in their structure, composition and photophysics. While microscopy provides a proxy for ultimate device function, past works have focused on neat thin-films on insulating substrates, missing crucial information about charge extraction losses and recombination losses introduced by transport layers. Here we use a multimodal operando microscopy toolkit to measure nanoscale current-voltage curves, recombination losses and chemical composition in an array of state-of-the-art perovskite solar cells before and after extended operational stress. We apply this toolkit to the same scan areas before and after extended operation to reveal that devices with the highest performance have the lowest initial performance spatial heterogeneity - a crucial link that is missed in conventional microscopy. We find that subtle compositional engineering of the perovskite has surprising effects on local disorder and resilience to operational stress. Minimising variations in local efficiency, rather than compositional disorder, is predictive of improved performance and stability. Modulating the interfaces with different contact layers or passivation treatments can increase initial performance but can also lead to dramatic nanoscale, interface-dominated degradation even in the presence of local performance homogeneity, inducing spatially varying transport, recombination, and electrical losses. These operando measurements of full devices act as screenable diagnostic tools, uniquely unveiling the microscopic mechanistic origins of device performance losses and degradation in an array of halide perovskite devices and treatments. This information in turn reveals guidelines for future improvements to both performance and stability.


title: Limits on Dark Matter Compact Objects implied by Supermagnified Stars in Lensing Clusters

abstract: Supermagnified stars are gravitationally lensed individual stars that are located close to a caustic of a lensing galaxy cluster, and have their flux magnified by a large enough factor (typically ~ 1000) to make them detectable with present telescopes. The maximum magnification is limited by microlensing caused by intracluster stars or other compact objects, which create a network of corrugated critical lines with an angular width proportional to the surface density of microlenses. We consider a set of 9 cases of supermagnified stars reported in the literature, and derive an upper limit on the surface density of compact objects, such as primordial black holes, that might be present as a fraction of the dark matter in addition to known intracluster stars. Any such additional compact objects would widen the corrugated critical line network and therefore the width of the distribution of supermagnified stars around the modeled critical lines of the lens. We find that any compact objects, including primordial black holes, with masses above $\sim 10^{-6}\, M_{\odot}$ (for which the microcaustics are smaller than the typical angular size of supermagnified stars) cannot account for more than ~ 3% of the dark matter.


title: Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generation

abstract: Text-to-image diffusion models have an unprecedented ability to generate diverse and high-quality images. However, they often struggle to faithfully capture the intended semantics of complex input prompts that include multiple subjects. Recently, numerous layout-to-image extensions have been introduced to improve user control, aiming to localize subjects represented by specific tokens. Yet, these methods often produce semantically inaccurate images, especially when dealing with multiple semantically or visually similar subjects. In this work, we study and analyze the causes of these limitations. Our exploration reveals that the primary issue stems from inadvertent semantic leakage between subjects in the denoising process. This leakage is attributed to the diffusion model's attention layers, which tend to blend the visual features of different subjects. To address these issues, we introduce Bounded Attention, a training-free method for bounding the information flow in the sampling process. Bounded Attention prevents detrimental leakage among subjects and enables guiding the generation to promote each subject's individuality, even with complex multi-subject conditioning. Through extensive experimentation, we demonstrate that our method empowers the generation of multiple subjects that better align with given prompts and layouts.


title: Geometric Thermodynamics of Collapse of Gels

abstract: Stimulus-induced volumetric phase transition in gels may be potentially exploited for various bio-engineering and mechanical engineering applications. Since the discovery of the phenomenon in the 1970s, extensive experimental research has helped in understanding the phase transition and related critical phenomena. Yet, little insight is available on the evolving microstructure. In this article, we aim at unravelling certain geometric aspects of the micromechanics underlying discontinuous phase transition in polyacrylamide gels. Towards this, we use geometric thermodynamics and a Landau-Ginzburg type free energy functional involving a squared gradient, in conjunction with Flory-Huggins theory. We specifically exploit Ruppeiner's approach of Riemannian geometry-enriched thermodynamic fluctuation theory that has been previously employed to investigate phase transitions in van der Waals fluids and black holes. The framework equips us with a scalar curvature that relates to the microstructural interactions of a gel during phase transition and at critical points. This curvature also provides an insight into the universality class of phase transition and the nature of polymer-polymer interactions.


title: Complex-Valued Signal Recovery using the Bayesian LASSO

abstract: Recovering complex-valued image recovery from noisy indirect data is important in applications such as ultrasound imaging and synthetic aperture radar. While there are many effective algorithms to recover point estimates of the magnitude, fewer are designed to recover the phase. Quantifying uncertainty in the estimate can also provide valuable information for real-time decision making. This investigation therefore proposes a new Bayesian inference method that recovers point estimates while also quantifying the uncertainty for complex-valued signals or images given noisy and indirect observation data. Our method is motivated by the Bayesian LASSO approach for real-valued sparse signals, and here we demonstrate that the Bayesian LASSO can be effectively adapted to recover complex-valued images whose magnitude is sparse in some (e.g.~the gradient) domain. Numerical examples demonstrate our algorithm's robustness to noise as well as its computational efficiency.


title: Comp4D: LLM-Guided Compositional 4D Scene Generation

abstract: Recent advancements in diffusion models for 2D and 3D content creation have sparked a surge of interest in generating 4D content. However, the scarcity of 3D scene datasets constrains current methodologies to primarily object-centric generation. To overcome this limitation, we present Comp4D, a novel framework for Compositional 4D Generation. Unlike conventional methods that generate a singular 4D representation of the entire scene, Comp4D innovatively constructs each 4D object within the scene separately. Utilizing Large Language Models (LLMs), the framework begins by decomposing an input text prompt into distinct entities and maps out their trajectories. It then constructs the compositional 4D scene by accurately positioning these objects along their designated paths. To refine the scene, our method employs a compositional score distillation technique guided by the pre-defined trajectories, utilizing pre-trained diffusion models across text-to-image, text-to-video, and text-to-3D domains. Extensive experiments demonstrate our outstanding 4D content creation capability compared to prior arts, showcasing superior visual quality, motion fidelity, and enhanced object interactions.


title: Mapping Image Transformations Onto Pixel Processor Arrays

abstract: Pixel Processor Arrays (PPA) present a new vision sensor/processor architecture consisting of a SIMD array of processor elements, each capable of light capture, storage, processing and local communication. Such a device allows visual data to be efficiently stored and manipulated directly upon the focal plane, but also demands the invention of new approaches and algorithms, suitable for the massively-parallel fine-grain processor arrays. In this paper we demonstrate how various image transformations, including shearing, rotation and scaling, can be performed directly upon a PPA. The implementation details are presented using the SCAMP-5 vision chip, that contains a 256x256 pixel-parallel array. Our approaches for performing the image transformations efficiently exploit the parallel computation in a cellular processor array, minimizing the number of SIMD instructions required. These fundamental image transformations are vital building blocks for many visual tasks. This paper aims to serve as a reference for future PPA research while demonstrating the flexibility of PPA architectures.


title: Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flows

abstract: Recent works have demonstrated success in controlling sentence attributes ($e.g.$, sentiment) and structure ($e.g.$, syntactic structure) based on the diffusion language model. A key component that drives theimpressive performance for generating high-quality samples from noise is iteratively denoise for thousands of steps. While beneficial, the complexity of starting from the noise and the learning steps has limited its implementation to many NLP real-world applications. This paper proposes Language Rectified Flow ({\ours}). Our method is based on the reformulation of the standard probabilistic flow models. Language rectified flow learns (neural) ordinary differential equation models to transport between the source distribution and the target distribution, hence providing a unified and effective solution to generative modeling and domain transfer. From the source distribution, our language rectified flow yields fast simulation and effectively decreases the inference time. Experiments on three challenging fine-grained control tasks and multiple high-quality text editing show that our method consistently outperforms its baselines. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.


title: DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Driving

abstract: End-to-end driving has made significant progress in recent years, demonstrating benefits such as system simplicity and competitive driving performance under both open-loop and closed-loop settings. Nevertheless, the lack of interpretability and controllability in its driving decisions hinders real-world deployment for end-to-end driving systems. In this paper, we collect a comprehensive end-to-end driving dataset named DriveCoT, leveraging the CARLA simulator. It contains sensor data, control decisions, and chain-of-thought labels to indicate the reasoning process. We utilize the challenging driving scenarios from the CARLA leaderboard 2.0, which involve high-speed driving and lane-changing, and propose a rule-based expert policy to control the vehicle and generate ground truth labels for its reasoning process across different driving aspects and the final decisions. This dataset can serve as an open-loop end-to-end driving benchmark, enabling the evaluation of accuracy in various chain-of-thought aspects and the final decision. In addition, we propose a baseline model called DriveCoT-Agent, trained on our dataset, to generate chain-of-thought predictions and final decisions. The trained model exhibits strong performance in both open-loop and closed-loop evaluations, demonstrating the effectiveness of our proposed dataset.


title: Composed Video Retrieval via Enriched Context and Discriminative Embeddings

abstract: Composed video retrieval (CoVR) is a challenging problem in computer vision which has recently highlighted the integration of modification text with visual queries for more sophisticated video search in large databases. Existing works predominantly rely on visual queries combined with modification text to distinguish relevant videos. However, such a strategy struggles to fully preserve the rich query-specific context in retrieved target videos and only represents the target video using visual embedding. We introduce a novel CoVR framework that leverages detailed language descriptions to explicitly encode query-specific contextual information and learns discriminative embeddings of vision only, text only and vision-text for better alignment to accurately retrieve matched target videos. Our proposed framework can be flexibly employed for both composed video (CoVR) and image (CoIR) retrieval tasks. Experiments on three datasets show that our approach obtains state-of-the-art performance for both CovR and zero-shot CoIR tasks, achieving gains as high as around 7% in terms of recall@K=1 score. Our code, models, detailed language descriptions for WebViD-CoVR dataset are available at \url{https://github.com/OmkarThawakar/composed-video-retrieval}


title: Understanding Long Videos in One Multimodal Language Model Pass

abstract: Large Language Models (LLMs), known to contain a strong awareness of world knowledge, have allowed recent approaches to achieve excellent performance on Long-Video Understanding benchmarks, but at high inference costs. In this work, we first propose Likelihood Selection, a simple technique that unlocks faster inference in autoregressive LLMs for multiple-choice tasks common in long-video benchmarks. In addition to faster inference, we discover the resulting models to yield surprisingly good accuracy on long-video tasks, even with no video specific information. Building on this, we inject video-specific object-centric information extracted from off-the-shelf pre-trained models and utilize natural language as a medium for information fusion. Our resulting Multimodal Video Understanding (MVU) framework demonstrates state-of-the-art performance across long-video and fine-grained action recognition benchmarks. Code available at: https://github.com/kahnchana/mvu


title: Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models

abstract: This paper presents Visual CoT, a novel pipeline that leverages the reasoning capabilities of multi-modal large language models (MLLMs) by incorporating visual Chain-of-Thought (CoT) reasoning. While MLLMs have shown promise in various visual tasks, they often lack interpretability and struggle with complex visual inputs. To address these challenges, we propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts. We collect and introduce the Visual CoT dataset comprising 373k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. Importantly, the introduced benchmark is capable of evaluating MLLMs in scenarios requiring specific local region identification. Extensive experiments demonstrate the effectiveness of our framework and shed light on better inference strategies. The Visual CoT dataset, benchmark, and pre-trained models are available to foster further research in this direction.


title: Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution

abstract: Diffusion models are just at a tipping point for image super-resolution task. Nevertheless, it is not trivial to capitalize on diffusion models for video super-resolution which necessitates not only the preservation of visual appearance from low-resolution to high-resolution videos, but also the temporal consistency across video frames. In this paper, we propose a novel approach, pursuing Spatial Adaptation and Temporal Coherence (SATeCo), for video super-resolution. SATeCo pivots on learning spatial-temporal guidance from low-resolution videos to calibrate both latent-space high-resolution video denoising and pixel-space video reconstruction. Technically, SATeCo freezes all the parameters of the pre-trained UNet and VAE, and only optimizes two deliberately-designed spatial feature adaptation (SFA) and temporal feature alignment (TFA) modules, in the decoder of UNet and VAE. SFA modulates frame features via adaptively estimating affine parameters for each pixel, guaranteeing pixel-wise guidance for high-resolution frame synthesis. TFA delves into feature interaction within a 3D local window (tubelet) through self-attention, and executes cross-attention between tubelet and its low-resolution counterpart to guide temporal feature alignment. Extensive experiments conducted on the REDS4 and Vid4 datasets demonstrate the effectiveness of our approach.


title: VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation

abstract: Recent innovations on text-to-3D generation have featured Score Distillation Sampling (SDS), which enables the zero-shot learning of implicit 3D models (NeRF) by directly distilling prior knowledge from 2D diffusion models. However, current SDS-based models still struggle with intricate text prompts and commonly result in distorted 3D models with unrealistic textures or cross-view inconsistency issues. In this work, we introduce a novel Visual Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the visual appearance knowledge in 2D visual prompt to boost text-to-3D generation. Instead of solely supervising SDS with text prompt, VP3D first capitalizes on 2D diffusion model to generate a high-quality image from input text, which subsequently acts as visual prompt to strengthen SDS optimization with explicit visual appearance. Meanwhile, we couple the SDS optimization with additional differentiable reward function that encourages rendering images of 3D models to better visually align with 2D visual prompt and semantically match with text prompt. Through extensive experiments, we show that the 2D Visual Prompt in our VP3D significantly eases the learning of visual appearance of 3D models and thus leads to higher visual fidelity with more detailed textures. It is also appealing in view that when replacing the self-generating visual prompt with a given reference image, VP3D is able to trigger a new task of stylized text-to-3D generation. Our project page is available at https://vp3d-cvpr24.github.io.


title: Addressing the Band Gap Problem with a Machine-Learned Exchange Functional

abstract: The systematic underestimation of band gaps is one of the most fundamental challenges in semilocal density functional theory (DFT). In addition to hindering the application of DFT to predicting electronic properties, the band gap problem is intimately related to self-interaction and delocalization errors, which make the study of charge transfer mechanisms with DFT difficult. In this work, we present two key innovations to address the band gap problem. First, we design an approach for machine learning density functionals based on Gaussian processes to explicitly fit single-particle energy levels. Second, we introduce novel nonlocal features of the density matrix that are expressive enough to fit these single-particle levels. Combining these developments, we train a machine-learned functional for the exact exchange energy that predicts molecular energy gaps and reaction energies of a wide range of molecules in excellent agreement with reference hybrid DFT calculations. In addition, while being trained solely on molecular data, our model predicts reasonable formation energies of polarons in solids, showcasing its transferability and robustness. Our approach generalizes straightforwardly to full exchange-correlation functionals, thus paving the way to the design of novel state-of-the-art functionals for the prediction of electronic properties of molecules and materials.


title: Multiple Chern bands in twisted MoTe$_2$ and possible non-Abelian states

abstract: We investigate the moir\'e band structures and possible even denominator fractional quantum Hall state in small angle twisted bilayer MoTe$_2$, using combined large-scale local basis density functional theory calculation and continuum model exact diagonalization. Via large-scale first principles calculations at $\theta=1.89^{\circ}$, we find a sequence of $C=1$ moir\'e Chern bands, in analogy to Landau levels. Constructing the continuum model with multiple Chern bands and uniform Berry curvature in the second moir\'e band, we undertake band-projected exact diagonalization using unscreened Coulomb repulsion to pinpoint possible $\nu=-3/2$ non-Abelian states across a wide range of twist angles below $\theta=2.5^{\circ}$.


title: SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer

abstract: Diffusion Transformer (DiT) has emerged as the new trend of generative diffusion models on image generation. In view of extremely slow convergence in typical DiT, recent breakthroughs have been driven by mask strategy that significantly improves the training efficiency of DiT with additional intra-image contextual learning. Despite this progress, mask strategy still suffers from two inherent limitations: (a) training-inference discrepancy and (b) fuzzy relations between mask reconstruction & generative diffusion process, resulting in sub-optimal training of DiT. In this work, we address these limitations by novelly unleashing the self-supervised discrimination knowledge to boost DiT training. Technically, we frame our DiT in a teacher-student manner. The teacher-student discriminative pairs are built on the diffusion noises along the same Probability Flow Ordinary Differential Equation (PF-ODE). Instead of applying mask reconstruction loss over both DiT encoder and decoder, we decouple DiT encoder and decoder to separately tackle discriminative and generative objectives. In particular, by encoding discriminative pairs with student and teacher DiT encoders, a new discriminative loss is designed to encourage the inter-image alignment in the self-supervised embedding space. After that, student samples are fed into student DiT decoder to perform the typical generative diffusion task. Extensive experiments are conducted on ImageNet dataset, and our method achieves a competitive balance between training cost and generative capacity.


title: TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models

abstract: Recent advances in text-to-video generation have demonstrated the utility of powerful diffusion models. Nevertheless, the problem is not trivial when shaping diffusion models to animate static image (i.e., image-to-video generation). The difficulty originates from the aspect that the diffusion process of subsequent animated frames should not only preserve the faithful alignment with the given image but also pursue temporal coherence among adjacent frames. To alleviate this, we present TRIP, a new recipe of image-to-video diffusion paradigm that pivots on image noise prior derived from static image to jointly trigger inter-frame relational reasoning and ease the coherent temporal modeling via temporal residual learning. Technically, the image noise prior is first attained through one-step backward diffusion process based on both static image and noised video latent codes. Next, TRIP executes a residual-like dual-path scheme for noise prediction: 1) a shortcut path that directly takes image noise prior as the reference noise of each frame to amplify the alignment between the first frame and subsequent frames; 2) a residual path that employs 3D-UNet over noised video and static image latent codes to enable inter-frame relational reasoning, thereby easing the learning of the residual noise for each frame. Furthermore, both reference and residual noise of each frame are dynamically merged via attention mechanism for final video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT datasets demonstrate the effectiveness of our TRIP for image-to-video generation. Please see our project page at https://trip-i2v.github.io/TRIP/.


title: Invertible Diffusion Models for Compressed Sensing

abstract: While deep neural networks (NN) significantly advance image compressed sensing (CS) by improving reconstruction quality, the necessity of training current CS NNs from scratch constrains their effectiveness and hampers rapid deployment. Although recent methods utilize pre-trained diffusion models for image reconstruction, they struggle with slow inference and restricted adaptability to CS. To tackle these challenges, this paper proposes Invertible Diffusion Models (IDM), a novel efficient, end-to-end diffusion-based CS method. IDM repurposes a large-scale diffusion sampling process as a reconstruction model, and finetunes it end-to-end to recover original images directly from CS measurements, moving beyond the traditional paradigm of one-step noise estimation learning. To enable such memory-intensive end-to-end finetuning, we propose a novel two-level invertible design to transform both (1) the multi-step sampling process and (2) the noise estimation U-Net in each step into invertible networks. As a result, most intermediate features are cleared during training to reduce up to 93.8% GPU memory. In addition, we develop a set of lightweight modules to inject measurements into noise estimator to further facilitate reconstruction. Experiments demonstrate that IDM outperforms existing state-of-the-art CS networks by up to 2.64dB in PSNR. Compared to the recent diffusion model-based approach DDNM, our IDM achieves up to 10.09dB PSNR gain and 14.54 times faster inference.


title: DreamLIP: Language-Image Pre-training with Long Captions

abstract: Language-image pre-training largely relies on how precisely and thoroughly a text describes its paired image. In practice, however, the contents of an image can be so rich that well describing them requires lengthy captions (e.g., with 10 sentences), which are usually missing in existing datasets. Consequently, there are currently no clear evidences on whether and how language-image pre-training could benefit from long captions. To figure this out, we first re-caption 30M images with detailed descriptions using a pre-trained Multi-modality Large Language Model (MLLM), and then study the usage of the resulting captions under a contrastive learning framework. We observe that, each sentence within a long caption is very likely to describe the image partially (e.g., an object). Motivated by this, we propose to dynamically sample sub-captions from the text label to construct multiple positive pairs, and introduce a grouping loss to match the embeddings of each sub-caption with its corresponding local image patches in a self-supervised manner. Experimental results on a wide rage of downstream tasks demonstrate the consistent superiority of our method, termed DreamLIP, over previous alternatives, highlighting its fine-grained representational capacity. It is noteworthy that, on the tasks of image-text retrieval and semantic segmentation, our model trained with 30M image-text pairs achieves on par or even better performance than CLIP trained with 400M pairs. Project page is available at https://zyf0619sjtu.github.io/dream-lip.


title: FlashFace: Human Image Personalization with High-fidelity Identity Preservation

abstract: This work presents FlashFace, a practical tool with which users can easily personalize their own photos on the fly by providing one or a few reference face images and a text prompt. Our approach is distinguishable from existing human photo customization methods by higher-fidelity identity preservation and better instruction following, benefiting from two subtle designs. First, we encode the face identity into a series of feature maps instead of one image token as in prior arts, allowing the model to retain more details of the reference faces (e.g., scars, tattoos, and face shape ). Second, we introduce a disentangled integration strategy to balance the text and image guidance during the text-to-image generation process, alleviating the conflict between the reference faces and the text prompts (e.g., personalizing an adult into a "child" or an "elder"). Extensive experimental results demonstrate the effectiveness of our method on various applications, including human image personalization, face swapping under language prompts, making virtual characters into real people, etc. Project Page: https://jshilong.github.io/flashface-page.


title: Optimizing LiDAR Placements for Robust Driving Perception in Adverse Conditions

abstract: The robustness of driving perception systems under unprecedented conditions is crucial for safety-critical usages. Latest advancements have prompted increasing interests towards multi-LiDAR perception. However, prevailing driving datasets predominantly utilize single-LiDAR systems and collect data devoid of adverse conditions, failing to capture the complexities of real-world environments accurately. Addressing these gaps, we proposed Place3D, a full-cycle pipeline that encompasses LiDAR placement optimization, data generation, and downstream evaluations. Our framework makes three appealing contributions. 1) To identify the most effective configurations for multi-LiDAR systems, we introduce a Surrogate Metric of the Semantic Occupancy Grids (M-SOG) to evaluate LiDAR placement quality. 2) Leveraging the M-SOG metric, we propose a novel optimization strategy to refine multi-LiDAR placements. 3) Centered around the theme of multi-condition multi-LiDAR perception, we collect a 364,000-frame dataset from both clean and adverse conditions. Extensive experiments demonstrate that LiDAR placements optimized using our approach outperform various baselines. We showcase exceptional robustness in both 3D object detection and LiDAR semantic segmentation tasks, under diverse adverse weather and sensor failure conditions. Code and benchmark toolkit are publicly available.


title: Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding

abstract: Safety-critical 3D scene understanding tasks necessitate not only accurate but also confident predictions from 3D perception models. This study introduces Calib3D, a pioneering effort to benchmark and scrutinize the reliability of 3D scene understanding models from an uncertainty estimation viewpoint. We comprehensively evaluate 28 state-of-the-art models across 10 diverse 3D datasets, uncovering insightful phenomena that cope with both the aleatoric and epistemic uncertainties in 3D scene understanding. We discover that despite achieving impressive levels of accuracy, existing models frequently fail to provide reliable uncertainty estimates -- a pitfall that critically undermines their applicability in safety-sensitive contexts. Through extensive analysis of key factors such as network capacity, LiDAR representations, rasterization resolutions, and 3D data augmentation techniques, we correlate these aspects directly with the model calibration efficacy. Furthermore, we introduce DeptS, a novel depth-aware scaling approach aimed at enhancing 3D model calibration. Extensive experiments across a wide range of configurations validate the superiority of our method. We hope this work could serve as a cornerstone for fostering reliable 3D scene understanding. Code and benchmark toolkits are publicly available.


title: SUDO: a framework for evaluating clinical artificial intelligence systems without ground-truth annotations

abstract: A clinical artificial intelligence (AI) system is often validated on a held-out set of data which it has not been exposed to before (e.g., data from a different hospital with a distinct electronic health record system). This evaluation process is meant to mimic the deployment of an AI system on data in the wild; those which are currently unseen by the system yet are expected to be encountered in a clinical setting. However, when data in the wild differ from the held-out set of data, a phenomenon referred to as distribution shift, and lack ground-truth annotations, it becomes unclear the extent to which AI-based findings can be trusted on data in the wild. Here, we introduce SUDO, a framework for evaluating AI systems without ground-truth annotations. SUDO assigns temporary labels to data points in the wild and directly uses them to train distinct models, with the highest performing model indicative of the most likely label. Through experiments with AI systems developed for dermatology images, histopathology patches, and clinical reports, we show that SUDO can be a reliable proxy for model performance and thus identify unreliable predictions. We also demonstrate that SUDO informs the selection of models and allows for the previously out-of-reach assessment of algorithmic bias for data in the wild without ground-truth annotations. The ability to triage unreliable predictions for further inspection and assess the algorithmic bias of AI systems can improve the integrity of research findings and contribute to the deployment of ethical AI systems in medicine.