title: On the Computational Complexity of Stackelberg Planning and Meta-Operator Verification: Technical Report

abstract: Stackelberg planning is a recently introduced single-turn two-player adversarial planning model, where two players are acting in a joint classical planning task, the objective of the first player being hampering the second player from achieving its goal. This places the Stackelberg planning problem somewhere between classical planning and general combinatorial two-player games. But, where exactly? All investigations of Stackelberg planning so far focused on practical aspects. We close this gap by conducting the first theoretical complexity analysis of Stackelberg planning. We show that in general Stackelberg planning is actually no harder than classical planning. Under a polynomial plan-length restriction, however, Stackelberg planning is a level higher up in the polynomial complexity hierarchy, suggesting that compilations into classical planning come with a worst-case exponential plan-length increase. In attempts to identify tractable fragments, we further study its complexity under various planning task restrictions, showing that Stackelberg planning remains intractable where classical planning is not. We finally inspect the complexity of meta-operator verification, a problem that has been recently connected to Stackelberg planning.


title: DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions

abstract: Generating natural hand-object interactions in 3D is challenging as the resulting hand and object motions are expected to be physically plausible and semantically meaningful. Furthermore, generalization to unseen objects is hindered by the limited scale of available hand-object interaction datasets. We propose DiffH2O, a novel method to synthesize realistic, one or two-handed object interactions from provided text prompts and geometry of the object. The method introduces three techniques that enable effective learning from limited data. First, we decompose the task into a grasping stage and a text-based interaction stage and use separate diffusion models for each. In the grasping stage, the model only generates hand motions, whereas in the interaction phase both hand and object poses are synthesized. Second, we propose a compact representation that tightly couples hand and object poses. Third, we propose two different guidance schemes to allow more control of the generated motions: grasp guidance and detailed textual guidance. Grasp guidance takes a single target grasping pose and guides the diffusion model to reach this grasp at the end of the grasping stage, which provides control over the grasping pose. Given a grasping motion from this stage, multiple different actions can be prompted in the interaction phase. For textual guidance, we contribute comprehensive text descriptions to the GRAB dataset and show that they enable our method to have more fine-grained control over hand-object interactions. Our quantitative and qualitative evaluation demonstrates that the proposed method outperforms baseline methods and leads to natural hand-object motions. Moreover, we demonstrate the practicality of our framework by utilizing a hand pose estimate from an off-the-shelf pose estimator for guidance, and then sampling multiple different actions in the interaction stage.


title: The Relativistic Spin Precession in the Compact Double Neutron Star System PSR~J1946+2052

abstract: We observe systematic profile changes in the visible pulsar of the compact double neutron star system PSR~J1946+2052 using observations with the Five-hundred-meter Aperture Spherical radio Telescope (FAST). The interpulse of PSR~J1946+2052 changed from single-peak to double-peak shape from 2018 to 2021. We attribute this evolution as the result of the relativistic spin precession of the pulsar. With the high sensitivity of FAST, we also measure significant polarization for the first time, allowing us to model this with the precessional rotating vector model. Assuming, to the first order, a circular hollow-cone-like emission beam pattern and taking the validity of general relativity, we derive the binary's orbital inclination angle (${63^\circ}^{+5^\circ}_{-3^\circ}$) and pulsar's spin geometry. Pulsar's spin vector and the orbital angular momentum vector are found to be only slightly misaligned (${0.21^\circ}^{+0.28^\circ}_{-0.10^\circ}$).The quoted uncertainties do not reflect the systematic uncertainties introduced by our model assumptions. By simulating future observations of profile and polarization evolution, we estimate that we could constrain the precession rate within a $43\%$ uncertainty in 9 years. Hence, we suggest that the system's profile evolution could be combined with precise pulsar timing to test general relativity in the future.


title: A modular framework of generalized Hurwitz class numbers

abstract: We discover a non-trivial rather simple relation between the mock modular generating functions of the level $1$ and level $N$ Hurwitz class numbers. This relation yields a holomorphic modular form of weight $\frac{3}{2}$ and level $4N$, where $N > 1$ is stipulated to be odd and square-free. We extend this observation to a non-holomorphic framework and obtain a higher level non-holomorphic Zagier Eisenstein series as well as a higher level preimage of it under the Bruinier--Funke operator $\xi_{\frac{1}{2}}$. All of these observations are deduced from a more general inspection of a certain weight $\frac{1}{2}$ Maass--Eisenstein series of level $4N$ at its spectral point $s=\frac{3}{4}$. This idea goes back to Duke, Imamo\={g}lu and T\'{o}th in level $4$ and relies on the theory of so-called sesquiharmonic Maass forms. We calculate the Fourier expansion of our sesquiharmonic preimage and of its shadow. We conclude by offering an example if $N=5$ or $N=7$ and we provide the SAGE code to compute the Fourier coefficients involved.


title: Assessment of Multimodal Large Language Models in Alignment with Human Values

abstract: Large Language Models (LLMs) aim to serve as versatile assistants aligned with human values, as defined by the principles of being helpful, honest, and harmless (hhh). However, in terms of Multimodal Large Language Models (MLLMs), despite their commendable performance in perception and reasoning tasks, their alignment with human values remains largely unexplored, given the complexity of defining hhh dimensions in the visual world and the difficulty in collecting relevant data that accurately mirrors real-world situations. To address this gap, we introduce Ch3Ef, a Compreh3ensive Evaluation dataset and strategy for assessing alignment with human expectations. Ch3Ef dataset contains 1002 human-annotated data samples, covering 12 domains and 46 tasks based on the hhh principle. We also present a unified evaluation strategy supporting assessment across various scenarios and different perspectives. Based on the evaluation results, we summarize over 10 key findings that deepen the understanding of MLLM capabilities, limitations, and the dynamic relationships between evaluation levels, guiding future advancements in the field.


title: Learning the Optimal Power Flow: Environment Design Matters

abstract: To solve the optimal power flow (OPF) problem, reinforcement learning (RL) emerges as a promising new approach. However, the RL-OPF literature is strongly divided regarding the exact formulation of the OPF problem as an RL environment. In this work, we collect and implement diverse environment design decisions from the literature regarding training data, observation space, episode definition, and reward function choice. In an experimental analysis, we show the significant impact of these environment design options on RL-OPF training performance. Further, we derive some first recommendations regarding the choice of these design decisions. The created environment framework is fully open-source and can serve as a benchmark for future research in the RL-OPF field.


title: The memory of Rayleigh-Taylor turbulence

abstract: In this work, we consider the problem of inferring the initial conditions of a Rayleigh-Taylor mixing zone by measuring the 0D turbulent quantities at an unspecified time. To this aim, we have generated a comprehensive dataset through direct numerical simulations (DNS), focusing on miscible fluids with slight density contrasts. The initial interface deformations in these simulations are characterized by an annular spectrum which is parametrized by four non dimensional numbers. %In order to study the sensitivity of 0D turbulent quantities to the initial interface perturbation distributions, we build a surrogate model for the simulations using a physics-informed neural network (PINN). This allows us to compute the Sobol indices for the turbulent quantities, disentangling the effects of the initial parameters on the growth of the mixing layer. Within a Bayesian framework, we use a Markov chain Monte-Carlo method to determine the posterior distributions of initial conditions given various state variables. %This sheds light on the inertial or diffusive trajectories along with how the initial conditions are progressively forgotten during transition to turbulence. Moreover, it identifies which turbulent quantities are better predictors for the dynamics of Rayleigh-Taylor mixing zones by more effectively retaining the memory of the flow. By inferring the initial conditions and forward propagating its maximum a posteriori (MAP) estimate, we propose a strategy to model the Rayleigh-Taylor transition to turbulence.


title: GPFL: A Gradient Projection-Based Client Selection Framework for Efficient Federated Learning

abstract: Federated learning client selection is crucial for determining participant clients while balancing model accuracy and communication efficiency. Existing methods have limitations in handling data heterogeneity, computational burdens, and independent client treatment. To address these challenges, we propose GPFL, which measures client value by comparing local and global descent directions. We also employ an Exploit-Explore mechanism to enhance performance. Experimental results on FEMINST and CIFAR-10 datasets demonstrate that GPFL outperforms baselines in Non-IID scenarios, achieving over 9\% improvement in FEMINST test accuracy. Moreover, GPFL exhibits shorter computation times through pre-selection and parameter reuse in federated learning.


title: A foundation model utilizing chest CT volumes and radiology reports for supervised-level zero-shot detection of abnormalities

abstract: A major challenge in computational research in 3D medical imaging is the lack of comprehensive datasets. Addressing this issue, our study introduces CT-RATE, the first 3D medical imaging dataset that pairs images with textual reports. CT-RATE consists of 25,692 non-contrast chest CT volumes, expanded to 50,188 through various reconstructions, from 21,304 unique patients, along with corresponding radiology text reports. Leveraging CT-RATE, we developed CT-CLIP, a CT-focused contrastive language-image pre-training framework. As a versatile, self-supervised model, CT-CLIP is designed for broad application and does not require task-specific training. Remarkably, CT-CLIP outperforms state-of-the-art, fully supervised methods in multi-abnormality detection across all key metrics, thus eliminating the need for manual annotation. We also demonstrate its utility in case retrieval, whether using imagery or textual queries, thereby advancing knowledge dissemination. The open-source release of CT-RATE and CT-CLIP marks a significant advancement in medical AI, enhancing 3D imaging analysis and fostering innovation in healthcare.


title: Implementing photometric stereo for scanning helium microscopy (SHeM) to reconstruct true-to-size 3D surfaces

abstract: Scanning Helium Microscopy (SHeM) offers a combination of spatial and angular resolution via a pinhole-collimated beam of thermal energy, neutral helium-4 atoms for non-destructive imaging. This thesis introduces a novel 3D imaging mode, "heliometric stereo", enabling true-to-size 3D surface reconstruction using an adapted photometric stereo algorithm. Stereolithography (SLA) 3D printed plastics are explored for SHeM pinhole plates due to limitations in traditional machining. FormLabs "Clear Resin" via SLA printing proves ideal for rapid prototyping of vacuum components, with a developed baking protocol ensuring vacuum compatibility. The study indicates re-wetting of such plastics is a surface process over weeks. Developing 3D image reconstruction for both single and multi-detector setups required a real-space point tracking method. The point tracking method facilitates facet angle measurement in various materials, including technological and biological crystals. It has since become integral to SHeM imaging protocols for sample manipulator debugging. The thesis also details a multi-detector SHeM instrument, referred to as B-SHeM. While primarily designed to perform heliometric stereo reconstructions, the instrument also enables the range of novel SHeM experiments such as mixed-species beams to investigate inelastic scattering. The heliometric stereo methods implemented in the work have motivated the development of a GPU accelerated version of the in-house Monte-Carlo based ray tracing framework, which is the de-facto standard for SHeM image analysis. GPU parallelisation was explored as a method for decreasing simulation time and enabling previously inaccessible simulations involving complex scattering distributions and high resolution, realistic sample geometries. Preliminary testing on an analogous problem yielded a potential performance increase of up to 380 times.


title: MBE-CASSCF Approach for the Accurate Treatment of Large Active Spaces

abstract: We present a novel implementation of the complete active space self-consistent field (CASSCF) method that makes use of the many-body expanded full configuration interaction (MBE-FCI) method to incrementally approximate electronic structures within large active spaces. On the basis of a hybrid first-order algorithm employing both Super-CI and quasi-Newton strategies for the optimization of molecular orbitals, we demonstrate both computational efficacy and high accuracy of the resulting MBE-CASSCF method. We assess the performance of our implementation on a set of established numerical tests before applying MBE-CASSCF in the investigation of the triplet-quintet spin gap of iron(II) porphyrin with active spaces as large as 50 electrons in 50 orbitals.


title: GTA-HDR: A Large-Scale Synthetic Dataset for HDR Image Reconstruction

abstract: High Dynamic Range (HDR) content (i.e., images and videos) has a broad range of applications. However, capturing HDR content from real-world scenes is expensive and time-consuming. Therefore, the challenging task of reconstructing visually accurate HDR images from their Low Dynamic Range (LDR) counterparts is gaining attention in the vision research community. A major challenge in this research problem is the lack of datasets, which capture diverse scene conditions (e.g., lighting, shadows, weather, locations, landscapes, objects, humans, buildings) and various image features (e.g., color, contrast, saturation, hue, luminance, brightness, radiance). To address this gap, in this paper, we introduce GTA-HDR, a large-scale synthetic dataset of photo-realistic HDR images sampled from the GTA-V video game. We perform thorough evaluation of the proposed dataset, which demonstrates significant qualitative and quantitative improvements of the state-of-the-art HDR image reconstruction methods. Furthermore, we demonstrate the effectiveness of the proposed dataset and its impact on additional computer vision tasks including 3D human pose estimation, human body part segmentation, and holistic scene segmentation. The dataset, data collection pipeline, and evaluation code are available at: https://github.com/HrishavBakulBarua/GTA-HDR.


title: Universal entropy transport far from equilibrium across the BCS-BEC crossover

abstract: The transport properties of strongly interacting fermionic systems can reveal exotic states of matter, but experiments and theory have predominantly focused on bulk systems in the hydrodynamic limit describable with linear response coefficients such as electrical and thermal conductivity. In a ballistic channel connecting two superfluid reservoirs, recent experiments revealed a far-from-equilibrium regime beyond linear hydrodynamics where particle and entropy currents respond nonlinearly to biases of chemical potential and temperature, and their ratio is robust to the channel geometry. However, the origin of this robustness and its relation to the strong interparticle interactions remain unknown. Here, we study the coupled transport of particles and entropy tuning the interaction across the Bardeen-Cooper-Schrieffer to Bose-Einstein condensate (BCS-BEC) crossover, the reservoir degeneracy across the superfluid phase transition, as well as the local potentials and confinement of the channel. Surprisingly, the entropy advectively transported per particle depends only on the interactions and reservoir degeneracy and not on the details of the channel, suggesting that this property has its origin in the universal equilibrium properties of the reservoirs. In contrast, the magnitudes of the advective and diffusive entropy currents vary significantly with the channel details. The advective current increases monotonically towards the BEC side, which can be largely explained by the estimated superfluid gap in the channel. The Wiedemann-Franz law that links the advective and diffusive currents in Fermi liquids is most egregiously violated at unitarity, suggesting a change in the nature of the excitations responsible for entropy diffusion near unitarity. These observations pose fundamental questions regarding transport phenomena in strongly interacting Fermi systems far from equilibrium.


title: ReMamber: Referring Image Segmentation with Mamba Twister

abstract: Referring Image Segmentation (RIS) leveraging transformers has achieved great success on the interpretation of complex visual-language tasks. However, the quadratic computation cost makes it resource-consuming in capturing long-range visual-language dependencies. Fortunately, Mamba addresses this with efficient linear complexity in processing. However, directly applying Mamba to multi-modal interactions presents challenges, primarily due to inadequate channel interactions for the effective fusion of multi-modal data. In this paper, we propose ReMamber, a novel RIS architecture that integrates the power of Mamba with a multi-modal Mamba Twister block. The Mamba Twister explicitly models image-text interaction, and fuses textual and visual features through its unique channel and spatial twisting mechanism. We achieve the state-of-the-art on three challenging benchmarks. Moreover, we conduct thorough analyses of ReMamber and discuss other fusion designs using Mamba. These provide valuable perspectives for future research.


title: Negative electronic compressibility in charge islands in twisted bilayer graphene

abstract: We report on the observation of negative electronic compressibility in twisted bilayer graphene for Fermi energies close to insulating states. To observe this negative compressibility, we take advantage of naturally occurring twist angle domains that emerge during the fabrication of the samples, leading to the formation of charge islands. We accurately measure their capacitance using Coulomb oscillations, from which we infer the compressibility of the electron gas. Notably, we not only observe the negative electronic compressibility near correlated insulating states at integer filling, but also prominently near the band insulating state at full filling, located at the edges of both the flat- and remote bands. Furthermore, the individual twist angle domains yield a well-defined carrier density, enabling us to quantify the strength of electronic interactions and verify the theoretical prediction that the inverse negative capacitance contribution is proportional to the average distance between the charge carriers. A detailed analysis of our findings suggests that Wigner crystallization is the most likely explanation for the observed negative electronic compressibility.


title: Normalized B-spline-like representation for low-degree Hermite osculatory interpolation problems

abstract: This paper deals with Hermite osculatory interpolating splines. For a partition of a real interval endowed with a refinement consisting in dividing each subinterval into two small subintervals, we consider a space of smooth splines with additional smoothness at the vertices of the initial partition, and of the lowest possible degree. A normalized B-spline-like representation for the considered spline space is provided. In addition, several quasi-interpolation operators based on blossoming and control polynomials have also been developed. Some numerical tests are presented and compared with some recent works to illustrate the performance of the proposed approach.


title: Experimental Realization of Discrete Time Quasi-Crystals

abstract: Floquet (periodically driven) systems can give rise to unique non-equilibrium phases of matter without equilibrium analogs. The most prominent example is the realization of discrete time crystals. An intriguing question emerges: what other novel phases can manifest when the constraint of time periodicity is relaxed? In this study, we explore quantum systems subjected to a quasi-periodic drive. Leveraging a strongly interacting spin ensemble in diamond, we identify the emergence of long-lived discrete time quasi-crystals. Unlike conventional time crystals, time quasi-crystals exhibit robust sub-harmonic responses at multiple incommensurate frequencies. Furthermore, we show that the multi-frequency nature of the quasi-periodic drive allows for the formation of diverse patterns associated with different discrete time quasi-crystalline phases. Our findings demonstrate the existence of non-equilibrium phases in quasi-Floquet settings, significantly broadening the catalog of novel phenomena in driven many-body quantum systems.


title: A Sociotechnical Framework For Addressing Stigma and Designing Personalized Digital Health Products

abstract: Stigma, a recognized global barrier to effective disease management, impacts social interactions, resource access, and psychological well-being. In this study, we developed a patient-centered framework for deriving design requirements and interventions for health conditions subject to social stigma. This study introduces a patient-centered framework, grounded in sociotechnical systems theory, to create tailored interventions and design requirements for health conditions influenced by social stigma. We tested this framework through a mixed-method study on chronic pelvic pain patients. Our approach led to the identification of ten design requirements that encompass behavioral and psychological support and strategies for day-to-day living. The findings reveal a preference among CPP patients for priming and social support interventions. This study underscores the value of a systems-based perspective in healthcare, advocating for a nuanced, patient-centered approach that addresses the complex nature of health conditions affected by social stigma. It contributes to the ongoing discourse on integrating STS theory into healthcare frameworks, highlighting the need for targeted strategies to combat the complexities of stigma in patient care.


title: Mechanistic Design and Scaling of Hybrid Architectures

abstract: The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation. We set out to simplify this process by grounding it in an end-to-end mechanistic architecture design (MAD) pipeline, encompassing small-scale capability unit tests predictive of scaling laws. Through a suite of synthetic token manipulation tasks such as compression and recall, designed to probe capabilities, we identify and test new hybrid architectures constructed from a variety of computational primitives. We experimentally validate the resulting architectures via an extensive compute-optimal and a new state-optimal scaling law analysis, training over 500 language models between 70M to 7B parameters. Surprisingly, we find MAD synthetics to correlate with compute-optimal perplexity, enabling accurate evaluation of new architectures via isolated proxy tasks. The new architectures found via MAD, based on simple ideas such as hybridization and sparsity, outperform state-of-the-art Transformer, convolutional, and recurrent architectures (Transformer++, Hyena, Mamba) in scaling, both at compute-optimal budgets and in overtrained regimes. Overall, these results provide evidence that performance on curated synthetic tasks can be predictive of scaling laws, and that an optimal architecture should leverage specialized layers via a hybrid topology.


title: TractOracle: towards an anatomically-informed reward function for RL-based tractography

abstract: Reinforcement learning (RL)-based tractography is a competitive alternative to machine learning and classical tractography algorithms due to its high anatomical accuracy obtained without the need for any annotated data. However, the reward functions so far used to train RL agents do not encapsulate anatomical knowledge which causes agents to generate spurious false positives tracts. In this paper, we propose a new RL tractography system, TractOracle, which relies on a reward network trained for streamline classification. This network is used both as a reward function during training as well as a mean for stopping the tracking process early and thus reduce the number of false positive streamlines. This makes our system a unique method that evaluates and reconstructs WM streamlines at the same time. We report an improvement of true positive ratios by almost 20\% and a reduction of 3x of false positive ratios on one dataset and an increase between 2x and 7x in the number true positive streamlines on another dataset.


title: Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation

abstract: Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, large-scale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting language-grounded robotic navigation. In this work, we present HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with open-vocabulary features. Our approach is able to represent multi-story buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within real-world multi-storage environments. We provide code and trial video data at http://hovsg.github.io/.


title: Climate Downscaling: A Deep-Learning Based Super-resolution Model of Precipitation Data with Attention Block and Skip Connections

abstract: Human activities accelerate consumption of fossil fuels and produce greenhouse gases, resulting in urgent issues today: global warming and the climate change. These indirectly cause severe natural disasters, plenty of lives suffering and huge losses of agricultural properties. To mitigate impacts on our lands, scientists are developing renewable, reusable, and clean energies and climatologists are trying to predict the extremes. Meanwhile, governments are publicizing resource-saving policies for a more eco-friendly society and arousing environment awareness. One of the most influencing factors is the precipitation, bringing condensed water vapor onto lands. Water resources are the most significant but basic needs in society, not only supporting our livings, but also economics. In Taiwan, although the average annual precipitation is up to 2,500 millimeter (mm), the water allocation for each person is lower than the global average due to drastically geographical elevation changes and uneven distribution through the year. Thus, it is crucial to track and predict the rainfall to make the most use of it and to prevent the floods. However, climate models have limited resolution and require intensive computational power for local-scale use. Therefore, we proposed a deep convolutional neural network with skip connections, attention blocks, and auxiliary data concatenation, in order to downscale the low-resolution precipitation data into high-resolution one. Eventually, we compare with other climate downscaling methods and show better performance in metrics of Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Pearson Correlation, structural similarity index (SSIM), and forecast indicators.


title: ArabicaQA: A Comprehensive Dataset for Arabic Question Answering

abstract: In this paper, we address the significant gap in Arabic natural language processing (NLP) resources by introducing ArabicaQA, the first large-scale dataset for machine reading comprehension and open-domain question answering in Arabic. This comprehensive dataset, consisting of 89,095 answerable and 3,701 unanswerable questions created by crowdworkers to look similar to answerable ones, along with additional labels of open-domain questions marks a crucial advancement in Arabic NLP resources. We also present AraDPR, the first dense passage retrieval model trained on the Arabic Wikipedia corpus, specifically designed to tackle the unique challenges of Arabic text retrieval. Furthermore, our study includes extensive benchmarking of large language models (LLMs) for Arabic question answering, critically evaluating their performance in the Arabic language context. In conclusion, ArabicaQA, AraDPR, and the benchmarking of LLMs in Arabic question answering offer significant advancements in the field of Arabic NLP. The dataset and code are publicly accessible for further research https://github.com/DataScienceUIBK/ArabicaQA.


title: Multi Agent Pathfinding for Noise Restricted Hybrid Fuel Unmanned Aerial Vehicles

abstract: Multi Agent Path Finding (MAPF) seeks the optimal set of paths for multiple agents from respective start to goal locations such that no paths conflict. We address the MAPF problem for a fleet of hybrid-fuel unmanned aerial vehicles which are subject to location-dependent noise restrictions. We solve this problem by searching a constraint tree for which the subproblem at each node is a set of shortest path problems subject to the noise and fuel constraints and conflict zone avoidance. A labeling algorithm is presented to solve this subproblem, including the conflict zones which are treated as dynamic obstacles. We present the experimental results of the algorithms for various graph sizes and number of agents.


title: A Mixed-Integer Linear Program to create the shifts in a supermarket

abstract: The shift design and the personnel scheduling problem is known to be a difficult problem. It is a real-world problem which has lots of applications in the organization of companies. Solutions are usually found by dividing the problem in two steps: first the shifts are created, then the employees are assigned to them by respecting a bunch of constraints. The assignment of different tasks increases the complexity, since we have to consider the skills of the single employee necessary to perform any activity. In this paper we present a mixed-integer linear programming formulation which models together the shift creation and the construction of rosters for employees, with the objective of minimizing the amount of uncovered demand. Finally we provide the results for three real-world instances, confirming that this approach is promising.


title: A survey on road safety and traffic efficiency vehicular applications based on C-V2X technologies

abstract: In recent years, the use of cellular network technologies to provide communication-based applications to vehicles has received considerable attention. 3GPP, the standardization body responsible for cellular networks specifications, is developing technologies to meet the requirements of vehicular communication applications, and the research community is testing and validating the ability of those technologies to implement different applications. This survey presents the body of work dealing with the use of cellular technologies to implement communication-based applications for the connected vehicle. We focus on basic and advanced road safety and traffic efficiency applications, which are critically important for the future of vehicular networks. We start by describing the different cellular-related technologies that have a role to play in providing services to the connected vehicle, propose a classification of types of communication used in vehicular applications, and then apply this classification to organize and present recent research work on the topic. Finally, we identify the main challenges in the use of cellular technologies to develop applications for the connected vehicle.


title: Counterfactual Fairness through Transforming Data Orthogonal to Bias

abstract: Machine learning models have shown exceptional prowess in solving complex issues across various domains. Nonetheless, these models can sometimes exhibit biased decision-making, leading to disparities in treatment across different groups. Despite the extensive research on fairness, the nuanced effects of multivariate and continuous sensitive variables on decision-making outcomes remain insufficiently studied. We introduce a novel data pre-processing algorithm, Orthogonal to Bias (OB), designed to remove the influence of a group of continuous sensitive variables, thereby facilitating counterfactual fairness in machine learning applications. Our approach is grounded in the assumption of a jointly normal distribution within a structural causal model (SCM), proving that counterfactual fairness can be achieved by ensuring the data is uncorrelated with sensitive variables. The OB algorithm is model-agnostic, catering to a wide array of machine learning models and tasks, and includes a sparse variant to enhance numerical stability through regularization. Through empirical evaluation on simulated and real-world datasets - including the adult income and the COMPAS recidivism datasets - our methodology demonstrates its capacity to enable fairer outcomes without compromising accuracy.


title: Using Domain Knowledge to Guide Dialog Structure Induction via Neural Probabilistic Soft Logic

abstract: Dialog Structure Induction (DSI) is the task of inferring the latent dialog structure (i.e., a set of dialog states and their temporal transitions) of a given goal-oriented dialog. It is a critical component for modern dialog system design and discourse analysis. Existing DSI approaches are often purely data-driven, deploy models that infer latent states without access to domain knowledge, underperform when the training corpus is limited/noisy, or have difficulty when test dialogs exhibit distributional shifts from the training domain. This work explores a neural-symbolic approach as a potential solution to these problems. We introduce Neural Probabilistic Soft Logic Dialogue Structure Induction (NEUPSL DSI), a principled approach that injects symbolic knowledge into the latent space of a generative neural model. We conduct a thorough empirical investigation on the effect of NEUPSL DSI learning on hidden representation quality, few-shot learning, and out-of-domain generalization performance. Over three dialog structure induction datasets and across unsupervised and semi-supervised settings for standard and cross-domain generalization, the injection of symbolic knowledge using NEUPSL DSI provides a consistent boost in performance over the canonical baselines.


title: Scaling of high-Rayleigh-number convection based on internal convective boundary layer

abstract: We propose a phenomenological model for thermal convection at high Rayleigh numbers. It hypothesizes existence of a high-Reynolds-number turbulent boundary layer near each horizontal plate, which is shown to be convective. The convective logarithmic friction law of Tong and Ding (2020) is used to relate the large-scale velocity to the induced friction velocity. The predicted scaling relations of the Nusselt ($Nu$) and Reynolds numbers ($Re$) on the Rayleigh number ($Ra$) do not have a power law form, each being a single function, suggesting a single flow regime with no transition, instead of multiple regimes. However, the predicted $Nu$ and $Re$ scaling is close to $Ra^{1/3}$ and $Ra^{4/9}$ for $Ra \sim 10^9$ to $10^{17}$, consistent with direct numerical simulation (DNS) results up to $Ra\sim 10^{15}$. For $Ra<10^9$, the model also correctly predicts the deviations observed in DNS from the above power law scaling. The model predicts deviations from $Ra^{1/3}$ beyond $Ra \sim 10^{17}$ with the local scaling exponent approaching $1/2$, which would likely require data at $Ra>10^{18}$ to verify.


title: Emittance preservation in a plasma-wakefield accelerator

abstract: Radio-frequency particle accelerators are engines of discovery, powering high-energy physics and photon science, but are also large and expensive due to their limited accelerating fields. Plasma-wakefield accelerators (PWFAs) provide orders-of-magnitude stronger fields in the charge-density wave behind a particle bunch travelling in a plasma, promising particle accelerators of greatly reduced size and cost. However, PWFAs can easily degrade the beam quality of the bunches they accelerate. Emittance, which determines how tightly beams can be focused, is a critical beam quality in for instance colliders and free-electron lasers, but is particularly prone to degradation. We demonstrate, for the first time, emittance preservation in a high-gradient and high-efficiency PWFA while simultaneously preserving charge and energy spread. This establishes that PWFAs can accelerate without degradation$\unicode{x2014}$essential for energy boosters in photon science and multistage facilities for compact high-energy particle colliders.


title: Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMs

abstract: Lexical-syntactic flexibility, in the form of conversion (or zero-derivation) is a hallmark of English morphology. In conversion, a word with one part of speech is placed in a non-prototypical context, where it is coerced to behave as if it had a different part of speech. However, while this process affects a large part of the English lexicon, little work has been done to establish the degree to which language models capture this type of generalization. This paper reports the first study on the behavior of large language models with reference to conversion. We design a task for testing lexical-syntactic flexibility -- the degree to which models can generalize over words in a construction with a non-prototypical part of speech. This task is situated within a natural language inference paradigm. We test the abilities of five language models -- two proprietary models (GPT-3.5 and GPT-4), three open-source models (Mistral 7B, Falcon 40B, and Llama 2 70B). We find that GPT-4 performs best on the task, followed by GPT-3.5, but that the open source language models are also able to perform it and that the 7B parameter Mistral displays as little difference between its baseline performance on the natural language inference task and the non-prototypical syntactic category task, as the massive GPT-4.


title: Ill-posedness of the hydrostatic Euler-Boussinesq equations and failure of hydrostatic limit

abstract: We investigate the hydrostatic approximation for inviscid stratified fluids, described by the two-dimensional Euler-Boussinesq equations in a periodic channel. Through a perturbative analysis of the hydrostatic homogeneous setting, we exhibit a stratified steady state violating the Miles-Howard criterion and generating a growing mode, both for the linearized hydrostatic and non-hydrostatic equations. By leveraging long-wave nonlinear instability for the original Euler-Boussinesq system, we demonstrate the breakdown of the hydrostatic limit around such unstable profiles. Finally, we establish the generic nonlinear ill-posedness of the limiting hydrostatic system in Sobolev spaces.


title: Parallelizable Parametric Nonlinear System Identification via tuning of a Moving Horizon State Estimator

abstract: This paper introduces a novel optimization-based approach for parametric nonlinear system identification. Building upon the prediction error method framework, traditionally used for linear system identification, we extend its capabilities to nonlinear systems. The predictions are computed using a moving horizon state estimator with a constant arrival cost. Eventually, both the system parameters and the arrival cost are estimated by minimizing the sum of the squared prediction errors. Since the predictions are induced by the state estimator, the method can be viewed as the tuning of a state estimator, based on its predictive capacities. The present extension of the prediction error method not only enhances performance for nonlinear systems but also enables learning from multiple trajectories with unknown initial states, broadening its applicability in practical scenarios. Additionally, the novel formulation leaves room for the design of efficient and parallelizable optimization algorithms, since each output prediction only depends on a fixed window of past actions and measurements. In the special case of linear time-invariant systems, we show an important property of the proposed method which suggests asymptotic consistency under reasonable assumptions. Numerical examples illustrate the effectiveness and practicality of the approach, and one of the examples also highlights the necessity for the arrival cost.


title: ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on Historical American Newspaper Pages

abstract: Question answering (QA) and Machine Reading Comprehension (MRC) tasks have significantly advanced in recent years due to the rapid development of deep learning techniques and, more recently, large language models. At the same time, many benchmark datasets have become available for QA and MRC tasks. However, most existing large-scale benchmark datasets have been created predominantly using synchronous document collections like Wikipedia or the Web. Archival document collections, such as historical newspapers, contain valuable information from the past that is still not widely used to train large language models. To further contribute to advancing QA and MRC tasks and to overcome the limitation of previous datasets, we introduce ChroniclingAmericaQA, a large-scale temporal QA dataset with 487K question-answer pairs created based on the historical newspaper collection Chronicling America. Our dataset is constructed from a subset of the Chronicling America newspaper collection spanning 120 years. One of the significant challenges for utilizing digitized historical newspaper collections is the low quality of OCR text. Therefore, to enable realistic testing of QA models, our dataset can be used in three different ways: answering questions from raw and noisy content, answering questions from cleaner, corrected version of the content, as well as answering questions from scanned images of newspaper pages. This and the fact that ChroniclingAmericaQA spans the longest time period among available QA datasets make it quite a unique and useful resource.


title: Exploring LLMs as a Source of Targeted Synthetic Textual Data to Minimize High Confidence Misclassifications

abstract: Natural Language Processing (NLP) models optimized for predictive performance often make high confidence errors and suffer from vulnerability to adversarial and out-of-distribution data. Existing work has mainly focused on mitigation of such errors using either humans or an automated approach. In this study, we explore the usage of large language models (LLMs) for data augmentation as a potential solution to the issue of NLP models making wrong predictions with high confidence during classification tasks. We compare the effectiveness of synthetic data generated by LLMs with that of human data obtained via the same procedure. For mitigation, humans or LLMs provide natural language characterizations of high confidence misclassifications to generate synthetic data, which are then used to extend the training set. We conduct an extensive evaluation of our approach on three classification tasks and demonstrate its effectiveness in reducing the number of high confidence misclassifications present in the model, all while maintaining the same level of accuracy. Moreover, we find that the cost gap between humans and LLMs surpasses an order of magnitude, as LLMs attain human-like performance while being more scalable.


title: Stealthy Deactivation of Safety Filters

abstract: Safety filters ensure that only safe control actions are executed. We propose a simple and stealthy false-data injection attack for deactivating such safety filters; in particular, we focus on deactivating safety filters that are based on control-barrier functions. The attack injects false sensor measurements to bias state estimates to the interior of a safety region, which makes the safety filter accept unsafe control actions. To detect such attacks, we also propose a detector that detects biases manufactured by the proposed attack policy, which complements conventional detectors when safety filters are used. The proposed attack policy and detector are illustrated on a double integrator example.


title: Proceedings Sixth Workshop on Models for Formal Analysis of Real Systems

abstract: This volume contains the proceedings of MARS 2024, the sixth workshop on Models for Formal Analysis of Real Systems, held as part of ETAPS 2024, the European Joint Conferences on Theory and Practice of Software. The MARS workshops bring together researchers from different communities who are developing formal models of real systems in areas where complex models occur, such as networks, cyber-physical systems, hardware/software co-design, biology, etc. The motivation and aim for MARS stem from the following two observations: (1) Large case studies are essential to show that specification formalisms and modelling techniques are applicable to real systems, whereas many research papers only consider toy examples or tiny case studies. (2) Developing an accurate model of a real system takes a large amount of time, often months or years. In most scientific papers, however, salient details of the model need to be skipped due to lack of space, and to leave room for formal verification methodologies and results. The MARS workshops aim at remedying these issues, emphasising modelling over verification, so as to retain lessons learnt from formal modelling, which are not usually discussed elsewhere.


title: An AI-Native Runtime for Multi-Wearable Environments

abstract: The miniaturization of AI accelerators is paving the way for next-generation wearable applications within wearable technologies. We introduce Mojito, an AI-native runtime with advanced MLOps designed to facilitate the development and deployment of these applications on wearable devices. It emphasizes the necessity of dynamic orchestration of distributed resources equipped with ultra-low-power AI accelerators to overcome challenges associated with unpredictable runtime environments. Through its innovative approaches, Mojito demonstrates how future wearable technologies can evolve to be more autonomous.


title: Synthetic training set generation using text-to-audio models for environmental sound classification

abstract: In the past few years, text-to-audio models have emerged as a significant advancement in automatic audio generation. Although they represent impressive technological progress, the effectiveness of their use in the development of audio applications remains uncertain. This paper aims to investigate these aspects, specifically focusing on the task of classification of environmental sounds. This study analyzes the performance of two different environmental classification systems when data generated from text-to-audio models is used for training. Two cases are considered: a) when the training dataset is augmented by data coming from two different text-to-audio models; and b) when the training dataset consists solely of synthetic audio generated. In both cases, the performance of the classification task is tested on real data. Results indicate that text-to-audio models are effective for dataset augmentation, whereas the performance of the models drops when relying on only generated audio.


title: X-ray Properties of the Luminous Quasar PG 1634+706 at z = 1.337 from SRG and XMM-Newton Data

abstract: In the fall of 2019, during the in-flight calibration phase of the SRG observatory, the onboard eROSITA and Mikhail Pavlinsky ART-XC telescopes carried out a series of observations of PG 1634+706 - one of the most luminous (an X-ray luminosity $\sim 10^{46}$ erg/s) quasars in the Universe at $z<2$. Approximately at the same dates this quasar was also observed by the XMM-Newton observatory. Although the object had already been repeatedly studied in X-rays previously, its new observations allowed its energy spectrum to be measured more accurately in the wide range $1-30$ keV (in the quasar rest frame). Its spectrum can be described by a two-component model that consists of a power-law continuum with a slope $\Gamma\approx 1.9$ and a broadened iron emission line at an energy of about 6.4 keV. The X-ray variability of the quasar was also investigated. On time scales of the order of several hours (here and below, in the source rest frame) the X-ray luminosity does not exhibit a statistically significant variability. However, it changed noticeably from observation to observation in the fall of 2019, having increased approximately by a factor of 1.5 in 25 days. A comparison of the new SRG and XMM-Newton measurements with the previous measurements of other X-ray observatories has shown that in the entire 17-year history of observations of the quasar PG 1634+706 its X-ray luminosity has varied by no more than a factor of 2.5, while the variations on time scales of several weeks and several years are comparable in amplitude.


title: A Floquet analysis perspective of driven light-matter interaction models

abstract: In this paper, we analyze the harmonically driven Jaynes-Cummings and Lipkin-Meshkov-Glick models using both numerical integration of time-dependent Hamiltonians and Floquet theory. For a separation of time-scales between the drive and intrinsic Rabi oscillations in the former model, the driving results in an effective periodic reversal of time. The corresponding Floquet Hamiltonian is a Wannier-Stark model, which can be analytically solved. Despite the chaotic nature of the driven Lipkin-Meshkov-Glick model, moderate system sizes can display qualitatively different behaviors under varying system parameters. Ergodicity arises in systems that are neither adiabatic nor diabatic, owing to repeated multi-level Landau-Zener transitions. Chaotic behavior, observed in slow driving, manifests as random jumps in the magnetization, suggesting potential utility as a random number generator. Furthermore, we discuss both models in terms of what we call Floquet Fock state lattices.


title: The Adams conjecture and intersections of local Arthur packets

abstract: The Adams conjecture states that the local theta correspondence sends a local Arthur packet to another local Arthur packet. M{\oe}glin confirmed the conjecture when lifting to groups of sufficiently high rank and also showed that it fails in low rank. Recently, Baki\'c and Hanzer described when the Adams conjecture holds in low rank for a representation in a fixed local Arthur packet. However, a representation may lie in many local Arthur packets and each gives a minimal rank for which the Adams conjecture holds. In this paper, we study the interplay of intersections of local Arthur packets with the Adams conjecture.


title: An invitation to the sample complexity of quantum hypothesis testing

abstract: Quantum hypothesis testing (QHT) has been traditionally studied from the information-theoretic perspective, wherein one is interested in the optimal decay rate of error probabilities as a function of the number of samples of an unknown state. In this paper, we study the sample complexity of QHT, wherein the goal is to determine the minimum number of samples needed to reach a desired error probability. By making use of the wealth of knowledge that already exists in the literature on QHT, we characterize the sample complexity of binary QHT in the symmetric and asymmetric settings, and we provide bounds on the sample complexity of multiple QHT. In more detail, we prove that the sample complexity of symmetric binary QHT depends logarithmically on the inverse error probability and inversely on the negative logarithm of the fidelity. As a counterpart of the quantum Stein's lemma, we also find that the sample complexity of asymmetric binary QHT depends logarithmically on the inverse type II error probability and inversely on the quantum relative entropy, provided that the type II error probability is sufficiently small. We then provide lower and upper bounds on the sample complexity of multiple QHT, with it remaining an intriguing open question to improve these bounds. The final part of our paper outlines and reviews how sample complexity of QHT is relevant to a broad swathe of research areas and can enhance understanding of many fundamental concepts, including quantum algorithms for simulation and search, quantum learning and classification, and foundations of quantum mechanics. As such, we view our paper as an invitation to researchers coming from different communities to study and contribute to the problem of sample complexity of QHT, and we outline a number of open directions for future research.

categories: quant-ph cs.IT cs.LG math.IT math.ST stat.TH


title: To Supervise or Not to Supervise: Understanding and Addressing the Key Challenges of 3D Transfer Learning

abstract: Transfer learning has long been a key factor in the advancement of many fields including 2D image analysis. Unfortunately, its applicability in 3D data processing has been relatively limited. While several approaches for 3D transfer learning have been proposed in recent literature, with contrastive learning gaining particular prominence, most existing methods in this domain have only been studied and evaluated in limited scenarios. Most importantly, there is currently a lack of principled understanding of both when and why 3D transfer learning methods are applicable. Remarkably, even the applicability of standard supervised pre-training is poorly understood. In this work, we conduct the first in-depth quantitative and qualitative investigation of supervised and contrastive pre-training strategies and their utility in downstream 3D tasks. We demonstrate that layer-wise analysis of learned features provides significant insight into the downstream utility of trained networks. Informed by this analysis, we propose a simple geometric regularization strategy, which improves the transferability of supervised pre-training. Our work thus sheds light onto both the specific challenges of 3D transfer learning, as well as strategies to overcome them.


title: Boosting Diffusion Models with Moving Average Sampling in Frequency Domain

abstract: Diffusion models have recently brought a powerful revolution in image generation. Despite showing impressive generative capabilities, most of these models rely on the current sample to denoise the next one, possibly resulting in denoising instability. In this paper, we reinterpret the iterative denoising process as model optimization and leverage a moving average mechanism to ensemble all the prior samples. Instead of simply applying moving average to the denoised samples at different timesteps, we first map the denoised samples to data space and then perform moving average to avoid distribution shift across timesteps. In view that diffusion models evolve the recovery from low-frequency components to high-frequency details, we further decompose the samples into different frequency components and execute moving average separately on each component. We name the complete approach "Moving Average Sampling in Frequency domain (MASF)". MASF could be seamlessly integrated into mainstream pre-trained diffusion models and sampling schedules. Extensive experiments on both unconditional and conditional diffusion models demonstrate that our MASF leads to superior performances compared to the baselines, with almost negligible additional complexity cost.


title: On a combinatorial classification of fine compactified universal Jacobians

abstract: Extending the definition of $V$-stability conditions, given by Viviani in a recent preprint, we introduce the notion of \emph{universal stability conditions}. Building on results by Pagani and Tommasi, we show that fine compactified universal Jacobians, that is, fine compactified Jacobians over the moduli spaces of stable pointed curves $\overline{\mathcal{M}}_{g,n}$, are combinatorially classified by universal stability conditions. We use these stability conditions to show the following. The inclusion of fine compactified universal Jacobians of type $(g,n)$ whose fibres over geometric points are classical, that is, they are constructed by some numerical polarisation, into the class of all fine compactified universal Jacobians, is strict, in general, for any $g\geq 2$. This answers a question of Pagani and Tommasi.


title: The relation between the gonality and the Clifford index of a chain of cycles

abstract: For a chain of cycles $\Gamma$ we prove Cliff($\Gamma$)=gon($\Gamma$)-2.


title: Addressing Social Misattributions of Large Language Models: An HCXAI-based Approach

abstract: Human-centered explainable AI (HCXAI) advocates for the integration of social aspects into AI explanations. Central to the HCXAI discourse is the Social Transparency (ST) framework, which aims to make the socio-organizational context of AI systems accessible to their users. In this work, we suggest extending the ST framework to address the risks of social misattributions in Large Language Models (LLMs), particularly in sensitive areas like mental health. In fact LLMs, which are remarkably capable of simulating roles and personas, may lead to mismatches between designers' intentions and users' perceptions of social attributes, risking to promote emotional manipulation and dangerous behaviors, cases of epistemic injustice, and unwarranted trust. To address these issues, we propose enhancing the ST framework with a fifth 'W-question' to clarify the specific social attributions assigned to LLMs by its designers and users. This addition aims to bridge the gap between LLM capabilities and user perceptions, promoting the ethically responsible development and use of LLM-based technology.


title: Scaling Mixed-Integer Programming for Certification of Neural Network Controllers Using Bounds Tightening

abstract: Neural networks offer a computationally efficient approximation of model predictive control, but they lack guarantees on the resulting controlled system's properties. Formal certification of neural networks is crucial for ensuring safety, particularly in safety-critical domains such as autonomous vehicles. One approach to formally certify properties of neural networks is to solve a mixed-integer program based on the network. This approach suffers from scalability issues due to the complexity of solving the resulting mixed-integer programs. Nevertheless, these issues can be (partially) mitigated via bound-tightening techniques prior to forming the mixed-integer program, which results in tighter formulations and faster optimisation. This paper presents bound-tightening techniques in the context of neural network explicit control policies. Bound tightening is particularly important when considering problems spanning multiple time steps of a controlled system, as the bounds must be propagated through the problem depth. Several strategies for bound tightening are evaluated in terms of both computational complexity and tightness of the bounds.


title: The Solution to an Impulse Control Problem Motivated by Optimal Harvesting

abstract: We consider a stochastic impulse control problem that is motivated by applications such as the optimal exploitation of a natural resource. In particular, we consider a stochastic system whose uncontrolled state dynamics are modelled by a non-explosive positive linear diffusion. The control that can be applied to this system takes the form of one-sided impulsive action. The objective of the control problem is to maximise a discounted performance criterion that rewards the effect of control action but involves a fixed cost at each time of a control intervention. We derive the complete solution to this problem under general assumptions. It turns out that the solution can take four qualitatively different forms, several of which have not been observed in the literature. In two of the four cases, there exist only $\varepsilon$-optimal control strategies. We also show that the boundary classification of 0 may play a critical role in the solution of the problem. Furthermore, we develop a way for establishing the strong solution to a stochastic impulse control problem's optimally controlled SDE.


title: MIND Your Language: A Multilingual Dataset for Cross-lingual News Recommendation

abstract: Digital news platforms use news recommenders as the main instrument to cater to the individual information needs of readers. Despite an increasingly language-diverse online community, in which many Internet users consume news in multiple languages, the majority of news recommendation focuses on major, resource-rich languages, and English in particular. Moreover, nearly all news recommendation efforts assume monolingual news consumption, whereas more and more users tend to consume information in at least two languages. Accordingly, the existing body of work on news recommendation suffers from a lack of publicly available multilingual benchmarks that would catalyze development of news recommenders effective in multilingual settings and for low-resource languages. Aiming to fill this gap, we introduce xMIND, an open, multilingual news recommendation dataset derived from the English MIND dataset using machine translation, covering a set of 14 linguistically and geographically diverse languages, with digital footprints of varying sizes. Using xMIND, we systematically benchmark several state-of-the-art content-based neural news recommenders (NNRs) in both zero-shot (ZS-XLT) and few-shot (FS-XLT) cross-lingual transfer scenarios, considering both monolingual and bilingual news consumption patterns. Our findings reveal that (i) current NNRs, even when based on a multilingual language model, suffer from substantial performance losses under ZS-XLT and that (ii) inclusion of target-language data in FS-XLT training has limited benefits, particularly when combined with a bilingual news consumption. Our findings thus warrant a broader research effort in multilingual and cross-lingual news recommendation. The xMIND dataset is available at https://github.com/andreeaiana/xMIND.


title: Identifying codes in graphs of given maximum degree. II. Triangle-free graphs

abstract: An $\textit{identifying code}$ of a closed-twin-free graph $G$ is a set $S$ of vertices of $G$ such that any two vertices in $G$ have a distinct intersection between their closed neighborhood and $S$. It was conjectured that there exists a constant $c$ such that for every connected closed-twin-free graph $G$ of order $n$ and maximum degree $\Delta$, the graph $G$ admits an identifying code of size at most $\left( \frac{\Delta-1}{\Delta} \right) n+c$. In a companion paper, we proved the conjecture for all trees. We show that the conjecture holds for all triangle-free graphs, with the same lists of exceptional graphs needing $c>0$ as for trees: for $\Delta\ge 3$, $c=1/3$ suffices and there is only a set of 12 trees requiring $c>0$ for $\Delta=3$, and when $\Delta\ge 4$ this set is reduced to the $\Delta$-star only. Our proof is by induction, whose starting point is the above result for trees. Along the way, we prove a generalized version of Bondy's theorem on induced subsets [J. A. Bondy. Induced subsets. Journal of Combinatorial Theory, Series B, 1972] that we use as a tool in our proofs. We also use our main result for triangle-free graphs, to prove the upper bound $\left( \frac{\Delta-1}{\Delta} \right) n+1/\Delta+4t$ for graphs that can be made triangle-free by the removal of $t$ edges.


title: Empowering Data Mesh with Federated Learning

abstract: The evolution of data architecture has seen the rise of data lakes, aiming to solve the bottlenecks of data management and promote intelligent decision-making. However, this centralized architecture is limited by the proliferation of data sources and the growing demand for timely analysis and processing. A new data paradigm, Data Mesh, is proposed to overcome these challenges. Data Mesh treats domains as a first-class concern by distributing the data ownership from the central team to each data domain, while keeping the federated governance to monitor domains and their data products. Many multi-million dollar organizations like Paypal, Netflix, and Zalando have already transformed their data analysis pipelines based on this new architecture. In this decentralized architecture where data is locally preserved by each domain team, traditional centralized machine learning is incapable of conducting effective analysis across multiple domains, especially for security-sensitive organizations. To this end, we introduce a pioneering approach that incorporates Federated Learning into Data Mesh. To the best of our knowledge, this is the first open-source applied work that represents a critical advancement toward the integration of federated learning methods into the Data Mesh paradigm, underscoring the promising prospects for privacy-preserving and decentralized data analysis strategies within Data Mesh architecture.


title: Low-Latency Neural Stereo Streaming

abstract: The rise of new video modalities like virtual reality or autonomous driving has increased the demand for efficient multi-view video compression methods, both in terms of rate-distortion (R-D) performance and in terms of delay and runtime. While most recent stereo video compression approaches have shown promising performance, they compress left and right views sequentially, leading to poor parallelization and runtime performance. This work presents Low-Latency neural codec for Stereo video Streaming (LLSS), a novel parallel stereo video coding method designed for fast and efficient low-latency stereo video streaming. Instead of using a sequential cross-view motion compensation like existing methods, LLSS introduces a bidirectional feature shifting module to directly exploit mutual information among views and encode them effectively with a joint cross-view prior model for entropy coding. Thanks to this design, LLSS processes left and right views in parallel, minimizing latency; all while substantially improving R-D performance compared to both existing neural and conventional codecs.


title: Double polytropic cosmic acceleration from the Murnaghan equation of state

abstract: We consider a double polytropic cosmological fluid and demonstrate that, when one constituent resembles a bare cosmological constant while the other emulates a generalized Chaplygin gas, a good description of the Universe's large-scale dynamics is obtained. In particular, our double polytropic reduces to the Murnaghan equation of state, whose applications are already well established in solid state physics and classical thermodynamics. Intriguingly, our model approximates the conventional $\Lambda$CDM paradigm while reproducing the collective effects of logotropic and generalized Chaplygin fluids across different regimes. To check the goodness of our fluid description, we analyze first order density perturbations, refining our model through various orders of approximation, utilizing $\sigma_8$ data alongside other cosmological data sets. Encouraging results suggest that our model, based on the Murnaghan equation of state, outperforms the standard cosmological background within specific approximate regimes and, on the whole, surpasses the standard phenomenological reconstruction of dark energy.


title: Deepfake Generation and Detection: A Benchmark and Survey

abstract: Deepfake is a technology dedicated to creating highly realistic facial images and videos under specific conditions, which has significant application potential in fields such as entertainment, movie production, digital human creation, to name a few. With the advancements in deep learning, techniques primarily represented by Variational Autoencoders and Generative Adversarial Networks have achieved impressive generation results. More recently, the emergence of diffusion models with powerful generation capabilities has sparked a renewed wave of research. In addition to deepfake generation, corresponding detection technologies continuously evolve to regulate the potential misuse of deepfakes, such as for privacy invasion and phishing attacks. This survey comprehensively reviews the latest developments in deepfake generation and detection, summarizing and analyzing current state-of-the-arts in this rapidly evolving field. We first unify task definitions, comprehensively introduce datasets and metrics, and discuss developing technologies. Then, we discuss the development of several related sub-fields and focus on researching four representative deepfake fields: face swapping, face reenactment, talking face generation, and facial attribute editing, as well as forgery detection. Subsequently, we comprehensively benchmark representative methods on popular datasets for each field, fully evaluating the latest and influential published works. Finally, we analyze challenges and future research directions of the discussed fields.


title: On the properties of distance covariance for categorical data: Robustness, sure screening, and approximate null distributions

abstract: Pearson's Chi-squared test, though widely used for detecting association between categorical variables, exhibits low statistical power in large sparse contingency tables. To address this limitation, two novel permutation tests have been recently developed: the distance covariance permutation test and the U-statistic permutation test. Both leverage the distance covariance functional but employ different estimators. In this work, we explore key statistical properties of the distance covariance for categorical variables. Firstly, we show that unlike Chi-squared, the distance covariance functional is B-robust for any number of categories (fixed or diverging). Second, we establish the strong consistency of distance covariance screening under mild conditions, and simulations confirm its advantage over Chi-squared screening, especially for large sparse tables. Finally, we derive an approximate null distribution for a bias-corrected distance correlation estimate, demonstrating its effectiveness through simulations.


title: Superior and Pragmatic Talking Face Generation with Teacher-Student Framework

abstract: Talking face generation technology creates talking videos from arbitrary appearance and motion signal, with the "arbitrary" offering ease of use but also introducing challenges in practical applications. Existing methods work well with standard inputs but suffer serious performance degradation with intricate real-world ones. Moreover, efficiency is also an important concern in deployment. To comprehensively address these issues, we introduce SuperFace, a teacher-student framework that balances quality, robustness, cost and editability. We first propose a simple but effective teacher model capable of handling inputs of varying qualities to generate high-quality results. Building on this, we devise an efficient distillation strategy to acquire an identity-specific student model that maintains quality with significantly reduced computational load. Our experiments validate that SuperFace offers a more comprehensive solution than existing methods for the four mentioned objectives, especially in reducing FLOPs by 99\% with the student model. SuperFace can be driven by both video and audio and allows for localized facial attributes editing.


title: Sen2Fire: A Challenging Benchmark Dataset for Wildfire Detection using Sentinel Data

abstract: Utilizing satellite imagery for wildfire detection presents substantial potential for practical applications. To advance the development of machine learning algorithms in this domain, our study introduces the \textit{Sen2Fire} dataset--a challenging satellite remote sensing dataset tailored for wildfire detection. This dataset is curated from Sentinel-2 multi-spectral data and Sentinel-5P aerosol product, comprising a total of 2466 image patches. Each patch has a size of 512$\times$512 pixels with 13 bands. Given the distinctive sensitivities of various wavebands to wildfire responses, our research focuses on optimizing wildfire detection by evaluating different wavebands and employing a combination of spectral indices, such as normalized burn ratio (NBR) and normalized difference vegetation index (NDVI). The results suggest that, in contrast to using all bands for wildfire detection, selecting specific band combinations yields superior performance. Additionally, our study underscores the positive impact of integrating Sentinel-5 aerosol data for wildfire detection. The code and dataset are available online (https://zenodo.org/records/10881058).


title: Empirical Analysis of EIP-3675: Miner Dynamics, Transaction Fees, and Transaction Time

abstract: The Ethereum Improvement Proposal 3675 (EIP-3675) marks a significant shift, transitioning from a Proof of Work (PoW) to a Proof of Stake (PoS) consensus mechanism. This transition resulted in a staggering 99.95% decrease in energy consumption. However, the transition prompts two critical questions: (1). How does EIP-3675 affect miners' dynamics? and (2). How do users determine priority fees, considering that paying too little may cause delays or non-inclusion, yet paying too much wastes money with little to no benefits? To address the first question, we present a comprehensive empirical study examining EIP-3675's effect on miner dynamics (i.e., miner participation, distribution, and the degree of randomness in miner selection). Our findings reveal that the transition has encouraged broader participation of miners in block append operation, resulting in a larger pool of unique miners ($\approx50\times$ PoW), and the change in miner distribution with the increased number of unique small category miners ($\approx60\times$ PoW). However, there is an unintended consequence: a reduction in the miner selection randomness, which signifies the negative impact of the transition to PoS-Ethereum on network decentralization. Regarding the second question, we employed regression-based machine learning models; the Gradient Boosting Regressor performed best in predicting priority fees, while the K-Neighbours Regressor was worst.


title: Neural Embedding Compression For Efficient Multi-Task Earth Observation Modelling

abstract: As repositories of large scale data in earth observation (EO) have grown, so have transfer and storage costs for model training and inference, expending significant resources. We introduce Neural Embedding Compression (NEC), based on the transfer of compressed embeddings to data consumers instead of raw data. We adapt foundation models (FM) through learned neural compression to generate multi-task embeddings while navigating the tradeoff between compression rate and embedding utility. We update only a small fraction of the FM parameters (10%) for a short training period (1% of the iterations of pre-training). We evaluate NEC on two EO tasks: scene classification and semantic segmentation. Compared with applying traditional compression to the raw data, NEC achieves similar accuracy with a 75% to 90% reduction in data. Even at 99.7% compression, performance drops by only 5% on the scene classification task. Overall, NEC is a data-efficient yet performant approach for multi-task EO modelling.


title: The Unreasonable Ineffectiveness of the Deeper Layers

abstract: We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed. To prune these models, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. In particular, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single A100 GPU. From a practical perspective, these results suggest that layer pruning methods can complement other PEFT strategies to further reduce computational resources of finetuning on the one hand, and can improve the memory and latency of inference on the other hand. From a scientific perspective, the robustness of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play a critical role in storing knowledge.


title: 2D Gaussian Splatting for Geometrically Accurate Radiance Fields

abstract: 3D Gaussian Splatting (3DGS) has recently revolutionized radiance field reconstruction, achieving high quality novel view synthesis and fast rendering speed without baking. However, 3DGS fails to accurately represent surfaces due to the multi-view inconsistent nature of 3D Gaussians. We present 2D Gaussian Splatting (2DGS), a novel approach to model and reconstruct geometrically accurate radiance fields from multi-view images. Our key idea is to collapse the 3D volume into a set of 2D oriented planar Gaussian disks. Unlike 3D Gaussians, 2D Gaussians provide view-consistent geometry while modeling surfaces intrinsically. To accurately recover thin surfaces and achieve stable optimization, we introduce a perspective-correct 2D splatting process utilizing ray-splat intersection and rasterization. Additionally, we incorporate depth distortion and normal consistency terms to further enhance the quality of the reconstructions. We demonstrate that our differentiable renderer allows for noise-free and detailed geometry reconstruction while maintaining competitive appearance quality, fast training speed, and real-time rendering.


title: Large scale paired antibody language models

abstract: Antibodies are proteins produced by the immune system that can identify and neutralise a wide variety of antigens with high specificity and affinity, and constitute the most successful class of biotherapeutics. With the advent of next-generation sequencing, billions of antibody sequences have been collected in recent years, though their application in the design of better therapeutics has been constrained by the sheer volume and complexity of the data. To address this challenge, we present IgBert and IgT5, the best performing antibody-specific language models developed to date which can consistently handle both paired and unpaired variable region sequences as input. These models are trained comprehensively using the more than two billion unpaired sequences and two million paired sequences of light and heavy chains present in the Observed Antibody Space dataset. We show that our models outperform existing antibody and protein language models on a diverse range of design and regression tasks relevant to antibody engineering. This advancement marks a significant leap forward in leveraging machine learning, large scale data sets and high-performance computing for enhancing antibody design for therapeutic development.


title: Searching for large dark matter clumps using the Galileo Satnav clock variations

abstract: This study presents bounds on transient variations of fundamental constants, with typical timescales ranging from minutes to months, using clocks in space. The underlying phenomenology describing such transient variations relies on models for Dark Matter (DM) which suggest possible encounters of macroscopic compact objects with the Earth, due to the motion of the solar system in the galactic halo. If such compact objects possess an effective feeble interaction with the ordinary matter beyond the gravitational one, it may result in effective transient variations of fundamental constants. Such variations leave signatures on clocks onboard GNSS satellites. In this paper, we introduce a phenomenological study dedicated to the search for such DM transient objects using the network of passive hydrogen masers (H-Masers) onboard Galileo satellites. We first model the signature of transient variations of fundamental constants as a frequency modulation in the difference between two satellite clocks, considering the satellite trajectories relative to the transient event. Then, we present first results based on a fast analysis method, the maximum reach analysis. The main result is a significant extension of the discovery range for DM transients, with a sensitivity never achieved before. We investigate indeed the range of transient sizes from $10^5$ to $10^9$ kilometres, which, apart from indirect and model-dependent non-transient effects, has never been explored previously.

categories: astro-ph.CO gr-qc hep-ph physics.atom-ph


title: Image-based Novel Fault Detection with Deep Learning Classifiers using Hierarchical Labels

abstract: One important characteristic of modern fault classification systems is the ability to flag the system when faced with previously unseen fault types. This work considers the unknown fault detection capabilities of deep neural network-based fault classifiers. Specifically, we propose a methodology on how, when available, labels regarding the fault taxonomy can be used to increase unknown fault detection performance without sacrificing model performance. To achieve this, we propose to utilize soft label techniques to improve the state-of-the-art deep novel fault detection techniques during the training process and novel hierarchically consistent detection statistics for online novel fault detection. Finally, we demonstrated increased detection performance on novel fault detection in inspection images from the hot steel rolling process, with results well replicated across multiple scenarios and baseline detection methods.


title: Density of group languages in shift spaces

abstract: We study the density of group languages (i.e. rational languages recognized by morphisms onto finite groups) inside shift spaces. The density of a rational language can be understood as the frequency of some "pattern" in the shift space, for example a pattern like "words with an even number of a given letter." In this paper, we handle density of group languages via ergodicity of skew products between the shift space and the recognizing group. We consider both the cases of shifts of finite type (with a suitable notion of irreducibility), and of minimal shifts. In the latter case, our main result is a closed formula for the density which holds whenever the skew product has minimal closed invariant subsets which are ergodic under the product of the original measure and the uniform probability measure on the group. The formula is derived in part from a characterization of minimal closed invariant subsets for skew products relying on notions of cocycles and coboundaries. In the case where the whole skew product itself is ergodic under the product measure, then the density is completely determined by the cardinality of the image of the language inside the recognizing group. We provide sufficient conditions for the skew product to have minimal closed invariant subsets that are ergodic under the product measure. Finally, we investigate the link between minimal closed invariant subsets, return words and bifix codes.


title: A Survey on 3D Egocentric Human Pose Estimation

abstract: Egocentric human pose estimation aims to estimate human body poses and develop body representations from a first-person camera perspective. It has gained vast popularity in recent years because of its wide range of applications in sectors like XR-technologies, human-computer interaction, and fitness tracking. However, to the best of our knowledge, there is no systematic literature review based on the proposed solutions regarding egocentric 3D human pose estimation. To that end, the aim of this survey paper is to provide an extensive overview of the current state of egocentric pose estimation research. In this paper, we categorize and discuss the popular datasets and the different pose estimation models, highlighting the strengths and weaknesses of different methods by comparative analysis. This survey can be a valuable resource for both researchers and practitioners in the field, offering insights into key concepts and cutting-edge solutions in egocentric pose estimation, its wide-ranging applications, as well as the open problems with future scope.


title: Anomalous supercurrent and diode effect in locally perturbed topological Josephson junctions

abstract: The simultaneous breaking of time-reversal and inversion symmetry can lead to peculiar effects in Josephson junctions, such as the anomalous Josephson effect or supercurrent rectification, which is a dissipationless analog of the diode effect. Due to their impact in new quantum technologies, it is important to find robust platforms and external means to manipulate the above effects in a controlled way. Here, we theoretically consider a Josephson junction based on a quantum spin Hall system as the normal channel, subjected to a magnetic field in the direction defined by spin-momentum locking, and in presence of a local tip in close proximity to one of the metallic edges in the normal region. We consider different local perturbations, model normal and magnetic tips, and study how they affect the Josephson response of the device. In particular, we argue that magnetic tips are a useful tool that allows for tunability of both $\phi_0$ response and supercurrent rectification.


title: Global asymptotics for $\beta$-Krawtchouk corners processes via multi-level loop equations

abstract: We introduce a two-parameter family of probability distributions, indexed by $\beta/2 = \theta > 0$ and $K \in \mathbb{Z}_{\geq 0}$, that are called $\beta$-Krawtchouk corners processes. These measures are related to Jack symmetric functions, and can be thought of as integrable discretizations of $\beta$-corners processes from random matrix theory, or alternatively as non-determinantal measures on lozenge tilings of infinite domains. We show that as $K$ tends to infinity the height function of these models concentrates around an explicit limit shape, and prove that its fluctuations are asymptotically described by a pull-back of the Gaussian free field, which agrees with the one for Wigner matrices. The main tools we use to establish our results are certain multi-level loop equations introduced in our earlier work arXiv:2108.07710.


title: Dynamical evolution of the Uranian satellite system I. From the 5/3 Ariel-Umbriel mean motion resonance to the present

abstract: Mutual gravitational interactions between the five major Uranian satellites raise small quasi-periodic fluctuations on their orbital elements. At the same time, tidal interactions between the satellites and the planet induce a slow outward drift of the orbits, while damping the eccentricities and the inclinations. In this paper, we revisit the current and near past evolution of this system using a N-body integrator, including spin evolution and tidal dissipation with the weak friction model. We update the secular eigenmodes of the system and show that it is unlikely that any of the main satellites were recently captured into a high obliquity Cassini state. We rather expect that the Uranian satellites are in a low obliquity Cassini state and compute their values. We also show that the current eccentricities of the satellites are not forced, and estimate the free eccentricities and inclinations. We constrain the quality factor of Uranus to be $Q_U = (8.6 \pm 2.9)\times10^3$, and that of the satellites to be $Q_S \sim 500$. We find that the system most likely encountered the 5/3 mean motion resonance between Ariel and Umbriel in the past, at about ($0.7\pm0.2$) Gyr ago. We additionally determine the eccentricities and inclinations of all satellites just after the resonance passage that comply with the current system. We finally show that, from the crossing of the 5/3 MMR to the present, the evolution of the system is mostly peaceful and dominated by tides raised on Uranus by the satellites.


title: Dynamical evolution of the Uranian satellite system II. Crossing of the 5/3 Ariel-Umbriel mean motion resonance

abstract: At present, the main satellites of Uranus are not involved in any low order mean motion resonance (MMR). However, owing to tides raised in the planet, Ariel and Umbriel most likely crossed the 5/3 MMR in the past. Previous studies on this resonance passage relied on limited time-consuming N-body simulations or simplified models focusing solely on the effects of the eccentricity or the inclination. In this paper, we aim to provide a more comprehensive view on how the system evaded capture in the 5/3 MMR. For that purpose, we developed a secular resonant two-satellite model with low eccentricities and low inclinations, including tides using the weak friction model. By performing a large number of numerical simulations, we show that capture in the 5/3 MMR is certain if the initial eccentricities of Ariel, $e_1$, and Umbriel, $e_2$, are related through $(e_1^2 + e_2^2)^{1/2} < 0.007$. Moreover, we observe that the eccentricity of Ariel is the key variable to evade the 5/3 MMR with a high probability. We determine that for $e_1 > 0.015$ and $e_2 < 0.01$, the system avoids capture in at least 60\% of the cases. We also show that, to replicate the currently observed system, the initial inclinations of Ariel and Umbriel must lay within $I_1 \leq 0.05^{\circ}$ and $0.06^{\circ} \leq I_2 \leq 0.11^{\circ}$, respectively. We checked these results using a complete N-body model with the five main satellites and did not observe any significant differences.


title: Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians

abstract: The recent 3D Gaussian splatting (3D-GS) has shown remarkable rendering fidelity and efficiency compared to NeRF-based neural scene representations. While demonstrating the potential for real-time rendering, 3D-GS encounters rendering bottlenecks in large scenes with complex details due to an excessive number of Gaussian primitives located within the viewing frustum. This limitation is particularly noticeable in zoom-out views and can lead to inconsistent rendering speeds in scenes with varying details. Moreover, it often struggles to capture the corresponding level of details at different scales with its heuristic density control operation. Inspired by the Level-of-Detail (LOD) techniques, we introduce Octree-GS, featuring an LOD-structured 3D Gaussian approach supporting level-of-detail decomposition for scene representation that contributes to the final rendering results. Our model dynamically selects the appropriate level from the set of multi-resolution anchor points, ensuring consistent rendering performance with adaptive LOD adjustments while maintaining high-fidelity rendering results.


title: A severe local flood and social events show a similar impact on human mobility

abstract: While a social event, such as a concert or a food festival, is a common experience to people, a natural disaster is experienced by a fewer individuals. The ordinary and common ground experience of social events could be therefore used to better understand the complex impacts of uncommon, but devastating natural events on society, such as floods. Based on this idea, we present a comparison - in terms of human mobility -, between an extreme local flood that occurred in 2017 in Switzerland, and social events which took place in the same region, in the weeks before and after the inundation. Using mobile phone location data, we show that the severe local flood and social events have a similar impact on human mobility, both at the national scale and at a local scale. At the national level, we found a small difference between the distributions of visitors and their travelled distances among the several weeks in which the events took place. At the local level, instead, we detected the anomalies (in time series) in the number of people travelling each road and railway, and we found that the distributions of anomalies, and of their clusters, are comparable between the flood and the social events. Hence, our findings suggest that the knowledge on ubiquitous social events can be employed to characterise the impacts of rare natural disasters on human mobility. The proposed methods at the local level can thus be used to analyse the disturbances in complex spatial networks and, in general, as complementary approaches for the analyses of complex systems.


title: On the Dynamics of Point-Vortices with Positive Intensities collapsing with the boundary

abstract: In this paper we study the point-vortex dynamics with positive intensities. We show that in the half-plane and in a disk, collapses of point-vortices with the boundary in finite time are impossible, hence the solution of the dynamics is global in time. We also give some necessary conditions for the existence of collapses with the boundary in general smooth bounded domains, in particular that the trajectory of at least one point-vortex has no limit. Some minor results are obtained with signed intensities.


title: Search and Society: Reimagining Information Access for Radical Futures

abstract: Information retrieval (IR) technologies and research are undergoing transformative changes. It is our perspective that the community should accept this opportunity to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others. In this perspective paper, we motivate why the community must consider this radical shift in how we do research and what we work on, and sketch a path forward towards this transformation.


title: Serpent: Scalable and Efficient Image Restoration via Multi-scale Structured State Space Models

abstract: The landscape of computational building blocks of efficient image restoration architectures is dominated by a combination of convolutional processing and various attention mechanisms. However, convolutional filters, while efficient, are inherently local and therefore struggle with modeling long-range dependencies in images. In contrast, attention excels at capturing global interactions between arbitrary image regions, but suffers from a quadratic cost in image dimension. In this work, we propose Serpent, an efficient architecture for high-resolution image restoration that combines recent advances in state space models (SSMs) with multi-scale signal processing in its core computational block. SSMs, originally introduced for sequence modeling, can maintain a global receptive field with a favorable linear scaling in input size. We propose a novel hierarchical architecture inspired by traditional signal processing principles, that converts the input image into a collection of sequences and processes them in a multi-scale fashion. Our experimental results demonstrate that Serpent can achieve reconstruction quality on par with state-of-the-art techniques, while requiring orders of magnitude less compute (up to $150$ fold reduction in FLOPS) and a factor of up to $5\times$ less GPU memory while maintaining a compact model size. The efficiency gains achieved by Serpent are especially notable at high image resolutions.