title: Automatic Identification of Driving Maneuver Patterns using a Robust Hidden Semi-Markov Models

abstract: There is an increase in interest to model driving maneuver patterns via the automatic unsupervised clustering of naturalistic sequential kinematic driving data. The patterns learned are often used in transportation research areas such as eco-driving, road safety, and intelligent vehicles. One such model capable of modeling these patterns is the Hierarchical Dirichlet Process Hidden Semi-Markov Model (HDP-HSMM), as it is often used to estimate data segmentation, state duration, and transition probabilities. While this model is a powerful tool for automatically clustering observed sequential data, the existing HDP-HSMM estimation suffers from an inherent tendency to overestimate the number of states. This can result in poor estimation, which can potentially impact impact transportation research through incorrect inference of driving patterns. In this paper, a new robust HDP-HSMM (rHDP-HSMM) method is proposed to reduce the number of redundant states and improve the consistency of the model's estimation. Both a simulation study and a case study using naturalistic driving data are presented to demonstrate the effectiveness of the proposed rHDP-HSMM in identifying and inference of driving maneuver patterns.


title: Wind Roche-lobe Overflow in Low-Mass Binaries: Exploring the Origin of Rapidly Rotating Blue Lurkers

abstract: Wind Roche-Lobe Overflow (WRLOF) is a mass-transfer mechanism proposed by Mohamed and Podsiadlowski (2007) for stellar binaries wherein the wind acceleration zone of the donor star exceeds its Roche lobe radius, allowing stellar wind material to be transferred to the accretor at enhanced rates. WRLOF may explain characteristics observed in blue lurkers and blue stragglers. While WRLOF has been implemented in rapid population synthesis codes, it has yet to be explored thoroughly in detailed binary models such as MESA (a 1D stellar evolution code), and over a wide range of initial binary configurations. We incorporate WRLOF accretion in MESA to investigate wide low-mass binaries at solar metallicity. We perform a parameter study over the initial orbital period and stellar mass. In most of the models where we consider angular momentum transfer during accretion, the accretor is spun up to the critical (or break-up) rotation rate. Then we assume the star develops a boosted wind to efficiently reduce the angular momentum so that it could maintain a sub-critical rotation. Balanced by boosted wind loss, the accretor only gains $\sim 2\%$ of its total mass, but can maintain a near-critical rotation rate during WRLOF. Notably, the mass-transfer efficiency is significantly smaller than in previous studies in which the rotation of the accretor is ignored. We compare our results to observational data of blue lurkers in M67 and find that the WRLOF mechanism can qualitatively explain the origin of their rapid rotation, their location on the HR diagram and their orbital periods.


title: An optical tweezer array of ultracold polyatomic molecules

abstract: Polyatomic molecules have rich structural features that make them uniquely suited to applications in quantum information science, quantum simulation, ultracold chemistry, and searches for physics beyond the Standard Model. However, a key challenge is fully controlling both the internal quantum state and the motional degrees of freedom of the molecules. Here, we demonstrate the creation of an optical tweezer array of individual polyatomic molecules, CaOH, with quantum control of their internal quantum state. The complex quantum structure of CaOH results in a non-trivial dependence of the molecules' behavior on the tweezer light wavelength. We control this interaction and directly and nondestructively image individual molecules in the tweezer array with >90% fidelity. The molecules are manipulated at the single internal quantum state level, thus demonstrating coherent state control in a tweezer array. The platform demonstrated here will enable a variety of experiments using individual polyatomic molecules with arbitrary spatial arrangement.


title: Retrievals Applied To A Decision Tree Framework Can Characterize Earth-like Exoplanet Analogs

abstract: Exoplanet characterization missions planned for the future will soon enable searches for life beyond our solar system. Critical to the search will be the development of life detection strategies that can search for biosignatures while maintaining observational efficiency. In this work, we adopted a newly developed biosignature decision tree strategy for remote characterization of Earth-like exoplanets. The decision tree offers a step-by-step roadmap for detecting exoplanet biosignatures and excluding false positives based on Earth's biosphere and its evolution over time. We followed the pathways for characterizing a modern Earth-like planet and an Archean Earth-like planet and evaluated the observational trades associated with coronagraph bandpass combinations of designs consistent with The Habitable Worlds Observatory (HWO) precursor studies. With retrieval analyses of each bandpass (or combination), we demonstrate the utility of the decision tree and evaluated the uncertainty on a suite of biosignature chemical species and habitability indicators (i.e., the gas abundances of H$_2$O, O$_2$, O$_3$, CH$_4$, and CO$_2$). Notably for modern Earth, less than an order of magnitude spread in the 1-$\sigma$ uncertainties were achieved for the abundances of H$_2$O and O$_2$, planetary surface pressure, and atmospheric temperature with three strategically placed bandpasses (two in the visible and one in the near-infrared). For the Archean, CH$_4$ and H$_2$O were detectable in the visible with a single bandpass.


title: On compatible Hom-Lie triple systems

abstract: In this paper, we consider compatible Hom-Lie triple systems. Compatible Hom-Lie triple systems are characterized as Maurer-Cartan elements in a suitable bidifferential graded Lie algebra. We also define a cohomology theory for compatible Hom-Lie triple systems. As applications of cohomology, we study abelian extensions and deformations of compatible Hom-Lie triple systems.


title: It's Not Easy Being Wrong: Large Language Models Struggle with Process of Elimination Reasoning

abstract: Chain-of-thought (COT) prompting can help large language models (LLMs) reason toward correct answers, but its efficacy in reasoning toward incorrect answers is unexplored. This process of elimination (PoE), when used with COT, can enhance self-consistency, interpretability, and tasks such as medical diagnoses of exclusion. Thus, we propose PoE with COT, where LLMs must reason toward incorrect options on multiple-choice questions. We evaluate the ability of GPT-3.5, LLaMA-2, and Falcon to perform PoE with COT on a total of four commonsense and scientific reasoning datasets. We find that the strategy of PoE always underperforms the strategy of choosing the correct answer. The agreement of these strategies is also lower than the self-consistency of each strategy. To study these issues further, we conduct error analyses and give suggestions for future work.


title: Transfer learning electronic structure and continuum model for twisted bilayer MoTe2

abstract: We investigate the lattice relaxation effect on moir\'e band structures in twisted bilayer MoTe$_2$ with two approaches: (a) large-scale plane-wave basis first principle calculation down to $2.88^{\circ}$, (b) transfer learning structure relaxation + local-basis first principles calculation down to $1.1^{\circ}$. Two types of van der Waals corrections have been examined: the D2 method of Grimme and the density-dependent energy correction. We note the density-dependent energy correction yields a continuous evolution of bandwidth with twist angles. Including second harmonic of intralayer potential/interlayer tunneling and the strain induced gauge field, we develop a more complete continuum model with a single set of parameters for a wide range of twist angles, \textcolor{red}{ and perform many-body simulations at $\nu=-1,-2/3, -1/3$. We further identify a series of $C=1$ Chern bands around $2^{\circ}$ from Wilson loop and edges state calculation, which serves as the foundation point for even-denominator non-Abelian states.


title: Unsupervised Musical Object Discovery from Audio

abstract: Current object-centric learning models such as the popular SlotAttention architecture allow for unsupervised visual scene decomposition. Our novel MusicSlots method adapts SlotAttention to the audio domain, to achieve unsupervised music decomposition. Since concepts of opacity and occlusion in vision have no auditory analogues, the softmax normalization of alpha masks in the decoders of visual object-centric models is not well-suited for decomposing audio objects. MusicSlots overcomes this problem. We introduce a spectrogram-based multi-object music dataset tailored to evaluate object-centric learning on western tonal music. MusicSlots achieves good performance on unsupervised note discovery and outperforms several established baselines on supervised note property prediction tasks.


title: Causality Diagrams using Hybrid Vector Clocks

abstract: Causality in distributed systems is a concept that has long been explored and numerous approaches have been made to use causality as a way to trace distributed system execution. Traditional approaches usually used system profiling and newer approaches profiled clocks of systems to detect failures and construct timelines that caused those failures. Since the advent of logical clocks, these profiles have become more and more accurate with ways to characterize concurrency and distributions, with accurate diagrams for message passing. Vector clocks addressed the shortcomings of using traditional logical clocks, by storing information about other processes in the system as well. Hybrid vector clocks are a novel approach to this concept where clocks need not store all the process information. Rather, we store information of processes within an acceptable skew of the focused process. This gives us an efficient way of profiling with substantially reduced costs to the system. Building on this idea, we propose the idea of building causal traces using information generated from the hybrid vector clock. The hybrid vector clock would provide us with a strong sense of concurrency and distribution, and we theorize that all the information generated from the clock is sufficient to develop a causal trace for debugging. We post-process and parse the clocks generated from an execution trace to develop a swimlane on a web interface, that traces the points of failure of a distributed system. We also provide an API to reuse this concept for any generic distributed system framework.


title: A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering

abstract: The emergence of multimodal large models (MLMs) has significantly advanced the field of visual understanding, offering remarkable capabilities in the realm of visual question answering (VQA). Yet, the true challenge lies in the domain of knowledge-intensive VQA tasks, which necessitate not just recognition of visual elements, but also a deep comprehension of the visual information in conjunction with a vast repository of learned knowledge. To uncover such capabilities of MLMs, particularly the newly introduced GPT-4V, we provide an in-depth evaluation from three perspectives: 1) Commonsense Knowledge, which assesses how well models can understand visual cues and connect to general knowledge; 2) Fine-grained World Knowledge, which tests the model's skill in reasoning out specific knowledge from images, showcasing their proficiency across various specialized fields; 3) Comprehensive Knowledge with Decision-making Rationales, which examines model's capability to provide logical explanations for its inference, facilitating a deeper analysis from the interpretability perspective. Extensive experiments indicate that GPT-4V achieves SOTA performance on above three tasks. Interestingly, we find that: a) GPT-4V demonstrates enhanced reasoning and explanation when using composite images as few-shot; b) GPT-4V produces severe hallucinations when dealing with world knowledge, highlighting the future need for advancements in this research direction.


title: Estimating optical vegetation indices with Sentinel-1 SAR data and AutoML

abstract: Current optical vegetation indices (VIs) for monitoring forest ecosystems are widely used in various applications. However, continuous monitoring based on optical satellite data can be hampered by atmospheric effects such as clouds. On the contrary, synthetic aperture radar (SAR) data can offer insightful and systematic forest monitoring with complete time series due to signal penetration through clouds and day and night acquisitions. The goal of this work is to overcome the issues affecting optical data with SAR data and serve as a substitute for estimating optical VIs for forests using machine learning. Time series of four VIs (LAI, FAPAR, EVI and NDVI) were estimated using multitemporal Sentinel-1 SAR and ancillary data. This was enabled by creating a paired multi-temporal and multi-modal dataset in Google Earth Engine (GEE), including temporally and spatially aligned Sentinel-1, Sentinel-2, digital elevation model (DEM), weather and land cover datasets (MMT-GEE). The use of ancillary features generated from DEM and weather data improved the results. The open-source Automatic Machine Learning (AutoML) approach, auto-sklearn, outperformed Random Forest Regression for three out of four VIs, while a 1-hour optimization length was enough to achieve sufficient results with an R2 of 69-84% low errors (0.05-0.32 of MAE depending on VI). Great agreement was also found for selected case studies in the time series analysis and in the spatial comparison between the original and estimated SAR-based VIs. In general, compared to VIs from currently freely available optical satellite data and available global VI products, a better temporal resolution (up to 240 measurements/year) and a better spatial resolution (20 m) were achieved using estimated SAR-based VIs. A great advantage of the SAR-based VI is the ability to detect abrupt forest changes with a sub-weekly temporal accuracy.


title: Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers

abstract: Recent approaches have explored language-guided classifiers capable of classifying examples from novel tasks when provided with task-specific natural language explanations, instructions or prompts (Sanh et al., 2022; R. Menon et al., 2022). While these classifiers can generalize in zero-shot settings, their task performance often varies substantially between different language explanations in unpredictable ways (Lu et al., 2022; Gonen et al., 2022). Also, current approaches fail to leverage unlabeled examples that may be available in many scenarios. Here, we introduce TALC, a framework that uses data programming to adapt a language-guided classifier for a new task during inference when provided with explanations from multiple teachers and unlabeled test examples. Our results show that TALC consistently outperforms a competitive baseline from prior work by an impressive 9.3% (relative improvement). Further, we demonstrate the robustness of TALC to variations in the quality and quantity of provided explanations, highlighting its potential in scenarios where learning from multiple teachers or a crowd is involved. Our code is available at: https://github.com/WeiKangda/TALC.git.


title: Crepant resolutions of stratified varieties via gluing

abstract: Let $X$ be a variety with a stratification $\mathcal{S}$ into smooth locally closed subvarieties such that $X$ is locally a product along each stratum (e.g., a symplectic singularity). We prove that assigning to each open subset $U \subset X$ the set of isomorphism classes of locally projective crepant resolutions of $U$ defines an $\mathcal{S}$-constructible sheaf of sets. Thus, for each stratum $S$ and basepoint $s \in S$, the fundamental group acts on the set of germs of projective crepant resolutions at $s$, leaving invariant the germs extending to the entire stratum. Global locally projective crepant resolutions correspond to compatible such choices for all strata. For example, if the local projective crepant resolutions are unique, they automatically glue uniquely. We give criteria for a locally projective crepant resolution $\rho: \tilde X \to X$ to be globally projective. We show that the sheafification of the presheaf $U \mapsto \operatorname{Pic}(\rho^{-1}(U)/U)$ of relative Picard classes is also constructible. The resolution is globally projective only if there exist local relatively ample bundles whose classes glue to a global section of this sheaf. The obstruction to lifting this section to a global ample line bundle is encoded by a gerbe on the singularity $X$. We show the gerbes are automatically trivial if $X$ is a symplectic quotient singularity. Our main results hold in the more general setting of partial crepant resolutions, that need not have smooth source. We apply the theory to symmetric powers and Hilbert schemes of surfaces with du Val singularities, finite quotients of tori, multiplicative and Nakajima quiver varieties, as well as to canonical threefold singularities.


title: Finding planted cliques using Markov chain Monte Carlo

abstract: The planted clique problem is a paradigmatic model of statistical-to-computational gaps: the planted clique is information-theoretically detectable if its size $k\ge 2\log_2 n$ but polynomial-time algorithms only exist for the recovery task when $k= \Omega(\sqrt{n})$. By now, there are many simple and fast algorithms that succeed as soon as $k = \Omega(\sqrt{n})$. Glaringly, however, no MCMC approach to the problem had been shown to work, including the Metropolis process on cliques studied by Jerrum since 1992. In fact, Chen, Mossel, and Zadik recently showed that any Metropolis process whose state space is the set of cliques fails to find any sub-linear sized planted clique in polynomial time if initialized naturally from the empty set. Here, we redeem MCMC performance for the planted clique problem by relaxing the state space to all vertex subsets and adding a corresponding energy penalty for missing edges. With that, we prove that energy-minimizing Markov chains (gradient descent and a low-temperature relaxation of it) succeed at recovering planted cliques of size $k = \Omega(\sqrt{n})$ if initialized from the full graph. Importantly, initialized from the empty set, the relaxation still does not help the gradient descent find sub-linear planted cliques. We also demonstrate robustness of these Markov chain approaches under a natural contamination model.


title: mlscorecheck: Testing the consistency of reported performance scores and experiments in machine learning

abstract: Addressing the reproducibility crisis in artificial intelligence through the validation of reported experimental results is a challenging task. It necessitates either the reimplementation of techniques or a meticulous assessment of papers for deviations from the scientific method and best statistical practices. To facilitate the validation of reported results, we have developed numerical techniques capable of identifying inconsistencies between reported performance scores and various experimental setups in machine learning problems, including binary/multiclass classification and regression. These consistency tests are integrated into the open-source package mlscorecheck, which also provides specific test bundles designed to detect systematically recurring flaws in various fields, such as retina image processing and synthetic minority oversampling.


title: Liouville theorems for conformally invariant fully nonlinear equations. I

abstract: A fundamental theorem of Liouville asserts that positive entire harmonic functions in Euclidean spaces must be constant. A remarkable Liouville-type theorem of Caffarelli-Gidas-Spruck states that positive entire solutions of $-\Delta u=u^{ \frac {n+2}{n-2} }$, $n\ge 3$, are unique modulo M\"obius transformations. Far-reaching extensions were established for general fully nonlinear conformally invariant equations through the works of Chang-Gursky-Yang, Li-Li, Li, and Viaclovsky. In this paper, we derive necessary and sufficient conditions for the validity of such Liouville-type theorems. This leads to necessary and sufficient conditions for local gradient estimates of solutions to hold, assuming a one-sided bound on the solutions, for a wide class of fully nonlinear elliptic equations involving Schouten tensors. A pivotal advancement in proving these Liouville-type theorems is our enhanced understanding of solutions to such equations near isolated singularities. In particular, we utilize earlier results of Caffarelli-Li-Nirenberg on lower- and upper-conical singularities. For general conformally invariant fully nonlinear elliptic equations, we prove that a viscosity super- (sub-)solution can be extended across an isolated singularity if and only if it is a lower- (upper-)conical singularity. We also provide necessary and sufficient conditions for lower- (upper-)conical behavior of a function near isolated singularities in terms of its conformal Hessian.


title: $q$-Analogue of the degree zero part of a rational Cherednik algebra

abstract: Inside the double affine Hecke algebra $\mathbb{H}_{n, q, \tau}$ of type $GL_n$, we define a subalgebra $\mathbb{H}^{\mathfrak{gl}_n}$ that may be thought of as a $q$-deformation of the degree zero part of the corresponding rational Cherednik algebra. We prove that the algebra $\mathbb{H}^{\mathfrak{gl}_n}$ is a flat $\tau$-deformation of the semi-direct product of the group algebra $\mathbb{C} \mathfrak{S}_n$ of the symmetric group with the image of the Drinfeld-Jimbo quantum group $U_q(\mathfrak{gl}_n)$ under the $q$-oscillator (Jordan-Schwinger) representation. We find all the defining relations and an explicit PBW basis for the algebra $\mathbb{H}^{\mathfrak{gl}_n}$. We describe its centre and establish a double centraliser property. Further, we develop the connection with integrable systems introduced by van Diejen, which we also generalise.


title: Non-affinity of liquid networks and bicontinuous mesophases

abstract: Amphiphiles self-assemble into a variety of bicontinuous mesophases whose equilibrium structures take the form of high-symmetry cubic networks. Here, we show that the symmetry-breaking distortions in these systems give rise to anomalously large, non-affine collective deformations, which we argue to be a generic consequence of mass equilibration within deformed networks. We propose and study a minimal liquid network model of bicontinuous networks, in which acubic distortions are modeled by the relaxation of residually-stressed mechanical networks with constant-tension bonds. We show that non-affinity is strongly dependent on the valency of the network as well as the degree of strain-softening/stiffening force in the bonds. Taking diblock copolymer melts as a model system, liquid network theory captures quantitative features of two bicontinuous phases based on comparison with self-consistent field theory predictions and direct experimental characterization of acubic distortions, which are likely to be pronounced in soft amphiphilic systems more generally.

categories: cond-mat.soft cond-mat.mes-hall cond-mat.mtrl-sci cond-mat.stat-mech


title: Optical manifestations of topological Euler class in electronic materials

abstract: We analyze quantum-geometric bounds on optical weights in topological phases with pairs of bands hosting non-trivial Euler class, a multi-gap invariant characterizing non-Abelian band topology. We show how the bounds constrain the combined optical weights of the Euler bands at different dopings and further restrict the size of the adjacent band gaps. In this process, we also consider the associated interband contributions to DC conductivities in the flat-band limit. We physically validate these results by recasting the bound in terms of transition rates associated with the optical absorption of light, and demonstrate how the Euler connections and curvatures can be determined through the use of momentum and frequency-resolved optical measurements, allowing for a direct measurement of this multi-band invariant. Additionally, we prove that the bound holds beyond the degenerate limit of Euler bands, resulting in nodal topology captured by the patch Euler class. In this context, we deduce optical manifestations of Euler topology within $\vec{k} \cdot \vec{p}$ models, which include AC conductivity, and third-order jerk photoconductivities in doped Euler semimetals. We showcase our findings with numerical validation in lattice-regularized models that benchmark effective theories for real materials and are themselves directly realizable in metamaterials and optical lattices.

categories: cond-mat.mes-hall cond-mat.quant-gas cond-mat.str-el physics.optics quant-ph


title: Towards a covariant framework for post-Newtonian expansions for radiative sources

abstract: We consider the classic problem of a compact fluid source that behaves non-relativistically and that radiates gravitational waves. The problem consists of determining the metric close to the source as well as far away from it. The non-relativistic nature of the source leads to a separation of scales resulting in an overlap region where both the $1/c$ and (multipolar) $G$-expansions are valid. Standard approaches to this problem (the Blanchet--Damour and the DIRE approach) use the harmonic gauge. We define a `post-Newtonian' class of gauges that admit a Newtonian regime in inertial coordinates. In this paper we set up a formalism to solve for the metric for any post-Newtonian gauge choice. Our methods are based on previous work on the covariant theory of non-relativistic gravity (a $1/c$-expansion of general relativity that uses post-Newton-Cartan variables). At the order of interest in the $1/c$ and $G$-expansions we split the variables into two sets: transverse and longitudinal. We show that for the transverse variables the problem can be reduced to inverting Laplacian and d'Alembertian operators on their respective domains subject to appropriate boundary conditions. The latter are regularity in the interior and asymptotic flatness with a Sommerfeld no-incoming radiation condition imposed at past null infinity. The longitudinal variables follow from the gauge choice. The full solution is then obtained by the method of matched asymptotic expansion. We show that our methods reproduce existing results in harmonic gauge to 2.5PN order.


title: GPT-4V(ision) as A Social Media Analysis Engine

abstract: Recent research has offered insights into the extraordinary capabilities of Large Multimodal Models (LMMs) in various general vision and language tasks. There is growing interest in how LMMs perform in more specialized domains. Social media content, inherently multimodal, blends text, images, videos, and sometimes audio. Understanding social multimedia content remains a challenging problem for contemporary machine learning frameworks. In this paper, we explore GPT-4V(ision)'s capabilities for social multimedia analysis. We select five representative tasks, including sentiment analysis, hate speech detection, fake news identification, demographic inference, and political ideology detection, to evaluate GPT-4V. Our investigation begins with a preliminary quantitative analysis for each task using existing benchmark datasets, followed by a careful review of the results and a selection of qualitative samples that illustrate GPT-4V's potential in understanding multimodal social media content. GPT-4V demonstrates remarkable efficacy in these tasks, showcasing strengths such as joint understanding of image-text pairs, contextual and cultural awareness, and extensive commonsense knowledge. Despite the overall impressive capacity of GPT-4V in the social media domain, there remain notable challenges. GPT-4V struggles with tasks involving multilingual social multimedia comprehension and has difficulties in generalizing to the latest trends in social media. Additionally, it exhibits a tendency to generate erroneous information in the context of evolving celebrity and politician knowledge, reflecting the known hallucination problem. The insights gleaned from our findings underscore a promising future for LMMs in enhancing our comprehension of social media content and its users through the analysis of multimodal information.


title: Interpretable Fine-Tuning for Graph Neural Network Surrogate Models

abstract: Data-driven surrogate modeling has surged in capability in recent years with the emergence of graph neural networks (GNNs), which can operate directly on mesh-based representations of data. The goal of this work is to introduce an interpretable fine-tuning strategy for GNNs, with application to unstructured mesh-based fluid dynamics modeling. The end result is an enhanced fine-tuned model that isolates regions in physical space, corresponding to sub-graphs, that are intrinsically linked to the forecasting task while retaining the predictive capability of the baseline. These structures, identified by the fine-tuned GNNs, are adaptively produced in the forward pass and serve as explainable links between the baseline model architecture, the optimization goal, and known problem-specific physics. Additionally, through a regularization procedure, the fine-tuned GNNs can also be used to identify, during inference, graph nodes that correspond to a majority of the anticipated forecasting error, adding a novel interpretable error-tagging capability to baseline models. Demonstrations are performed using unstructured flow field data sourced from flow over a backward-facing step at high Reynolds numbers.


title: Singularities of orthogonal and symplectic determinantal varieties

abstract: Let either $GL(E)\times SO(F)$ or $GL(E)\times Sp(F)$ act naturally on the space of matrices $E\otimes F$. There are only finitely many orbits, and the orbit closures are orthogonal and symplectic generalizations of determinantal varieties, which can be described similarly using rank conditions. In this paper, we study the singularities of these varieties and describe their defining equations. We prove that in the symplectic case, the orbit closures are normal with good filtrations, and in characteristic $0$ have rational singularities. In the orthogonal case we show that most orbit closures will have the same properties, and determine precisely the exceptions to this.


title: Tabdoor: Backdoor Vulnerabilities in Transformer-based Neural Networks for Tabular Data

abstract: Deep Neural Networks (DNNs) have shown great promise in various domains. Alongside these developments, vulnerabilities associated with DNN training, such as backdoor attacks, are a significant concern. These attacks involve the subtle insertion of triggers during model training, allowing for manipulated predictions. More recently, DNNs for tabular data have gained increasing attention due to the rise of transformer models. Our research presents a comprehensive analysis of backdoor attacks on tabular data using DNNs, mainly focusing on transformers. We also propose a novel approach for trigger construction: an in-bounds attack, which provides excellent attack performance while maintaining stealthiness. Through systematic experimentation across benchmark datasets, we uncover that transformer-based DNNs for tabular data are highly susceptible to backdoor attacks, even with minimal feature value alterations. We also verify that our attack can be generalized to other models, like XGBoost and DeepFM. Our results demonstrate up to 100% attack success rate with negligible clean accuracy drop. Furthermore, we evaluate several defenses against these attacks, identifying Spectral Signatures as the most effective. Nevertheless, our findings highlight the need to develop tabular data-specific countermeasures to defend against backdoor attacks.


title: Low regularity well-posedness for the generalized surface quasi-geostrophic front equation

abstract: We consider the well-posedness of the generalized surface quasi-geostrophic (gSQG) front equation. By using the null structure of the equation via a paradifferential normal form analysis, we obtain balanced energy estimates, which allow us to prove the local well-posedness of the non-periodic gSQG front equation at a low level of regularity (in the SQG case, at only one-half derivatives above scaling). In addition, we establish global well-posedness for small and localized rough initial data, as well as modified scattering, by using the testing by wave packet approach of Ifrim-Tataru.


title: The Unsettled Number: Hubble's Tension

abstract: One of main sources of uncertainty in modern cosmology is the present rate of the universe's expansion, H0, called the Hubble constant. Once again, different observational techniques bring about different results, causing new 'Hubble tension'. In the present work, we review the historical roots of the Hubble constant from the beginning of the twentieth century, when modern cosmology originated, to the present. We develop the arguments that gave rise to the importance of measuring the expansion of the Universe and its discovery, and we describe the different pioneering works attempting to measure it. There has been a long dispute on this matter, even in the present epoch, which is marked by high-tech instrumentation and, therefore, in smaller uncertainties in the relevant parameters. It is, again, currently necessary to conduct a careful and critical revision of the different methods before one invokes new physics to solve the so-called Hubble tension.


title: An Extensive Study on Adversarial Attack against Pre-trained Models of Code

abstract: Transformer-based pre-trained models of code (PTMC) have been widely utilized and have achieved state-of-the-art performance in many mission-critical applications. However, they can be vulnerable to adversarial attacks through identifier substitution or coding style transformation, which can significantly degrade accuracy and may further incur security concerns. Although several approaches have been proposed to generate adversarial examples for PTMC, the effectiveness and efficiency of such approaches, especially on different code intelligence tasks, has not been well understood. To bridge this gap, this study systematically analyzes five state-of-the-art adversarial attack approaches from three perspectives: effectiveness, efficiency, and the quality of generated examples. The results show that none of the five approaches balances all these perspectives. Particularly, approaches with a high attack success rate tend to be time-consuming; the adversarial code they generate often lack naturalness, and vice versa. To address this limitation, we explore the impact of perturbing identifiers under different contexts and find that identifier substitution within for and if statements is the most effective. Based on these findings, we propose a new approach that prioritizes different types of statements for various tasks and further utilizes beam search to generate adversarial examples. Evaluation results show that it outperforms the state-of-the-art ALERT in terms of both effectiveness and efficiency while preserving the naturalness of the generated adversarial examples.


title: Fast and Space-Efficient Parallel Algorithms for Influence Maximization

abstract: Influence Maximization (IM) is a crucial problem in data science. The goal is to find a fixed-size set of highly-influential seed vertices on a network to maximize the influence spread along the edges. While IM is NP-hard on commonly-used diffusion models, a greedy algorithm can achieve $(1-1/e)$-approximation, repeatedly selecting the vertex with the highest marginal gain in influence as the seed. Due to theoretical guarantees, rich literature focuses on improving the performance of the greedy algorithm. To estimate the marginal gain, existing work either runs Monte Carlo (MC) simulations of influence spread or pre-stores hundreds of sketches (usually per-vertex information). However, these approaches can be inefficient in time (MC simulation) or space (storing sketches), preventing the ideas from scaling to today's large-scale graphs. This paper significantly improves the scalability of IM using two key techniques. The first is a sketch-compression technique for the independent cascading model on undirected graphs. It allows combining the simulation and sketching approaches to achieve a time-space tradeoff. The second technique includes new data structures for parallel seed selection. Using our new approaches, we implemented PaC-IM: Parallel and Compressed IM. We compare PaC-IM with state-of-the-art parallel IM systems on a 96-core machine with 1.5TB memory. PaC-IM can process large-scale graphs with up to 900M vertices and 74B edges in about 2 hours. On average across all tested graphs, our uncompressed version is 5--18$\times$ faster and about 1.4$\times$ more space-efficient than existing parallel IM systems. Using compression further saves 3.8$\times$ space with only 70% overhead in time on average.


title: On Bounding and Approximating Functions of Multiple Expectations using Quasi-Monte Carlo

abstract: Monte Carlo and Quasi-Monte Carlo methods present a convenient approach for approximating the expected value of a random variable. Algorithms exist to adaptively sample the random variable until a user defined absolute error tolerance is satisfied with high probability. This work describes an extension of such methods which supports adaptive sampling to satisfy general error criteria for functions of a common array of expectations. Although several functions involving multiple expectations are being evaluated, only one random sequence is required, albeit sometimes of larger dimension than the underlying randomness. These enhanced Monte Carlo and Quasi-Monte Carlo algorithms are implemented in the QMCPy Python package with support for economic and parallel function evaluation. We exemplify these capabilities on problems from machine learning and global sensitivity analysis.


title: Using Natural Language Explanations to Improve Robustness of In-context Learning

abstract: Recent studies demonstrated that large language models (LLMs) can excel in many tasks via in-context learning (ICL). However, recent works show that ICL-prompted models tend to produce inaccurate results when presented with adversarial inputs. In this work, we investigate whether augmenting ICL with natural language explanations (NLEs) improves the robustness of LLMs on adversarial datasets covering natural language inference and paraphrasing identification. We prompt LLMs with a small set of human-generated NLEs to produce further NLEs, yielding more accurate results than both a zero-shot-ICL setting and using only human-generated NLEs. Our results on five popular LLMs (GPT3.5-turbo, Llama2, Vicuna, Zephyr, and Mistral) show that our approach yields over 6% improvement over baseline approaches for eight adversarial datasets: HANS, ISCS, NaN, ST, PICD, PISP, ANLI, and PAWS. Furthermore, previous studies have demonstrated that prompt selection strategies significantly enhance ICL on in-distribution test sets. However, our findings reveal that these strategies do not match the efficacy of our approach for robustness evaluations, resulting in an accuracy drop of 8% compared to the proposed approach.


title: Backreaction of axion-SU(2) dynamics during inflation

abstract: We consider the effects of backreaction on axion-SU(2) dynamics during inflation. We use the linear evolution equations for the gauge field modes and compute their backreaction on the background quantities numerically using the Hartree approximation. We show that the spectator chromo-natural inflation attractor is unstable when back-reaction becomes important. Working within the constraints of the linear mode equations, we find a new dynamical attractor solution for the axion field and the vacuum expectation value of the gauge field, where the latter has an opposite sign with respect to the chromo-natural inflation solution. Our findings are of particular interest to the phenomenology of axion-SU(2) inflation, as they demonstrate the instability of the usual trajectory due to large backreaction effects. The viable parameter space of the model becomes significantly altered, provided future non-Abelian lattice simulations confirm the existence of the new dynamical attractor. In addition, the backreaction effects lead to characteristic oscillatory features in the primordial gravitational wave background that are potentially detectable with upcoming gravitational wave detectors.


title: Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learning

abstract: We introduce PACOH-RL, a novel model-based Meta-Reinforcement Learning (Meta-RL) algorithm designed to efficiently adapt control policies to changing dynamics. PACOH-RL meta-learns priors for the dynamics model, allowing swift adaptation to new dynamics with minimal interaction data. Existing Meta-RL methods require abundant meta-learning data, limiting their applicability in settings such as robotics, where data is costly to obtain. To address this, PACOH-RL incorporates regularization and epistemic uncertainty quantification in both the meta-learning and task adaptation stages. When facing new dynamics, we use these uncertainty estimates to effectively guide exploration and data collection. Overall, this enables positive transfer, even when access to data from prior tasks or dynamic settings is severely limited. Our experiment results demonstrate that PACOH-RL outperforms model-based RL and model-based Meta-RL baselines in adapting to new dynamic conditions. Finally, on a real robotic car, we showcase the potential for efficient RL policy adaptation in diverse, data-scarce conditions.


title: Sound Gradual Verification with Symbolic Execution

abstract: Gradual verification, which supports explicitly partial specifications and verifies them with a combination of static and dynamic checks, makes verification more incremental and provides earlier feedback to developers. While an abstract, weakest precondition-based approach to gradual verification was previously proven sound, the approach did not provide sufficient guidance for implementation and optimization of the required run-time checks. More recently, gradual verification was implemented using symbolic execution techniques, but the soundness of the approach (as with related static checkers based on implicit dynamic frames) was an open question. This paper puts practical gradual verification on a sound footing with a formalization of symbolic execution, optimized run-time check generation, and run time execution. We prove our approach is sound; our proof also covers a core subset of the Viper tool, for which we are aware of no previous soundness result. Our formalization enabled us to find a soundness bug in an implemented gradual verification tool and describe the fix necessary to make it sound.


title: Scanning the moduli of smooth hypersurfaces

abstract: We study the locus of smooth hypersurfaces inside the Hilbert scheme of a smooth projective complex variety. In the spirit of scanning, we construct a map to a continuous section space of a projective bundle, and show that it induces an isomorphism in integral homology in a range of degrees growing with the ampleness of the hypersurfaces. When the ambient variety is a curve, this recovers a result of McDuff about configuration spaces. We compute the rational cohomology of the section space and exhibit a phenomenon of homological stability for hypersurfaces with first Chern class going to infinity. For simply connected varieties, the rational cohomology is shown to agree with the stable cohomology of a moduli space of hypersurfaces, with a peculiar tangential structure, as studied by Galatius and Randal-Williams.


title: Fast Normalized Cross-Correlation for Template Matching with Rotations

abstract: Normalized cross-correlation is the reference approach to carry out template matching on images. When it is computed in Fourier space, it can handle efficiently template translations but it cannot do so with template rotations. Including rotations requires sampling the whole space of rotations, repeating the computation of the correlation each time. This article develops an alternative mathematical theory to handle efficiently, at the same time, rotations and translations. Our proposal has a reduced computational complexity because it does not require to repeatedly sample the space of rotations. To do so, we integrate the information relative to all rotated versions of the template into a unique symmetric tensor template -which is computed only once per template-. Afterward, we demonstrate that the correlation between the image to be processed with the independent tensor components of the tensorial template contains enough information to recover template instance positions and rotations. Our proposed method has the potential to speed up conventional template matching computations by a factor of several magnitude orders for the case of 3D images.


title: GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

abstract: We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical user interface (GUI) navigation task. MM-Navigator can interact with a smartphone screen as human users, and determine subsequent actions to fulfill given instructions. Our findings demonstrate that large multimodal models (LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities. We first benchmark MM-Navigator on our collected iOS screen dataset. According to human assessments, the system exhibited a 91\% accuracy rate in generating reasonable action descriptions and a 75\% accuracy rate in executing the correct actions for single-step instructions on iOS. Additionally, we evaluate the model on a subset of an Android screen navigation dataset, where the model outperforms previous GUI navigators in a zero-shot fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for future research into the GUI navigation task. The project page is at https://github.com/zzxslp/MM-Navigator.


title: Learning Control Policies of Hodgkin-Huxley Neuronal Dynamics

abstract: We present a neural network approach for closed-loop deep brain stimulation (DBS). We cast the problem of finding an optimal neurostimulation strategy as a control problem. In this setting, control policies aim to optimize therapeutic outcomes by tailoring the parameters of a DBS system, typically via electrical stimulation, in real time based on the patient's ongoing neuronal activity. We approximate the value function offline using a neural network to enable generating controls (stimuli) in real time via the feedback form. The neuronal activity is characterized by a nonlinear, stiff system of differential equations as dictated by the Hodgkin-Huxley model. Our training process leverages the relationship between Pontryagin's maximum principle and Hamilton-Jacobi-Bellman equations to update the value function estimates simultaneously. Our numerical experiments illustrate the accuracy of our approach for out-of-distribution samples and the robustness to moderate shocks and disturbances in the system.


title: Can Authorship Attribution Models Distinguish Speakers in Speech Transcripts?

abstract: Authorship verification is the task of determining if two distinct writing samples share the same author and is typically concerned with the attribution of written text. In this paper, we explore the attribution of transcribed speech, which poses novel challenges. The main challenge is that many stylistic features, such as punctuation and capitalization, are not informative in this setting. On the other hand, transcribed speech exhibits other patterns, such as filler words and backchannels (e.g., 'um', 'uh-huh'), which may be characteristic of different speakers. We propose a new benchmark for speaker attribution focused on human-transcribed conversational speech transcripts. To limit spurious associations of speakers with topic, we employ both conversation prompts and speakers participating in the same conversation to construct verification trials of varying difficulties. We establish the state of the art on this new benchmark by comparing a suite of neural and non-neural baselines, finding that although written text attribution models achieve surprisingly good performance in certain settings, they perform markedly worse as conversational topic is increasingly controlled. We present analyses of the impact of transcription style on performance as well as the ability of fine-tuning on speech transcripts to improve performance.


title: Exploration via linearly perturbed loss minimisation

abstract: We introduce exploration via linear loss perturbations (EVILL), a randomised exploration method for structured stochastic bandit problems that works by solving for the minimiser of a linearly perturbed regularised negative log-likelihood function. We show that, for the case of generalised linear bandits, EVILL reduces to perturbed history exploration (PHE), a method where exploration is done by training on randomly perturbed rewards. In doing so, we provide a simple and clean explanation of when and why random reward perturbations give rise to good bandit algorithms. We propose data-dependent perturbations not present in previous PHE-type methods that allow EVILL to match the performance of Thompson-sampling-style parameter-perturbation methods, both in theory and in practice. Moreover, we show an example outside generalised linear bandits where PHE leads to inconsistent estimates, and thus linear regret, while EVILL remains performant. Like PHE, EVILL can be implemented in just a few lines of code.


title: Generators of the preprojective CoHA of a quiver

abstract: In this short note, we refine a result of Schiffmann-Vasserot, by showing that the localized preprojective cohomological Hall algebra of any quiver is spherical, i.e. generated by elements of minimal dimension.


title: Higher Chow groups and not necessarily admissible cycles

abstract: We construct some analog of cubical Bloch's higher Chow groups. Instead of considering cycles in $X\times\mathbb A^n$ we consider varieties $Y$ over $X$ together with a distinguished element in the $n$-th exterior power of the multiplicative group of the field of fraction on $Y$. This definition allows us to make sense of a cycle in $X\times\mathbb A^n$ intersecting faces improperly as an element in this complex. We prove that this complex is well-defined and study its basic properties: flat pullback, the localization sequence etc. As an application we prove that the cohomology of this complex in degrees $m-1, m$ and weight $m$ isomorphic to the cohomology of polylogarithmic complex.


title: Feature emergence via margin maximization: case studies in algebraic tasks

abstract: Understanding the internal representations learned by neural networks is a cornerstone challenge in the science of machine learning. While there have been significant recent strides in some cases towards understanding how neural networks implement specific target functions, this paper explores a complementary question -- why do networks arrive at particular computational strategies? Our inquiry focuses on the algebraic learning tasks of modular addition, sparse parities, and finite group operations. Our primary theoretical findings analytically characterize the features learned by stylized neural networks for these algebraic tasks. Notably, our main technique demonstrates how the principle of margin maximization alone can be used to fully specify the features learned by the network. Specifically, we prove that the trained networks utilize Fourier features to perform modular addition and employ features corresponding to irreducible group-theoretic representations to perform compositions in general groups, aligning closely with the empirical observations of Nanda et al. and Chughtai et al. More generally, we hope our techniques can help to foster a deeper understanding of why neural networks adopt specific computational strategies.


title: Optimal Load Shedding for Public Safety Power Shutoffs

abstract: Public utilities are faced with situations where high winds can bring trees and debris into contact with energized power lines and other equipments, which could ignite wildfires. As a result, they need to turn off power during severe weather to help prevent wildfires. This is called Public Safety Power Shutoff (PSPS). We present a method for load reduction using a multi-step genetic algorithm for Public Safety Power Shutoff events. The proposed method optimizes load shedding using partial load shedding based on load importance (critical loads like hospitals, fire stations, etc). The multi-step genetic algorithm optimizes load shedding while minimizing the impact on important loads and preserving grid stability. The effectiveness of the method is demonstrated through network examples. The results show that the proposed method achieves minimal load shedding while maintaining the critical loads at acceptable levels. This approach will help utilities to effectively manage PSPS events and reduce the risk of wildfires caused by the power lines.


title: Epiperimetric inequalities in the obstacle problem for the fractional Laplacian

abstract: Using the epiperimetric inequalities approach, we study the obstacle problem $\min\{(-\Delta)^su,u-\varphi\}=0,$ for the fractional Laplacian $(-\Delta)^s$ with obstacle $\varphi\in C^{k,\gamma}(\mathbb{R}^n)$, $k\ge2$ and $\gamma\in(0,1)$. We prove an epiperimetric inequality for the Weiss' energy $W_{1+s}$ and a logarithmic epiperimetric inequality for the Weiss' energy $W_{2m}$. Moreover, we also prove two epiperimetric inequalities for negative energies $W_{1+s}$ and $W_{2m}$. By these epiperimetric inequalities, we deduce a frequency gap and a characterization of the blow-ups for the frequencies $\lambda=1+s$ and $\lambda=2m$. Finally, we give an alternative proof of the regularity of the points on the free boundary with frequency $1+s$ and we describe the structure of the points on the free boundary with frequency $2m$, with $m\in\mathbb{N}$ and $2m\le k.$


title: Universal signatures of Majorana zero modes in critical Kitaev chains

abstract: Many topological or critical aspects of the Kitaev chain are well known, with several classic results. In contrast, the study of the critical behavior of the strong Majorana zero modes (MZM) has been overlooked. Here we introduce two topological markers which, surprisingly, exhibit non-trivial signatures over the entire (1+1) Ising critical line. We first analytically compute the MZM fidelity ${\cal{F}}_{\rm MZM}$--a measure of the MZM mapping between parity sectors. It takes a universal value along the (1+1) Ising critical line, ${\cal{F}}_{\rm MZM}=\sqrt{8}/\pi$, independent of the energy. We also obtain an exact analytical result for the critical MZM occupation number ${{\cal N}}_{\rm MZM}$ which depends on the Catalan's constant ${\cal G}\approx 0.91596559$, for both the ground-state (${{\cal N}}_{\rm MZM}=1/2-4{\cal{G}}/\pi^2\approx 0.12877$) and the first excited state (${{\cal N}}_{\rm MZM}=1/2+(8-4{\cal{G}})/\pi^2\approx 0.93934$). We further compute finite-size corrections which identically vanish for the special ratio $\Delta/t=\sqrt{2}-1$ between pairing and hopping in the critical Kitaev chain.

categories: cond-mat.str-el cond-mat.stat-mech cond-mat.supr-con quant-ph


title: The local moduli space of the Einstein-Yang-Mills system

abstract: We study the deformation theory of the Einstein-Yang-Mills system on a principal bundle with a compact structure group over a compact manifold. We first construct, as an application of the general slice theorem of Diez and Rudolph, a smooth slice in the tame Fr\'echet category for the coupled action of bundle automorphisms on metrics and connections. Using this result, together with a careful analysis of the linearization of the Einstein-Yang-Mills system, we realize the moduli space of Einstein-Yang-Mills pairs modulo automorphism as an analytic set in a finite-dimensional tame Fr\'echet manifold, extending classical results of Koiso for Einstein metrics and Yang-Mills connections to the Einstein-Yang-Mills system. Furthermore, we introduce the notion of \emph{essential deformation} of an Einstein-Yang-Mills pair, which we characterize in full generality and explore in more detail in the four-dimensional case, proving a decoupling result for trace deformations when the underlying Einstein-Yang-Mills pair is a Ricci-flat metric coupled to an anti-self-dual instanton. In particular, we find a novel obstruction that does not occur in the separate Einstein or Yang-Mills moduli problems. Finally, we prove that every essential deformation of the four-dimensional Einstein-Yang-Mills system based on a Calabi-Yau metric coupled to an instanton is of restricted type. Notable examples of Einstein-Yang-Mills pairs include $\mathrm{G}_2$ or $\mathrm{Spin}(7)$ holonomy metrics coupled to $\mathrm{G}_2$ or $\mathrm{Spin}(7)$ instantons, respectively, or zero-slope polystable holomorphic vector bundles on Calabi-Yau manifolds.


title: Realizability of Free Spaces of Curves

abstract: The free space diagram is a popular tool to compute the well-known Fr\'echet distance. As the Fr\'echet distance is used in many different fields, many variants have been established to cover the specific needs of these applications. Often, the question arises whether a certain pattern in the free space diagram is "realizable", i.e., whether there exists a pair of polygonal chains whose free space diagram corresponds to it. The answer to this question may help in deciding the computational complexity of these distance measures, as well as allowing to design more efficient algorithms for restricted input classes that avoid certain free space patterns. Therefore, we study the inverse problem: Given a potential free space diagram, do there exist curves that generate this diagram? Our problem of interest is closely tied to the classic Distance Geometry problem. We settle the complexity of Distance Geometry in $\mathbb{R}^{> 2}$, showing $\exists\mathbb{R}$-hardness. We use this to show that for curves in $\mathbb{R}^{\ge 2}$, the realizability problem is $\exists\mathbb{R}$-complete, both for continuous and for discrete Fr\'echet distance. We prove that the continuous case in $\mathbb{R}^1$ is only weakly NP-hard, and we provide a pseudo-polynomial time algorithm and show that it is fixed-parameter tractable. Interestingly, for the discrete case in $\mathbb{R}^1$, we show that the problem becomes solvable in polynomial time.


title: To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning

abstract: Existing visual instruction tuning methods typically prompt large language models with textual descriptions to generate instruction-following data. Despite the promising performance achieved, these descriptions are derived from image annotations, which are oftentimes coarse-grained. Furthermore, the instructions might even contradict the visual content without observing the entire visual context. To address this challenge, we introduce a fine-grained visual instruction dataset, LVIS-Instruct4V, which contains 220K visually aligned and context-aware instructions produced by prompting the powerful GPT-4V with images from LVIS. Through experimental validation and case studies, we demonstrate that high-quality visual instructional data could improve the performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a wide spectrum of benchmarks by clear margins. Notably, by simply replacing the LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA on most challenging LMM benchmarks, e.g., LLaVA$^w$ (76.7 vs. 70.7) and MM-Vet (40.2 vs. 35.4). We release our data and model at https://github.com/X2FD/LVIS-INSTRUCT4V.


title: SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

abstract: We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings. First, for stronger vision-language alignment, we unfreeze the large language model (LLM) during pre-training, and introduce a weight mix strategy between LLMs trained by real-world and synthetic data. By directly integrating the weights from two domains, the mixed LLM can efficiently incorporate diverse semantics with favorable robustness. Then, to enable multi-purpose capabilities, we mix a variety of tasks for joint visual instruction tuning, and design task-specific instructions to avoid inter-task conflict. In addition to the basic visual question answering, we include more challenging tasks such as region-level understanding, caption grounding, document layout detection, and human pose estimation, contributing to mutual enhancement over different scenarios. Additionally, we propose to extract comprehensive visual embeddings from various network architectures, pre-training paradigms, and information granularity, providing language models with more robust image representations. Based on our proposed joint mixing, SPHINX exhibits superior multi-modal understanding capabilities on a wide range of applications. On top of this, we further propose an efficient strategy aiming to better capture fine-grained appearances of high-resolution images. With a mixing of different scales and high-resolution sub-images, SPHINX attains exceptional visual parsing and reasoning performance on existing evaluation benchmarks. We hope our work may cast a light on the exploration of joint mixing in future MLLM research. Code is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.


title: On the connection between AGN radiative feedback and massive black hole spin

abstract: We present a novel implementation for active galactic nucleus (AGN) feedback through ultra-fast winds in the code gizmo. Our feedback recipe accounts for the angular dependence of radiative feedback upon black hole spin. We self-consistently evolve in time i) the gas accretion process from resolved scales to an unresolved AGN disc, ii) the evolution of the spin of the massive black hole (MBH), iii) the injection of AGN-driven winds into the resolved scales, and iv) the spin-induced anisotropy of the overall feedback process. We test our implementation by following the propagation of the wind-driven outflow into a homogeneous medium, and we compare the results against simple analytical models. Then, we consider an isolated galaxy setup and there we study the impact of the AGN feedback on the evolution of the MBH and the of the host galaxy. We find that: i) AGN feedback limits the gas inflow that powers the MBH, with a consequent weak impact on the host galaxy characterized by a star formation (SF) suppression of about a factor of two in the nuclear region; ii) the impact of AGN feedback on the host galaxy and on MBH growth is primarily determined by the AGN luminosity, rather than by its angular pattern set by the MBH spin; iii) the imprint of the angular pattern of the AGN radiation emission manifest in a more clear way at high accretion rates. At such high rates the more isotropic angular patterns, proper to higher spin values, sweep away gas in the nuclear region more easily, hence causing a slower MBH mass and spin growths and a higher quenching of the SF. We argue that the influence of spin-dependent anisotropy of AGN feedback on MBH and galaxy evolution is likely to be relevant in those scenarios characterized by high and prolonged MBH accretion episodes and by high AGN wind-galaxy coupling. Such conditions are more frequently met in galaxy mergers and/or high redshift galaxies.


title: Algorithms for Object Detection in Substations

abstract: Inspection of high-voltage power equipment is an effective way to ensure power supply reliability. Object recognition, one of the key technologies in automatic power equipment inspection, attracts attention of many researchers and engineers. Although quite a few existing models have some their own advantages, object relationship between equipment which is very important in this task is scarcely considered. This paper combining object relationship modeling and Transformer Model proposes a Relation Transformer Model. It has four parts -- backbone, encoder, decoder and prediction heads. With this structure, the proposed method shows in experiments a much better performance than other three commonly used models in object recognition in substation, largely promoting the development of automatic power equipment inspection.


title: A Metacognitive Approach to Out-of-Distribution Detection for Segmentation

abstract: Despite outstanding semantic scene segmentation in closed-worlds, deep neural networks segment novel instances poorly, which is required for autonomous agents acting in an open world. To improve out-of-distribution (OOD) detection for segmentation, we introduce a metacognitive approach in the form of a lightweight module that leverages entropy measures, segmentation predictions, and spatial context to characterize the segmentation model's uncertainty and detect pixel-wise OOD data in real-time. Additionally, our approach incorporates a novel method of generating synthetic OOD data in context with in-distribution data, which we use to fine-tune existing segmentation models with maximum entropy training. This further improves the metacognitive module's performance without requiring access to OOD data while enabling compatibility with established pre-trained models. Our resulting approach can reliably detect OOD instances in a scene, as shown by state-of-the-art performance on OOD detection for semantic segmentation benchmarks.


title: Relative intrinsic dimensionality is intrinsic to learning

abstract: High dimensional data can have a surprising property: pairs of data points may be easily separated from each other, or even from arbitrary subsets, with high probability using just simple linear classifiers. However, this is more of a rule of thumb than a reliable property as high dimensionality alone is neither necessary nor sufficient for successful learning. Here, we introduce a new notion of the intrinsic dimension of a data distribution, which precisely captures the separability properties of the data. For this intrinsic dimension, the rule of thumb above becomes a law: high intrinsic dimension guarantees highly separable data. We extend this notion to that of the relative intrinsic dimension of two data distributions, which we show provides both upper and lower bounds on the probability of successfully learning and generalising in a binary classification problem


title: Noise-robust latent vector reconstruction in ptychography using deep generative models

abstract: Computational imaging is increasingly vital for a broad spectrum of applications, ranging from biological to material sciences. This includes applications where the object is known and sufficiently sparse, allowing it to be described with a reduced number of parameters. When no explicit parameterization is available, a deep generative model can be trained to represent an object in a low-dimensional latent space. In this paper, we harness this dimensionality reduction capability of autoencoders to search for the object solution within the latent space rather than the object space. We demonstrate a novel approach to ptychographic image reconstruction by integrating a deep generative model obtained from a pre-trained autoencoder within an Automatic Differentiation Ptychography (ADP) framework. This approach enables the retrieval of objects from highly ill-posed diffraction patterns, offering an effective method for noise-robust latent vector reconstruction in ptychography. Moreover, the mapping into a low-dimensional latent space allows us to visualize the optimization landscape, which provides insight into the convexity and convergence behavior of the inverse problem. With this work, we aim to facilitate new applications for sparse computational imaging such as when low radiation doses or rapid reconstructions are essential.


title: Crown ether decorated silicon photonics for safeguarding against lead poisoning

abstract: Lead (Pb2+) toxification in society is one of the most concerning public health crisis that remains unaddressed. The exposure to Pb2+ poisoning leads to a multitude of enduring health issues, even at the part-per-billion scale (ppb). Yet, public action dwarfs its impact. Pb2+ poisoning is estimated to account for 1 million deaths per year globally, which is in addition to its chronic impact on children. With their ring-shaped cavities, crown ethers are uniquely capable of selectively binding to specific ions. In this work, for the first time, the synergistic integration of highly-scalable silicon photonics, with crown ether amine conjugation via Fischer esterification in an environmentally-friendly fashion is demonstrated. This realises a photonic platform that enables the in-situ, highly-selective and quantitative detection of various ions. The development dispels the existing notion that Fischer esterification is restricted to organic compounds, laying the ground for subsequent amine conjugation for various crown ethers. In this work, the platform is engineered for Pb2+ detection, demonstrating a large dynamic detection range of 1 - 262000 ppb with high selectivity against a wide range of relevant ions. These results indicate the potential for the pervasive implementation of the technology to safeguard against ubiquitous lead poisoning in our society.


title: Evaluating the Potential of Leading Large Language Models in Reasoning Biology Questions

abstract: Recent advances in Large Language Models (LLMs) have presented new opportunities for integrating Artificial General Intelligence (AGI) into biological research and education. This study evaluated the capabilities of leading LLMs, including GPT-4, GPT-3.5, PaLM2, Claude2, and SenseNova, in answering conceptual biology questions. The models were tested on a 108-question multiple-choice exam covering biology topics in molecular biology, biological techniques, metabolic engineering, and synthetic biology. Among the models, GPT-4 achieved the highest average score of 90 and demonstrated the greatest consistency across trials with different prompts. The results indicated GPT-4's proficiency in logical reasoning and its potential to aid biology research through capabilities like data analysis, hypothesis generation, and knowledge integration. However, further development and validation are still required before the promise of LLMs in accelerating biological discovery can be realized.


title: Cross-Dialect Sentence Transformation: A Comparative Analysis of Language Models for Adapting Sentences to British English

abstract: This study explores linguistic distinctions among American, Indian, and Irish English dialects and assesses various Language Models (LLMs) in their ability to generate British English translations from these dialects. Using cosine similarity analysis, the study measures the linguistic proximity between original British English translations and those produced by LLMs for each dialect. The findings reveal that Indian and Irish English translations maintain notably high similarity scores, suggesting strong linguistic alignment with British English. In contrast, American English exhibits slightly lower similarity, reflecting its distinct linguistic traits. Additionally, the choice of LLM significantly impacts translation quality, with Llama-2-70b consistently demonstrating superior performance. The study underscores the importance of selecting the right model for dialect translation, emphasizing the role of linguistic expertise and contextual understanding in achieving accurate translations.


title: Performance Prediction of Data-Driven Knowledge summarization of High Entropy Alloys (HEAs) literature implementing Natural Language Processing algorithms

abstract: The ability to interpret spoken language is connected to natural language processing. It involves teaching the AI how words relate to one another, how they are meant to be used, and in what settings. The goal of natural language processing (NLP) is to get a machine intelligence to process words the same way a human brain does. This enables machine intelligence to interpret, arrange, and comprehend textual data by processing the natural language. The technology can comprehend what is communicated, whether it be through speech or writing because AI pro-cesses language more quickly than humans can. In the present study, five NLP algorithms, namely, Geneism, Sumy, Luhn, Latent Semantic Analysis (LSA), and Kull-back-Liebler (KL) al-gorithm, are implemented for the first time for the knowledge summarization purpose of the High Entropy Alloys (HEAs). The performance prediction of these algorithms is made by using the BLEU score and ROUGE score. The results showed that the Luhn algorithm has the highest accuracy score for the knowledge summarization tasks compared to the other used algorithms.


title: Input Reconstruction Attack against Vertical Federated Large Language Models

abstract: Recently, large language models (LLMs) have drawn extensive attention from academia and the public, due to the advent of the ChatGPT. While LLMs show their astonishing ability in text generation for various tasks, privacy concerns limit their usage in real-life businesses. More specifically, either the user's inputs (the user sends the query to the model-hosting server) or the model (the user downloads the complete model) itself will be revealed during the usage. Vertical federated learning (VFL) is a promising solution to this kind of problem. It protects both the user's input and the knowledge of the model by splitting the model into a bottom part and a top part, which is maintained by the user and the model provider, respectively. However, in this paper, we demonstrate that in LLMs, VFL fails to protect the user input since it is simple and cheap to reconstruct the input from the intermediate embeddings. Experiments show that even with a commercial GPU, the input sentence can be reconstructed in only one second. We also discuss several possible solutions to enhance the privacy of vertical federated LLMs.


title: Event Detection on Twitter

abstract: Detecting events by using social media has been an active research problem. In this work, we investigate and compare the performance of two methods for event detection in Twitter by using Apache Storm as the stream processing infrastructure. The first event detection method is based on identifying uncommonly common words inside tweet blocks, and the second one is based on clustering tweets to detect a cluster as an event. Each of the methods has its own characteristics. Uncommonly common word based method relies on the burst of words and hence is not affected from concurrency problems in distributed environment. On the other hand, clustering based method includes a finer grained analysis, but it is sensitive to the concurrent processing. We investigate the effect of stream processing and concurrency handling support provided by Apace Storm on event detection by these methods.


title: Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?

abstract: We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment. This problem is comprised of arithmetic questions posed in natural language, with an arbitrary adversarial string inserted before the question is complete. Even in the simple setting of 1-digit addition problems, it is easy to find adversarial prompts that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and even to steer models to a particular wrong answer. We additionally provide a simple algorithm for finding successful attacks by querying those same models, which we name "prompt inversion rejection sampling" (PIRS). We finally show that models can be partially hardened against these attacks via reinforcement learning and via agentic constitutional loops. However, we were not able to make a language model fully robust against adversarial arithmetic attacks.


title: NLQxform: A Language Model-based Question to SPARQL Transformer

abstract: In recent years, scholarly data has grown dramatically in terms of both scale and complexity. It becomes increasingly challenging to retrieve information from scholarly knowledge graphs that include large-scale heterogeneous relationships, such as authorship, affiliation, and citation, between various types of entities, e.g., scholars, papers, and organizations. As part of the Scholarly QALD Challenge, this paper presents a question-answering (QA) system called NLQxform, which provides an easy-to-use natural language interface to facilitate accessing scholarly knowledge graphs. NLQxform allows users to express their complex query intentions in natural language questions. A transformer-based language model, i.e., BART, is employed to translate questions into standard SPARQL queries, which can be evaluated to retrieve the required information. According to the public leaderboard of the Scholarly QALD Challenge at ISWC 2023 (Task 1: DBLP-QUAD - Knowledge Graph Question Answering over DBLP), NLQxform achieved an F1 score of 0.85 and ranked first on the QA task, demonstrating the competitiveness of the system.


title: Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sources

abstract: To address the data scarcity issue in Conversational question answering (ConvQA), a dialog inpainting method, which utilizes documents to generate ConvQA datasets, has been proposed. However, the original dialog inpainting model is trained solely on the dialog reconstruction task, resulting in the generation of questions with low contextual relevance due to insufficient learning of question-answer alignment. To overcome this limitation, we propose a novel framework called Dialogizer, which has the capability to automatically generate ConvQA datasets with high contextual relevance from textual sources. The framework incorporates two training tasks: question-answer matching (QAM) and topic-aware dialog generation (TDG). Moreover, re-ranking is conducted during the inference phase based on the contextual relevance of the generated questions. Using our framework, we produce four ConvQA datasets by utilizing documents from multiple domains as the primary source. Through automatic evaluation using diverse metrics, as well as human evaluation, we validate that our proposed framework exhibits the ability to generate datasets of higher quality compared to the baseline dialog inpainting model.


title: Large Language Models can Strategically Deceive their Users when Put Under Pressure

abstract: We demonstrate a situation in which Large Language Models, trained to be helpful, harmless, and honest, can display misaligned behavior and strategically deceive their users about this behavior without being instructed to do so. Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management. When reporting to its manager, the model consistently hides the genuine reasons behind its trading decision. We perform a brief investigation of how this behavior varies under changes to the setting, such as removing model access to a reasoning scratchpad, attempting to prevent the misaligned behavior by changing system instructions, changing the amount of pressure the model is under, varying the perceived risk of getting caught, and making other simple changes to the environment. To our knowledge, this is the first demonstration of Large Language Models trained to be helpful, harmless, and honest, strategically deceiving their users in a realistic situation without direct instructions or training for deception.


title: Identification of Books That are Suitable for Middle School Students Using Artificial Neural Networks

abstract: Reading right books contributes to children's imagination and brain development, enhances their language and emotional comprehension abilities, and strengthens their relationships with others. Building upon the critical role of reading books in individual development, this paper aims to develop an algorithm that determines the suitability of books for middle school students by analyzing their structural and semantic features. Using methods described, an algorithm will be created that can be utilized by institutions and individuals responsible for children's education, such as the Ministry of National Education officials and schools. This algorithm will facilitate the selection of books to be taught at the middle school level. With the algorithm, the book selection process for the middle school curriculum can be expedited, and it will serve as a preliminary reference source for those who evaluate books by reading them. In this paper, the Python programming language was employed, utilizing natural language processing methods. Additionally, an artificial neural network (ANN) was trained using the data which had been preprocessed to construct an original dataset. To train this network, suitable books for middle school students were provided by the MEB, Oxford and Cambridge and with content assessed based on the "R" criterion, and inappropriate books for middle school students in terms of content were included. This trained neural network achieved a 90.06% consistency rate in determining the appropriateness of the test-provided books. Considering the obtained findings, it can be concluded that the developed software has achieved the desired objective.


title: Hallucination-minimized Data-to-answer Framework for Financial Decision-makers

abstract: Large Language Models (LLMs) have been applied to build several automation and personalized question-answering prototypes so far. However, scaling such prototypes to robust products with minimized hallucinations or fake responses still remains an open challenge, especially in niche data-table heavy domains such as financial decision making. In this work, we present a novel Langchain-based framework that transforms data tables into hierarchical textual data chunks to enable a wide variety of actionable question answering. First, the user-queries are classified by intention followed by automated retrieval of the most relevant data chunks to generate customized LLM prompts per query. Next, the custom prompts and their responses undergo multi-metric scoring to assess for hallucinations and response confidence. The proposed system is optimized with user-query intention classification, advanced prompting, data scaling capabilities and it achieves over 90% confidence scores for a variety of user-queries responses ranging from {What, Where, Why, How, predict, trend, anomalies, exceptions} that are crucial for financial decision making applications. The proposed data to answers framework can be extended to other analytical domains such as sales and payroll to ensure optimal hallucination control guardrails.


title: Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification

abstract: A promising approach for improving the performance of vision-language models like CLIP for image classification is to extend the class descriptions (i.e., prompts) with related attributes, e.g., using brown sparrow instead of sparrow. However, current zero-shot methods select a subset of attributes regardless of commonalities between the target classes, potentially providing no useful information that would have helped to distinguish between them. For instance, they may use color instead of bill shape to distinguish between sparrows and wrens, which are both brown. We propose Follow-up Differential Descriptions (FuDD), a zero-shot approach that tailors the class descriptions to each dataset and leads to additional attributes that better differentiate the target classes. FuDD first identifies the ambiguous classes for each image, and then uses a Large Language Model (LLM) to generate new class descriptions that differentiate between them. The new class descriptions resolve the initial ambiguity and help predict the correct label. In our experiments, FuDD consistently outperforms generic description ensembles and naive LLM-generated descriptions on 12 datasets. We show that differential descriptions are an effective tool to resolve class ambiguities, which otherwise significantly degrade the performance. We also show that high quality natural language class descriptions produced by FuDD result in comparable performance to few-shot adaptation methods.


title: How to Bridge the Gap between Modalities: A Comprehensive Survey on Multimodal Large Language Model

abstract: This review paper explores Multimodal Large Language Models (MLLMs), which integrate Large Language Models (LLMs) like GPT-4 to handle multimodal data such as text and vision. MLLMs demonstrate capabilities like generating image narratives and answering image-based questions, bridging the gap towards real-world human-computer interactions and hinting at a potential pathway to artificial general intelligence. However, MLLMs still face challenges in processing the semantic gap in multimodality, which may lead to erroneous generation, posing potential risks to society. Choosing the appropriate modality alignment method is crucial, as improper methods might require more parameters with limited performance improvement. This paper aims to explore modality alignment methods for LLMs and their existing capabilities. Implementing modality alignment allows LLMs to address environmental issues and enhance accessibility. The study surveys existing modal alignment methods in MLLMs into four groups: (1) Multimodal Converters that change data into something LLMs can understand; (2) Multimodal Perceivers to improve how LLMs perceive different types of data; (3) Tools Assistance for changing data into one common format, usually text; and (4) Data-Driven methods that teach LLMs to understand specific types of data in a dataset. This field is still in a phase of exploration and experimentation, and we will organize and update various existing research methods for multimodal information alignment.


title: A Decision Support System for Liver Diseases Prediction: Integrating Batch Processing, Rule-Based Event Detection and SPARQL Query

abstract: Liver diseases pose a significant global health burden, impacting a substantial number of individuals and exerting substantial economic and social consequences. Rising liver problems are considered a fatal disease in many countries, such as Egypt, Molda, etc. The objective of this study is to construct a predictive model for liver illness using Basic Formal Ontology (BFO) and detection rules derived from a decision tree algorithm. Based on these rules, events are detected through batch processing using the Apache Jena framework. Based on the event detected, queries can be directly processed using SPARQL. To make the ontology operational, these Decision Tree (DT) rules are converted into Semantic Web Rule Language (SWRL). Using this SWRL in the ontology for predicting different types of liver disease with the help of the Pellet and Drool inference engines in Protege Tools, a total of 615 records are taken from different liver diseases. After inferring the rules, the result can be generated for the patient according to the DT rules, and other patient-related details along with different precautionary suggestions can be obtained based on these results. Combining query results of batch processing and ontology-generated results can give more accurate suggestions for disease prevention and detection. This work aims to provide a comprehensive approach that is applicable for liver disease prediction, rich knowledge graph representation, and smart querying capabilities. The results show that combining RDF data, SWRL rules, and SPARQL queries for analysing and predicting liver disease can help medical professionals to learn more about liver diseases and make a Decision Support System (DSS) for health care.


title: Graph GOSPA metric: a metric to measure the discrepancy between graphs of different sizes

abstract: This paper proposes a metric to measure the dissimilarity between graphs that may have a different number of nodes. The proposed metric extends the generalised optimal subpattern assignment (GOSPA) metric, which is a metric for sets, to graphs. The proposed graph GOSPA metric includes costs associated with node attribute errors for properly assigned nodes, missed and false nodes and edge mismatches between graphs. The computation of this metric is based on finding the optimal assignments between nodes in the two graphs, with the possibility of leaving some of the nodes unassigned. We also propose a lower bound for the metric, which is also a metric for graphs and is computable in polynomial time using linear programming. The metric is first derived for undirected unweighted graphs and it is then extended to directed and weighted graphs. The properties of the metric are demonstrated via simulated and empirical datasets.


title: Enhancing Actuarial Non-Life Pricing Models via Transformers

abstract: Currently, there is a lot of research in the field of neural networks for non-life insurance pricing. The usual goal is to improve the predictive power via neural networks while building upon the generalized linear model, which is the current industry standard. Our paper contributes to this current journey via novel methods to enhance actuarial non-life models with transformer models for tabular data. We build here upon the foundation laid out by the combined actuarial neural network as well as the localGLMnet and enhance those models via the feature tokenizer transformer. The manuscript demonstrates the performance of the proposed methods on a real-world claim frequency dataset and compares them with several benchmark models such as generalized linear models, feed-forward neural networks, combined actuarial neural networks, LocalGLMnet, and pure feature tokenizer transformer. The paper shows that the new methods can achieve better results than the benchmark models while preserving certain generalized linear model advantages. The paper also discusses the practical implications and challenges of applying transformer models in actuarial settings.


title: Multi-Label Topic Model for Financial Textual Data

abstract: This paper presents a multi-label topic model for financial texts like ad-hoc announcements, 8-K filings, finance related news or annual reports. I train the model on a new financial multi-label database consisting of 3,044 German ad-hoc announcements that are labeled manually using 20 predefined, economically motivated topics. The best model achieves a macro F1 score of more than 85%. Translating the data results in an English version of the model with similar performance. As application of the model, I investigate differences in stock market reactions across topics. I find evidence for strong positive or negative market reactions for some topics, like announcements of new Large Scale Projects or Bankruptcy Filings, while I do not observe significant price effects for some other topics. Furthermore, in contrast to previous studies, the multi-label structure of the model allows to analyze the effects of co-occurring topics on stock market reactions. For many cases, the reaction to a specific topic depends heavily on the co-occurrence with other topics. For example, if allocated capital from a Seasoned Equity Offering (SEO) is used for restructuring a company in the course of a Bankruptcy Proceeding, the market reacts positively on average. However, if that capital is used for covering unexpected, additional costs from the development of new drugs, the SEO implies negative reactions on average.


title: Testing LLMs on Code Generation with Varying Levels of Prompt Specificity

abstract: Large language models (LLMs) have demonstrated unparalleled prowess in mimicking human-like text generation and processing. Among the myriad of applications that benefit from LLMs, automated code generation is increasingly promising. The potential to transform natural language prompts into executable code promises a major shift in software development practices and paves the way for significant reductions in manual coding efforts and the likelihood of human-induced errors. This paper reports the results of a study that evaluates the performance of various LLMs, such as Bard, ChatGPT-3.5, ChatGPT-4, and Claude-2, in generating Python for coding problems. We focus on how levels of prompt specificity impact the accuracy, time efficiency, and space efficiency of the generated code. A benchmark of 104 coding problems, each with four types of prompts with varying degrees of tests and specificity, was employed to examine these aspects comprehensively. Our results indicate significant variations in performance across different LLMs and prompt types, and its key contribution is to reveal the ideal prompting strategy for creating accurate Python functions. This study lays the groundwork for further research in LLM capabilities and suggests practical implications for utilizing LLMs in automated code generation tasks and test-driven development.


title: Polarimetric PatchMatch Multi-View Stereo

abstract: PatchMatch Multi-View Stereo (PatchMatch MVS) is one of the popular MVS approaches, owing to its balanced accuracy and efficiency. In this paper, we propose Polarimetric PatchMatch multi-view Stereo (PolarPMS), which is the first method exploiting polarization cues to PatchMatch MVS. The key of PatchMatch MVS is to generate depth and normal hypotheses, which form local 3D planes and slanted stereo matching windows, and efficiently search for the best hypothesis based on the consistency among multi-view images. In addition to standard photometric consistency, our PolarPMS evaluates polarimetric consistency to assess the validness of a depth and normal hypothesis, motivated by the physical property that the polarimetric information is related to the object's surface normal. Experimental results demonstrate that our PolarPMS can improve the accuracy and the completeness of reconstructed 3D models, especially for texture-less surfaces, compared with state-of-the-art PatchMatch MVS methods.


title: Online Advertisements with LLMs: Opportunities and Challenges

abstract: This paper explores the potential for leveraging Large Language Models (LLM) in the realm of online advertising systems. We delve into essential requirements including privacy, latency, reliability as well as the satisfaction of users and advertisers that such a system must fulfill. We further introduce a general framework for LLM advertisement, consisting of modification, bidding, prediction, and auction modules. Different design considerations for each module are presented. Fundamental questions regarding practicality, efficiency, and implementation challenges of these designs are raised for future research. Finally, we explore the prospect of LLM-based dynamic creative optimization as a means to significantly enhance the appeal of advertisements to users and discuss its additional challenges.


title: Cache Optimization and Performance Modeling of Batched, Small, and Rectangular Matrix Multiplication on Intel, AMD, and Fujitsu Processors

abstract: Factorization and multiplication of dense matrices and tensors are critical, yet extremely expensive pieces of the scientific toolbox. Careful use of low rank approximation can drastically reduce the computation and memory requirements of these operations. In addition to a lower arithmetic complexity, such methods can, by their structure, be designed to efficiently exploit modern hardware architectures. The majority of existing work relies on batched BLAS libraries to handle the computation of many small dense matrices. We show that through careful analysis of the cache utilization, register accumulation using SIMD registers and a redesign of the implementation, one can achieve significantly higher throughput for these types of batched low-rank matrices across a large range of block and batch sizes. We test our algorithm on 3 CPUs using diverse ISAs -- the Fujitsu A64FX using ARM SVE, the Intel Xeon 6148 using AVX-512 and AMD EPYC 7502 using AVX-2, and show that our new batching methodology is able to obtain more than twice the throughput of vendor optimized libraries for all CPU architectures and problem sizes.


title: PECoP: Parameter Efficient Continual Pretraining for Action Quality Assessment

abstract: The limited availability of labelled data in Action Quality Assessment (AQA), has forced previous works to fine-tune their models pretrained on large-scale domain-general datasets. This common approach results in weak generalisation, particularly when there is a significant domain shift. We propose a novel, parameter efficient, continual pretraining framework, PECoP, to reduce such domain shift via an additional pretraining stage. In PECoP, we introduce 3D-Adapters, inserted into the pretrained model, to learn spatiotemporal, in-domain information via self-supervised learning where only the adapter modules' parameters are updated. We demonstrate PECoP's ability to enhance the performance of recent state-of-the-art methods (MUSDL, CoRe, and TSA) applied to AQA, leading to considerable improvements on benchmark datasets, JIGSAWS ($\uparrow6.0\%$), MTL-AQA ($\uparrow0.99\%$), and FineDiving ($\uparrow2.54\%$). We also present a new Parkinson's Disease dataset, PD4T, of real patients performing four various actions, where we surpass ($\uparrow3.56\%$) the state-of-the-art in comparison. Our code, pretrained models, and the PD4T dataset are available at https://github.com/Plrbear/PECoP.