title: Stabilizing perpendicular magnetic anisotropy with strong exchange bias in PtMn/Co by magneto-ionics

abstract: Electric field control of magnetic properties offers a broad and promising toolbox for enabling ultra-low power electronics. A key challenge with high technological relevance is to master the interplay between the magnetic anisotropy of a ferromagnet and the exchange coupling to an adjacent antiferromagnet. Here, we demonstrate that magneto-ionic gating can be used to achieve a very stable out-of-plane (OOP) oriented magnetization with strong exchange bias in samples with as-deposited preferred in-plane (IP) magnetization. We show that the perpendicular interfacial anisotropy can be increased by more than a factor 2 in the stack Ta/Pt/PtMn/Co/HfO2 by applying -2.5 V gate voltage over 3 nm HfO2, causing a reorientation of the magnetization from IP to OOP with a strong OOP exchange bias of more than 50 mT. Comparing two thicknesses of PtMn, we identify a notable trade-off: while thicker PtMn yields a significantly larger exchange bias, it also results in a slower response to ionic liquid gating within the accessible gate voltage window. These results pave the way for post-deposition electrical tailoring of magnetic anisotropy and exchange bias in samples requiring significant exchange bias.


title: Event activity correlations and jet measurements in p+Au collisions at $\sqrt{s_\mathrm{NN}}$ = 200 GeV

abstract: With the STAR experiment at RHIC, we characterize $\sqrt{s_\mathrm{NN}}$ = 200 GeV $p$+Au collisions by event activity (EA) measured within the pseudorapidity range $\eta\in[-5,-3.4]$ in the Au-going direction and report correlations between this EA and hard- and soft-scale particle production at mid-rapidity ($\eta\in[-1,1]$). At the soft scale, charged particle production in low-EA $p$+Au collisions is comparable to that in \pp collisions and increases monotonically with increasing EA. At the hard scale, we report measurements of high transverse momentum ($p_\mathrm{T}$) jets in events of different EAs. In contrast to the soft particle production, high-$p_\mathrm{T}$ particle production and EA are found to be inversely related. To investigate whether this is a signal of jet quenching in high-EA events, we also report ratios of $p_\mathrm{T}$ imbalance and azimuthal separation of dijets in high- and low-EA events. Within our measurement precision, no significant differences are observed, disfavoring the presence of jet quenching in the highest 30% EA $p$+Au collisions at $\sqrt{s_\mathrm{NN}}$ = 200 GeV.


title: Under pressure: learning-based analog gauge reading in the wild

abstract: We propose an interpretable framework for reading analog gauges that is deployable on real world robotic systems. Our framework splits the reading task into distinct steps, such that we can detect potential failures at each step. Our system needs no prior knowledge of the type of gauge or the range of the scale and is able to extract the units used. We show that our gauge reading algorithm is able to extract readings with a relative reading error of less than 2%.


title: NeuroLGP-SM: Scalable Surrogate-Assisted Neuroevolution for Deep Neural Networks

abstract: Evolutionary Algorithms (EAs) play a crucial role in the architectural configuration and training of Artificial Deep Neural Networks (DNNs), a process known as neuroevolution. However, neuroevolution is hindered by its inherent computational expense, requiring multiple generations, a large population, and numerous epochs. The most computationally intensive aspect lies in evaluating the fitness function of a single candidate solution. To address this challenge, we employ Surrogate-assisted EAs (SAEAs). While a few SAEAs approaches have been proposed in neuroevolution, none have been applied to truly large DNNs due to issues like intractable information usage. In this work, drawing inspiration from Genetic Programming semantics, we use phenotypic distance vectors, outputted from DNNs, alongside Kriging Partial Least Squares (KPLS), an approach that is effective in handling these large vectors, making them suitable for search. Our proposed approach, named Neuro-Linear Genetic Programming surrogate model (NeuroLGP-SM), efficiently and accurately estimates DNN fitness without the need for complete evaluations. NeuroLGP-SM demonstrates competitive or superior results compared to 12 other methods, including NeuroLGP without SM, convolutional neural networks, support vector machines, and autoencoders. Additionally, it is worth noting that NeuroLGP-SM is 25% more energy-efficient than its NeuroLGP counterpart. This efficiency advantage adds to the overall appeal of our proposed NeuroLGP-SM in optimising the configuration of large DNNs.


title: Radiation hardness of open Fabry-Perot microcavities

abstract: High-finesse microcavities offer a platform for compact, high-precision sensing by employing high-reflectivity, low-loss mirrors to create effective optical path lengths that are orders of magnitude larger than the device geometry. Here, we investigate the radiation hardness of Fabry-Perot microcavities formed from dielectric mirrors deposited on the tips of optical fibers. The microcavities are irradiated under both conventional (~0.1 Gy/s) and ultrahigh (FLASH, ~20 Gy/s) radiotherapy dose rates. Within our measurement sensitivity of ~40 ppm loss, we observe no degradation in the mirror absorption after irradiation with over 300 Gy accumulated dose. This result highlights the excellent radiation hardness of the dielectric mirrors forming the cavities, enabling new optics-based, real-time, in-vivo, tissue-equivalent radiation dosimeters with ~10 micron spatial resolution (our motivation), as well as other applications in high-radiation environments.


title: Detecting AI-Generated Images via CLIP

abstract: As AI-generated image (AIGI) methods become more powerful and accessible, it has become a critical task to determine if an image is real or AI-generated. Because AIGI lack the signatures of photographs and have their own unique patterns, new models are needed to determine if an image is AI-generated. In this paper, we investigate the ability of the Contrastive Language-Image Pre-training (CLIP) architecture, pre-trained on massive internet-scale data sets, to perform this differentiation. We fine-tune CLIP on real images and AIGI from several generative models, enabling CLIP to determine if an image is AI-generated and, if so, determine what generation method was used to create it. We show that the fine-tuned CLIP architecture is able to differentiate AIGI as well or better than models whose architecture is specifically designed to detect AIGI. Our method will significantly increase access to AIGI-detecting tools and reduce the negative effects of AIGI on society, as our CLIP fine-tuning procedures require no architecture changes from publicly available model repositories and consume significantly less GPU resources than other AIGI detection models.


title: Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes

abstract: Particle filters flexibly represent multiple posterior modes nonparametrically, via a collection of weighted samples, but have classically been applied to tracking problems with known dynamics and observation likelihoods. Such generative models may be inaccurate or unavailable for high-dimensional observations like images. We instead leverage training data to discriminatively learn particle-based representations of uncertainty in latent object states, conditioned on arbitrary observations via deep neural network encoders. While prior discriminative particle filters have used heuristic relaxations of discrete particle resampling, or biased learning by truncating gradients at resampling steps, we achieve unbiased and low-variance gradient estimates by representing posteriors as continuous mixture densities. Our theory and experiments expose dramatic failures of existing reparameterization-based estimators for mixture gradients, an issue we address via an importance-sampling gradient estimator. Unlike standard recurrent neural networks, our mixture density particle filter represents multimodal uncertainty in continuous latent states, improving accuracy and robustness. On a range of challenging tracking and robot localization problems, our approach achieves dramatic improvements in accuracy, while also showing much greater stability across multiple training runs.


title: Central Configurations with Dihedral Symmetry

abstract: As an application of the representation theory for the dihedral groups, we study the symmetric central configurations in the n-body problem where $n$ equal masses are placed at the vertices of a regular $n$-gon. Since the Hessian matrices at these configurations are typically very large, particularly when $n$ is large, computations of their eigenvalues present a challenging problem. However, by decomposing the action of the dihedral groups into irreducible representations, we show that the Hessians can be simplified to a block-diagonal matrix with small blocks, of the sizes at most 2*2. This is due to the fact that the action of a dihedral group can be represented as a block-diagonal matrix with small irreducible blocks. In the end, the eigenvalues can be explicitly obtained by simply computing eigenvalues of these small block matrices.


title: Handling Reward Misspecification in the Presence of Expectation Mismatch

abstract: Detecting and handling misspecified objectives, such as reward functions, has been widely recognized as one of the central challenges within the domain of Artificial Intelligence (AI) safety research. However, even with the recognition of the importance of this problem, we are unaware of any works that attempt to provide a clear definition for what constitutes (a) misspecified objectives and (b) successfully resolving such misspecifications. In this work, we use the theory of mind, i.e., the human user's beliefs about the AI agent, as a basis to develop a formal explanatory framework called Expectation Alignment (EAL) to understand the objective misspecification and its causes. Our \EAL\ framework not only acts as an explanatory framework for existing works but also provides us with concrete insights into the limitations of existing methods to handle reward misspecification and novel solution strategies. We use these insights to propose a new interactive algorithm that uses the specified reward to infer potential user expectations about the system behavior. We show how one can efficiently implement this algorithm by mapping the inference problem into linear programs. We evaluate our method on a set of standard Markov Decision Process (MDP) benchmarks.


title: Convergence of coordinate ascent variational inference for log-concave measures via optimal transport

abstract: Mean field variational inference (VI) is the problem of finding the closest product (factorized) measure, in the sense of relative entropy, to a given high-dimensional probability measure $\rho$. The well known Coordinate Ascent Variational Inference (CAVI) algorithm aims to approximate this product measure by iteratively optimizing over one coordinate (factor) at a time, which can be done explicitly. Despite its popularity, the convergence of CAVI remains poorly understood. In this paper, we prove the convergence of CAVI for log-concave densities $\rho$. If additionally $\log \rho$ has Lipschitz gradient, we find a linear rate of convergence, and if also $\rho$ is strongly log-concave, we find an exponential rate. Our analysis starts from the observation that mean field VI, while notoriously non-convex in the usual sense, is in fact displacement convex in the sense of optimal transport when $\rho$ is log-concave. This allows us to adapt techniques from the optimization literature on coordinate descent algorithms in Euclidean space.

categories: stat.ML cs.LG math.OC math.PR math.ST stat.TH


title: JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models

abstract: The proliferation of large language models (LLMs) has underscored concerns regarding their security vulnerabilities, notably against jailbreak attacks, where adversaries design jailbreak prompts to circumvent safety mechanisms for potential misuse. Addressing these concerns necessitates a comprehensive analysis of jailbreak prompts to evaluate LLMs' defensive capabilities and identify potential weaknesses. However, the complexity of evaluating jailbreak performance and understanding prompt characteristics makes this analysis laborious. We collaborate with domain experts to characterize problems and propose an LLM-assisted framework to streamline the analysis process. It provides automatic jailbreak assessment to facilitate performance evaluation and support analysis of components and keywords in prompts. Based on the framework, we design JailbreakLens, a visual analysis system that enables users to explore the jailbreak performance against the target model, conduct multi-level analysis of prompt characteristics, and refine prompt instances to verify findings. Through a case study, technical evaluations, and expert interviews, we demonstrate our system's effectiveness in helping users evaluate model security and identify model weaknesses.


title: Beam Test Performance Studies of CMS Phase-2 Outer Tracker Module Prototypes

abstract: A new tracking detector will be installed as part of the Phase-2 upgrade of the CMS detector for the high-luminosity LHC era. This tracking detector includes the Inner Tracker, equipped with silicon pixel sensor modules, and the Outer Tracker, consisting of modules with two parallel stacked silicon sensors. The Outer Tracker front-end ASICs will be able to correlate hits from charged particles in these two sensors to perform on-module discrimination of transverse momenta $p_\mathrm{T}$. The $p_\mathrm{T}$ information is generated at a frequency of 40 MHz and will be used in the Level-1 trigger decision of CMS. Prototypes of the so-called 2S modules were tested at the Test Beam Facility at DESY Hamburg between 2019 and 2020. These modules use the final front-end ASIC, the CMS Binary Chip (CBC), and for the first time the Concentrator Integrated Circuit (CIC), optical readout and on-module power conversion. In total, seven modules were tested, one of which was assembled with sensors irradiated with protons. An important aspect was to show that it is possible to read out modules synchronously. A cluster hit efficiency of about 99.75% was achieved for all modules. The CBC $p_\mathrm{T}$ discrimination mechanism has been verified to work together with the CIC and optical readout. The measured module performance meets the requirements for operation in the upgraded CMS tracking detector.


title: Microscale Hydrogen, Carbon, and Nitrogen Isotopic Diversity of Organic Matter in Asteroid Ryugu

abstract: We report the H, C, and N isotopic compositions of microscale (0.2 to 2$\mu$m) organic matter in samples of asteroid Ryugu and the Orgueil CI carbonaceous chondrite. Three regolith particles of asteroid Ryugu, returned by the Hayabusa2 spacecraft, and several fragments of Orgueil were analyzed by NanoSIMS isotopic imaging. The isotopic distributions of the Ryugu samples from two different collection spots are closely similar to each other and to the Orgueil samples, strengthening the proposed Ryugu-CI chondrite connection. Most individual sub-$\mu$m organic grains have isotopic compositions within error of bulk values, but 2-8% of them are outliers exhibiting large isotopic enrichments or depletions in D, $^{15}$N, and/or $^{13}$C. The H, C and N isotopic compositions of the outliers are not correlated with each other: while some C-rich grains are both D- and $^{15}$N-enriched, many are enriched or depleted in one or the other system. This most likely points to a diversity in isotopic fractionation pathways and thus diversity in the local formation environments for the individual outlier grains. The observation of a relatively small population of isotopic outlier grains can be explained either by escape from nebular and/or parent body homogenization of carbonaceous precursor material or addition of later isotopic outlier grains. The strong chemical similarity of isotopically typical and isotopically outlying grains, as reflected by synchrotron x-ray absorption spectra, suggests a genetic connection and thus favors the former, homogenization scenario. However, the fact that even the least altered meteorites show the same pattern of a small population of outliers on top of a larger population of homogenized grains indicates that some or most of the homogenization occurred prior to accretion of the macromolecular organic grains into asteroidal parent bodies.


title: The Elephant in the Room: Rethinking the Usage of Pre-trained Language Model in Sequential Recommendation

abstract: Sequential recommendation (SR) has seen significant advancements with the help of Pre-trained Language Models (PLMs). Some PLM-based SR models directly use PLM to encode user historical behavior's text sequences to learn user representations, while there is seldom an in-depth exploration of the capability and suitability of PLM in behavior sequence modeling. In this work, we first conduct extensive model analyses between PLMs and PLM-based SR models, discovering great underutilization and parameter redundancy of PLMs in behavior sequence modeling. Inspired by this, we explore different lightweight usages of PLMs in SR, aiming to maximally stimulate the ability of PLMs for SR while satisfying the efficiency and usability demands of practical systems. We discover that adopting behavior-tuned PLMs for item initializations of conventional ID-based SR models is the most economical framework of PLM-based SR, which would not bring in any additional inference cost but could achieve a dramatic performance boost compared with the original version. Extensive experiments on five datasets show that our simple and universal framework leads to significant improvement compared to classical SR and SOTA PLM-based SR models without additional inference costs.


title: Diffusion-Based Joint Temperature and Precipitation Emulation of Earth System Models

abstract: Earth system models (ESMs) are the principal tools used in climate science to generate future climate projections under various atmospheric emissions scenarios on a global or regional scale. Generative deep learning approaches are suitable for emulating these tools due to their computational efficiency and ability, once trained, to generate realizations in a fraction of the time required by ESMs. We extend previous work that used a generative probabilistic diffusion model to emulate ESMs by targeting the joint emulation of multiple variables, temperature and precipitation, by a single diffusion model. Joint generation of multiple variables is critical to generate realistic samples of phenomena resulting from the interplay of multiple variables. The diffusion model emulator takes in the monthly mean-maps of temperature and precipitation and produces the daily values of each of these variables that exhibit statistical properties similar to those generated by ESMs. Our results show the outputs from our extended model closely resemble those from ESMs on various climate metrics including dry spells and hot streaks, and that the joint distribution of temperature and precipitation in our sample closely matches those of ESMs.


title: Transient Classifiers for Fink: Benchmarks for LSST

abstract: The upcoming Legacy Survey of Space and Time (LSST) at the Vera Rubin Observatory is expected to detect a few million transients per night, which will generate a live alert stream during the entire 10 years of the survey. This will be distributed via community brokers whose task is to select subsets of the stream and direct them to scientific communities. Given the volume and complexity of data, machine learning (ML) algorithms will be paramount for this task. We present the infrastructure tests and classification methods developed within the {\sc Fink} broker in preparation for LSST. This work aims to provide detailed information regarding the underlying assumptions, and methods, behind each classifier, enabling users to make informed follow-up decisions from {\sc Fink} photometric classifications. Using simulated data from the Extended LSST Astronomical Time-series Classification Challenge (ELAsTiCC), we showcase the performance of binary and multi-class ML classifiers available in {\sc Fink}. These include tree-based classifiers coupled with tailored feature extraction strategies, as well as deep learning algorithms. We introduce the CBPF Alert Transient Search (CATS), a deep learning architecture specifically designed for this task. Results show that {\sc Fink} classifiers are able to handle the extra complexity which is expected from LSST data. CATS achieved $97\%$ accuracy on a multi-class classification while our best performing binary classifier achieve $99\%$ when classifying the Periodic class. ELAsTiCC was an important milestone in preparing {\sc Fink} infrastructure to deal with LSST-like data. Our results demonstrate that {\sc Fink} classifiers are well prepared for the arrival of the new stream; this experience also highlights that transitioning from current infrastructures to Rubin will require significant adaptation of currently available tools.


title: Semantic Approach to Quantifying the Consistency of Diffusion Model Image Generation

abstract: In this study, we identify the need for an interpretable, quantitative score of the repeatability, or consistency, of image generation in diffusion models. We propose a semantic approach, using a pairwise mean CLIP (Contrastive Language-Image Pretraining) score as our semantic consistency score. We applied this metric to compare two state-of-the-art open-source image generation diffusion models, Stable Diffusion XL and PixArt-{\alpha}, and we found statistically significant differences between the semantic consistency scores for the models. Agreement between the Semantic Consistency Score selected model and aggregated human annotations was 94%. We also explored the consistency of SDXL and a LoRA-fine-tuned version of SDXL and found that the fine-tuned model had significantly higher semantic consistency in generated images. The Semantic Consistency Score proposed here offers a measure of image generation alignment, facilitating the evaluation of model architectures for specific tasks and aiding in informed decision-making regarding model selection.


title: Radioastrometry at different frequencies

abstract: Very long baseline interferometry (VLBI) technique allows us to determine positions of thousands of radio sources using the absolute astrometry approach. I have investigated the impact of a selection of observing frequencies in a range from 2 to 43 GHz in single-band, dual-band, and quad-band observing modes on astrometric results. I processed seven datasets in a range of 72 thousands to 6.9 million observations, estimated source positions, and compared them. I found that source positions derived from dual-band, quad-band, and 23.6 GHz single-band data agree at a level below 0.2 mas. Comparison of independent datasets allowed me to assess the error level of individual catalogues: 0.05-0.07 mas per position component. Further comparison showed that individual catalogues have systematic errors at the same level. Positions from 23.6 GHz single-band data show systematic errors related to the residual ionosphere contribution. Analysis of source positions differences revealed systematic errors along the jet direction at a level of 0.09 mas. Network related systematic errors affect all the data regardless of frequency. Comparison of position estimates allowed me to derive the stochastic error model that closes the error budget. Based on collected evidence, I made a conclusion that development of frequency-dependent reference frames of the entire sky is not warranted. In most cases dual-band, quad-band, and single-band data at frequency 22 GHz and higher can be used interchangeably, which allows us to exploit the strength of a specific frequency setup for given objects. Mixing observations at different frequencies causes errors not exceeding 0.07 mas.


title: Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length

abstract: The quadratic complexity and weak length extrapolation of Transformers limits their ability to scale to long sequences, and while sub-quadratic solutions like linear attention and state space models exist, they empirically underperform Transformers in pretraining efficiency and downstream task accuracy. We introduce Megalodon, a neural architecture for efficient sequence modeling with unlimited context length. Megalodon inherits the architecture of Mega (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability and stability, including complex exponential moving average (CEMA), timestep normalization layer, normalized attention mechanism and pre-norm with two-hop residual configuration. In a controlled head-to-head comparison with Llama2, Megalodon achieves better efficiency than Transformer in the scale of 7 billion parameters and 2 trillion training tokens. Megalodon reaches a training loss of 1.70, landing mid-way between Llama2-7B (1.75) and 13B (1.67). Code: https://github.com/XuezheMax/megalodon


title: Energy stability of magnetohydrodynamic flow in channels and ducts

abstract: We study the energy stability of pressure-driven laminar magnetohydrodynamic flow in a rectangular duct with transverse homogeneous magnetic field and electrically insulating walls. For sufficiently strong fields, the laminar velocity distribution has a uniform core and convex Hartmann and Shercliff boundary layers on the walls perpendicular and parallel to the magnetic field. The problem is discretized by a double expansion in Chebyshev polynomials in the cross-stream coordinates. The linear eigenvalue problem for the critical Reynolds number depends on the streamwise wavenumber, Hartmann number and the aspect ratio. We consider the limits of small and large aspect ratios in order to compare with stability models based on one-dimensional base flows. For large aspect ratios we find good numerical agreement with results based on the quasi-two-dimensional approximation. The lift-up mechanism dominates in the limit of zero streamwise wavenumber and provides a linear dependence between the critical Reynolds and Hartmann number in the duct. The duct results for small aspect ratio converge to Orr's original energy stability result for spanwise uniform perturbations imposed on the plane Poiseuille base flow. We also examine different possible symmetries of eigenmodes as well as the purely hydrodynamic case in the duct geometry.


title: Random walks on simplicial complexes

abstract: The notion of Laplacian of a graph can be generalized to simplicial complexes and hypergraphs, and contains information on the topology of these structures. Even for a graph, the consideration of associated simplicial complexes is interesting to understand its shape. Whereas the Laplacian of a graph has a simple probabilistic interpretation as the generator of a continuous time Markov chain on the graph, things are not so direct when considering simplicial complexes. We define here new Markov chains on simplicial complexes. For a given order~$k$, the state space is the set of $k$-cycles that are chains of $k$-simplexes with null boundary. This new framework is a natural generalization of the canonical Markov chains on graphs. We show that the generator of our Markov chain is the upper Laplacian defined in the context of algebraic topology for discrete structure. We establish several key properties of this new process: in particular, when the number of vertices is finite, the Markov chain is positive recurrent. This result is not trivial, since the cycles can loop over themselves an unbounded number of times. We study the diffusive limits when the simplicial complexes under scrutiny are a sequence of ever refining triangulations of the flat torus. Using the analogy between singular and Hodge homologies, we express this limit as valued in the set of currents. The proof of tightness and the identification of the limiting martingale problem make use of the flat norm and carefully controls of the error terms in the convergence of the generator. Uniqueness of the solution to the martingale problem is left open. An application to hole detection is carried.


title: Hydroxide Transport and Mechanical Properties of Polyolefin-Based Anion Exchange Membranes from Atomistic Molecular Dynamics Simulations

abstract: Anion exchange membranes are used in alkaline fuel cells and offer a promising alternative to the more expensive proton exchange membrane fuel cells. However, hydroxide ion conductivity in anion exchange membranes is low, and the quest for membranes with superior ion conductivity, mechanical robustness, and chemical stability is ongoing. In this study, we use classical molecular dynamics simulations to study hydroxide ion transport and mechanical properties of eight different hydrated polyolefin-based membranes, to provide a molecular-level understanding of the structure-function relationships in these systems. We examine the microstructure of the membranes and find that polymers with narrow cavity size distribution have tighter packing of water molecules around hydroxide ions. We estimate the self-diffusion coefficient of water and hydroxide ions and find that water molecules have a higher diffusion than hydroxide ions across all systems. The trends in hydroxide diffusion align well with experimental conductivity measurements. Water facilitates hydroxide diffusion, and this is clearly observed when the hydration level is varied for the same polymer chemistry. In systems with narrow cavities and tightly bound hydroxide ions, hydroxide diffusion is the lowest, underscoring the fact that water channels facilitate hydroxide transport. Finally, we apply uniaxial deformation to calculate the mechanical properties of these systems and find that polymers with higher hydration levels show poor mechanical properties. Atomistic molecular dynamics models can accurately capture the trade-off between hydroxide transport and mechanical performance in anion exchange membranes and allow us to screen new candidates more efficiently.


title: Real-time guidewire tracking and segmentation in intraoperative x-ray

abstract: During endovascular interventions, physicians have to perform accurate and immediate operations based on the available real-time information, such as the shape and position of guidewires observed on the fluoroscopic images, haptic information and the patients' physiological signals. For this purpose, real-time and accurate guidewire segmentation and tracking can enhance the visualization of guidewires and provide visual feedback for physicians during the intervention as well as for robot-assisted interventions. Nevertheless, this task often comes with the challenge of elongated deformable structures that present themselves with low contrast in the noisy fluoroscopic image sequences. To address these issues, a two-stage deep learning framework for real-time guidewire segmentation and tracking is proposed. In the first stage, a Yolov5s detector is trained, using the original X-ray images as well as synthetic ones, which is employed to output the bounding boxes of possible target guidewires. More importantly, a refinement module based on spatiotemporal constraints is incorporated to robustly localize the guidewire and remove false detections. In the second stage, a novel and efficient network is proposed to segment the guidewire in each detected bounding box. The network contains two major modules, namely a hessian-based enhancement embedding module and a dual self-attention module. Quantitative and qualitative evaluations on clinical intra-operative images demonstrate that the proposed approach significantly outperforms our baselines as well as the current state of the art and, in comparison, shows higher robustness to low quality images.


title: CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation

abstract: Large Language Models (LLMs) have proved effective and efficient in generating code, leading to their utilization within the hardware design process. Prior works evaluating LLMs' abilities for register transfer level code generation solely focus on functional correctness. However, the creativity associated with these LLMs, or the ability to generate novel and unique solutions, is a metric not as well understood, in part due to the challenge of quantifying this quality. To address this research gap, we present CreativeEval, a framework for evaluating the creativity of LLMs within the context of generating hardware designs. We quantify four creative sub-components, fluency, flexibility, originality, and elaboration, through various prompting and post-processing techniques. We then evaluate multiple popular LLMs (including GPT models, CodeLlama, and VeriGen) upon this creativity metric, with results indicating GPT-3.5 as the most creative model in generating hardware designs.


title: Plasma Dynamics and Nonthermal Particle Acceleration in 3D Nonrelativistic Magnetic Reconnection

abstract: Understanding plasma dynamics and nonthermal particle acceleration in 3D magnetic reconnection has been a long-standing challenge. In this paper, we explore these problems by performing large-scale fully kinetic simulations of multi-xline plasmoid reconnection with various parameters in both the weak and strong guide field regimes. In each regime, we have identified its unique 3D dynamics that leads to field-line chaos and efficient acceleration, and we have achieved nonthermal acceleration of both electrons and protons into power-law spectra. The spectral indices agree well with a simple Fermi acceleration theory that includes guide field dependence. In the low-guide-field regime, the flux-rope kink instability governs the 3D dynamics for efficient acceleration. The weak dependence of the spectra on the ion-to-electron mass ratio and $\beta$ ($\ll1$) implies that the particles are sufficiently magnetized for Fermi acceleration in our simulations. While both electrons and protons are injected at reconnection exhausts, protons are primarily injected by perpendicular electric fields through Fermi reflections and electrons are injected by a combination of perpendicular and parallel electric fields. The magnetic power spectra agree with in-situ magnetotail observations, and the spectral index may reflect a reconnection-driven size distribution of plasmoids instead of Goldreich-Sridhar vortex cascade. As the guide field becomes stronger, the oblique flux ropes of large sizes capture the main 3D dynamics for efficient acceleration. Intriguingly, the oblique flux ropes can also run into flux-rope kink instability to drive extra 3D dynamics. This work has broad implications for 3D reconnection dynamics and particle acceleration in heliophysics and astrophysics.


title: Enhancing IoT Malware Detection through Adaptive Model Parallelism and Resource Optimization

abstract: The widespread integration of IoT devices has greatly improved connectivity and computational capabilities, facilitating seamless communication across networks. Despite their global deployment, IoT devices are frequently targeted for security breaches due to inherent vulnerabilities. Among these threats, malware poses a significant risk to IoT devices. The lack of built-in security features and limited resources present challenges for implementing effective malware detection techniques on IoT devices. Moreover, existing methods assume access to all device resources for malware detection, which is often not feasible for IoT devices deployed in critical real-world scenarios. To overcome this challenge, this study introduces a novel approach to malware detection tailored for IoT devices, leveraging resource and workload awareness inspired by model parallelism. Initially, the device assesses available resources for malware detection using a lightweight regression model. Based on resource availability, ongoing workload, and communication costs, the malware detection task is dynamically allocated either on-device or offloaded to neighboring IoT nodes with sufficient resources. To uphold data integrity and user privacy, instead of transferring the entire malware detection task, the classifier is divided and distributed across multiple nodes, then integrated at the parent node for detection. Experimental results demonstrate that this proposed technique achieves a significant speedup of 9.8 x compared to on-device inference, while maintaining a high malware detection accuracy of 96.7%.


title: Leveraging viscous Hamilton-Jacobi PDEs for uncertainty quantification in scientific machine learning

abstract: Uncertainty quantification (UQ) in scientific machine learning (SciML) combines the powerful predictive power of SciML with methods for quantifying the reliability of the learned models. However, two major challenges remain: limited interpretability and expensive training procedures. We provide a new interpretation for UQ problems by establishing a new theoretical connection between some Bayesian inference problems arising in SciML and viscous Hamilton-Jacobi partial differential equations (HJ PDEs). Namely, we show that the posterior mean and covariance can be recovered from the spatial gradient and Hessian of the solution to a viscous HJ PDE. As a first exploration of this connection, we specialize to Bayesian inference problems with linear models, Gaussian likelihoods, and Gaussian priors. In this case, the associated viscous HJ PDEs can be solved using Riccati ODEs, and we develop a new Riccati-based methodology that provides computational advantages when continuously updating the model predictions. Specifically, our Riccati-based approach can efficiently add or remove data points to the training set invariant to the order of the data and continuously tune hyperparameters. Moreover, neither update requires retraining on or access to previously incorporated data. We provide several examples from SciML involving noisy data and \textit{epistemic uncertainty} to illustrate the potential advantages of our approach. In particular, this approach's amenability to data streaming applications demonstrates its potential for real-time inferences, which, in turn, allows for applications in which the predicted uncertainty is used to dynamically alter the learning process.


title: Stokes problem with slip boundary conditions using stabilized finite elements combined with Nitsche

abstract: We discuss how slip conditions for the Stokes equation can be handled using Nitsche method, for a stabilized finite element discretization. Emphasis is made on the interplay between stabilization and Nitsche terms. Well-posedness of the discrete problem and optimal convergence rates, in natural norm for the velocity and the pressure, are established, and illustrated with various numerical experiments. The proposed method fits naturally in the context of a finite element implementation while being accurate, and allows an increased flexibility in the choice of the finite element pairs.


title: Reducing the Barriers to Entry for Foundation Model Training

abstract: The world has recently witnessed an unprecedented acceleration in demands for Machine Learning and Artificial Intelligence applications. This spike in demand has imposed tremendous strain on the underlying technology stack in supply chain, GPU-accelerated hardware, software, datacenter power density, and energy consumption. If left on the current technological trajectory, future demands show insurmountable spending trends, further limiting market players, stifling innovation, and widening the technology gap. To address these challenges, we propose a fundamental change in the AI training infrastructure throughout the technology ecosystem. The changes require advancements in supercomputing and novel AI training approaches, from high-end software to low-level hardware, microprocessor, and chip design, while advancing the energy efficiency required by a sustainable infrastructure. This paper presents the analytical framework that quantitatively highlights the challenges and points to the opportunities to reduce the barriers to entry for training large language models.


title: A Typology of Decision-Making Tasks for Visualization

abstract: Despite decision-making being a vital goal of data visualization, little work has been done to differentiate the decision-making tasks within our field. While visualization task taxonomies and typologies exist, they are often too granular for describing complex decision goals and decision-making processes, thus limiting their potential use in designing decision-support tools. In this paper, we contribute a typology of decision-making tasks that were iteratively refined from a list of design goals distilled from a literature review. Our typology is concise and consists of only three tasks: choose, activate, and create. Originally proposed by the scientific community, we extend and provide definitions for these tasks that are suitable for the visualization community. Our proposed typology offers two benefits. First, it facilitates the composition of decisions using these three tasks, allowing for flexible and clear descriptions across varying complexities and domains. Second, diagrams created using this typology encourage productive discourse between visualization designers and domain experts by abstracting the intricacies of data, thereby promoting clarity and rigorous analysis of decision-making processes. We motivate the use of our typology through four case studies and demonstrate the benefits of our approach through semi-structured interviews conducted with experienced members of the visualization community, comprising academic and industry experts, who have contributed to developing or publishing decision support systems for domain experts. Our interviewees composed diagrams using our typology to delineate the decision-making processes that drive their decision-support tools, demonstrating its descriptive capacity and effectiveness.


title: Interactive Sonification for Health and Energy using ChucK and Unity

abstract: Sonification can provide valuable insights about data but most existing approaches are not designed to be controlled by the user in an interactive fashion. Interactions enable the designer of the sonification to more rapidly experiment with sound design and allow the sonification to be modified in real-time by interacting with various control parameters. In this paper, we describe two case studies of interactive sonification that utilize publicly available datasets that have been described recently in the International Conference on Auditory Display (ICAD). They are from the health and energy domains: electroencephalogram (EEG) alpha wave data and air pollutant data consisting of nitrogen dioxide, sulfur dioxide, carbon monoxide, and ozone. We show how these sonfications can be recreated to support interaction utilizing a general interactive sonification framework built using ChucK, Unity, and Chunity. In addition to supporting typical sonification methods that are common in existing sonification toolkits, our framework introduces novel methods such as supporting discrete events, interleaved playback of multiple data streams for comparison, and using frequency modulation (FM) synthesis in terms of one data attribute modulating another. We also describe how these new functionalities can be used to improve the sonification experience of the two datasets we have investigated.


title: E3: Ensemble of Expert Embedders for Adapting Synthetic Image Detectors to New Generators Using Limited Data

abstract: As generative AI progresses rapidly, new synthetic image generators continue to emerge at a swift pace. Traditional detection methods face two main challenges in adapting to these generators: the forensic traces of synthetic images from new techniques can vastly differ from those learned during training, and access to data for these new generators is often limited. To address these issues, we introduce the Ensemble of Expert Embedders (E3), a novel continual learning framework for updating synthetic image detectors. E3 enables the accurate detection of images from newly emerged generators using minimal training data. Our approach does this by first employing transfer learning to develop a suite of expert embedders, each specializing in the forensic traces of a specific generator. Then, all embeddings are jointly analyzed by an Expert Knowledge Fusion Network to produce accurate and reliable detection decisions. Our experiments demonstrate that E3 outperforms existing continual learning methods, including those developed specifically for synthetic image detection.


title: Star exponentials from propagators and path integrals

abstract: In this paper we address the relation between the star exponentials emerging within the Deformation Quantization formalism and Feynman's path integrals associated with propagators in quantum dynamics. In order to obtain such a relation, we start by visualizing the quantum propagator as an integral transform of the star exponential by means of the symbol corresponding to the time evolution operator and, thus, we introduce Feynman's path integral representation of the propagator as a sum over all the classical histories. The star exponential thus constructed has the advantage that it does not depend on the convergence of formal series, as commonly understood within the context of Deformation Quantization. We include some basic examples to illustrate our findings, recovering standard results reported in the literature. Further, for an arbitrary finite dimensional system, we use the star exponential introduced here in order to find a particular representation of the star product which resembles the one encountered in the context of the quantum field theory for a Poisson sigma model.


title: Evaluating the Quality of Answers in Political Q&A Sessions with Large Language Models

abstract: This paper presents a new approach to evaluating the quality of answers in political question-and-answer sessions. We propose to measure an answer's quality based on the degree to which it allows us to infer the initial question accurately. This conception of answer quality inherently reflects their relevance to initial questions. Drawing parallels with semantic search, we argue that this measurement approach can be operationalized by fine-tuning a large language model on the observed corpus of questions and answers without additional labeled data. We showcase our measurement approach within the context of the Question Period in the Canadian House of Commons. Our approach yields valuable insights into the correlates of the quality of answers in the Question Period. We find that answer quality varies significantly based on the party affiliation of the members of Parliament asking the questions and uncover a meaningful correlation between answer quality and the topics of the questions.


title: Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance

abstract: This paper revisits recent code similarity evaluation metrics, particularly focusing on the application of Abstract Syntax Tree (AST) editing distance in diverse programming languages. In particular, we explore the usefulness of these metrics and compare them to traditional sequence similarity metrics. Our experiments showcase the effectiveness of AST editing distance in capturing intricate code structures, revealing a high correlation with established metrics. Furthermore, we explore the strengths and weaknesses of AST editing distance and prompt-based GPT similarity scores in comparison to BLEU score, execution match, and Jaccard Similarity. We propose, optimize, and publish an adaptable metric that demonstrates effectiveness across all tested languages, representing an enhanced version of Tree Similarity of Edit Distance (TSED).


title: Empowering Malware Detection Efficiency within Processing-in-Memory Architecture

abstract: The widespread integration of embedded systems across various industries has facilitated seamless connectivity among devices and bolstered computational capabilities. Despite their extensive applications, embedded systems encounter significant security threats, with one of the most critical vulnerabilities being malicious software, commonly known as malware. In recent times, malware detection techniques leveraging Machine Learning have gained popularity. Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) have proven particularly efficient in image processing tasks. However, one major drawback of neural network architectures is their substantial computational resource requirements. Continuous training of malware detection models with updated malware and benign samples demands immense computational resources, presenting a challenge for real-world applications. In response to these concerns, we propose a Processing-in-Memory (PIM)-based architecture to mitigate memory access latency, thereby reducing the resources consumed during model updates. To further enhance throughput and minimize energy consumption, we incorporate precision scaling techniques tailored for CNN models. Our proposed PIM architecture exhibits a 1.09x higher throughput compared to existing Lookup Table (LUT)-based PIM architectures. Additionally, precision scaling combined with PIM enhances energy efficiency by 1.5x compared to full-precision operations, without sacrificing performance. This innovative approach offers a promising solution to the resource-intensive nature of malware detection model updates, paving the way for more efficient and sustainable cybersecurity practices.


title: The Illusion of State in State-Space Models

abstract: State-space models (SSMs) have emerged as a potential alternative architecture for building large language models (LLMs) compared to the previously ubiquitous transformer architecture. One theoretical weakness of transformers is that they cannot express certain kinds of sequential computation and state tracking (Merrill & Sabharwal, 2023), which SSMs are explicitly designed to address via their close architectural similarity to recurrent neural networks (RNNs). But do SSMs truly have an advantage (over transformers) in expressive power for state tracking? Surprisingly, the answer is no. Our analysis reveals that the expressive power of SSMs is limited very similarly to transformers: SSMs cannot express computation outside the complexity class $\mathsf{TC}^0$. In particular, this means they cannot solve simple state-tracking problems like permutation composition. It follows that SSMs are provably unable to accurately track chess moves with certain notation, evaluate code, or track entities in a long narrative. To supplement our formal analysis, we report experiments showing that Mamba-style SSMs indeed struggle with state tracking. Thus, despite its recurrent formulation, the "state" in an SSM is an illusion: SSMs have similar expressiveness limitations to non-recurrent models like transformers, which may fundamentally limit their ability to solve real-world state-tracking problems.


title: Single-image driven 3d viewpoint training data augmentation for effective wine label recognition

abstract: Confronting the critical challenge of insufficient training data in the field of complex image recognition, this paper introduces a novel 3D viewpoint augmentation technique specifically tailored for wine label recognition. This method enhances deep learning model performance by generating visually realistic training samples from a single real-world wine label image, overcoming the challenges posed by the intricate combinations of text and logos. Classical Generative Adversarial Network (GAN) methods fall short in synthesizing such intricate content combination. Our proposed solution leverages time-tested computer vision and image processing strategies to expand our training dataset, thereby broadening the range of training samples for deep learning applications. This innovative approach to data augmentation circumvents the constraints of limited training resources. Using the augmented training images through batch-all triplet metric learning on a Vision Transformer (ViT) architecture, we can get the most discriminative embedding features for every wine label, enabling us to perform one-shot recognition of existing wine labels in the training classes or future newly collected wine labels unavailable in the training. Experimental results show a significant increase in recognition accuracy over conventional 2D data augmentation techniques.


title: Constrained C-Test Generation via Mixed-Integer Programming

abstract: This work proposes a novel method to generate C-Tests; a deviated form of cloze tests (a gap filling exercise) where only the last part of a word is turned into a gap. In contrast to previous works that only consider varying the gap size or gap placement to achieve locally optimal solutions, we propose a mixed-integer programming (MIP) approach. This allows us to consider gap size and placement simultaneously, achieving globally optimal solutions, and to directly integrate state-of-the-art models for gap difficulty prediction into the optimization problem. A user study with 40 participants across four C-Test generation strategies (including GPT-4) shows that our approach (MIP) significantly outperforms two of the baseline strategies (based on gap placement and GPT-4); and performs on-par with the third (based on gap size). Our analysis shows that GPT-4 still struggles to fulfill explicit constraints during generation and that MIP produces C-Tests that correlate best with the perceived difficulty. We publish our code, model, and collected data consisting of 32 English C-Tests with 20 gaps each (totaling 3,200 individual gap responses) under an open source license.


title: The characteristic sequence of the integers that are the sum of two squares is not morphic

abstract: Let $(s_2(n))_{n\in \mathbb{N}}$ be a $0,1$-sequence such that, for any natural number $n$, $s_2(n) = 1$ if and only if $n$ is a sum of two squares. In a recent article, Tahay proved that the sequence $(s_2(n))_{n\in \mathbb{N}}$ is not $k$-automatic for any integer $k$, and asked if this sequence can be morphic. In this note, we give a negative answer to this question.


title: Solving high-dimensional Kolmogorov backward equations with functional hierarchical tensor operators

abstract: Solving high-dimensional partial differential equations necessitates methods free of exponential scaling in the dimension of the problem. This work introduces a tensor network approach for the Kolmogorov backward equation via approximating directly the Markov operator. We show that the high-dimensional Markov operator can be obtained under a functional hierarchical tensor (FHT) ansatz with a hierarchical sketching algorithm. When the terminal condition admits an FHT ansatz, the proposed operator outputs an FHT ansatz for the PDE solution through an efficient functional tensor network contraction procedure. In addition, the proposed operator-based approach also provides an efficient way to solve the Kolmogorov forward equation when the initial distribution is in an FHT ansatz. We apply the proposed approach successfully to two challenging time-dependent Ginzburg-Landau models with hundreds of variables.


title: Classification of Convex Ancient Solutions to Free Boundary Curve Shortening Flow in Convex Domains

abstract: We classify convex ancient curve shortening flows with free boundary on general bounded convex domains.


title: Inverse Kinematics for Neuro-Robotic Grasping with Humanoid Embodied Agents

abstract: This paper introduces a novel zero-shot motion planning method that allows users to quickly design smooth robot motions in Cartesian space. A B\'ezier curve-based Cartesian plan is transformed into a joint space trajectory by our neuro-inspired inverse kinematics (IK) method CycleIK, for which we enable platform independence by scaling it to arbitrary robot designs. The motion planner is evaluated on the physical hardware of the two humanoid robots NICO and NICOL in a human-in-the-loop grasping scenario. Our method is deployed with an embodied agent that is a large language model (LLM) at its core. We generalize the embodied agent, that was introduced for NICOL, to also be embodied by NICO. The agent can execute a discrete set of physical actions and allows the user to verbally instruct various different robots. We contribute a grasping primitive to its action space that allows for precise manipulation of household objects. The new CycleIK method is compared to popular numerical IK solvers and state-of-the-art neural IK methods in simulation and is shown to be competitive with or outperform all evaluated methods when the algorithm runtime is very short. The grasping primitive is evaluated on both NICOL and NICO robots with a reported grasp success of 72% to 82% for each robot, respectively.


title: Strongly Tail-Optimal Scheduling in the Light-Tailed M/G/1

abstract: We study the problem of scheduling jobs in a queueing system, specifically an M/G/1 with light-tailed job sizes, to asymptotically optimize the response time tail. This means scheduling to make $\mathbf{P}[T > t]$, the chance a job's response time exceeds $t$, decay as quickly as possible in the $t \to \infty$ limit. For some time, the best known policy was First-Come First-Served (FCFS), which has an asymptotically exponential tail: $\mathbf{P}[T > t] \sim C e^{-\gamma t}$. FCFS achieves the optimal *decay rate* $\gamma$, but its *tail constant* $C$ is suboptimal. Only recently have policies that improve upon FCFS's tail constant been discovered. But it is unknown what the optimal tail constant is, let alone what policy might achieve it. In this paper, we derive a closed-form expression for the optimal tail constant $C$, and we introduce *$\gamma$-Boost*, a new policy that achieves this optimal tail constant. Roughly speaking, $\gamma$-Boost operates similarly to FCFS, but it pretends that small jobs arrive earlier than their true arrival times. This significantly reduces the response time of small jobs without unduly delaying large jobs, improving upon FCFS's tail constant by up to 50% with only moderate job size variability, with even larger improvements for higher variability. While these results are for systems with full job size information, we also introduce and analyze a version of $\gamma$-Boost that works in settings with partial job size information, showing it too achieves significant gains over FCFS. Finally, we show via simulation that $\gamma$-Boost has excellent practical performance.


title: "Don't forget to put the milk back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situations

abstract: Home robots intend to make their users lives easier. Our work assists in this goal by enabling robots to inform their users of dangerous or unsanitary anomalies in their home. Some examples of these anomalies include the user leaving their milk out, forgetting to turn off the stove, or leaving poison accessible to children. To move towards enabling home robots with these abilities, we have created a new dataset, which we call SafetyDetect. The SafetyDetect dataset consists of 1000 anomalous home scenes, each of which contains unsafe or unsanitary situations for an agent to detect. Our approach utilizes large language models (LLMs) alongside both a graph representation of the scene and the relationships between the objects in the scene. Our key insight is that this connected scene graph and the object relationships it encodes enables the LLM to better reason about the scene -- especially as it relates to detecting dangerous or unsanitary situations. Our most promising approach utilizes GPT-4 and pursues a categorization technique where object relations from the scene graph are classified as normal, dangerous, unsanitary, or dangerous for children. This method is able to correctly identify over 90% of anomalous scenarios in the SafetyDetect Dataset. Additionally, we conduct real world experiments on a ClearPath TurtleBot where we generate a scene graph from visuals of the real world scene, and run our approach with no modification. This setup resulted in little performance loss. The SafetyDetect Dataset and code will be released to the public upon this papers publication.


title: Hindsight PRIORs for Reward Learning from Human Preferences

abstract: Preference based Reinforcement Learning (PbRL) removes the need to hand specify a reward function by learning a reward from preference feedback over policy behaviors. Current approaches to PbRL do not address the credit assignment problem inherent in determining which parts of a behavior most contributed to a preference, which result in data intensive approaches and subpar reward functions. We address such limitations by introducing a credit assignment strategy (Hindsight PRIOR) that uses a world model to approximate state importance within a trajectory and then guides rewards to be proportional to state importance through an auxiliary predicted return redistribution objective. Incorporating state importance into reward learning improves the speed of policy learning, overall policy performance, and reward recovery on both locomotion and manipulation tasks. For example, Hindsight PRIOR recovers on average significantly (p<0.05) more reward on MetaWorld (20%) and DMC (15%). The performance gains and our ablations demonstrate the benefits even a simple credit assignment strategy can have on reward learning and that state importance in forward dynamics prediction is a strong proxy for a state's contribution to a preference decision. Code repository can be found at https://github.com/apple/ml-rlhf-hindsight-prior.


title: Measuring the Predictability of Recommender Systems using Structural Complexity Metrics

abstract: Recommender systems (RS) are central to the filtering and curation of online content. These algorithms predict user ratings for unseen items based on past preferences. Despite their importance, the innate predictability of RS has received limited attention. This study introduces data-driven metrics to measure the predictability of RS based on the structural complexity of the user-item rating matrix. A low predictability score indicates complex and unpredictable user-item interactions, while a high predictability score reveals less complex patterns with predictive potential. We propose two strategies that use singular value decomposition (SVD) and matrix factorization (MF) to measure structural complexity. By perturbing the data and evaluating the prediction of the perturbed version, we explore the structural consistency indicated by the SVD singular vectors. The assumption is that a random perturbation of highly structured data does not change its structure. Empirical results show a high correlation between our metrics and the accuracy of the best-performing prediction algorithms on real data sets.


title: Strongly Gauduchon Hyperbolicity and Two Other Types of Hyperbolicity

abstract: This paper proposes sG-hyperbolicity as a new tool for studying hyperbolicity on complex manifolds. It demonstrates that this notion leads to a wider class of divisorially hyperbolic manifolds compared to balanced hyperbolicity. We also introduce weakly p-K\"ahler hyperbolic structures and pluriclosed star split hyperbolic metrics as possible new avenues for exploration.


title: Structured Model Pruning for Efficient Inference in Computational Pathology

abstract: Recent years have seen significant efforts to adopt Artificial Intelligence (AI) in healthcare for various use cases, from computer-aided diagnosis to ICU triage. However, the size of AI models has been rapidly growing due to scaling laws and the success of foundational models, which poses an increasing challenge to leverage advanced models in practical applications. It is thus imperative to develop efficient models, especially for deploying AI solutions under resource-constrains or with time sensitivity. One potential solution is to perform model compression, a set of techniques that remove less important model components or reduce parameter precision, to reduce model computation demand. In this work, we demonstrate that model pruning, as a model compression technique, can effectively reduce inference cost for computational and digital pathology based analysis with a negligible loss of analysis performance. To this end, we develop a methodology for pruning the widely used U-Net-style architectures in biomedical imaging, with which we evaluate multiple pruning heuristics on nuclei instance segmentation and classification, and empirically demonstrate that pruning can compress models by at least 70% with a negligible drop in performance.


title: Measurements of Titan's Stratospheric Winds during the 2009 Equinox with the eSMA

abstract: Saturn's moon Titan possesses stratospheric zonal winds that places it among a sparse class of planetary bodies known to have superrotation in their atmospheres. Few measurements have been made of these speeds in the upper stratosphere, leaving their seasonal variations still not well understood. We examined observations made with the extended Submillimeter Array (eSMA) in 2009 March (L\textsubscript{s}=355\textdegree{}) and 2010 February (L\textsubscript{s}=5\textdegree{}), shortly before and after Titan's northern spring equinox. Cassini observations and atmospheric models find equinoctial periods to be especially dynamic. Zonal wind calculations, derived from the Doppler frequency shift of CH\textsubscript{3}CN near 349.4 GHz, yielded speeds of 128 $\pm$ 27 m s\textsuperscript{-1} in 2009 and 209 $\pm$ 48 m s\textsuperscript{-1} in 2010. We estimated the measured emission to originate from vertical altitudes of $336^{+112}_{-88}$ kilometers, equivalent to pressures of $3.8^{+19.2}_{-3.4}$ Pa, commensurate with Titan's upper stratosphere/lower mesosphere. This suggests a possible increase in zonal speeds during this period. The results are then compared to those from previous Cassini-inferred and direct-interferometric observations of winds, as well as general circulation model simulations, to form a more complete picture of the seasonal cycle of stratospheric zonal winds.


title: Breaking Boundaries: A Chronology with Future Directions of Women in Exercise Physiology Research, Centred on Pregnancy

abstract: Historically, females were excluded from clinical research due to their reproductive roles, hindering medical understanding and healthcare quality. Despite guidelines promoting equal participation, females are underrepresented in exercise science, perpetuating misconceptions about female physiology. Even less attention has been given to exercise in the pregnant population. Research on pregnancy and exercise has evolved considerably from the initial bedrest prescriptions but concerns about exercise risks during pregnancy persisted for many decades. Recent guidelines endorse moderate-intensity physical activity during pregnancy, supported by considerable evidence of its safety and benefits. Mental health during pregnancy, often overlooked, is gaining traction, with exercise showing promise in reducing depression and anxiety. While pregnancy guidelines recommend moderate-intensity physical activity, there remains limited understanding of optimal frequency, intensity, type and time (duration) for extremes like elite athletes or those with complications. Female participation in elite sport and physically demanding jobs is rising, but research on their specific needs is lacking. Traditional practices like bed rest for high-risk pregnancies are being questioned, as evidence suggests it may not improve outcomes. Historical neglect of gestational parents in research perpetuated stereotypes of female frailty, but recent years have seen a shift towards recognizing the benefits of an active pregnancy. Closing knowledge gaps and inclusivity in research are crucial for ensuring guidelines reflect the diverse needs of gestational parents. Therefore, the purpose of this review is to summarize the evolution of exercise physiology and pregnancy research along with future directions for this novel field.


title: Modeling Melt Pool Geometry in Metal Additive Manufacturing Using Goldak's Semi-Ellipsoidal Heat Source: A Data-driven Computational Approach

abstract: This analytical solution, based on Goldak's Semi-Ellipsoidal Heat Source model, captures the dynamic temperature evolution from a semi-ellipsoidal power density moving heat source within a semi-infinite body. It tackles the convection-diffusion heat transfer equation by integrating an instantaneous point heat source across the volume of the ellipsoidal shape. The model's precision is validated by the excellent match between the predicted transient temperatures and empirical data from bead-on-plate specimens, enhancing its capability to accurately predict in-process temperature profiles in laser-based metal additive manufacturing (AM) operations. Developed in Python, the model offers customized calculations from setup to boundary conditions, adapting to variations in material properties under intense heat gradients. It considers the temperature dependency of thermal material properties and AM-process parameters, accounting for significant temperature gradients and changes in heat transfer mechanisms. The model also includes phase changes of melting or solidification with adjusted heat capacity to accurately reflect these transformations. Additionally, it considers the effects of variable laser power, scanning speed, and timing across each scanning pattern segment, acknowledging the thermal interactions between successive layers and their impact on heat transfer. This comprehensive analytical model is ready for applications including thermal stress analysis, microstructure modeling, and simulation of AM processes, predicting residual stresses and distortions.


title: Transposing cartesian and other structure in double categories

abstract: The cartesian structure possessed by relations, spans, profunctors, and other such morphisms is elegantly expressed by universal properties in double categories. Though cartesian double categories were inspired in part by the older program of cartesian bicategories, the precise relationship between the double-categorical and bicategorical approaches has so far remained mysterious, except in special cases. We provide a formal connection by showing that every double category with iso-strong finite products, and in particular every cartesian equipment, has an underlying cartesian bicategory. To do so, we develop broadly applicable techniques for transposing natural transformations and adjunctions between double categories, extending a line of previous work rooted in the concepts of companions and conjoints.


title: BERT-LSH: Reducing Absolute Compute For Attention

abstract: This study introduces a novel BERT-LSH model that incorporates Locality Sensitive Hashing (LSH) to approximate the attention mechanism in the BERT architecture. We examine the computational efficiency and performance of this model compared to a standard baseline BERT model. Our findings reveal that BERT-LSH significantly reduces computational demand for the self-attention layer while unexpectedly outperforming the baseline model in pretraining and fine-tuning tasks. These results suggest that the LSH-based attention mechanism not only offers computational advantages but also may enhance the model's ability to generalize from its training data. For more information, visit our GitHub repository: https://github.com/leo4life2/algoml-final


title: Vehicle-to-Vehicle Charging: Model, Complexity, and Heuristics

abstract: The rapid adoption of Electric Vehicles (EVs) poses challenges for electricity grids to accommodate or mitigate peak demand. Vehicle-to-Vehicle Charging (V2VC) has been recently adopted by popular EVs, posing new opportunities and challenges to the management and operation of EVs. We present a novel V2VC model that allows decision-makers to take V2VC into account when optimizing their EV operations. We show that optimizing V2VC is NP-Complete and find that even small problem instances are computationally challenging. We propose R-V2VC, a heuristic that takes advantage of the resulting totally unimodular constraint matrix to efficiently solve problems of realistic sizes. Our results demonstrate that R-V2VC presents a linear growth in the solution time as the problem size increases, while achieving solutions of optimal or near-optimal quality. R-V2VC can be used for real-world operations and to study what-if scenarios when evaluating the costs and benefits of V2VC.


title: Predicting Traffic Congestion at Urban Intersections Using Data-Driven Modeling

abstract: Traffic congestion at intersections is a significant issue in urban areas, leading to increased commute times, safety hazards, and operational inefficiencies. This study aims to develop a predictive model for congestion at intersections in major U.S. cities, utilizing a dataset of trip-logging metrics from commercial vehicles across 4,800 intersections. The dataset encompasses 27 features, including intersection coordinates, street names, time of day, and traffic metrics (Kashyap et al., 2019). Additional features, such as rainfall/snowfall percentage, distance from downtown and outskirts, and road types, were incorporated to enhance the model's predictive power. The methodology involves data exploration, feature transformation, and handling missing values through low-rank models and label encoding. The proposed model has the potential to assist city planners and governments in anticipating traffic hot spots, optimizing operations, and identifying infrastructure challenges.


title: Multiply-Robust Causal Change Attribution

abstract: Comparing two samples of data, we observe a change in the distribution of an outcome variable. In the presence of multiple explanatory variables, how much of the change can be explained by each possible cause? We develop a new estimation strategy that, given a causal model, combines regression and re-weighting methods to quantify the contribution of each causal mechanism. Our proposed methodology is multiply robust, meaning that it still recovers the target parameter under partial misspecification. We prove that our estimator is consistent and asymptotically normal. Moreover, it can be incorporated into existing frameworks for causal attribution, such as Shapley values, which will inherit the consistency and large-sample distribution properties. Our method demonstrates excellent performance in Monte Carlo simulations, and we show its usefulness in an empirical application. Our method is implemented as part of the Python library DoWhy (arXiv:2011.04216, arXiv:2206.06821).


title: On Nash resolution of (singular) Lie algebroids

abstract: Any Lie algebroid $A$ admits a Nash-type blow-up $\mathrm{Nash}(A)$ that sits in a nice short exact sequence of Lie algebroids $0\rightarrow K\rightarrow \mathrm{Nash}(A)\rightarrow \mathcal{D}\rightarrow 0$ with $K$ a Lie algebra bundle and $\mathcal{D}$ a Lie algebroid whose anchor map is injective on an open dense subset. The base variety is a blowup determined by the singular foliation of $A$. We provide concrete examples. Moreover, we extend the construction following Mohsen's to singular subalgebroids in the sense of Androulidakis-Zambon.


title: Mal'tsev products of varieties, I

abstract: We investigate the Mal'tsev product $\mathcal{V} \circ \mathcal{W}$ of two varieties $\mathcal{V}$ and $\mathcal{W}$ of the same similarity type. Such a product is usually a quasivariety but not necessarily a variety. We give an equational base for the variety generated by $\mathcal{V} \circ \mathcal{W}$ in terms of identities satisfied in $\mathcal{V}$ and $\mathcal{W}$. Then the main result provides a new sufficient condition for $\mathcal{V} \circ \mathcal{W}$ to be a variety: If $\mathcal{W}$ is an idempotent variety and there are terms $f(x,y)$ and $g(x,y)$ such that $\mathcal{W}$ satisfies the identity $f(x,y) = g(x,y)$ and $\mathcal{V}$ satisfies the identities $f(x,y) = x$ and $g(x,y) = y$, then $\mathcal{V} \circ \mathcal{W}$ is a variety. We also provide a number of examples and applications of this result.


title: Practical Safe Extremum Seeking with Assignable Rate of Attractivity to the Safe Set

abstract: We present Assignably Safe Extremum Seeking (ASfES), an algorithm designed to minimize a measured objective function while maintaining a measured metric of safety (a control barrier function or CBF) be positive in a practical sense. We ensure that for trajectories with safe initial conditions, the violation of safety can be made arbitrarily small with appropriately chosen design constants. We also guarantee an assignable ``attractivity'' rate: from unsafe initial conditions, the trajectories approach the safe set, in the sense of the measured CBF, at a rate no slower than a user-assigned rate. Similarly, from safe initial conditions, the trajectories approach the unsafe set, in the sense of the CBF, no faster than the assigned attractivity rate. The feature of assignable attractivity is not present in the semiglobal version of safe extremum seeking, where the semiglobality of convergence is achieved by slowing the adaptation. We also demonstrate local convergence of the parameter to a neighborhood of the minimum of the objective function constrained to the safe set. The ASfES algorithm and analysis are multivariable, but we also extend the algorithm to a Newton-Based ASfES scheme (NB-ASfES) which we show is only useful in the scalar case. The proven properties of the designs are illustrated through simulation examples.


title: Mal'tsev products of varieties, II

abstract: The Mal'tsev product of two varieties of the same similarity type is not in general a variety, because it can fail to be closed under homomorphic images. In the previous paper we provided a new sufficient condition for such a product to be a variety. In this paper we extend that result by weakening the assumptions regarding the two varieties. We also explore the various special cases of our new result and provide a number of examples of its application.


title: Multi-fingered Robotic Hand Grasping in Cluttered Environments through Hand-object Contact Semantic Mapping

abstract: The integration of optimization method and generative models has significantly advanced dexterous manipulation techniques for five-fingered hand grasping. Yet, the application of these techniques in cluttered environments is a relatively unexplored area. To address this research gap, we have developed a novel method for generating five-fingered hand grasp samples in cluttered settings. This method emphasizes simulated grasp quality and the nuanced interaction between the hand and surrounding objects. A key aspect of our approach is our data generation method, capable of estimating contact spatial and semantic representations and affordance grasps based on object affordance information. Furthermore, our Contact Semantic Conditional Variational Autoencoder (CoSe-CVAE) network is adept at creating comprehensive contact maps from point clouds, incorporating both spatial and semantic data. We introduce a unique grasp detection technique that efficiently formulates mechanical hand grasp poses from these maps. Additionally, our evaluation model is designed to assess grasp quality and collision probability, significantly improving the practicality of five-fingered hand grasping in complex scenarios. Our data generation method outperforms previous datasets in grasp diversity, scene diversity, modality diversity. Our grasp generation method has demonstrated remarkable success, outperforming established baselines with 81.0% average success rate in real-world single-object grasping and 75.3% success rate in multi-object grasping. The dataset and supplementary materials can be found at https://sites.google.com/view/ffh-clutteredgrasping, and we will release the code upon publication.


title: Enumerating 6D supergravities with $T\leq 1$

abstract: The space of 6D supergravities with minimal supersymmetry is greatly constrained by anomaly cancellation. Nevertheless, a large number of models satisfy all low-energy consistency conditions and in this work we make progress towards exhaustively enumerating all anomaly-free models with at most one tensor multiplet. Generalizing previous techniques, we describe a general algorithm using multi-hypergraphs and simplicial complexes to systematically enumerate anomaly-free models with gauge groups of any number of simple factors and with hypermultiplets falling into any representations. Using these new ideas, we obtain a \emph{complete} list of anomaly-free models for $T\leq 1$, the only simplifying assumption being that the gauge group contains no $\operatorname{U}(1)$, $\operatorname{SU}(2)$, $\operatorname{SU}(3)$ or $\operatorname{Sp}(2)$ factors. We also study how many/which models in this ensemble satisfy several UV and swampland bounds which have been proposed and previously utilized to great effect, finding that none are ruled out for $T=0$ and $\approx\!50\%$ are inconsistent with quantum gravity for $T=1$.


title: Experimental Design for Active Transductive Inference in Large Language Models

abstract: One emergent ability of large language models (LLMs) is that query-specific examples can be included in the prompt at inference time. In this work, we use active learning for adaptive prompt design and call it Active In-context Prompt Design (AIPD). We design the LLM prompt by adaptively choosing few-shot examples from a training set to optimize performance on a test set. The training examples are initially unlabeled and we obtain the label of the most informative ones, which maximally reduces uncertainty in the LLM prediction. We propose two algorithms, GO and SAL, which differ in how the few-shot examples are chosen. We analyze these algorithms in linear models: first GO and then use its equivalence with SAL. We experiment with many different tasks in small, medium-sized, and large language models; and show that GO and SAL outperform other methods for choosing few-shot examples in the LLM prompt at inference time.


title: LazyDP: Co-Designing Algorithm-Software for Scalable Training of Differentially Private Recommendation Models

abstract: Differential privacy (DP) is widely being employed in the industry as a practical standard for privacy protection. While private training of computer vision or natural language processing applications has been studied extensively, the computational challenges of training of recommender systems (RecSys) with DP have not been explored. In this work, we first present our detailed characterization of private RecSys training using DP-SGD, root-causing its several performance bottlenecks. Specifically, we identify DP-SGD's noise sampling and noisy gradient update stage to suffer from a severe compute and memory bandwidth limitation, respectively, causing significant performance overhead in training private RecSys. Based on these findings, we propose LazyDP, an algorithm-software co-design that addresses the compute and memory challenges of training RecSys with DP-SGD. Compared to a state-of-the-art DP-SGD training system, we demonstrate that LazyDP provides an average 119x training throughput improvement while also ensuring mathematically equivalent, differentially private RecSys models to be trained.


title: Establishing Criteria for the Transition from Kinetic to Fluid Modeling in Hollow Cathode Analysis

abstract: In this study, we conduct 2D3V Particle-In-Cell simulations of hollow cathodes, encompassing both the channel and plume region, with an emphasis on plasma switch applications. The plasma in the hollow cathode channel can exhibit kinetic effects depending on how fast electrons emitted from the insert are thermalized via Coulomb collisions. The criterion that determines whether the plasma operates in a Duid or kinetic regime is given as follows. When Coulomb collisions occur at a much greater rate than ionization or excitation events, the Electron Energy Distribution Function relaxes to aMaxwellian distribution and the plasmawithin the channel can be describedwith aDuidmodel. In contrast, if inelastic processes are much faster, then the Electron Energy Distribution Function in the channel exhibits a notable high-energy tail, and a kinetic treatment is required. This criterion is applied to other kinds of hollow cathodes from the literature, revealing that a Duid approach is suitable for most electric propulsion applications, whereas a kinetic treatment might be necessary for plasma switches. Additionally, amomentumbalance reveals that a diffusion equation is sufAcient to predict the plasma plume expansion, a crucial input in the design of hollow cathodes for plasma switch applications.


title: Mixed-Precision Computing in the GRIST Dynamical Core for Weather and Climate Modelling

abstract: Atmosphere modelling applications become increasingly memory-bound due to the inconsistent development rates between processor speeds and memory bandwidth. In this study, we mitigate memory bottlenecks and reduce the computational load of the GRIST dynamical core by adopting the mixed-precision computing strategy. Guided by a limited-degree of iterative development principle, we identify the equation terms that are precision insensitive and modify them from double- to single-precision. The results show that most precision-sensitive terms are predominantly linked to pressure-gradient and gravity terms, while most precision-insensitive terms are advective terms. The computational cost is reduced without compromising the solver accuracy. The runtime of the model's hydrostatic solver, non-hydrostatic solver, and tracer transport solver is reduced by 24%, 27%, and 44%, respectively. A series of idealized tests, real-world weather and climate modelling tests, has been performed to assess the optimized model performance qualitatively and quantitatively. In particular, in the high-resolution weather forecast simulation, the model sensitivity to the precision level is mainly dominated by the small-scale features. While in long-term climate simulation, the precision-induced sensitivity can form at the large scale.


title: Assessing Economic Viability: A Comparative Analysis of Total Cost of Ownership for Domain-Adapted Large Language Models versus State-of-the-art Counterparts in Chip Design Coding Assistance

abstract: This paper presents a comparative analysis of total cost of ownership (TCO) and performance between domain-adapted large language models (LLM) and state-of-the-art (SoTA) LLMs , with a particular emphasis on tasks related to coding assistance for chip design. We examine the TCO and performance metrics of a domain-adaptive LLM, ChipNeMo, against two leading LLMs, Claude 3 Opus and ChatGPT-4 Turbo, to assess their efficacy in chip design coding generation. Through a detailed evaluation of the accuracy of the model, training methodologies, and operational expenditures, this study aims to provide stakeholders with critical information to select the most economically viable and performance-efficient solutions for their specific needs. Our results underscore the benefits of employing domain-adapted models, such as ChipNeMo, that demonstrate improved performance at significantly reduced costs compared to their general-purpose counterparts. In particular, we reveal the potential of domain-adapted LLMs to decrease TCO by approximately 90%-95%, with the cost advantages becoming increasingly evident as the deployment scale expands. With expansion of deployment, the cost benefits of ChipNeMo become more pronounced, making domain-adaptive LLMs an attractive option for organizations with substantial coding needs supported by LLMs


title: Mid-infrared 2D nonredundant optical phased array of mirror emitters in an InGaAs/InP platform

abstract: The extension of photonic technologies such as lidar and free-space optical communications from the traditional visible and near-infrared wavelengths to longer wavelengths can improve performance in adverse environments such as haze, fog, smoke, or strong solar background. Non-mechanical beam steerers will be a critical component of the low size, weight, and power modules needed for the portable or unmanned systems deployed in these environments. In this work, we demonstrate the first 2D optical phased array for non-mechanical beam steering in the mid-infrared spectral region. We combine a total internal-reflection mirror emitter with a non-redundant array of 30 elements to carry out 2D beam steering at a single wavelength of 4.6 $\mu$m. The experiment yielded 600 resolvable far-field points, with 2400 points over a $28^\circ \times 28^\circ$ field of view calculated theoretically. Moreover, the device was fabricated in a passive InGaAs/InP platform, contributing another advance in the ongoing development of quantum cascade laser-based photonic integration.


title: Complex variable solution on over-/under-break shallow tunnelling in gravitational geomaterial with reasonable far-field displacement

abstract: Over-/under-break excavation is a common phenomenon in shallow tunnelling, which is nonetheless not generally considered in existing complex variable solutions. In this paper, a new equilibrium mechanical model on over-/under-break shallow tunnelling in gravitational geomaterial is established by fixing far-field ground surface to form a corresponding mixed boundary problem. With integration of a newly proposed bidirectional composite conformal mapping using Charge Simulation Method, a complex variable solution of infinite complex potential series is subsequently derived using analytic continuation to tranform the mixed boundaries into a homogenerous Riemann-Hilbert problem, which is iteratively solved to obtain the stress and displacement in geomaterial. The infinite complex potential series of the complex variable solution are truncated to obtain numerical results, which is rectified by Lanczos filtering to reduce the oscillation of Gibbs phenomena. The bidirectional conformal mapping is discussed and validated via several numerical cases, and the subsequent complex variable solution is verified by examining the Lanczos filtering and solution convergence, and comparing with corresponding finite element solution and existing analytical solution. Further discussions are made to disclose possible defects of the proposed solution for objectivity.


title: Uncertainty Quantification in Detecting Choroidal Metastases on MRI via Evolutionary Strategies

abstract: Uncertainty quantification plays a vital role in facilitating the practical implementation of AI in radiology by addressing growing concerns around trustworthiness. Given the challenges associated with acquiring large, annotated datasets in this field, there is a need for methods that enable uncertainty quantification in small data AI approaches tailored to radiology images. In this study, we focused on uncertainty quantification within the context of the small data evolutionary strategies-based technique of deep neuroevolution (DNE). Specifically, we employed DNE to train a simple Convolutional Neural Network (CNN) with MRI images of the eyes for binary classification. The goal was to distinguish between normal eyes and those with metastatic tumors called choroidal metastases. The training set comprised 18 images with choroidal metastases and 18 without tumors, while the testing set contained a tumor-to-normal ratio of 15:15. We trained CNN model weights via DNE for approximately 40,000 episodes, ultimately reaching a convergence of 100% accuracy on the training set. We saved all models that achieved maximal training set accuracy. Then, by applying these models to the testing set, we established an ensemble method for uncertainty quantification.The saved set of models produced distributions for each testing set image between the two classes of normal and tumor-containing. The relative frequencies permitted uncertainty quantification of model predictions. Intriguingly, we found that subjective features appreciated by human radiologists explained images for which uncertainty was high, highlighting the significance of uncertainty quantification in AI-driven radiological analyses.


title: gnss_lib_py: Analyzing GNSS Data with Python

abstract: This paper presents gnss_lib_py, a Python library used to parse, analyze, and visualize data from a variety of GNSS (Global Navigation Satellite Systems) data sources. The gnss_lib_py library's ease of use, modular capabilities, testing coverage, and extensive documentation make it an attractive tool not only for scientific and industry users wanting a quick, out-of-the-box solution but also for advanced GNSS users developing new GNSS algorithms. gnss_lib_py has already demonstrated its usefulness and impact through presentation in academic conferences, use in research papers, and adoption in graduate-level university course curricula.


title: WROOM: An Autonomous Driving Approach for Off-Road Navigation

abstract: Off-road navigation is a challenging problem both at the planning level to get a smooth trajectory and at the control level to avoid flipping over, hitting obstacles, or getting stuck at a rough patch. There have been several recent works using classical approaches involving depth map prediction followed by smooth trajectory planning and using a controller to track it. We design an end-to-end reinforcement learning (RL) system for an autonomous vehicle in off-road environments using a custom-designed simulator in the Unity game engine. We warm-start the agent by imitating a rule-based controller and utilize Proximal Policy Optimization (PPO) to improve the policy based on a reward that incorporates Control Barrier Functions (CBF), facilitating the agent's ability to generalize effectively to real-world scenarios. The training involves agents concurrently undergoing domain-randomized trials in various environments. We also propose a novel simulation environment to replicate off-road driving scenarios and deploy our proposed approach on a real buggy RC car. Videos and additional results: https://sites.google.com/view/wroom-utd/home


title: On Speculative Decoding for Multimodal Large Language Models

abstract: Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing components from the draft model. Our experiments across three different tasks show that speculative decoding can achieve a memory-bound speedup of up to 2.37$\times$ using a 115M parameter language model that we trained from scratch. Additionally, we introduce a compact LLaVA draft model incorporating an image adapter, which shows marginal performance gains in image captioning while maintaining comparable results in other tasks.


title: Voice Attribute Editing with Text Prompt

abstract: Despite recent advancements in speech generation with text prompt providing control over speech style, voice attributes in synthesized speech remain elusive and challenging to control. This paper introduces a novel task: voice attribute editing with text prompt, with the goal of making relative modifications to voice attributes according to the actions described in the text prompt. To solve this task, VoxEditor, an end-to-end generative model, is proposed. In VoxEditor, addressing the insufficiency of text prompt, a Residual Memory (ResMem) block is designed, that efficiently maps voice attributes and these descriptors into the shared feature space. Additionally, the ResMem block is enhanced with a voice attribute degree prediction (VADP) block to align voice attributes with corresponding descriptors, addressing the imprecision of text prompt caused by non-quantitative descriptions of voice attributes. We also establish the open-source VCTK-RVA dataset, which leads the way in manual annotations detailing voice characteristic differences among different speakers. Extensive experiments demonstrate the effectiveness and generalizability of our proposed method in terms of both objective and subjective metrics. The dataset and audio samples are available on the website.


title: A Lightweight Spatiotemporal Network for Online Eye Tracking with Event Camera

abstract: Event-based data are commonly encountered in edge computing environments where efficiency and low latency are critical. To interface with such data and leverage their rich temporal features, we propose a causal spatiotemporal convolutional network. This solution targets efficient implementation on edge-appropriate hardware with limited resources in three ways: 1) deliberately targets a simple architecture and set of operations (convolutions, ReLU activations) 2) can be configured to perform online inference efficiently via buffering of layer outputs 3) can achieve more than 90% activation sparsity through regularization during training, enabling very significant efficiency gains on event-based processors. In addition, we propose a general affine augmentation strategy acting directly on the events, which alleviates the problem of dataset scarcity for event-based systems. We apply our model on the AIS 2024 event-based eye tracking challenge, reaching a score of 0.9916 p10 accuracy on the Kaggle private testset.