title: Device Scheduling for Relay-assisted Over-the-Air Aggregation in Federated Learning

abstract: Federated learning (FL) leverages data distributed at the edge of the network to enable intelligent applications. The efficiency of FL can be improved by using over-the-air computation (AirComp) technology in the process of gradient aggregation. In this paper, we propose a relay-assisted large-scale FL framework, and investigate the device scheduling problem in relay-assisted FL systems under the constraints of power consumption and mean squared error (MSE). we formulate a joint device scheduling, and power allocation problem to maximize the number of scheduled devices. We solve the resultant non-convex optimization problem by transforming the optimization problem into multiple sparse optimization problems. By the proposed device scheduling algorithm, these sparse sub-problems are solved and the maximum number of federated learning edge devices is obtained. The simulation results demonstrate the effectiveness of the proposed scheme as compared with other benchmark schemes.


title: LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset

abstract: Instance shape reconstruction from a 3D scene involves recovering the full geometries of multiple objects at the semantic instance level. Many methods leverage data-driven learning due to the intricacies of scene complexity and significant indoor occlusions. Training these methods often requires a large-scale, high-quality dataset with aligned and paired shape annotations with real-world scans. Existing datasets are either synthetic or misaligned, restricting the performance of data-driven methods on real data. To this end, we introduce LASA, a Large-scale Aligned Shape Annotation Dataset comprising 10,412 high-quality CAD annotations aligned with 920 real-world scene scans from ArkitScenes, created manually by professional artists. On this top, we propose a novel Diffusion-based Cross-Modal Shape Reconstruction (DisCo) method. It is empowered by a hybrid feature aggregation design to fuse multi-modal inputs and recover high-fidelity object geometries. Besides, we present an Occupancy-Guided 3D Object Detection (OccGOD) method and demonstrate that our shape annotations provide scene occupancy clues that can further improve 3D object detection. Supported by LASA, extensive experiments show that our methods achieve state-of-the-art performance in both instance-level scene reconstruction and 3D object detection tasks.


title: Scene-Conditional 3D Object Stylization and Composition

abstract: Recently, 3D generative models have made impressive progress, enabling the generation of almost arbitrary 3D assets from text or image inputs. However, these approaches generate objects in isolation without any consideration for the scene where they will eventually be placed. In this paper, we propose a framework that allows for the stylization of an existing 3D asset to fit into a given 2D scene, and additionally produce a photorealistic composition as if the asset was placed within the environment. This not only opens up a new level of control for object stylization, for example, the same assets can be stylized to reflect changes in the environment, such as summer to winter or fantasy versus futuristic settings-but also makes the object-scene composition more controllable. We achieve this by combining modeling and optimizing the object's texture and environmental lighting through differentiable ray tracing with image priors from pre-trained text-to-image diffusion models. We demonstrate that our method is applicable to a wide variety of indoor and outdoor scenes and arbitrary objects.


title: Searching for Axial Neutral Current Non-Standard Interactions of neutrinos by DUNE-like experiments

abstract: The increasingly precise neutrino experiments raise the hope for searching for new physics through studying the impact of Neutral Current (NC) Non-Standard Interactions (NSI) of neutrinos with matter fields. Neutrino oscillation experiments along with the Elastic Coherent $\nu$ Nucleus Scattering (CE$\nu$NS) experiments already set strong bounds on all the flavor elements of the "vector" NC NSI. However, "axial" NC NSI can hide from these experiments. We show how a DUNE-like experiment can probe these couplings by studying NC Deep Inelastic Scattering (DIS) events. We find that strong bounds can be set on the axial NC NSI of neutrinos with the $u$, $d$, and $s$ quarks. We show that using both the near and far detectors, a DUNE-like experiment can significantly improve the present bounds on all the flavor elements.


title: Nanocrystal Programmable Assembly Beyond Hard Spheres (or Shapes) and Other (Simple) Potentials

abstract: Ligands are the key to almost any strategy in the assembly of programmable nanocrystals (or nanoparticles) and must be accurately considered in any predictive model. Hard Spheres (or Shapes) provide the simplest and yet quite successful approach to assembly, with remarkable sophisticated predictions verified in experiments. There are, however, many situations where hard spheres/shapes predictions fail. This prompts three important questions: {\em In what situations should hard spheres/shapes models be expected to work?} and when they do not work, {\em Is there a general model that successfully corrects hard sphere/shape predictions?} and given other successful models where ligands are included explicitly, and of course, numerical simulations, {\em can we unify hard sphere/shape models, explicit ligand models and all atom simulations?}. The Orbifold Topological Model (OTM) provides a positive answer to these three questions. In this paper, I give a detailed review of OTM, describing the concept of ligand vortices and how it leads to spontaneous valence and nanoparticle "eigenshapes" while providing a prediction of the lattice structure, without fitting parameters, which accounts for many body effects not captured in (two-body) potentials. I present a thorough survey of experiments and simulations and show that, to this date, they are in full agreement with the OTM predictions. I will conclude with a discussion on whether NC superlattices are equilibrium structures and some significant challenges in structure prediction.


title: Terrapin Attack: Breaking SSH Channel Integrity By Sequence Number Manipulation

abstract: The SSH protocol provides secure access to network services, particularly remote terminal login and file transfer within organizational networks and to over 15 million servers on the open internet. SSH uses an authenticated key exchange to establish a secure channel between a client and a server, which protects the confidentiality and integrity of messages sent in either direction. The secure channel prevents message manipulation, replay, insertion, deletion, and reordering. At the network level, SSH uses the Binary Packet Protocol over TCP. In this paper, we show that the SSH Binary Packet Protocol is no longer a secure channel: SSH channel integrity (INT-PST, aINT-PTXT, and INT-sfCTF) is broken for three widely used encryption modes. This allows prefix truncation attacks where encrypted packets at the beginning of the SSH channel can be deleted without the client or server noticing it. We demonstrate several real-world applications of this attack. We show that we can fully break SSH extension negotiation (RFC 8308), such that an attacker can downgrade the public key algorithms for user authentication or turn off a new countermeasure against keystroke timing attacks introduced in OpenSSH 9.5. Further, we identify an implementation flaw in AsyncSSH that, together with prefix truncation, allows an attacker to redirect the victim's login into a shell controlled by the attacker. We also performed an internet-wide scan and found that 71.6% of SSH servers support a vulnerable encryption mode, while 63.2% even list it as their preferred choice. We identify two root causes that enable these attacks: First, the SSH handshake supports optional messages that are not authenticated. Second, SSH does not reset message sequence numbers when activating encryption keys. Based on this analysis, we propose effective and backward-compatible changes to SSH that mitigate our attacks.


title: Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model

abstract: The ability of large language models (LLMs) to process visual inputs has given rise to general-purpose vision systems, unifying various vision-language (VL) tasks by instruction tuning. However, due to the enormous diversity in input-output formats in the vision domain, existing general-purpose models fail to successfully integrate segmentation and multi-image inputs with coarse-level tasks into a single framework. In this work, we introduce VistaLLM, a powerful visual system that addresses coarse- and fine-grained VL tasks over single and multiple input images using a unified framework. VistaLLM utilizes an instruction-guided image tokenizer that filters global embeddings using task descriptions to extract compressed and refined features from numerous images. Moreover, VistaLLM employs a gradient-aware adaptive sampling technique to represent binary segmentation masks as sequences, significantly improving over previously used uniform sampling. To bolster the desired capability of VistaLLM, we curate CoinIt, a comprehensive coarse-to-fine instruction tuning dataset with 6.8M samples. We also address the lack of multi-image grounding datasets by introducing a novel task, AttCoSeg (Attribute-level Co-Segmentation), which boosts the model's reasoning and grounding capability over multiple input images. Extensive experiments on a wide range of V- and VL tasks demonstrate the effectiveness of VistaLLM by achieving consistent state-of-the-art performance over strong baselines across all downstream tasks. Our project page can be found at https://shramanpramanick.github.io/VistaLLM/.


title: Loop corrections in the separate universe picture

abstract: In inflationary models that produce a spike of power on short scales, back-reaction of small-scale substructure onto large-scale modes is enhanced. We argue that the separate universe framework provides a highly convenient tool to compute loop corrections that quantify this back-reaction. Each loop of interest is characterized by large hierarchies in wavenumber and horizon exit time. The separate universe framework highlights important factorizations involving these hierarchies. We interpret each loop correction in terms of a simple, classical, back-reaction model, and clarify the meaning of the different volume scalings that have been reported in the literature. We argue that significant back-reaction requires both short-scale nonlinearities and long-short couplings that modulate the short-scale power spectrum. In the absence of long-short couplings, only incoherent shot noise-like effects are present, which are volume-suppressed. Dropping the shot noise, back-reaction from a particular scale is controlled by a product of $f_{NL}$-like parameters: an equilateral configuration measuring the nonlinearity of the short-scale modes, and a squeezed configuration measuring the long-short coupling. These may carry important scale dependence controlling the behaviour of the loop in the decoupling limit where the hierarchy of scales becomes large. In single-field models the long-short coupling may be suppressed by this hierarchy, in which case the net back-reaction would be safely suppressed. We illustrate our framework using explicit computations in a 3-phase ultra-slow-roll scenario. Finally, we discuss different choices for the smoothing scale used in the separate universe framework and argue the effect can be absorbed into a renormalization of local operators.


title: SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process

abstract: In this paper, we explore a principal way to enhance the quality of object masks produced by different segmentation models. We propose a model-agnostic solution called SegRefiner, which offers a novel perspective on this problem by interpreting segmentation refinement as a data generation process. As a result, the refinement process can be smoothly implemented through a series of denoising diffusion steps. Specifically, SegRefiner takes coarse masks as inputs and refines them using a discrete diffusion process. By predicting the label and corresponding states-transition probabilities for each pixel, SegRefiner progressively refines the noisy masks in a conditional denoising manner. To assess the effectiveness of SegRefiner, we conduct comprehensive experiments on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation. The results demonstrate the superiority of our SegRefiner from multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics across different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, it exhibits a strong capability to capture extremely fine details when refining high-resolution images. The source code and trained models are available at https://github.com/MengyuWang826/SegRefiner.


title: ROSE: A reduced-order scattering emulator for optical models

abstract: A new generation of phenomenological optical potentials requires robust calibration and uncertainty quantification, motivating the use of Bayesian statistical methods. These Bayesian methods usually require calculating observables for thousands or even millions of parameter sets, making fast and accurate emulators highly desirable or even essential. Emulating scattering across different energies or with interactions such as optical potentials is challenging because of the non-affine parameter dependence, meaning the parameters do not all factorize from individual operators. Here we introduce and demonstrate the Reduced Order Scattering Emulator (ROSE) framework, a reduced basis emulator that can handle non-affine problems. ROSE is fully extensible and works within the publicly available BAND Framework software suite for calibration, model mixing, and experimental design. As a demonstration problem, we use ROSE to calibrate a realistic nucleon-target scattering model through the calculation of elastic cross sections. This problem shows the practical value of the ROSE framework for Bayesian uncertainty quantification with controlled trade-offs between emulator speed and accuracy as compared to high-fidelity solvers. Planned extensions of ROSE are discussed.


title: Enhancing quantum utility: simulating large-scale quantum spin chains on superconducting quantum computers

abstract: We present the quantum simulation of the frustrated quantum spin-$\frac{1}{2}$ antiferromagnetic Heisenberg spin chain with competing nearest-neighbor $(J_1)$ and next-nearest-neighbor $(J_2)$ exchange interactions in the real superconducting quantum computer with qubits ranging up to 100. In particular, we implement, for the first time, the Hamiltonian with the next-nearest neighbor exchange interaction in conjunction with the nearest neighbor interaction on IBM's superconducting quantum computer and carry out the time evolution of the spin chain by employing first-order Trotterization. Furthermore, our novel implementation of second-order Trotterization for the isotropic Heisenberg spin chain, involving only nearest-neighbor exchange interaction, enables precise measurement of the expectation values of staggered magnetization observable across a range of up to 100 qubits. Notably, in both cases, our approach results in a constant circuit depth in each Trotter step, independent of the initial number of qubits. Our demonstration of the accurate measurement of expectation values for the large-scale quantum system using superconducting quantum computers designates the quantum utility of these devices for investigating various properties of many-body quantum systems. This will be a stepping stone to achieving the quantum advantage over classical ones in simulating quantum systems before the fault tolerance quantum era.


title: The "visible" Wigner matrix

abstract: We consider the ``visible'' Wigner matrix, a Wigner matrix whose $(i, j)$-th entry is coerced to zero if $i, j$ are co-prime. Using a recent result from elementary number theory on co-primality patterns in integers, we show that the limiting spectral distribution of this matrix exists, and give explicit descriptions of its moments in terms of infinite products over primes $p$ of certain polynomials evaluated at $1/p$. We also consider the complementary ``invisible'' Wigner matrix.

categories: math.PR cond-mat.dis-nn math-ph math.MP math.NT


title: The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment: Official Splits and Benchmark

abstract: This technical report provides a detailed overview of Endoscapes, a dataset of laparoscopic cholecystectomy (LC) videos with highly intricate annotations targeted at automated assessment of the Critical View of Safety (CVS). Endoscapes comprises 201 LC videos with frames annotated sparsely but regularly with segmentation masks, bounding boxes, and CVS assessment by three different clinical experts. Altogether, there are 11090 frames annotated with CVS and 1933 frames annotated with tool and anatomy bounding boxes from the 201 videos, as well as an additional 422 frames from 50 of the 201 videos annotated with tool and anatomy segmentation masks. In this report, we provide detailed dataset statistics (size, class distribution, dataset splits, etc.) and a comprehensive performance benchmark for instance segmentation, object detection, and CVS prediction. The dataset and model checkpoints are publically available at https://github.com/CAMMA-public/Endoscapes.


title: Efficient Title Reranker for Fast and Improved Knowledge-Intense NLP

abstract: In recent RAG approaches, rerankers play a pivotal role in refining retrieval accuracy with the ability of revealing logical relations for each pair of query and text. However, existing rerankers are required to repeatedly encode the query and a large number of long retrieved text. This results in high computational costs and limits the number of retrieved text, hindering accuracy. As a remedy of the problem, we introduce the Efficient Title Reranker via Broadcasting Query Encoder, a novel technique for title reranking that achieves a 20x-40x speedup over the vanilla passage reranker. Furthermore, we introduce Sigmoid Trick, a novel loss function customized for title reranking. Combining both techniques, we empirically validated their effectiveness, achieving state-of-the-art results on all four datasets we experimented with from the KILT knowledge benchmark.


title: On Inference Stability for Diffusion Models

abstract: Denoising Probabilistic Models (DPMs) represent an emerging domain of generative models that excel in generating diverse and high-quality images. However, most current training methods for DPMs often neglect the correlation between timesteps, limiting the model's performance in generating images effectively. Notably, we theoretically point out that this issue can be caused by the cumulative estimation gap between the predicted and the actual trajectory. To minimize that gap, we propose a novel \textit{sequence-aware} loss that aims to reduce the estimation gap to enhance the sampling quality. Furthermore, we theoretically show that our proposed loss function is a tighter upper bound of the estimation loss in comparison with the conventional loss in DPMs. Experimental results on several benchmark datasets including CIFAR10, CelebA, and CelebA-HQ consistently show a remarkable improvement of our proposed method regarding the image generalization quality measured by FID and Inception Score compared to several DPM baselines. Our code and pre-trained checkpoints are available at \url{https://github.com/VinAIResearch/SA-DPM}.


title: Orisometry formalism reveals duality and exotic nonuniform response in origami sheets

abstract: Origami metamaterial design enables drastic qualitative changes in the response properties of a thin sheet via the addition of a repeating pattern of folds based around a rigid folding motion. Known also as a mechanism, this folding motion will have a very small energy cost when applied uniformly; and yet uniform activation of such remains highly difficult to observe, these sheets instead generically displaying nonuniform response patterns which are not yet well understood. Here, we present a purely geometric continuum theory which captures the nonuniform, nonlinear response to generic loading as composed locally of the planar mechanism, as well as previously identified ``twist'' and ``bend'' modes which enable the patterned sheet to curve out of the plane across long distances. Our numerical analysis confirms that these three modes govern the observed nonuniform response, varying smoothly across the sheet according to three PDEs which guarantee compatibility. In analogy with the recently solved case of planar mechanism metamaterials, these ``Orisometries'' (origami + isometries, so named by us) are subextensive but infinite in number, with each mode displaying, in the linear limit, ``sheared analytic'' spatial patterns which are controlled by the Poisson's ratio of the uniform folding mechanism. Furthermore, the ``planar'' mechanism-based deformation patterns superimpose with a mathematically dual space of ``non-planar'' twist/bend deformations to span the available soft linear response. Together, our findings furnish the first quantification of the number of soft response modes available, as well as the first intuitive quantification of their spatial distribution.

categories: cond-mat.soft cond-mat.mtrl-sci math-ph math.DG math.MP


title: TAO-Amodal: A Benchmark for Tracking Any Object Amodally

abstract: Amodal perception, the ability to comprehend complete object structures from partial visibility, is a fundamental skill, even for infants. Its significance extends to applications like autonomous driving, where a clear understanding of heavily occluded objects is essential. However, modern detection and tracking algorithms often overlook this critical capability, perhaps due to the prevalence of \textit{modal} annotations in most benchmarks. To address the scarcity of amodal benchmarks, we introduce TAO-Amodal, featuring 833 diverse categories in thousands of video sequences. Our dataset includes \textit{amodal} and modal bounding boxes for visible and partially or fully occluded objects, including those that are partially out of the camera frame. We investigate the current lay of the land in both amodal tracking and detection by benchmarking state-of-the-art modal trackers and amodal segmentation methods. We find that existing methods, even when adapted for amodal tracking, struggle to detect and track objects under heavy occlusion. To mitigate this, we explore simple finetuning schemes that can increase the amodal tracking and detection metrics of occluded objects by 2.1\% and 3.3\%.


title: Velocity reconstruction in the era of DESI and Rubin (part II): Realistic samples on the light cone

abstract: Reconstructing the galaxy peculiar velocity field from the distribution of large-scale structure plays an important role in cosmology. On one hand, it gives us an insight into structure formation and gravity; on the other, it allows us to selectively extract the kinetic Sunyaev-Zeldovich (kSZ) effect from cosmic microwave background (CMB) maps. In this work, we employ high-accuracy synthetic galaxy catalogs on the light cone to investigate how well we can recover the velocity field when utilizing the three-dimensional spatial distribution of the galaxies in a modern large-scale structure experiment such as the Dark Energy Spectroscopic Instrument (DESI) and Rubin Observatory (LSST). In particular, we adopt the standard technique used in baryon acoustic oscillation (BAO) analysis for reconstructing the Zeldovich displacements of galaxies through the continuity equation, which yields a first-order approximation to their large-scale velocities. We investigate variations in the number density, bias, mask, area, redshift noise, and survey depth, and smoothing. Since our main goal is to provide guidance for planned kSZ analysis between DESI and the Atacama Cosmology Telescope (ACT), we apply velocity reconstruction to a faithful representation of DESI spectroscopic and photometric targets. We report the cross-correlation coefficient between the reconstructed and the true velocities along the line of sight. For the DESI Y1 spectroscopic survey, we expect the correlation coefficient to be $r \approx 0.64$, while for a photometric survey with $\sigma_z/(1+z) = 0.02$, as is approximately the case for the Legacy Survey used in the target selection of DESI galaxies, $r$ shrinks by half to $r \approx 0.31$. We hope the results in this paper can be used to inform future kSZ stacking studies. All scripts used in this paper can be found here: \url{https://github.com/boryanah/abacus_kSZ_recon}.


title: Velocity reconstruction in the era of DESI and Rubin (part I): Exploring spectroscopic, photometric & hybrid samples

abstract: Peculiar velocities of galaxies and halos can be reconstructed from their spatial distribution alone. This technique is analogous to the baryon acoustic oscillations (BAO) reconstruction, using the continuity equation to connect density and velocity fields. The resulting reconstructed velocities can be used to measure imprints of galaxy velocities on the cosmic microwave background (CMB) like the kinematic Sunyaev-Zel'dovich (kSZ) effect or the moving lens effect. As the precision of these measurements increases, characterizing the performance of the velocity reconstruction becomes crucial to allow unbiased and statistically optimal inference. In this paper, we quantify the relevant performance metrics: the variance of the reconstructed velocities and their correlation coefficient with the true velocities. We show that the relevant velocities to reconstruct for kSZ and moving lens are actually the halo -- rather than galaxy -- velocities. We quantify the impact of redshift-space distortions, photometric redshift errors, satellite galaxy fraction, incorrect cosmological parameter assumptions and smoothing scale on the reconstruction performance. We also investigate hybrid reconstruction methods, where velocities inferred from spectroscopic samples are evaluated at the positions of denser photometric samples. We find that using exclusively the photometric sample is better than performing a hybrid analysis. The 2 Gpc$/h$ length simulations from AbacusSummit with realistic galaxy samples for DESI and Rubin LSST allow us to perform this analysis in a controlled setting. In the companion paper Hadzhiyska et al. 2024, we further include the effects of evolution along the light cone and give realistic performance estimates for DESI luminous red galaxies (LRGs), emission line galaxies (ELGs), and Rubin LSST-like samples.


title: A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise

abstract: The surge of interest towards Multi-modal Large Language Models (MLLMs), e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both academia and industry. They endow Large Language Models (LLMs) with powerful capabilities in visual understanding, enabling them to tackle diverse multi-modal tasks. Very recently, Google released Gemini, its newest and most capable MLLM built from the ground up for multi-modality. In light of the superior reasoning capabilities, can Gemini challenge GPT-4V's leading position in multi-modal learning? In this paper, we present a preliminary exploration of Gemini Pro's visual understanding proficiency, which comprehensively covers four domains: fundamental perception, advanced cognition, challenging vision tasks, and various expert capacities. We compare Gemini Pro with the state-of-the-art GPT-4V to evaluate its upper limits, along with the latest open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and black-box systems. The qualitative samples indicate that, while GPT-4V and Gemini showcase different answering styles and preferences, they can exhibit comparable visual reasoning capabilities, and Sphinx still trails behind them concerning domain generalizability. Specifically, GPT-4V tends to elaborate detailed explanations and intermediate steps, and Gemini prefers to output a direct and concise answer. The quantitative evaluation on the popular MME benchmark also demonstrates the potential of Gemini to be a strong challenger to GPT-4V. Our early investigation of Gemini also observes some common issues of MLLMs, indicating that there still remains a considerable distance towards artificial general intelligence. Our project for tracking the progress of MLLM is released at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.


title: Weakly Supervised Open-Vocabulary Object Detection

abstract: Despite weakly supervised object detection (WSOD) being a promising step toward evading strong instance-level annotations, its capability is confined to closed-set categories within a single training dataset. In this paper, we propose a novel weakly supervised open-vocabulary object detection framework, namely WSOVOD, to extend traditional WSOD to detect novel concepts and utilize diverse datasets with only image-level annotations. To achieve this, we explore three vital strategies, including dataset-level feature adaptation, image-level salient object localization, and region-level vision-language alignment. First, we perform data-aware feature extraction to produce an input-conditional coefficient, which is leveraged into dataset attribute prototypes to identify dataset bias and help achieve cross-dataset generalization. Second, a customized location-oriented weakly supervised region proposal network is proposed to utilize high-level semantic layouts from the category-agnostic segment anything model to distinguish object boundaries. Lastly, we introduce a proposal-concept synchronized multiple-instance network, i.e., object mining and refinement with visual-semantic alignment, to discover objects matched to the text embeddings of concepts. Extensive experiments on Pascal VOC and MS COCO demonstrate that the proposed WSOVOD achieves new state-of-the-art compared with previous WSOD methods in both close-set object localization and detection tasks. Meanwhile, WSOVOD enables cross-dataset and open-vocabulary learning to achieve on-par or even better performance than well-established fully-supervised open-vocabulary object detection (FSOVOD).


title: Efficient fidelity estimation: Alternative derivation and related applications

abstract: In [Phys. Rev. A 107, 012427 (2023)], A. J. Baldwin and J. A. Jones proved that Uhlmann-Jozsa's fidelity between two quantum states $\rho$ and $\sigma$, i.e., $F(\rho,\sigma)~:=~(Tr\sqrt{\sqrt{\rho}\sigma\sqrt{\rho}})^2$, can be written in a simplified form as $F(\rho,\sigma) = (Tr\sqrt{\rho\sigma})^2$. In this article, we give an alternative proof of this result, using a function power series expansion and the properties of the trace function. Our approach not only reinforces the validity of the simplified expression but also facilitates the exploration of novel dissimilarity functions for quantum states and more complex trace functions of a density operator.


title: Single-pixel 3D imaging based on fusion temporal data of single photon detector and millimeter-wave radar

abstract: Recently, there has been increased attention towards 3D imaging using single-pixel single-photon detection (also known as temporal data) due to its potential advantages in terms of cost and power efficiency. However, to eliminate the symmetry blur in the reconstructed images, a fixed background is required. This paper proposes a fusion-data-based 3D imaging method that utilizes a single-pixel single-photon detector and a millimeter-wave radar to capture temporal histograms of a scene from multiple perspectives. Subsequently, the 3D information can be reconstructed from the one-dimensional fusion temporal data by using Artificial Neural Network (ANN). Both the simulation and experimental results demonstrate that our fusion method effectively eliminates symmetry blur and improves the quality of the reconstructed images.


title: Intervals and Outer Measure on $\mathbb{R}$

abstract: This article gives some properties of intervals in $\mathbb{R}$ and discusses some problems involving intervals for which the concept of outer measure on $\mathbb{R}$ provides a more efficient solution than an elementary approach. The outer measure is then defined and some of its main properties in relation to intervals are developed, culminating in the countable additivity of outer measure on the 'system of intervals' $\mathcal{I} = \{$ all countable unions of intervals in $\mathbb{R}\ \}$. This demonstrates early on how the outer measure on $\mathbb{R}$ is naturally countably additive on a quite large class of sets, and motivates the Borel algebra $\mathcal{B}$ as an extension of that class which provides an additional desired property of outer measure, namely closure of its domain under set complementation -- for example as developed in [Axler, Chap 2]. Details are given of how one of the intervals problems solved by the outer measure allows proof prior to the Lebesgue integration theory of the Bounded Convergence, Monotone Convergence, and Dominated Convergence Theorems for Riemann integrals. One application of the latter is the proof of Stirling's Formula given in [Conrad]. Some further details on handling double series are provided than is normally given, based on the textbook 'Theory and Application of Infinite Series' by [Knopp] and the article [4]. The term 'countable union' of sets will mean a union of an infinite sequence of sets. $\overline{\mathbb{R}}$ will denote the extended real number system $\mathbb{R}\ \cup \{\infty, -\infty\}$.


title: DiffSpectralNet : Unveiling the Potential of Diffusion Models for Hyperspectral Image Classification

abstract: Hyperspectral images (HSI) have become popular for analysing remotely sensed images in multiple domain like agriculture, medical. However, existing models struggle with complex relationships and characteristics of spectral-spatial data due to the multi-band nature and data redundancy of hyperspectral data. To address this limitation, we propose a new network called DiffSpectralNet, which combines diffusion and transformer techniques. Our approach involves a two-step process. First, we use an unsupervised learning framework based on the diffusion model to extract both high-level and low-level spectral-spatial features. The diffusion method is capable of extracting diverse and meaningful spectral-spatial features, leading to improvement in HSI classification. Then, we employ a pretrained denoising U-Net to extract intermediate hierarchical features for classification. Finally, we use a supervised transformer-based classifier to perform the HSI classification. Through comprehensive experiments on HSI datasets, we evaluate the classification performance of DiffSpectralNet. The results demonstrate that our framework significantly outperforms existing approaches, achieving state-of-the-art performance.


title: Hierarchical Classification System for Breast Cancer Specimen Report (HCSBC) -- an end-to-end model for characterizing severity and diagnosis

abstract: Automated classification of cancer pathology reports can extract information from unstructured reports and categorize each report into structured diagnosis and severity categories. Thus, such system can reduce the burden for populating tumor registries, help registration for clinical trial as well as developing large dataset for deep learning model development using true pathologic ground truth. However, the content of breast pathology reports can be difficult for categorize due to the high linguistic variability in content and wide variety of potential diagnoses >50. Existing NLP models are primarily focused on developing classifier for primary breast cancer types (e.g. IDC, DCIS, ILC) and tumor characteristics, and ignore the rare diagnosis of cancer subtypes. We then developed a hierarchical hybrid transformer-based pipeline (59 labels) - Hierarchical Classification System for Breast Cancer Specimen Report (HCSBC), which utilizes the potential of the transformer context-preserving NLP technique and compared our model to several state of the art ML and DL models. We trained the model on the EUH data and evaluated our model's performance on two external datasets - MGH and Mayo Clinic. We publicly release the code and a live application under Huggingface spaces repository


title: Enhancing Understanding of Driving Attributes through Quantitative Assessment of Driver Cognition

abstract: This paper presents a novel approach for analysing EEG data from drivers in a simulated driving test. We focused on the Hurst exponent, Shannon entropy, and fractal dimension as markers of the nonlinear dynamics of the brain. The results show significant trends: Shannon Entropy and Fractal Dimension exhibit variations during driving condition transitions, whereas the Hurst exponent reflects memory retention portraying learning patterns. These findings suggest that the tools of Non-linear Dynamical (NLD) Theory as indicators of cognitive state and driving memory changes for assessing driver performance and advancing the understanding of non-linear dynamics of human cognition in the context of driving and beyond. Our study reveals the potential of NLD tools to elucidate brain state and system variances, enabling their integration into current Deep Learning and Machine Learning models. This integration can extend beyond driving applications and be harnessed for cognitive learning, thereby improving overall productivity and accuracy levels.


title: What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?

abstract: Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to improve the learning efficiency and generalization ability of policies learned from pixels. To that end, past work has favored large object interaction datasets, such as first-person videos of humans completing diverse tasks, in pursuit of manipulation-relevant features. Although this approach improves the efficiency of policy learning, it remains unclear how reliable these representations are in the presence of distribution shifts that arise commonly in robotic applications. Surprisingly, we find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture or the introduction of distractor objects. To understand what properties do lead to robust representations, we compare the performance of 15 pre-trained vision models under different visual appearances. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models. The rank order induced by this metric is more predictive than metrics that have previously guided generalization research within computer vision and machine learning, such as downstream ImageNet accuracy, in-domain accuracy, or shape-bias as evaluated by cue-conflict performance. We test this finding extensively on a suite of distribution shifts in ten tasks across two simulated manipulation environments. On the ALOHA setup, segmentation score predicts real-world performance after offline training with 50 demonstrations.


title: Comparison of two efficient numerical techniques based on Chelyshkov polynomial for solving stochastic It\^o-Volterra integral equation

abstract: In this study, two reliable approaches to solving the nonlinear stochastic It\^o-Volterra integral equation are provided. These equations have been evaluated using the orthonormal Chelyshkov spectral collocation technique and the orthonormal Chelyshkov spectral Galerkin method. The techniques presented here transform this problem into a collection of nonlinear algebraic equations that have been numerically solved using the Newton method. Also, the convergence analysis has been studied for both approaches. Two illustrative examples have been provided to show the efficacy, plausibility, proficiency, and applicability of the current approaches.


title: High-energy electron measurements with thin Si detectors

abstract: A technique for measuring high-energy electrons using Si detectors of various thicknesses that are much smaller than the range of the examined electrons is presented. The advantages of the method are discussed on the basis of electron-positron pair creation recently studied in deuteron-deuteron fusion reactions at very low energies. Careful Geant 4 Monte Carlo simulations enabled the identification of the main spectral contributions of emitted electrons and positrons resulting from the energy loss mechanisms and scattering processes within the target, detector and their holders. Significant changes in the intensity of the detected electrons, depending on the detector thickness and the thicknesses of absorption foils placed in the front of the detector could be observed. The corresponding correction factors have been calculated and can be used for different applications in basic and applied research.


title: An Unexpected Class of 5+gon-free Line Patterns

abstract: Let $S$ be a finite subset of ${\mathbb R}^2 \setminus (0,0)$. Generally, one would expect the pattern of lines $Ax + By = 1$, where $(A, B) \in S$ to contain polygons of all shapes and sizes. We show, however, that when $S$ is a rectangular subset of the integer lattice or a closely related set, no polygons with more than 4 sides occur. In the process, we develop a general theorem that explains how to find the next side as one travels around the boundary of a cell.


title: On the diagonal product of special unitary matrices

abstract: We study the image of $\textrm{SU}(n)$ under the diagonal product map. Using only elementary tools of linear algebra, analysis and general topology, we find the analytical formula for the boundary of this image and find all special unitary matrices whose diagonal product lies on that boundary.


title: Rota-Baxter operators and Loday-type algebras on the BiHom-associative conformal algebras

abstract: (Tri)dendriform algebras, Rota-Baxter operators, and closely related NS-algebras have a number of dominant applications in physics, especially in quantum field theory. Proceeding from the recent study relating these structures, this paper considers (tri)dendriform algebras, NS-algebras, and (twisted)Rota-Baxter operators in the context of BiHom-associative conformal algebras. A comprehensive investigation of the BiHom-(tri)dendriform conformal algebras and their characterization in terms of conformal bimodule has been conducted. The study of BiHom-NS-conformal algebra reveals that it is not only a generalization of NS-conformal algebra using two structural maps but is also the generalization of BiHom-(tri)dendriform conformal algebras. Additionally, it is found to have a close proximity between BiHom-twisted Rota-Baxter operators and BiHom-NS-conformal algebras. The comparative study to Rota-Baxter operators on BiHom-associative conformal algebras and Rota-Baxter operators on BiHom-(tri)dendriform conformal algebras reveals a relationship between BiHom-quadri conformal algebra and Rota-Baxter operators. In the end, the concept of Rota-Baxter system (a generalization of the Rota-Baxter operator) for BiHom-associative conformal algebras and BiHom-dendriform conformal algebras is narrated, where the interconnections of these algebras are depicted. Furthermore, a connection is established between BiHom-quadri conformal algebras and Rota-Baxter systems for BiHom-dendriform conformal algebras.


title: Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions

abstract: A significant amount of research is focused on developing and evaluating large language models for a variety of code synthesis tasks. These include synthesizing code from natural language, synthesizing tests from code, and synthesizing explanations of code. In contrast, the behavior of instructional code editing with LLMs is understudied. These are tasks in which the model is provided a block of code and an instruction to modify the code. The editing instruction may ask for a feature to be added or removed, describe a bug and ask for a fix, or ask for a different kind of solution. We introduce a carefully crafted benchmark of code editing tasks and use it to evaluate several cutting edge LLMs. Our evaluation exposes a significant gap between the capabilities of state-of-the-art open and closed models. For example, even GPT-3.5-Turbo is better than the best open model at code editing tasks. We also introduce a new, carefully curated, permissively licensed training dataset of code editing tasks coupled with natural language instructions. Using this training dataset, we show that we can fine-tune open Code LLMs to significantly improve their code editing capabilities, closing the gap between open and closed models. All code, data, and models are available at https://github.com/nuprl/CanItEdit.


title: The detectability of single spinless stellar-mass black holes through gravitational lensing of gravitational waves with advanced LIGO

abstract: We investigate the detectability of gravitational waves that have been lensed by a spinless stellar-mass black hole, with respect to the advanced LIGO. By solving the full relativistic linear wave equations in the spacetime of a Schwarzschild black hole, we find that the strong gravity can create unique signals in the lensed waveform, particularly during the merger and ringdown stages. The differences in terms of fitting factor between the lensed waveform and best-fitted unlensed general relativity template with spin-precessing and higher-order multipoles are greater than $5\%$ for the lens black hole mass within $70M_{\odot}<M_{\rm lens}<133.33 M_{\odot}$ under advanced LIGO's sensitivity. This is up to 5 times more detectable than the previous analysis based on the weak field approximation for a point mass and covers most part of the black hole mass gap predicted by stellar evolution theory. Based on Bayesian inference, the lensing feature can be distinguished with a signal-to-noise ratio of 12.5 for $M_{\rm lens}=70 M_{\odot}$ and 19.2 for $M_{\rm lens}=250 M_{\odot}$, which is attainable for advanced LIGO.


title: Boundary Chaos: Spectral Form Factor

abstract: Random matrix spectral correlations is a defining feature of quantum chaos. Here, we study such correlations in a minimal model of chaotic many-body quantum dynamics where interactions are confined to the system's boundary, dubbed \textit{boundary chaos}, in terms of the spectral form factor and its fluctuations. We exactly calculate the latter in the limit of large local Hilbert space dimension $q$ for different classes of random boundary interactions and find it to coincide with random matrix theory, possibly after a non-zero Thouless time. The latter effect is due to a drastic enhancement of the spectral form factor, when integer time and system size fulfill a resonance condition. We compare our semiclassical (large $q$) results with numerics at small local Hilbert space dimension ($q=2,3$) and observe qualitatively similar features as in the semiclassical regime.

categories: quant-ph cond-mat.stat-mech hep-th nlin.CD


title: Overdrawing Urns using Categories of Signed Probabilities

abstract: A basic experiment in probability theory is drawing without replacement from an urn filled with multiple balls of different colours. Clearly, it is physically impossible to overdraw, that is, to draw more balls from the urn than it contains. This paper demonstrates that overdrawing does make sense mathematically, once we allow signed distributions with negative probabilities. A new (conservative) extension of the familiar hypergeometric ('draw-and-delete') distribution is introduced that allows draws of arbitrary sizes, including overdraws. The underlying theory makes use of the dual basis functions of the Bernstein polynomials, which play a prominent role in computer graphics. Negative probabilities are treated systematically in the framework of categorical probability and the central role of datastructures such as multisets and monads is emphasised.


title: Some characterizations of ergodicity in Riesz spaces

abstract: In the recent surge of papers on ergodic theory within Riesz spaces, this article contributes by introducing enhanced characterizations of ergodicity. Our work extends and strengthens prior results from both the authors and Homann, Kuo, and Watson. Specifically, we show that in a conditional expectation preserving system (E,T,S,e), S can be extended to the natural domain of T and operates as an isometry on L^{p}(T) spaces.


title: FengWu-4DVar: Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation

abstract: Weather forecasting is a crucial yet highly challenging task. With the maturity of Artificial Intelligence (AI), the emergence of data-driven weather forecasting models has opened up a new paradigm for the development of weather forecasting systems. Despite the significant successes that have been achieved (e.g., surpassing advanced traditional physical models for global medium-range forecasting), existing data-driven weather forecasting models still rely on the analysis fields generated by the traditional assimilation and forecasting system, which hampers the significance of data-driven weather forecasting models regarding both computational cost and forecasting accuracy. In this work, we explore the possibility of coupling the data-driven weather forecasting model with data assimilation by integrating the global AI weather forecasting model, FengWu, with one of the most popular assimilation algorithms, Four-Dimensional Variational (4DVar) assimilation, and develop an AI-based cyclic weather forecasting system, FengWu-4DVar. FengWu-4DVar can incorporate observational data into the data-driven weather forecasting model and consider the temporal evolution of atmospheric dynamics to obtain accurate analysis fields for making predictions in a cycling manner without the help of physical models. Owning to the auto-differentiation ability of deep learning models, FengWu-4DVar eliminates the need of developing the cumbersome adjoint model, which is usually required in the traditional implementation of the 4DVar algorithm. Experiments on the simulated observational dataset demonstrate that FengWu-4DVar is capable of generating reasonable analysis fields for making accurate and efficient iterative predictions.


title: PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

abstract: This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. This distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated across inputs, while the majority, cold neurons, vary based on specific inputs. PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. PowerInfer further integrates adaptive predictors and neuron-aware sparse operators, optimizing the efficiency of neuron activation and computational sparsity. Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy.


title: Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling

abstract: While generative AI excels in content generation, it does not always increase user engagement. This can be attributed to two main factors. First, generative AI generates content without incorporating explicit or implicit feedback about user interactions. Even if the generated content seems to be more informative or well-written, it does not necessarily lead to an increase in user activities, such as clicks. Second, there is a concern with the quality of the content generative AI produces, which often lacks the distinctiveness and authenticity that human-created content possesses. These two factors can lead to content that fails to meet specific needs and preferences of users, ultimately reducing its potential to be engaging. This paper presents a generic framework of how to improve user engagement with generative AI by leveraging user feedback. Our solutions employ rejection sampling, a technique used in reinforcement learning, to boost engagement metrics. We leveraged the framework in the context of email notification subject lines generation for an online social network, and achieved significant engagement metric lift including +1% Session and +0.4% Weekly Active Users. We believe our work offers a universal framework that enhances user engagement with generative AI, particularly when standard generative AI reaches its limits in terms of enhancing content to be more captivating. To the best of our knowledge, this represents an early milestone in the industry's successful use of generative AI to enhance user engagement.


title: When Parameter-efficient Tuning Meets General-purpose Vision-language Models

abstract: Instruction tuning has shown promising potential for developing general-purpose AI capabilities by using large-scale pre-trained models and boosts growing research to integrate multimodal information for creative applications. However, existing works still face two main limitations: the high training costs and heavy computing resource dependence of full model fine-tuning, and the lack of semantic information in instructions, which hinders multimodal alignment. Addressing these challenges, this paper proposes a novel approach to utilize Parameter-Efficient Tuning for generAl-purpose vision-Language models, namely PETAL. PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique, which significantly reduces the training costs and reliance on heavy computing resources. Furthermore, PETAL enhances the semantic depth of instructions in two innovative ways: 1) by introducing adaptive instruction mixture-of-experts(MOEs), and 2) by fortifying the score-based linkage between parameter-efficient tuning and mutual information. Our extensive experiments across five multimodal downstream benchmarks reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness. Additionally, our approach demonstrates remarkable advantages in few-shot settings, backed by comprehensive visualization analyses. Our source code is available at: https://github. com/melonking32/PETAL.


title: Prediction of Crash Injury Severity in Florida's Interstate-95

abstract: Drivers can sustain serious injuries in traffic accidents. In this study, traffic crashes on Florida's Interstate-95 from 2016 to 2021 were gathered, and several classification methods were used to estimate the severity of driver injuries. In the feature selection method, logistic regression was applied. To compare model performances, various model assessment matrices such as accuracy, recall, and area under curve (AUC) were developed. The Adaboost algorithm outperformed the others in terms of recall and AUC. SHAP values were also generated to explain the classification model's results. This analytical study can be used to examine factors that contribute to the severity of driver injuries in crashes.


title: Democratize with Care: The need for fairness specific features in user-interface based open source AutoML tools

abstract: AI is increasingly playing a pivotal role in businesses and organizations, impacting the outcomes and interests of human users. Automated Machine Learning (AutoML) streamlines the machine learning model development process by automating repetitive tasks and making data-driven decisions, enabling even non-experts to construct high-quality models efficiently. This democratization allows more users (including non-experts) to access and utilize state-of-the-art machine-learning expertise. However, AutoML tools may also propagate bias in the way these tools handle the data, model choices, and optimization approaches adopted. We conducted an experimental study of User-interface-based open source AutoML tools (DataRobot, H2O Studio, Dataiku, and Rapidminer Studio) to examine if they had features to assist users in developing fairness-aware machine learning models. The experiments covered the following considerations for the evaluation of features: understanding use case context, data representation, feature relevance and sensitivity, data bias and preprocessing techniques, data handling capabilities, training-testing split, hyperparameter handling, and constraints, fairness-oriented model development, explainability and ability to download and edit models by the user. The results revealed inadequacies in features that could support in fairness-aware model development. Further, the results also highlight the need to establish certain essential features for promoting fairness in AutoML tools.


title: Bird Movement Prediction Using Long Short-Term Memory Networks to Prevent Bird Strikes with Low Altitude Aircraft

abstract: The number of collisions between aircraft and birds in the airspace has been increasing at an alarming rate over the past decade due to increasing bird population, air traffic and usage of quieter aircraft. Bird strikes with aircraft are anticipated to increase dramatically when emerging Advanced Air Mobility aircraft start operating in the low altitude airspace where probability of bird strikes is the highest. Not only do such bird strikes can result in human and bird fatalities, but they also cost the aviation industry millions of dollars in damages to aircraft annually. To better understand the causes and effects of bird strikes, research to date has mainly focused on analyzing factors which increase the probability of bird strikes, identifying high risk birds in different locations, predicting the future number of bird strike incidents, and estimating cost of bird strike damages. However, research on bird movement prediction for use in flight planning algorithms to minimize the probability of bird strikes is very limited. To address this gap in research, we implement four different types of Long Short-Term Memory (LSTM) models to predict bird movement latitudes and longitudes. A publicly available data set on the movement of pigeons is utilized to train the models and evaluate their performances. Using the bird flight track predictions, aircraft departures from Cleveland Hopkins airport are simulated to be delayed by varying amounts to avoid potential bird strikes with aircraft during takeoff. Results demonstrate that the LSTM models can predict bird movement with high accuracy, achieving a Mean Absolute Error of less than 100 meters, outperforming linear and nonlinear regression models. Our findings indicate that incorporating bird movement prediction into flight planning can be highly beneficial.


title: Towards an end-to-end artificial intelligence driven global weather forecasting system

abstract: The weather forecasting system is important for science and society, and significant achievements have been made in applying artificial intelligence (AI) to medium-range weather forecasting. However, existing AI-based weather forecasting models rely on analysis or reanalysis products from traditional numerical weather prediction (NWP) systems as initial conditions for making predictions. Initial states are typically generated by traditional data assimilation components, which are computational expensive and time-consuming. Here we present an AI-based data assimilation model, i.e., Adas, for global weather variables. By introducing the confidence matrix, Adas employs gated convolution to handle sparse observations and gated cross-attention for capturing the interactions between the background and observations. Further, we combine Adas with the advanced AI-based forecasting model (i.e., FengWu) to construct the first end-to-end AI-based global weather forecasting system: FengWu-Adas. We demonstrate that Adas can assimilate global observations to produce high-quality analysis, enabling the system operate stably for long term. Moreover, we are the first to apply the methods to real-world scenarios, which is more challenging and has considerable practical application potential. We have also achieved the forecasts based on the analyses generated by AI with a skillful forecast lead time exceeding that of the IFS for the first time.


title: Open Vocabulary Semantic Scene Sketch Understanding

abstract: We study the underexplored but fundamental vision problem of machine understanding of abstract freehand scene sketches. We introduce a sketch encoder that results in semantically-aware feature space, which we evaluate by testing its performance on a semantic sketch segmentation task. To train our model we rely only on the availability of bitmap sketches with their brief captions and do not require any pixel-level annotations. To obtain generalization to a large set of sketches and categories, we build on a vision transformer encoder pretrained with the CLIP model. We freeze the text encoder and perform visual-prompt tuning of the visual encoder branch while introducing a set of critical modifications. Firstly, we augment the classical key-query (k-q) self-attention blocks with value-value (v-v) self-attention blocks. Central to our model is a two-level hierarchical network design that enables efficient semantic disentanglement: The first level ensures holistic scene sketch encoding, and the second level focuses on individual categories. We, then, in the second level of the hierarchy, introduce a cross-attention between textual and visual branches. Our method outperforms zero-shot CLIP pixel accuracy of segmentation results by 37 points, reaching an accuracy of $85.5\%$ on the FS-COCO sketch dataset. Finally, we conduct a user study that allows us to identify further improvements needed over our method to reconcile machine and human understanding of scene sketches.


title: Towards Better Serialization of Tabular Data for Few-shot Classification with Large Language Models

abstract: We present a study on the integration of Large Language Models (LLMs) in tabular data classification, emphasizing an efficient framework. Building upon existing work done in TabLLM (arXiv:2210.10723), we introduce three novel serialization techniques, including the standout LaTeX serialization method. This method significantly boosts the performance of LLMs in processing domain-specific datasets, Our method stands out for its memory efficiency and ability to fully utilize complex data structures. Through extensive experimentation, including various serialization approaches like feature combination and importance, we demonstrate our work's superiority in accuracy and efficiency over traditional models.


title: Clustering of Global Magnetospheric Observations

abstract: The use of supervised methods in space science have demonstrated powerful capability in classification tasks, but unsupervised methods have been less utilized for the clustering of spacecraft observations. We use a combination of unsupervised methods, being principal component analysis, self-organizing maps, and hierarchical agglomerative clustering, to make predictions on if THEMIS and MMS observations occurred in the magnetosphere, magnetosheath, or the solar wind. The resulting predictions are validated visually by analyzing the distribution of predictions and studying individual time series. Particular nodes in the self organizing map are studied to see what data they represent. The capability of deeper hierarchical analysis using this model is briefly explored. Finally, the changes in region prediction can be used to infer magnetopause and bow shock crossings, which can act as an additional method of validation, and are saved for their utility in solar wind validation, understanding magnetopause processes, and the potential to develop a bow shock model.


title: Users Approach on Providing Feedback for Smart Home Devices

abstract: Smart Home technology has accomplished extraordinary interest in making individuals' lives more straightforward and more relaxing as of late. Technology as of late brought about delivering numerous savvy and refined frameworks which advanced clever living innovation. In this paper, we will be investigating the behavioural intention of user's approach on providing feedback for smart home devices. We will be conducting an online survey for sample of three to five students selected by simple random sampling to study the user's motto for giving feedback on smart home devices and their expectations. We have observed that most users are ready to share their feedback on smart home devices actively to improvise the service and quality of the product to fulfill the user needs and make their lives easier.


title: Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformer

abstract: Recently, many mesh-based graph neural network (GNN) models have been proposed for modeling complex high-dimensional physical systems. Remarkable achievements have been made in significantly reducing the solving time compared to traditional numerical solvers. These methods are typically designed to i) reduce the computational cost in solving physical dynamics and/or ii) propose techniques to enhance the solution accuracy in fluid and rigid body dynamics. However, it remains under-explored whether they are effective in addressing the challenges of flexible body dynamics, where instantaneous collisions occur within a very short timeframe. In this paper, we present Hierarchical Contact Mesh Transformer (HCMT), which uses hierarchical mesh structures and can learn long-range dependencies (occurred by collisions) among spatially distant positions of a body -- two close positions in a higher-level mesh correspond to two distant positions in a lower-level mesh. HCMT enables long-range interactions, and the hierarchical mesh structure quickly propagates collision effects to faraway positions. To this end, it consists of a contact mesh Transformer and a hierarchical mesh Transformer (CMT and HMT, respectively). Lastly, we propose a flexible body dynamics dataset, consisting of trajectories that reflect experimental settings frequently used in the display industry for product designs. We also compare the performance of several baselines using well-known benchmark datasets. Our results show that HCMT provides significant performance improvements over existing methods. Our code is available at https://github.com/yuyudeep/hcmt.


title: MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers

abstract: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in real applications. To address these issues, this paper breaks down the text-based video editing task into two stages. First, we leverage an pre-trained text-to-image diffusion model to simultaneously edit few keyframes in an zero-shot way. Second, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the edited keyframes, using the structural guidance from intermediate frames. Experimental results suggest that our MaskINT achieves comparable performance with diffusion-based methodologies, while significantly improve the inference time. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.


title: Distilling Autoregressive Models to Obtain High-Performance Non-Autoregressive Solvers for Vehicle Routing Problems with Faster Inference Speed

abstract: Neural construction models have shown promising performance for Vehicle Routing Problems (VRPs) by adopting either the Autoregressive (AR) or Non-Autoregressive (NAR) learning approach. While AR models produce high-quality solutions, they generally have a high inference latency due to their sequential generation nature. Conversely, NAR models generate solutions in parallel with a low inference latency but generally exhibit inferior performance. In this paper, we propose a generic Guided Non-Autoregressive Knowledge Distillation (GNARKD) method to obtain high-performance NAR models having a low inference latency. GNARKD removes the constraint of sequential generation in AR models while preserving the learned pivotal components in the network architecture to obtain the corresponding NAR models through knowledge distillation. We evaluate GNARKD by applying it to three widely adopted AR models to obtain NAR VRP solvers for both synthesized and real-world instances. The experimental results demonstrate that GNARKD significantly reduces the inference time (4-5 times faster) with acceptable performance drop (2-3\%). To the best of our knowledge, this study is first-of-its-kind to obtain NAR VRP solvers from AR ones through knowledge distillation.


title: Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation

abstract: Referring Remote Sensing Image Segmentation (RRSIS) is a new challenge that combines computer vision and natural language processing, delineating specific regions in aerial images as described by textual queries. Traditional Referring Image Segmentation (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery, leading to suboptimal segmentation results. To address these challenges, we introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS. RMSIN incorporates an Intra-scale Interaction Module (IIM) to effectively address the fine-grained detail required at multiple scales and a Cross-scale Interaction Module (CIM) for integrating these details coherently across the network. Furthermore, RMSIN employs an Adaptive Rotated Convolution (ARC) to account for the diverse orientations of objects, a novel contribution that significantly enhances segmentation accuracy. To assess the efficacy of RMSIN, we have curated an expansive dataset comprising 17,402 image-caption-mask triplets, which is unparalleled in terms of scale and variety. This dataset not only presents the model with a wide range of spatial and rotational scenarios but also establishes a stringent benchmark for the RRSIS task, ensuring a rigorous evaluation of performance. Our experimental evaluations demonstrate the exceptional performance of RMSIN, surpassing existing state-of-the-art models by a significant margin. All datasets and code are made available at https://github.com/Lsan2401/RMSIN.


title: Atlantis: Enabling Underwater Depth Estimation with Stable Diffusion

abstract: Monocular depth estimation has experienced significant progress on terrestrial images in recent years, largely due to deep learning advancements. However, it remains inadequate for underwater scenes, primarily because of data scarcity. Given the inherent challenges of light attenuation and backscattering in water, acquiring clear underwater images or precise depth information is notably difficult and costly. Consequently, learning-based approaches often rely on synthetic data or turn to unsupervised or self-supervised methods to mitigate this lack of data. Nonetheless, the performance of these methods is often constrained by the domain gap and looser constraints. In this paper, we propose a novel pipeline for generating photorealistic underwater images using accurate terrestrial depth data. This approach facilitates the training of supervised models for underwater depth estimation, effectively reducing the performance disparity between terrestrial and underwater environments. Contrary to prior synthetic datasets that merely apply style transfer to terrestrial images without altering the scene content, our approach uniquely creates vibrant, non-existent underwater scenes by leveraging terrestrial depth data through the innovative Stable Diffusion model. Specifically, we introduce a unique Depth2Underwater ControlNet, trained on specially prepared \{Underwater, Depth, Text\} data triplets, for this generation task. Our newly developed dataset enables terrestrial depth estimation models to achieve considerable improvements, both quantitatively and qualitatively, on unseen underwater images, surpassing their terrestrial pre-trained counterparts. Moreover, the enhanced depth accuracy for underwater scenes also aids underwater image restoration techniques that rely on depth maps, further demonstrating our dataset's utility. The dataset will be available at https://github.com/zkawfanx/Atlantis.


title: A Performance Evaluation of a Quantized Large Language Model on Various Smartphones

abstract: This paper explores the feasibility and performance of on-device large language model (LLM) inference on various Apple iPhone models. Amidst the rapid evolution of generative AI, on-device LLMs offer solutions to privacy, security, and connectivity challenges inherent in cloud-based models. Leveraging existing literature on running multi-billion parameter LLMs on resource-limited devices, our study examines the thermal effects and interaction speeds of a high-performing LLM across different smartphone generations. We present real-world performance results, providing insights into on-device inference capabilities.


title: A Study on Social Robot Behavior in Group Conversation

abstract: Recently, research in human-robot interaction began to consider a robot's influence at the group level. Despite the recent growth in research investigating the effects of robots within groups of people, our overall understanding of what happens when robots are placed within groups or teams of people is still limited. This paper investigates several key problems for social robots that manage conversations in a group setting, where the number of participants is more than two. In a group setting, the conversation dynamics are a lot more complicated than the conventional one-to-one conversation, thus, there are more challenges need to be solved.


title: Principled Weight Initialisation for Input-Convex Neural Networks

abstract: Input-Convex Neural Networks (ICNNs) are networks that guarantee convexity in their input-output mapping. These networks have been successfully applied for energy-based modelling, optimal transport problems and learning invariances. The convexity of ICNNs is achieved by using non-decreasing convex activation functions and non-negative weights. Because of these peculiarities, previous initialisation strategies, which implicitly assume centred weights, are not effective for ICNNs. By studying signal propagation through layers with non-negative weights, we are able to derive a principled weight initialisation for ICNNs. Concretely, we generalise signal propagation theory by removing the assumption that weights are sampled from a centred distribution. In a set of experiments, we demonstrate that our principled initialisation effectively accelerates learning in ICNNs and leads to better generalisation. Moreover, we find that, in contrast to common belief, ICNNs can be trained without skip-connections when initialised correctly. Finally, we apply ICNNs to a real-world drug discovery task and show that they allow for more effective molecular latent space exploration.


title: Learning to Reweight for Graph Neural Network

abstract: Graph Neural Networks (GNNs) show promising results for graph tasks. However, existing GNNs' generalization ability will degrade when there exist distribution shifts between testing and training graph data. The cardinal impetus underlying the severe degeneration is that the GNNs are architected predicated upon the I.I.D assumptions. In such a setting, GNNs are inclined to leverage imperceptible statistical correlations subsisting in the training set to predict, albeit it is a spurious correlation. In this paper, we study the problem of the generalization ability of GNNs in Out-Of-Distribution (OOD) settings. To solve this problem, we propose the Learning to Reweight for Generalizable Graph Neural Network (L2R-GNN) to enhance the generalization ability for achieving satisfactory performance on unseen testing graphs that have different distributions with training graphs. We propose a novel nonlinear graph decorrelation method, which can substantially improve the out-of-distribution generalization ability and compares favorably to previous methods in restraining the over-reduced sample size. The variables of the graph representation are clustered based on the stability of the correlation, and the graph decorrelation method learns weights to remove correlations between the variables of different clusters rather than any two variables. Besides, we interpose an efficacious stochastic algorithm upon bi-level optimization for the L2R-GNN framework, which facilitates simultaneously learning the optimal weights and GNN parameters, and avoids the overfitting problem. Experimental results show that L2R-GNN greatly outperforms baselines on various graph prediction benchmarks under distribution shifts.


title: DSAF: A Dual-Stage Adaptive Framework for Numerical Weather Prediction Downscaling

abstract: While widely recognized as one of the most substantial weather forecasting methodologies, Numerical Weather Prediction (NWP) usually suffers from relatively coarse resolution and inevitable bias due to tempo-spatial discretization, physical parametrization process, and computation limitation. With the roaring growth of deep learning-based techniques, we propose the Dual-Stage Adaptive Framework (DSAF), a novel framework to address regional NWP downscaling and bias correction tasks. DSAF uniquely incorporates adaptive elements in its design to ensure a flexible response to evolving weather conditions. Specifically, NWP downscaling and correction are well-decoupled in the framework and can be applied independently, which strategically guides the optimization trajectory of the model. Utilizing a multi-task learning mechanism and an uncertainty-weighted loss function, DSAF facilitates balanced training across various weather factors. Additionally, our specifically designed attention-centric learnable module effectively integrates geographic information, proficiently managing complex interrelationships. Experimental validation on the ECMWF operational forecast (HRES) and reanalysis (ERA5) archive demonstrates DSAF's superior performance over existing state-of-the-art models and shows substantial improvements when existing models are augmented using our proposed modules. Code is publicly available at https://github.com/pengwei07/DSAF.


title: When Graph Neural Network Meets Causality: Opportunities, Methodologies and An Outlook

abstract: Graph Neural Networks (GNNs) have emerged as powerful representation learning tools for capturing complex dependencies within diverse graph-structured data. Despite their success in a wide range of graph mining tasks, GNNs have raised serious concerns regarding their trustworthiness, including susceptibility to distribution shift, biases towards certain populations, and lack of explainability. Recently, integrating causal learning techniques into GNNs has sparked numerous ground-breaking studies since many GNN trustworthiness issues can be alleviated by capturing the underlying data causality rather than superficial correlations. In this survey, we comprehensively review recent research efforts on Causality-Inspired GNNs (CIGNNs). Specifically, we first employ causal tools to analyze the primary trustworthiness risks of existing GNNs, underscoring the necessity for GNNs to comprehend the causal mechanisms within graph data. Moreover, we introduce a taxonomy of CIGNNs based on the type of causal learning capability they are equipped with, i.e., causal reasoning and causal representation learning. Besides, we systematically introduce typical methods within each category and discuss how they mitigate trustworthiness risks. Finally, we summarize useful resources and discuss several future directions, hoping to shed light on new research opportunities in this emerging field. The representative papers, along with open-source data and codes, are available in https://github.com/usail-hkust/Causality-Inspired-GNNs.


title: ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval

abstract: The goal of Universal Cross-Domain Retrieval (UCDR) is to achieve robust performance in generalized test scenarios, wherein data may belong to strictly unknown domains and categories during training. Recently, pre-trained models with prompt tuning have shown strong generalization capabilities and attained noteworthy achievements in various downstream tasks, such as few-shot learning and video-text retrieval. However, applying them directly to UCDR may not sufficiently to handle both domain shift (i.e., adapting to unfamiliar domains) and semantic shift (i.e., transferring to unknown categories). To this end, we propose \textbf{Pro}mpting-to-\textbf{S}imulate (ProS), the first method to apply prompt tuning for UCDR. ProS employs a two-step process to simulate Content-aware Dynamic Prompts (CaDP) which can impact models to produce generalized features for UCDR. Concretely, in Prompt Units Learning stage, we introduce two Prompt Units to individually capture domain and semantic knowledge in a mask-and-align way. Then, in Context-aware Simulator Learning stage, we train a Content-aware Prompt Simulator under a simulated test scenarios to produce the corresponding CaDP. Extensive experiments conducted on three benchmark datasets show that our method achieves new state-of-the-art performance without bringing excessive parameters. Our method is publicly available at https://github.com/fangkaipeng/ProS.


title: Zero-shot Building Attribute Extraction from Large-Scale Vision and Language Models

abstract: Existing building recognition methods, exemplified by BRAILS, utilize supervised learning to extract information from satellite and street-view images for classification and segmentation. However, each task module requires human-annotated data, hindering the scalability and robustness to regional variations and annotation imbalances. In response, we propose a new zero-shot workflow for building attribute extraction that utilizes large-scale vision and language models to mitigate reliance on external annotations. The proposed workflow contains two key components: image-level captioning and segment-level captioning for the building images based on the vocabularies pertinent to structural and civil engineering. These two components generate descriptive captions by computing feature representations of the image and the vocabularies, and facilitating a semantic match between the visual and textual representations. Consequently, our framework offers a promising avenue to enhance AI-driven captioning for building attribute extraction in the structural and civil engineering domains, ultimately reducing reliance on human annotations while bolstering performance and adaptability.


title: Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation

abstract: Continual Test-Time Adaptation (CTTA) is proposed to migrate a source pre-trained model to continually changing target distributions, addressing real-world dynamism. Existing CTTA methods mainly rely on entropy minimization or teacher-student pseudo-labeling schemes for knowledge extraction in unlabeled target domains. However, dynamic data distributions cause miscalibrated predictions and noisy pseudo-labels in existing self-supervised learning methods, hindering the effective mitigation of error accumulation and catastrophic forgetting problems during the continual adaptation process. To tackle these issues, we propose a continual self-supervised method, Adaptive Distribution Masked Autoencoders (ADMA), which enhances the extraction of target domain knowledge while mitigating the accumulation of distribution shifts. Specifically, we propose a Distribution-aware Masking (DaM) mechanism to adaptively sample masked positions, followed by establishing consistency constraints between the masked target samples and the original target samples. Additionally, for masked tokens, we utilize an efficient decoder to reconstruct a hand-crafted feature descriptor (e.g., Histograms of Oriented Gradients), leveraging its invariant properties to boost task-relevant representations. Through conducting extensive experiments on four widely recognized benchmarks, our proposed method attains state-of-the-art performance in both classification and segmentation CTTA tasks. Our project page: https://sites.google.com/view/continual-mae/home.


title: Unveiling Spaces: Architecturally meaningful semantic descriptions from images of interior spaces

abstract: There has been a growing adoption of computer vision tools and technologies in architectural design workflows over the past decade. Notable use cases include point cloud generation, visual content analysis, and spatial awareness for robotic fabrication. Multiple image classification, object detection, and semantic pixel segmentation models have become popular for the extraction of high-level symbolic descriptions and semantic content from two-dimensional images and videos. However, a major challenge in this regard has been the extraction of high-level architectural structures (walls, floors, ceilings windows etc.) from diverse imagery where parts of these elements are occluded by furniture, people, or other non-architectural elements. This project aims to tackle this problem by proposing models that are capable of extracting architecturally meaningful semantic descriptions from two-dimensional scenes of populated interior spaces. 1000 virtual classrooms are parametrically generated, randomized along key spatial parameters such as length, width, height, and door/window positions. The positions of cameras, and non-architectural visual obstructions (furniture/objects) are also randomized. A Generative Adversarial Network (GAN) for image-to-image translation (Pix2Pix) is trained on synthetically generated rendered images of these enclosures, along with corresponding image abstractions representing high-level architectural structure. The model is then tested on unseen synthetic imagery of new enclosures, and outputs are compared to ground truth using pixel-wise comparison for evaluation. A similar model evaluation is also carried out on photographs of existing indoor enclosures, to measure its performance in real-world settings.


title: New Horizons: Pioneering Pharmaceutical R&D with Generative AI from lab to the clinic -- an industry perspective

abstract: The rapid advance of generative AI is reshaping the strategic vision for R&D across industries. The unique challenges of pharmaceutical R&D will see applications of generative AI deliver value along the entire value chain from early discovery to regulatory approval. This perspective reviews these challenges and takes a three-horizon approach to explore the generative AI applications already delivering impact, the disruptive opportunities which are just around the corner, and the longer-term transformation which will shape the future of the industry. Selected applications are reviewed for their potential to drive increase productivity, accelerate timelines, improve the quality of research, data and decision making, and support a sustainable future for the industry. Recommendations are given for Pharma R&D leaders developing a generative AI strategy today which will lay the groundwork for getting real value from the technology and safeguarding future growth. Generative AI is today providing new, efficient routes to accessing and combining organisational data to drive productivity. Next, this impact will reach clinical development, enhancing the patient experience, driving operational efficiency, and unlocking digital innovation to better tackle the future burden of disease. Looking to the furthest horizon, rapid acquisition of rich multi-omics data, which capture the 'language of life', in combination with next generation AI technologies will allow organisations to close the loop around phases of the pipeline through rapid, automated generation and testing of hypotheses from bench to bedside. This provides a vision for the future of R&D with sustainability at the core, with reduced timescales and reduced dependency on resources, while offering new hope to patients to treat the untreatable and ultimately cure diseases.


title: SCoTTi: Save Computation at Training Time with an adaptive framework

abstract: On-device training is an emerging approach in machine learning where models are trained on edge devices, aiming to enhance privacy protection and real-time performance. However, edge devices typically possess restricted computational power and resources, making it challenging to perform computationally intensive model training tasks. Consequently, reducing resource consumption during training has become a pressing concern in this field. To this end, we propose SCoTTi (Save Computation at Training Time), an adaptive framework that addresses the aforementioned challenge. It leverages an optimizable threshold parameter to effectively reduce the number of neuron updates during training which corresponds to a decrease in memory and computation footprint. Our proposed approach demonstrates superior performance compared to the state-of-the-art methods regarding computational resource savings on various commonly employed benchmarks and popular architectures, including ResNets, MobileNet, and Swin-T.


title: SkyMask: Attack-agnostic Robust Federated Learning with Fine-grained Learnable Masks

abstract: Federated Learning (FL) is becoming a popular paradigm for leveraging distributed data and preserving data privacy. However, due to the distributed characteristic, FL systems are vulnerable to Byzantine attacks that compromised clients attack the global model by uploading malicious model updates. Most existing Byzantine-robust FL systems statistically analyze the weights of whole individual model updates uploaded by clients to defend against Byzantine attacks. With the development of layer-level and parameter-level fine-grained attacks, the attacks' stealthiness and effectiveness have been significantly improved. Due to unawareness or overreaction, the existing model-level defense methods degrade the training efficiency and model performance. To address this problem, we propose SkyMask, a new attack-agnostic robust FL system that leverages fine-grained learnable masks to identify malicious model updates at the parameter-level. Specifically, the FL server applies parameter-level masks to model updates uploaded by clients and trains the masks over a small clean dataset (i.e., root dataset) to learn the subtle difference between benign and malicious model updates in a high-dimension space. Our extensive experiments involve different models on three public datasets under state-of-the-art (SOTA) attacks, where the results show that SkyMask achieves up to 10% higher testing accuracy compared with SOTA defense strategies and successfully defends against attacks with malicious clients of a high fraction up to 80%. In the meantime, the experimental results demonstrate the scalability of our approach and the weak dependence on the data distribution of the root dataset.


title: Learning Deterministic Surrogates for Robust Convex QCQPs

abstract: Decision-focused learning is a promising development for contextual optimisation. It enables us to train prediction models that reflect the contextual sensitivity structure of the problem. However, there have been limited attempts to extend this paradigm to robust optimisation. We propose a double implicit layer model for training prediction models with respect to robust decision loss in uncertain convex quadratically constrained quadratic programs (QCQP). The first layer solves a deterministic version of the problem, the second layer evaluates the worst case realisation for an uncertainty set centred on the observation given the decisions obtained from the first layer. This enables us to learn model parameterisations that lead to robust decisions while only solving a simpler deterministic problem at test time. Additionally, instead of having to solve a robust counterpart we solve two smaller and potentially easier problems in training. The second layer (worst case problem) can be seen as a regularisation approach for predict-and-optimise by fitting to a neighbourhood of problems instead of just a point observation. We motivate relaxations of the worst-case problem in cases of uncertainty sets that would otherwise lead to trust region problems, and leverage various relaxations to deal with uncertain constraints. Both layers are typically strictly convex in this problem setting and thus have meaningful gradients almost everywhere. We demonstrate an application of this model on simulated experiments. The method is an effective regularisation tool for decision-focused learning for uncertain convex QCQPs.


title: Vision-Based Automatic Groceries Tracking System -- Smart Homes

abstract: With advanced AI, while every industry is growing at rocket speed, the smart home industry has not reached the next generation. There is still a huge leap of innovation that needs to happen before we call a home a Smart home. A Smart home should predict residents' needs and fulfill them in a timely manner. One of the important tasks of maintaining a home is timely grocery tracking and supply maintenance. Grocery tracking models are very famous in the retail industry but they are nonexistent in the common household. Groceries detection in household refrigerators or storage closets is very complicated compared to retail shelving data. In this paper, home grocery tracking problem is resolved by combining retail shelving data and fruits dataset with real-time 360 view data points collected from home groceries storage. By integrating this vision-based object detection system along with supply chain and user food interest prediction systems, complete automation of groceries ordering can be achieved.


title: Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models

abstract: This paper presents a comprehensive study on the role of Classifier-Free Guidance (CFG) in text-conditioned diffusion models from the perspective of inference efficiency. In particular, we relax the default choice of applying CFG in all diffusion steps and instead search for efficient guidance policies. We formulate the discovery of such policies in the differentiable Neural Architecture Search framework. Our findings suggest that the denoising steps proposed by CFG become increasingly aligned with simple conditional steps, which renders the extra neural network evaluation of CFG redundant, especially in the second half of the denoising process. Building upon this insight, we propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively omits network evaluations when the denoising process displays convergence. Our experiments demonstrate that AG preserves CFG's image quality while reducing computation by 25%. Thus, AG constitutes a plug-and-play alternative to Guidance Distillation, achieving 50% of the speed-ups of the latter while being training-free and retaining the capacity to handle negative prompts. Finally, we uncover further redundancies of CFG in the first half of the diffusion process, showing that entire neural function evaluations can be replaced by simple affine transformations of past score estimates. This method, termed LinearAG, offers even cheaper inference at the cost of deviating from the baseline model. Our findings provide insights into the efficiency of the conditional denoising process that contribute to more practical and swift deployment of text-conditioned diffusion models.


title: Foreseeing Reconstruction Quality of Gradient Inversion: An Optimization Perspective

abstract: Gradient inversion attacks can leak data privacy when clients share weight updates with the server in federated learning (FL). Existing studies mainly use L2 or cosine distance as the loss function for gradient matching in the attack. Our empirical investigation shows that the vulnerability ranking varies with the loss function used. Gradient norm, which is commonly used as a vulnerability proxy for gradient inversion attack, cannot explain this as it remains constant regardless of the loss function for gradient matching. In this paper, we propose a loss-aware vulnerability proxy (LAVP) for the first time. LAVP refers to either the maximum or minimum eigenvalue of the Hessian with respect to gradient matching loss at ground truth. This suggestion is based on our theoretical findings regarding the local optimization of the gradient inversion in proximity to the ground truth, which corresponds to the worst case attack scenario. We demonstrate the effectiveness of LAVP on various architectures and datasets, showing its consistent superiority over the gradient norm in capturing sample vulnerabilities. The performance of each proxy is measured in terms of Spearman's rank correlation with respect to several similarity scores. This work will contribute to enhancing FL security against any potential loss functions beyond L2 or cosine distance in the future.