title: Effective Brane Field Theory with Higher-form Symmetry

abstract: We propose an effective field theory for branes with higher-form symmetry as a generalization of ordinary Landau theory, which is an extension of the previous work by Iqbal and McGreevy for one-dimensional objects to an effective theory for $p$-dimensional objects. In the case of a $p$-form symmetry, the fundamental field $\psi[C_p^{}]$ is a functional of $p$-dimensional closed brane $C_p^{}$ embedded in a spacetime. As a natural generalization of ordinary field theory, we call this theory the brane field theory. In order to construct an action that is invariant under higher-form transformation, we generalize the idea of area derivative for one-dimensional objects to higher-dimensional ones. Following this, we discuss various fundamental properties of the brane field based on the higher-form invariant action. It is shown that the classical solution exhibits the area law in the unbroken phase of $\mathrm{U}(1)$ $p$-form symmetry, while it indicates a constant behavior in the broken phase for the large volume limit of $C_p^{}$. In the latter case, the low-energy effective theory is described by the $p$-form Maxwell theory. We also discuss brane-field theories with a discrete higher-form symmetry and show that the low-energy effective theory becomes a BF-type topological field theory, resulting in topological order. Finally, we present a concrete brane-field model that describes a superconductor from the point of view of higher-form symmetry.


title: On the Capacity of Reconfigurable Intelligence Surface: the Sparse Channel Case

abstract: Reconfigurable intelligent surface (RIS) is an important candidate technology for 6G. We provide an analysis of RIS-assisted MIMO communication in sparse channel typically found in the mmW or THz range. By exploring the sparse property, we maximize the capacity in the singular space of the channel and developed efficient algorithms for SU-MIMO or DL MU-MIMO. We also proved it is more difficult to support high rank transmission in the RIS reflection channel than in the traditional MIMO channel.


title: HeightFormer: A Multilevel Interaction and Image-adaptive Classification-regression Network for Monocular Height Estimation with Aerial Images

abstract: Height estimation has long been a pivotal topic within measurement and remote sensing disciplines, proving critical for endeavours such as 3D urban modelling, MR and autonomous driving. Traditional methods utilise stereo matching or multisensor fusion, both well-established techniques that typically necessitate multiple images from varying perspectives and adjunct sensors like SAR, leading to substantial deployment costs. Single image height estimation has emerged as an attractive alternative, boasting a larger data source variety and simpler deployment. However, current methods suffer from limitations such as fixed receptive fields, a lack of global information interaction, leading to noticeable instance-level height deviations. The inherent complexity of height prediction can result in a blurry estimation of object edge depth when using mainstream regression methods based on fixed height division. This paper presents a comprehensive solution for monocular height estimation in remote sensing, termed HeightFormer, combining multilevel interactions and image-adaptive classification-regression. It features the Multilevel Interaction Backbone (MIB) and Image-adaptive Classification-regression Height Generator (ICG). MIB supplements the fixed sample grid in CNN of the conventional backbone network with tokens of different interaction ranges. It is complemented by a pixel-, patch-, and feature map-level hierarchical interaction mechanism, designed to relay spatial geometry information across different scales and introducing a global receptive field to enhance the quality of instance-level height estimation. The ICG dynamically generates height partition for each image and reframes the traditional regression task, using a refinement from coarse to fine classification-regression that significantly mitigates the innate ill-posedness issue and drastically improves edge sharpness.


title: Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learning

abstract: This work identifies a simple pre-training mechanism that leads to representations exhibiting better continual and transfer learning. This mechanism -- the repeated resetting of weights in the last layer, which we nickname "zapping" -- was originally designed for a meta-continual-learning procedure, yet we show it is surprisingly applicable in many settings beyond both meta-learning and continual learning. In our experiments, we wish to transfer a pre-trained image classifier to a new set of classes, in a few shots. We show that our zapping procedure results in improved transfer accuracy and/or more rapid adaptation in both standard fine-tuning and continual learning settings, while being simple to implement and computationally efficient. In many cases, we achieve performance on par with state of the art meta-learning without needing the expensive higher-order gradients, by using a combination of zapping and sequential learning. An intuitive explanation for the effectiveness of this zapping procedure is that representations trained with repeated zapping learn features that are capable of rapidly adapting to newly initialized classifiers. Such an approach may be considered a computationally cheaper type of, or alternative to, meta-learning rapidly adaptable features with higher-order gradients. This adds to recent work on the usefulness of resetting neural network parameters during training, and invites further investigation of this mechanism.


title: PG-NeuS: Robust and Efficient Point Guidance for Multi-View Neural Surface Reconstruction

abstract: Recently, learning multi-view neural surface reconstruction with the supervision of point clouds or depth maps has been a promising way. However, due to the underutilization of prior information, current methods still struggle with the challenges of limited accuracy and excessive time complexity. In addition, prior data perturbation is also an important but rarely considered issue. To address these challenges, we propose a novel point-guided method named PG-NeuS, which achieves accurate and efficient reconstruction while robustly coping with point noise. Specifically, aleatoric uncertainty of the point cloud is modeled to capture the distribution of noise, leading to noise robustness. Furthermore, a Neural Projection module connecting points and images is proposed to add geometric constraints to implicit surface, achieving precise point guidance. To better compensate for geometric bias between volume rendering and point modeling, high-fidelity points are filtered into a Bias Network to further improve details representation. Benefiting from the effective point guidance, even with a lightweight network, the proposed PG-NeuS achieves fast convergence with an impressive 11x speedup compared to NeuS. Extensive experiments show that our method yields high-quality surfaces with high efficiency, especially for fine-grained details and smooth regions, outperforming the state-of-the-art methods. Moreover, it exhibits strong robustness to noisy data and sparse data.


title: A Novel Statistical Measure for Out-of-Distribution Detection in Data Quality Assurance

abstract: Data outside the problem domain poses significant threats to the security of AI-based intelligent systems. Aiming to investigate the data domain and out-of-distribution (OOD) data in AI quality management (AIQM) study, this paper proposes to use deep learning techniques for feature representation and develop a novel statistical measure for OOD detection. First, to extract low-dimensional representative features distinguishing normal and OOD data, the proposed research combines the deep auto-encoder (AE) architecture and neuron activation status for feature engineering. Then, using local conditional probability (LCP) in data reconstruction, a novel and superior statistical measure is developed to calculate the score of OOD detection. Experiments and evaluations are conducted on image benchmark datasets and an industrial dataset. Through comparative analysis with other common statistical measures in OOD detection, the proposed research is validated as feasible and effective in OOD and AIQM studies.


title: LEMON: Lossless model expansion

abstract: Scaling of deep neural networks, especially Transformers, is pivotal for their surging performance and has further led to the emergence of sophisticated reasoning capabilities in foundation models. Such scaling generally requires training large models from scratch with random initialization, failing to leverage the knowledge acquired by their smaller counterparts, which are already resource-intensive to obtain. To tackle this inefficiency, we present $\textbf{L}$ossl$\textbf{E}$ss $\textbf{MO}$del Expansio$\textbf{N}$ (LEMON), a recipe to initialize scaled models using the weights of their smaller but pre-trained counterparts. This is followed by model training with an optimized learning rate scheduler tailored explicitly for the scaled models, substantially reducing the training time compared to training from scratch. Notably, LEMON is versatile, ensuring compatibility with various network structures, including models like Vision Transformers and BERT. Our empirical results demonstrate that LEMON reduces computational costs by 56.7% for Vision Transformers and 33.2% for BERT when compared to training from scratch.


title: Scaling relations in quasi-static magnetoconvection with a strong vertical magnetic field

abstract: The scaling law for the horizontal length scale $\ell$ relative to the domain height $L$, originating from the linear theory of quasi-static magnetoconvection, $\ell/L \sim Q^{-1/6}$, has been verified through two-dimensional (2D) direct numerical simulation (DNS), particularly at high values of the Chandrasekhar number ($Q$). This relationship remains valid within a specific flow regime characterized by columnar structures aligned with the magnetic field. Expanding upon the $Q$-dependence of the horizontal length scale, we have derived scaling laws for the Nusselt number $Nu$ and the Reynolds number $Re$ as functions of the driving forces (Rayleigh number $Ra$ and $Q$) in quasi-static magnetoconvection influenced by a strong magnetic field. These scaling relations, $Nu \sim Ra/Q$ and $Re \sim Ra Q^{-5/6}$, have been successfully validated using 2D DNS data spanning a wide range of five decades in $Q$, ranging from $10^5$ to $10^9$. The successful validation of the relations at large $Q$ values, combined with our theoretical analysis of dissipation rates and the incorporation of the horizontal length scale's influence on scaling behavior, presents a \textcolor{black}{valid} approach for deriving scaling laws under various conditions.


title: Performance Analysis of Vertical Axis Wind Turbine Clusters: Effect of Inter-Turbine Spacing and Turbine Rotation

abstract: Wind energy has emerged as a viable alternative to fossil fuels, with vertical axis wind turbines (VAWTs) gaining popularity due to their efficiency and adaptability. Combining the Actuator Line Method (ALM) with Large Eddy Simulation (LES) enables accurate performance evaluations, facilitating the design and optimization of wind turbines. The present study invokes ALM-based methodology to perform calculations for the VAWTs. The results of the LES simulations of the VAWTs have been extensively validated against the available experimental and numerical data. The study further explores a VAWT cluster of three turbines by investigating the influence of turbine spacing (in both in-line and staggered configuration) on cluster performance. The study shows that the configuration with a streamwise separation (Xsep) of 0.34D and a transverse separation (Ysep) of 2.5D exhibits superior performance to other combinations owing to increased kinetic energy in the wake for the downstream turbines. Further, we have presented the effect of varying the rotation direction (in combinations of Clockwise and Counter-Clockwise rotation) for the individual turbines in the 3-turbine cluster for the two configurations: in-line (Xsep = 0D, Ysep = 2.5D) and staggered (Xsep = 0.34D, Ysep = 2.5D). Staggered counter-rotating turbine cases show reduced performance compared to co-rotating cases, specifically, the clockwise co-rotating (C-C-C) configuration. In the in-line configuration, counter-rotating setups outperform co-rotating ones. Counter-rotation analysis reveals that reducing streamwise separation allows turbines to align in line without sacrificing performance, thereby increasing the power density of the turbine cluster


title: MLP-AMDC: An MLP Architecture for Adaptive-Mask-based Dual-Camera snapshot hyperspectral imaging

abstract: Coded Aperture Snapshot Spectral Imaging (CASSI) system has great advantages over traditional methods in dynamically acquiring Hyper-Spectral Image (HSI), but there are the following problems. 1) Traditional mask relies on random patterns or analytical design, both of which limit the performance improvement of CASSI. 2) Existing high-quality reconstruction algorithms are slow in reconstruction and can only reconstruct scene information offline. To address the above two problems, this paper designs the AMDC-CASSI system, introducing RGB camera with CASSI based on Adaptive-Mask as multimodal input to improve the reconstruction quality. The existing SOTA reconstruction schemes are based on transformer, but the operation of self-attention pulls down the operation efficiency of the network. In order to improve the inference speed of the reconstruction network, this paper proposes An MLP Architecture for Adaptive-Mask-based Dual-Camera (MLP-AMDC) to replace the transformer structure of the network. Numerous experiments have shown that MLP performs no less well than transformer-based structures for HSI reconstruction, while MLP greatly improves the network inference speed and has less number of parameters and operations, our method has a 8 db improvement over SOTA and at least a 5-fold improvement in reconstruction speed. (https://github.com/caizeyu1992/MLP-AMDC.)


title: Giant Acceleration of Diffusion in Soft Matter Potential

abstract: Diffusion of Brownian particles in the tilted periodic potential, usually referred to the washboard potential (WBP), is a well-known model to describe physical systems out of equilibrium. Considering that the biological medium is flexible and thermally fluctuating, a new model, namely the soft matter potential (SMP), is proposed to describe the biological medium. Compared to the washboard potential (WBP), SMP allows Brownian particles to actively modify the structure of the biological medium. Brenner's homogenization theory is applied to predict the diffusivity and velocity of Brownian particles driven by external forces in SMP. Thermodynamic uncertainty relation (TUR) is analyzed for Brownian particles in SMP. It is found that, compared to WBP, Brownian particles in SMP require a lower energy cost $\langle q \rangle$ to achieve accuracy $\mathcal{A}$, i.e. Brownian particles in SMP have higher transport efficiency when driven by external forces.


title: On the Rational Degree of Boolean Functions and Applications

abstract: We study a natural complexity measure of Boolean functions known as the (exact) rational degree. For total functions $f$, it is conjectured that $\mathrm{rdeg}(f)$ is polynomially related to $\mathrm{deg}(f)$, where $\mathrm{deg}(f)$ is the Fourier degree. Towards this conjecture, we show that symmetric functions have rational degree at least $\mathrm{deg}(f)/2$ and monotone functions have rational degree at least $\sqrt{\mathrm{deg}(f)}$. We observe that both of these lower bounds are tight. In addition, we show that all read-once depth-$d$ Boolean formulae have rational degree at least $\Omega(\mathrm{deg}(f)^{1/d})$. Furthermore, we show that almost every Boolean function on $n$ variables has rational degree at least $n/2 - O(\sqrt{n})$. In contrast to total functions, we exhibit partial functions that witness unbounded separations between rational and approximate degree, in both directions. As a consequence, we show that for quantum computers, post-selection and bounded-error are incomparable resources in the black-box model.


title: Uniqueness of blowups for forced mean curvature flow

abstract: We prove uniqueness of tangent cones for forced mean curvature flow, at both closed self-shrinkers and round cylindrical self-shrinkers, in any codimension. The corresponding results for mean curvature flow in Euclidean space were proven by Schulze and Colding-Minicozzi respectively. We adapt their methods to handle the presence of the forcing term, which vanishes in the blow-up limit but complicates the analysis along the rescaled flow. Our results naturally include the case of mean curvature flows in Riemannian manifolds.


title: MCPNS: A Macropixel Collocated Position and Its Neighbors Search for Plenoptic 2.0 Video Coding

abstract: Recently, it was demonstrated that a newly focused plenoptic 2.0 camera can capture much higher spatial resolution owing to its effective light field sampling, as compared to a traditional unfocused plenoptic 1.0 camera. However, due to the nature difference of the optical structure between the plenoptic 1.0 and 2.0 cameras, the existing fast motion estimation (ME) method for plenoptic 1.0 videos is expected to be sub-optimal for encoding plenoptic 2.0 videos. In this paper, we point out the main motion characteristic differences between plenoptic 1.0 and 2.0 videos and then propose a new fast ME, called macropixel collocated position and its neighbors search (MCPNS) for plenoptic 2.0 videos. In detail, we propose to reduce the number of macropixel collocated position (MCP) search candidates based on the new observation of center-biased motion vector distribution at macropixel resolution. After that, due to large motion deviation behavior around each MCP location in plenoptic 2.0 videos, we propose to select a certain number of key MCP locations with the lowest matching cost to perform the neighbors MCP search to improve the motion search accuracy. Different from existing methods, our method can achieve better performance without requiring prior knowledge of microlens array orientations. Our simulation results confirmed the effectiveness of the proposed algorithm in terms of both bitrate savings and computational costs compared to existing methods.


title: Scanning quantum correlation microscopy with few emitters

abstract: Optical superresolution microscopy is an important field, where nonlinear optical processes or prior information is used to defeat the classical diffraction limit of light. Quantum correlation microscopy uses photon arrival statistics from single photon emitters to aid in the determination of properties including the number of emitters and their relative brightness. Here we model quantum correlation microscopy in the few emitter regime, i.e. around four single photon emitters below the diffraction limit. We use the Akaike Information Criterion to determine the number of emitters and we vary the relative contributions of intensity to quantum correlation information to determine contribution that provides optimal imaging. Our results show diffraction unlimited performance and a change in localisation scaling behaviour dependant on emitter closeness.


title: Effects of Human Adversarial and Affable Samples on BERT Generalization

abstract: BERT-based models have had strong performance on leaderboards, yet have been demonstrably worse in real-world settings requiring generalization. Limited quantities of training data is considered a key impediment to achieving generalizability in machine learning. In this paper, we examine the impact of training data quality, not quantity, on a model's generalizability. We consider two characteristics of training data: the portion of human-adversarial (h-adversarial), i.e., sample pairs with seemingly minor differences but different ground-truth labels, and human-affable (h-affable) training samples, i.e., sample pairs with minor differences but the same ground-truth label. We find that for a fixed size of training samples, as a rule of thumb, having 10-30% h-adversarial instances improves the precision, and therefore F1, by up to 20 points in the tasks of text classification and relation extraction. Increasing h-adversarials beyond this range can result in performance plateaus or even degradation. In contrast, h-affables may not contribute to a model's generalizability and may even degrade generalization performance.


title: Dual-Stream Knowledge-Preserving Hashing for Unsupervised Video Retrieval

abstract: Unsupervised video hashing usually optimizes binary codes by learning to reconstruct input videos. Such reconstruction constraint spends much effort on frame-level temporal context changes without focusing on video-level global semantics that are more useful for retrieval. Hence, we address this problem by decomposing video information into reconstruction-dependent and semantic-dependent information, which disentangles the semantic extraction from reconstruction constraint. Specifically, we first design a simple dual-stream structure, including a temporal layer and a hash layer. Then, with the help of semantic similarity knowledge obtained from self-supervision, the hash layer learns to capture information for semantic retrieval, while the temporal layer learns to capture the information for reconstruction. In this way, the model naturally preserves the disentangled semantics into binary codes. Validated by comprehensive experiments, our method consistently outperforms the state-of-the-arts on three video benchmarks.


title: Triple points and phase transitions of D-dimensional dyonic AdS black holes with quasitopological electromagnetism in Einstein-Gauss-Bonnet gravity

abstract: By considering the negative cosmological constant {\Lambda} as a thermodynamic pressure, we study the thermodynamics and phase transitions of the D-dimensional dyonic AdS black holes (BHs) with quasitopological electromagnetism in Einstein-Gauss-Bonnet (EGB) gravity. The results indicate that the small/large BH phase transition that is similar to the van der Waals (vdW) liquid/gas phase transition always exist for any space-time dimensions. But interestingly, we then find that this BH system exhibits a more complex phase structure in 6-dimensional case that missing in other dimensions. Specifically, it shows for D = 6 that we observed the small/intermediate/large BH phase transitions in a specific parameter region with the triple point naturally appeared. However, for the dyonic AdS BHs with quasitopological electromagnetism in Einstein gravity, the novel phase structure composed of two separate coexistence curves observed in [1] disappeared in EGB gravity. In addition, it is also true that the critical exponents calculated near the critical points possess identical values as mean field theory. Finally, we conclude that these findings shall provide some deep insights into the intriguing thermodynamic properties of the dyonic AdS BHs with quasitopological electromagnetism in EGB gravity.


title: A Study of Rarely Appeared Instructions in an Executable Binary

abstract: An executable binary typically contains a large number of machine instructions. Although the statistics of popular instructions is well known, the distribution of non-popular instructions has been relatively under explored. Our finding shows that an arbitrary group of binaries com es with both i) a similar distribution of common machine instructions, and ii) quite a few rarely appeared instructions (e.g., less than five occurrences) apart from the distribution. Their infrequency may represent the signature of a code chunk or the footprint of a binary. In this work, we investigate such rare instructions with an in-depth analysis at the source level, clas sifying them into four categories.


title: AutoFHE: Automated Adaption of CNNs for Efficient Evaluation over FHE

abstract: Secure inference of deep convolutional neural networks (CNNs) under RNS-CKKS involves polynomial approximation of unsupported non-linear activation functions. However, existing approaches have three main limitations: 1) Inflexibility: The polynomial approximation and associated homomorphic evaluation architecture are customized manually for each CNN architecture and do not generalize to other networks. 2) Suboptimal Approximation: Each activation function is approximated instead of the function represented by the CNN. 3) Restricted Design: Either high-degree or low-degree polynomial approximations are used. The former retains high accuracy but slows down inference due to bootstrapping operations, while the latter accelerates ciphertext inference but compromises accuracy. To address these limitations, we present AutoFHE, which automatically adapts standard CNNs for secure inference under RNS-CKKS. The key idea is to adopt layerwise mixed-degree polynomial activation functions, which are optimized jointly with the homomorphic evaluation architecture in terms of the placement of bootstrapping operations. The problem is modeled within a multi-objective optimization framework to maximize accuracy and minimize the number of bootstrapping operations. AutoFHE can be applied flexibly on any CNN architecture, and it provides diverse solutions that span the trade-off between accuracy and latency. Experimental evaluation over RNS-CKKS encrypted CIFAR datasets shows that AutoFHE accelerates secure inference by $1.32\times$ to $1.8\times$ compared to methods employing high-degree polynomials. It also improves accuracy by up to 2.56% compared to methods using low-degree polynomials. Lastly, AutoFHE accelerates inference and improves accuracy by $103\times$ and 3.46%, respectively, compared to CNNs under TFHE.


title: The Newlander-Nirenberg theorem for complex $b$-manifolds

abstract: Melrose defined the $b$-tangent bundle of a smooth manifold $M$ with boundary as the vector bundle whose sections are vector fields on $M$ tangent to the boundary. Mendoza defined a complex $b$-manifold as a manifold with boundary together with an involutive splitting of the complexified $b$-tangent bundle into complex conjugate factors. In this article, we prove complex $b$-manifolds have a single local model depending only on dimension. This can be thought of as the Newlander-Nirenberg theorem for complex $b$-manifolds: there are no ``local invariants''. Our proof uses Mendoza's existing result that complex $b$-manifolds do not have ``formal local invariants'' and a singular coordinate change trick to leverage the classical Newlander-Nirenberg theorem.


title: On Automorphisms of Complex $b^k$-Manifolds

abstract: The $b$-calculus of Melrose is a tool for studying structures on a smooth manifold with a first order degeneracy at a given hypersurface. In this framework, Mendoza defined complex $b$-manifolds. In the spirit of work of Scott, we extend Mendoza's definition to the case of higher-order degeneracies, introducing the notion of a complex $b^k$-manifold for $k$ a positive integer. We then investigate the local and global automorphisms of complex $b^k$-manifolds. We also propose $b^k$-analogues for some classical spaces of holomorphic functions.


title: Why Train More? Effective and Efficient Membership Inference via Memorization

abstract: Membership Inference Attacks (MIAs) aim to identify specific data samples within the private training dataset of machine learning models, leading to serious privacy violations and other sophisticated threats. Many practical black-box MIAs require query access to the data distribution (the same distribution where the private data is drawn) to train shadow models. By doing so, the adversary obtains models trained "with" or "without" samples drawn from the distribution, and analyzes the characteristics of the samples under consideration. The adversary is often required to train more than hundreds of shadow models to extract the signals needed for MIAs; this becomes the computational overhead of MIAs. In this paper, we propose that by strategically choosing the samples, MI adversaries can maximize their attack success while minimizing the number of shadow models. First, our motivational experiments suggest memorization as the key property explaining disparate sample vulnerability to MIAs. We formalize this through a theoretical bound that connects MI advantage with memorization. Second, we show sample complexity bounds that connect the number of shadow models needed for MIAs with memorization. Lastly, we confirm our theoretical arguments with comprehensive experiments; by utilizing samples with high memorization scores, the adversary can (a) significantly improve its efficacy regardless of the MIA used, and (b) reduce the number of shadow models by nearly two orders of magnitude compared to state-of-the-art approaches.


title: What can we learn from the experiment of electrostatic conveyor belt for excitons?

abstract: Motivated by the experiment of electrostatic conveyor belt for indirect excitons [A. G. Winbow, \textit{et al.}, Phys. Rev. Lett. \textbf{106}, 196806 (2011)], we study the exciton patterns for understanding the exciton dynamics. By analyzing the exciton diffusion, we find that the patterns mainly come from the photoluminescence of two kinds of excitons. The patterns near the laser spot come from the hot excitons which can be regarded as the classical particles. However, the patterns far from the laser spot come from the cooled excitons or coherent excitons. Taking into account of the finite lifetime of Bosonic excitons and of the interactions between them, we build a time-dependent nonlinear Schr\"{o}dinger equation including the non-Hermitian dissipation to describe the coherent exciton dynamics. The real-time and imaginary-time evolutions are used alternately to solve the Schr\"{o}dinger equation in order to simulate the exciton diffusion accompanied with the exciton cooling in the moving lattices. By calculating the escape probability, we obtain the transport distances of the coherent excitons in the conveyor which are consistent with the experimental data. The cooling speed of excitons is found to be important in the coherent exciton transport. Moreover, the plateau in the average transport distance cannot be explained by the dynamical localization-delocalization transition induced by the disorders.


title: Harnessing Large Language Models' Empathetic Response Generation Capabilities for Online Mental Health Counselling Support

abstract: Large Language Models (LLMs) have demonstrated remarkable performance across various information-seeking and reasoning tasks. These computational systems drive state-of-the-art dialogue systems, such as ChatGPT and Bard. They also carry substantial promise in meeting the growing demands of mental health care, albeit relatively unexplored. As such, this study sought to examine LLMs' capability to generate empathetic responses in conversations that emulate those in a mental health counselling setting. We selected five LLMs: version 3.5 and version 4 of the Generative Pre-training (GPT), Vicuna FastChat-T5, Pathways Language Model (PaLM) version 2, and Falcon-7B-Instruct. Based on a simple instructional prompt, these models responded to utterances derived from the EmpatheticDialogues (ED) dataset. Using three empathy-related metrics, we compared their responses to those from traditional response generation dialogue systems, which were fine-tuned on the ED dataset, along with human-generated responses. Notably, we discovered that responses from the LLMs were remarkably more empathetic in most scenarios. We position our findings in light of catapulting advancements in creating empathetic conversational systems.


title: Gromov-Witten Generating Series of Elliptic Curves and Iterated Integrals of Eisenstein-Kronecker Forms

abstract: We compute various types of iterated integrals of Eisenstein-Kronecker forms that are constructed from the Kronecker theta function. Furthermore, we relate the generating series of Gromov-Witten invariants of elliptic curves to these iterated integrals.


title: Robust 1-bit Compressed Sensing with Iterative Hard Thresholding

abstract: In 1-bit compressed sensing, the aim is to estimate a $k$-sparse unit vector $x\in S^{n-1}$ within an $\epsilon$ error (in $\ell_2$) from minimal number of linear measurements that are quantized to just their signs, i.e., from measurements of the form $y = \mathrm{Sign}(\langle a, x\rangle).$ In this paper, we study a noisy version where a fraction of the measurements can be flipped, potentially by an adversary. In particular, we analyze the Binary Iterative Hard Thresholding (BIHT) algorithm, a proximal gradient descent on a properly defined loss function used for 1-bit compressed sensing, in this noisy setting. It is known from recent results that, with $\tilde{O}(\frac{k}{\epsilon})$ noiseless measurements, BIHT provides an estimate within $\epsilon$ error. This result is optimal and universal, meaning one set of measurements work for all sparse vectors. In this paper, we show that BIHT also provides better results than all known methods for the noisy setting. We show that when up to $\tau$-fraction of the sign measurements are incorrect (adversarial error), with the same number of measurements as before, BIHT agnostically provides an estimate of $x$ within an $\tilde{O}(\epsilon+\tau)$ error, maintaining the universality of measurements. This establishes stability of iterative hard thresholding in the presence of measurement error. To obtain the result, we use the restricted approximate invertibility of Gaussian matrices, as well as a tight analysis of the high-dimensional geometry of the adversarially corrupted measurements.

categories: cs.IT cs.DS cs.LG eess.SP math.IT stat.ML


title: Assessing Copula Models for Mixed Continuous-Ordinal Variables

abstract: Vine pair-copula constructions exist for a mix of continuous and ordinal variables. In some steps, this can involve estimating a bivariate copula for a pair of mixed continuous-ordinal variables. To assess the adequacy of copula fits for such a pair, diagnostic and visualization methods based on normal score plots and conditional Q-Q plots are proposed. The former utilizes a latent continuous variable for the ordinal variable. Using the Kullback-Leibler divergence, existing probability models for mixed continuous-ordinal variable pair are assessed for the adequacy of fit with simple parametric copula families. The effectiveness of the proposed visualization and diagnostic methods is illustrated on simulated and real datasets.


title: Channel-robust Automatic Modulation Classification Using Spectral Quotient Cumulants

abstract: Automatic modulation classification (AMC) is to identify the modulation format of the received signal corrupted by the channel effects and noise. Most existing works focus on the impact of noise while relatively little attention has been paid to the impact of channel effects. However, the instability posed by multipath fading channels leads to significant performance degradation. To mitigate the adverse effects of the multipath channel, we propose a channel-robust modulation classification framework named spectral quotient cumulant classification (SQCC) for orthogonal frequency division multiplexing (OFDM) systems. Specifically, we first transform the received signal to the spectral quotient (SQ) sequence by spectral circular shift division operations. Secondly, an outlier detector is proposed to filter the outliers in the SQ sequence. At last, we extract spectral quotient cumulants (SQCs) from the filtered SQ sequence as the inputs to train the artificial neural network (ANN) classifier and use the trained ANN to make the final decisions. Simulation results show that our proposed SQCC method exhibits classification robustness and superiority under various unknown Rician multipath fading channels compared with other existing methods. Specifically, the SQCC method achieves nearly 90% classification accuracy at the signal to noise ratio (SNR) of 4dB when testing under multiple channels but training under AWGN channel.


title: Joshi--Malafarina--Narayan singularity in weak magnetic field

abstract: The importance and significance of magnetic fields in the astrophysical scenario is well known. Many domains of astrophysical black hole physics such as polarized shadow image, high energy emitting processes and jet formation are dependent on the behavior of the magnetic fields in the vicinity of the compact objects. In light of this, we determine the master equation and master differential equation that determine the spatial behavior of the magnetic field inside a matter distribution or vacuum region, of general spherically symmetric metric, which is immersed in a test magnetic field. We also investigate here the case of JMN-1 singularity immersed in a uniform weak magnetic field and determine the behavior of magnetic fields by defining electromagnetic four potential vector. We find that the tangential component of the magnetic field is discontinuous at the matching surface of the JMN-1 singularity with the external Schwarzschild metric, resulting in surface currents. We define the covariant expression of surface current density in this scenario. We also analyze the behavior of center-of-mass energy of two oppositely charged particles in the geometry of the magnetized JMN-1 singularity. We briefly discuss the possible scenarios which would possess a discontinuous magnetic field and implications of the same and future possibilities in the realm of astrophysics are indicated.


title: VERTICO and IllustrisTNG: The spatially resolved effects of environment on galactic gas

abstract: It has been shown in previous publications that the TNG100 simulation quantitatively reproduces the observed reduction in each of the total atomic and total molecular hydrogen gas for galaxies within massive halos, i.e.~dense environments. In this Letter, we study how well TNG50 reproduces the resolved effects of a Virgo-like cluster environment on the gas surface densities of satellite galaxies with $m_* > \! 10^9\,{\rm M}_\odot$ and ${\rm SFR} \! > 0.05\,{\rm M}_\odot\,{\rm yr}^{-1}$. We select galaxies in the simulation that are analogous to those in the HERACLES and VERTICO surveys, and mock-observe them to the common specifications of the data. Although TNG50 does not quantitatively match the observed gas surface densities in the centers of galaxies, the simulation does qualitatively reproduce the trends of gas truncation and central density suppression seen in VERTICO in both HI and H$_2$. This result promises that modern cosmological hydrodynamic simulations can be used to reliably model the post-infall histories of cluster satellite galaxies.


title: Effective potential engineering by emergent anisotropy in a tunable open-access microcavity

abstract: Photonic spin-orbit (SO) coupling is an important physical mechanism leading to numerous interesting phenomena in the systems of microcavity photons and exciton-polaritons. We report the effect of SO coupling in a tunable open-access microcavity embedded with anisotropic active media. The SO coupling associated with the TE-TM splitting results in an emergent anisotropy, which further leads to fine energy splittings allowing clear observation of the full set of eigenstates, in sharp contrast with the isotropic situation which leads to the isotropic eigenstates of spin vortices. We show that the photonic potential can be engineered by playing with the relation between the emergent anisotropy and the cavity ellipticity. All the experimental results are well reproduced by the degenerate perturbation theory. Our results constitute a significant extension to the research field of microcavity spinoptronics, with potential applications in polarization control and optical property measurement of photonic devices and materials.


title: Visualizing Why Nondeterministic Finite-State Automata Reject

abstract: Students find their first course in Formal Languages and Automata Theory challenging. In addition to the development of formal arguments, most students struggle to understand nondeterministic computation models. In part, the struggle stems from the course exposing them for the first time to nondeterminism. Often, students find it difficult to understand why a nondeterministic machine accepts or rejects a word. Furthermore, they may feel uncomfortable with there being multiple computations on the same input and with a machine not consuming all of its input. This article describes a visualization tool developed to help students understand nondeterministic behavior. The tool is integrated into, FSM, a domain-specific language for the Automata Theory classroom. The strategy is based on the automatic generation of computation graphs given a machine and an input word. Unlike previous visualization tools, the computation graphs generated reflect the structure of the given machine's transition relation and not the structure of the computation tree.


title: Beyond Sharing Weights in Decoupling Feature Learning Network for UAV RGB-Infrared Vehicle Re-Identification

abstract: Owing to the capacity of performing full-time target search, cross-modality vehicle re-identification (Re-ID) based on unmanned aerial vehicle (UAV) is gaining more attention in both video surveillance and public security. However, this promising and innovative research has not been studied sufficiently due to the data inadequacy issue. Meanwhile, the cross-modality discrepancy and orientation discrepancy challenges further aggravate the difficulty of this task. To this end, we pioneer a cross-modality vehicle Re-ID benchmark named UAV Cross-Modality Vehicle Re-ID (UCM-VeID), containing 753 identities with 16015 RGB and 13913 infrared images. Moreover, to meet cross-modality discrepancy and orientation discrepancy challenges, we present a hybrid weights decoupling network (HWDNet) to learn the shared discriminative orientation-invariant features. For the first challenge, we proposed a hybrid weights siamese network with a well-designed weight restrainer and its corresponding objective function to learn both modality-specific and modality shared information. In terms of the second challenge, three effective decoupling structures with two pretext tasks are investigated to learn orientation-invariant feature. Comprehensive experiments are carried out to validate the effectiveness of the proposed method. The dataset and codes will be released at https://github.com/moonstarL/UAV-CM-VeID.


title: Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection

abstract: Out-of-distribution (OOD) detection is essential for reliable and trustworthy machine learning. Recent multi-modal OOD detection leverages textual information from in-distribution (ID) class names for visual OOD detection, yet it currently neglects the rich contextual information of ID classes. Large language models (LLMs) encode a wealth of world knowledge and can be prompted to generate descriptive features for each class. Indiscriminately using such knowledge causes catastrophic damage to OOD detection due to LLMs' hallucinations, as is observed by our analysis. In this paper, we propose to apply world knowledge to enhance OOD detection performance through selective generation from LLMs. Specifically, we introduce a consistency-based uncertainty calibration method to estimate the confidence score of each generation. We further extract visual objects from each image to fully capitalize on the aforementioned world knowledge. Extensive experiments demonstrate that our method consistently outperforms the state-of-the-art.


title: Time-resolved ARPES with probe energy of 6.0/7.2 eV and switchable resolution configuration

abstract: We present a detailed exposition of the design for time- and angle-resolved photoemission spectroscopy using a UV probe laser source that combines the nonlinear effects of \b{eta}-BaB2O4 and KBe2BO3F2 optical crystals. The photon energy of the probe laser can be switched between 6.0 and 7.2 eV, with the flexibility to operate each photon energy setting under two distinct resolution configurations. Under the fully optimized energy resolution configuration, we achieve an energy resolution of 8.5 meV at 6.0 eV and 10 meV at 7.2 eV. Alternatively, switching to the other configuration enhances temporal resolution, yielding a temporal resolution of 72 fs for 6.0 eV and 185 fs for 7.2 eV. We validated the performance and reliability of our system by applying it to measuring two typical materials: the topological insulator MnBi2Te4 and excitonic insulator candidate Ta2NiSe5.


title: Increasing the Earth's Albedo: The K\"ohler Equation at Sea

abstract: Increasing marine haze and clouds has been considered as a possible means of increasing the Earth's albedo. This would reduce Solar heating and global warming, counteracting the effects of the anthropogenic increase in greenhouse gases. One proposed method of doing so would inject small droplets of seawater or condensation nuclei into the marine boundary layer, creating artificial haze and cloud. The equilibrium size of such droplets is described by the K\"{o}hler equation that includes the vapor pressure reduction attributable to the solute according to Raoult's law and the vapor pressure increase of a small droplet as a result of surface tension according to Kelvin. Here we apply this classic result to small droplets in the marine boundary layer, where the partial pressure of water vapor is less than the equilibrium vapor pressure because it is in equilibrium with the saline ocean. We calculate the equilibrium size of a droplet containing dissolved ions and find that the radius of a droplet of seawater shrinks greatly before it achieves equilibrium.


title: An improved dynamical Poisson equation solver for self-gravity

abstract: Since self-gravity is crucial in the structure formation of the universe, many hydrodynamics simulations with the effect of self-gravity have been conducted. The multigrid method is widely used as a solver for the Poisson equation of the self-gravity; however, the parallelization efficiency of the multigrid method becomes worse when we use a massively parallel computer, and it becomes inefficient with more than $10^4$ cores, even for highly tuned codes. To perform large-scale parallel simulations ($> 10^4$ cores), developing a new gravity solver with good parallelization efficiency is beneficial. In this article, we develop a new self-gravity solver using the telegraph equation with a damping coefficient, $\kappa$. Parallelization is much easier than the case of the elliptic Poisson equation since the telegraph equation is a hyperbolic partial differential equation. We analyze convergence tests of our telegraph equations solver and determine that the best non-dimensional damping coefficient of the telegraph equations is $\tilde{\kappa} \simeq 2.5$. We also show that our method can maintain high parallelization efficiency even for massively parallel computations due to the hyperbolic nature of the telegraphic equation by weak-scaling tests. If the time step of the calculation is determined by heating/cooling or chemical reactions, rather than the CFL condition, our method may provide the method for calculating self-gravity faster than other previously known methods such as the fast Fourier transform and multigrid iteration solvers because gravitational phase velocity determined by the CFL condition using these timescales is much larger than the fluid velocity plus sound speed.


title: Local Graph Clustering with Noisy Labels

abstract: The growing interest in machine learning problems over graphs with additional node information such as texts, images, or labels has popularized methods that require the costly operation of processing the entire graph. Yet, little effort has been made to the development of fast local methods (i.e. without accessing the entire graph) that extract useful information from such data. To that end, we propose a study of local graph clustering using noisy node labels as a proxy for additional node information. In this setting, nodes receive initial binary labels based on cluster affiliation: 1 if they belong to the target cluster and 0 otherwise. Subsequently, a fraction of these labels is flipped. We investigate the benefits of incorporating noisy labels for local graph clustering. By constructing a weighted graph with such labels, we study the performance of graph diffusion-based local clustering method on both the original and the weighted graphs. From a theoretical perspective, we consider recovering an unknown target cluster with a single seed node in a random graph with independent noisy node labels. We provide sufficient conditions on the label noise under which, with high probability, using diffusion in the weighted graph yields a more accurate recovery of the target cluster. This approach proves more effective than using the given labels alone or using diffusion in the label-free original graph. Empirically, we show that reliable node labels can be obtained with just a few samples from an attributed graph. Moreover, utilizing these labels via diffusion in the weighted graph leads to significantly better local clustering performance across several real-world datasets, improving F1 scores by up to 13%.


title: Incorporating Domain Knowledge Graph into Multimodal Movie Genre Classification with Self-Supervised Attention and Contrastive Learning

abstract: Multimodal movie genre classification has always been regarded as a demanding multi-label classification task due to the diversity of multimodal data such as posters, plot summaries, trailers and metadata. Although existing works have made great progress in modeling and combining each modality, they still face three issues: 1) unutilized group relations in metadata, 2) unreliable attention allocation, and 3) indiscriminative fused features. Given that the knowledge graph has been proven to contain rich information, we present a novel framework that exploits the knowledge graph from various perspectives to address the above problems. As a preparation, the metadata is processed into a domain knowledge graph. A translate model for knowledge graph embedding is adopted to capture the relations between entities. Firstly we retrieve the relevant embedding from the knowledge graph by utilizing group relations in metadata and then integrate it with other modalities. Next, we introduce an Attention Teacher module for reliable attention allocation based on self-supervised learning. It learns the distribution of the knowledge graph and produces rational attention weights. Finally, a Genre-Centroid Anchored Contrastive Learning module is proposed to strengthen the discriminative ability of fused features. The embedding space of anchors is initialized from the genre entities in the knowledge graph. To verify the effectiveness of our framework, we collect a larger and more challenging dataset named MM-IMDb 2.0 compared with the MM-IMDb dataset. The experimental results on two datasets demonstrate that our model is superior to the state-of-the-art methods. We will release the code in the near future.


title: A Mean Field Game approach to Large Deviations for empirical measures under common noise

abstract: We present a Mean Field Game approach to obtain the rate function for the empirical measure of interacting particles under McKean-Vlasov dynamics. Although the result is well known, our approach relies on PDE methods and provides another example of the well known connection between Mean Field Games and Large Deviations.


title: Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles

abstract: The fusion of human-centric design and artificial intelligence (AI) capabilities has opened up new possibilities for next-generation autonomous vehicles that go beyond transportation. These vehicles can dynamically interact with passengers and adapt to their preferences. This paper proposes a novel framework that leverages Large Language Models (LLMs) to enhance the decision-making process in autonomous vehicles. By utilizing LLMs' linguistic and contextual understanding abilities with specialized tools, we aim to integrate the language and reasoning capabilities of LLMs into autonomous vehicles. Our research includes experiments in HighwayEnv, a collection of environments for autonomous driving and tactical decision-making tasks, to explore LLMs' interpretation, interaction, and reasoning in various scenarios. We also examine real-time personalization, demonstrating how LLMs can influence driving behaviors based on verbal commands. Our empirical results highlight the substantial advantages of utilizing chain-of-thought prompting, leading to improved driving decisions, and showing the potential for LLMs to enhance personalized driving experiences through ongoing verbal feedback. The proposed framework aims to transform autonomous vehicle operations, offering personalized support, transparent decision-making, and continuous learning to enhance safety and effectiveness. We achieve user-centric, transparent, and adaptive autonomous driving ecosystems supported by the integration of LLMs into autonomous vehicles.


title: BaSAL: Size-Balanced Warm Start Active Learning for LiDAR Semantic Segmentation

abstract: Active learning strives to reduce the need for costly data annotation, by repeatedly querying an annotator to label the most informative samples from a pool of unlabeled data, and then training a model from these samples. We identify two problems with existing active learning methods for LiDAR semantic segmentation. First, they overlook the severe class imbalance inherent in LiDAR semantic segmentation datasets. Second, to bootstrap the active learning loop when there is no labeled data available, they train their initial model from randomly selected data samples, leading to low performance. This situation is referred to as the cold start problem. To address these problems we propose BaSAL, a size-balanced warm start active learning model, based on the observation that each object class has a characteristic size. By sampling object clusters according to their size, we can thus create a size-balanced dataset that is also more class-balanced. Furthermore, in contrast to existing information measures like entropy or CoreSet, size-based sampling does not require a pretrained model, thus addressing the cold start problem effectively. Results show that we are able to improve the performance of the initial model by a large margin. Combining warm start and size-balanced sampling with established information measures, our approach achieves comparable performance to training on the entire SemanticKITTI dataset, despite using only 5% of the annotations, outperforming existing active learning methods. We also match the existing state-of-the-art in active learning on nuScenes. Our code is available at: https://github.com/Tony-WJR/BaSAL.


title: ZEST: Attention-based Zero-Shot Learning for Unseen IoT Device Classification

abstract: Recent research works have proposed machine learning models for classifying IoT devices connected to a network. However, there is still a practical challenge of not having all devices (and hence their traffic) available during the training of a model. This essentially means, during the operational phase, we need to classify new devices not seen in the training phase. To address this challenge, we propose ZEST -- a ZSL (zero-shot learning) framework based on self-attention for classifying both seen and unseen devices. ZEST consists of i) a self-attention based network feature extractor, termed SANE, for extracting latent space representations of IoT traffic, ii) a generative model that trains a decoder using latent features to generate pseudo data, and iii) a supervised model that is trained on the generated pseudo data for classifying devices. We carry out extensive experiments on real IoT traffic data; our experiments demonstrate i) ZEST achieves significant improvement (in terms of accuracy) over the baselines; ii) SANE is able to better extract meaningful representations than LSTM which has been commonly used for modeling network traffic.


title: Impact Factors for Computer Science Conferences

abstract: An increasing number of CS researchers are employed in academic non-CS departments where publication output is measured in terms of journal impact factors. To foster recognition of publications in peer-reviewed CS conference proceedings, we analyzed more than 40,000 CS publications and computed journal impact factors for 88 top-ranking conferences across a representative range of fields, finding that some conferences have impact factors corresponding to those of high-ranking journals.


title: Continual Learning via Manifold Expansion Replay

abstract: In continual learning, the learner learns multiple tasks in sequence, with data being acquired only once for each task. Catastrophic forgetting is a major challenge to continual learning. To reduce forgetting, some existing rehearsal-based methods use episodic memory to replay samples of previous tasks. However, in the process of knowledge integration when learning a new task, this strategy also suffers from catastrophic forgetting due to an imbalance between old and new knowledge. To address this problem, we propose a novel replay strategy called Manifold Expansion Replay (MaER). We argue that expanding the implicit manifold of the knowledge representation in the episodic memory helps to improve the robustness and expressiveness of the model. To this end, we propose a greedy strategy to keep increasing the diameter of the implicit manifold represented by the knowledge in the buffer during memory management. In addition, we introduce Wasserstein distance instead of cross entropy as distillation loss to preserve previous knowledge. With extensive experimental validation on MNIST, CIFAR10, CIFAR100, and TinyImageNet, we show that the proposed method significantly improves the accuracy in continual learning setup, outperforming the state of the arts.


title: Rethinking Large-scale Pre-ranking System: Entire-chain Cross-domain Models

abstract: Industrial systems such as recommender systems and online advertising, have been widely equipped with multi-stage architectures, which are divided into several cascaded modules, including matching, pre-ranking, ranking and re-ranking. As a critical bridge between matching and ranking, existing pre-ranking approaches mainly endure sample selection bias (SSB) problem owing to ignoring the entire-chain data dependence, resulting in sub-optimal performances. In this paper, we rethink pre-ranking system from the perspective of the entire sample space, and propose Entire-chain Cross-domain Models (ECM), which leverage samples from the whole cascaded stages to effectively alleviate SSB problem. Besides, we design a fine-grained neural structure named ECMM to further improve the pre-ranking accuracy. Specifically, we propose a cross-domain multi-tower neural network to comprehensively predict for each stage result, and introduce the sub-networking routing strategy with $L0$ regularization to reduce computational costs. Evaluations on real-world large-scale traffic logs demonstrate that our pre-ranking models outperform SOTA methods while time consumption is maintained within an acceptable level, which achieves better trade-off between efficiency and effectiveness.


title: SEE-OoD: Supervised Exploration For Enhanced Out-of-Distribution Detection

abstract: Current techniques for Out-of-Distribution (OoD) detection predominantly rely on quantifying predictive uncertainty and incorporating model regularization during the training phase, using either real or synthetic OoD samples. However, methods that utilize real OoD samples lack exploration and are prone to overfit the OoD samples at hand. Whereas synthetic samples are often generated based on features extracted from training data, rendering them less effective when the training and OoD data are highly overlapped in the feature space. In this work, we propose a Wasserstein-score-based generative adversarial training scheme to enhance OoD detection accuracy, which, for the first time, performs data augmentation and exploration simultaneously under the supervision of limited OoD samples. Specifically, the generator explores OoD spaces and generates synthetic OoD samples using feedback from the discriminator, while the discriminator exploits both the observed and synthesized samples for OoD detection using a predefined Wasserstein score. We provide theoretical guarantees that the optimal solutions of our generative scheme are statistically achievable through adversarial training in empirical settings. We then demonstrate that the proposed method outperforms state-of-the-art techniques on various computer vision datasets and exhibits superior generalizability to unseen OoD data.


title: QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models

abstract: Large Language Models (LLMs) excel in NLP, but their demands hinder their widespread deployment. While Quantization-Aware Training (QAT) offers a solution, its extensive training costs make Post-Training Quantization (PTQ) a more practical approach for LLMs. In existing studies, activation outliers in particular channels are identified as the bottleneck to PTQ accuracy. They propose to transform the magnitudes from activations to weights, which however offers limited alleviation or suffers from unstable gradients, resulting in a severe performance drop at low-bitwidth. In this paper, we propose QLLM, an accurate and efficient low-bitwidth PTQ method designed for LLMs. QLLM introduces an adaptive channel reassembly technique that reallocates the magnitude of outliers to other channels, thereby mitigating their impact on the quantization range. This is achieved by channel disassembly and channel assembly, which first breaks down the outlier channels into several sub-channels to ensure a more balanced distribution of activation magnitudes. Then similar channels are merged to maintain the original channel number for efficiency. Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly. To further compensate for the performance loss caused by quantization, we propose an efficient tuning method that only learns a small number of low-rank weights while freezing the pre-trained quantized model. After training, these low-rank parameters can be fused into the frozen weights without affecting inference. Extensive experiments on LLaMA-1 and LLaMA-2 show that QLLM can obtain accurate quantized models efficiently. For example, QLLM quantizes the 4-bit LLaMA-2-70B within 10 hours on a single A100-80G GPU, outperforming the previous state-of-the-art method by 7.89% on the average accuracy across five zero-shot tasks.


title: X-HRNet: Towards Lightweight Human Pose Estimation with Spatially Unidimensional Self-Attention

abstract: High-resolution representation is necessary for human pose estimation to achieve high performance, and the ensuing problem is high computational complexity. In particular, predominant pose estimation methods estimate human joints by 2D single-peak heatmaps. Each 2D heatmap can be horizontally and vertically projected to and reconstructed by a pair of 1D heat vectors. Inspired by this observation, we introduce a lightweight and powerful alternative, Spatially Unidimensional Self-Attention (SUSA), to the pointwise (1x1) convolution that is the main computational bottleneck in the depthwise separable 3c3 convolution. Our SUSA reduces the computational complexity of the pointwise (1x1) convolution by 96% without sacrificing accuracy. Furthermore, we use the SUSA as the main module to build our lightweight pose estimation backbone X-HRNet, where `X' represents the estimated cross-shape attention vectors. Extensive experiments on the COCO benchmark demonstrate the superiority of our X-HRNet, and comprehensive ablation studies show the effectiveness of the SUSA modules. The code is publicly available at https://github.com/cool-xuan/x-hrnet.


title: Understanding and Controlling a Maze-Solving Policy Network

abstract: To understand the goals and goal representations of AI systems, we carefully study a pretrained reinforcement learning policy that solves mazes by navigating to a range of target squares. We find this network pursues multiple context-dependent goals, and we further identify circuits within the network that correspond to one of these goals. In particular, we identified eleven channels that track the location of the goal. By modifying these channels, either with hand-designed interventions or by combining forward passes, we can partially control the policy. We show that this network contains redundant, distributed, and retargetable goal representations, shedding light on the nature of goal-direction in trained policy networks.


title: EC-Depth: Exploring the consistency of self-supervised monocular depth estimation in challenging scenes

abstract: Self-supervised monocular depth estimation holds significant importance in the fields of autonomous driving and robotics. However, existing methods are typically trained and tested on standard datasets, overlooking the impact of various adverse conditions prevalent in real-world applications, such as rainy days. As a result, it is commonly observed that these methods struggle to handle these challenging scenarios. To address this issue, we present EC-Depth, a novel self-supervised two-stage framework to achieve a robust depth estimation. In the first stage, we propose depth consistency regularization to propagate reliable supervision from standard to challenging scenes. In the second stage, we adopt the Mean Teacher paradigm and propose a novel consistency-based pseudo-label filtering strategy to improve the quality of pseudo-labels, further improving both the accuracy and robustness of our model. Extensive experiments demonstrate that our method achieves accurate and consistent depth predictions in both standard and challenging scenarios, surpassing existing state-of-the-art methods on KITTI, KITTI-C, DrivingStereo, and NuScenes-Night benchmarks.


title: Model Predictive Inferential Control of Neural State-Space Models for Autonomous Vehicle Motion Planning

abstract: Model predictive control (MPC) has proven useful in enabling safe and optimal motion planning for autonomous vehicles. In this paper, we investigate how to achieve MPC-based motion planning when a neural state-space model represents the vehicle dynamics. As the neural state-space model will lead to highly complex, nonlinear and nonconvex optimization landscapes, mainstream gradient-based MPC methods will be computationally too heavy to be a viable solution. In a departure, we propose the idea of model predictive inferential control (MPIC), which seeks to infer the best control decisions from the control objectives and constraints. Following the idea, we convert the MPC problem for motion planning into a Bayesian state estimation problem. Then, we develop a new particle filtering/smoothing approach to perform the estimation. This approach is implemented as banks of unscented Kalman filters/smoothers and offers high sampling efficiency, fast computation, and estimation accuracy. We evaluate the MPIC approach through a simulation study of autonomous driving in different scenarios, along with an exhaustive comparison with gradient-based MPC. The results show that the MPIC approach has considerable computational efficiency, regardless of complex neural network architectures, and shows the capability to solve large-scale MPC problems for neural state-space models.


title: Dynamics of particles with electric charge and magnetic dipole moment near Schwarzschild-MOG black hole

abstract: Investigations of electromagnetic interactions between test-charged and magnetized particles are important in the dynamics of the particles in strong gravitational fields around black holes. Here, we study the dynamics of a particle having an electric charge and a magnetic dipole moment in the spacetime of a Schwarzschild black hole in modified gravity (MOG), called Schwarzschild-MOG black hole. First, we provide a solution of Maxwell equations for the angular component of electromagnetic four potentials in the Schwarzschild-MOG spacetime. Then, we derive equations of motion and effective potential for circular motion of such particles using a hybrid form of the Hamilton-Jacobi equation which includes both interactions of electric charge and magnetic dipole moment with the external magnetic field assumed as asymptotically uniform, and interaction between the particles and the MOG field. Also, we investigate the effects of the three types of interactions on the radius of innermost stable circular orbits (ISCOs) and the energy \& angular momentum of the particles at their corresponding ISCOs. Finally, we provide detailed analyses of the effects of the three interactions mentioned above on the center of mass energy in the collisions between neutral, electrically charged, and magnetized particles.


title: Three Paths to Rational Curves with Rational Arc Length

abstract: We solve the so far open problem of constructing all spatial rational curves with rational arc length functions. More precisely, we present three different methods for this construction. The first method adapts a recent approach of (Kalkan et al. 2022) to rational PH curves and requires solving a modestly sized system of linear equations. The second constructs the curve by imposing zero-residue conditions, thus extending ideas of previous papers by (Farouki and Sakkalis 2019) and the authors themselves (Schr\"ocker and \v{S}\'ir 2023). The third method generalizes the dual approach of (Pottmann 1995) from planar to spatial curves. The three methods share the same quaternion based representation in which not only the PH curve but also its arc length function are compactly expressed. We also present a new proof based on the quaternion polynomial factorization theory of the well known characterization of the Pythagorean quadruples.


title: Semi-Classical Asymptotics of Bergman and Spectral Kernels for (0,q)-forms

abstract: In this paper, we develop a new scaling method to study spectral and Bergman kernels for the k-th tensor power of a line bundle over a complex manifold under local spectral gap condition. In particular, we establish a simple proof of the pointwise asymptotics of spectral and Bergman kernels. As a new result, in the function case, we obtain the leading term of Bergman kernel under spectral gap with exponential decay. Moreover, in the general cases of (0,q)-forms, the asymptotics remain valid while the curvature of the line bundle is degenerate.


title: Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability

abstract: What is the relationship between model architecture and the ability to perform in-context learning? In this empirical study, we take the first steps toward answering this question. We evaluate thirteen model architectures capable of causal language modeling across a suite of synthetic in-context learning tasks. These selected architectures represent a broad range of paradigms, including recurrent and convolution-based neural networks, transformers, state space model inspired, and other emerging attention alternatives. We discover that all the considered architectures can perform in-context learning under a wider range of conditions than previously documented. Additionally, we observe stark differences in statistical efficiency and consistency by varying the number of in-context examples and task difficulty. We also measure each architecture's predisposition towards in-context learning when presented with the option to memorize rather than leverage in-context examples. Finally, and somewhat surprisingly, we find that several attention alternatives are sometimes competitive with or better in-context learners than transformers. However, no single architecture demonstrates consistency across all tasks, with performance either plateauing or declining when confronted with a significantly larger number of in-context examples than those encountered during gradient-based training.


title: $\mathfrak{g}_0$-Geometry and Its Relationship with Stable Representations

abstract: This paper explores the sheaves with the action of a lie algebra and computes their cohomology in a new category. Then in the following sections, We try to generalize a classical result in [GM, Ch. IV] about exterior algebra. We add the action of $\mathfrak{g}_{0}$ and prove that there is still a faithful functor.


title: LGL-BCI: A Lightweight Geometric Learning Framework for Motor Imagery-Based Brain-Computer Interfaces

abstract: Brain-Computer Interfaces (BCIs) are a groundbreaking technology for interacting with external devices using brain signals. Despite advancements, electroencephalogram (EEG)-based Motor Imagery (MI) tasks face challenges like amplitude and phase variability, and complex spatial correlations, with a need for smaller model size and faster inference. This study introduces the LGL-BCI framework, employing a Geometric Deep Learning Framework for EEG processing in non-Euclidean metric spaces, particularly the Symmetric Positive Definite (SPD) Manifold space. LGL-BCI offers robust EEG data representation and captures spatial correlations. We propose an EEG channel selection solution via a feature decomposition algorithm to reduce SPD matrix dimensionality, with a lossless transformation boosting inference speed. Extensive experiments show LGL-BCI's superior accuracy and efficiency compared to current solutions, highlighting geometric deep learning's potential in MI-BCI applications. The efficiency, assessed on two public EEG datasets and two real-world EEG devices, significantly outperforms the state-of-the-art solution in accuracy ($82.54\%$ versus $62.22\%$) with fewer parameters (64.9M compared to 183.7M).


title: Collisional excitation of PO$^+$ by $para$-H$_2$

abstract: We report the derivation of rate coefficients for the rotational (de-)excitation of PO$^+$ induced by collisions with H$_2$. The calculations were performed on a four-dimensional potential energy surface, obtained on top of highly accurate $ab$ $initio$ energy points. Preliminary tests pointed out the low influence of the coupling between $j=0$ and the higher rotational levels of H$_2$ on the cross sections values, thus allowing to neglect the rotational structure of H$_2$. On this basis, state-to-state collisional rate coefficients were derived for temperatures ranging from 5 to 200 K. Radiative transfer calculations have been used to model the recent observation of PO$^+$ in the G+0.693-0.027 molecular cloud, in order to evaluate the possible impact of non-LTE models on the determination of its physical conditions. The derived column density was found to be approximately $\sim 3.7\times10^{11}$ cm$^{-2}$, which is 60\% (a factor of $\sim 1.7$) smaller than the previously LTE-derived value. Extensive simulations show that PO$^+$ low-$j$ rotational lines exhibit maser behavior at densities between $10^4$ and $10^6$ cm$^{-3}$, thus highlighting the importance of a proper treatment of the molecular collisions to accurately model PO$^+$ emissions in the interstellar medium.


title: Lattice real-time simulations with learned optimal kernels

abstract: We present a simulation strategy for the real-time dynamics of quantum fields, inspired by reinforcement learning. It builds on the complex Langevin approach, which it amends with system specific prior information, a necessary prerequisite to overcome this exceptionally severe sign problem. The optimization process underlying our machine learning approach is made possible by deploying inherently stable solvers of the complex Langevin stochastic process and a novel optimality criterion derived from insight into so-called boundary terms. This conceptual and technical progress allows us to both significantly extend the range of real-time simulations in 1+1d scalar field theory beyond the state-of-the-art and to avoid discretization artifacts that plagued previous real-time field theory simulations. Limitations of and promising future directions are discussed.

categories: hep-lat cond-mat.other hep-th nucl-th stat.ML


title: Proton-mediated reversible switching of metastable ferroelectric phases with low operation voltages

abstract: The exploration of ferroelectric phase transitions enables an in-depth understanding of ferroelectric switching and promising applications in information storage. However, controllably tuning the dynamics of ferroelectric phase transitions remains challenging owing to inaccessible hidden phases. Here, using protonic gating technology, we create a series of metastable ferroelectric phases and demonstrate their reversible transitions in van der Waals ferroelectric {\alpha}-In2Se3 transistors. By varying the gate bias, protons can be incrementally injected or extracted, achieving controllable tuning of the ferroelectric {\alpha}-In2Se3 protonic dynamics across the channel and obtaining numerous intermediate phases. Combining piezoelectric force microscopy and electrical hysteresis measurements, we surprisingly discover that the gate tuning of {\alpha}-In2Se3 protonation is volatile, unlike the conventional gating effect, and the created new phases remain polar. Their origin, revealed by first-principles calculations, is related to the formation of metastable hydrogen-stabilized {\alpha}-In2Se3 phases with switchable dipoles. Furthermore, our approach enables ultralow gate-voltage switching of different phases (below 0.4 V), and easy integration with transistor components. This work provides fundamental insights into the interactions between protons and ferroelectric phase transitions, providing a possible avenue for accessing hidden phases in ferroelectric switching and for developing energy-efficient, multilevel-switching memory devices.


title: Log-Gaussian Gamma Processes for Training Bayesian Neural Networks in Raman and CARS Spectroscopies

abstract: We propose an approach utilizing gamma-distributed random variables, coupled with log-Gaussian modeling, to generate synthetic datasets suitable for training neural networks. This addresses the challenge of limited real observations in various applications. We apply this methodology to both Raman and coherent anti-Stokes Raman scattering (CARS) spectra, using experimental spectra to estimate gamma process parameters. Parameter estimation is performed using Markov chain Monte Carlo methods, yielding a full Bayesian posterior distribution for the model which can be sampled for synthetic data generation. Additionally, we model the additive and multiplicative background functions for Raman and CARS with Gaussian processes. We train two Bayesian neural networks to estimate parameters of the gamma process which can then be used to estimate the underlying Raman spectrum and simultaneously provide uncertainty through the estimation of parameters of a probability distribution. We apply the trained Bayesian neural networks to experimental Raman spectra of phthalocyanine blue, aniline black, naphthol red, and red 264 pigments and also to experimental CARS spectra of adenosine phosphate, fructose, glucose, and sucrose. The results agree with deterministic point estimates for the underlying Raman and CARS spectral signatures.


title: Learning from Label Proportions: Bootstrapping Supervised Learners via Belief Propagation

abstract: Learning from Label Proportions (LLP) is a learning problem where only aggregate level labels are available for groups of instances, called bags, during training, and the aim is to get the best performance at the instance-level on the test data. This setting arises in domains like advertising and medicine due to privacy considerations. We propose a novel algorithmic framework for this problem that iteratively performs two main steps. For the first step (Pseudo Labeling) in every iteration, we define a Gibbs distribution over binary instance labels that incorporates a) covariate information through the constraint that instances with similar covariates should have similar labels and b) the bag level aggregated label. We then use Belief Propagation (BP) to marginalize the Gibbs distribution to obtain pseudo labels. In the second step (Embedding Refinement), we use the pseudo labels to provide supervision for a learner that yields a better embedding. Further, we iterate on the two steps again by using the second step's embeddings as new covariates for the next iteration. In the final iteration, a classifier is trained using the pseudo labels. Our algorithm displays strong gains against several SOTA baselines (up to 15%) for the LLP Binary Classification problem on various dataset types - tabular and Image. We achieve these improvements with minimal computational overhead above standard supervised learning due to Belief Propagation, for large bag sizes, even for a million samples.


title: Structural balance and spectral properties of generalized corona product of signed graphs

abstract: In this paper, we extend our earlier proposal of corona product of signed graphs into generalized corona product of signed graphs inspired by the generalized corona product of unsigned graphs. Then we study structural balance and spectral properties of these graphs. Utilizing the notion of coronal of a graph, we determine computable formulae of characteristic, Laplacian, and signless Laplacian polynomials of generalized corona product of signed graphs. Finally, we provide sufficient conditions for the generalized corona product of some distinct collections of signed graphs to be co-spectral.


title: Viscosity solutions to a Cauchy type problem for timelike Lorentzian eikonal equation

abstract: In this paper, we propose a Cauchy type problem to the timelike Lorentzian eikonal equation on a globally hyperbolic space-time. For this equation, as the value of the solution on a Cauchy surface is known, we prove the existence of viscosity solutions on the past set (future set) of the Cauchy surface. Furthermore, when the time orientation of viscosity solution is consistent, the uniqueness and stability of viscosity solutions are also obtained.


title: Stability of Periodic Waves for the Defocusing Fractional Cubic Nonlinear Schr\"odinger Equation

abstract: In this paper, we determine the spectral instability of periodic odd waves for the defocusing fractional cubic nonlinear Schr\"odinger equation. Our approach is based on periodic perturbations that have the same period as the standing wave solution, and we construct real periodic waves by minimizing a suitable constrained problem. The odd solution generates three negative simple eigenvalues for the associated linearized operator, and we obtain all this spectral information by using tools related to the oscillation theorem for fractional Hill operators. Newton's iteration method is presented to generate the odd periodic standing wave solutions and numerical results have been used to apply the spectral stability theory via Krein signature as established in [22] and [23].


title: Volumes of Subvarieties of Complex Ball Quotients and Sparsity of Rational Points

abstract: Let $X=\Gamma \backslash \mathbb{B}^{n} $ be an $n$-dimensional complex ball quotient by a torsion-free non-uniform lattice $\Gamma$ whose parabolic subgroups are unipotent. We prove that the volumes of subvarieties of $X$ are controlled by the systole of $X,$ which is the length of the shortest closed geodesic of $X$. There are a number of arithmetic and geometric consequences: the systole of $X$ controls the growth rate of rational points on $X,$ uniformly in the field of definition. Also, we obtain effective global generation and very ampleness results for multiples of the canonical bundle $K_{\overline{X}},$ where $\overline{X}$ is the toroidal compactification of $X.$ These results follow from the bound we find for the Seshadri constant of $K_{\overline{X}}$ in terms of the systole.


title: ETDock: A Novel Equivariant Transformer for Protein-Ligand Docking

abstract: Predicting the docking between proteins and ligands is a crucial and challenging task for drug discovery. However, traditional docking methods mainly rely on scoring functions, and deep learning-based docking approaches usually neglect the 3D spatial information of proteins and ligands, as well as the graph-level features of ligands, which limits their performance. To address these limitations, we propose an equivariant transformer neural network for protein-ligand docking pose prediction. Our approach involves the fusion of ligand graph-level features by feature processing, followed by the learning of ligand and protein representations using our proposed TAMformer module. Additionally, we employ an iterative optimization approach based on the predicted distance matrix to generate refined ligand poses. The experimental results on real datasets show that our model can achieve state-of-the-art performance.


title: Reconstructing jets in the Phase-2 upgrade of the CMS Level-1 Trigger with a seeded cone algorithm

abstract: The Phase-2 Upgrade of the CMS Level-1 Trigger (L1T) will reconstruct particles using the Particle Flow algorithm, connecting information from the tracker, muon, and calorimeter detectors, and enabling fine-grained reconstruction of high level physics objects like jets. We have developed a jet reconstruction algorithm using a cone centred on an energetic seed from these Particle Flow candidates. The implementation is designed to find up to 16 jets in each Xilinx Ultrascale+ FPGA, with a latency of less than 1 {\mu}s, and event throughput of 6.7 MHz to fit within the L1T system constraints. Pipelined processing enables reconstruction of jet collections with different cone sizes for little additional resource cost. The design of the algorithm also provides a platform for additional computation using the jet constituents, such as jet tagging using neural networks. We will describe the implementation, its jet reconstruction performance, computational metrics, and the developments towards jet tagging.


title: Inference for Nonlinear Endogenous Treatment Effects Accounting for High-Dimensional Covariate Complexity

abstract: Nonlinearity and endogeneity are prevalent challenges in causal analysis using observational data. This paper proposes an inference procedure for a nonlinear and endogenous marginal effect function, defined as the derivative of the nonparametric treatment function, with a primary focus on an additive model that includes high-dimensional covariates. Using the control function approach for identification, we implement a regularized nonparametric estimation to obtain an initial estimator of the model. Such an initial estimator suffers from two biases: the bias in estimating the control function and the regularization bias for the high-dimensional outcome model. Our key innovation is to devise the double bias correction procedure that corrects these two biases simultaneously. Building on this debiased estimator, we further provide a confidence band of the marginal effect function. Simulations and an empirical study of air pollution and migration demonstrate the validity of our procedures.


title: Age Estimation Based on Graph Convolutional Networks and Multi-head Attention Mechanisms

abstract: Age estimation technology is a part of facial recognition and has been applied to identity authentication. This technology achieves the development and application of a juvenile anti-addiction system by authenticating users in the game. Convolutional Neural Network (CNN) and Transformer algorithms are widely used in this application scenario. However, these two models cannot flexibly extract and model features of faces with irregular shapes, and they are ineffective in capturing key information. Furthermore, the above methods will contain a lot of background information while extracting features, which will interfere with the model. In consequence, it is easy to extract redundant information from images. In this paper, a new modeling idea is proposed to solve this problem, which can flexibly model irregular objects. The Graph Convolutional Network (GCN) is used to extract features from irregular face images effectively, and multi-head attention mechanisms are added to avoid redundant features and capture key region information in the image. This model can effectively improve the accuracy of age estimation and reduce the MAE error value to about 3.64, which is better than the effect of today's age estimation model, to improve the accuracy of face recognition and identity authentication.


title: Very-Large-Scale-Integrated High-$Q$ Nanoantenna Pixels (VINPix)

abstract: Metasurfaces provide a versatile and compact approach to free-space optical manipulation and wavefront shaping. Comprised of arrays of judiciously-arranged dipolar resonators, metasurfaces precisely control the amplitude, polarization, and phase of light, with applications spanning imaging, sensing, modulation, and computing. Three crucial performance metrics of metasurfaces and their constituent resonators are the quality factor ($Q$-factor), mode-volume ($V_m$), and the ability to control far-field radiation. Often, resonators face a trade-off between these parameters: a reduction in $V_m$ leads to an equivalent reduction in $Q$, albeit with more control over radiation. Here, we demonstrate that this perceived compromise is not inevitable $-$ high-$Q$, subwavelength $V_m$, and controlled dipole-like radiation can be achieved, simultaneously. We design high-$Q$, very-large-scale integrated silicon nanoantenna pixels $-$ VINPix $-$ that combine guided mode resonance waveguides with photonic crystal cavities. With optimized nanoantennas, we achieve $Q$-factors exceeding 1500 with $V_m$ less than 0.1 $(\lambda/n_{\text{air}})^3$. Each nanoantenna is individually addressable by free-space light, and exhibits dipole-like scattering to the far-field. Resonator densities exceeding a million nanoantennas per $\text{cm}^2$ can be achieved, as demonstrated by our fabrication of an 8 mm x 8 mm VINPix array. As a proof-of-concept application, we demonstrate spectrometer-free, spatially localized, refractive-index sensing utilizing a VINPix array. Our platform provides a foundation for compact, densely multiplexed devices such as spatial light modulators, computational spectrometers, and in-situ environmental sensors.


title: The Search-and-Mix Paradigm in Approximate Nash Equilibrium Algorithms

abstract: AI in Math deals with mathematics in a constructive manner so that reasoning becomes automated, less laborious, and less error-prone. For algorithms, the question becomes how to automate analyses for specific problems. For the first time, this work provides an automatic method for approximation analysis on a well-studied problem in theoretical computer science: computing approximate Nash equilibria in two-player games. We observe that such algorithms can be reformulated into a search-and-mix paradigm, which involves a search phase followed by a mixing phase. By doing so, we are able to fully automate the procedure of designing and analyzing the mixing phase. For example, we illustrate how to perform our method with a program to analyze the approximation bounds of all the algorithms in the literature. Same approximation bounds are computed without any hand-written proof. Our automatic method heavily relies on the LP-relaxation structure in approximate Nash equilibria. Since many approximation algorithms and online algorithms adopt the LP relaxation, our approach may be extended to automate the analysis of other algorithms.


title: GameGPT: Multi-agent Collaborative Framework for Game Development

abstract: The large language model (LLM) based agents have demonstrated their capacity to automate and expedite software development processes. In this paper, we focus on game development and propose a multi-agent collaborative framework, dubbed GameGPT, to automate game development. While many studies have pinpointed hallucination as a primary roadblock for deploying LLMs in production, we identify another concern: redundancy. Our framework presents a series of methods to mitigate both concerns. These methods include dual collaboration and layered approaches with several in-house lexicons, to mitigate the hallucination and redundancy in the planning, task identification, and implementation phases. Furthermore, a decoupling approach is also introduced to achieve code generation with better precision.


title: Frequency-Aware Re-Parameterization for Over-Fitting Based Image Compression

abstract: Over-fitting-based image compression requires weights compactness for compression and fast convergence for practical use, posing challenges for deep convolutional neural networks (CNNs) based methods. This paper presents a simple re-parameterization method to train CNNs with reduced weights storage and accelerated convergence. The convolution kernels are re-parameterized as a weighted sum of discrete cosine transform (DCT) kernels enabling direct optimization in the frequency domain. Combined with L1 regularization, the proposed method surpasses vanilla convolutions by achieving a significantly improved rate-distortion with low computational cost. The proposed method is verified with extensive experiments of over-fitting-based image restoration on various datasets, achieving up to -46.12% BD-rate on top of HEIF with only 200 iterations.


title: Rethinking Negative Pairs in Code Search

abstract: Recently, contrastive learning has become a key component in fine-tuning code search models for software development efficiency and effectiveness. It pulls together positive code snippets while pushing negative samples away given search queries. Among contrastive learning, InfoNCE is the most widely used loss function due to its better performance. However, the following problems in negative samples of InfoNCE may deteriorate its representation learning: 1) The existence of false negative samples in large code corpora due to duplications. 2). The failure to explicitly differentiate between the potential relevance of negative samples. As an example, a bubble sorting algorithm example is less ``negative'' than a file saving function for the quick sorting algorithm query. In this paper, we tackle the above problems by proposing a simple yet effective Soft-InfoNCE loss that inserts weight terms into InfoNCE. In our proposed loss function, we apply three methods to estimate the weights of negative pairs and show that the vanilla InfoNCE loss is a special case of Soft-InfoNCE. Theoretically, we analyze the effects of Soft-InfoNCE on controlling the distribution of learnt code representations and on deducing a more precise mutual information estimation. We furthermore discuss the superiority of proposed loss functions with other design alternatives. Extensive experiments demonstrate the effectiveness of Soft-InfoNCE and weights estimation methods under state-of-the-art code search models on a large-scale public dataset consisting of six programming languages. Source code is available at \url{https://github.com/Alex-HaochenLi/Soft-InfoNCE}.


title: Tight Time-Space Lower Bounds for Constant-Pass Learning

abstract: In his breakthrough paper, Raz showed that any parity learning algorithm requires either quadratic memory or an exponential number of samples [FOCS'16, JACM'19]. A line of work that followed extended this result to a large class of learning problems. Until recently, all these results considered learning in the streaming model, where each sample is drawn independently, and the learner is allowed a single pass over the stream of samples. Garg, Raz, and Tal [CCC'19] considered a stronger model, allowing multiple passes over the stream. In the $2$-pass model, they showed that learning parities of size $n$ requires either a memory of size $n^{1.5}$ or at least $2^{\sqrt{n}}$ samples. (Their result also generalizes to other learning problems.) In this work, for any constant $q$, we prove tight memory-sample lower bounds for any parity learning algorithm that makes $q$ passes over the stream of samples. We show that such a learner requires either $\Omega(n^{2})$ memory size or at least $2^{\Omega(n)}$ samples. Beyond establishing a tight lower bound, this is the first non-trivial lower bound for $q$-pass learning for any $q\ge 3$. Similar to prior work, our results extend to any learning problem with many nearly-orthogonal concepts. We complement the lower bound with an upper bound, showing that parity learning with $q$ passes can be done efficiently with $O(n^2/\log q)$ memory.


title: Learning Transferable Conceptual Prototypes for Interpretable Unsupervised Domain Adaptation

abstract: Despite the great progress of unsupervised domain adaptation (UDA) with the deep neural networks, current UDA models are opaque and cannot provide promising explanations, limiting their applications in the scenarios that require safe and controllable model decisions. At present, a surge of work focuses on designing deep interpretable methods with adequate data annotations and only a few methods consider the distributional shift problem. Most existing interpretable UDA methods are post-hoc ones, which cannot facilitate the model learning process for performance enhancement. In this paper, we propose an inherently interpretable method, named Transferable Conceptual Prototype Learning (TCPL), which could simultaneously interpret and improve the processes of knowledge transfer and decision-making in UDA. To achieve this goal, we design a hierarchically prototypical module that transfers categorical basic concepts from the source domain to the target domain and learns domain-shared prototypes for explaining the underlying reasoning process. With the learned transferable prototypes, a self-predictive consistent pseudo-label strategy that fuses confidence, predictions, and prototype information, is designed for selecting suitable target samples for pseudo annotations and gradually narrowing down the domain gap. Comprehensive experiments show that the proposed method can not only provide effective and intuitive explanations but also outperform previous state-of-the-arts.