title: Towards Multi-Task Multi-Modal Models: A Video Generative Perspective

abstract: Advancements in language foundation models have primarily fueled the recent surge in artificial intelligence. In contrast, generative learning of non-textual modalities, especially videos, significantly trails behind language modeling. This thesis chronicles our endeavor to build multi-task models for generating videos and other modalities under diverse conditions, as well as for understanding and compression applications. Given the high dimensionality of visual data, we pursue concise and accurate latent representations. Our video-native spatial-temporal tokenizers preserve high fidelity. We unveil a novel approach to mapping bidirectionally between visual observation and interpretable lexical terms. Furthermore, our scalable visual token representation proves beneficial across generation, compression, and understanding tasks. This achievement marks the first instances of language models surpassing diffusion models in visual synthesis and a video tokenizer outperforming industry-standard codecs. Within these multi-modal latent spaces, we study the design of multi-task generative models. Our masked multi-task transformer excels at the quality, efficiency, and flexibility of video generation. We enable a frozen language model, trained solely on text, to generate visual content. Finally, we build a scalable generative multi-modal transformer trained from scratch, enabling the generation of videos containing high-fidelity motion with the corresponding audio given diverse conditions. Throughout the course, we have shown the effectiveness of integrating multiple tasks, crafting high-fidelity latent representation, and generating multiple modalities. This work suggests intriguing potential for future exploration in generating non-textual data and enabling real-time, interactive experiences across various media forms.


title: Free-Space Optical Channel Turbulence Prediction: A Machine Learning Approach

abstract: Channel turbulence presents a formidable obstacle for free-space optical (FSO) communication. Anticipation of turbulence levels is highly important for mitigating disruptions. We study the application of machine learning (ML) to FSO data streams to rapidly predict channel turbulence levels with no additional sensing hardware. An optical bit stream was transmitted through a controlled channel in the lab under six distinct turbulence levels, and the efficacy of using ML to classify turbulence levels was examined. ML-based turbulence level classification was found to be >98% accurate with multiple ML training parameters, but highly dependent upon the timescale of changes between turbulence levels.


title: Latent Energy-Based Odyssey: Black-Box Optimization via Expanded Exploration in the Energy-Based Latent Space

abstract: Offline Black-Box Optimization (BBO) aims at optimizing a black-box function using the knowledge from a pre-collected offline dataset of function values and corresponding input designs. However, the high-dimensional and highly-multimodal input design space of black-box function pose inherent challenges for most existing methods that model and operate directly upon input designs. These issues include but are not limited to high sample complexity, which relates to inaccurate approximation of black-box function; and insufficient coverage and exploration of input design modes, which leads to suboptimal proposal of new input designs. In this work, we consider finding a latent space that serves as a compressed yet accurate representation of the design-value joint space, enabling effective latent exploration of high-value input design modes. To this end, we formulate an learnable energy-based latent space, and propose Noise-intensified Telescoping density-Ratio Estimation (NTRE) scheme for variational learning of an accurate latent space model without costly Markov Chain Monte Carlo. The optimization process is then exploration of high-value designs guided by the learned energy-based model in the latent space, formulated as gradient-based sampling from a latent-variable-parameterized inverse model. We show that our particular parameterization encourages expanded exploration around high-value design modes, motivated by inversion thinking of a fundamental result of conditional covariance matrix typically used for variance reduction. We observe that our method, backed by an accurately learned informative latent space and an expanding-exploration model design, yields significant improvements over strong previous methods on both synthetic and real world datasets such as the design-bench suite.


title: Pretraining with Random Noise for Fast and Robust Learning without Weight Transport

abstract: The brain prepares for learning even before interacting with the environment, by refining and optimizing its structures through spontaneous neural activity that resembles random noise. However, the mechanism of such a process has yet to be thoroughly understood, and it is unclear whether this process can benefit the algorithm of machine learning. Here, we study this issue using a neural network with a feedback alignment algorithm, demonstrating that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport. First, we found that random noise training modifies forward weights to match backward synaptic feedback, which is necessary for teaching errors by feedback alignment. As a result, a network with pre-aligned weights learns notably faster than a network without random noise training, even reaching a convergence speed comparable to that of a backpropagation algorithm. Sequential training with both random noise and data brings weights closer to synaptic feedback than training solely with data, enabling more precise credit assignment and faster learning. We also found that each readout probability approaches the chance level and that the effective dimensionality of weights decreases in a network pretrained with random noise. This pre-regularization allows the network to learn simple solutions of a low rank, reducing the generalization loss during subsequent training. This also enables the network robustly to generalize a novel, out-of-distribution dataset. Lastly, we confirmed that random noise pretraining reduces the amount of meta-loss, enhancing the network ability to adapt to various tasks. Overall, our results suggest that random noise training with feedback alignment offers a straightforward yet effective method of pretraining that facilitates quick and reliable learning without weight transport.


title: The Collusion of Memory and Nonlinearity in Stochastic Approximation With Constant Stepsize

abstract: In this work, we investigate stochastic approximation (SA) with Markovian data and nonlinear updates under constant stepsize $\alpha>0$. Existing work has primarily focused on either i.i.d. data or linear update rules. We take a new perspective and carefully examine the simultaneous presence of Markovian dependency of data and nonlinear update rules, delineating how the interplay between these two structures leads to complications that are not captured by prior techniques. By leveraging the smoothness and recurrence properties of the SA updates, we develop a fine-grained analysis of the correlation between the SA iterates $\theta_k$ and Markovian data $x_k$. This enables us to overcome the obstacles in existing analysis and establish for the first time the weak convergence of the joint process $(x_k, \theta_k)_{k\geq0}$. Furthermore, we present a precise characterization of the asymptotic bias of the SA iterates, given by $\mathbb{E}[\theta_\infty]-\theta^\ast=\alpha(b_\text{m}+b_\text{n}+b_\text{c})+O(\alpha^{3/2})$. Here, $b_\text{m}$ is associated with the Markovian noise, $b_\text{n}$ is tied to the nonlinearity, and notably, $b_\text{c}$ represents a multiplicative interaction between the Markovian noise and nonlinearity, which is absent in previous works. As a by-product of our analysis, we derive finite-time bounds on higher moment $\mathbb{E}[\|\theta_k-\theta^\ast\|^{2p}]$ and present non-asymptotic geometric convergence rates for the iterates, along with a Central Limit Theorem.


title: SIC-POVMs and the Knaster's Conjecture

abstract: Symmetric Informationally Complete Positive Operator-Valued Measures (SIC-POVMs) have been constructed in many dimensions using the Weyl-Heisenberg group. In the quantum information community, it is commonly believed that SCI-POVMs exist in all dimensions; however, the general proof of their existence is still an open problem. The Bloch sphere representation of SIC-POVMs allows for a general geometric description of the set of operators, where they form the vertices of a regular simplex oriented based on a continuous function. We use this perspective of the SIC-POVMs to prove the Knaster's conjecture for the geometry of SIC-POVMs and prove the existence of a continuous family of generalized SIC-POVMs where $(n^2-1)$ of the matrices have the same value of $Tr(\rho^k)$. Furthermore, by using numerical methods, we show that a regular simplex can be constructed such that all its vertices map to the same value of $Tr(\rho^3)$ on the Bloch sphere of $3$ and $4$ dimensional Hilbert spaces. In the $3$-dimensional Hilbert space, we generate $10^4$ generalized SIC-POVMs for randomly chosen $Tr(\rho^3)$ values such that all the elements are equivalent up to unitary transformations.


title: Faster Sampling via Stochastic Gradient Proximal Sampler

abstract: Stochastic gradients have been widely integrated into Langevin-based methods to improve their scalability and efficiency in solving large-scale sampling problems. However, the proximal sampler, which exhibits much faster convergence than Langevin-based algorithms in the deterministic setting Lee et al. (2021), has yet to be explored in its stochastic variants. In this paper, we study the Stochastic Proximal Samplers (SPS) for sampling from non-log-concave distributions. We first establish a general framework for implementing stochastic proximal samplers and establish the convergence theory accordingly. We show that the convergence to the target distribution can be guaranteed as long as the second moment of the algorithm trajectory is bounded and restricted Gaussian oracles can be well approximated. We then provide two implementable variants based on Stochastic gradient Langevin dynamics (SGLD) and Metropolis-adjusted Langevin algorithm (MALA), giving rise to SPS-SGLD and SPS-MALA. We further show that SPS-SGLD and SPS-MALA can achieve $\epsilon$-sampling error in total variation (TV) distance within $\tilde{\mathcal{O}}(d\epsilon^{-2})$ and $\tilde{\mathcal{O}}(d^{1/2}\epsilon^{-2})$ gradient complexities, which outperform the best-known result by at least an $\tilde{\mathcal{O}}(d^{1/3})$ factor. This enhancement in performance is corroborated by our empirical studies on synthetic data with various dimensions, demonstrating the efficiency of our proposed algorithm.


title: Limited-perception games

abstract: We study rational agents with different perception capabilities in strategic games. We focus on a class of one-shot limited-perception games. These games extend simultaneous-move normal-form games by presenting each player with an individualized perception of all players' payoff functions. The accuracy of a player's perception is determined by the player's capability level. Capability levels are countable and totally ordered, with a higher level corresponding to a more accurate perception. We study the rational behavior of players in these games and formalize relevant equilibria conditions. In contrast to equilibria in conventional bimatrix games, which can be represented by a pair of mixed strategies, in our limited perception games a higher-order response function captures how the lower-capability player uses their (less accurate) perception of the payoff function to reason about the (more accurate) possible perceptions of the higher-capability opponent. This response function characterizes, for each possible perception of the higher-capability player (from the perspective of the lower-capability player), the best response of the higher capability player for that perception. Since the domain of the response function can be exponentially large or even infinite, finding one equilibrium may be computationally intractable or even undecidable. Nevertheless, we show that for any $\epsilon$, there exists an $\epsilon$-equilibrium with a compact, tractable representation whose size is independent of the size of the response function's domain. We further identify classes of zero-sum limited-perception games in which finding an equilibrium becomes a (typically tractable) nonsmooth convex optimization problem.


title: A Separation in Heavy-Tailed Sampling: Gaussian vs. Stable Oracles for Proximal Samplers

abstract: We study the complexity of heavy-tailed sampling and present a separation result in terms of obtaining high-accuracy versus low-accuracy guarantees i.e., samplers that require only $O(\log(1/\varepsilon))$ versus $\Omega(\text{poly}(1/\varepsilon))$ iterations to output a sample which is $\varepsilon$-close to the target in $\chi^2$-divergence. Our results are presented for proximal samplers that are based on Gaussian versus stable oracles. We show that proximal samplers based on the Gaussian oracle have a fundamental barrier in that they necessarily achieve only low-accuracy guarantees when sampling from a class of heavy-tailed targets. In contrast, proximal samplers based on the stable oracle exhibit high-accuracy guarantees, thereby overcoming the aforementioned limitation. We also prove lower bounds for samplers under the stable oracle and show that our upper bounds cannot be fundamentally improved.


title: Zooming in on discrete space

abstract: Although we lack complete understanding of quantum aspects of gravitation, it is usually agreed, using general arguments, that a final quantum gravity theory will endow space and time with some (fundamental or effective) notion of discreteness. This granular character is supposed to lie on space and time scales of $l_P \sim 10^{-33}$ cm and $\tau_P\sim 10^{-42}$ s, respectively -- the Planck scale -- , far beyond any hope of direct assessment. Here, by modeling displacements of particles on a discrete underlying space as Poisson processes, we speculate on the possibility of amplifying the effects of space discreteness (if existent) by several orders of magnitude, using the statistical variance of correlated displacements of particles/systems with very different masses. Although still out of reach by current technology, the analysis presented here suggests that it may be possible to see hints of space(time) discreteness at larger scales than one would usually expect.


title: CARL: A Framework for Equivariant Image Registration

abstract: Image registration estimates spatial correspondences between a pair of images. These estimates are typically obtained via numerical optimization or regression by a deep network. A desirable property of such estimators is that a correspondence estimate (e.g., the true oracle correspondence) for an image pair is maintained under deformations of the input images. Formally, the estimator should be equivariant to a desired class of image transformations. In this work, we present careful analyses of the desired equivariance properties in the context of multi-step deep registration networks. Based on these analyses we 1) introduce the notions of $[U,U]$ equivariance (network equivariance to the same deformations of the input images) and $[W,U]$ equivariance (where input images can undergo different deformations); we 2) show that in a suitable multi-step registration setup it is sufficient for overall $[W,U]$ equivariance if the first step has $[W,U]$ equivariance and all others have $[U,U]$ equivariance; we 3) show that common displacement-predicting networks only exhibit $[U,U]$ equivariance to translations instead of the more powerful $[W,U]$ equivariance; and we 4) show how to achieve multi-step $[W,U]$ equivariance via a coordinate-attention mechanism combined with displacement-predicting refinement layers (CARL). Overall, our approach obtains excellent practical registration performance on several 3D medical image registration tasks and outperforms existing unsupervised approaches for the challenging problem of abdomen registration.


title: Oracle-Efficient Reinforcement Learning for Max Value Ensembles

abstract: Reinforcement learning (RL) in large or infinite state spaces is notoriously challenging, both theoretically (where worst-case sample and computational complexities must scale with state space cardinality) and experimentally (where function approximation and policy gradient techniques often scale poorly and suffer from instability and high variance). One line of research attempting to address these difficulties makes the natural assumption that we are given a collection of heuristic base or $\textit{constituent}$ policies upon which we would like to improve in a scalable manner. In this work we aim to compete with the $\textit{max-following policy}$, which at each state follows the action of whichever constituent policy has the highest value. The max-following policy is always at least as good as the best constituent policy, and may be considerably better. Our main result is an efficient algorithm that learns to compete with the max-following policy, given only access to the constituent policies (but not their value functions). In contrast to prior work in similar settings, our theoretical results require only the minimal assumption of an ERM oracle for value function approximation for the constituent policies (and not the global optimal policy or the max-following policy itself) on samplable distributions. We illustrate our algorithm's experimental effectiveness and behavior on several robotic simulation testbeds.


title: PP-SAM: Perturbed Prompts for Robust Adaptation of Segment Anything Model for Polyp Segmentation

abstract: The Segment Anything Model (SAM), originally designed for general-purpose segmentation tasks, has been used recently for polyp segmentation. Nonetheless, fine-tuning SAM with data from new imaging centers or clinics poses significant challenges. This is because this necessitates the creation of an expensive and time-intensive annotated dataset, along with the potential for variability in user prompts during inference. To address these issues, we propose a robust fine-tuning technique, PP-SAM, that allows SAM to adapt to the polyp segmentation task with limited images. To this end, we utilize variable perturbed bounding box prompts (BBP) to enrich the learning context and enhance the model's robustness to BBP perturbations during inference. Rigorous experiments on polyp segmentation benchmarks reveal that our variable BBP perturbation significantly improves model resilience. Notably, on Kvasir, 1-shot fine-tuning boosts the DICE score by 20% and 37% with 50 and 100-pixel BBP perturbations during inference, respectively. Moreover, our experiments show that 1-shot, 5-shot, and 10-shot PP-SAM with 50-pixel perturbations during inference outperform a recent state-of-the-art (SOTA) polyp segmentation method by 26%, 7%, and 5% DICE scores, respectively. Our results motivate the broader applicability of our PP-SAM for other medical imaging tasks with limited samples. Our implementation is available at https://github.com/SLDGroup/PP-SAM.


title: A Study on Magnetic-sensitivity Wavelength Position of the Working Line Used by the Full-Disk Magnetograph onboard the Advanced Space based Solar Observatory (ASO-S/FMG)

abstract: Utilizing data from the $Solar$ $Magnetism$ and $Activity$ $Telescope$ (SMAT), analytical solutions of polarized radiative transfer equations, and in-orbit test data from the Full-disk Magnetograph (FMG) onboard the Advanced Space based Solar Observatory (ASO-S), this study reveals the magnetic-sensitivity spectral positions for the Fe {\sc i} $\lambda$5234.19 A, working line used by FMG. From the experimental data of SMAT, it is found that the most sensitivity position is located at the line center for linear polarization (Stokes-Q/U), while it is about -0.07 A away from the line center for circular polarization (Stokes-V). Moreover, both the theoretical analysis and the in-orbit test data analysis of FMG prove again the above results. Additionally, the theoretical analysis suggests the presence of distinct spectral pockets (centered at 0.08-0.15 A) from the line, harboring intense magnetic sensitivity across all three Stokes parameters. Striking a balance between high sensitivity for both linear and circular polarization while capturing additional valuable information, a spectral position of -0.08 A emerges as the champion for routine FMG magnetic-field observations.


title: Free energy formulas for confined nematic liquid crystals based on analogies with Kirchhoff-Routh theory in vortex dynamics

abstract: Active nematics are influenced by alignment angle singularities called topological defects. The localization of these defects is of major interest for biological applications. The total distortion of alignment angles due to defects is evaluated using Frank free energy, which is one of the criteria used to determine the location and stability of these defects. Previous work used the line integrals of a complex potential associated with the alignments for the energy calculation (Miyazako and Nara, R. Soc. Open Sci., 2022), which has a high computational cost. We propose analytical formulas for the free energy in the presence of multiple topological defects in confined geometries. The formulas derived here are an analogue of Kirchhoff-Routh functions in vortex dynamics. The proposed formulas are explicit with respect to the defect locations and conformal maps, which enables the explicit calculation of the energy extrema. The formulas are applied to calculate the locations of defects in so-called doublets and triplets by solving simple polynomial formulas. A stability analysis is also conducted to detect whether defect pairs with charges $\pm 1/2$ are stable or unstable in triplet regions. Our numerical results are shown to match the experimental results (Ienaga {\em et al.,} Soft Matter, 2023).


title: Tuning the Electronic and Optical Properties of Impurity-Engineered Two-Dimensional Graphullerene Half-Semiconductors

abstract: A novel material consisting of a monolayer of C$_{60}$ buckyballs with hexagonal symmetry has recently been observed experimentally, named graphullerene. In this study, we present a comprehensive \textit{ab-initio} theoretical analysis of the electronic and optical properties of both pristine and impurity-engineered monolayer graphullerene using spin-dependent density functional theory (spin-DFT). Our findings reveal that graphullerene is a direct band gap semiconductor with a band gap of approximately 1.5 eV at the $\Gamma$ point, agreeing well with experimental data. Notably, we demonstrate that by adding impurities, in particular substitutional nitrogen, substitutional boron, or adsorbent hydrogen, to graphullerene results in the formation of spin-dependent deep donor and deep acceptor levels, thereby giving rise to a variety of half-semiconductors. All the impurities exhibit a magnetic moment of approximately $\mu_B$ per impurity. This impurity engineering enables the tuning of spin-polarized exciton properties in graphullerene, with spin-dependent band gap energies ranging from 0.43 eV ($\lambda \sim$ 2.9 $\mu$m) to 1.5 eV ($\lambda \sim$ 820 nm), covering the near-infrared (NIR) and short-wavelength infrared (SWIR) regimes. Our results suggest that both pristine and impurity-engineered graphullerene have significant potential for the development of carbon-based 2D semiconductor spintronic and opto-spintronic devices.


title: CH_3OH and Its Deuterated Species in the Disk/Envelope System of the Low-Mass Protostellar Source B335

abstract: Deuterium fractionation in the closest vicinity of a protostar is important in understanding its potential heritage to a planetary system. Here, we have detected the spectral line emission of CH3OH and its three deuterated species, CH2DOH, CHD2OH, and CH3OD, toward the low-mass protostellar source B335 at a resolution of 0.''03 (5 au) with Atacama Large Millimeter/submillimeter Array. They have a ring distribution within the radius of 24 au with the intensity depression at the continuum peak. We derive the column densities and abundance ratios of the above species at 6 positions in the disk/envelope system as well as the continuum peak. The D/H ratio of CH3OH is ~[0.03-0.13], which is derived by correcting the statistical weight of 3 for CH2DOH. The [CHD2OH]/[CH2DOH] ratio is derived to be higher ([0.14-0.29]). On the other hand, the [CH2DOH]/[CH3OD] ratio ([4.9-15]) is higher than the statistical ratio of 3, and is comparable to those reported for other low-mass sources. We study the physical structure on a few au scale in B335 by analyzing the CH3OH (183,15-182,16, A) and HCOOH (120,12-110,11) line emission. Velocity structures of these lines are reasonably explained as the infalling-rotating motion. The protostellar mass and the upper limit to centrifugal barrier are thus derived to be 0.03-0.07 M_{\odot} and <7 au, respectively, showing that B335 harbors a young protostar with a tiny disk structure. Such youth of the protostar may be related to the relatively high [CH2DOH]/[CH3OH] ratio.


title: On Rankin-Cohen Brackets of Hecke Eigenforms and Modular Forms of Half-Integral Weight

abstract: We generalize the linear relation formula between the square of normalized Hecke eigenforms of weight $k$ and normalized Hecke eigenforms of weight $2k$, to Rankin-Cohen brackets of general degree. As an ingredient of the proof, we also generalize a formula of Zagier on the Petersson inner product of Rankin-Cohen brackets involving Eisenstein series.


title: Ecosystem of Large Language Models for Code

abstract: The availability of vast amounts of publicly accessible data of source code and the advances in modern language models, coupled with increasing computational resources, have led to a remarkable surge in the development of large language models for code (LLM4Code, for short). The interaction between code datasets and models gives rise to a complex ecosystem characterized by intricate dependencies that are worth studying. This paper introduces a pioneering analysis of the code model ecosystem. Utilizing Hugging Face -- the premier hub for transformer-based models -- as our primary source, we curate a list of datasets and models that are manually confirmed to be relevant to software engineering. By analyzing the ecosystem, we first identify the popular and influential datasets, models, and contributors. The popularity is quantified by various metrics, including the number of downloads, the number of likes, the number of reuses, etc. The ecosystem follows a power-law distribution, indicating that users prefer widely recognized models and datasets. Then, we manually categorize how models in the ecosystem are reused into nine categories, analyzing prevalent model reuse practices. The top 3 most popular reuse types are fine-tuning, architecture sharing, and quantization. We also explore the practices surrounding the publication of LLM4Code, specifically focusing on documentation practice and license selection. We find that the documentation in the ecosystem contains less information than that in general artificial intelligence (AI)-related repositories hosted on GitHub. Additionally, the license usage is also different from other software repositories. Models in the ecosystem adopt some AI-specific licenses, e.g., RAIL (Responsible AI Licenses) and AI model license agreement.


title: Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective

abstract: The two-stage fine-tuning (FT) method, linear probing then fine-tuning (LP-FT), consistently outperforms linear probing (LP) and FT alone in terms of accuracy for both in-distribution (ID) and out-of-distribution (OOD) data. This success is largely attributed to the preservation of pre-trained features, achieved through a near-optimal linear head obtained during LP. However, despite the widespread use of large language models, the exploration of complex architectures such as Transformers remains limited. In this paper, we analyze the training dynamics of LP-FT for classification models on the basis of the neural tangent kernel (NTK) theory. Our analysis decomposes the NTK matrix into two components, highlighting the importance of the linear head norm alongside the prediction accuracy at the start of the FT stage. We also observe a significant increase in the linear head norm during LP, stemming from training with the cross-entropy (CE) loss, which effectively minimizes feature changes. Furthermore, we find that this increased norm can adversely affect model calibration, a challenge that can be addressed by temperature scaling. Additionally, we extend our analysis with the NTK to the low-rank adaptation (LoRA) method and validate its effectiveness. Our experiments with a Transformer-based model on natural language processing tasks across multiple benchmarks confirm our theoretical analysis and demonstrate the effectiveness of LP-FT in fine-tuning language models. Code is available at https://github.com/tom4649/lp-ft_ntk.


title: Hypergraph Laplacian Eigenmaps and Face Recognition Problems

abstract: Face recognition is a very important topic in data science and biometric security research areas. It has multiple applications in military, finance, and retail, to name a few. In this paper, the novel hypergraph Laplacian Eigenmaps will be proposed and combine with the k nearest-neighbor method and/or with the kernel ridge regression method to solve the face recognition problem. Experimental results illustrate that the accuracy of the combination of the novel hypergraph Laplacian Eigenmaps and one specific classification system is similar to the accuracy of the combination of the old symmetric normalized hypergraph Laplacian Eigenmaps method and one specific classification system.


title: DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models

abstract: Pretrained diffusion models (DMs) have recently been popularly used in solving inverse problems (IPs). The existing methods mostly interleave iterative steps in the reverse diffusion process and iterative steps to bring the iterates closer to satisfying the measurement constraint. However, such interleaving methods struggle to produce final results that look like natural objects of interest (i.e., manifold feasibility) and fit the measurement (i.e., measurement feasibility), especially for nonlinear IPs. Moreover, their capabilities to deal with noisy IPs with unknown types and levels of measurement noise are unknown. In this paper, we advocate viewing the reverse process in DMs as a function and propose a novel plug-in method for solving IPs using pretrained DMs, dubbed DMPlug. DMPlug addresses the issues of manifold feasibility and measurement feasibility in a principled manner, and also shows great potential for being robust to unknown types and levels of noise. Through extensive experiments across various IP tasks, including two linear and three nonlinear IPs, we demonstrate that DMPlug consistently outperforms state-of-the-art methods, often by large margins especially for nonlinear IPs. The code is available at https://github.com/sun-umn/DMPlug.


title: Realization of 2/3-layer transition metal dichalcogenides

abstract: Layered van der Waals transition metal dichalcogenides (TMDCs), generally composed of three atomic X-M-X planes in each layer (M = transition metal, X = chalcogen), provide versatile platforms for exploring diverse quantum phenomena. In each MX2 layer, the M-X bonds are predominantly covalent in nature, as a result, the cleavage of TMDC crystals always occurring between the layers. Here we report the controllable realization of fractional-layer WTe2 via an in-situ scanning tunnelling microscopy (STM) tip manipulation technique. By applying STM tip pulses, hundreds of the topmost Te atoms are removed to form a nanoscale monolayer Te pit in the 1T'-WTe2, thus realizing a brand-new 2/3-layer WTe2. Such a unique configuration undergoes a spontaneous atomic reconstruction, yielding an energy-dependent unidirectional charge-density-wave state with the wavevector and geometry quite distinct from that of pristine 1T'-WTe2. Our results expand the conventional understanding of the TMDCs and are expected to stimulate the research on extraordinary structures and properties based on fractional-layer TMDCs.


title: LLM-Based Cooperative Agents using Information Relevance and Plan Validation

abstract: We address the challenge of multi-agent cooperation, where agents achieve a common goal by interacting with a 3D scene and cooperating with decentralized agents under complex partial observations. This involves managing communication costs and optimizing interaction trajectories in dynamic environments. Our research focuses on three primary limitations of existing cooperative agent systems. Firstly, current systems demonstrate inefficiency in managing acquired information through observation, resulting in declining planning performance as the environment becomes more complex with additional objects or goals. Secondly, the neglect of false plans in partially observable settings leads to suboptimal cooperative performance, as agents struggle to adapt to environmental changes influenced by the unseen actions of other agents. Lastly, the failure to incorporate spatial data into decision-making processes restricts the agent's ability to construct optimized trajectories. To overcome these limitations, we propose the RElevance and Validation-Enhanced Cooperative Language Agent (REVECA), a novel cognitive architecture powered by GPT-3.5. REVECA leverages relevance assessment, plan validation, and spatial information to enhance the efficiency and robustness of agent cooperation in dynamic and partially observable environments while minimizing continuous communication costs and effectively managing irrelevant dummy objects. Our extensive experiments demonstrate the superiority of REVECA over previous approaches, including those driven by GPT-4.0. Additionally, a user study highlights REVECA's potential for achieving trustworthy human-AI cooperation. We expect that REVECA will have significant applications in gaming, XR applications, educational tools, and humanoid robots, contributing to substantial economic, commercial, and academic advancements.


title: Model Ensembling for Constrained Optimization

abstract: There is a long history in machine learning of model ensembling, beginning with boosting and bagging and continuing to the present day. Much of this history has focused on combining models for classification and regression, but recently there is interest in more complex settings such as ensembling policies in reinforcement learning. Strong connections have also emerged between ensembling and multicalibration techniques. In this work, we further investigate these themes by considering a setting in which we wish to ensemble models for multidimensional output predictions that are in turn used for downstream optimization. More precisely, we imagine we are given a number of models mapping a state space to multidimensional real-valued predictions. These predictions form the coefficients of a linear objective that we would like to optimize under specified constraints. The fundamental question we address is how to improve and combine such models in a way that outperforms the best of them in the downstream optimization problem. We apply multicalibration techniques that lead to two provably efficient and convergent algorithms. The first of these (the white box approach) requires being given models that map states to output predictions, while the second (the \emph{black box} approach) requires only policies (mappings from states to solutions to the optimization problem). For both, we provide convergence and utility guarantees. We conclude by investigating the performance and behavior of the two algorithms in a controlled experimental setting.


title: Multi-answer Constrained Optimal Querying: Maximum Information Gain Coding

abstract: As the rapidly developments of artificial intelligence and machine learning, behavior tree design in multiagent system or AI game become more important. The behavior tree design problem is highly related to the source coding in information theory. "Twenty Questions" problem is a typical example for the behavior tree design, usually used to explain the source coding application in information theory and can be solved by Huffman coding. In some realistic scenarios, there are some constraints on the asked questions. However, for general question set, finding the minimum expected querying length is an open problem, belongs to NP-hard. Recently, a new coding scheme has been proposed to provide a near optimal solution for binary cases with some constraints, named greedy binary separation coding (GBSC). In this work, we shall generalize it to D-ary cases and propose maximum information gain coding (MIGC) approach to solve the multi-answer decision constrained querying problem. The optimality of the proposed MIGC is discussed in theory. Later on, we also apply MIGC to discuss three practical scenarios and showcase that MIGC has better performance than GBSC and Shannon Coding in terms of bits persymbol.


title: Adaptive VIO: Deep Visual-Inertial Odometry with Online Continual Learning

abstract: Visual-inertial odometry (VIO) has demonstrated remarkable success due to its low-cost and complementary sensors. However, existing VIO methods lack the generalization ability to adjust to different environments and sensor attributes. In this paper, we propose Adaptive VIO, a new monocular visual-inertial odometry that combines online continual learning with traditional nonlinear optimization. Adaptive VIO comprises two networks to predict visual correspondence and IMU bias. Unlike end-to-end approaches that use networks to fuse the features from two modalities (camera and IMU) and predict poses directly, we combine neural networks with visual-inertial bundle adjustment in our VIO system. The optimized estimates will be fed back to the visual and IMU bias networks, refining the networks in a self-supervised manner. Such a learning-optimization-combined framework and feedback mechanism enable the system to perform online continual learning. Experiments demonstrate that our Adaptive VIO manifests adaptive capability on EuRoC and TUM-VI datasets. The overall performance exceeds the currently known learning-based VIO methods and is comparable to the state-of-the-art optimization-based methods.


title: CHESS: Contextual Harnessing for Efficient SQL Synthesis

abstract: Utilizing large language models (LLMs) for transforming natural language questions into SQL queries (text-to-SQL) is a promising yet challenging approach, particularly when applied to real-world databases with complex and extensive schemas. In particular, effectively incorporating data catalogs and database values for SQL generation remains an obstacle, leading to suboptimal solutions. We address this problem by proposing a new pipeline that effectively retrieves relevant data and context, selects an efficient schema, and synthesizes correct and efficient SQL queries. To increase retrieval precision, our pipeline introduces a hierarchical retrieval method leveraging model-generated keywords, locality-sensitive hashing indexing, and vector databases. Additionally, we have developed an adaptive schema pruning technique that adjusts based on the complexity of the problem and the model's context size. Our approach generalizes to both frontier proprietary models like GPT-4 and open-source models such as Llama-3-70B. Through a series of ablation studies, we demonstrate the effectiveness of each component of our pipeline and its impact on the end-to-end performance. Our method achieves new state-of-the-art performance on the cross-domain challenging BIRD dataset.


title: Symmetry-Informed Governing Equation Discovery

abstract: Despite the advancements in learning governing differential equations from observations of dynamical systems, data-driven methods are often unaware of fundamental physical laws, such as frame invariance. As a result, these algorithms may search an unnecessarily large space and discover equations that are less accurate or overly complex. In this paper, we propose to leverage symmetry in automated equation discovery to compress the equation search space and improve the accuracy and simplicity of the learned equations. Specifically, we derive equivariance constraints from the time-independent symmetries of ODEs. Depending on the types of symmetries, we develop a pipeline for incorporating symmetry constraints into various equation discovery algorithms, including sparse regression and genetic programming. In experiments across a diverse range of dynamical systems, our approach demonstrates better robustness against noise and recovers governing equations with significantly higher probability than baselines without symmetry.


title: Electron Transport along Screw Dislocations in a Strong Topological Insulator

abstract: In a three-dimensional strong topological insulator, gapless helical surface states appear everywhere on its surface. In the presence of a screw dislocation, gapless helical modes also appear in the vicinity of the corresponding dislocation line. Let us focus on a case where a pair of screw dislocations connects the top and bottom surfaces of a strong topological insulator with a shape of rectangular parallelepiped. The dislocation-induced helical modes are expected to act as one-dimensional conduction channels connecting the top and bottom surfaces. To examine this expectation, we calculate the two-terminal conductance between a pair of electrodes placed on the top and bottom surfaces. We found that the dislocation-induced helical modes and the helical surface states on the side surface contribute to the two-terminal conductance. The contribution of the dislocation-induced helical modes becomes more dominant than that of the helical surface states in certain situations.


title: Geometrically Thick Disks around Kerr Black Holes in a Swirling Universe

abstract: We investigate geometrically thick disks around Kerr black holes in a swirling universe. This stationary and axisymmetric spacetime is composed of a Kerr black hole, which is immersed in a swirling background. The swirling background is characterized by an odd $\mathcal{Z}_2$ symmetry, where the northern and southern hemispheres are rotating in opposite directions. The Kerr and swirling rotations can lead to the emergence of complex spin-spin interactions, which heavily influence the spacetime properties. In order to study this influence, we analyze prograde as well as retrograde circular orbits and geometrically thick disks for different spacetime solutions, which are classified by their Kerr parameter $a$ and the swirling parameter $j$. We find stabilizing effects on prograde circular orbits and destabilizing effects on retrograde circular orbits, which originate from the spin-spin interaction and depend mainly on the Kerr rotation. Furthermore, the background rotation leads to an emergence of static orbits and the symmetry breaking regarding the equatorial plane causes a vertical distribution of the circular orbits and thick torus solutions. Due to the destabilizing effect of the swirling rotation, an outer marginally stable orbit appears, which heavily downsizes the parameter space for disk solutions. Dependent on the Keplerian specific angular momentum distribution of the spacetime and on the disk momentum $\ell_0$, different types of disk are possible which we classify and which differ in terms of the complex disk dynamics that could arise due to the presence of an inner and outer cusp or just an outer cusp.


title: Greedy Growing Enables High-Resolution Pixel-Based Diffusion Models

abstract: We address the long-standing problem of how to learn effective pixel-based image diffusion models at scale, introducing a remarkably simple greedy growing method for stable training of large-scale, high-resolution models. without the needs for cascaded super-resolution components. The key insight stems from careful pre-training of core components, namely, those responsible for text-to-image alignment {\it vs.} high-resolution rendering. We first demonstrate the benefits of scaling a {\it Shallow UNet}, with no down(up)-sampling enc(dec)oder. Scaling its deep core layers is shown to improve alignment, object structure, and composition. Building on this core model, we propose a greedy algorithm that grows the architecture into high-resolution end-to-end models, while preserving the integrity of the pre-trained representation, stabilizing training, and reducing the need for large high-resolution datasets. This enables a single stage model capable of generating high-resolution images without the need of a super-resolution cascade. Our key results rely on public datasets and show that we are able to train non-cascaded models up to 8B parameters with no further regularization schemes. Vermeer, our full pipeline model trained with internal datasets to produce 1024x1024 images, without cascades, is preferred by 44.0% vs. 21.4% human evaluators over SDXL.


title: Graphon Particle Systems, Part I: Spatio-Temporal Approximation and Law of Large Numbers

abstract: We study a class of graphon particle systems with time-varying random coefficients. In a graphon particle system, the interactions among particles are characterized by the coupled mean field terms through an underlying graphon and the randomness of the coefficients comes from the stochastic processes associated with the particle labels. By constructing two-level approximated sequences converging in 2-Wasserstein distance, we prove the existence and uniqueness of the solution to the system. Besides, by constructing two-level approximated functions converging to the graphon mean field terms, we establish the law of large numbers, which reveals that if the number of particles tends to infinity and the discretization step tends to zero, then the discrete-time interacting particle system over the large-scale network converges to the graphon particle system. As a byproduct, we discover that the graphon particle system can describe the dynamic evolution of the distributed stochastic gradient descent algorithm over the large-scale network and prove that if the gradients of the local cost functions are Lipschitz continuous, then the graphon particle system can be regarded as the spatio-temporal approximation of the discrete-time distributed stochastic gradient descent algorithm as the number of network nodes tends to infinity and the algorithm step size tends to zero.


title: Masked Face Recognition with Generative-to-Discriminative Representations

abstract: Masked face recognition is important for social good but challenged by diverse occlusions that cause insufficient or inaccurate representations. In this work, we propose a unified deep network to learn generative-to-discriminative representations for facilitating masked face recognition. To this end, we split the network into three modules and learn them on synthetic masked faces in a greedy module-wise pretraining manner. First, we leverage a generative encoder pretrained for face inpainting and finetune it to represent masked faces into category-aware descriptors. Attribute to the generative encoder's ability in recovering context information, the resulting descriptors can provide occlusion-robust representations for masked faces, mitigating the effect of diverse masks. Then, we incorporate a multi-layer convolutional network as a discriminative reformer and learn it to convert the category-aware descriptors into identity-aware vectors, where the learning is effectively supervised by distilling relation knowledge from off-the-shelf face recognition model. In this way, the discriminative reformer together with the generative encoder serves as the pretrained backbone, providing general and discriminative representations towards masked faces. Finally, we cascade one fully-connected layer following by one softmax layer into a feature classifier and finetune it to identify the reformed identity-aware vectors. Extensive experiments on synthetic and realistic datasets demonstrate the effectiveness of our approach in recognizing masked faces.


title: Addressing Discretization-Induced Bias in Demographic Prediction

abstract: Racial and other demographic imputation is necessary for many applications, especially in auditing disparities and outreach targeting in political campaigns. The canonical approach is to construct continuous predictions -- e.g., based on name and geography -- and then to $\textit{discretize}$ the predictions by selecting the most likely class (argmax). We study how this practice produces $\textit{discretization bias}$. In particular, we show that argmax labeling, as used by a prominent commercial voter file vendor to impute race/ethnicity, results in a substantial under-count of African-American voters, e.g., by 28.2% points in North Carolina. This bias can have substantial implications in downstream tasks that use such labels. We then introduce a $\textit{joint optimization}$ approach -- and a tractable $\textit{data-driven thresholding}$ heuristic -- that can eliminate this bias, with negligible individual-level accuracy loss. Finally, we theoretically analyze discretization bias, show that calibrated continuous models are insufficient to eliminate it, and that an approach such as ours is necessary. Broadly, we warn researchers and practitioners against discretizing continuous demographic predictions without considering downstream consequences.


title: Transport of Algebraic Structure to Latent Embeddings

abstract: Machine learning often aims to produce latent embeddings of inputs which lie in a larger, abstract mathematical space. For example, in the field of 3D modeling, subsets of Euclidean space can be embedded as vectors using implicit neural representations. Such subsets also have a natural algebraic structure including operations (e.g., union) and corresponding laws (e.g., associativity). How can we learn to "union" two sets using only their latent embeddings while respecting associativity? We propose a general procedure for parameterizing latent space operations that are provably consistent with the laws on the input space. This is achieved by learning a bijection from the latent space to a carefully designed mirrored algebra which is constructed on Euclidean space in accordance with desired laws. We evaluate these structural transport nets for a range of mirrored algebras against baselines that operate directly on the latent space. Our experiments provide strong evidence that respecting the underlying algebraic structure of the input space is key for learning accurate and self-consistent operations.


title: One-dimensional SRBM with discontinuous state-dependent drift and variance and its stationary distribution

abstract: A semi-martingale reflecting Brownian motion is a popular process for diffusion approximations of queueing models including their networks. In this paper, we are concerned with the case that it lives on the nonnegative half-line but its variance and drift discontinuously change depending on its position. This reflecting diffusion process naturally arises from a state-dependent single server queue, studied by the author. Our main interest is in its stationary distribution, which is important for application. We define this reflecting diffusion process as the solution of a stochastic integral equation. Its existence is recently shown by Atar, Castiel and Reiman (2022,2023) under a more general condition than we have studied here, and its uniqueness is obtained also by them under one of the conditions of this paper. We here give elementary proofs for both of them. In this paper, we consider its stability, that is, the condition that the reflecting diffusion process has a stationary distribution, and derive its stationary distribution under the stability condition. Our basic tools are the generalized Ito formula for a convex function and local time. Proofs are rather elementary once it is understood how to use the Ito formula and local time.


title: Study of Robust Direction Finding Based on Joint Sparse Representation

abstract: Standard Direction of Arrival (DOA) estimation methods are typically derived based on the Gaussian noise assumption, making them highly sensitive to outliers. Therefore, in the presence of impulsive noise, the performance of these methods may significantly deteriorate. In this paper, we model impulsive noise as Gaussian noise mixed with sparse outliers. By exploiting their statistical differences, we propose a novel DOA estimation method based on sparse signal recovery (SSR). Furthermore, to address the issue of grid mismatch, we utilize an alternating optimization approach that relies on the estimated outlier matrix and the on-grid DOA estimates to obtain the off-grid DOA estimates. Simulation results demonstrate that the proposed method exhibits robustness against large outliers.


title: Reframing the Relationship in Out-of-Distribution Detection

abstract: The remarkable achievements of Large Language Models (LLMs) have captivated the attention of both academia and industry, transcending their initial role in dialogue generation. The utilization of LLMs as intermediary agents in various tasks has yielded promising results, sparking a wave of innovation in artificial intelligence. Building on these breakthroughs, we introduce a novel approach that integrates the agent paradigm into the Out-of-distribution (OOD) detection task, aiming to enhance its robustness and adaptability. Our proposed method, Concept Matching with Agent (CMA), employs neutral prompts as agents to augment the CLIP-based OOD detection process. These agents function as dynamic observers and communication hubs, interacting with both In-distribution (ID) labels and data inputs to form vector triangle relationships. This triangular framework offers a more nuanced approach than the traditional binary relationship, allowing for better separation and identification of ID and OOD inputs. Our extensive experimental results showcase the superior performance of CMA over both zero-shot and training-required methods in a diverse array of real-world scenarios.


title: Oblivious Monitoring for Discrete-Time STL via Fully Homomorphic Encryption

abstract: When monitoring a cyber-physical system (CPS) from a remote server, keeping the monitored data secret is crucial, particularly when they contain sensitive information, e.g., biological or location data. Recently, Banno et al. (CAV'22) proposed a protocol for online LTL monitoring that keeps data concealed from the server using Fully Homomorphic Encryption (FHE). We build on this protocol to allow arithmetic operations over encrypted values, e.g., to compute a safety measurement combining distance, velocity, and so forth. Overall, our protocol enables oblivious online monitoring of discrete-time real-valued signals against signal temporal logic (STL) formulas. Our protocol combines two FHE schemes, CKKS and TFHE, leveraging their respective strengths. We employ CKKS to evaluate arithmetic predicates in STL formulas while utilizing TFHE to process them using a DFA derived from the STL formula. We conducted case studies on monitoring blood glucose levels and vehicles' behavior against the Responsibility-Sensitive Safety (RSS) rules. Our results suggest the practical relevance of our protocol.


title: Time-dependent complex variable solution on quasi three-dimensional shallow tunnelling in gravititational geomaterial with reasonable far-field displacement

abstract: Three-dimensional effect of tunnel face and gravitational excavation generally occur in shallow tunnelling, which are nevertheless not adequately considered in present complex variable solutions. In this paper, a new time-dependent complex variable solution on quasi three-dimensional shallow tunnelling in gravitational geomaterial is derived, and the far-field displacement singularity is eliminated by fixed far-field ground surface in the whole excavation time span. With an equivalent coefficient of three-dimensional effect, the quasi three-dimensional shallow tunnelling is transformed into a plane strain problem with time-dependent virtual traction along tunnel periphery. The mixed boundaries of fixed far-field ground surface and nearby free segment form a homogenerous Riemann-Hilbert problem with extra constraints of the virtual traction along tunnel periphery, which is simultaneously solved using an iterative linear system with good numerical stability. The mixed boundary conditions along the ground surface in the whole excavation time span are well satisified in a numerical case, which is further examined by comparing with corresponding finite element solution. The results are in good agreements, and the proposed solution illustrates high efficiency. More discussions are made on excavation rate, viscosity, and solution convergence. A latent paradox is disclosed for objectivity.


title: Learning phase transitions by siamese neural network

abstract: The wide application of machine learning (ML) techniques in statistics physics has presented new avenues for research in this field. In this paper, we introduce a semi-supervised learning method based on Siamese Neural Networks (SNN), trying to explore the potential of neural network (NN) in the study of critical behaviors beyond the approaches of supervised and unsupervised learning. By focusing on the (1+1) dimensional bond directed percolation (DP) model of nonequilibrium phase transition, we use the SNN to predict the critical values and critical exponents of the system. Different from traditional ML methods, the input of SNN is a set of configuration data pairs and the output prediction is similarity, which prompts to find an anchor point of data for pair comparison during the test. In our study, during test we set different bond probability $p$ as anchors, and discuss the impact of the configurations at this anchors on predictions. More, we use an iterative method to find the optimal training interval to make the algorithm more efficient, and the prediction results are comparable to other ML methods.


title: Physics informed cell representations for variational formulation of multiscale problems

abstract: With the rapid advancement of graphical processing units, Physics-Informed Neural Networks (PINNs) are emerging as a promising tool for solving partial differential equations (PDEs). However, PINNs are not well suited for solving PDEs with multiscale features, particularly suffering from slow convergence and poor accuracy. To address this limitation of PINNs, this article proposes physics-informed cell representations for resolving multiscale Poisson problems using a model architecture consisting of multilevel multiresolution grids coupled with a multilayer perceptron (MLP). The grid parameters (i.e., the level-dependent feature vectors) and the MLP parameters (i.e., the weights and biases) are determined using gradient-descent based optimization. The variational (weak) form based loss function accelerates computation by allowing the linear interpolation of feature vectors within grid cells. This cell-based MLP model also facilitates the use of a decoupled training scheme for Dirichlet boundary conditions and a parameter-sharing scheme for periodic boundary conditions, delivering superior accuracy compared to conventional PINNs. Furthermore, the numerical examples highlight improved speed and accuracy in solving PDEs with nonlinear or high-frequency boundary conditions and provide insights into hyperparameter selection. In essence, by cell-based MLP model along with the parallel tiny-cuda-nn library, our implementation improves convergence speed and numerical accuracy.


title: ARC: A Generalist Graph Anomaly Detector with In-Context Learning

abstract: Graph anomaly detection (GAD), which aims to identify abnormal nodes that differ from the majority within a graph, has garnered significant attention. However, current GAD methods necessitate training specific to each dataset, resulting in high training costs, substantial data requirements, and limited generalizability when being applied to new datasets and domains. To address these limitations, this paper proposes ARC, a generalist GAD approach that enables a ``one-for-all'' GAD model to detect anomalies across various graph datasets on-the-fly. Equipped with in-context learning, ARC can directly extract dataset-specific patterns from the target dataset using few-shot normal samples at the inference stage, without the need for retraining or fine-tuning on the target dataset. ARC comprises three components that are well-crafted for capturing universal graph anomaly patterns: 1) smoothness-based feature Alignment module that unifies the features of different datasets into a common and anomaly-sensitive space; 2) ego-neighbor Residual graph encoder that learns abnormality-related node embeddings; and 3) cross-attentive in-Context anomaly scoring module that predicts node abnormality by leveraging few-shot normal samples. Extensive experiments on multiple benchmark datasets from various domains demonstrate the superior anomaly detection performance, efficiency, and generalizability of ARC.


title: Balancing User Preferences by Social Networks: A Condition-Guided Social Recommendation Model for Mitigating Popularity Bias

abstract: Social recommendation models weave social interactions into their design to provide uniquely personalized recommendation results for users. However, social networks not only amplify the popularity bias in recommendation models, resulting in more frequent recommendation of hot items and fewer long-tail items, but also include a substantial amount of redundant information that is essentially meaningless for the model's performance. Existing social recommendation models fail to address the issues of popularity bias and the redundancy of social information, as they directly characterize social influence across the entire social network without making targeted adjustments. In this paper, we propose a Condition-Guided Social Recommendation Model (named CGSoRec) to mitigate the model's popularity bias by denoising the social network and adjusting the weights of user's social preferences. More specifically, CGSoRec first includes a Condition-Guided Social Denoising Model (CSD) to remove redundant social relations in the social network for capturing users' social preferences with items more precisely. Then, CGSoRec calculates users' social preferences based on denoised social network and adjusts the weights in users' social preferences to make them can counteract the popularity bias present in the recommendation model. At last, CGSoRec includes a Condition-Guided Diffusion Recommendation Model (CGD) to introduce the adjusted social preferences as conditions to control the recommendation results for a debiased direction. Comprehensive experiments on three real-world datasets demonstrate the effectiveness of our proposed method. The code is in: https://github.com/hexin5515/CGSoRec.


title: On the origin of infrared bands attributed to tryptophan in Spitzer observations of IC 348

abstract: Infrared emission features toward interstellar gas of the IC 348 star cluster in Perseus have been recently proposed to originate from the amino acid tryptophan. The assignment was based on laboratory infrared spectra of tryptophan pressed into pellets, a method which is known to cause large frequency shifts compared to the gas phase. We assess the validity of the assignment based on the original Spitzer data as well as new data from JWST. In addition, we report new spectra of tryptophan condensed in para-hydrogen matrices to compare with the observed spectra. The JWST MIRI data do not show evidence for tryptophan, despite deeper integration toward IC 348. In addition, we show that several of the lines attributed to tryptophan are likely due to instrumental artifacts. This, combined with the new laboratory data, allows us to conclude that there is no compelling evidence for the tryptophan assignment.


title: Probabilistic Height Grid Terrain Mapping for Mining Shovels using LiDAR

abstract: This paper explores the question of creating and maintaining terrain maps in environments where the terrain changes. The specific example explored is the construction of terrain maps from 3D LiDAR measurements on an electric rope shovel. The approach extends the height grid representation of terrain to include a Hidden Markov Model in each cell, enabling confidence-based mapping of constantly changing terrain. There are inherent difficulties in this problem, including semantic labelling of the LiDAR measurements associated with machinery and determining the pose of the sensor. Solutions to both of these problems are explored. The significance of this work lies in the need for accurate terrain mapping to support autonomous machine operation.


title: The Chern-Simons Functional Integral, Kauffman's Bracket Polynomial, and other link invariants

abstract: We study Chern-Simons Gauge Theory in axial gauge on ${\mathbb R}^3.$ This theory has a quadratic Lagrangian and therefore expectations can be computed nonperturbatively by explicit formulas, giving an (unbounded) linear functional on a space of polynomial functions in the gauge fields, as a mathematically well-defined avatar of the formal functional integral. We use differential-geometric methods to extend the definition of this linear functional to expectations of products of Wilson loops corresponding to oriented links in ${\mathbb R}^3,$ and derive skein relations for them. In the case $G=SU(2)$ we show that these skein relations are closely related to those of the Kauffman bracket polynomial, which is closely related to the Jones polynomial. We also study the case of groups of higher rank. We note that in the absence of a cubic term in the action, there is no quantization condition on the coupling $\lambda,$ which can be any complex number. This is in line with the fact that the Jones polynomial, in contrast to the manifold invariants of Witten and Reshetikhin-Turaev, is defined for any value of the coupling. The appearance of the parameter $e^{\frac1{2\lambda}}$ in the expectations and skein relations is also natural. Likewise, the extension of the theory to noncompact groups presents no difficulties. Finally we show how computations similar to ours, but for gauge fields in two dimensions, yield the Goldman bracket.


title: A new proof of the Bondal-Orlov reconstruction using Matsui spectra

abstract: In 2005, Balmer defined the ringed space $\operatorname{Spec}_\otimes \mathcal{T}$ for a given tensor triangulated category, while in 2023, the second author introduced the ringed space $\operatorname{Spec}_\vartriangle \mathcal{T}$ for a given triangulated category. In the algebro-geometric context, these spectra provided several reconstruction theorems using derived categories. In this paper, we prove that $\operatorname{Spec}_{\otimes_X^\mathbb{L}} \operatorname{Perf} X$ is an open ringed subspace of $\operatorname{Spec}_\vartriangle \operatorname{Perf} X$ for a quasi-projective variety $X$. As an application, we provide a new proof of the Bondal-Orlov and Ballard reconstruction theorems in terms of these spectra. Recently, the first author introduced the Fourier-Mukai locus $\operatorname{Spec}^\mathsf{FM} \operatorname{Perf} X$ for a smooth projective variety $X$, which is constructed by gluing Fourier-Mukai partners of $X$ inside $\operatorname{Spec}_\vartriangle \operatorname{Perf} X$. As another application of our main theorem, we demonstrate that $\operatorname{Spec}^\mathsf{FM} \operatorname{Perf} X$ can be viewed as an open ringed subspace of $\operatorname{Spec}_\vartriangle \operatorname{Perf} X$. As a result, we show that all the Fourier-Mukai partners of an abelian variety $X$ can be reconstructed by topologically identifying the Fourier-Mukai locus within $\operatorname{Spec}_\vartriangle \operatorname{Perf} X$.


title: Coverage Analysis of Downlink Transmission in Multi-Connectivity Cellular V2X Networks

abstract: With the increasing of connected vehicles in the fifth-generation mobile communication networks (5G) and beyond 5G (B5G), ensuring the reliable and high-speed cellular vehicle-to-everything (C-V2X) communication has posed significant challenges due to the high mobility of vehicles. For improving the network performance and reliability, multi-connectivity technology has emerged as a crucial transmission mode for C-V2X in the 5G era. To this end, this paper proposes a framework for analyzing the performance of multi-connectivity in C-V2X downlink transmission, with a focus on the performance indicators of joint distance distribution and coverage probability. Specifically, we first derive the joint distance distribution of multi-connectivity. By leveraging the tools of stochastic geometry, we then obtain the analytical expressions of coverage probability based on the previous results for general multi-connectivity cases in C-V2X. Subsequently, we evaluate the effect of path loss exponent and downlink base station density on coverage probability based on the proposed analytical framework. Finally, extensive Monte Carlo simulations are conducted to validate the effectiveness of the proposed analytical framework and the simulation results reveal that multi-connectivity technology can significantly enhance the coverage probability in C-V2X.


title: Unusual switch from low-temperature T-quadratic resistivity in the underdoped pseudogap phase of cuprate superconductors to low-temperature T-linear resistivity in the overdoped strange-metal phase

abstract: The transport experiments demonstrate a dramatic switch from the low-temperature T-linear resistivity in the overdoped strange-metal phase to the T-quadratic resistivity in the underdoped pseudogap phase of cuprate superconductors, however, a consensus on the origin of this switch is still lacking. Here the low-temperature resistivity in the underdoped pseudogap phase of cuprate superconductors is investigated using the Boltzmann transport equation. The low-temperature resistivity originates from the electron umklapp scattering mediated by the spin excitation. However, the dominant contribution to the resistivity mainly comes from the antinodal umklapp scattering. In particular, a low temperature $T_{scale}$ scales with $\Delta^{2}_{p}$ in the underdoped regime due to the opening of a momentum-dependent spin pseudogap, where $\Delta_{p}$ is the minimal umklapp vector at the antinode. Notably, this $T_{scale}$ as a function of doping presents a similar behavior of the antinodal spin pseudogap crossover temperature, i.e., $T_{scale}$ decreases with the increase of doping in the underdoped regime, and then is reduced to a very low temperature in the overdoped regime. In the underdoped regime, the resistivity is T-quadratic in the low temperatures below $T_{scale}$ with the strength of the T-quadratic resistivity that weakens as the doping is raised. However, in the overdoped regime, the resistivity is T-linear in the low temperatures above $T_{scale}$. The result in this paper together with the recent study on the electrical transport in the overdoped regime therefore show that the electron Umklapp scattering from a spin excitation responsible for the low-temperature T-linear resistivity in the overdoped strange-metal phase naturally produces the low-temperature T-quadratic resistivity in the underdoped pseudogap phase resulting from the opening of a momentum dependent spin pseudogap.


title: Simple obstructions and cone reduction

abstract: Let $X$ be a Deligne-Mumford stack locally of finite type over an algebraically closed field $k$ of characteristic zero. We show that the intrinsic normal cone $C_X$ of $X$ is supported in the subcone $\mathbb{V}(\Omega_X[-1])$ ($h^1/h^0((\Omega^1_X)^\vee)$) of its intrinsic normal sheaf $N_X$. This leads to an alternative proof of cone reduction by cosections for $C_X$. We also discuss vanishing of simple obstructions under the Buchweitz-Flenner semiregularity map for sheaves.


title: Analysis of Broken Randomized Experiments by Principal Stratification

abstract: Although randomized controlled trials have long been regarded as the ``gold standard'' for evaluating treatment effects, there is no natural prevention from post-treatment events. For example, non-compliance makes the actual treatment different from the assigned treatment, truncation-by-death renders the outcome undefined or ill-defined, and missingness prevents the outcomes from being measured. In this paper, we develop a statistical analysis framework using principal stratification to investigate the treatment effect in broken randomized experiments. The average treatment effect in compliers and always-survivors is adopted as the target causal estimand. We establish the asymptotic property for the estimator. We apply the framework to study the effect of training on earnings in the Job Corps Study and find that the training program does not have an effect on employment but possibly have an effect on improving the earnings after employment.


title: Emergent topological magnetism in Hund's excitonic insulator

abstract: Analogous to the charged electron-electron pair condensation in superconductors, an excitonic insulator (EI) represents Fermi surface instability due to spontaneous formation and condensation of charge-neutral electron-hole pair (exciton). Unlike in superconductors, however, the charge-neutral nature of exciton makes probing emergent EI phase via macroscopic physical properties generally difficult. Here, we propose a van der Waals coupled antiferromagnetic semiconductor GdGaI (GGI) as a new material category leading to emergent multi-q magnet intertwined with spontaneous exciton formation/condensation. Before excitonic band hybridization, a simple picture for the parent electronic state consists of electron (Gd-derived 5d) and hole (Ga-derived 4p) delocalized bands, together with Gd-derived 4f localized antiferromagnets with S = 7/2 classical nature. Through intra Gd atom 4f-5d Hund's coupling, a notable finding is the emergent minimum length scale (2a) Skyrmion-like spin texture resulting from spontaneous condensation/formation of spin-polarized exciton with BCS-BEC crossover phenomenology. This discovered platform is promising for realizing valuable quantum matter on the nanoscale; our finding will provide significant insight into designing the atomic scale topological magnetism out of itinerant systems.


title: Quantum equivalence of the Freedman-Townsend model and the principal chiral $\sigma$-model

abstract: The Freedman-Townsend model is quantized using the Batalin-Vilkovisky approach to Lagrangian quantization of gauge theories with linearly dependent generators. Path integral arguments are then applied to demonstrate the quantum equivalence of the Freedman-Townsend model to the principal chiral $\sigma$-model.


title: TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Models

abstract: One key challenge in backdoor attacks against large foundation models is the resource limits. Backdoor attacks usually require retraining the target model, which is impractical for very large foundation models. Existing backdoor attacks are mainly designed for supervised classifiers or small foundation models (e.g., BERT). None of these attacks has successfully compromised a very large foundation model, such as Llama-3-70B, especially with limited computational resources. In this paper, we propose TrojFM, a novel backdoor attack tailored for very large foundation models. Our primary technical contribution is the development of a novel backdoor injection method. This method forces a backdoored model to generate similar hidden representations for poisoned inputs regardless of their actual semantics. Our approach injects such backdoors by fine-tuning only a very small proportion of model parameters. This enables TrojFM to efficiently launch downstream task-agnostic backdoor attacks against very large foundation models under limited computational resources. Moreover, we optimize the fine-tuning process with our customized QLoRA technique, enabling launching our attack via only~\textit{one A100 GPU}. Furthermore, we design a new trigger injection method to ensure our attack stealthiness. Through extensive experiments, we first demonstrate that TrojFM can launch effective backdoor attacks against widely used large GPT-style models without jeopardizing their normal functionalities (and outperforming existing attacks on BERT-style models). Furthermore, we show that TrojFM is resilient to SOTA defenses and is insensitive to changes in key hyper-parameters. Finally, we conduct a resource analysis to quantify that our method can significantly save computational and memory costs compared to existing backdoor attacks.


title: The second-order zero differential uniformity of the swapped inverse functions over finite fields

abstract: The Feistel Boomerang Connectivity Table (FBCT) was proposed as the feistel counterpart of the Boomerang Connectivity Table. The entries of the FBCT are actually related to the second-order zero differential spectrum. Recently, several results on the second-order zero differential uniformity of some functions were introduced. However, almost all of them were focused on power functions, and there are only few results on non-power functions. In this paper, we investigate the second-order zero differential uniformity of the swapped inverse functions, which are functions obtained from swapping two points in the inverse function. We also present the second-order zero differential spectrum of the swapped inverse functions for certain cases. In particular, this paper is the first result to characterize classes of non-power functions with the second-order zero differential uniformity equal to 4, in even characteristic.


title: PromptFix: You Prompt and We Fix the Photo

abstract: Diffusion models equipped with language models demonstrate excellent controllability in image generation tasks, allowing image processing to adhere to human instructions. However, the lack of diverse instruction-following data hampers the development of models that effectively recognize and execute user-customized instructions, particularly in low-level tasks. Moreover, the stochastic nature of the diffusion process leads to deficiencies in image generation or editing tasks that require the detailed preservation of the generated images. To address these limitations, we propose PromptFix, a comprehensive framework that enables diffusion models to follow human instructions to perform a wide variety of image-processing tasks. First, we construct a large-scale instruction-following dataset that covers comprehensive image-processing tasks, including low-level tasks, image editing, and object creation. Next, we propose a high-frequency guidance sampling method to explicitly control the denoising process and preserve high-frequency details in unprocessed areas. Finally, we design an auxiliary prompting adapter, utilizing Vision-Language Models (VLMs) to enhance text prompts and improve the model's task generalization. Experimental results show that PromptFix outperforms previous methods in various image-processing tasks. Our proposed model also achieves comparable inference efficiency with these baseline models and exhibits superior zero-shot capabilities in blind restoration and combination tasks. The dataset and code will be aviliable at https://github.com/yeates/PromptFix.


title: Fractal behavior of tensor powers of the two dimensional space in prime characteristic

abstract: We study the number of indecomposable summands in tensor powers of the vector representation of SL2. Our main focus is on positive characteristic where this sequence of numbers and its generating function show fractal behavior akin to Mahler functions.


title: Classical and quantum thermodynamics described as a system-bath model: The dimensionless minimum work principle

abstract: We formulate a thermodynamic theory applicable to both classical and quantum systems. These systems are depicted as thermodynamic system-bath models capable of handling isothermal, isentropic, thermostatic, and entropic processes. Our approach is based on the use of a dimensionless thermodynamic potential expressed as a function of the intensive and extensive thermodynamic variables. Using the principles of dimensionless minimum work and dimensionless maximum entropy derived from quasi-static changes of external perturbations and temperature, we obtain the Massieu-Planck potentials as entropic potentials and the Helmholtz-Gibbs potentials as free energy. These potentials can be interconverted through time-dependent Legendre transformations. Our results are verified numerically for an anharmonic Brownian system described in phase space using the low-temperature quantum Fokker-Planck equations in the quantum case and the Kramers equation in the classical case, both developed for the thermodynamic system-bath model. Thus, we clarify the conditions for thermodynamics to be valid even for small systems described by Hamiltonians and establish a basis for extending thermodynamics to non-equilibrium conditions.


title: 3D Reconstruction with Fast Dipole Sums

abstract: We introduce a technique for the reconstruction of high-fidelity surfaces from multi-view images. Our technique uses a new point-based representation, the dipole sum, which generalizes the winding number to allow for interpolation of arbitrary per-point attributes in point clouds with noisy or outlier points. Using dipole sums allows us to represent implicit geometry and radiance fields as per-point attributes of a point cloud, which we initialize directly from structure from motion. We additionally derive Barnes-Hut fast summation schemes for accelerated forward and reverse-mode dipole sum queries. These queries facilitate the use of ray tracing to efficiently and differentiably render images with our point-based representations, and thus update their point attributes to optimize scene geometry and appearance. We evaluate this inverse rendering framework against state-of-the-art alternatives, based on ray tracing of neural representations or rasterization of Gaussian point-based representations. Our technique significantly improves reconstruction quality at equal runtimes, while also supporting more general rendering techniques such as shadow rays for direct illumination. In the supplement, we provide interactive visualizations of our results.


title: NoteLLM-2: Multimodal Large Representation Models for Recommendation

abstract: Large Language Models (LLMs) have demonstrated exceptional text understanding. Existing works explore their application in text embedding tasks. However, there are few works utilizing LLMs to assist multimodal representation tasks. In this work, we investigate the potential of LLMs to enhance multimodal representation in multimodal item-to-item (I2I) recommendations. One feasible method is the transfer of Multimodal Large Language Models (MLLMs) for representation tasks. However, pre-training MLLMs usually requires collecting high-quality, web-scale multimodal data, resulting in complex training procedures and high costs. This leads the community to rely heavily on open-source MLLMs, hindering customized training for representation scenarios. Therefore, we aim to design an end-to-end training method that customizes the integration of any existing LLMs and vision encoders to construct efficient multimodal representation models. Preliminary experiments show that fine-tuned LLMs in this end-to-end method tend to overlook image content. To overcome this challenge, we propose a novel training framework, NoteLLM-2, specifically designed for multimodal representation. We propose two ways to enhance the focus on visual information. The first method is based on the prompt viewpoint, which separates multimodal content into visual content and textual content. NoteLLM-2 adopts the multimodal In-Content Learning method to teach LLMs to focus on both modalities and aggregate key information. The second method is from the model architecture, utilizing a late fusion mechanism to directly fuse visual information into textual information. Extensive experiments have been conducted to validate the effectiveness of our method.


title: SCSim: A Realistic Spike Cameras Simulator

abstract: Spike cameras, with their exceptional temporal resolution, are revolutionizing high-speed visual applications. Large-scale synthetic datasets have significantly accelerated the development of these cameras, particularly in reconstruction and optical flow. However, current synthetic datasets for spike cameras lack sophistication. Addressing this gap, we introduce SCSim, a novel and more realistic spike camera simulator with a comprehensive noise model. SCSim is adept at autonomously generating driving scenarios and synthesizing corresponding spike streams. To enhance the fidelity of these streams, we've developed a comprehensive noise model tailored to the unique circuitry of spike cameras. Our evaluations demonstrate that SCSim outperforms existing simulation methods in generating authentic spike streams. Crucially, SCSim simplifies the creation of datasets, thereby greatly advancing spike-based visual tasks like reconstruction. Our project refers to https://github.com/Acnext/SCSim.


title: Joint Node Selection and Resource Allocation Optimization for Cooperative Sensing with a Shared Wireless Backhaul

abstract: In this paper, we consider a cooperative sensing framework in the context of future multi-functional network with both communication and sensing ability, where one base station (BS) serves as a sensing transmitter and several nearby BSs serve as sensing receivers. Each receiver receives the sensing signal reflected by the target and communicates with the fusion center (FC) through a wireless multiple access channel (MAC) for cooperative target localization. To improve the localization performance, we present a hybrid information-signal domain cooperative sensing (HISDCS) design, where each sensing receiver transmits both the estimated time delay/effective reflecting coefficient and the received sensing signal sampled around the estimated time delay to the FC. Then, we propose to minimize the number of channel uses by utilizing an efficient Karhunen-Lo\'eve transformation (KLT) encoding scheme for signal quantization and proper node selection, under the Cram\'er-Rao lower bound (CRLB) constraint and the capacity limits of MAC. A novel matrix-inequality constrained successive convex approximation (MCSCA) algorithm is proposed to optimize the wireless backhaul resource allocation, together with a greedy strategy for node selection. Despite the high non-convexness of the considered problem, we prove that the proposed MCSCA algorithm is able to converge to the set of Karush-Kuhn-Tucker (KKT) solutions of a relaxed problem obtained by relaxing the discrete variables. Besides, a low-complexity quantization bit reallocation algorithm is designed, which does not perform explicit node selection, and is able to harvest most of the performance gain brought by HISDCS. Finally, numerical simulations are presented to show that the proposed HISDCS design is able to significantly outperform the baseline schemes.


title: Laurel: Generating Dafny Assertions Using Large Language Models

abstract: Dafny is a popular verification language, which automates proofs by outsourcing them to an SMT solver. This automation is not perfect, however, and the solver often requires guidance in the form of helper assertions creating a burden for the proof engineer. In this paper, we propose Laurel, a tool that uses large language models (LLMs) to automatically generate helper assertions for Dafny programs. To improve the success rate of LLMs in this task, we design two domain-specific prompting techniques. First, we help the LLM determine the location of the missing assertion by analyzing the verifier's error message and inserting an assertion placeholder at that location. Second, we provide the LLM with example assertions from the same codebase, which we select based on a new lemma similarity metric. We evaluate our techniques on a dataset of helper assertions we extracted from three real-world Dafny codebases. Our evaluation shows that Laurel is able to generate over 50% of the required helper assertions given only a few attempts, making LLMs a usable and affordable tool to further automate practical program verification.


title: Multiphoton quantum sensing

abstract: While the fundamental principles of light-matter interaction are well-understood and drive countless technologies, the world of multiphoton processes remains a fascinating puzzle, holding the potential to drastically alter our understanding of how light interacts with matter at its most basic level. This rich interplay of light and matter unveils novel phenomena that can be harnessed for sensing with exceptional precision, as exemplified by multiphoton quantum sensing. This thesis delves into the applications of multiphoton quantum protocols, particularly in imaging, communication, and plasmonic sensing, to surpass classical limitations and achieve enhanced sensitivity. We explore the potential of multiphoton quantum processes, particularly in the nanoscale regime and within subsystems of macroscopic systems, where novel and ultra-sensitive sensing methodologies emerge. Subsequent chapters of this thesis demonstrate the transformative potential of multiphoton quantum sensing, elucidating the design, implementation, and experimental results of specific sensing protocols tailored to diverse applications. Our analysis combines experimental observations and theoretical predictions to assess the sensitivity and performance of these protocols. Additionally, the thesis discusses potential future directions and advancements in the field, envisioning applications in biomolecule detection, environmental monitoring, and fundamental studies of light-matter interactions at the nanoscale. Concluding reflections highlight the implications of multiphoton quantum sensing across scientific disciplines and lay the groundwork for future research endeavors.


title: X(3872) revisited: the roles of OPEP and the quark degrees of freedom

abstract: The $X(3872)$ is investigated by employing the quark-hadron hybrid model, that consists of the $c\bar c$ core, $D^{(*)}\bar D{}^*$, $J/\psi\omega$, and $J/\psi\rho$ two-meson states. Due to the attraction from the $c\bar c$-$D\bar D{}^*$ coupling and from the OPEP tensor coupling, a very thin peak can appear at the $D^{0}\bar D{}^{*0}$ threshold. The energy of the corresponding pole of the scattering matrix is $E=(+0.06-0.14i)$ MeV, which is on the physical sheet and above the threshold, the same as the one of the poles from the LHCb data analysis.


title: Physics-informed Inverse Design of Multi-bit Programmable Metasurfaces

abstract: Emerging reconfigurable metasurfaces offer various possibilities in programmatically manipulating electromagnetic waves across spatial, spectral, and temporal domains, showcasing great potential for enhancing terahertz applications. However, they are hindered by limited tunability, particularly evident in relatively small phase tuning over 270o, due to the design constraints with time-intensive forward design methodologies. Here, we demonstrate a multi-bit programmable metasurface capable of terahertz beam steering, facilitated by a developed physics-informed inverse design (PIID) approach. Through integrating a modified coupled mode theory (MCMT) into residual neural networks, our PIID algorithm not only significantly increases the design accuracy compared to conventional neural networks but also elucidates the intricate physical relations between the geometry and the modes. Without decreasing the reflection intensity, our method achieves the enhanced phase tuning as large as 300o. Additionally, we experimentally validate the inverse designed programmable beam steering metasurface, which is adaptable across 1-bit, 2-bit, and tri-state coding schemes, yielding a deflection angle up to 68o and broadened steering coverage. Our demonstration provides a promising pathway for rapidly exploring advanced metasurface devices, with potentially great impact on communication and imaging technologies.


title: DualContrast: Unsupervised Disentangling of Content and Transformations with Implicit Parameterization

abstract: Unsupervised disentanglement of content and transformation has recently drawn much research, given their efficacy in solving downstream unsupervised tasks like clustering, alignment, and shape analysis. This problem is particularly important for analyzing shape-focused real-world scientific image datasets, given their significant relevance to downstream tasks. The existing works address the problem by explicitly parameterizing the transformation factors, significantly reducing their expressiveness. Moreover, they are not applicable in cases where transformations can not be readily parametrized. An alternative to such explicit approaches is self-supervised methods with data augmentation, which implicitly disentangles transformations and content. We demonstrate that the existing self-supervised methods with data augmentation result in the poor disentanglement of content and transformations in real-world scenarios. Therefore, we developed a novel self-supervised method, DualContrast, specifically for unsupervised disentanglement of content and transformations in shape-focused image datasets. Our extensive experiments showcase the superiority of DualContrast over existing self-supervised and explicit parameterization approaches. We leveraged DualContrast to disentangle protein identities and protein conformations in cellular 3D protein images. Moreover, we also disentangled transformations in MNIST, viewpoint in the Linemod Object dataset, and human movement deformation in the Starmen dataset as transformations using DualContrast.


title: A Real-Time Voice Activity Detection Based On Lightweight Neural

abstract: Voice activity detection (VAD) is the task of detecting speech in an audio stream, which is challenging due to numerous unseen noises and low signal-to-noise ratios in real environments. Recently, neural network-based VADs have alleviated the degradation of performance to some extent. However, the majority of existing studies have employed excessively large models and incorporated future context, while neglecting to evaluate the operational efficiency and latency of the models. In this paper, we propose a lightweight and real-time neural network called MagicNet, which utilizes casual and depth separable 1-D convolutions and GRU. Without relying on future features as input, our proposed model is compared with two state-of-the-art algorithms on synthesized in-domain and out-domain test datasets. The evaluation results demonstrate that MagicNet can achieve improved performance and robustness with fewer parameter costs.


title: Exploring Fairness in Educational Data Mining in the Context of the Right to be Forgotten

abstract: In education data mining (EDM) communities, machine learning has achieved remarkable success in discovering patterns and structures to tackle educational challenges. Notably, fairness and algorithmic bias have gained attention in learning analytics of EDM. With the increasing demand for the right to be forgotten, there is a growing need for machine learning models to forget sensitive data and its impact, particularly within the realm of EDM. The paradigm of selective forgetting, also known as machine unlearning, has been extensively studied to address this need by eliminating the influence of specific data from a pre-trained model without complete retraining. However, existing research assumes that interactive data removal operations are conducted in secure and reliable environments, neglecting potential malicious unlearning requests to undermine the fairness of machine learning systems. In this paper, we introduce a novel class of selective forgetting attacks designed to compromise the fairness of learning models while maintaining their predictive accuracy, thereby preventing the model owner from detecting the degradation in model performance. Additionally, we propose an innovative optimization framework for selective forgetting attacks, capable of generating malicious unlearning requests across various attack scenarios. We validate the effectiveness of our proposed selective forgetting attacks on fairness through extensive experiments using diverse EDM datasets.


abstract: Knowledge tracing has been widely used in online learning systems to guide the students' future learning. However, most existing KT models primarily focus on extracting abundant information from the question sets and explore the relationships between them, but ignore the personalized student behavioral information in the learning process. This will limit the model's ability to accurately capture the personalized knowledge states of students and reasonably predict their performances. To alleviate this limitation, we explicitly models the personalized learning process by incorporating the emotions, a representative personalized behavior in the learning process, into KT framework. Specifically, we present a novel Dual-State Personalized Knowledge Tracing with Emotional Incorporation model to achieve this goal: Firstly, we incorporate emotional information into the modeling process of knowledge state, resulting in the Knowledge State Boosting Module. Secondly, we design an Emotional State Tracing Module to monitor students' personalized emotional states, and propose an emotion prediction method based on personalized emotional states. Finally, we apply the predicted emotions to enhance students' response prediction. Furthermore, to extend the generalization capability of our model across different datasets, we design a transferred version of DEKT, named Transfer Learning-based Self-loop model (T-DEKT). Extensive experiments show our method achieves the state-of-the-art performance.