title: Multirobot Watchman Routes in a Simple Polygon

abstract: The well-known \textsc{Watchman Route} problem seeks a shortest route in a polygonal domain from which every point of the domain can be seen. In this paper, we study the cooperative variant of the problem, namely the \textsc{$k$-Watchmen Routes} problem, in a simple polygon $P$. We look at both the version in which the $k$ watchmen must collectively see all of $P$, and the quota version in which they must see a predetermined fraction of $P$'s area. We give an exact pseudopolynomial time algorithm for the \textsc{$k$-Watchmen Routes} problem in a simple orthogonal polygon $P$ with the constraint that watchmen must move on axis-parallel segments, and there is a given common starting point on the boundary. Further, we give a fully polynomial-time approximation scheme and a constant-factor approximation for unconstrained movement. For the quota version, we give a constant-factor approximation in a simple polygon, utilizing the solution to the (single) \textsc{Quota Watchman Route} problem.


title: Photoinduced Charge Transfer in Transition Metal Dichalcogenide Quantum Dots

abstract: In this paper, a charge transfer mechanism in transition metal dichalcogenide (TMDC) quantum dots (QDs) is studied. A common dye, Rhodamine 6G (R6G), has been used as fluorescent molecule and QDs of molybdenum disulfide ($\rm{MoS_2}$) and tungsten disulfide ($\rm{WS_2}$) have been used as acceptors of electrons in the photo-induced charge transfer process. The MoS$_2$ and WS$_2$ QDs have been synthesized using ultrasonication method. These QDs have been characterized using transmission electron microscope (TEM), UV-Vis spectrophotometer, and fluorimeter. The TEM images of these QDs show well dispersed particles of size $~$2 nm. These QDs show intense fluorescence emission when excited with light having wavelength, $\lambda < $ 350 nm. In presence of light, the photons help in creating charges in fluorescent dye molecule, and TMDCs QDs felicitate the charge transfer in the process. The charge transfer phenomenon has been studied using time correlated single photon counting (TCSPC) set-up, which is a time resolved fluorescence spectroscopic technique. The time resolved fluorescence study shows a drastic change in the fluorescence (FL) lifetime of R6G molecules in the presence of QDs. This decrease in FL lifetime of R6G shows that the MoS$_2$, and WS$_2$ QDs provide additional path to the photo-generated electrons in the excited state of R6G. This study can be further extended to optoelectronic devices, where charge transfer plays an important role in the device efficiency and performance.


title: A-PETE: Adaptive Prototype Explanations of Tree Ensembles

abstract: The need for interpreting machine learning models is addressed through prototype explanations within the context of tree ensembles. An algorithm named Adaptive Prototype Explanations of Tree Ensembles (A-PETE) is proposed to automatise the selection of prototypes for these classifiers. Its unique characteristics is using a specialised distance measure and a modified k-medoid approach. Experiments demonstrated its competitive predictive accuracy with respect to earlier explanation algorithms. It also provides a a sufficient number of prototypes for the purpose of interpreting the random forest classifier.


title: Introducing sgboost: A Practical Guide and Implementation of sparse-group boosting in R

abstract: This paper introduces the sgboost package in R, which implements sparse-group boosting for modeling high-dimensional data with natural groupings in covariates. Sparse-group boosting offers a flexible approach for both group and individual variable selection, reducing overfitting and enhancing model interpretability. The package uses regularization techniques based on the degrees of freedom of individual and group base-learners, and is designed to be used in conjunction with the mboost package. Through comparisons with existing methods and demonstration of its unique functionalities, this paper provides a practical guide on utilizing sparse-group boosting in R, accompanied by code examples to facilitate its application in various research domains. Overall, this paper serves as a valuable resource for researchers and practitioners seeking to use sparse-group boosting for efficient and interpretable high-dimensional data analysis.


title: A Multi-wavelength, Multi-epoch Monitoring Campaign of Accretion Variability in T Tauri Stars from the ODYSSEUS Survey. I. HST FUV and NUV Spectra

abstract: The Classical T Tauri Star (CTTS) stage is a critical phase of the star and planet formation process. In an effort to better understand the mass accretion process, which can dictate further stellar evolution and planet formation, a multi-epoch, multi-wavelength photometric and spectroscopic monitoring campaign of four CTTSs (TW Hya, RU Lup, BP Tau, and GM Aur) was carried out in 2021 and 2022/2023 as part of the Outflows and Disks Around Young Stars: Synergies for the Exploration of ULYSSES Spectra (ODYSSEUS) program. Here we focus on the HST UV spectra obtained by the HST Director's Discretionary Time UV Legacy Library of Young Stars as Essential Standards (ULLYSES) program. Using accretion shock modeling, we find that all targets exhibit accretion variability, varying from short increases in accretion rate by up to a factor of 3 within 48 hours, to longer decreases in accretion rate by a factor of 2.5 over the course of 1 year. This is despite the generally consistent accretion morphology within each target. Additionally, we test empirical relationships between accretion rate and UV luminosity and find stark differences, showing that these relationships should not be used to estimate the accretion rate for individual target. Our work reinforces that future multi-epoch and simultaneous multi-wavelength studies are critical in our understanding of the accretion process in low-mass star formation.


title: Fibonacci sequence and Pythagorean triples in the composition of functions for integer solutions from certain operator

abstract: The following article summarizes research where theorems and their respective demonstrations are postulated based on quadratic equations with special properties given by the Pythagorean triplets and the Fibonacci sequence given the second order of equations where integer solutions are found an environment in number theory and its applications to calculus.


title: Direct Alignment of Language Models via Quality-Aware Self-Refinement

abstract: Reinforcement Learning from Human Feedback (RLHF) has been commonly used to align the behaviors of Large Language Models (LLMs) with human preferences. Recently, a popular alternative is Direct Policy Optimization (DPO), which replaces an LLM-based reward model with the policy itself, thus obviating the need for extra memory and training time to learn the reward model. However, DPO does not consider the relative qualities of the positive and negative responses, and can lead to sub-optimal training outcomes. To alleviate this problem, we investigate the use of intrinsic knowledge within the on-the-fly fine-tuning LLM to obtain relative qualities and help to refine the loss function. Specifically, we leverage the knowledge of the LLM to design a refinement function to estimate the quality of both the positive and negative responses. We show that the constructed refinement function can help self-refine the loss function under mild assumptions. The refinement function is integrated into DPO and its variant Identity Policy Optimization (IPO). Experiments across various evaluators indicate that they can improve the performance of the fine-tuned models over DPO and IPO.


title: Interferometry of quantum correlation functions to access quasiprobability distribution of work

abstract: The Kirkwood-Dirac quasiprobability distribution emerges from the quantum correlation function of two observables measured at distinct times and is therefore relevant for fundamental physics and quantum technologies. These quasiprobabilities follow all but one of Kolmogorov axioms for joint probability distributions: they can take non-positive values. Their experimental reconstruction becomes challenging when expectation values of incompatible observables are involved. Previous strategies aimed to reconstruct them using weak measurements or combining strong measurements. Here, we use a more direct approach, an interferometric scheme aided by an auxiliary system, to reconstruct the Kirkwood-Dirac quasiprobability distribution. We experimentally demonstrate the interferometric scheme in an electron-nuclear spin system associated with a nitrogen-vacancy center in diamond. By measuring the characteristic function, we reconstruct the quasiprobability distribution of the work and analyze the behavior of the first and second moments of work. Our results clarify the physical meaning of the work quasiprobability distribution in the context of quantum thermodynamics. Finally, having measured the real and imaginary parts of the Kirkwood-Dirac quasiprobability of work, we are also able to study the uncertainty of measuring the Hamiltonian of the system at two times, via the Robertson-Schr{\"o}dinger uncertainty relation, for different initial states.


title: Comparing information content of representation spaces for disentanglement with VAE ensembles

abstract: Disentanglement is the endeavour to use machine learning to divide information about a dataset into meaningful fragments. In practice these fragments are representation (sub)spaces, often the set of channels in the latent space of a variational autoencoder (VAE). Assessments of disentanglement predominantly employ metrics that are coarse-grained at the model level, but this approach can obscure much about the process of information fragmentation. Here we propose to study the learned channels in aggregate, as the fragments of information learned by an ensemble of repeat training runs. Additionally, we depart from prior work where measures of similarity between individual subspaces neglected the nature of data embeddings as probability distributions. Instead, we view representation subspaces as communication channels that perform a soft clustering of the data; consequently, we generalize two classic information-theoretic measures of similarity between clustering assignments to compare representation spaces. We develop a lightweight method of estimation based on fingerprinting representation subspaces by their ability to distinguish dataset samples, allowing us to identify, analyze, and leverage meaningful structure in ensembles of VAEs trained on synthetic and natural datasets. Using this fully unsupervised pipeline we identify "hotspots" in the space of information fragments: groups of nearly identical representation subspaces that appear repeatedly in an ensemble of VAEs, particularly as regularization is increased. Finally, we leverage the proposed methodology to achieve ensemble learning with VAEs, boosting the information content of a set of weak learners -- a capability not possible with previous methods of assessing channel similarity.


title: Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation

abstract: We prove that the combination of a target network and over-parameterized linear function approximation establishes a weaker convergence condition for bootstrapped value estimation in certain cases, even with off-policy data. Our condition is naturally satisfied for expected updates over the entire state-action space or learning with a batch of complete trajectories from episodic Markov decision processes. Notably, using only a target network or an over-parameterized model does not provide such a convergence guarantee. Additionally, we extend our results to learning with truncated trajectories, showing that convergence is achievable for all tasks with minor modifications, akin to value truncation for the final states in trajectories. Our primary result focuses on temporal difference estimation for prediction, providing high-probability value estimation error bounds and empirical analysis on Baird's counterexample and a Four-room task. Furthermore, we explore the control setting, demonstrating that similar convergence conditions apply to Q-learning.


title: Designing for Fairness in Human-Robot Interactions

abstract: The foundation of successful human collaboration is deeply rooted in the principles of fairness. As robots are increasingly prevalent in various parts of society where they are working alongside groups and teams of humans, their ability to understand and act according to principles of fairness becomes crucial for their effective integration. This is especially critical when robots are part of multi-human teams, where they must make continuous decisions regarding the allocation of resources. These resources can be material, such as tools, or communicative, such as gaze direction, and must be distributed fairly among team members to ensure optimal team performance and healthy group dynamics. Therefore, our research focuses on understanding how robots can effectively participate within human groups by making fair decisions while contributing positively to group dynamics and outcomes. In this paper, I discuss advances toward ensuring that robots are capable of considering human notions of fairness in their decision-making.


title: An Attention-Based Multi-Context Convolutional Encoder-Decoder Neural Network for Work Zone Traffic Impact Prediction

abstract: Work zone is one of the major causes of non-recurrent traffic congestion and road incidents. Despite the significance of its impact, studies on predicting the traffic impact of work zones remain scarce. In this paper, we propose a data integration pipeline that enhances the utilization of work zone and traffic data from diversified platforms, and introduce a novel deep learning model to predict the traffic speed and incident likelihood during planned work zone events. The proposed model transforms traffic patterns into 2D space-time images for both model input and output and employs an attention-based multi-context convolutional encoder-decoder architecture to capture the spatial-temporal dependencies between work zone events and traffic variations. Trained and validated on four years of archived work zone traffic data from Maryland, USA, the model demonstrates superior performance over baseline models in predicting traffic speed, incident likelihood, and inferred traffic attributes such as queue length and congestion timings (i.e., start time and duration). Specifically, the proposed model outperforms the baseline models by reducing the prediction error of traffic speed by 5% to 34%, queue length by 11% to 29%, congestion timing by 6% to 17%, and increasing the accuracy of incident predictions by 5% to 7%. Consequently, this model offers substantial promise for enhancing the planning and traffic management of work zones.


title: Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

abstract: Reinforcement learning from human feedback (RLHF) has emerged as a central tool for language model alignment. We consider online exploration in RLHF, which exploits interactive access to human or AI feedback by deliberately encouraging the model to produce diverse, maximally informative responses. By allowing RLHF to confidently stray from the pre-trained model, online exploration offers the possibility of novel, potentially super-human capabilities, but its full potential as a paradigm for language model training has yet to be realized, owing to computational and statistical bottlenecks in directly adapting existing reinforcement learning techniques. We propose a new algorithm for online exploration in RLHF, Exploratory Preference Optimization (XPO), which is simple and practical -- a one-line change to (online) Direct Preference Optimization (DPO; Rafailov et al., 2023) -- yet enjoys the strongest known provable guarantees and promising empirical performance. XPO augments the DPO objective with a novel and principled exploration bonus, empowering the algorithm to explore outside the support of the initial model and human feedback data. In theory, we show that XPO is provably sample-efficient and converges to a near-optimal language model policy under natural exploration conditions, irrespective of whether the initial model has good coverage. Our analysis, which builds on the observation that DPO implicitly performs a form of $Q^{\star}$-approximation (or, Bellman error minimization), combines previously disparate techniques from language modeling and theoretical reinforcement learning in a serendipitous fashion through the perspective of KL-regularized Markov decision processes. Empirically, we find that XPO is more sample-efficient than non-exploratory DPO variants in a preliminary evaluation.


title: Grammar-Aligned Decoding

abstract: Large Language Models (LLMs) struggle with reliably generating highly structured outputs, such as program code, mathematical formulas, or well-formed markup. Constrained decoding approaches mitigate this problem by greedily restricting what tokens an LLM can output at each step to guarantee that the output matches a given constraint. Specifically, in grammar-constrained decoding (GCD), the LLM's output must follow a given grammar. In this paper we demonstrate that GCD techniques (and in general constrained decoding techniques) can distort the LLM's distribution, leading to outputs that are grammatical but appear with likelihoods that are not proportional to the ones given by the LLM, and so ultimately are low-quality. We call the problem of aligning sampling with a grammar constraint, grammar-aligned decoding (GAD), and propose adaptive sampling with approximate expected futures (ASAp), a decoding algorithm that guarantees the output to be grammatical while provably producing outputs that match the conditional probability of the LLM's distribution conditioned on the given grammar constraint. Our algorithm uses prior sample outputs to soundly overapproximate the future grammaticality of different output prefixes. Our evaluation on code generation and structured NLP tasks shows how ASAp often produces outputs with higher likelihood (according to the LLM's distribution) than existing GCD techniques, while still enforcing the desired grammatical constraints.


title: Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling

abstract: Diffusion models have emerged as a powerful tool for generating high-quality images from textual descriptions. Despite their successes, these models often exhibit limited diversity in the sampled images, particularly when sampling with a high classifier-free guidance weight. To address this issue, we present Kaleido, a novel approach that enhances the diversity of samples by incorporating autoregressive latent priors. Kaleido integrates an autoregressive language model that encodes the original caption and generates latent variables, serving as abstract and intermediary representations for guiding and facilitating the image generation process. In this paper, we explore a variety of discrete latent representations, including textual descriptions, detection bounding boxes, object blobs, and visual tokens. These representations diversify and enrich the input conditions to the diffusion models, enabling more diverse outputs. Our experimental results demonstrate that Kaleido effectively broadens the diversity of the generated image samples from a given textual description while maintaining high image quality. Furthermore, we show that Kaleido adheres closely to the guidance provided by the generated latent variables, demonstrating its capability to effectively control and direct the image generation process.


title: Boil-off of red supergiants: mass loss and type II-P supernovae

abstract: The mass loss mechanism of red supergiant stars is not well understood, even though it has crucial consequences for their stellar evolution and the appearance of supernovae that occur upon core-collapse. We argue that outgoing shock waves launched near the photosphere can support a dense chromosphere between the star's surface and the dust formation radius at several stellar radii. We derive analytic expressions for the time-averaged density profile of the chromosphere, and we use these to estimate mass loss rates due to winds launched by radiation pressure at the dust formation radius. These mass loss rates are similar to recent observations, possibly explaining the upward kink in mass loss rates of luminous red supergiants. Our models predict that low-mass red supergiants lose less mass than commonly assumed, while high-mass red supergiants lose more. The chromospheric mass of our models is $\sim$0.01 solar masses, most of which lies within a few stellar radii. This can help explain the early light curves and spectra of type-II P supernovae without requiring extreme pre-supernova mass loss. We discuss implications for stellar evolution, type II-P supernovae, SN 2023ixf, and Betelgeuse.


title: Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models

abstract: Adapting large-scale pre-trained generative models in a parameter-efficient manner is gaining traction. Traditional methods like low rank adaptation achieve parameter efficiency by imposing constraints but may not be optimal for tasks requiring high representation capacity. We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. Using the Kronecker product and efficient Stiefel optimizers, we achieve parameter-efficient adaptation of orthogonal matrices. We introduce Spectral Orthogonal Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity. Extensive evaluations on text-to-image diffusion models demonstrate SODA's effectiveness, offering a spectrum-aware alternative to existing fine-tuning methods.


title: Good Modelling Software Practices

abstract: In socio-environmental sciences, models are frequently used as tools to represent, understand, project and predict the behaviour of these complex systems. Along the modelling chain, Good Modelling Practices have been evolving that ensure -- amongst others -- that models are transparent and replicable. Whenever such models are represented in software, good modelling meets Good software Practices, such as a tractable development workflow, good code, collaborative development and governance, continuous integration and deployment, and Good Scientific Practices, such as attribution of copyrights and acknowledgement of intellectual property, publication of a software paper and archiving. Too often in existing socio-environmental model software, these practices have been regarded as an add-on to be considered at a later stage only; in fact, many modellers have shied away from publishing their model as open source out of fear that having to add good practices is too demanding. We here argue for making a habit of following a list of simple and not so simple practices early on in the implementation of the model life cycle. We contextualise cherry-picked and hands-on practices for supporting Good Modelling Practices, and we demonstrate their application in the example context of the Viable North Sea fisheries socio-ecological systems model.


title: RydbergGPT

abstract: We introduce a generative pretained transformer (GPT) designed to learn the measurement outcomes of a neutral atom array quantum computer. Based on a vanilla transformer, our encoder-decoder architecture takes as input the interacting Hamiltonian, and outputs an autoregressive sequence of qubit measurement probabilities. Its performance is studied in the vicinity of a quantum phase transition in Rydberg atoms in a square lattice array. We explore the ability of the architecture to generalize, by producing groundstate measurements for Hamiltonian parameters not seen in the training set. We focus on examples of physical observables obtained from inference on three different models, trained in fixed compute time on a single NVIDIA A100 GPU. These can act as benchmarks for the scaling of larger RydbergGPT models in the future. Finally, we provide RydbergGPT open source, to aid in the development of foundation models based off of a wide variety of quantum computer interactions and data sets in the future.


title: Dimension formulas for period spaces via motives and species

abstract: We apply the structure theory of finite dimensional algebras in order to deduce dimension formulas for spaces of period numbers, i.e., complex numbers defined by integrals of algebraic nature. We get a complete and conceptually clear answer in the case of $1$-periods, generalising classical results like Baker's theorem on the logarithms of algebraic numbers and partial results in Huber--W{\"u}stholz \cite{huber-wuestholz}. The application to the case of Mixed Tate Motives (i.e., Multiple Zeta Values) recovers the dimension estimates of Deligne--Goncharov \cite{deligne-goncharov}.


title: The First Billion Years, According to JWST

abstract: With stunning clarity, JWST has revealed the Universe's first billion years. The scientific community is analyzing a wealth of JWST imaging and spectroscopic data from that era, and is in the process of rewriting the astronomy textbooks. Here, 1.5 years into the JWST science mission, we provide a snapshot of the great progress made towards understanding the initial chapters of our cosmic history. We highlight discoveries and breakthroughs, topics and issues that are not yet understood, and questions that will be addressed in the coming years, as JWST continues its revolutionary observations of the Early Universe. While this compendium is written by a small number of authors, invited to ISSI Bern in March 2024 as part of the 2024 ISSI Breakthrough Workshop, we acknowledge the work of a large community that is advancing our collective understanding of the evolution of the Early Universe.


title: Factors Influencing Performance of Students in Software Automated Test Tools Course

abstract: Formal software testing education is important for building efficient QA professionals. Various aspects of quality assurance approaches are usually covered in courses for training software testing students. Automated Test Tools is one of the core courses in the software testing post-graduate curriculum due to the high demand for automated testers in the workforce. It is important to understand which factors are affecting student performance in the automated testing course to be able to assist the students early on based on their needs. Various metrics that are considered for predicting student performance in this testing course are student engagement, grades on individual deliverables, and prerequisite courses. This study identifies the impact of assessing students based on individual vs. group activities, theoretical vs. practical components, and the effect of having taken prerequisite courses in their final grade. To carry out this research, student data was collected from the automated test tools course of a community college-based postgraduate certificate program in software testing. The dataset contained student records from the years 2021 to 2022 and consisted of information from five different semesters. Various machine learning algorithms were applied to develop an effective model for predicting students performance in the automated software testing tools course, and finally, important features affecting the students performance were identified. The predictive performance model of the automated test tools course that was developed by applying the logistic regression technique, showed the best performance, with an accuracy score of 90%.


title: An Organic Weed Control Prototype using Directed Energy and Deep Learning

abstract: Organic weed control is a vital to improve crop yield with a sustainable approach. In this work, a directed energy weed control robot prototype specifically designed for organic farms is proposed. The robot uses a novel distributed array robot (DAR) unit for weed treatment. Soybean and corn databases are built to train deep learning neural nets to perform weed recognition. The initial deep learning neural nets show a high performance in classifying crops. The robot uses a patented directed energy plant eradication recipe that is completely organic and UV-C free, with no chemical damage or physical disturbance to the soil. The deep learning can classify 8 common weed species in a soybean field under natural environment with up to 98% accuracy.


title: On the number of Regge trajectories for dual amplitudes

abstract: Regge poles connect the analytic structure of scattering amplitudes, analytically continued in spin, to the high-energy limit in momentum space. Dual models are expected to have only Regge poles, and string theory suggests there should be an infinite number of them. In this study, we investigate the number of Regge trajectories these models may have. We prove, based solely on crossing symmetry and unitarity, that meromorphic amplitudes, with or without subtractions, cannot produce a reggeizing amplitude if they contain any finite number of Regge trajectories. We argue that this should exclude the existence of such amplitudes altogether. Additionally, we develop and apply a linear programming dual bootstrap method to exclude these amplitudes directly in momentum space.


title: Quantum state preparation for multivariate functions

abstract: A fundamental step of any quantum algorithm is the preparation of qubit registers in a suitable initial state. Often qubit registers represent a discretization of continuous variables and the initial state is defined by a multivariate function. We develop protocols for preparing quantum states whose amplitudes encode multivariate functions by linearly combining block-encodings of Fourier and Chebyshev basis functions. Without relying on arithmetic circuits, quantum Fourier transforms, or multivariate quantum signal processing, our algorithms are simpler and more effective than previous proposals. We analyze requirements both asymptotically and pragmatically in terms of near/medium-term resources. Numerically, we prepare bivariate Student's t-distributions, 2D Ricker wavelets and electron wavefunctions in a 3D Coulomb potential, which are initial states with potential applications in finance, physics and chemistry simulations. Finally, we prepare bivariate Gaussian distributions on the Quantinuum H2-1 trapped-ion quantum processor using 24 qubits and up to 237 two-qubit gates.


title: Unified Directly Denoising for Both Variance Preserving and Variance Exploding Diffusion Models

abstract: Previous work has demonstrated that, in the Variance Preserving (VP) scenario, the nascent Directly Denoising Diffusion Models (DDDM) can generate high-quality images in one step while achieving even better performance in multistep sampling. However, the Pseudo-LPIPS loss used in DDDM leads to concerns about the bias in assessment. Here, we propose a unified DDDM (uDDDM) framework that generates images in one-step/multiple steps for both Variance Preserving (VP) and Variance Exploding (VE) cases. We provide theoretical proofs of the existence and uniqueness of the model's solution paths, as well as the non-intersecting property of the sampling paths. Additionally, we propose an adaptive Pseudo-Huber loss function to balance the convergence to the true solution and the stability of convergence process.Through a comprehensive evaluation, we demonstrate that uDDDMs achieve FID scores comparable to the best-performing methods available for CIFAR-10 in both VP and VE. Specifically, uDDDM achieves one-step generation on CIFAR10 with FID of 2.63 and 2.53 for VE and VP respectively. By extending the sampling to 1000 steps, we further reduce FID score to 1.71 and 1.65 for VE and VP respectively, setting state-of-the-art performance in both cases.


title: Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

abstract: While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices. Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.


abstract: The Transformer architecture has recently gained considerable attention in the field of graph representation learning, as it naturally overcomes several limitations of Graph Neural Networks (GNNs) with customized attention mechanisms or positional and structural encodings. Despite making some progress, existing works tend to overlook external information of graphs, specifically the correlation between graphs. Intuitively, graphs with similar structures should have similar representations. Therefore, we propose Graph External Attention (GEA) -- a novel attention mechanism that leverages multiple external node/edge key-value units to capture inter-graph correlations implicitly. On this basis, we design an effective architecture called Graph External Attention Enhanced Transformer (GEAET), which integrates local structure and global interaction information for more comprehensive graph representations. Extensive experiments on benchmark datasets demonstrate that GEAET achieves state-of-the-art empirical performance. The source code is available for reproducibility at: https://github.com/icm1018/GEAET.


title: Algebra of global sections of $\psi$-bundles on $\bar{M}_{0,n}$

abstract: We consider the ${\mathbb Z}^n$-graded algebra of global sections of line bundles generated by the standard line bundles $L_1,\ldots,L_n$ on $\bar{M}_{0,n}$. We find a simple presentation of this algebra by generators and quadratic relations. As an application we prove that the moduli space $\bar{M}_{0,n}[\psi]$ of $\psi$-stable curves of genus $0$ is Cohen-Macaulay and normal, and the natural map $\bar{M}_{0,n}\to \bar{M}_{0,n}[\psi]$ is a rational resolution.


title: Neural Network Verification with Branch-and-Bound for General Nonlinearities

abstract: Branch-and-bound (BaB) is among the most effective methods for neural network (NN) verification. However, existing works on BaB have mostly focused on NNs with piecewise linear activations, especially ReLU networks. In this paper, we develop a general framework, named GenBaB, to conduct BaB for general nonlinearities in general computational graphs based on linear bound propagation. To decide which neuron to branch, we design a new branching heuristic which leverages linear bounds as shortcuts to efficiently estimate the potential improvement after branching. To decide nontrivial branching points for general nonlinear functions, we propose to optimize branching points offline, which can be efficiently leveraged during verification with a lookup table. We demonstrate the effectiveness of our GenBaB on verifying a wide range of NNs, including networks with activation functions such as Sigmoid, Tanh, Sine and GeLU, as well as networks involving multi-dimensional nonlinear operations such as multiplications in LSTMs and Vision Transformers. Our framework also allows the verification of general nonlinear computation graphs and enables verification applications beyond simple neural networks, particularly for AC Optimal Power Flow (ACOPF). GenBaB is part of the latest $\alpha,\!\beta$-CROWN, the winner of the 4th International Verification of Neural Networks Competition (VNN-COMP 2023).


title: Recurrent neural networks: vanishing and exploding gradients are not the end of the story

abstract: Recurrent neural networks (RNNs) notoriously struggle to learn long-term memories, primarily due to vanishing and exploding gradients. The recent success of state-space models (SSMs), a subclass of RNNs, to overcome such difficulties challenges our theoretical understanding. In this paper, we delve into the optimization challenges of RNNs and discover that, as the memory of a network increases, changes in its parameters result in increasingly large output variations, making gradient-based learning highly sensitive, even without exploding gradients. Our analysis further reveals the importance of the element-wise recurrence design pattern combined with careful parametrizations in mitigating this effect. This feature is present in SSMs, as well as in other architectures, such as LSTMs. Overall, our insights provide a new explanation for some of the difficulties in gradient-based learning of RNNs and why some architectures perform better than others.


title: Single-beam grating-chip 3D and 1D optical lattices

abstract: Ultracold atoms are crucial for unlocking truly precise and accurate quantum metrology, and provide an essential platform for quantum computing, communication and memories. One of the largest ongoing challenges is the miniaturization of these quantum devices. Here, we show that the typically macroscopic optical lattice architecture at the heart of many ultra-precise quantum technologies can be realized with a single input laser beam on the same diffractive chip already used to create the ultracold atoms. Moreover, this inherently ultra-stable platform enables access to a plethora of new lattice dimensionalities and geometries, ideally suited for the design of high-accuracy, portable quantum devices.


title: Mixed Diffusion for 3D Indoor Scene Synthesis

abstract: Realistic conditional 3D scene synthesis significantly enhances and accelerates the creation of virtual environments, which can also provide extensive training data for computer vision and robotics research among other applications. Diffusion models have shown great performance in related applications, e.g., making precise arrangements of unordered sets. However, these models have not been fully explored in floor-conditioned scene synthesis problems. We present MiDiffusion, a novel mixed discrete-continuous diffusion model architecture, designed to synthesize plausible 3D indoor scenes from given room types, floor plans, and potentially pre-existing objects. We represent a scene layout by a 2D floor plan and a set of objects, each defined by its category, location, size, and orientation. Our approach uniquely implements structured corruption across the mixed discrete semantic and continuous geometric domains, resulting in a better conditioned problem for the reverse denoising step. We evaluate our approach on the 3D-FRONT dataset. Our experimental results demonstrate that MiDiffusion substantially outperforms state-of-the-art autoregressive and diffusion models in floor-conditioned 3D scene synthesis. In addition, our models can handle partial object constraints via a corruption-and-masking strategy without task specific training. We show MiDiffusion maintains clear advantages over existing approaches in scene completion and furniture arrangement experiments.


title: Brightening and Fading in the Youngest Galactic Supernova Remnant G1.9+0.3: 13 years of monitoring with the Chandra X-ray Observatory

abstract: We report results from 13 years of Chandra monitoring of nonthermal X-ray emission from the youngest Galactic supernova remnant G1.9+0.3, the only remnant known to be increasing in brightness. We confirm the spatially-integrated flux increase rate of $(1.2 \pm 0.2)$% yr$^{-1}$ between 1 and 7 keV, but find large spatial variations, from decreases of $-3$% yr$^{-1}$ to increases of 7% yr$^{-1}$, over length scales as small as $10''$ or smaller. We observe relatively little change in spectral slope, though one region shows significant hardening (photon index $\Delta \Gamma \sim 0.4$) as it brightens by 1% yr$^{-1}$. Such rates of change can be accommodated by any of several explanations, including steady evolution of the blast wave, expansion or compression of discrete plasma blobs, strong magnetic turbulence, or variations in magnetic-field aspect angle. Our results do not constrain the mean magnetic-field strength, but a self-consistent picture of the spatially averaged rate of increase can be produced in which the maximum energies of accelerated particles are limited by the remnant age (applying both to electrons and to ions) to about 20 TeV, and the remnant-averaged magnetic field strength is about 30 $\mu$G. The deceleration parameter $m$ (average shock radius varying as $t^m$) is about 0.7, consistent with estimates from overall expansion dynamics, and confirming an explosion date of about 1900 CE. Shock-efficiency factors $\epsilon_e$ and $\epsilon_B$ (fractions of shock energy in relativistic electrons and magnetic field) are 0.003 and 0.0002 in this picture. However, the large range of rates of brightness change indicates that such a global model is oversimplified. Temporal variations of photon index, expected to be small but measurable with longer time baselines, can discriminate among possible models.


title: Code Pretraining Improves Entity Tracking Abilities of Language Models

abstract: Recent work has provided indirect evidence that pretraining language models on code improves the ability of models to track state changes of discourse entities expressed in natural language. In this work, we systematically test this claim by comparing pairs of language models on their entity tracking performance. Critically, the pairs consist of base models and models trained on top of these base models with additional code data. We extend this analysis to additionally examine the effect of math training, another highly structured data type, and alignment tuning, an important step for enhancing the usability of models. We find clear evidence that models additionally trained on large amounts of code outperform the base models. On the other hand, we find no consistent benefit of additional math training or alignment tuning across various model families.


title: Very Low Complexity Speech Synthesis Using Framewise Autoregressive GAN (FARGAN) with Pitch Prediction

abstract: Neural vocoders are now being used in a wide range of speech processing applications. In many of those applications, the vocoder can be the most complex component, so finding lower complexity algorithms can lead to significant practical benefits. In this work, we propose FARGAN, an autoregressive vocoder that takes advantage of long-term pitch prediction to synthesize high-quality speech in small subframes, without the need for teacher-forcing. Experimental results show that the proposed 600~MFLOPS FARGAN vocoder can achieve both higher quality and lower complexity than existing low-complexity vocoders. The quality even matches that of existing higher-complexity vocoders.


title: Generalization Beyond Data Imbalance: A Controlled Study on CLIP for Transferable Insights

abstract: Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community. The findings are validated in both supervised and self-supervised learning, enabling models trained on imbalanced data to achieve CLIP-level performance on diverse recognition tasks. Code and data are available at: https://github.com/CVMI-Lab/clip-beyond-tail.


title: A Multi-wavelength, Multi-epoch Monitoring Campaign of Accretion Variability in T Tauri Stars from the ODYSSEUS Survey. II. Photometric Light Curves

abstract: Classical T Tauri Stars (CTTSs) are young, low-mass stars which accrete material from their surrounding protoplanetary disk. To better understand accretion variability, we conducted a multi-epoch, multi-wavelength photometric monitoring campaign of four CTTSs: TW Hya, RU Lup, BP Tau, and GM Aur, in 2021 and 2022, contemporaneous with HST UV and optical spectra. We find that all four targets display significant variability in their light curves, generally on days-long timescales (but in some cases year-to-year) often due to periodicity associated with stellar rotation and to stochastic accretion variability. Their is a strong connection between mass accretion and photometric variability in all bands, but the relationship varies per target and epoch. Thus, photometry should be used with caution as a direct measure of accretion in CTTSs.


title: Fast inspirals and the treatment of orbital resonances

abstract: Extreme mass ratio inspirals (EMRIs), where a compact object orbits a massive black hole, are a key source of gravitational waves for the future Laser Interferometer Space Antenna (LISA). Due to their small mass ratio, ($\epsilon \sim 10^{-4}$--$10^{-7}$), the binary evolves slowly and EMRI signals will be in-band for years. Additionally, astrophysical EMRIs are expected to have complex dynamics featuring both spin-precession and eccentricity. A standard approach to modelling these inspirals is via the method of osculating geodesics (OG) which we employ along with a toy model for the gravitational self-force. Using this method requires resolving tens of thousands radial and polar orbital librations over the long duration of the signal which makes the inspiral trajectory expensive to compute. In this work we accelerate these calculations by employing Near-Identity (averaging) Transformations. However, this averaging technique breaks down at orbital resonances where the radial and polar frequencies are an integer ratio of each other. Thus, we switch to a partial averaging transformation in the vicinity of the resonance where the dynamics are characterised by the slow evolution of the so-called "resonant phase". Additionally, we develop an optimal switching criterion to minimise the computation time while maximising accuracy. We find the error in the waveform phase is improved from $\mathcal{O}(\epsilon^{-1/2})$ in the fully averaged scheme to $\mathcal{O}(\epsilon^{4/7})$ in the switching scheme. At the same time, this scheme improves the scaling of the computation time from being inversely proportional to $\epsilon$ using OG, to a very weak scaling with $\epsilon$. This results in a speed-up of at least two orders of magnitude for LISA EMRIs with room for further optimisation.


title: Toward Quantum Analogue Simulation of Many-Body Supersymmetry with Rydberg Atom Arrays

abstract: A topological quantum number, the Witten index, characterizes supersymmetric models by probing for zero energy modes and the possibility of supersymmetry breaking. We propose an averaging method to infer the Witten index in quantum analogue simulators. Motivated by recent work on Rydberg atoms trapped in optical tweezer arrays, we consider a related supersymmetric XXZ spin model. We show how to infer the Witten index from open system averaging and numerically demonstrate its topological robustness in this model. Our work defines a route for quantum analogue simulators to directly identify many-body topological physics.

categories: quant-ph cond-mat.quant-gas cond-mat.stat-mech physics.atom-ph


title: Latent Intrinsics Emerge from Training to Relight

abstract: Image relighting is the task of showing what a scene from a source image would look like if illuminated differently. Inverse graphics schemes recover an explicit representation of geometry and a set of chosen intrinsics, then relight with some form of renderer. However error control for inverse graphics is difficult, and inverse graphics methods can represent only the effects of the chosen intrinsics. This paper describes a relighting method that is entirely data-driven, where intrinsics and lighting are each represented as latent variables. Our approach produces SOTA relightings of real scenes, as measured by standard metrics. We show that albedo can be recovered from our latent intrinsics without using any example albedos, and that the albedos recovered are competitive with SOTA methods.


title: Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

abstract: In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing benchmarks through four key features: 1) Diversity in video types, spanning 6 primary visual domains with 30 subfields to ensure broad scenario generalizability; 2) Duration in temporal dimension, encompassing both short-, medium-, and long-term videos, ranging from 11 seconds to 1 hour, for robust contextual dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides video frames, including subtitles and audios, to unveil the all-round capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual labeling by expert annotators to facilitate precise and reliable model assessment. 900 videos with a total of 254 hours are manually selected and annotated by repeatedly viewing all the video content, resulting in 2,700 question-answer pairs. With Video-MME, we extensively evaluate various state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as open-source image models like InternVL-Chat-V1.5 and video models like LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the best-performing commercial model, significantly outperforming the open-source models. Our dataset along with these findings underscores the need for further improvements in handling longer sequences and multi-modal data. Project Page: https://video-mme.github.io


title: The Velocity Dispersion Function for Quiescent Galaxies in Massive Clusters from IllustrisTNG

abstract: We derive the central stellar velocity dispersion function for quiescent galaxies in 280 massive clusters with $\log (M_{200} / M_{\odot}) > 14$ in IllustrisTNG300. The velocity dispersion function is an independent tracer of the dark matter mass distribution of subhalos in galaxy clusters. Based on the IllustrisTNG cluster catalog, we select quiescent member subhalos with a specific star formation rate $< 2 \times 10^{-11}$ yr${^-1}$ and stellar mass $\log (M_{*} / M_{\odot}) > 9$. We then simulate fiber spectroscopy to measure the stellar velocity dispersion of the simulated galaxies; we compute the line-of-sight velocity dispersions of star particles within a cylindrical volume that penetrates the core of each subhalo. We construct the velocity dispersion functions for quiescent subhalos within $R_{200}$. The simulated cluster velocity dispersion function exceeds the simulated field velocity dispersion function for $\log \sigma_{*} > 2.2$, indicating the preferential formation of large velocity dispersion galaxies in dense environments. The excess is similar in simulations and in the observations. We also compare the simulated velocity dispersion function for the three most massive clusters with $\log (M_{200} / M_{\odot}) > 15$ with the observed velocity dispersion function for the two most massive clusters in the local universe, Coma and A2029. Intriguingly, the simulated velocity dispersion functions are significantly lower for $\log \sigma_{*} > 2.0$. This discrepancy results from 1) a smaller number of subhalos with $\log (M_{*} / M_{\odot}) > 10$ in TNG300 compared to the observed clusters, and 2) a significant offset between the observed and simulated $M_{*} - \sigma_{*}$ relations. The velocity dispersion function offers a unique window on galaxy and structure formation in simulations.


title: PhyPlan: Generalizable and Rapid Physical Task Planning with Physics Informed Skill Networks for Robot Manipulators

abstract: Given the task of positioning a ball-like object to a goal region beyond direct reach, humans can often throw, slide, or rebound objects against the wall to attain the goal. However, enabling robots to reason similarly is non-trivial. Existing methods for physical reasoning are data-hungry and struggle with complexity and uncertainty inherent in the real world. This paper presents PhyPlan, a novel physics-informed planning framework that combines physics-informed neural networks (PINNs) with modified Monte Carlo Tree Search (MCTS) to enable embodied agents to perform dynamic physical tasks. PhyPlan leverages PINNs to simulate and predict outcomes of actions in a fast and accurate manner and uses MCTS for planning. It dynamically determines whether to consult a PINN-based simulator (coarse but fast) or engage directly with the actual environment (fine but slow) to determine optimal policy. Given an unseen task, PhyPlan can infer the sequence of actions and learn the latent parameters, resulting in a generalizable approach that can rapidly learn to perform novel physical tasks. Evaluation with robots in simulated 3D environments demonstrates the ability of our approach to solve 3D-physical reasoning tasks involving the composition of dynamic skills. Quantitatively, PhyPlan excels in several aspects: (i) it achieves lower regret when learning novel tasks compared to the state-of-the-art, (ii) it expedites skill learning and enhances the speed of physical reasoning, (iii) it demonstrates higher data efficiency compared to a physics un-informed approach.


title: VR Isle Academy: A VR Digital Twin Approach for Robotic Surgical Skill Development

abstract: Contemporary progress in the field of robotics, marked by improved efficiency and stability, has paved the way for the global adoption of surgical robotic systems (SRS). While these systems enhance surgeons' skills by offering a more accurate and less invasive approach to operations, they come at a considerable cost. Moreover, SRS components often involve heavy machinery, making the training process challenging due to limited access to such equipment. In this paper we introduce a cost-effective way to facilitate training for a simulator of a SRS via a portable, device-agnostic, ultra realistic simulation with hand tracking and feet tracking support. Error assessment is accessible in both real-time and offline, which enables the monitoring and tracking of users' performance. The VR application has been objectively evaluated by several untrained testers showcasing significant reduction in error metrics as the number of training sessions increases. This indicates that the proposed VR application denoted as VR Isle Academy operates efficiently, improving the robot - controlling skills of the testers in an intuitive and immersive way towards reducing the learning curve at minimal cost.


title: pH-sensitive spontaneous decay of functionalised carbon dots in solutions

abstract: Carbon quantum dots have become attractive in various applications, such as drug delivery, biological sensing, photocatalysis, and solar cells. Among these, pH sensing via luminescence lifetime measurements of surface-functionalised carbon dots is one application currently investigated for their long lifetime and autonomous operation. In this manuscript, we explore the theoretical connection between excitation lifetimes and the pH value of the surrounding liquid via the protonation and deprotonation of functional groups. Example calculations applied to m-phenylenediamine, phloroglucinol and tethered disperse blue 1 are shown by applying a separation approach treating the electronic wavefunction of functional groups separately from the internal electronic structure of the (large) carbon dot. The bulk of the carbon dot is treated as an environment characterised by its optical spectrum that shifts the transition rates of the functional group. A simple relationship between pH, pKa and mixed fluorescence lifetime is derived from transition rates of the protonated and deprotonated states. pH sensitivity improves when the difference in transition rates is greatest between protonated and deprotonated species, with the greatest sensitivity found where the pKa is close to the pH region of interest. The introduced model can directly be extended to consider multicomponent liquids and multiple protonation states.


title: Navigating the Future of Federated Recommendation Systems with Foundation Models

abstract: In recent years, the integration of federated learning (FL) and recommendation systems (RS), known as Federated Recommendation Systems (FRS), has attracted attention for preserving user privacy by keeping private data on client devices. However, FRS faces inherent limitations such as data heterogeneity and scarcity, due to the privacy requirements of FL and the typical data sparsity issues of RSs. Models like ChatGPT are empowered by the concept of transfer learning and self-supervised learning, so they can be easily applied to the downstream tasks after fine-tuning or prompting. These models, so-called Foundation Models (FM), fouce on understanding the human's intent and perform following their designed roles in the specific tasks, which are widely recognized for producing high-quality content in the image and language domains. Thus, the achievements of FMs inspire the design of FRS and suggest a promising research direction: integrating foundation models to address the above limitations. In this study, we conduct a comprehensive review of FRSs with FMs. Specifically, we: 1) summarise the common approaches of current FRSs and FMs; 2) review the challenges posed by FRSs and FMs; 3) discuss potential future research directions; and 4) introduce some common benchmarks and evaluation metrics in the FRS field. We hope that this position paper provides the necessary background and guidance to explore this interesting and emerging topic.


title: Disentangling Specificity for Abstractive Multi-document Summarization

abstract: Multi-document summarization (MDS) generates a summary from a document set. Each document in a set describes topic-relevant concepts, while per document also has its unique contents. However, the document specificity receives little attention from existing MDS approaches. Neglecting specific information for each document limits the comprehensiveness of the generated summaries. To solve this problem, in this paper, we propose to disentangle the specific content from documents in one document set. The document-specific representations, which are encouraged to be distant from each other via a proposed orthogonal constraint, are learned by the specific representation learner. We provide extensive analysis and have interesting findings that specific information and document set representations contribute distinctive strengths and their combination yields a more comprehensive solution for the MDS. Also, we find that the common (i.e. shared) information could not contribute much to the overall performance under the MDS settings. Implemetation codes are available at https://github.com/congboma/DisentangleSum.


title: A Prompt-driven Task Planning Method for Multi-drones based on Large Language Model

abstract: With the rapid development of drone technology, the application of multi-drones is becoming increasingly widespread in various fields. However, the task planning technology for multi-drones still faces challenges such as the complexity of remote operation and the convenience of human-machine interaction. To address these issues, this paper proposes a prompt-driven task planning method for multi-drones based on large language models. By introducing the Prompt technique, appropriate prompt information is provided for the multi-drone system.


title: PyTorch-IE: Fast and Reproducible Prototyping for Information Extraction

abstract: The objective of Information Extraction (IE) is to derive structured representations from unstructured or semi-structured documents. However, developing IE models is complex due to the need of integrating several subtasks. Additionally, representation of data among varied tasks and transforming datasets into task-specific model inputs presents further challenges. To streamline this undertaking for researchers, we introduce PyTorch-IE, a deep-learning-based framework uniquely designed to enable swift, reproducible, and reusable implementations of IE models. PyTorch-IE offers a flexible data model capable of creating complex data structures by integrating interdependent layers of annotations derived from various data types, like plain text or semi-structured text, and even images. We propose task modules to decouple the concerns of data representation and model-specific representations, thereby fostering greater flexibility and reusability of code. PyTorch-IE also extends support for widely used libraries such as PyTorch-Lightning for training, HuggingFace datasets for dataset reading, and Hydra for experiment configuration. Supplementary libraries and GitHub templates for the easy setup of new projects are also provided. By ensuring functionality and versatility, PyTorch-IE provides vital support to the research community engaged in Information Extraction.


title: KnowledgeHub: An end-to-end Tool for Assisted Scientific Discovery

abstract: This paper describes the KnowledgeHub tool, a scientific literature Information Extraction (IE) and Question Answering (QA) pipeline. This is achieved by supporting the ingestion of PDF documents that are converted to text and structured representations. An ontology can then be constructed where a user defines the types of entities and relationships they want to capture. A browser-based annotation tool enables annotating the contents of the PDF documents according to the ontology. Named Entity Recognition (NER) and Relation Classification (RC) models can be trained on the resulting annotations and can be used to annotate the unannotated portion of the documents. A knowledge graph is constructed from these entity and relation triples which can be queried to obtain insights from the data. Furthermore, we integrate a suite of Large Language Models (LLMs) that can be used for QA and summarisation that is grounded in the included documents via a retrieval component. KnowledgeHub is a unique tool that supports annotation, IE and QA, which gives the user full insight into the knowledge discovery pipeline.


title: ULTra-AV: A Unified Longitudinal Trajectory Dataset for Automated Vehicle

abstract: Automated Vehicles (AVs) promise significant advances in transportation. Critical to these improvements is understanding AVs' longitudinal behavior, relying heavily on real-world trajectory data. Existing open-source trajectory datasets of AV, however, often fall short in refinement, reliability, and completeness, hindering effective performance metrics analysis and model development. This study addresses these challenges by creating a Unified Longitudinal TRAjectory dataset for AVs (Ultra-AV) to analyze their microscopic longitudinal driving behaviors. This dataset compiles data from 13 distinct sources, encompassing various AV types, test sites, and experiment scenarios. We established a three-step data processing: 1. extraction of longitudinal trajectory data, 2. general data cleaning, and 3. data-specific cleaning to obtain the longitudinal trajectory data and car-following trajectory data. The validity of the processed data is affirmed through performance evaluations across safety, mobility, stability, and sustainability, along with an analysis of the relationships between variables in car-following models. Our work not only furnishes researchers with standardized data and metrics for longitudinal AV behavior studies but also sets guidelines for data collection and model development.


title: EnterpriseEM: Fine-tuned Embeddings for Enterprise Semantic Search

abstract: Enterprises grapple with the significant challenge of managing proprietary unstructured data, hindering efficient information retrieval. This has led to the emergence of AI-driven information retrieval solutions, designed to adeptly extract relevant insights to address employee inquiries. These solutions often leverage pre-trained embedding models and generative models as foundational components. While pre-trained embeddings may exhibit proximity or disparity based on their original training objectives, they might not fully align with the unique characteristics of enterprise-specific data, leading to suboptimal alignment with the retrieval goals of enterprise environments. In this paper, we propose a methodology to fine-tune pre-trained embedding models specifically for enterprise environments. By adapting the embeddings to better suit the retrieval tasks prevalent in enterprises, we aim to enhance the performance of information retrieval solutions. We discuss the process of fine-tuning, its effect on retrieval accuracy, and the potential benefits for enterprise information management. Our findings demonstrate the efficacy of fine-tuned embedding models in improving the precision and relevance of search results in enterprise settings.


title: DisCo: Towards Harmonious Disentanglement and Collaboration between Tabular and Semantic Space for Recommendation

abstract: Recommender systems play important roles in various applications such as e-commerce, social media, etc. Conventional recommendation methods usually model the collaborative signals within the tabular representation space. Despite the personalization modeling and the efficiency, the latent semantic dependencies are omitted. Methods that introduce semantics into recommendation then emerge, injecting knowledge from the semantic representation space where the general language understanding are compressed. However, existing semantic-enhanced recommendation methods focus on aligning the two spaces, during which the representations of the two spaces tend to get close while the unique patterns are discarded and not well explored. In this paper, we propose DisCo to Disentangle the unique patterns from the two representation spaces and Collaborate the two spaces for recommendation enhancement, where both the specificity and the consistency of the two spaces are captured. Concretely, we propose 1) a dual-side attentive network to capture the intra-domain patterns and the inter-domain patterns, 2) a sufficiency constraint to preserve the task-relevant information of each representation space and filter out the noise, and 3) a disentanglement constraint to avoid the model from discarding the unique information. These modules strike a balance between disentanglement and collaboration of the two representation spaces to produce informative pattern vectors, which could serve as extra features and be appended to arbitrary recommendation backbones for enhancement. Experiment results validate the superiority of our method against different models and the compatibility of DisCo over different backbones. Various ablation studies and efficiency analysis are also conducted to justify each model component.


title: Extracting Essential and Disentangled Knowledge for Recommendation Enhancement

abstract: Recommender models play a vital role in various industrial scenarios, while often faced with the catastrophic forgetting problem caused by the fast shifting data distribution, e.g., the evolving user interests, click signals fluctuation during sales promotions, etc. To alleviate this problem, a common approach is to reuse knowledge from the historical data. However, preserving the vast and fast-accumulating data is hard, which causes dramatic storage overhead. Memorizing old data through a parametric knowledge base is then proposed, which compresses the vast amount of raw data into model parameters. Despite the flexibility, how to improve the memorization and generalization capabilities of the parametric knowledge base is challenging. In this paper, we propose two constraints to extract Essential and Disentangled Knowledge from past data for rational and generalized recommendation enhancement, which improves the capabilities of the parametric knowledge base without increasing the size of it. The essential principle helps to compress the input into representative vectors that capture the task-relevant information and filter out the noisy information. The disentanglement principle reduces the redundancy of stored information and pushes the knowledge base to focus on capturing the disentangled invariant patterns. These two rules together promote rational compression of information for robust and generalized knowledge representations. Extensive experiments on two datasets justify the effectiveness of the proposed method.


title: Thesis: Document Summarization with applications to Keyword extraction and Image Retrieval

abstract: Automatic summarization is the process of reducing a text document in order to generate a summary that retains the most important points of the original document. In this work, we study two problems - i) summarizing a text document as set of keywords/caption, for image recommedation, ii) generating opinion summary which good mix of relevancy and sentiment with the text document. Intially, we present our work on an recommending images for enhancing a substantial amount of existing plain text news articles. We use probabilistic models and word similarity heuristics to generate captions and extract Key-phrases which are re-ranked using a rank aggregation framework with relevance feedback mechanism. We show that such rank aggregation and relevant feedback which are typically used in Tagging Documents, Text Information Retrieval also helps in improving image retrieval. These queries are fed to the Yahoo Search Engine to obtain relevant images 1. Our proposed method is observed to perform better than all existing baselines. Additonally, We propose a set of submodular functions for opinion summarization. Opinion summarization has built in it the tasks of summarization and sentiment detection. However, it is not easy to detect sentiment and simultaneously extract summary. The two tasks conflict in the sense that the demand of compression may drop sentiment bearing sentences, and the demand of sentiment detection may bring in redundant sentences. However, using submodularity we show how to strike a balance between the two requirements. Our functions generate summaries such that there is good correlation between document sentiment and summary sentiment along with good ROUGE score. We also compare the performances of the proposed submodular functions.


title: KU-DMIS at EHRSQL 2024:Generating SQL query via question templatization in EHR

abstract: Transforming natural language questions into SQL queries is crucial for precise data retrieval from electronic health record (EHR) databases. A significant challenge in this process is detecting and rejecting unanswerable questions that request information beyond the database's scope or exceed the system's capabilities. In this paper, we introduce a novel text-to-SQL framework that robustly handles out-of-domain questions and verifies the generated queries with query execution.Our framework begins by standardizing the structure of questions into a templated format. We use a powerful large language model (LLM), fine-tuned GPT-3.5 with detailed prompts involving the table schemas of the EHR database system. Our experimental results demonstrate the effectiveness of our framework on the EHRSQL-2024 benchmark benchmark, a shared task in the ClinicalNLP workshop. Although a straightforward fine-tuning of GPT shows promising results on the development set, it struggled with the out-of-domain questions in the test set. With our framework, we improve our system's adaptability and achieve competitive performances in the official leaderboard of the EHRSQL-2024 challenge.


title: Use of natural language processing to extract and classify papillary thyroid cancer features from surgical pathology reports

abstract: Background We aim to use Natural Language Processing (NLP) to automate the extraction and classification of thyroid cancer risk factors from pathology reports. Methods We analyzed 1,410 surgical pathology reports from adult papillary thyroid cancer patients at Mayo Clinic, Rochester, MN, from 2010 to 2019. Structured and non-structured reports were used to create a consensus-based ground truth dictionary and categorized them into modified recurrence risk levels. Non-structured reports were narrative, while structured reports followed standardized formats. We then developed ThyroPath, a rule-based NLP pipeline, to extract and classify thyroid cancer features into risk categories. Training involved 225 reports (150 structured, 75 unstructured), with testing on 170 reports (120 structured, 50 unstructured) for evaluation. The pipeline's performance was assessed using both strict and lenient criteria for accuracy, precision, recall, and F1-score. Results In extraction tasks, ThyroPath achieved overall strict F-1 scores of 93% for structured reports and 90 for unstructured reports, covering 18 thyroid cancer pathology features. In classification tasks, ThyroPath-extracted information demonstrated an overall accuracy of 93% in categorizing reports based on their corresponding guideline-based risk of recurrence: 76.9% for high-risk, 86.8% for intermediate risk, and 100% for both low and very low-risk cases. However, ThyroPath achieved 100% accuracy across all thyroid cancer risk categories with human-extracted pathology information. Conclusions ThyroPath shows promise in automating the extraction and risk recurrence classification of thyroid pathology reports at large scale. It offers a solution to laborious manual reviews and advancing virtual registries. However, it requires further validation before implementation.


title: Exploration of Attention Mechanism-Enhanced Deep Learning Models in the Mining of Medical Textual Data

abstract: The research explores the utilization of a deep learning model employing an attention mechanism in medical text mining. It targets the challenge of analyzing unstructured text information within medical data. This research seeks to enhance the model's capability to identify essential medical information by incorporating deep learning and attention mechanisms. This paper reviews the basic principles and typical model architecture of attention mechanisms and shows the effectiveness of their application in the tasks of disease prediction, drug side effect monitoring, and entity relationship extraction. Aiming at the particularity of medical texts, an adaptive attention model integrating domain knowledge is proposed, and its ability to understand medical terms and process complex contexts is optimized. The experiment verifies the model's effectiveness in improving task accuracy and robustness, especially when dealing with long text. The future research path of enhancing model interpretation, realizing cross-domain knowledge transfer, and adapting to low-resource scenarios is discussed in the research outlook, which provides a new perspective and method support for intelligent medical information processing and clinical decision assistance. Finally, cross-domain knowledge transfer and adaptation strategies for low-resource scenarios, providing theoretical basis and technical reference for promoting the development of intelligent medical information processing and clinical decision support systems.


title: PTA: Enhancing Multimodal Sentiment Analysis through Pipelined Prediction and Translation-based Alignment

abstract: Multimodal aspect-based sentiment analysis (MABSA) aims to understand opinions in a granular manner, advancing human-computer interaction and other fields. Traditionally, MABSA methods use a joint prediction approach to identify aspects and sentiments simultaneously. However, we argue that joint models are not always superior. Our analysis shows that joint models struggle to align relevant text tokens with image patches, leading to misalignment and ineffective image utilization. In contrast, a pipeline framework first identifies aspects through MATE (Multimodal Aspect Term Extraction) and then aligns these aspects with image patches for sentiment classification (MASC: Multimodal Aspect-Oriented Sentiment Classification). This method is better suited for multimodal scenarios where effective image use is crucial. We present three key observations: (a) MATE and MASC have different feature requirements, with MATE focusing on token-level features and MASC on sequence-level features; (b) the aspect identified by MATE is crucial for effective image utilization; and (c) images play a trivial role in previous MABSA methods due to high noise. Based on these observations, we propose a pipeline framework that first predicts the aspect and then uses translation-based alignment (TBA) to enhance multimodal semantic consistency for better image utilization. Our method achieves state-of-the-art (SOTA) performance on widely used MABSA datasets Twitter-15 and Twitter-17. This demonstrates the effectiveness of the pipeline approach and its potential to provide valuable insights for future MABSA research. For reproducibility, the code and checkpoint will be released.


title: Large Language Models' Detection of Political Orientation in Newspapers

abstract: Democratic opinion-forming may be manipulated if newspapers' alignment to political or economical orientation is ambiguous. Various methods have been developed to better understand newspapers' positioning. Recently, the advent of Large Language Models (LLM), and particularly the pre-trained LLM chatbots like ChatGPT or Gemini, hold disruptive potential to assist researchers and citizens alike. However, little is know on whether LLM assessment is trustworthy: do single LLM agrees with experts' assessment, and do different LLMs answer consistently with one another? In this paper, we address specifically the second challenge. We compare how four widely employed LLMs rate the positioning of newspapers, and compare if their answers align with one another. We observe that this is not the case. Over a woldwide dataset, articles in newspapers are positioned strikingly differently by single LLMs, hinting to inconsistent training or excessive randomness in the algorithms. We thus raise a warning when deciding which tools to use, and we call for better training and algorithm development, to cover such significant gap in a highly sensitive matter for democracy and societies worldwide. We also call for community engagement in benchmark evaluation, through our open initiative navai.pro.


title: EHR-SeqSQL : A Sequential Text-to-SQL Dataset For Interactively Exploring Electronic Health Records

abstract: In this paper, we introduce EHR-SeqSQL, a novel sequential text-to-SQL dataset for Electronic Health Record (EHR) databases. EHR-SeqSQL is designed to address critical yet underexplored aspects in text-to-SQL parsing: interactivity, compositionality, and efficiency. To the best of our knowledge, EHR-SeqSQL is not only the largest but also the first medical text-to-SQL dataset benchmark to include sequential and contextual questions. We provide a data split and the new test set designed to assess compositional generalization ability. Our experiments demonstrate the superiority of a multi-turn approach over a single-turn approach in learning compositionality. Additionally, our dataset integrates specially crafted tokens into SQL queries to improve execution efficiency. With EHR-SeqSQL, we aim to bridge the gap between practical needs and academic research in the text-to-SQL domain.


title: Harmful Speech Detection by Language Models Exhibits Gender-Queer Dialect Bias

abstract: Content moderation on social media platforms shapes the dynamics of online discourse, influencing whose voices are amplified and whose are suppressed. Recent studies have raised concerns about the fairness of content moderation practices, particularly for aggressively flagging posts from transgender and non-binary individuals as toxic. In this study, we investigate the presence of bias in harmful speech classification of gender-queer dialect online, focusing specifically on the treatment of reclaimed slurs. We introduce a novel dataset, QueerReclaimLex, based on 109 curated templates exemplifying non-derogatory uses of LGBTQ+ slurs. Dataset instances are scored by gender-queer annotators for potential harm depending on additional context about speaker identity. We systematically evaluate the performance of five off-the-shelf language models in assessing the harm of these texts and explore the effectiveness of chain-of-thought prompting to teach large language models (LLMs) to leverage author identity context. We reveal a tendency for these models to inaccurately flag texts authored by gender-queer individuals as harmful. Strikingly, across all LLMs the performance is poorest for texts that show signs of being written by individuals targeted by the featured slur (F1 <= 0.24). We highlight an urgent need for fairness and inclusivity in content moderation systems. By uncovering these biases, this work aims to inform the development of more equitable content moderation practices and contribute to the creation of inclusive online spaces for all users.


title: CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning

abstract: This paper presents CrossVoice, a novel cascade-based Speech-to-Speech Translation (S2ST) system employing advanced ASR, MT, and TTS technologies with cross-lingual prosody preservation through transfer learning. We conducted comprehensive experiments comparing CrossVoice with direct-S2ST systems, showing improved BLEU scores on tasks such as Fisher Es-En, VoxPopuli Fr-En and prosody preservation on benchmark datasets CVSS-T and IndicTTS. With an average mean opinion score of 3.75 out of 4, speech synthesized by CrossVoice closely rivals human speech on the benchmark, highlighting the efficacy of cascade-based systems and transfer learning in multilingual S2ST with prosody transfer.


title: Multilingual Prosody Transfer: Comparing Supervised & Transfer Learning

abstract: The field of prosody transfer in speech synthesis systems is rapidly advancing. This research is focused on evaluating learning methods for adapting pre-trained monolingual text-to-speech (TTS) models to multilingual conditions, i.e., Supervised Fine-Tuning (SFT) and Transfer Learning (TL). This comparison utilizes three distinct metrics: Mean Opinion Score (MOS), Recognition Accuracy (RA), and Mel Cepstral Distortion (MCD). Results demonstrate that, in comparison to SFT, TL leads to significantly enhanced performance, with an average MOS higher by 1.53 points, a 37.5% increase in RA, and approximately a 7.8-point improvement in MCD. These findings are instrumental in helping build TTS models for low-resource languages.


title: LocMoE+: Enhanced Router with Token Feature Awareness for Efficient LLM Pre-Training

abstract: Mixture-of-Experts (MoE) architectures have recently gained increasing popularity within the domain of large language models (LLMs) due to their ability to significantly reduce training and inference overhead. However, MoE architectures face challenges, such as significant disparities in the number of tokens assigned to each expert and a tendency toward homogenization among experts, which adversely affects the model's semantic generation capabilities. In this paper, we introduce LocMoE+, a refined version of the low-overhead LocMoE, incorporating the following enhancements: (1) Quantification and definition of the affinity between experts and tokens. (2) Implementation of a global-level adaptive routing strategy to rearrange tokens based on their affinity scores. (3) Reestimation of the lower bound for expert capacity, which has been shown to progressively decrease as the token feature distribution evolves. Experimental results demonstrate that, without compromising model convergence or efficacy, the number of tokens each expert processes can be reduced by over 60%. Combined with communication optimizations, this leads to an average improvement in training efficiency ranging from 5.4% to 46.6%. After fine-tuning, LocMoE+ exhibits a performance improvement of 9.7% to 14.1% across the GDAD, C-Eval, and TeleQnA datasets.


title: Embedding-Aligned Language Models

abstract: We propose a novel approach for training large language models (LLMs) to adhere to objectives defined within a latent embedding space. Our method leverages reinforcement learning (RL), treating a pre-trained LLM as an environment. Our embedding-aligned guided language (EAGLE) agent is trained to iteratively steer the LLM's generation towards optimal regions of the latent embedding space, w.r.t. some predefined criterion. We demonstrate the effectiveness of the EAGLE agent using the MovieLens 25M dataset to surface content gaps that satisfy latent user demand. We also demonstrate the benefit of using an optimal design of a state-dependent action set to improve EAGLE's efficiency. Our work paves the way for controlled and grounded text generation using LLMs, ensuring consistency with domain-specific knowledge and data representations.


title: SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models

abstract: Large Language Models (LLMs) have become increasingly popular, transforming a wide range of applications across various domains. However, the real-world effectiveness of their query cache systems has not been thoroughly investigated. In this work, we for the first time conducted an analysis on real-world human-to-LLM interaction data, identifying key challenges in existing caching solutions for LLM-based chat services. Our findings reveal that current caching methods fail to leverage semantic connections, leading to inefficient cache performance and extra token costs. To address these issues, we propose SCALM, a new cache architecture that emphasizes semantic analysis and identifies significant cache entries and patterns. We also detail the implementations of the corresponding cache storage and eviction strategies. Our evaluations show that SCALM increases cache hit ratios and reduces operational costs for LLMChat services. Compared with other state-of-the-art solutions in GPTCache, SCALM shows, on average, a relative increase of 63% in cache hit ratio and a relative improvement of 77% in tokens savings.


title: Easy real-time collision detection

abstract: This article presents an easy distance field-based collision detection scheme to detect collisions of an object with its environment. Through the clever use of back-face culling and z-buffering, the solution is precise and very easy to implement. Since the complete scheme relies on the graphics pipeline, the collision detection is performed by the GPU. It is easy to use and only requires the meshes of the object and the scene; it does not rely on special representations. It can natively handle collision with primitives emitted directly on the pipeline. Our scheme is efficient and we expose many possible variants (especially an adaptation to certain particle systems). The main limitation of our scheme is that it imposes some restrictions on the shape of the considered objects - but not on their environment. We evaluate our scheme by first, comparing with the FCL, second, testing a more complete scene (involving geometry shader, tessellation and compute shader) and last, illustrating with a particle system.


title: Adapting PromptORE for Modern History: Information Extraction from Hispanic Monarchy Documents of the XVIth Century

abstract: Semantic relations among entities are a widely accepted method for relation extraction. PromptORE (Prompt-based Open Relation Extraction) was designed to improve relation extraction with Large Language Models on generalistic documents. However, it is less effective when applied to historical documents, in languages other than English. In this study, we introduce an adaptation of PromptORE to extract relations from specialized documents, namely digital transcripts of trials from the Spanish Inquisition. Our approach involves fine-tuning transformer models with their pretraining objective on the data they will perform inference. We refer to this process as "biasing". Our Biased PromptORE addresses complex entity placements and genderism that occur in Spanish texts. We solve these issues by prompt engineering. We evaluate our method using Encoder-like models, corroborating our findings with experts' assessments. Additionally, we evaluate the performance using a binomial classification benchmark. Our results show a substantial improvement in accuracy -up to a 50% improvement with our Biased PromptORE models in comparison to the baseline models using standard PromptORE.


title: Persian Homograph Disambiguation: Leveraging ParsBERT for Enhanced Sentence Understanding with a Novel Word Disambiguation Dataset

abstract: Homograph disambiguation, the task of distinguishing words with identical spellings but different meanings, poses a substantial challenge in natural language processing. In this study, we introduce a novel dataset tailored for Persian homograph disambiguation. Our work encompasses a thorough exploration of various embeddings, evaluated through the cosine similarity method and their efficacy in downstream tasks like classification. Our investigation entails training a diverse array of lightweight machine learning and deep learning models for phonograph disambiguation. We scrutinize the models' performance in terms of Accuracy, Recall, and F1 Score, thereby gaining insights into their respective strengths and limitations. The outcomes of our research underscore three key contributions. First, we present a newly curated Persian dataset, providing a solid foundation for future research in homograph disambiguation. Second, our comparative analysis of embeddings highlights their utility in different contexts, enriching the understanding of their capabilities. Third, by training and evaluating a spectrum of models, we extend valuable guidance for practitioners in selecting suitable strategies for homograph disambiguation tasks. In summary, our study unveils a new dataset, scrutinizes embeddings through diverse perspectives, and benchmarks various models for homograph disambiguation. These findings empower researchers and practitioners to navigate the intricate landscape of homograph-related challenges effectively.


title: Clustered Retrieved Augmented Generation (CRAG)

abstract: Providing external knowledge to Large Language Models (LLMs) is a key point for using these models in real-world applications for several reasons, such as incorporating up-to-date content in a real-time manner, providing access to domain-specific knowledge, and contributing to hallucination prevention. The vector database-based Retrieval Augmented Generation (RAG) approach has been widely adopted to this end. Thus, any part of external knowledge can be retrieved and provided to some LLM as the input context. Despite RAG approach's success, it still might be unfeasible for some applications, because the context retrieved can demand a longer context window than the size supported by LLM. Even when the context retrieved fits into the context window size, the number of tokens might be expressive and, consequently, impact costs and processing time, becoming impractical for most applications. To address these, we propose CRAG, a novel approach able to effectively reduce the number of prompting tokens without degrading the quality of the response generated compared to a solution using RAG. Through our experiments, we show that CRAG can reduce the number of tokens by at least 46\%, achieving more than 90\% in some cases, compared to RAG. Moreover, the number of tokens with CRAG does not increase considerably when the number of reviews analyzed is higher, unlike RAG, where the number of tokens is almost 9x higher when there are 75 reviews compared to 4 reviews.


title: Large Language Model Pruning

abstract: We surely enjoy the larger the better models for their superior performance in the last couple of years when both the hardware and software support the birth of such extremely huge models. The applied fields include text mining and others. In particular, the success of LLMs on text understanding and text generation draws attention from researchers who have worked on NLP and related areas for years or even decades. On the side, LLMs may suffer from problems like model overfitting, hallucination, and device limitation to name a few. In this work, we suggest a model pruning technique specifically focused on LLMs. The proposed methodology emphasizes the explainability of deep learning models. By having the theoretical foundation, we obtain a trustworthy deep model so that huge models with a massive number of model parameters become not quite necessary. A mutual information-based estimation is adopted to find neurons with redundancy to eliminate. Moreover, an estimator with well-tuned parameters helps to find precise estimation to guide the pruning procedure. At the same time, we also explore the difference between pruning on large-scale models vs. pruning on small-scale models. The choice of pruning criteria is sensitive in small models but not for large-scale models. It is a novel finding through this work. Overall, we demonstrate the superiority of the proposed model to the state-of-the-art models.


title: AMGPT: a Large Language Model for Contextual Querying in Additive Manufacturing

abstract: Generalized large language models (LLMs) such as GPT-4 may not provide specific answers to queries formulated by materials science researchers. These models may produce a high-level outline but lack the capacity to return detailed instructions on manufacturing and material properties of novel alloys. Enhancing a smaller model with specialized domain knowledge may provide an advantage over large language models which cannot be retrained quickly enough to keep up with the rapid pace of research in metal additive manufacturing (AM). We introduce "AMGPT," a specialized LLM text generator designed for metal AM queries. The goal of AMGPT is to assist researchers and users in navigating the extensive corpus of literature in AM. Instead of training from scratch, we employ a pre-trained Llama2-7B model from Hugging Face in a Retrieval-Augmented Generation (RAG) setup, utilizing it to dynamically incorporate information from $\sim$50 AM papers and textbooks in PDF format. Mathpix is used to convert these PDF documents into TeX format, facilitating their integration into the RAG pipeline managed by LlamaIndex. Expert evaluations of this project highlight that specific embeddings from the RAG setup accelerate response times and maintain coherence in the generated text.


title: Paths of A Million People: Extracting Life Trajectories from Wikipedia

abstract: Notable people's life trajectories have been a focus of study -- the locations and times of various activities, such as birth, death, education, marriage, competition, work, delivering a speech, making a scientific discovery, finishing a masterpiece, and fighting a battle, and how these people interact with others, carry important messages for the broad research related to human dynamics. However, the scarcity of trajectory data in terms of volume, density, and inter-person interactions, limits relevant studies from being comprehensive and interactive. We mine millions of biography pages from Wikipedia and tackle the generalization problem stemming from the variety and heterogeneity of the trajectory descriptions. Our ensemble model COSMOS, which combines the idea of semi-supervised learning and contrastive learning, achieves an F1 score of 85.95%. For this task, we also create a hand-curated dataset, WikiLifeTrajectory, consisting of 8,852 (person, time, location) triplets as ground truth. Besides, we perform an empirical analysis on the trajectories of 8,272 historians to demonstrate the validity of the extracted results. To facilitate the research on trajectory extractions and help the analytical studies to construct grand narratives, we make our code, the million-level extracted trajectories, and the WikiLifeTrajectory dataset publicly available.


title: Retrieval-Augmented Conversational Recommendation with Prompt-based Semi-Structured Natural Language State Tracking

abstract: Conversational recommendation (ConvRec) systems must understand rich and diverse natural language (NL) expressions of user preferences and intents, often communicated in an indirect manner (e.g., "I'm watching my weight"). Such complex utterances make retrieving relevant items challenging, especially if only using often incomplete or out-of-date metadata. Fortunately, many domains feature rich item reviews that cover standard metadata categories and offer complex opinions that might match a user's interests (e.g., "classy joint for a date"). However, only recently have large language models (LLMs) let us unlock the commonsense connections between user preference utterances and complex language in user-generated reviews. Further, LLMs enable novel paradigms for semi-structured dialogue state tracking, complex intent and preference understanding, and generating recommendations, explanations, and question answers. We thus introduce a novel technology RA-Rec, a Retrieval-Augmented, LLM-driven dialogue state tracking system for ConvRec, showcased with a video, open source GitHub repository, and interactive Google Colab notebook.