title: MDA: Availability-Aware Federated Learning Client Selection

abstract: Recently, a new distributed learning scheme called Federated Learning (FL) has been introduced. FL is designed so that server never collects user-owned data meaning it is great at preserving privacy. FL's process starts with the server sending a model to clients, then the clients train that model using their data and send the updated model back to the server. Afterward, the server aggregates all the updates and modifies the global model. This process is repeated until the model converges. This study focuses on an FL setting called cross-device FL, which trains based on a large number of clients. Since many devices may be unavailable in cross-device FL, and communication between the server and all clients is extremely costly, only a fraction of clients gets selected for training at each round. In vanilla FL, clients are selected randomly, which results in an acceptable accuracy but is not ideal from the overall training time perspective, since some clients are slow and can cause some training rounds to be slow. If only fast clients get selected the learning would speed up, but it will be biased toward only the fast clients' data, and the accuracy degrades. Consequently, new client selection techniques have been proposed to improve the training time by considering individual clients' resources and speed. This paper introduces the first availability-aware selection strategy called MDA. The results show that our approach makes learning faster than vanilla FL by up to 6.5%. Moreover, we show that resource heterogeneity-aware techniques are effective but can become even better when combined with our approach, making it faster than the state-of-the-art selectors by up to 16%. Lastly, our approach selects more unique clients for training compared to client selectors that only select fast clients, which reduces our technique's bias.


title: Ultra-Fast Semi-Empirical Quantum Chemistry for High-Throughput Computational Campaigns with Sparrow

abstract: Semi-empirical quantum chemical approaches are known to compromise accuracy for feasibility of calculations on huge molecules. However, the need for ultrafast calculations in interactive quantum mechanical studies, high-throughput virtual screening, and for data-driven machine learning has shifted the emphasis towards calculation runtimes recently. This comes with new constraints for the software implementation as many fast calculations would suffer from a large overhead of manual setup and other procedures that are comparatively fast when studying a single molecular structure, but which become prohibitively slow for high-throughput demands. In this work, we discuss the effect of various well-established semi-empirical approximations on calculation speed and relate this to data transfer rates from the raw-data source computer to the results visualization front end. For the former, we consider desktop computers, local high performance computing, as well as remote cloud services in order to elucidate the effect on interactive calculations, for web and cloud interfaces in local applications, and in world-wide interactive virtual sessions. The models discussed in this work have been implemented into our open-source software SCINE Sparrow.


title: FedSysID: A Federated Approach to Sample-Efficient System Identification

abstract: We study the problem of learning a linear system model from the observations of $M$ clients. The catch: Each client is observing data from a different dynamical system. This work addresses the question of how multiple clients collaboratively learn dynamical models in the presence of heterogeneity. We pose this problem as a federated learning problem and characterize the tension between achievable performance and system heterogeneity. Furthermore, our federated sample complexity result provides a constant factor improvement over the single agent setting. Finally, we describe a meta federated learning algorithm, FedSysID, that leverages existing federated algorithms at the client level.


title: Link Prediction with Non-Contrastive Learning

abstract: A recent focal area in the space of graph neural networks (GNNs) is graph self-supervised learning (SSL), which aims to derive useful node representations without labeled data. Notably, many state-of-the-art graph SSL methods are contrastive methods, which use a combination of positive and negative samples to learn node representations. Owing to challenges in negative sampling (slowness and model sensitivity), recent literature introduced non-contrastive methods, which instead only use positive samples. Though such methods have shown promising performance in node-level tasks, their suitability for link prediction tasks, which are concerned with predicting link existence between pairs of nodes (and have broad applicability to recommendation systems contexts) is yet unexplored. In this work, we extensively evaluate the performance of existing non-contrastive methods for link prediction in both transductive and inductive settings. While most existing non-contrastive methods perform poorly overall, we find that, surprisingly, BGRL generally performs well in transductive settings. However, it performs poorly in the more realistic inductive settings where the model has to generalize to links to/from unseen nodes. We find that non-contrastive models tend to overfit to the training graph and use this analysis to propose T-BGRL, a novel non-contrastive framework that incorporates cheap corruptions to improve the generalization ability of the model. This simple modification strongly improves inductive performance in 5/6 of our datasets, with up to a 120% improvement in Hits@50--all with comparable speed to other non-contrastive baselines and up to 14x faster than the best-performing contrastive baseline. Our work imparts interesting findings about non-contrastive learning for link prediction and paves the way for future researchers to further expand upon this area.


title: Deep Learning Training Procedure Augmentations

abstract: Recent advances in Deep Learning have greatly improved performance on various tasks such as object detection, image segmentation, sentiment analysis. The focus of most research directions up until very recently has been on beating state-of-the-art results. This has materialized in the utilization of bigger and bigger models and techniques which help the training procedure to extract more predictive power out of a given dataset. While this has lead to great results, many of which with real-world applications, other relevant aspects of deep learning have remained neglected and unknown. In this work, we will present several novel deep learning training techniques which, while capable of offering significant performance gains they also reveal several interesting analysis results regarding convergence speed, optimization landscape smoothness, and adversarial robustness. The methods presented in this work are the following: $\bullet$ Perfect Ordering Approximation; a generalized model agnostic curriculum learning approach. The results show the effectiveness of the technique for improving training time as well as offer some new insight into the training process of deep networks. $\bullet$ Cascading Sum Augmentation; an extension of mixup capable of utilizing more data points for linear interpolation by leveraging a smoother optimization landscape. This can be used for computer vision tasks in order to improve both prediction performance as well as improve passive model robustness.


title: Non-invasive Liver Fibrosis Screening on CT Images using Radiomics

abstract: Objectives: To develop and evaluate a radiomics machine learning model for detecting liver fibrosis on CT of the liver. Methods: For this retrospective, single-centre study, radiomic features were extracted from Regions of Interest (ROIs) on CT images of patients who underwent simultaneous liver biopsy and CT examinations. Combinations of contrast, normalization, machine learning model, and feature selection method were determined based on their mean test Area Under the Receiver Operating Characteristic curve (AUC) on randomly placed ROIs. The combination and selected features with the highest AUC were used to develop a final liver fibrosis screening model. Results: The study included 101 male and 68 female patients (mean age = 51.2 years $\pm$ 14.7 [SD]). When averaging the AUC across all combinations, non-contrast enhanced (NC) CT (AUC, 0.6100; 95% CI: 0.5897, 0.6303) outperformed contrast-enhanced CT (AUC, 0.5680; 95% CI: 0.5471, 0.5890). The combination of hyperparameters and features that yielded the highest AUC was a logistic regression model with inputs features of maximum, energy, kurtosis, skewness, and small area high gray level emphasis extracted from non-contrast enhanced NC CT normalized using Gamma correction with $\gamma$ = 1.5 (AUC, 0.7833; 95% CI: 0.7821, 0.7845), (sensitivity, 0.9091; 95% CI: 0.9091, 0.9091). Conclusions: Radiomics-based machine learning models allow for the detection of liver fibrosis with reasonable accuracy and high sensitivity on NC CT. Thus, these models can be used to non-invasively screen for liver fibrosis, contributing to earlier detection of the disease at a potentially curable stage.


title: Improved Factorization for Threshold Resummation in Heavy Quark to Heavy Quark Decays

abstract: We consider the resummation of soft-gluon effects in heavy quark to heavy quark decays, namely the processes Q1 -> Q2 + (non QCD partons), where Q1 and Q2 are two different heavy quarks. We construct a new factorization scheme for threshold resummed spectra, which allows us to consistently evaluate the distribution of the final hadron invariant mass mX in all the kinematic regions, i.e. when mX is smaller, of the same order, or larger than the mass of the final quark Q2. A dependence of the Improved Coefficient Function on the threshold variable is introduced, which can however be relegated to a small interval of this variable by means of the so-called Partition of Unity. We explicitly apply our improved scheme to the b -> X_s + gamma decay at next-to-leading logarithmic accuracy.


title: Connecting photometric and spectroscopic granulation signals with CHEOPS and ESPRESSO

abstract: Stellar granulation generates fluctuations in photometric and spectroscopic data whose properties depend on the stellar type, composition, and evolutionary state. In this study, we aim to detect the signatures of stellar granulation, link spectroscopic and photometric signatures of convection for main-sequence stars, and test predictions from 3D hydrodynamic models. For the first time, we observed two bright stars (Teff = 5833 K and 6205 K) with high-precision observations taken simultaneously with CHEOPS and ESPRESSO. We analyzed the properties of the stellar granulation signal in each individual data set. We compared them to Kepler observations and 3D hydrodynamic models. While isolating the granulation-induced changes by attenuating the p-mode oscillation signals, we studied the relationship between photometric and spectroscopic observables. The signature of stellar granulation is detected and precisely characterized for the hotter F star in the CHEOPS and ESPRESSO observations. For the cooler G star, we obtain a clear detection in the CHEOPS dataset only. The TESS observations are blind to this stellar signal. Based on CHEOPS observations, we show that the inferred properties of stellar granulation are in agreement with both Kepler observations and hydrodynamic models. Comparing their periodograms, we observe a strong link between spectroscopic and photometric observables. Correlations of this stellar signal in the time domain (flux vs RV) and with specific spectroscopic observables (shape of the cross-correlation functions) are however difficult to isolate due to signal-to-noise dependent variations. In the context of the upcoming PLATO mission and the extreme precision RV surveys, a thorough understanding of the properties of the stellar granulation signal is needed. The CHEOPS and ESPRESSO observations pave the way for detailed analyses of this stellar process.


title: $S$-unit equation in two variables and Pad\'{e} approximations

abstract: In this article, we use Pad\'{e} approximations constructed for binomial functions, to give a new upper bound for the number of the solutions of the $S$-unit equation. Combining explicit formulae of these Pad\'{e} approximants with a simple argument relying on Mahler measure and on the local height, we refine the bound due to J.-H. Evertse.


title: Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

abstract: Let $\Omega = [0,1]^d$ be the unit cube in $\mathbb{R}^d$. We study the problem of how efficiently, in terms of the number of parameters, deep neural networks with the ReLU activation function can approximate functions in the Sobolev spaces $W^s(L_q(\Omega))$ and Besov spaces $B^s_r(L_q(\Omega))$, with error measured in the $L_p(\Omega)$ norm. This problem is important when studying the application of neural networks in a variety of fields, including scientific computing and signal processing, and has previously been solved only when $p=q=\infty$. Our contribution is to provide a complete solution for all $1\leq p,q\leq \infty$ and $s > 0$ for which the corresponding Sobolev or Besov space compactly embeds into $L_p$. The key technical tool is a novel bit-extraction technique which gives an optimal encoding of sparse vectors. This enables us to obtain sharp upper bounds in the non-linear regime where $p > q$. We also provide a novel method for deriving $L_p$-approximation lower bounds based upon VC-dimension when $p < \infty$. Our results show that very deep ReLU networks significantly outperform classical methods of approximation in terms of the number of parameters, but that this comes at the cost of parameters which are not encodable.


title: Elements of effective machine learning datasets in astronomy

abstract: In this work, we identify elements of effective machine learning datasets in astronomy and present suggestions for their design and creation. Machine learning has become an increasingly important tool for analyzing and understanding the large-scale flood of data in astronomy. To take advantage of these tools, datasets are required for training and testing. However, building machine learning datasets for astronomy can be challenging. Astronomical data is collected from instruments built to explore science questions in a traditional fashion rather than to conduct machine learning. Thus, it is often the case that raw data, or even downstream processed data is not in a form amenable to machine learning. We explore the construction of machine learning datasets and we ask: what elements define effective machine learning datasets? We define effective machine learning datasets in astronomy to be formed with well-defined data points, structure, and metadata. We discuss why these elements are important for astronomical applications and ways to put them in practice. We posit that these qualities not only make the data suitable for machine learning, they also help to foster usable, reusable, and replicable science practices.


title: An Analysis of Social Biases Present in BERT Variants Across Multiple Languages

abstract: Although large pre-trained language models have achieved great success in many NLP tasks, it has been shown that they reflect human biases from their pre-training corpora. This bias may lead to undesirable outcomes when these models are applied in real-world settings. In this paper, we investigate the bias present in monolingual BERT models across a diverse set of languages (English, Greek, and Persian). While recent research has mostly focused on gender-related biases, we analyze religious and ethnic biases as well and propose a template-based method to measure any kind of bias, based on sentence pseudo-likelihood, that can handle morphologically complex languages with gender-based adjective declensions. We analyze each monolingual model via this method and visualize cultural similarities and differences across different dimensions of bias. Ultimately, we conclude that current methods of probing for bias are highly language-dependent, necessitating cultural insights regarding the unique ways bias is expressed in each language and culture (e.g. through coded language, synecdoche, and other similar linguistic concepts). We also hypothesize that higher measured social biases in the non-English BERT models correlate with user-generated content in their training.


title: Nonlinear Schwarz preconditioning for Quasi-Newton methods

abstract: We propose the nonlinear restricted additive Schwarz (RAS) preconditioning strategy to improve the convergence speed of limited memory quasi-Newton (QN) methods. We consider both "left-preconditioning" and "right-preconditioning" strategies. As the application of the nonlinear preconditioning changes the standard gradients and Hessians to their preconditioned counterparts, the standard secant pairs cannot be used to approximate the preconditioned Hessians. We discuss how to construct the secant pairs in the preconditioned QN framework. Finally, we demonstrate the robustness and efficiency of the preconditioned QN methods using numerical experiments.


title: Dynamical stability in a non-Hermitian kicked rotor model

abstract: We investigate the quantum irreversibility and quantum diffusion in a non-Hermitian kicked rotor model for which the kicking strength is complex. Our results show that the exponential decay of Loschmidt echo gradually disappears with increasing the strength of the imaginary part of non-Hermitian driven potential, demonstrating the suppress of the exponential instability by non-Hermiticity. The quantum diffusion exhibits the dynamical localization in momentum space, namely, the mean square of momentum increases to saturation with time evolution, which decreases with the increase of the strength of the imaginary part of the kicking. This clearly reveals the enhancement of dynamical localization by non-Hermiticity. We find, both analytically and numerically, that the quantum state are mainly populated on a very few quasieigenstates with significantly large value of the imaginary part of quasienergies. Interestingly, the average value of the inverse participation ratio of quasieigenstates decreases with the increase of the strength of the imaginary part of the kicking potential, which implies that the feature of quasieigenstates determines the stability of wavepacket's dynamics and the dynamical localization of energy diffusion.


title: Learning Branching Heuristics from Graph Neural Networks

abstract: Backtracking has been widely used for solving problems in artificial intelligence (AI), including constraint satisfaction problems and combinatorial optimization problems. Good branching heuristics can efficiently improve the performance of backtracking by helping prune the search space and leading the search to the most promising direction. In this paper, we first propose a new graph neural network (GNN) model designed using the probabilistic method. From the GNN model, we introduce an approach to learn a branching heuristic for combinatorial optimization problems. In particular, our GNN model learns appropriate probability distributions on vertices in given graphs from which the branching heuristic is extracted and used in a backtracking search. Our experimental results for the (minimum) dominating-clique problem show that this learned branching heuristic performs better than the minimum-remaining-values heuristic in terms of the number of branches of the whole search tree. Our approach introduces a new way of applying GNNs towards enhancing the classical backtracking algorithm used in AI.


title: Exploring Temporal Information Dynamics in Spiking Neural Networks

abstract: Most existing Spiking Neural Network (SNN) works state that SNNs may utilize temporal information dynamics of spikes. However, an explicit analysis of temporal information dynamics is still missing. In this paper, we ask several important questions for providing a fundamental understanding of SNNs: What are temporal information dynamics inside SNNs? How can we measure the temporal information dynamics? How do the temporal information dynamics affect the overall learning performance? To answer these questions, we estimate the Fisher Information of the weights to measure the distribution of temporal information during training in an empirical manner. Surprisingly, as training goes on, Fisher information starts to concentrate in the early timesteps. After training, we observe that information becomes highly concentrated in earlier few timesteps, a phenomenon we refer to as temporal information concentration. We observe that the temporal information concentration phenomenon is a common learning feature of SNNs by conducting extensive experiments on various configurations such as architecture, dataset, optimization strategy, time constant, and timesteps. Furthermore, to reveal how temporal information concentration affects the performance of SNNs, we design a loss function to change the trend of temporal information. We find that temporal information concentration is crucial to building a robust SNN but has little effect on classification accuracy. Finally, we propose an efficient iterative pruning method based on our observation on temporal information concentration. Code is available at https://github.com/Intelligent-Computing-Lab-Yale/Exploring-Temporal-Information-Dynamics-in-Spiking-Neural-Networks.


title: Faster Algorithm for Structured John Ellipsoid Computation

abstract: Computing John Ellipsoid is a fundamental problem in machine learning and convex optimization, where the goal is to compute the ellipsoid with maximal volume that lies in a given convex centrally symmetric polytope defined by a matrix $A \in \mathbb{R}^{n \times d}$. In this work, we show two faster algorithms for approximating the John Ellipsoid. $\bullet$ For sparse matrix $A$, we can achieve nearly input sparsity time $\mathrm{nnz}(A) + d^{\omega}$, where $\omega$ is exponent of matrix multiplication. Currently, $\omega \approx 2.373$. $\bullet$ For the matrix $A$ which has small treewidth $\tau$, we can achieve $n \tau^2$ time. Therefore, we significantly improves the state-of-the-art results on approximating the John Ellipsoid for centrally symmetric polytope [Cohen, Cousins, Lee, and Yang COLT 2019] which takes $nd^2$ time.


title: When is the complement of the diagonal of a LOTS functionally countable?

abstract: In a 2021 paper, Vladimir Tkachuk asked whether there is a non-separable LOTS $X$ such that $X^2\setminus\{\langle x,x\rangle\colon x\in X\}$ is functionally countable. In this paper we prove that such a space, if it exists, must be an Aronszajn line and admits a $\leq 2$-to-$1$ retraction to a subspace that is a Suslin line. After this, assuming the existence of a Suslin line, we prove that there is Suslin line that is functionally countable. Finally, we present an example of a functionally countable Suslin line $L$ such that $L^2\setminus\{\langle x,x\rangle\colon x\in L\}$ is not functionally countable.


title: Domain-Independent Dynamic Programming: Generic State Space Search for Combinatorial Optimization

abstract: For combinatorial optimization problems, model-based approaches such as mixed-integer programming (MIP) and constraint programming (CP) aim to decouple modeling and solving a problem: the 'holy grail' of declarative problem solving. We propose domain-independent dynamic programming (DIDP), a new model-based paradigm based on dynamic programming (DP). While DP is not new, it has typically been implemented as a problem-specific method. We propose Dynamic Programming Description Language (DyPDL), a formalism to define DP models, and develop Cost-Algebraic A* Solver for DyPDL (CAASDy), a generic solver for DyPDL using state space search. We formalize existing problem-specific DP and state space search methods for combinatorial optimization problems as DP models in DyPDL. Using CAASDy and commercial MIP and CP solvers, we experimentally compare the DP models with existing MIP and CP models, showing that, despite its nascent nature, CAASDy outperforms MIP and CP on a number of common problem classes.


title: Topological transitions, turbulent-like motion and long-time-tails driven by cell division in biological tissues

abstract: The complex spatiotemporal flow patterns in living tissues, driven by active forces, have many of the characteristics associated with inertial turbulence even though the Reynolds number is extremely low. Analyses of experimental data from two-dimensional epithelial monolayers in combination with agent-based simulations show that cell division and apoptosis lead to directed cell motion for hours, resulting in rapid topological transitions in neighboring cells. These transitions in turn generate both long ranged and long lived clockwise and anticlockwise vortices, which gives rise to turbulent-like flows. Both experiments and simulations show that at long wavelengths the wave vector ($k$) dependent energy spectrum $E(k) \approx k^{-5/3}$, coinciding with the Kolmogorov scaling in fully developed inertial turbulence. Using theoretical arguments and simulations, we show that long-lived vortices lead to long-time tails in the velocity auto-correlation function, $C_v(t) \sim t^{-1/2}$, which has the same structure as in classical 2D fluids but with a different scaling exponent.


title: c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization

abstract: Hyperparameter optimization (HPO) is crucial for strong performance of deep learning algorithms and real-world applications often impose some constraints, such as memory usage, or latency on top of the performance requirement. In this work, we propose constrained TPE (c-TPE), an extension of the widely-used versatile Bayesian optimization method, tree-structured Parzen estimator (TPE), to handle these constraints. Our proposed extension goes beyond a simple combination of an existing acquisition function and the original TPE, and instead includes modifications that address issues that cause poor performance. We thoroughly analyze these modifications both empirically and theoretically, providing insights into how they effectively overcome these challenges. In the experiments, we demonstrate that c-TPE exhibits the best average rank performance among existing methods with statistical significance on 81 expensive HPO with inequality constraints. Due to the lack of baselines, we only discuss the applicability of our method to hard-constrained optimization in Appendix D.


title: Charged black holes from a family of Born-Infeld-type electrodynamics models

abstract: We present a family of nonlinear electrodynamics models that are free of the infinite self-energy of the point charge. Each model is dependent on a dimensional nonlinearity parameter and is determined by the integer value of a dimensionless parameter n. The Born-Infeld model is recovered when n = 1. Some of the characteristics of this family are studied. In addition, we study the solutions of electrically charged AdS black holes, which result from coupling these electrodynamic models to General Relativity. A Smarr formula consistent with the first law of thermodynamics is also obtained in an extended phase space.


title: Engineering ultra-strong Mg-Li-Al-based light-weight alloys from first principles

abstract: Light-weight alloys are essential pillars of transportation technologies. They also play a crucial role to achieve a more green and cost-effective aerospace technologies. Magnesium-lithium-aluminum (Mg-Li-Al) alloys are auspicious candidates due to their promising mechanical strengths at low densities. We herein present a systematic first-principles investigation of the Mg-Li-Al-based alloys to provide insights for designing ultra-strong light-weight alloys. Initial analysis indicates that the Mg-Li-Al mixtures are not thermally stabilized into random-solid solutions. Following this hint, the base-centered cubic (BCC)-based intermetallics of Mg, Li and Al are investigated for their thermal and elastic stabilities.Three simple figures of merits are used to further assess their mechanical strengths. The most-frequently observed intermetallics are used to predict the yield strength of the hetero-structures from the recent experimental works. The rule of mixing works reasonable well to predict the mechanical properties of complex structures starting from isolated intermetallics.


title: Isoclinism of skew braces

abstract: We define isoclinism of skew braces and present several applications. We study some properties of skew braces that are invariant under isoclinism. For example, we prove that right nilpotency is an isoclinism invariant. This result has application in the theory of set-theoretic solutions to the Yang-Baxter equation. We define isoclinic solutions and study multipermutation solutions under isoclinism.


title: Triggered Ion-acoustic Waves

abstract: The Parker Solar Probe is in a solar orbit with a perihelion for orbit 12 at 13.3 solar radii. The electric field experiment on this satellite observes what we call triggered ion-acoustic waves as the most dominant wave mode above a few Hz within the solar radial distance of 15-25 solar radii. In this mode, a few Hz electrostatic wave is typically accompanied by bursts of a few hundred Hz wave whose bursts are phase locked with each low frequency wave period. Plasma density fluctuations with {\Delta}n/n~0.1 accompany these waves and they have no magnetic field component. The wave durations can be hours and their field and density fluctuations are nearly pure sine waves. They are identified as ion-acoustic waves. The low and high frequency waves are measured to have the same phase velocity within experimental uncertainties, which is a requirement associated with their phase locked relationship. From the measured wavelength, the potential associated with the low frequency wave is estimated to be ~10 Volts, which can result in electron heating via the Landau resonance that is in agreement with observations of the core electron temperature increases at times of such waves. Their phase locked relationship and pure frequency are surprising features that characterize a new regime of instability and evolution of ion-acoustic waves that may not have been reported previously. That these waves are an instrumental effect unrelated to natural processes is considered. While this is unlikely, the possibility that these waves are artificial cannot be rule out


title: Realistic Spin Model for Multiferroic NiI$_2$

abstract: A realistic first-principle-based spin Hamiltonian is constructed for the type-II multiferroic NiI$_2$, using a symmetry-adapted cluster expansion method. Besides single ion anisotropy and isotropic Heisenberg terms, this model further includes the Kitaev interaction and a biquadratic term, and can well reproduce striking features of the experimental helical ground state, that are, {\it e.g.}, a proper screw state, canting of rotation plane, propagation direction and period. Using this model to build a phase diagram, it is demonstrated that, (i) the in-plane propagation direction of $\langle1\bar10\rangle$ is determined by the Kitaev interaction, instead of the long-believed exchange frustrations; and (ii) the canting of rotation plane is also dominantly determined by Kitaev interaction, rather than interlayer couplings. Furthermore, additional Monte Carlo simulations reveal three equivalent domains and different topological defects. Since the ferroelectricity is induced by spins in type-II multiferroics, our work also implies that Kitaev interaction is closely related to the multiferroicity of NiI$_2$.


title: EasyMLServe: Easy Deployment of REST Machine Learning Services

abstract: Various research domains use machine learning approaches because they can solve complex tasks by learning from data. Deploying machine learning models, however, is not trivial and developers have to implement complete solutions which are often installed locally and include Graphical User Interfaces (GUIs). Distributing software to various users on-site has several problems. Therefore, we propose a concept to deploy software in the cloud. There are several frameworks available based on Representational State Transfer (REST) which can be used to implement cloud-based machine learning services. However, machine learning services for scientific users have special requirements that state-of-the-art REST frameworks do not cover completely. We contribute an EasyMLServe software framework to deploy machine learning services in the cloud using REST interfaces and generic local or web-based GUIs. Furthermore, we apply our framework on two real-world applications, \ie, energy time-series forecasting and cell instance segmentation. The EasyMLServe framework and the use cases are available on GitHub.


title: Picard group of moduli of parabolic Higgs bundles

abstract: Let $X$ be a compact Riemann surface of genus $g\geq 3$ and let $\mathcal{M}_{\mathrm{par}}$ be the moduli space of semistable parabolic bundles over $X$. Let $\mathcal{M}_{\mathrm{parH}}$ denote the moduli space of semistable parabolic Higgs bundles over $X$. In this article, we study the Picard group of $\mathcal{M}_{\mathrm{par}}$ and $\mathcal{M}_{\mathrm{parH}}$.


title: Panoramic Video Salient Object Detection with Ambisonic Audio Guidance

abstract: Video salient object detection (VSOD), as a fundamental computer vision problem, has been extensively discussed in the last decade. However, all existing works focus on addressing the VSOD problem in 2D scenarios. With the rapid development of VR devices, panoramic videos have been a promising alternative to 2D videos to provide immersive feelings of the real world. In this paper, we aim to tackle the video salient object detection problem for panoramic videos, with their corresponding ambisonic audios. A multimodal fusion module equipped with two pseudo-siamese audio-visual context fusion (ACF) blocks is proposed to effectively conduct audio-visual interaction. The ACF block equipped with spherical positional encoding enables the fusion in the 3D context to capture the spatial correspondence between pixels and sound sources from the equirectangular frames and ambisonic audios. Experimental results verify the effectiveness of our proposed components and demonstrate that our method achieves state-of-the-art performance on the ASOD60K dataset.


title: Photo Rater: Photographs Auto-Selector with Deep Learning

abstract: Photo Rater is a computer vision project that uses neural networks to help photographers select the best photo among those that are taken based on the same scene. This process is usually referred to as "culling" in photography, and it can be tedious and time-consuming if done manually. Photo Rater utilizes three separate neural networks to complete such a task: one for general image quality assessment, one for classifying whether the photo is blurry (either due to unsteady hands or out-of-focusness), and one for assessing general aesthetics (including the composition of the photo, among others). After feeding the image through each neural network, Photo Rater outputs a final score for each image, ranking them based on this score and presenting it to the user.


title: Introducci\'on a la teor\'ia de categor\'ias, l\'ogica y topos elementales para una mente curiosa

abstract: This paper presents a study of how the theory of categories leads to the creation of non classical logical systems. In particular, the case of the elementary topos of graphs, where there are three other truth values different from false and true. The approach in this article to the theory of categories avoids specialized mathematical training to understand it, since it seeks to make accessible the main ideas of this branch of mathematics to other disciplines of knowledge. This work was presented by the second author, under the title of "Logic and Categories", at the I Colloquium on Logic, Epistemology and Methodology organized by the School of Philosophy of the University sity of Costa Rica.


title: Quantitative Method for Security Situation of the Power Information Network Based on the Evolutionary Neural Network

abstract: Cybersecurity is the security cornerstone of digital transformation of the power grid and construction of new power systems. The traditional network security situation quantification method only analyzes from the perspective of network performance, ignoring the impact of various power application services on the security situation, so the quantification results cannot fully reflect the power information network risk state. This study proposes a method for quantifying security situation of the power information network based on the evolutionary neural network. First, the security posture system architecture is designed by analyzing the business characteristics of power information network applications. Second, combining the importance of power application business, the spatial element index system of coupled interconnection is established from three dimensions of network reliability, threat, and vulnerability. Then, the BP neural network optimized by the genetic evolutionary algorithm is incorporated into the element index calculation process, and the quantitative model of security posture of the power information network based on the evolutionary neural network is constructed. Finally, a simulation experiment environment is built according to a power sector network topology, and the effectiveness and robustness of the method proposed in the study are verified.


title: Competing multiferroic phases in monolayer and few-layer NiI$_{2}$

abstract: A recent experiment reported type-II multiferroicity in monolayer (ML) NiI$_{2}$ based on a presumed spiral magnetic configuration (Spiral-B), which is, as we found here, under debate in the ML limit. Freestanding ML NiI$_{2}$ breaks its C$_{3}$ symmetry, as it prefers a striped antiferromagnetic order (AABB-AFM) along with an intralayer antiferroelectric (AFE) order. However, substrate confinement may preserve the C$_{3}$ symmetry and/or apply tensile strain to the ML. This leads to another spiral magnetic order (Spiral-$IV^X$), while 2L shows a different order (Spiral-$V^Y$) and Spiral-B dominates in thicker layers. Thus, three multiferroic phases, namely, Spiral-B+FE, Spiral-$IV^X$ +FE, Spiral-$V^Y$+FE, and an anti-multiferroic AABB-AFM+AFE one, show layer-thickness-dependent and geometry-dependent dominance, ascribed to competitions among thickness-dependent Kitaev, biquadratic, and Heisenberg spin-exchange interactions and single-ion magnetic anisotropy. Our theoretical results clarify the debate on the multiferroicity of ML NiI$_{2}$ and shed light on the role of layer-stacking-induced changes in noncollinear spin-exchange interactions and magnetic anisotropy in thickness-dependent magnetism.


title: Supervised Contrastive Prototype Learning: Augmentation Free Robust Neural Network

abstract: Transformations in the input space of Deep Neural Networks (DNN) lead to unintended changes in the feature space. Almost perceptually identical inputs, such as adversarial examples, can have significantly distant feature representations. On the contrary, Out-of-Distribution (OOD) samples can have highly similar feature representations to training set samples. Our theoretical analysis for DNNs trained with a categorical classification head suggests that the inflexible logit space restricted by the classification problem size is one of the root causes for the lack of $\textit{robustness}$. Our second observation is that DNNs over-fit to the training augmentation technique and do not learn $\textit{nuance invariant}$ representations. Inspired by the recent success of prototypical and contrastive learning frameworks for both improving robustness and learning nuance invariant representations, we propose a training framework, $\textbf{Supervised Contrastive Prototype Learning}$ (SCPL). We use N-pair contrastive loss with prototypes of the same and opposite classes and replace a categorical classification head with a $\textbf{Prototype Classification Head}$ (PCH). Our approach is $\textit{sample efficient}$, does not require $\textit{sample mining}$, can be implemented on any existing DNN without modification to their architecture, and combined with other training augmentation techniques. We empirically evaluate the $\textbf{clean}$ robustness of our method on out-of-distribution and adversarial samples. Our framework outperforms other state-of-the-art contrastive and prototype learning approaches in $\textit{robustness}$.


title: PatchGT: Transformer over Non-trainable Clusters for Learning Graph Representations

abstract: Recently the Transformer structure has shown good performances in graph learning tasks. However, these Transformer models directly work on graph nodes and may have difficulties learning high-level information. Inspired by the vision transformer, which applies to image patches, we propose a new Transformer-based graph neural network: Patch Graph Transformer (PatchGT). Unlike previous transformer-based models for learning graph representations, PatchGT learns from non-trainable graph patches, not from nodes directly. It can help save computation and improve the model performance. The key idea is to segment a graph into patches based on spectral clustering without any trainable parameters, with which the model can first use GNN layers to learn patch-level representations and then use Transformer to obtain graph-level representations. The architecture leverages the spectral information of graphs and combines the strengths of GNNs and Transformers. Further, we show the limitations of previous hierarchical trainable clusters theoretically and empirically. We also prove the proposed non-trainable spectral clustering method is permutation invariant and can help address the information bottlenecks in the graph. PatchGT achieves higher expressiveness than 1-WL-type GNNs, and the empirical study shows that PatchGT achieves competitive performances on benchmark datasets and provides interpretability to its predictions. The implementation of our algorithm is released at our Github repo: https://github.com/tufts-ml/PatchGT.


title: A Critical Review of Traffic Signal Control and A Novel Unified View of Reinforcement Learning and Model Predictive Control Approaches for Adaptive Traffic Signal Control

abstract: Recent years have witnessed substantial growth in adaptive traffic signal control (ATSC) methodologies that improve transportation network efficiency, especially in branches leveraging artificial intelligence based optimization and control algorithms such as reinforcement learning as well as conventional model predictive control. However, lack of cross-domain analysis and comparison of the effectiveness of applied methods in ATSC research limits our understanding of existing challenges and research directions. This chapter proposes a novel unified view of modern ATSCs to identify common ground as well as differences and shortcomings of existing methodologies with the ultimate goal to facilitate cross-fertilization and advance the state-of-the-art. The unified view applies the mathematical language of the Markov decision process, describes the process of controller design from both the world (problem) and solution modeling perspectives. The unified view also analyses systematic issues commonly ignored in existing studies and suggests future potential directions to resolve these issues.


title: Unit-log-symmetric models: Characterization, statistical properties and its use in analyzing internet access data

abstract: We present here a unit-log-symmetric model based on the bivariate log-symmetric distribution. It is a flexible family of distributions over the interval $(0, 1)$. We then discuss its mathematical properties such as stochastic representation, symmetry, modality, moments, quantile function, entropy and maximum likelihood estimators, paying particular attention to the special cases of unit log-normal, unit-log-Student-$t$ and unit-log-Laplace distributions. Finally, some empirical results and practical illustrations are presented.


title: Utility Assessment of Synthetic Data Generation Methods

abstract: Big data analysis poses the dual problem of privacy preservation and utility, i.e., how accurate data analyses remain after transforming original data in order to protect the privacy of the individuals that the data is about - and whether they are accurate enough to be meaningful. In this paper, we thus investigate across several datasets whether different methods of generating fully synthetic data vary in their utility a priori (when the specific analyses to be performed on the data are not known yet), how closely their results conform to analyses on original data a posteriori, and whether these two effects are correlated. We find some methods (decision-tree based) to perform better than others across the board, sizeable effects of some choices of imputation parameters (notably the number of released datasets), no correlation between broad utility metrics and analysis accuracy, and varying correlations for narrow metrics. We did get promising findings for classification tasks when using synthetic data for training machine learning models, which we consider worth exploring further also in terms of mitigating privacy attacks against ML models such as membership inference and model inversion.


title: Supervised Pretraining for Molecular Force Fields and Properties Prediction

abstract: Machine learning approaches have become popular for molecular modeling tasks, including molecular force fields and properties prediction. Traditional supervised learning methods suffer from scarcity of labeled data for particular tasks, motivating the use of large-scale dataset for other relevant tasks. We propose to pretrain neural networks on a dataset of 86 millions of molecules with atom charges and 3D geometries as inputs and molecular energies as labels. Experiments show that, compared to training from scratch, fine-tuning the pretrained model can significantly improve the performance for seven molecular property prediction tasks and two force field tasks. We also demonstrate that the learned representations from the pretrained model contain adequate information about molecular structures, by showing that linear probing of the representations can predict many molecular information including atom types, interatomic distances, class of molecular scaffolds, and existence of molecular fragments. Our results show that supervised pretraining is a promising research direction in molecular modeling


title: Effectiveness of an Online Course in Programming in a State University in the Philippines

abstract: Online courses, as a pedagogical approach to teaching, boomed during this Coronavirus Disease 2019 pandemic era. Universities shifted from traditional face to face classes to online distance learning due to the cause of the pandemic. This study aimed to determine how effective an online course is in learning a programming course. The study utilized mixed method research applied through a validated survey questionnaire consisting of closed and open ended questions. Python programming was the course selected to undergo the study and underwent an evaluation to determine the students' responses. Student respondents are from Bulacan State University, a state university in the Philippines, under the Bachelor of Science in Information Technology program. Based on their responses, the students found that the online Python programming was Very Effective, with an overall mean of 4.49. This result shows that students found the online course effective, provided the proper course design and content, allowed them to spend enough time finishing tasks, and provided communication and interaction with their instructor and fellow students. Additionally, students gave overwhelmingly positive responses when asked what their instructors had done well on the course delivery and provided insightful and constructive comments for further enhancement and delivery of the course. This study found that most students strongly agreed and believed in the effectiveness of delivering the Python Programming course asynchronously. With such positive results from the student's perspective and evaluation, the course can be enhanced to continue providing quality education at Bulacan State University.


title: Efficient and Accurate Calibration to FX Market Skew with Fully Parameterized Local Volatility Model

abstract: When trading American and Asian options in the FX derivatives market, banks must calculate prices using a complex mathematical model. It is often observed that different models produce varying prices for the same exotic option, which violates the non-arbitrage requirement of derivative risk management. To address this issue, we have studied a fully parameterized local volatility model for pricing American/Asian options. This model, when implemented using a grid or Monte-Carlo numerical method, can be efficiently and accurately calibrated to FX market skew volatilities. As a result, the model can provide reliable prices for exotic options during daily trading activities.


title: A1 SLAM: Quadruped SLAM using the A1's Onboard Sensors

abstract: Quadrupeds are robots that have been of interest in the past few years due to their versatility in navigating across various terrain and utility in several applications. For quadrupeds to navigate without a predefined map a priori, they must rely on SLAM approaches to localize and build the map of the environment. Despite the surge of interest and research development in SLAM and quadrupeds, there still has yet to be an open-source package that capitalizes on the onboard sensors of an affordable quadruped. This motivates the A1 SLAM package, which is an open-source ROS package that provides the Unitree A1 quadruped with real-time, high performing SLAM capabilities using the default sensors shipped with the robot. A1 SLAM solves the PoseSLAM problem using the factor graph paradigm to optimize for the poses throughout the trajectory. A major design feature of the algorithm is using a sliding window of fully connected LiDAR odometry factors. A1 SLAM has been benchmarked against Google's Cartographer and has showed superior performance especially with trajectories experiencing aggressive motion.


title: Electromagnetic draping of merging neutron stars

abstract: We first derive a set of equations describing general stationary configurations of relativistic force-free plasma, without assuming any geometric symmetries. We then demonstrate that electromagnetic interaction of merging neutron stars is necessarily dissipative due to the effect of electromagnetic draping - creation of dissipative regions near the star (in the single-magnetized case) or at the magnetospheric boundary (in the double-magnetized case). Our results indicate that even in the single magnetized case we expect that relativistic jets (or ``tongues'') are produced, with correspondingly beamed emission pattern.


title: Multi-Step Short-Term Wind Speed Prediction with Rank Pooling and Fast Fourier Transformation

abstract: Short-term wind speed prediction is essential for economical wind power utilization. The real-world wind speed data is typically intermittent and fluctuating, presenting great challenges to existing shallow models. In this paper, we present a novel deep hybrid model for multi-step wind speed prediction, namely LR-FFT-RP-MLP/LSTM (Linear Fast Fourier Transformation Rank Pooling Multiple-Layer Perception/Long Short-Term Memory). Our hybrid model processes the local and global input features simultaneously. We leverage Rank Pooling (RP) for the local feature extraction to capture the temporal structure while maintaining the temporal order. Besides, to understand the wind periodic patterns, we exploit Fast Fourier Transformation (FFT) to extract global features and relevant frequency components in the wind speed data. The resulting local and global features are respectively integrated with the original data and are fed into an MLP/LSTM layer for the initial wind speed predictions. Finally, we leverage a linear regression layer to collaborate these initial predictions to produce the final wind speed prediction. The proposed hybrid model is evaluated using real wind speed data collected from 2010 to 2020, demonstrating superior forecasting capabilities when compared to state-of-the-art single and hybrid models. Overall, this study presents a promising approach for improving the accuracy of wind speed forecasting.


title: Experimental realization of tunable Poisson's ratio in deployable origami metamaterials

abstract: Origami metamaterials are known to display highly tunable Poisson's ratio values depending on their folded state. Most studies on the Poisson effects in deployable origami tessellations are restricted to theory and simulation. Experimental realization of the desired Poisson effects in origami metamaterials requires special attention to the boundary conditions to enable deployable nonlinear deformations that give rise to tunability. In this work, we present a novel experimental setup suitable to study the Poisson effects in 2D origami tessellations that undergo simultaneous deformations in both the applied and transverse directions. The setup comprises a gripping mechanism, which we call a Saint-Venant fixture, to eliminate Saint-Venant end effects during uniaxial testing experiment. Using this setup, we conduct Poisson's ratio measurements of the Morph origami pattern whose configuration space combines features of the Miura-ori and Eggbox parent patterns. We experimentally observe the Poisson's ratio sign switching capability of the Morph pattern, along with its ability to display either completely positive or completely negative values of Poisson's ratio by virtue of topological transformations. To demonstrate the versatility of the novel setup we also perform experiments on the standard Miura-ori and the standard Eggbox patterns. Our results demonstrate the agreement between the theory, the simulations, and the experiments on the Poisson's ratio measurement and its tunability in origami metamaterials. The proposed experimental technique can be adopted for investigating other tunable properties of origami metamaterials in static and in dynamic regimes, such as finite-strain Poisson's ratios, elastic thermal expansion, and wave propagation control.


title: Towards Optimal Control of Amyloid Fibrillation

abstract: EGCG, as a representative of amyloid inhibitors, has shown a promising ability against Abeta fibrillation by directly degradating the mature fibrils. Most previous studies have been focusing on its functional mechanisms, meanwhile its optimal dosage has been seldom considered. To solve this critical issue, we refer to the generalized Logistics model for amyloid fibrillation and inhibition and adopt the optimal control theory to balance the effectiveness and cost (or toxicity) of inhibitors. The optimal control trajectory of inhibitors is analytically solved, based on which the influence of model parameters, the difference between the optimal control strategy and several traditional drug dosing strategies are systematically compared and validated through experiments. Our results will shed light on the rational usage of amyloid inhibitors in clinic.


title: Unsupervised User-Based Insider Threat Detection Using Bayesian Gaussian Mixture Models

abstract: Insider threats are a growing concern for organizations due to the amount of damage that their members can inflict by combining their privileged access and domain knowledge. Nonetheless, the detection of such threats is challenging, precisely because of the ability of the authorized personnel to easily conduct malicious actions and because of the immense size and diversity of audit data produced by organizations in which the few malicious footprints are hidden. In this paper, we propose an unsupervised insider threat detection system based on audit data using Bayesian Gaussian Mixture Models. The proposed approach leverages a user-based model to optimize specific behaviors modelization and an automatic feature extraction system based on Word2Vec for ease of use in a real-life scenario. The solution distinguishes itself by not requiring data balancing nor to be trained only on normal instances, and by its little domain knowledge required to implement. Still, results indicate that the proposed method competes with state-of-the-art approaches, presenting a good recall of 88\%, accuracy and true negative rate of 93%, and a false positive rate of 6.9%. For our experiments, we used the benchmark dataset CERT version 4.2.


title: BERN-NN: Tight Bound Propagation For Neural Networks Using Bernstein Polynomial Interval Arithmetic

abstract: In this paper, we present BERN-NN as an efficient tool to perform bound propagation of Neural Networks (NNs). Bound propagation is a critical step in wide range of NN model checkers and reachability analysis tools. Given a bounded input set, bound propagation algorithms aim to compute tight bounds on the output of the NN. So far, linear and convex optimizations have been used to perform bound propagation. Since neural networks are highly non-convex, state-of-the-art bound propagation techniques suffer from introducing large errors. To circumvent such drawback, BERN-NN approximates the bounds of each neuron using a class of polynomials called Bernstein polynomials. Bernstein polynomials enjoy several interesting properties that allow BERN-NN to obtain tighter bounds compared to those relying on linear and convex approximations. BERN-NN is efficiently parallelized on graphic processing units (GPUs). Extensive numerical results show that bounds obtained by BERN-NN are orders of magnitude tighter than those obtained by state-of-the-art verifiers such as linear programming and linear interval arithmetic. Moreoveer, BERN-NN is both faster and produces tighter outputs compared to convex programming approaches like alpha-CROWN.


title: Incentive-boosted Federated Crowdsourcing

abstract: Crowdsourcing is a favorable computing paradigm for processing computer-hard tasks by harnessing human intelligence. However, generic crowdsourcing systems may lead to privacy-leakage through the sharing of worker data. To tackle this problem, we propose a novel approach, called iFedCrowd (incentive-boosted Federated Crowdsourcing), to manage the privacy and quality of crowdsourcing projects. iFedCrowd allows participants to locally process sensitive data and only upload encrypted training models, and then aggregates the model parameters to build a shared server model to protect data privacy. To motivate workers to build a high-quality global model in an efficacy way, we introduce an incentive mechanism that encourages workers to constantly collect fresh data to train accurate client models and boosts the global model training. We model the incentive-based interaction between the crowdsourcing platform and participating workers as a Stackelberg game, in which each side maximizes its own profit. We derive the Nash Equilibrium of the game to find the optimal solutions for the two sides. Experimental results confirm that iFedCrowd can complete secure crowdsourcing projects with high quality and efficiency.


title: Don't Watch Me: A Spatio-Temporal Trojan Attack on Deep-Reinforcement-Learning-Augment Autonomous Driving

abstract: Deep reinforcement learning (DRL) is one of the most popular algorithms to realize an autonomous driving (AD) system. The key success factor of DRL is that it embraces the perception capability of deep neural networks which, however, have been proven vulnerable to Trojan attacks. Trojan attacks have been widely explored in supervised learning (SL) tasks (e.g., image classification), but rarely in sequential decision-making tasks solved by DRL. Hence, in this paper, we explore Trojan attacks on DRL for AD tasks. First, we propose a spatio-temporal DRL algorithm based on the recurrent neural network and attention mechanism to prove that capturing spatio-temporal traffic features is the key factor to the effectiveness and safety of a DRL-augment AD system. We then design a spatial-temporal Trojan attack on DRL policies, where the trigger is hidden in a sequence of spatial and temporal traffic features, rather than a single instant state used in existing Trojan on SL and DRL tasks. With our Trojan, the adversary acts as a surrounding normal vehicle and can trigger attacks via specific spatial-temporal driving behaviors, rather than physical or wireless access. Through extensive experiments, we show that while capturing spatio-temporal traffic features can improve the performance of DRL for different AD tasks, they suffer from Trojan attacks since our designed Trojan shows high stealthy (various spatio-temporal trigger patterns), effective (less than 3.1\% performance variance rate and more than 98.5\% attack success rate), and sustainable to existing advanced defenses.


title: An alternative derivation of orbital-free density functional theory

abstract: Polymer self-consistent field theory techniques are used to derive quantum density functional theory without the use of the theorems of density functional theory. Instead, a free energy is obtained from a partition function that is constructed directly from a Hamiltonian, so that the results are, in principle, valid at finite temperatures. The main governing equations are found to be a set of modified diffusion equations, and the set of self-consistent equations are essentially identical to those of a ring polymer system. The equations are shown to be equivalent to Kohn-Sham density functional theory, and to reduce to classical density functional theory, each under appropriate conditions. The obtained non-interacting kinetic energy functional is, in principle, exact, but suffers from the usual orbital-free approximation of the Pauli exclusion principle in additional to the exchange-correlation approximation. The equations are solved using the spectral method of polymer self-consistent field theory, which allows the set of modified diffusion equations to be evaluated for the same computational cost as solving a single diffusion equation. A simple exchange-correlation functional is chosen, together with a shell-structure-based Pauli potential, in order to compare the ensemble average electron densities of several isolated atom systems to known literature results. The agreement is excellent, justifying the alternative formalism and numerical method. Some speculation is provided on considering the time-like parameter in the diffusion equations, which is related to temperature, as having dimensional significance, and thus picturing point-like quantum particles instead as non-local, polymer-like, threads in a higher dimensional thermal-space. A consideration of the double-slit experiment from this point of view is speculated to provide results equivalent to the Copenhagen interpretation.


title: Contract-Backed Digital Cash

abstract: We characterize digital cash as the digital equivalent of physical cash: secure, fungible, decentralized, directly controlled, privacy-preserving; but enhanced with qualitatively new functionality. It is extremely efficiently transferable and, most importantly, transactional or, more generally, contract-backed. This facilitates fully automated, guaranteed transactional execution of atomic resource exchanges and more complex contracts, without a multitude of intermediaries and expensive or slow semi-manual processes. A didactic objective is separating money characteristics from technology aspects such as specific blockchain and distributed ledger systems to help disentangle discussions of digital money design from implementation techniques. We finally discuss the power and role of programmable (contract-backed) digital money in case studies: tokenization of invoice debt using smart contracts on Ethereum, with stablecoins serving as digital money; smart contracts for disbursing payments transparently and reliably in accordance with social legislation; and a Danish e-krone for crowdfunding public and private community projets. These contributions are made in independent chapters by participants of the Working Group on Digital Cash at Copenhagen FinTech in 2018 and 2019, which have not been published before. Collectively, the contributions illustrate the design space and potential of digital money when powered by smart digital contracts that effectively eliminate both counterparty risk (somebody does not pay or does not deliver) and settlement risk (a trade fails and needs to be aborted) orders of magnitude faster than in current financial practice.


title: Siamese based Neural Network for Offline Writer Identification on word level data

abstract: Handwriting recognition is one of the desirable attributes of document comprehension and analysis. It is concerned with the documents writing style and characteristics that distinguish the authors. The diversity of text images, notably in images with varying handwriting, makes the process of learning good features difficult in cases where little data is available. In this paper, we propose a novel scheme to identify the author of a document based on the input word image. Our method is text independent and does not impose any constraint on the size of the input image under examination. To begin with, we detect crucial components in handwriting and extract regions surrounding them using Scale Invariant Feature Transform (SIFT). These patches are designed to capture individual writing features (including allographs, characters, or combinations of characters) that are likely to be unique for an individual writer. These features are then passed through a deep Convolutional Neural Network (CNN) in which the weights are learned by applying the concept of Similarity learning using Siamese network. Siamese network enhances the discrimination power of CNN by mapping similarity between different pairs of input image. Features learned at different scales of the extracted SIFT key-points are encoded using Sparse PCA, each components of the Sparse PCA is assigned a saliency score signifying its level of significance in discriminating different writers effectively. Finally, the weighted Sparse PCA corresponding to each SIFT key-points is combined to arrive at a final classification score for each writer. The proposed algorithm was evaluated on two publicly available databases (namely IAM and CVL) and is able to achieve promising result, when compared with other deep learning based algorithm.


title: MiftyCoin (MFT): A Cryptocurrency Mined with Proof of Human Work

abstract: We present in this paper a cryptocurrency called Mobile Fungible Token (MFT) or "MiftyCoin", which is mined with Proof of Human Work (PoH). Blocks in MFT's blockchain are mined by users solving unique 24-tile puzzles autogenerated as a function of block hash values. Each tile in the puzzle is a 4-sided domino-like square, where the number of dots per square is a function of a subset of the bits of a corresponding byte in the block's hash value. The objective is to find a set of tile moves that end up in an arrangement with an optimal score; where each matching tile side increases the score by one. The block with the highest score gets a reward. More information about MiftyCoin is available at: https://www.miftycoin.com.


title: LAPTNet: LiDAR-Aided Perspective Transform Network

abstract: Semantic grids are a useful representation of the environment around a robot. They can be used in autonomous vehicles to concisely represent the scene around the car, capturing vital information for downstream tasks like navigation or collision assessment. Information from different sensors can be used to generate these grids. Some methods rely only on RGB images, whereas others choose to incorporate information from other sensors, such as radar or LiDAR. In this paper, we present an architecture that fuses LiDAR and camera information to generate semantic grids. By using the 3D information from a LiDAR point cloud, the LiDAR-Aided Perspective Transform Network (LAPTNet) is able to associate features in the camera plane to the bird's eye view without having to predict any depth information about the scene. Compared to state-of-theart camera-only methods, LAPTNet achieves an improvement of up to 8.8 points (or 38.13%) over state-of-art competing approaches for the classes proposed in the NuScenes dataset validation split.


title: Sign Language to Text Conversion in Real Time using Transfer Learning

abstract: The people in the world who are hearing impaired face many obstacles in communication and require an interpreter to comprehend what a person is saying. There has been constant scientific research and the existing models lack the ability to make accurate predictions. So we propose a deep learning model trained on ASL i.e. American Sign Language which will take actions in the form of ASL as input and translate it into text. To achieve the translation a Convolution Neural Network model and a transfer learning model based on the VGG16 architecture are used. There has been an improvement in accuracy from 94% of CNN to 98.7% of Transfer Learning, an improvement of 5%. An application with the deep learning model integrated has also been built.


title: Sentence-Level Sign Language Recognition Framework

abstract: We present two solutions to sentence-level SLR. Sentence-level SLR required mapping videos of sign language sentences to sequences of gloss labels. Connectionist Temporal Classification (CTC) has been used as the classifier level of both models. CTC is used to avoid pre-segmenting the sentences into individual words. The first model is an LRCN-based model, and the second model is a Multi-Cue Network. LRCN is a model in which a CNN as a feature extractor is applied to each frame before feeding them into an LSTM. In the first approach, no prior knowledge has been leveraged. Raw frames are fed into an 18-layer LRCN with a CTC on top. In the second approach, three main characteristics (hand shape, hand position, and hand movement information) associated with each sign have been extracted using Mediapipe. 2D landmarks of hand shape have been used to create the skeleton of the hands and then are fed to a CONV-LSTM model. Hand locations and hand positions as relative distance to head are fed to separate LSTMs. All three sources of information have been then integrated into a Multi-Cue network with a CTC classification layer. We evaluated the performance of proposed models on RWTH-PHOENIX-Weather. After performing an excessive search on model hyper-parameters such as the number of feature maps, input size, batch size, sequence length, LSTM memory cell, regularization, and dropout, we were able to achieve 35 Word Error Rate (WER).


title: How to Backpropagate through Hungarian in Your DETR?

abstract: The DEtection TRansformer (DETR) approach, which uses a transformer encoder-decoder architecture and a set-based global loss, has become a building block in many transformer based applications. However, as originally presented, the assignment cost and the global loss are not aligned, i.e., reducing the former is likely but not guaranteed to reduce the latter. And the issue of gradient is ignored when a combinatorial solver such as Hungarian is used. In this paper we show that the global loss can be expressed as the sum of an assignment-independent term, and an assignment-dependent term which can be used to define the assignment cost matrix. Recent results on generalized gradients of optimal assignment cost with respect to parameters of an assignment problem are then used to define generalized gradients of the loss with respect to network parameters, and backpropagation is carried out properly. Our experiments using the same loss weights show interesting convergence properties and a potential for further performance improvements.


title: PatchBlender: A Motion Prior for Video Transformers

abstract: Transformers have become one of the dominant architectures in the field of computer vision. However, there are yet several challenges when applying such architectures to video data. Most notably, these models struggle to model the temporal patterns of video data effectively. Directly targeting this issue, we introduce PatchBlender, a learnable blending function that operates over patch embeddings across the temporal dimension of the latent space. We show that our method is successful at enabling vision transformers to encode the temporal component of video data. On Something-Something v2 and MOVi-A, we show that our method improves the baseline performance of video Transformers. PatchBlender has the advantage of being compatible with almost any Transformer architecture and since it is learnable, the model can adaptively turn on or off the prior. It is also extremely lightweight compute-wise, 0.005% the GFLOPs of a ViT-B.


title: Text-Aware Dual Routing Network for Visual Question Answering

abstract: Visual question answering (VQA) is a challenging task to provide an accurate natural language answer given an image and a natural language question about the image. It involves multi-modal learning, i.e., computer vision (CV) and natural language processing (NLP), as well as flexible answer prediction for free-form and open-ended answers. Existing approaches often fail in cases that require reading and understanding text in images to answer questions. In practice, they cannot effectively handle the answer sequence derived from text tokens because the visual features are not text-oriented. To address the above issues, we propose a Text-Aware Dual Routing Network (TDR) which simultaneously handles the VQA cases with and without understanding text information in the input images. Specifically, we build a two-branch answer prediction network that contains a specific branch for each case and further develop a dual routing scheme to dynamically determine which branch should be chosen. In the branch that involves text understanding, we incorporate the Optical Character Recognition (OCR) features into the model to help understand the text in the images. Extensive experiments on the VQA v2.0 dataset demonstrate that our proposed TDR outperforms existing methods, especially on the ''number'' related VQA questions.


title: GLAMI-1M: A Multilingual Image-Text Fashion Dataset

abstract: We introduce GLAMI-1M: the largest multilingual image-text classification dataset and benchmark. The dataset contains images of fashion products with item descriptions, each in 1 of 13 languages. Categorization into 191 classes has high-quality annotations: all 100k images in the test set and 75% of the 1M training set were human-labeled. The paper presents baselines for image-text classification showing that the dataset presents a challenging fine-grained classification problem: The best scoring EmbraceNet model using both visual and textual features achieves 69.7% accuracy. Experiments with a modified Imagen model show the dataset is also suitable for image generation conditioned on text. The dataset, source code and model checkpoints are published at https://github.com/glami/glami-1m


title: Web-based Management Information System of Cases Filed with the National Labor Relations Commission

abstract: This study was developed to describe the daily operations and encountered problems of the National Labor Relations Commission Regional Arbitration Branch No. IV (NLRC RAB IV) through conducted observations and interviews. These problems were addressed and analyzed to be the features of the developed web-based management information system (MIS) for cases. The research methodology utilized in this project was the descriptive developmental approach. The Agile Software Development methodology was followed to develop the system. It was used to quickly produce the desired output while allowing the user to go back through phases without finishing the whole cycle. The system covered managing filed complaints, Single-Entry Approach (SEnA), labor cases, and report generation. The findings, through the interview, of handling records were inconsistent and inaccurate. This study also focused on ensuring the Data Privacy Act of 2012, protecting the database's information using the XOR Cipher Algorithm. This study was evaluated using standard web evaluation criteria. Using the criteria, the study's overall mean was 4.27 and 4.43, with the descriptive meaning of Very Good, which showed that the system was accepted as perceived by experts and end-users, respectively. Management of filed cases is a vital process for the Commission. With that said, developing a web-based management information system could ease the internal operations of handling and managing filed labor cases. Moreover, respondents and complainants can easily determine their filed cases' status using the case status tracking system. For further improvements to the system, additional printable documents may be added that could be found needed by the Commission. Lastly, further research about the effectiveness of the web-based system may be conducted for further enhancements of the system.


title: Transform Once: Efficient Operator Learning in Frequency Domain

abstract: Spectral analysis provides one of the most effective paradigms for information-preserving dimensionality reduction, as simple descriptions of naturally occurring signals are often obtained via few terms of periodic basis functions. In this work, we study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time: frequency-domain models (FDMs). Existing FDMs are based on complex-valued transforms i.e. Fourier Transforms (FT), and layers that perform computation on the spectrum and input data separately. This design introduces considerable computational overhead: for each layer, a forward and inverse FT. Instead, this work introduces a blueprint for frequency domain learning through a single transform: transform once (T1). To enable efficient, direct learning in the frequency domain we derive a variance-preserving weight initialization scheme and investigate methods for frequency selection in reduced-order FDMs. Our results noticeably streamline the design process of FDMs, pruning redundant transforms, and leading to speedups of 3x to 10x that increase with data resolution and model size. We perform extensive experiments on learning the solution operator of spatio-temporal dynamics, including incompressible Navier-Stokes, turbulent flows around airfoils and high-resolution video of smoke. T1 models improve on the test performance of FDMs while requiring significantly less computation (5 hours instead of 32 for our large-scale experiment), with over 20% reduction in average predictive error across tasks.


title: Dual gradient method for ill-posed problems using multiple repeated measurement data

abstract: We consider determining $\R$-minimizing solutions of linear ill-posed problems $A x = y$, where $A: {\mathscr X} \to {\mathscr Y}$ is a bounded linear operator from a Banach space ${\mathscr X}$ to a Hilbert space ${\mathscr Y}$ and ${\mathcal R}: {\mathscr X} \to [0, \infty]$ is a proper strongly convex penalty function. Assuming that multiple repeated independent identically distributed unbiased data of $y$ are available, we consider a dual gradient method to reconstruct the ${\mathcal R}$-minimizing solution using the average of these data. By terminating the method by either an {\it a priori} stopping rule or a statistical variant of the discrepancy principle, we provide the convergence analysis and derive convergence rates when the sought solution satisfies certain variational source conditions. Various numerical results are reported to test the performance of the method.


title: Information Geometry of Dynamics on Graphs and Hypergraphs

abstract: We introduce a new information-geometric structure associated with the dynamics on discrete objects such as graphs and hypergraphs. The presented setup consists of two dually flat structures built on the vertex and edge spaces, respectively. The former is the conventional duality between density and potential, e.g., the probability density and its logarithmic form induced by a convex thermodynamic function. The latter is the duality between flux and force induced by a convex and symmetric dissipation function, which drives the dynamics of the density. These two are connected topologically by the homological algebraic relation induced by the underlying discrete objects. The generalized gradient flow in this doubly dual flat structure is an extension of the gradient flows on Riemannian manifolds, which include Markov jump processes and nonlinear chemical reaction dynamics as well as the natural gradient and mirror descent. The information-geometric projections on this doubly dual flat structure lead to information-geometric extensions of the Helmholtz-Hodge decomposition and the Otto structure in $L^{2}$ Wasserstein geometry. The structure can be extended to non-gradient nonequilibrium flows, from which we also obtain the induced dually flat structure on cycle spaces. This abstract but general framework can extend the applicability of information geometry to various problems of linear and nonlinear dynamics.

categories: cs.IT math.DG math.IT math.ST physics.chem-ph physics.data-an stat.TH


title: TetraSphere: A Neural Descriptor for O(3)-Invariant Point Cloud Analysis

abstract: In many practical applications, 3D point cloud analysis requires rotation invariance. In this paper, we present a learnable descriptor invariant under 3D rotations and reflections, i.e., the O(3) actions, utilizing the recently introduced steerable 3D spherical neurons and vector neurons. Specifically, we propose an embedding of the 3D spherical neurons into 4D vector neurons, which leverages end-to-end training of the model. In our approach, we perform TetraTransform--an equivariant embedding of the 3D input into 4D, constructed from the steerable neurons--and extract deeper O(3)-equivariant features using vector neurons. This integration of the TetraTransform into the VN-DGCNN framework, termed TetraSphere, negligibly increases the number of parameters by less than 0.0002%. TetraSphere sets a new state-of-the-art performance classifying randomly rotated real-world object scans of the challenging subsets of ScanObjectNN. Additionally, TetraSphere outperforms all equivariant methods on randomly rotated synthetic data: classifying objects from ModelNet40 and segmenting parts of the ShapeNet shapes. Thus, our results reveal the practical value of steerable 3D spherical neurons for learning in 3D Euclidean space. The code is available at https://github.com/pavlo-melnyk/tetrasphere.


title: Cohomological properties of vector-valued Lipschitz algebras and their second duals

abstract: Let $\frak{F}(X, A)$ be one of the Banach algebras $\hbox{Lip}(X, A)$ or $\hbox{lip}(X, A)$. In this paper, we show that $\frak{F}(X, A)$ is amenable if and only if $X$ is uniformly discrete and $A$ is amenable. We also prove that the result holds for $\hbox{lip}^\circ(X, A)$ instead of $\frak{F}(X, A)$. In the case where $A^*$ is separable, we establish that $\frak{F}(X, A)^{**}$ is amenable if and only if $X$ is uniformly discrete and $A^{**}$ is amenable, however, amenability of $\hbox{lip}^\circ(X, A)^{**}$ is equivalent to amenability of $A^{**}$ and finiteness of $X$. We prove that if $\hbox{Lip}(X, A)$ is point (respectively, weakly) amenable, then $X$ is uniformly discrete and $A$ is point (respectively, weakly) amenable. In particular, $\hbox{Lip}X$ is weakly amenable if and only if $X$ is discrete. We then investigate cohomological properties for vector-valued Banach algebras $C_0(X, A)$ and $L^1(G, A)$. Finally, we prove that biprojectivity (respectively, cyclically weak amenability) of $A^{**}$ implies biprojectivity (respectively, cyclically weak amenability) of $A$. This result holds for weak amenability and cyclic amenability when $A$ is commutative.


title: Probing the tidal disruption event iPTF16axa with CLOUDY and disc-wind models

abstract: We present both a disc-wind model on the optical/UV emission continuum and CLOUDY modelling on the spectral lines of the tidal disruption event (TDE) iPTF16axa to understand the disc-wind emission and the properties of the atmosphere that impacts the line luminosity of the TDE. Assuming the optical/UV emission from the wind due to the disc super-Eddington phase, we use the steady structured disc-wind model with a spherical wind with constant velocity to fit the observations on multiple days. The extracted parameters are stellar-mass $M_{\star} = 6.20 \pm 1.19 M_{\odot}$, disc radiative efficiency $\log_{\rm 10}(\eta) = -1.22 \pm 1.327$, wind inner radius $r_l = (2.013 \pm 0.551) \times 10^{14}~{\rm cm}$ and velocity $v_w = 18999.4 \pm 1785.1 ~{\rm km~s^{-1}}$. The photosphere temperature for wind emission is $ \sim 2 \times 10^4~{\rm K}$ and the disc single blackbody temperature is $\sim 0.995 \times 10^5~{\rm K}$. We also perform CLOUDY modelling to explain the observed He and H line luminosities that estimate a wind inner radius $r_l = 7.07 \times 10^{14}~{\rm cm}$ and velocity $v_w = 1.3 \times 10^4~{\rm km~s^{-1}}$. The independent analyses of iPTF16axa using CLOUDY and disc-wind models show comparable results that agree with observations. The CLOUDY modelling finds that both the super solar abundance of He and a smaller He II line optical depth is responsible for the enhancement of He II line luminosity over the H$\alpha$ line luminosity. The super-solar abundance of He II agrees with a relatively large stellar mass and suggests that the disrupted star might have been a red giant.


title: Transformer-based Model for Word Level Language Identification in Code-mixed Kannada-English Texts

abstract: Using code-mixed data in natural language processing (NLP) research currently gets a lot of attention. Language identification of social media code-mixed text has been an interesting problem of study in recent years due to the advancement and influences of social media in communication. This paper presents the Instituto Polit\'ecnico Nacional, Centro de Investigaci\'on en Computaci\'on (CIC) team's system description paper for the CoLI-Kanglish shared task at ICON2022. In this paper, we propose the use of a Transformer based model for word-level language identification in code-mixed Kannada English texts. The proposed model on the CoLI-Kenglish dataset achieves a weighted F1-score of 0.84 and a macro F1-score of 0.61.


title: Combining quantum noise reduction resources: a practical approach

abstract: Optomechanical sensors are capable of transducing external perturbations to resolvable optical signals. A particular regime of interest is that of high-bandwidth force detection, where an impulse is delivered to the system over a short period of time. Exceedingly sensitive impulse detection has been proposed to observe very weak signals like those for long range interactions with dark matter requiring much higher sensitivities than current sensors can provide. Quantum resources to go beyond the standard quantum limit of noise in these sensors include squeezing of the light used to transduce the signal, backaction evasion by measuring the optimum quadrature, and quantum nondemolition (QND) measurements which reduce backaction directly. However, it has been extremely difficult to determine a scheme where all these quantum resources contribute to noise reduction thereby exceeding the benefit of using only one quantum resource alone. We provide the theoretical limits to noise reduction while combining quantum enhanced readout techniques such as squeezing and QND measurements for these optomechanical sensors. We demonstrate that backaction evasion through QND techniques dramatically reduces the technical challenges presented when using squeezed light for broadband force detection, paving the way for combining multiple quantum noise reduction techniques for enhanced sensitivity in the context of impulse metrology.


title: CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion

abstract: Multi-modality (MM) image fusion aims to render fused images that maintain the merits of different modalities, e.g., functional highlight and detailed textures. To tackle the challenge in modeling cross-modality features and decomposing desirable modality-specific and modality-shared features, we propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. Firstly, CDDFuse uses Restormer blocks to extract cross-modality shallow features. We then introduce a dual-branch Transformer-CNN feature extractor with Lite Transformer (LT) blocks leveraging long-range attention to handle low-frequency global features and Invertible Neural Networks (INN) blocks focusing on extracting high-frequency local information. A correlation-driven loss is further proposed to make the low-frequency features correlated while the high-frequency features uncorrelated based on the embedded information. Then, the LT-based global fusion and INN-based local fusion layers output the fused image. Extensive experiments demonstrate that our CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. We also show that CDDFuse can boost the performance in downstream infrared-visible semantic segmentation and object detection in a unified benchmark. The code is available at https://github.com/Zhaozixiang1228/MMIF-CDDFuse.


title: Meta Architecture for Point Cloud Analysis

abstract: Recent advances in 3D point cloud analysis bring a diverse set of network architectures to the field. However, the lack of a unified framework to interpret those networks makes any systematic comparison, contrast, or analysis challenging, and practically limits healthy development of the field. In this paper, we take the initiative to explore and propose a unified framework called PointMeta, to which the popular 3D point cloud analysis approaches could fit. This brings three benefits. First, it allows us to compare different approaches in a fair manner, and use quick experiments to verify any empirical observations or assumptions summarized from the comparison. Second, the big picture brought by PointMeta enables us to think across different components, and revisit common beliefs and key design decisions made by the popular approaches. Third, based on the learnings from the previous two analyses, by doing simple tweaks on the existing approaches, we are able to derive a basic building block, termed PointMetaBase. It shows very strong performance in efficiency and effectiveness through extensive experiments on challenging benchmarks, and thus verifies the necessity and benefits of high-level interpretation, contrast, and comparison like PointMeta. In particular, PointMetaBase surpasses the previous state-of-the-art method by 0.7%/1.4/%2.1% mIoU with only 2%/11%/13% of the computation cost on the S3DIS datasets.


title: The existence of null circular geodesics outside extremal spherically symmetric asymptotically flat hairy black holes

abstract: The existence of null circular geodesics has been proved in the background of non-extremal spherically symmetric asymptotically flat black holes in previous works. Then it is an interesting question that whether extremal black holes possess null circular geodesics outside horizons. In the present paper, we pay attentions to the extremal spherically symmetric asymptotically flat hairy black holes. We show the existence of the fastest trajectory to circle a extremal black hole. As the fastest trajectory corresponds to the position of null circular geodesics, we prove that null circular geodesics exist outside extremal spherically symmetric asymptotically flat hairy black holes. We also point out that our proof also works for non-extremal black holes.


title: Metal-insulator phase separation in KTaO3-based two-dimensional electron gas

abstract: Electronic phase separation (EPS) originates from an incomplete transformation between electronic phases, causing the inhomogeneous spatial distribution of electronic properties. In the system of two-dimensional electron gas (2DEG), the EPS is usually identified based on a percolative metal-to-superconductor transition. Here, we report a metal-insulator transition (MIT) in KTaO3-based 2DEG with the width of conductive channel decreasing into micrometer scale. Hysteretic resistance-temperature relations are observed due to the competition between metallic and insulating phases, which is tunable by magnetic field. Such a size-dependent MIT effect is attributed to the coexistence and separation of metallic and insulating phases. Combining density functional theory calculation, we propose a theoretical model to simulate the dynamic process of the EPS using the percolation theory, demonstrating the mechanism of size-dependent MIT. Our work suggests a clear and simple 2DEG platform to achieve the spatial coexistence of metallic and insulating phases.


title: Effective Chiral Magnetic Effect from Neutrino Radiation

abstract: We develop an approach to chiral kinetic theories for electrons close to equilibrium and neutrinos away from equilibrium based on a systematic power counting scheme for different timescales of electromagnetic and weak interactions. Under this framework, we derive electric and energy currents along magnetic fields induced by neutrino radiation in general nonequilibrium states. This may be regarded as an effective chiral magnetic effect (CME), which is present without a chiral chemical potential, unlike the conventional CME. We also consider the so-called gain region of core-collapse supernovae as an example and find that the effective CME enhanced by persistent neutrino emission in time is sufficiently large to lead to the inverse cascade of magnetic and fluid kinetic energies and observed magnitudes of pulsar kicks. Our framework may also be applicable to other dense-matter systems involving nonequilibrium neutrinos.


title: SKDBERT: Compressing BERT via Stochastic Knowledge Distillation

abstract: In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain compact BERT-style language model dubbed SKDBERT. In each iteration, SKD samples a teacher model from a pre-defined teacher ensemble, which consists of multiple teacher models with multi-level capacities, to transfer knowledge into student model in an one-to-one manner. Sampling distribution plays an important role in SKD. We heuristically present three types of sampling distributions to assign appropriate probabilities for multi-level teacher models. SKD has two advantages: 1) it can preserve the diversities of multi-level teacher models via stochastically sampling single teacher model in each iteration, and 2) it can also improve the efficacy of knowledge distillation via multi-level teacher models when large capacity gap exists between the teacher model and the student model. Experimental results on GLUE benchmark show that SKDBERT reduces the size of a BERT$_{\rm BASE}$ model by 40% while retaining 99.5% performances of language understanding and being 100% faster.


title: Self-Supervised Surgical Instrument 3D Reconstruction from a Single Camera Image

abstract: Surgical instrument tracking is an active research area that can provide surgeons feedback about the location of their tools relative to anatomy. Recent tracking methods are mainly divided into two parts: segmentation and object detection. However, both can only predict 2D information, which is limiting for application to real-world surgery. An accurate 3D surgical instrument model is a prerequisite for precise predictions of the pose and depth of the instrument. Recent single-view 3D reconstruction methods are only used in natural object reconstruction and do not achieve satisfying reconstruction accuracy without 3D attribute-level supervision. Further, those methods are not suitable for the surgical instruments because of their elongated shapes. In this paper, we firstly propose an end-to-end surgical instrument reconstruction system -- Self-supervised Surgical Instrument Reconstruction (SSIR). With SSIR, we propose a multi-cycle-consistency strategy to help capture the texture information from a slim instrument while only requiring a binary instrument label map. Experiments demonstrate that our approach improves the reconstruction quality of surgical instruments compared to other self-supervised methods and achieves promising results.


title: Similarity-based cooperative equilibrium

abstract: As machine learning agents act more autonomously in the world, they will increasingly interact with each other. Unfortunately, in many social dilemmas like the one-shot Prisoner's Dilemma, standard game theory predicts that ML agents will fail to cooperate with each other. Prior work has shown that one way to enable cooperative outcomes in the one-shot Prisoner's Dilemma is to make the agents mutually transparent to each other, i.e., to allow them to access one another's source code (Rubinstein 1998, Tennenholtz 2004) -- or weights in the case of ML agents. However, full transparency is often unrealistic, whereas partial transparency is commonplace. Moreover, it is challenging for agents to learn their way to cooperation in the full transparency setting. In this paper, we introduce a more realistic setting in which agents only observe a single number indicating how similar they are to each other. We prove that this allows for the same set of cooperative outcomes as the full transparency setting. We also demonstrate experimentally that cooperation can be learned using simple ML methods.