title: Fourier Methods for Sufficient Dimension Reduction in Time Series

abstract: Dimensionality reduction has always been one of the most significant and challenging problems in the analysis of high-dimensional data. In the context of time series analysis, our focus is on the estimation and inference of conditional mean and variance functions. By using central mean and variance dimension reduction subspaces that preserve sufficient information about the response, one can effectively estimate the unknown mean and variance functions of the time series. While the literature presents several approaches to estimate the time series central mean and variance subspaces (TS-CMS and TS-CVS), these methods tend to be computationally intensive and infeasible for practical applications. By employing the Fourier transform, we derive explicit estimators for TS-CMS and TS-CVS. These proposed estimators are demonstrated to be consistent, asymptotically normal, and efficient. Simulation studies have been conducted to evaluate the performance of the proposed method. The results show that our method is significantly more accurate and computationally efficient than existing methods. Furthermore, the method has been applied to the Canadian Lynx dataset.


title: TriDeNT: Triple Deep Network Training for Privileged Knowledge Distillation in Histopathology

abstract: Computational pathology models rarely utilise data that will not be available for inference. This means most models cannot learn from highly informative data such as additional immunohistochemical (IHC) stains and spatial transcriptomics. We present TriDeNT, a novel self-supervised method for utilising privileged data that is not available during inference to improve performance. We demonstrate the efficacy of this method for a range of different paired data including immunohistochemistry, spatial transcriptomics and expert nuclei annotations. In all settings, TriDeNT outperforms other state-of-the-art methods in downstream tasks, with observed improvements of up to 101%. Furthermore, we provide qualitative and quantitative measurements of the features learned by these models and how they differ from baselines. TriDeNT offers a novel method to distil knowledge from scarce or costly data during training, to create significantly better models for routine inputs.


title: Distributed Optimization with Feasible Set Privacy

abstract: We consider the setup of a constrained optimization problem with two agents $E_1$ and $E_2$ who jointly wish to learn the optimal solution set while keeping their feasible sets $\mathcal{P}_1$ and $\mathcal{P}_2$ private from each other. The objective function $f$ is globally known and each feasible set is a collection of points from a global alphabet. We adopt a sequential symmetric private information retrieval (SPIR) framework where one of the agents (say $E_1$) privately checks in $\mathcal{P}_2$, the presence of candidate solutions of the problem constrained to $\mathcal{P}_1$ only, while learning no further information on $\mathcal{P}_2$ than the solution alone. Further, we extract an information theoretically private threshold PSI (ThPSI) protocol from our scheme and characterize its download cost. We show that, compared to privately acquiring the feasible set $\mathcal{P}_1\cap \mathcal{P}_2$ using an SPIR-based private set intersection (PSI) protocol, and finding the optimum, our scheme is better as it incurs less information leakage and less download cost than the former. Over all possible uniform mappings of $f$ to a fixed range of values, our scheme outperforms the former with a high probability.


title: A Framework for Symmetric Self-Intersecting Surfaces

abstract: 3D printing of surfaces has become an established method for prototyping and visualisation. However, surfaces often contain certain degenerations, such as self-intersecting faces or non-manifold parts, which pose problems in obtaining a 3D printable file. Therefore, it is necessary to examine these degenerations beforehand. Surfaces in three-dimensional space can be represented as embedded simplicial complexes describing a triangulation of the surface. We use this combinatorial description, and the notion of embedded simplicial surfaces (which can be understood as well-behaved surfaces) to give a framework for obtaining 3D printable files. This provides a new perspective on self-intersecting triangulated surfaces in three-dimensional space. Our method first retriangulates a surface using a minimal number of triangles, then computes its outer hull, and finally treats non-manifold parts. To this end, we prove an initialisation criterion for the computation of the outer hull. We also show how symmetry properties can be used to simplify computations. Implementations of the proposed algorithms are given in the computer algebra system GAP4. To verify our methods, we use a dataset of self-intersecting symmetric icosahedra. Exploiting the symmetry of the underlying embedded complex leads to a notable speed-up and enhanced numerical robustness when computing a retriangulation, compared to methods that do not take advantage of symmetry.


title: Transitions of Solutions and Their Efficiency

abstract: We broaden the basis of non-cooperative game theory by considering miscoordination on a solution concept. For any solution concept, we extend the solution set of a strategic-form game to a transition set. This set contains profiles where various agents simultaneously follow different solutions, e.g.~different Nash equilibria. This models the fact that in practice, complicated agents are rarely perfectly coordinated on the same equilibrium. We define two efficiency measures, called the price of transition anarchy and stability, and bound them. We also refine the notion of transition to the notion of limited transition, where only a limited number of solutions is simultaneously played, and to stable transitions, which allow for only minor lack of coordination. We compare the above mentioned efficiency measures and bound the efficiency of transitions in important cases, including the important cases of constant-sum and potential games, which span the set of finite games with the same number of strategies for each agent. We also prove tight efficiency bounds for routing games and coordination games on graphs. Finally, we study algorithms to find the transition degree required to make a given profile a transition, or to render all the profiles transitions. We conclude that for the sake of efficiency, it is crucial to avoid uncoordinated transitions, besides certain cases, such as constant-sum games, identical utility games, some types of routing games, limited transitions in potential games, and stable transitions in coordination games.


title: Exact solutions of Cotton Gravity in its Codazzi formulation

abstract: The "Codazzi formulation", based on a Codazzi tensor, provides a more robust and straightforward theoretical framework for "Cotton Gravity" (CG) than its original formulation in terms of the Cotton tensor. Using this formulation we provide a self-consistent procedure to generate non-trivial exact solutions in CG that generalize well known General Relativity (GR) solutions. We re-derive a known CG solution that generalizes the Schwarzschild solution of GR, showing that it is the unique vacuum solution of static spherical symmetry in CG, extending this result to a CG generalization of the Reissner-Nordstrom solution of GR, all of which places a strong case supporting the fulfillment of Birkhoff's theorem. When applied to Friedman-Lema\^\i tre-Robertson-Walker (FLRW) models CG naturally identifies the $\Lambda$CDM model as the unique FLRW dust model with constant negative spatial curvature. We also obtain CG generalizations of the Lema\^\i tre-Tolman-Bondi (LTB) and Szekeres dust solutions of GR, allowing for time and space dependent changes from decelerated to accelerated evolution, without necessarily assuming a dark energy source. The CG generalization of static perfect fluid spheres allows in the weak field regime to model the flattening of rotation velocities in spherical galactic systems without assuming dark matter. We also generalize non-static spherically symmetric perfect fluid solutions with a shear-free 4 velocity. Our results suggest the need for further research using the Codazzi formulation to explore the potential for applications of CG to current open problems in gravitational systems.


title: GIVT: Generative Infinite-Vocabulary Transformers

abstract: We introduce generative infinite-vocabulary transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose two surprisingly simple modifications to decoder-only transformers: 1) at the input, we replace the finite-vocabulary lookup table with a linear projection of the input vectors; and 2) at the output, we replace the logits prediction (usually mapped to a categorical distribution) with the parameters of a multivariate Gaussian mixture model. Inspired by the image-generation paradigm of VQ-GAN and MaskGIT, where transformers are used to model the discrete latent sequences of a VQ-VAE, we use GIVT to model the unquantized real-valued latent sequences of a $\beta$-VAE. In class-conditional image generation GIVT outperforms VQ-GAN (and improved variants thereof) as well as MaskGIT, and achieves performance competitive with recent latent diffusion models. Finally, we obtain strong results outside of image generation when applying GIVT to panoptic segmentation and depth estimation with a VAE variant of the UViM framework


title: Aspects of the Higgs phase in SU(2)xU(1) lattice gauge Higgs theory

abstract: Using a simplified lattice version of the electroweak sector of the standard model, with dynamical fermions excluded, we determine at fixed Weinberg angle the transition line between the confined phase and the Higgs phase, the latter defined as the region where the global center subgroup of the gauge group is spontaneously broken, and "separation of charge" confinement disappears. We then search, via lattice Monte Carlo simulations, for possible neutral vector bosons in the Higgs region, apart from the photon and Z. There are numerical indications of a "light Z" in the lattice data (along with the photon and the Z), but a lack of the expected scaling of the light mass particle excludes any firm conclusions about the physical spectrum.


title: When it Rains, it Pours: Modeling Media Storms and the News Ecosystem

abstract: Most events in the world receive at most brief coverage by the news media. Occasionally, however, an event will trigger a media storm, with voluminous and widespread coverage lasting for weeks instead of days. In this work, we develop and apply a pairwise article similarity model, allowing us to identify story clusters in corpora covering local and national online news, and thereby create a comprehensive corpus of media storms over a nearly two year period. Using this corpus, we investigate media storms at a new level of granularity, allowing us to validate claims about storm evolution and topical distribution, and provide empirical support for previously hypothesized patterns of influence of storms on media coverage and intermedia agenda setting.


title: Tree of Attacks: Jailbreaking Black-Box LLMs Automatically

abstract: While Large Language Models (LLMs) display versatile functionality, they continue to generate harmful, biased, and toxic content, as demonstrated by the prevalence of human-designed jailbreaks. In this work, we present Tree of Attacks with Pruning (TAP), an automated method for generating jailbreaks that only requires black-box access to the target LLM. TAP utilizes an LLM to iteratively refine candidate (attack) prompts using tree-of-thought reasoning until one of the generated prompts jailbreaks the target. Crucially, before sending prompts to the target, TAP assesses them and prunes the ones unlikely to result in jailbreaks. Using tree-of-thought reasoning allows TAP to navigate a large search space of prompts and pruning reduces the total number of queries sent to the target. In empirical evaluations, we observe that TAP generates prompts that jailbreak state-of-the-art LLMs (including GPT4 and GPT4-Turbo) for more than 80% of the prompts using only a small number of queries. Interestingly, TAP is also capable of jailbreaking LLMs protected by state-of-the-art guardrails, e.g., LlamaGuard. This significantly improves upon the previous state-of-the-art black-box method for generating jailbreaks.


title: Magicoder: Empowering Code Generation with OSS-Instruct

abstract: We introduce Magicoder, a series of fully open-source (code, weights, and data) Large Language Models (LLMs) for code that significantly closes the gap with top code models while having no more than 7B parameters. Magicoder models are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate diverse instruction data for code. Our main motivation is to mitigate the inherent bias of the synthetic data generated by LLMs through the wealth of open-source references for the production of more realistic and controllable data. The orthogonality of OSS-Instruct and other data generation methods like Evol-Instruct further enables us to build an enhanced MagicoderS. Both Magicoder and MagicoderS substantially outperform state-of-the-art code models with similar or even larger sizes on a wide range of coding benchmarks. Notably, MagicoderS-CL-7B based on CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in pass@1 ). Overall, OSS-Instruct opens a new direction for crafting diverse synthetic instruction data for code using abundant open-source references.


title: Mathematical Supplement for the $\texttt{gsplat}$ Library

abstract: This report provides the mathematical details of the gsplat library, a modular toolbox for efficient differentiable Gaussian splatting, as proposed by Kerbl et al. It provides a self-contained reference for the computations involved in the forward and backward passes of differentiable Gaussian splatting. To facilitate practical usage and development, we provide a user friendly Python API that exposes each component of the forward and backward passes in rasterization at github.com/nerfstudio-project/gsplat .


title: Prescribed $L_p$ curvature problem

abstract: In this paper, we establish the existence of smooth, origin-symmetric, strictly convex solutions to the prescribed even $L_p$ curvature problem.


title: In-situ tuning of optomechanical crystals with nano-oxidation

abstract: Optomechanical crystals are a promising device platform for quantum transduction and sensing. Precise targeting of the optical and acoustic resonance frequencies of these devices is crucial for future advances on these fronts. However, fabrication disorder in these wavelength-scale nanoscale devices typically leads to inhomogeneous resonance frequencies. Here we achieve in-situ, selective frequency tuning of optical and acoustic resonances in silicon optomechanical crystals via electric field-induced nano-oxidation using an atomic-force microscope. Our method can achieve a tuning range >2 nm (0.13%) for the optical resonance wavelength in the telecom C-band, and >60 MHz (1.2%) for the acoustic resonance frequency at 5 GHz. The tuning resolution of 1.1 nm for the optical wavelength, and $150$ kHz for the acoustic frequency allows us to spectrally align multiple optomechanical crystal resonators using optimal oxidation patterns determined via an inverse design protocol. Our results establish a method for precise post-fabrication tuning of optomechancical crystals. This technique can enable coupled optomechanical resonator arrays, scalable resonant optomechanical circuits, and frequency matching of microwave-optical quantum transducers.


title: VerA: Versatile Anonymization Fit for Clinical Facial Images

abstract: The escalating legislative demand for data privacy in facial image dissemination has underscored the significance of image anonymization. Recent advancements in the field surpass traditional pixelation or blur methods, yet they predominantly address regular single images. This leaves clinical image anonymization -- a necessity for illustrating medical interventions -- largely unaddressed. We present VerA, a versatile facial image anonymization that is fit for clinical facial images where: (1) certain semantic areas must be preserved to show medical intervention results, and (2) anonymizing image pairs is crucial for showing before-and-after results. VerA outperforms or is on par with state-of-the-art methods in de-identification and photorealism for regular images. In addition, we validate our results on paired anonymization, and on the anonymization of both single and paired clinical images with extensive quantitative and qualitative evaluation.


title: TPPoet: Transformer-Based Persian Poem Generation using Minimal Data and Advanced Decoding Techniques

abstract: Recent advances in language models (LMs), have demonstrated significant efficacy in tasks related to the arts and humanities. While LMs have exhibited exceptional performance across a wide range of natural language processing tasks, there are notable challenges associated with their utilization on small datasets and their ability to replicate more creative human capacities. In this study, we aim to address these challenges by training a Persian classical poetry generation model using a transformer architecture on a specialized dataset with no pretraining. Additionally, we propose a novel decoding method to enhance coherence and meaningfulness in the generated poetry, effectively managing the tradeoff between diversity and quality. Furthermore, the results of our training approach and the proposed decoding method are evaluated through comprehensive set of automatic and human evaluations and showed its superior capability to generate coherent and meaningful poetry in compare to other decoding methods and an existing Persian large language model (LLM).


title: SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM

abstract: Dense simultaneous localization and mapping (SLAM) is crucial for robotics and augmented reality applications. However, current methods are often hampered by the non-volumetric or implicit way they represent a scene. This work introduces SplaTAM, an approach that, for the first time, leverages explicit volumetric representations, i.e., 3D Gaussians, to enable high-fidelity reconstruction from a single unposed RGB-D camera, surpassing the capabilities of existing methods. SplaTAM employs a simple online tracking and mapping system tailored to the underlying Gaussian representation. It utilizes a silhouette mask to elegantly capture the presence of scene density. This combination enables several benefits over prior representations, including fast rendering and dense optimization, quickly determining if areas have been previously mapped, and structured map expansion by adding more Gaussians. Extensive experiments show that SplaTAM achieves up to 2x superior performance in camera pose estimation, map construction, and novel-view synthesis over existing methods, paving the way for more immersive high-fidelity SLAM applications.


title: Chaotic and Thermal Aspects in the $| HES \rangle$ S-Matrix

abstract: We compute tree level scattering amplitudes involving more than one highly excited states and tachyons in bosonic string theory. We use these amplitudes to understand chaotic and thermal aspects of the excited string states lending support to the Susskind-Horowitz-Polchinski correspondence principle. The unaveraged amplitudes exhibit chaos in the resonance distribution as a function of kinematic parameters, which can be described by random matrix theory. Upon coarse-graining these amplitudes are shown to exponentiate, and capture various thermal features, including features of a stringy version of the eigenstate thermalization hypothesis as well as notions of typicality. Further, we compute the effective string form factor corresponding to the highly excited states, and argue for the random walk behaviour of the long strings.


title: Can we truly transfer an actor's genuine happiness to avatars? An investigation into virtual, real, posed and spontaneous faces

abstract: A look is worth a thousand words is a popular phrase. And why is a simple look enough to portray our feelings about something or someone? Behind this question are the theoretical foundations of the field of psychology regarding social cognition and the studies of psychologist Paul Ekman. Facial expressions, as a form of non-verbal communication, are the primary way to transmit emotions between human beings. The set of movements and expressions of facial muscles that convey some emotional state of the individual to their observers are targets of studies in many areas. Our research aims to evaluate Ekman's action units in datasets of real human faces, posed and spontaneous, and virtual human faces resulting from transferring real faces into Computer Graphics faces. In addition, we also conducted a case study with specific movie characters, such as SheHulk and Genius. We intend to find differences and similarities in facial expressions between real and CG datasets, posed and spontaneous faces, and also to consider the actors' genders in the videos. This investigation can help several areas of knowledge, whether using real or virtual human beings, in education, health, entertainment, games, security, and even legal matters. Our results indicate that AU intensities are greater for posed than spontaneous datasets, regardless of gender. Furthermore, there is a smoothing of intensity up to 80 percent for AU6 and 45 percent for AU12 when a real face is transformed into CG.


title: Meromorphic CFTs have central charges c = 8$\mathbb{N}$: a proof based on the MLDE approach and Rademacher series

abstract: In this short note, we present a simple and elementary proof that meromorphic conformal field theories (CFTs) have central charges of the form: $c=8N$ with $N\in\mathbb{N}$ (the set of natural numbers) using the modular linear differential equations (MLDEs) approach. We first set up the 1-character MLDE for arbitrary value of the Wronskian index: $\ell$. From this we get the general form of the meromorphic CFT's character. We then study its modular transformations and the asymptotic value of it's Fourier coefficients -- using Rademacher series -- to conclude that odd values of $\ell$ make the character in-admissible implying that the central charge for admissible character has to be a multiple of 8.

categories: hep-th cond-mat.str-el math-ph math.MP math.NT math.QA


title: Fundamental Physics Opportunities with the Next-Generation Event Horizon Telescope

abstract: The Event Horizon Telescope (EHT) Collaboration recently published the first images of the supermassive black holes in the cores of the Messier 87 and Milky Way galaxies. These observations have provided a new means to study supermassive black holes and probe physical processes occurring in the strong-field regime. We review the prospects of future observations and theoretical studies of supermassive black hole systems with the next-generation Event Horizon Telescope (ngEHT), which will greatly enhance the capabilities of the existing EHT array. These enhancements will open up several previously inaccessible avenues of investigation, thereby providing important new insights into the properties of supermassive black holes and their environments. This review describes the current state of knowledge for five key science cases, summarising the unique challenges and opportunities for fundamental physics investigations that the ngEHT will enable.

categories: astro-ph.HE astro-ph.IM gr-qc hep-ph hep-th


title: Swampland Constraints on the SymTFT of Supergravity

abstract: We consider string/M-theory reductions on a compact space $X=X^\text{loc} \cup X^\circ$, where $X^\text{loc}$ contains the singular locus, and $X^\circ$ its complement. For the resulting supergravity theories, we construct a suitable Symmetry Topological Field Theory (SymTFT) associated with the boundary $\partial X^\text{loc} \coprod \partial X^\circ$. We propose that boundary conditions for different BPS branes wrapping the same boundary cycles must be correlated for the SymTFT to yield an absolute theory consistent with quantum gravity. Using heterotic/M-theory duality, this constraint can be translated into a field theoretic statement, which restricts the global structure of $d\geq 7$, $\mathcal{N}=1$ supergravity theories to reproduce precisely the landscape of untwisted toroidal heterotic compactifications. Furthermore, for 6d $(2,0)$ theories, we utilize a subtle interplay between gauged 0-, 2-, and 4-form symmetries to provide a bottom-up explanation of the correlated boundary conditions in K3 compactifications of type IIB.


title: Hot PATE: Private Aggregation of Distributions for Diverse Task

abstract: The Private Aggregation of Teacher Ensembles (PATE) framework is a versatile approach to privacy-preserving machine learning. In PATE, teacher models that are not privacy-preserving are trained on distinct portions of sensitive data. Privacy-preserving knowledge transfer to a student model is then facilitated by privately aggregating teachers' predictions on new examples. Employing PATE with generative auto-regressive models presents both challenges and opportunities. These models excel in open ended \emph{diverse} (aka hot) tasks with multiple valid responses. Moreover, the knowledge of models is often encapsulated in the response distribution itself and preserving this diversity is critical for fluid and effective knowledge transfer from teachers to student. In all prior designs, higher diversity resulted in lower teacher agreement and thus -- a tradeoff between diversity and privacy. Prior works with PATE thus focused on non-diverse settings or limiting diversity to improve utility. We propose \emph{hot PATE}, a design tailored for the diverse setting. In hot PATE, each teacher model produces a response distribution that can be highly diverse. We mathematically model the notion of \emph{preserving diversity} and propose an aggregation method, \emph{coordinated ensembles}, that preserves privacy and transfers diversity with \emph{no penalty} to privacy or efficiency. We demonstrate empirically the benefits of hot PATE for in-context learning via prompts and potential to unleash more of the capabilities of generative models.


title: Style Aligned Image Generation via Shared Attention

abstract: Large-scale Text-to-Image (T2I) models have rapidly gained prominence across creative fields, generating visually compelling outputs from textual prompts. However, controlling these models to ensure consistent style remains challenging, with existing methods necessitating fine-tuning and manual intervention to disentangle content and style. In this paper, we introduce StyleAligned, a novel technique designed to establish style alignment among a series of generated images. By employing minimal `attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. This approach allows for the creation of style-consistent images using a reference style through a straightforward inversion operation. Our method's evaluation across diverse styles and text prompts demonstrates high-quality synthesis and fidelity, underscoring its efficacy in achieving consistent style across various inputs.


title: GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians

abstract: We present GaussianAvatar, an efficient approach to creating realistic human avatars with dynamic 3D appearances from a single video. We start by introducing animatable 3D Gaussians to explicitly represent humans in various poses and clothing styles. Such an explicit and animatable representation can fuse 3D appearances more efficiently and consistently from 2D observations. Our representation is further augmented with dynamic properties to support pose-dependent appearance modeling, where a dynamic appearance network along with an optimizable feature tensor is designed to learn the motion-to-appearance mapping. Moreover, by leveraging the differentiable motion condition, our method enables a joint optimization of motions and appearances during avatar modeling, which helps to tackle the long-standing issue of inaccurate motion estimation in monocular settings. The efficacy of GaussianAvatar is validated on both the public dataset and our collected dataset, demonstrating its superior performances in terms of appearance quality and rendering efficiency.


title: Fast View Synthesis of Casual Videos

abstract: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.


title: BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation

abstract: Generating large-scale 3D scenes cannot simply apply existing 3D object synthesis technique since 3D scenes usually hold complex spatial configurations and consist of a number of objects at varying scales. We thus propose a practical and efficient 3D representation that incorporates an equivariant radiance field with the guidance of a bird's-eye view (BEV) map. Concretely, objects of synthesized 3D scenes could be easily manipulated through steering the corresponding BEV maps. Moreover, by adequately incorporating positional encoding and low-pass filters into the generator, the representation becomes equivariant to the given BEV map. Such equivariance allows us to produce large-scale, even infinite-scale, 3D scenes via synthesizing local scenes and then stitching them with smooth consistency. Extensive experiments on 3D scene datasets demonstrate the effectiveness of our approach. Our project website is at https://zqh0253.github.io/BerfScene/.


title: MANUS: Markerless Grasp Capture using Articulated 3D Gaussians

abstract: Understanding how we grasp objects with our hands has important applications in areas like robotics and mixed reality. However, this challenging problem requires accurate modeling of the contact between hands and objects. To capture grasps, existing methods use skeletons, meshes, or parametric models that does not represent hand shape accurately resulting in inaccurate contacts. We present MANUS, a method for Markerless Hand-Object Grasp Capture using Articulated 3D Gaussians. We build a novel articulated 3D Gaussians representation that extends 3D Gaussian splatting for high-fidelity representation of articulating hands. Since our representation uses Gaussian primitives, it enables us to efficiently and accurately estimate contacts between the hand and the object. For the most accurate results, our method requires tens of camera views that current datasets do not provide. We therefore build MANUS-Grasps, a new dataset that contains hand-object grasps viewed from 50+ cameras across 30+ scenes, 3 subjects, and comprising over 7M frames. In addition to extensive qualitative results, we also show that our method outperforms others on a quantitative contact evaluation method that uses paint transfer from the object to the hand.


title: Chiral algebra from worldsheet

abstract: The chiral algebra of a 4D $N\geq2$ superconformal field theory is a vertex operator algebra generated by the Schur subsector of the 4D theory and its rigid (yet rich) structure has been useful in constraining and classifying 4D N=2 SCFTs. We study how the chiral algebra arises from the worldsheet perspective. In the worldsheet CFT dual of 4D N=4 SYM at the free point, we extract the subsector that corresponds to the spacetime Schur operators at generic coupling, and show how they generate the chiral algebra. The result can be easily generalized to 4D N=2 superconformal field theories that arise as orbifolds of 4D N=4 SYM.


title: DiffiT: Diffusion Vision Transformers for Image Generation

abstract: Diffusion models with their powerful expressivity and high sample quality have achieved State-Of-The-Art (SOTA) performance in the generative domain. The pioneering Vision Transformer (ViT) has also demonstrated strong modeling capabilities and scalability, especially for recognition tasks. In this paper, we study the effectiveness of ViTs in diffusion-based generative learning and propose a new model denoted as Diffusion Vision Transformers (DiffiT). Specifically, we propose a methodology for finegrained control of the denoising process and introduce the Time-dependant Multihead Self Attention (TMSA) mechanism. DiffiT is surprisingly effective in generating high-fidelity images with significantly better parameter efficiency. We also propose latent and image space DiffiT models and show SOTA performance on a variety of class-conditional and unconditional synthesis tasks at different resolutions. The Latent DiffiT model achieves a new SOTA FID score of 1.73 on ImageNet-256 dataset while having 19.85%, 16.88% less parameters than other Transformer-based diffusion models such as MDT and DiT, respectively. Code: https://github.com/NVlabs/DiffiT


title: MMT/Binospec Spectroscopic Survey of Two $z\sim$ 0.8 Galaxy Clusters in the Eye of Horus Field

abstract: The discovery of the Eye of Horus (EoH), a rare double source-plane lens system ($z_{\rm lens}=$ 0.795; $z_{\rm src}=$ 1.302 and 1.988), has also led to the identification of two high-redshift ($z_{\rm phot}\sim$ 0.8) galaxy clusters in the same field based on the subsequent analysis of the Subaru/Hyper Suprime-Cam (HSC) optical and XMM-Newton X-ray data. The two brightest cluster galaxies (BCGs), one of which is the lensing galaxy of the EoH, are separated by only $\sim$100$"$ ($=$ 0.75 Mpc $<$ $r_{200}$) on the sky, raising the possibility that these two clusters may be physically associated. Here, we present a follow-up optical spectroscopic survey of this EoH field, obtaining 218 secure redshifts using MMT/Binospec. We have confirmed that there indeed exist two massive ($M_{\rm dyn}$ $>$ $10^{14}$ M$_\odot$) clusters of galaxies at $z$ $=$ 0.795 (the main cluster) and at $z=0.769$ (the NE cluster). However, these clusters have a velocity offset of $\sim$4300 km s$^{-1}$, suggesting that this two-cluster system is likely a line-of-sight projection rather than a physically-related association (e.g., a cluster merger). In terms of the properties of cluster-member galaxies, these two $z\sim0.8$ clusters appear well-developed, each harboring an old (age $=$ 3.6-6.0 Gyr) and massive ($M_\mathrm{*}$ $=$ 4.2-9.5 $\times$ $10^{11}$ M$_\odot$) BCG and exhibiting a well-established red sequence (RS). This study underscores the importance of conducting a spectroscopic follow-up for high-redshift cluster candidates because RS-based cluster selections are susceptible to such a projection effect in general.


title: iMatching: Imperative Correspondence Learning

abstract: Learning feature correspondence is a foundational task in computer vision, holding immense importance for downstream applications such as visual odometry and 3D reconstruction. Despite recent progress in data-driven models, feature correspondence learning is still limited by the lack of accurate per-pixel correspondence labels. To overcome this difficulty, we introduce a new self-supervised scheme, imperative learning (IL), for training feature correspondence. It enables correspondence learning on arbitrary uninterrupted videos without any camera pose or depth labels, heralding a new era for self-supervised correspondence learning. Specifically, we formulated the problem of correspondence learning as a bilevel optimization, which takes the reprojection error from bundle adjustment as a supervisory signal for the model. To avoid large memory and computation overhead, we leverage the stationary point to effectively back-propagate the implicit gradients through bundle adjustment. Through extensive experiments, we demonstrate superior performance on tasks including feature matching and pose estimation, in which we obtained an average of 30% accuracy gain over the state-of-the-art matching models.


title: Object Recognition as Next Token Prediction

abstract: We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels. To ground this prediction process in auto-regression, we customize a non-causal attention mask for the decoder, incorporating two key features: modeling tokens from different labels to be independent, and treating image tokens as a prefix. This masking mechanism inspires an efficient method - one-shot sampling - to simultaneously sample tokens of multiple labels in parallel and rank generated labels by their probabilities during inference. To further enhance the efficiency, we propose a simple strategy to construct a compact decoder by simply discarding the intermediate blocks of a pretrained language model. This approach yields a decoder that matches the full model's performance while being notably more efficient. The code is available at https://github.com/kaiyuyue/nxtp


title: Competition-Level Problems are Effective LLM Evaluators

abstract: Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about these abilities and the potential data contamination problem recently. This paper aims to evaluate the reasoning capacities of LLMs, specifically in solving recent competition-level programming problems in Codeforces, which are expert-crafted and unique, requiring deep understanding and robust reasoning skills. We first provide a comprehensive evaluation of GPT-4's peiceived zero-shot performance on this task, considering various aspects such as problems' release time, difficulties, and types of errors encountered. Surprisingly, the peiceived performance of GPT-4 has experienced a cliff like decline in problems after September 2021 consistently across all the difficulties and types of problems, which shows the potential data contamination, as well as the challenges for any existing LLM to solve unseen complex reasoning problems. We further explore various approaches such as fine-tuning, Chain-of-Thought prompting and problem description simplification, unfortunately none of them is able to consistently mitigate the challenges. Through our work, we emphasis the importance of this excellent data source for assessing the genuine reasoning capabilities of LLMs, and foster the development of LLMs with stronger reasoning abilities and better generalization in the future.


title: Optimizing Camera Configurations for Multi-View Pedestrian Detection

abstract: Jointly considering multiple camera views (multi-view) is very effective for pedestrian detection under occlusion. For such multi-view systems, it is critical to have well-designed camera configurations, including camera locations, directions, and fields-of-view (FoVs). Usually, these configurations are crafted based on human experience or heuristics. In this work, we present a novel solution that features a transformer-based camera configuration generator. Using reinforcement learning, this generator autonomously explores vast combinations within the action space and searches for configurations that give the highest detection accuracy according to the training dataset. The generator learns advanced techniques like maximizing coverage, minimizing occlusion, and promoting collaboration. Across multiple simulation scenarios, the configurations generated by our transformer-based model consistently outperform random search, heuristic-based methods, and configurations designed by human experts, shedding light on future camera layout optimization.


title: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation

abstract: Monocular depth estimation is a fundamental computer vision task. Recovering 3D depth from a single image is geometrically ill-posed and requires scene understanding, so it is not surprising that the rise of deep learning has led to a breakthrough. The impressive progress of monocular depth estimators has mirrored the growth in model capacity, from relatively modest CNNs to large Transformer architectures. Still, monocular depth estimators tend to struggle when presented with images with unfamiliar content and layout, since their knowledge of the visual world is restricted by the data seen during training, and challenged by zero-shot generalization to new domains. This motivates us to explore whether the extensive priors captured in recent generative diffusion models can enable better, more generalizable depth estimation. We introduce Marigold, a method for affine-invariant monocular depth estimation that is derived from Stable Diffusion and retains its rich prior knowledge. The estimator can be fine-tuned in a couple of days on a single GPU using only synthetic training data. It delivers state-of-the-art performance across a wide range of datasets, including over 20% performance gains in specific cases. Project page: https://marigoldmonodepth.github.io.


title: Learning Polynomial Problems with $SL(2,\mathbb{R})$ Equivariance

abstract: Optimizing and certifying the positivity of polynomials are fundamental primitives across mathematics and engineering applications, from dynamical systems to operations research. However, solving these problems in practice requires large semidefinite programs, with poor scaling in dimension and degree. In this work, we demonstrate for the first time that neural networks can effectively solve such problems in a data-driven fashion, achieving tenfold speedups while retaining high accuracy. Moreover, we observe that these polynomial learning problems are equivariant to the non-compact group $SL(2,\mathbb{R})$, which consists of area-preserving linear transformations. We therefore adapt our learning pipelines to accommodate this structure, including data augmentation, a new $SL(2,\mathbb{R})$-equivariant architecture, and an architecture equivariant with respect to its maximal compact subgroup, $SO(2, \mathbb{R})$. Surprisingly, the most successful approaches in practice do not enforce equivariance to the entire group, which we prove arises from an unusual lack of architecture universality for $SL(2,\mathbb{R})$ in particular. A consequence of this result, which is of independent interest, is that there exists an equivariant function for which there is no sequence of equivariant polynomials multiplied by arbitrary invariants that approximates the original function. This is a rare example of a symmetric problem where data augmentation outperforms a fully equivariant architecture, and provides interesting lessons in both theory and practice for other problems with non-compact symmetries.


title: Rejuvenating image-GPT as Strong Visual Representation Learners

abstract: This paper enhances image-GPT (iGPT), one of the pioneering works that introduce autoregressive pretraining to predict next pixels for visual representation learning. Two simple yet essential changes are made. First, we shift the prediction target from raw pixels to semantic tokens, enabling a higher-level understanding of visual content. Second, we supplement the autoregressive modeling by instructing the model to predict not only the next tokens but also the visible tokens. This pipeline is particularly effective when semantic tokens are encoded by discriminatively trained models, such as CLIP. We introduce this novel approach as D-iGPT. Extensive experiments showcase that D-iGPT excels as a strong learner of visual representations: A notable achievement of D-iGPT is its compelling performance on the ImageNet-1K dataset -- by training on publicly available datasets, D-iGPT achieves 89.5\% top-1 accuracy with a vanilla ViT-Large model. This model also shows strong generalization on the downstream task and robustness on out-of-distribution samples. Code is avaiable at \href{https://github.com/OliverRensu/D-iGPT}{https://github.com/OliverRensu/D-iGPT}.


title: A double copy from twisted (co)homology at genus one

abstract: We study the twisted (co)homology of a family of genus-one integrals -- the so called Riemann-Wirtinger integrals. These integrals are closely related to one-loop string amplitudes in chiral splitting where one leaves the loop-momentum, modulus and all but one puncture un-integrated. While not actual one-loop string integrals, they share many properties and are simple enough that the associated twisted (co)homologies have been completely characterized [Goto2022]. Using intersection numbers -- an inner product on the vector space of allowed differential forms -- we derive the Gauss-Manin connection for two bases of the twisted cohomology providing an independent check of [Mano&Watanabe2012]. We also use the intersection index -- an inner product on the vector space of allowed contours -- to derive a double-copy formula for complex genus-one integrals over a torus. Similar to the celebrated KLT formula between open- and closed-string tree-level amplitudes, the intersection indices form a genus-one KLT kernel defining bilinears in meromorphic Riemann-Wirtinger integrals that are equal to their complex counterparts.


title: Generative Powers of Ten

abstract: We present a method that uses a text-to-image model to generate consistent content across multiple image scales, enabling extreme semantic zooms into a scene, e.g., ranging from a wide-angle landscape view of a forest to a macro shot of an insect sitting on one of the tree branches. We achieve this through a joint multi-scale diffusion sampling approach that encourages consistency across different scales while preserving the integrity of each individual sampling process. Since each generated scale is guided by a different text prompt, our method enables deeper levels of zoom than traditional super-resolution methods that may struggle to create new contextual structure at vastly different scales. We compare our method qualitatively with alternative techniques in image super-resolution and outpainting, and show that our method is most effective at generating consistent multi-scale content.


title: Readout Guidance: Learning Control from Diffusion Features

abstract: We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity. Furthermore, by comparing the readout estimates to a user-defined target, and back-propagating the gradient through the readout head, these estimates can be used to guide the sampling process. Compared to prior methods for conditional generation, Readout Guidance requires significantly fewer added parameters and training samples, and offers a convenient and simple recipe for reproducing different forms of conditional control under a single framework, with a single architecture and sampling procedure. We showcase these benefits in the applications of drag-based manipulation, identity-consistent generation, and spatially aligned control. Project page: https://readout-guidance.github.io.


title: Guarding Barlow Twins Against Overfitting with Mixed Samples

abstract: Self-supervised Learning (SSL) aims to learn transferable feature representations for downstream applications without relying on labeled data. The Barlow Twins algorithm, renowned for its widespread adoption and straightforward implementation compared to its counterparts like contrastive learning methods, minimizes feature redundancy while maximizing invariance to common corruptions. Optimizing for the above objective forces the network to learn useful representations, while avoiding noisy or constant features, resulting in improved downstream task performance with limited adaptation. Despite Barlow Twins' proven effectiveness in pre-training, the underlying SSL objective can inadvertently cause feature overfitting due to the lack of strong interaction between the samples unlike the contrastive learning approaches. From our experiments, we observe that optimizing for the Barlow Twins objective doesn't necessarily guarantee sustained improvements in representation quality beyond a certain pre-training phase, and can potentially degrade downstream performance on some datasets. To address this challenge, we introduce Mixed Barlow Twins, which aims to improve sample interaction during Barlow Twins training via linearly interpolated samples. This results in an additional regularization term to the original Barlow Twins objective, assuming linear interpolation in the input space translates to linearly interpolated features in the feature space. Pre-training with this regularization effectively mitigates feature overfitting and further enhances the downstream performance on CIFAR-10, CIFAR-100, TinyImageNet, STL-10, and ImageNet datasets. The code and checkpoints are available at: https://github.com/wgcban/mix-bt.git


title: Steerers: A framework for rotation equivariant keypoint descriptors

abstract: Image keypoint descriptions that are discriminative and matchable over large changes in viewpoint are vital for 3D reconstruction. However, descriptions output by learned descriptors are typically not robust to camera rotation. While they can be made more robust by, e.g., data augmentation, this degrades performance on upright images. Another approach is test-time augmentation, which incurs a significant increase in runtime. Instead, we learn a linear transform in description space that encodes rotations of the input image. We call this linear transform a steerer since it allows us to transform the descriptions as if the image was rotated. From representation theory, we know all possible steerers for the rotation group. Steerers can be optimized (A) given a fixed descriptor, (B) jointly with a descriptor or (C) we can optimize a descriptor given a fixed steerer. We perform experiments in these three settings and obtain state-of-the-art results on the rotation invariant image matching benchmarks AIMS and Roto-360. We publish code and model weights at https://github.com/georg-bn/rotation-steerers.


title: Aligning and Prompting Everything All at Once for Universal Visual Perception

abstract: Vision foundation models have been explored recently to build general-purpose vision systems. However, predominant paradigms, driven by casting instance-level tasks as an object-word alignment, bring heavy cross-modality interaction, which is not effective in prompting object detection and visual grounding. Another line of work that focuses on pixel-level tasks often encounters a large annotation gap of things and stuff, and suffers from mutual interference between foreground-object and background-class segmentation. In stark contrast to the prevailing methods, we present APE, a universal visual perception model for aligning and prompting everything all at once in an image to perform diverse tasks, i.e., detection, segmentation, and grounding, as an instance-level sentence-object matching paradigm. Specifically, APE advances the convergence of detection and grounding by reformulating language-guided grounding as open-vocabulary detection, which efficiently scales up model prompting to thousands of category vocabularies and region descriptions while maintaining the effectiveness of cross-modality fusion. To bridge the granularity gap of different pixel-level tasks, APE equalizes semantic and panoptic segmentation to proxy instance learning by considering any isolated regions as individual instances. APE aligns vision and language representation on broad data with natural and challenging characteristics all at once without task-specific fine-tuning. The extensive experiments on over 160 datasets demonstrate that, with only one-suit of weights, APE outperforms (or is on par with) the state-of-the-art models, proving that an effective yet universal perception for anything aligning and prompting is indeed feasible. Codes and trained models are released at https://github.com/shenyunhang/APE.


title: Momentum-space formulae for AdS correlators for diverse theories in diverse dimensions

abstract: In this paper, we explore correlators of a series of theories in anti-de Sitter space: we present comprehensive results for interactions involving scalars, gluons, and gravitons in multiple dimensions. One aspect of our investigation is the establishment of an intriguing connection between the kinematic factors of these theories; indeed, such a connection directly relates these theories among themselves and with other theories of higher spin fields. Besides providing several explicit results throughout the paper, we also highlight the interconnections and relationships between these different theories, providing valuable insights into their similarities and distinctions.


title: GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis

abstract: We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner. The proposed method enables 2K-resolution rendering under a sparse-view camera setting. Unlike the original Gaussian Splatting or neural implicit rendering methods that necessitate per-subject optimizations, we introduce Gaussian parameter maps defined on the source views and regress directly Gaussian Splatting properties for instant novel view synthesis without any fine-tuning or optimization. To this end, we train our Gaussian parameter regression module on a large amount of human scan data, jointly with a depth estimation module to lift 2D parameter maps to 3D space. The proposed framework is fully differentiable and experiments on several datasets demonstrate that our method outperforms state-of-the-art methods while achieving an exceeding rendering speed.


title: Latent Feature-Guided Diffusion Models for Shadow Removal

abstract: Recovering textures under shadows has remained a challenging problem due to the difficulty of inferring shadow-free scenes from shadow images. In this paper, we propose the use of diffusion models as they offer a promising approach to gradually refine the details of shadow regions during the diffusion process. Our method improves this process by conditioning on a learned latent feature space that inherits the characteristics of shadow-free images, thus avoiding the limitation of conventional methods that condition on degraded images only. Additionally, we propose to alleviate potential local optima during training by fusing noise features with the diffusion network. We demonstrate the effectiveness of our approach which outperforms the previous best method by 13% in terms of RMSE on the AISTD dataset. Further, we explore instance-level shadow removal, where our model outperforms the previous best method by 82% in terms of RMSE on the DESOBA dataset.


title: Mesh-Guided Neural Implicit Field Editing

abstract: Neural implicit fields have emerged as a powerful 3D representation for reconstructing and rendering photo-realistic views, yet they possess limited editability. Conversely, explicit 3D representations, such as polygonal meshes, offer ease of editing but may not be as suitable for rendering high-quality novel views. To harness the strengths of both representations, we propose a new approach that employs a mesh as a guiding mechanism in editing the neural radiance field. We first introduce a differentiable method using marching tetrahedra for polygonal mesh extraction from the neural implicit field and then design a differentiable color extractor to assign colors obtained from the volume renderings to this extracted mesh. This differentiable colored mesh allows gradient back-propagation from the explicit mesh to the implicit fields, empowering users to easily manipulate the geometry and color of neural implicit fields. To enhance user control from coarse-grained to fine-grained levels, we introduce an octree-based structure into its optimization. This structure prioritizes the edited regions and the surface part, making our method achieve fine-grained edits to the neural implicit field and accommodate various user modifications, including object additions, component removals, specific area deformations, and adjustments to local and global colors. Through extensive experiments involving diverse scenes and editing operations, we have demonstrated the capabilities and effectiveness of our method. Our project page is: \url{https://cassiepython.github.io/MNeuEdit/}


title: PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness

abstract: We propose the task of Panoptic Scene Completion (PSC) which extends the recently popular Semantic Scene Completion (SSC) task with instance-level information to produce a richer understanding of the 3D scene. Our PSC proposal utilizes a hybrid mask-based technique on the non-empty voxels from sparse multi-scale completions. Whereas the SSC literature overlooks uncertainty which is critical for robotics applications, we instead propose an efficient ensembling to estimate both voxel-wise and instance-wise uncertainties along PSC. This is achieved by building on a multi-input multi-output (MIMO) strategy, while improving performance and yielding better uncertainty for little additional compute. Additionally, we introduce a technique to aggregate permutation-invariant mask predictions. Our experiments demonstrate that our method surpasses all baselines in both Panoptic Scene Completion and uncertainty estimation on three large-scale autonomous driving datasets. Our code and data are available at https://astra-vision.github.io/PaSCo .


title: Spectral Temporal Graph Neural Network for massive MIMO CSI Prediction

abstract: In the realm of 5G communication systems, the accuracy of Channel State Information (CSI) prediction is vital for optimizing performance. This letter introduces a pioneering approach: the Spectral-Temporal Graph Neural Network (STEM GNN), which fuses spatial relationships and temporal dynamics of the wireless channel using the Graph Fourier Transform. We compare the STEM GNN approach with conventional Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) models for CSI prediction. Our findings reveal a significant enhancement in overall communication system performance through STEM GNNs. For instance, in one scenario, STEM GNN achieves a sum rate of 5.009 bps/Hz which is $11.9\%$ higher than that of LSTM and $35\%$ higher than that of RNN. The spectral-temporal analysis capabilities of STEM GNNs capture intricate patterns often overlooked by traditional models, offering improvements in beamforming, interference mitigation, and ultra-reliable low-latency communication (URLLC).


title: Coding for the unsourced A-channel with erasures: the linked loop code

abstract: The A-channel is a noiseless multiple access channel in which users simultaneously transmit Q-ary symbols and the receiver observes the set of transmitted symbols, but not their multiplicities. An A-channel is said to be unsourced if, additionally, users transmissions are encoded across time using a common codebook and decoding of the transmitted messages is done without regard to the identities of the active users. An interesting variant of the unsourced A-channel is the unsourced A-channel with erasures (UACE), in which transmitted symbols are erased with a given independent and identically distributed probability. In this paper, we focus on designing a code that enables a list of transmitted codewords to be recovered despite the erasures of some of the transmitted symbols. To this end, we propose the linked-loop code (LLC), which uses parity bits to link each symbol to the previous M symbols in a tail-biting manner, i.e., the first symbols of the transmission are linked to the last ones. The decoding process occurs in two phases: the first phase decodes the codewords that do not suffer from any erasures, and the second phase attempts to recover the erased symbols using the available parities. We compare the performance of the LLC over the UACE with other codes in the literature and argue for the effectiveness of the construction. Our motivation for studying the UACE comes from its relevance in machine-type communication and coded compressed sensing.


title: Efficient LDPC Decoding using Physical Computation

abstract: Due to 5G deployment, there is significant interest in LDPC decoding. While much research is devoted on efficient hardwiring of algorithms based on Belief Propagation (BP), it has been shown that LDPC decoding can be formulated as a combinatorial optimization problem, which could benefit from significant acceleration of physical computation mechanisms such as Ising machines. This approach has so far resulted in poor performance. This paper shows that the reason is not fundamental but suboptimal hardware and formulation. A co-designed Ising machine-based system can improve speed by 3 orders of magnitude. As a result, a physical computation approach can outperform hardwiring state-of-the-art algorithms. In this paper, we show such an augmented Ising machine that is 4.4$\times$ more energy efficient than the state of the art in the literature.


title: Formulas for 2-D Beltrami Operators

abstract: In this article we consider 2-dimensional surfaces. We define some new operators which enable us to evaluate quantities of the surface, such invariants, in a more systematic way.


title: Cooperation Based Joint Active and Passive Sensing with Asynchronous Transceivers for Perceptive Mobile Networks

abstract: Perceptive mobile network (PMN) is an emerging concept for next-generation wireless networks capable of conducting integrated sensing and communication (ISAC). A major challenge for realizing high performance sensing in PMNs is how to deal with spatially separated asynchronous transceivers. Asynchronicity results in timing offsets (TOs) and carrier frequency offsets (CFOs), which further cause ambiguity in ranging and velocity sensing. Most existing algorithms mitigate TOs and CFOs based on the line-of-sight (LOS) propagation path between sensing transceivers. However, LOS paths may not exist in realistic scenarios. In this paper, we propose a cooperation based joint active and passive sensing scheme for the non-LOS (NLOS) scenarios having asynchronous transceivers. This scheme relies on the cross-correlation cooperative sensing (CCCS) algorithm, which regards active sensing as a reference and mitigates TOs and CFOs by correlating active and passive sensing information. Another major challenge for realizing high performance sensing in PMNs is how to realize high accuracy angle-of-arrival (AoA) estimation with low complexity. Correspondingly, we propose a low complexity AoA algorithm based on cooperative sensing, which comprises coarse AoA estimation and fine AoA estimation. Analytical and numerical simulation results verify the performance advantages of the proposed CCCS algorithm and the low complexity AoA estimation algorithm.


title: Driver Safety Reward with Cooperative Platooning using Blockchain

abstract: Cooperative driving (or Platooning) focuses on improving the safety and efficiency by connecting two or more vehicles on a road by vehicular communication protocols. The leader is crucial as it manages the platoon, establishes communication between cars, and perform platoon maneuvers. In this paper, we proposed a driver incentive model which encourages platooning on roads leading to driver safety. As, the leader of platoon have multiple responsibilities than followers, our model rewards more incentives to leader than followers. These incentives will be rewarded as crypto tokens. This digital monetization method for both leaders and followers of a platoon is accomplished by secure transactions using blockchain.


title: Teaching mathematical modeling for sustainability: Enhancing interdisciplinary skills in students

abstract: We developed a pilot course focused on mathematical modeling within the tertiary education framework, with a distinct emphasis on sustainability and sustainable development. While an applicable textbook exists for this liberal arts course, it is noticeable that numerous examples within it are not directly aligned with sustainability concerns. To address this gap, our study strategically integrated the teaching and learning of modeling by carefully selecting classroom examples that closely align with the context of sustainability. By employing an innovative and adaptable approach to the course content delivery, we fostered interdisciplinary collaboration among students, improved their comprehension, and enhanced their interdisciplinary skills.


title: A nonlinear population model

abstract: This paper considers a nonlinear model for population dynamics with age structure. The fertility rate with respect to age is non constant and has the form proposed by [17]. Moreover, its multiplicative structure and the multiplicative structure of mortality makes the model separable. In this setting it is shown that the number of births in unit time is given by a system of nonlinear ordinary differential equations. The steady state solution together with the equilibrium solution is found explicitly.


title: Uncertainty Quantification in Machine Learning Based Segmentation: A Post-Hoc Approach for Left Ventricle Volume Estimation in MRI

abstract: Recent studies have confirmed cardiovascular diseases remain responsible for highest death toll amongst non-communicable diseases. Accurate left ventricular (LV) volume estimation is critical for valid diagnosis and management of various cardiovascular conditions, but poses significant challenge due to inherent uncertainties associated with segmentation algorithms in magnetic resonance imaging (MRI). Recent machine learning advancements, particularly U-Net-like convolutional networks, have facilitated automated segmentation for medical images, but struggles under certain pathologies and/or different scanner vendors and imaging protocols. This study proposes a novel methodology for post-hoc uncertainty estimation in LV volume prediction using It\^{o} stochastic differential equations (SDEs) to model path-wise behavior for the prediction error. The model describes the area of the left ventricle along the heart's long axis. The method is agnostic to the underlying segmentation algorithm, facilitating its use with various existing and future segmentation technologies. The proposed approach provides a mechanism for quantifying uncertainty, enabling medical professionals to intervene for unreliable predictions. This is of utmost importance in critical applications such as medical diagnosis, where prediction accuracy and reliability can directly impact patient outcomes. The method is also robust to dataset changes, enabling application for medical centers with limited access to labeled data. Our findings highlight the proposed uncertainty estimation methodology's potential to enhance automated segmentation robustness and generalizability, paving the way for more reliable and accurate LV volume estimation in clinical settings as well as opening new avenues for uncertainty quantification in biomedical image segmentation, providing promising directions for future research.


title: The SVHN Dataset Is Deceptive for Probabilistic Generative Models Due to a Distribution Mismatch

abstract: The Street View House Numbers (SVHN) dataset is a popular benchmark dataset in deep learning. Originally designed for digit classification tasks, the SVHN dataset has been widely used as a benchmark for various other tasks including generative modeling. However, with this work, we aim to warn the community about an issue of the SVHN dataset as a benchmark for generative modeling tasks: we discover that the official split into training set and test set of the SVHN dataset are not drawn from the same distribution. We empirically show that this distribution mismatch has little impact on the classification task (which may explain why this issue has not been detected before), but it severely affects the evaluation of probabilistic generative models, such as Variational Autoencoders and diffusion models. As a workaround, we propose to mix and re-split the official training and test set when SVHN is used for tasks other than classification. We publish a new split and the indices we used to create it at https://jzenn.github.io/svhn-remix/ .


title: Pura Vida Neutrosophic Algebra

abstract: We introduce Pura Vida Neutrosophic Algebra, an algebraic structure consisting of neutrosophic numbers equipped with two binary operations namely addition and multiplication. The addition can be calculated sometimes with the function min and other times with the max function. The multiplication operation is the usual sum between numbers. Pura Vida Neutrosophic Algebra is an extension of both Tropical Algebra (also known as Min-Plus, or Min-Algebra) and Max-Plus Algebra (also known as Max-algebra). Tropical and Max-Plus algebras are algebraic structures included in semirings and their operations can be used in matrices and vectors. Pura Vida Neutrosophic Algebra is included in Neutrosophic semirings and can be used in Neutrosophic matrices and vectors


title: A 5G DMRS-based Signal for Integrated Sensing and Communication System

abstract: Integrated sensing and communication (ISAC) is considered as the potential key technology of the future mobile communication systems. The signal design is fundamental for the ISAC system. The reference signals in mobile communication systems have good detection performance, which is worth further research. Existing studies applied the single reference signal to radar sensing. In this paper, a multiple reference signals collaborative sensing scheme is designed. Specifically, we jointly apply channel state information reference signal (CSI-RS), positioning reference signal (PRS) and demodulation reference signal (DMRS) in radar sensing, which improve the performance of radar sensing via obtaining continuous time-frequency resource mapping. Cr\'amer-Rao lower bound (CRLB) of the joint reference signal for distance and velocity estimation is derived. The impacts of carrier frequency and subcarrier spacing on the performance of distance and velocity estimation are revealed. The results of simulation experiments show that compared with the single reference signal sensing scheme, the multiple reference signals collaborative sensing scheme effectively improves the sensing accuracy. Moreover, because of the discontinuous OFDM symbols, the accuracy of velocity estimation could be further improved via compressed sensing (CS). This paper has verified that multiple reference signals, instead of single reference signal, have much more superior performance on radar sensing, which is a practical and efficient approach in designing ISAC signal.


title: LpiCT: A logic security analysis framework for protocols

abstract: The pi calculus is a basic theory of mobile communication based on the notion of interaction, which, aimed at analyzing and modelling the behaviors of communication process in communicating and mobile systems, is widely applied to the security analysis of cryptographic protocol's design and implementation. But the pi calculus does not provide perfect logic security analysis, so the logic flaws in the design and the implementation of a cryptographic protocol can not be discovered in time. The aim is to analyze whether there are logic flaws in the design and the implementation of a cryptographic protocol, so as to ensure the security of the cryptographic protocol when it is encoded into a software and implemented. This paper introduces logic rules and proofs, binary tree and the KMP algorithm, and proposes a new extension the pi calculus theory, a logic security analysis framework and an algorithm. This paper presents the logic security proof and analysis of TLS1.3 protocol's interactional implementation process. Empirical results show that the new extension theory, the logic security analysis framework and the algorithm can effectively analyze whether there are logic flaws in the design and the implementation of a cryptographic protocol. The security of cryptographic protocols depends not only on cryptographic primitives, but also on the coding of cryptographic protocols and the environment in which they are implemented. The security analysis framework of cryptographic protocol implementation proposed in this paper can ensure the security of protocol implementation.


title: Mercury: A modeling, simulation, and optimization framework for data stream-oriented IoT applications

abstract: The Internet of Things is transforming our society by monitoring users and infrastructures' behavior to enable new services that will improve life quality and resource management. These applications require a vast amount of localized information to be processed in real-time so, the deployment of new fog computing infrastructures that bring computing closer to the data sources is a major concern. In this context, we present Mercury, a Modeling, Simulation, and Optimization (M&S&O) framework to analyze the dimensioning and the dynamic operation of real-time fog computing scenarios. Our research proposes a location-aware solution that supports data stream analytics applications including FaaS-based computation offloading. Mercury implements a detailed structural and behavioral simulation model, providing fine-grained simulation outputs, and is described using the Discrete Event System Specification (DEVS) mathematical formalism, helping to validate the model's implementation. Finally, we present a case study using real traces from a driver assistance scenario, offering a detailed comparison with other state-of-the-art simulators.


title: TailorMe: Self-Supervised Learning of an Anatomically Constrained Volumetric Human Shape Model

abstract: Human shape spaces have been extensively studied, as they are a core element of human shape and pose inference tasks. Classic methods for creating a human shape model register a surface template mesh to a database of 3D scans and use dimensionality reduction techniques, such as Principal Component Analysis, to learn a compact representation. While these shape models enable global shape modifications by correlating anthropometric measurements with the learned subspace, they only provide limited localized shape control. We instead register a volumetric anatomical template, consisting of skeleton bones and soft tissue, to the surface scans of the CAESAR database. We further enlarge our training data to the full Cartesian product of all skeletons and all soft tissues using physically plausible volumetric deformation transfer. This data is then used to learn an anatomically constrained volumetric human shape model in a self-supervised fashion. The resulting TailorMe model enables shape sampling, localized shape manipulation, and fast inference from given surface scans.


title: Insolvability of $x^x = a$ in elementary functions

abstract: In a recent paper "Solvability of equations in elementary functions" [arXiv:1911.10409] the insolvability in elementary functions of equation $\tan(x) - x = a$ was proved. This work applies the same topological method to prove the insolvability of equation $x^x = a$.


title: Wavefront Transformation-based Near-field Channel Prediction for Extremely Large Antenna Array with Mobility

abstract: This paper addresses the mobility problem in extremely large antenna array (ELAA) communication systems. In order to account for the performance loss caused by the spherical wavefront of ELAA in the mobility scenario, we propose a wavefront transformation-based matrix pencil (WTMP) channel prediction method. In particular, we design a matrix to transform the spherical wavefront into a new wavefront, which is closer to the plane wave. We also design a time-frequency projection matrix to capture the time-varying path delay due to user movement. Furthermore, we adopt the matrix pencil (MP) method to estimate channel parameters. Our proposed WTMP method can mitigate the effect of near-field radiation when predicting future channels. Theoretical analysis shows that the designed matrix is asymptotically determined by the distance between the base station (BS) antenna array and the scatterers or the user when the number of BS antennas is large enough. For an ELAA communication system in the mobility scenario, we prove that the prediction error converges to zero with the increasing number of BS antennas. Simulation results demonstrate that our designed transform matrix efficiently mitigates the near-field effect, and that our proposed WTMP method can overcome the ELAA mobility challenge and approach the performance in stationary setting.


title: Channel Scheduling for IoT Access with Spatial Correlation

abstract: Spatially correlated device activation is a typical feature of the Internet of Things (IoT). This motivates the development of channel scheduling (CS) methods that mitigate device collisions efficiently in such scenarios, which constitutes the scope of this work. Specifically, we present a quadratic program (QP) formulation for the CS problem considering the joint activation probabilities among devices. This formulation allows the devices to stochastically select the transmit channels, thus, leading to a soft-clustering approach. We prove that the optimal QP solution can only be attained when it is transformed into a hard-clustering problem, leading to a pure integer QP, which we transform into a pure integer linear program (PILP). We leverage the branch-and-cut (B&C) algorithm to solve PILP optimally. Due to the high computational cost of B&C, we resort to some sub-optimal clustering methods with low computational costs to tackle the clustering problem in CS. Our findings demonstrate that the CS strategy, sourced from B&C, significantly outperforms those derived from sub-optimal clustering methods, even amidst increased device correlation.


title: Entropy generating function for past lifetime and its properties

abstract: The past entropy is considered as an uncertainty measure for the past lifetime distribution. Generating function approach to entropy become popular in recent time as it generate several well-known entropy measures. In this paper, we introduce the past entropy-generating function. We study certain properties of this measure. It is shown that the past entropy-generating function uniquely determines the distribution. Further, we present characterizations for some lifetime models using the relationship between reliability concepts and the past entropy-generating function.


title: Hierarchical ML Codebook Design for Extreme MIMO Beam Management

abstract: Beam management is a strategy to unify beamforming and channel state information (CSI) acquisition with large antenna arrays in 5G. Codebooks serve multiple uses in beam management including beamforming reference signals, CSI reporting, and analog beam training. In this paper, we propose and evaluate a machine learning-refined codebook design process for extremely large multiple-input multiple-output (X-MIMO) systems. We propose a neural network and beam selection strategy to design the initial access and refinement codebooks using end-to-end learning from beamspace representations. The algorithm, called Extreme-Beam Management (X-BM), can significantly improve the performance of extremely large arrays as envisioned for 6G and capture realistic wireless and physical layer aspects. Our results show an 8dB improvement in initial access and overall effective spectral efficiency improvements compared to traditional codebook methods.

categories: eess.SP cs.IT cs.LG cs.SY eess.SY math.IT


title: Training Chain-of-Thought via Latent-Variable Inference

abstract: Large language models (LLMs) solve problems more accurately and interpretably when instructed to work out the answer step by step using a ``chain-of-thought'' (CoT) prompt. One can also improve LLMs' performance on a specific task by supervised fine-tuning, i.e., by using gradient ascent on some tunable parameters to maximize the average log-likelihood of correct answers from a labeled training set. Naively combining CoT with supervised tuning requires supervision not just of the correct answers, but also of detailed rationales that lead to those answers; these rationales are expensive to produce by hand. Instead, we propose a fine-tuning strategy that tries to maximize the \emph{marginal} log-likelihood of generating a correct answer using CoT prompting, approximately averaging over all possible rationales. The core challenge is sampling from the posterior over rationales conditioned on the correct answer; we address it using a simple Markov-chain Monte Carlo (MCMC) expectation-maximization (EM) algorithm inspired by the self-taught reasoner (STaR), memoized wake-sleep, Markovian score climbing, and persistent contrastive divergence. This algorithm also admits a novel control-variate technique that drives the variance of our gradient estimates to zero as the model improves. Applying our technique to GSM8K and the tasks in BIG-Bench Hard, we find that this MCMC-EM fine-tuning technique typically improves the model's accuracy on held-out examples more than STaR or prompt-tuning with or without CoT.


title: The Role of Low-energy (< 20 eV) Secondary Electrons in the Extraterrestrial Synthesis of Prebiotic Molecules

abstract: We demonstrate for the first time that Galactic cosmic rays with energies as high as 1e10 eV can trigger a cascade of low-energy (< 20 eV) secondary electrons that could be a significant contributor to the interstellar synthesis of prebiotic molecules whose delivery by comets, meteorites, and interplanetary dust particles may have kick-started life on Earth. We explore the relative importance of low-energy (< 20 eV) secondary electrons--agents of radiation chemistry--and low-energy (< 10 eV), non-ionizing photons--instigators of photochemistry. Our calculations indicate fluxes of 100 electrons/cm2/s for low-energy secondary electrons produced within interstellar ices due to incident attenuated Galactic cosmic-ray (CR) protons. Consequently, in certain star-forming regions where internal high-energy radiation sources produce ionization rates that are observed to be a thousand times greater than the typical interstellar Galactic ionization rate, the flux of low-energy secondary electrons should far exceed that of non-ionizing photons. Because reaction cross-sections can be several orders of magnitude larger for electrons than for photons, even in the absence of such enhancement our calculations indicate that secondary low-energy electrons are at least as significant as low-energy (< 10 eV) non-ionizing photons in the interstellar synthesis of prebiotic molecules. Most importantly, our results demonstrate the pressing need for explicitly incorporating low-energy electrons in current and future astrochemical simulations of cosmic ices. Such models are critically important for interpreting James Webb Space Telescope infrared measurements, which are currently being used to probe the origins of life by studying complex organic molecules found in ices near star-forming regions.


title: How Generative-AI can be Effectively used in Government Chatbots

abstract: With the rapid development of artificial intelligence and breakthroughs in machine learning and natural language processing, intelligent question-answering robots have become widely used in government affairs. This paper conducts a horizontal comparison between Guangdong Province's government chatbots, ChatGPT, and Wenxin Ernie, two large language models, to analyze the strengths and weaknesses of existing government chatbots and AIGC technology. The study finds significant differences between government chatbots and large language models. China's government chatbots are still in an exploratory stage and have a gap to close to achieve "intelligence." To explore the future direction of government chatbots more deeply, this research proposes targeted optimization paths to help generative AI be effectively applied in government chatbot conversations.


title: Adam-like Algorithm with Smooth Clipping Attains Global Minima: Analysis Based on Ergodicity of Functional SDEs

abstract: In this paper, we prove that an Adam-type algorithm with smooth clipping approaches the global minimizer of the regularized non-convex loss function. Adding smooth clipping and taking the state space as the set of all trajectories, we can apply the ergodic theory of Markov semigroups for this algorithm and investigate its asymptotic behavior. The ergodic theory we establish in this paper reduces the problem of evaluating the convergence, generalization error and discretization error of this algorithm to the problem of evaluating the difference between two functional stochastic differential equations (SDEs) with different drift coefficients. As a result of our analysis, we have shown that this algorithm minimizes the the regularized non-convex loss function with errors of the form $n^{-1/2}$, $\eta^{1/4}$, $\beta^{-1} \log (\beta + 1)$ and $e^{- c t}$. Here, $c$ is a constant and $n$, $\eta$, $\beta$ and $t$ denote the size of the training dataset, learning rate, inverse temperature and time, respectively.


title: Study of cosmogenic activation above ground of Ar for DarkSide-20k

abstract: The production of long-lived radioactive isotopes due to the exposure to cosmic rays on the Earth's surface is an hazard for experiments searching for rare events like the direct detection of galactic dark matter particles. The use of large amounts of liquid Argon is foreseen in different projects, like the DarkSide-20k experiment, intended to look for Weakly Interacting Massive Particles at the Laboratori Nazionali del Gran Sasso. Here, results from the study of the cosmogenic activation of Argon carried out in the context of DarkSide-20k are presented. The induced activity of several isotopes, including 39Ar, and the expected counting rates in the detector have been deduced, considering exposure conditions as realistic as possible.


title: Channel-Feedback-Free Transmission for Downlink FD-RAN: A Radio Map based Complex-valued Precoding Network Approach

abstract: As the demand for high-quality services proliferates, an innovative network architecture, the fully-decoupled RAN (FD-RAN), has emerged for more flexible spectrum resource utilization and lower network costs. However, with the decoupling of uplink base stations and downlink base stations in FD-RAN, the traditional transmission mechanism, which relies on real-time channel feedback, is not suitable as the receiver is not able to feedback accurate and timely channel state information to the transmitter. This paper proposes a novel transmission scheme without relying on physical layer channel feedback. Specifically, we design a radio map based complex-valued precoding network~(RMCPNet) model, which outputs the base station precoding based on user location. RMCPNet comprises multiple subnets, with each subnet responsible for extracting unique modal features from diverse input modalities. Furthermore, the multi-modal embeddings derived from these distinct subnets are integrated within the information fusion layer, culminating in a unified representation. We also develop a specific RMCPNet training algorithm that employs the negative spectral efficiency as the loss function. We evaluate the performance of the proposed scheme on the public DeepMIMO dataset and show that RMCPNet can achieve 16\% and 76\% performance improvements over the conventional real-valued neural network and statistical codebook approach, respectively.


title: Virtual Fusion with Contrastive Learning for Single Sensor-based Activity Recognition

abstract: Various types of sensors can be used for Human Activity Recognition (HAR), and each of them has different strengths and weaknesses. Sometimes a single sensor cannot fully observe the user's motions from its perspective, which causes wrong predictions. While sensor fusion provides more information for HAR, it comes with many inherent drawbacks like user privacy and acceptance, costly set-up, operation, and maintenance. To deal with this problem, we propose Virtual Fusion - a new method that takes advantage of unlabeled data from multiple time-synchronized sensors during training, but only needs one sensor for inference. Contrastive learning is adopted to exploit the correlation among sensors. Virtual Fusion gives significantly better accuracy than training with the same single sensor, and in some cases, it even surpasses actual fusion using multiple sensors at test time. We also extend this method to a more general version called Actual Fusion within Virtual Fusion (AFVF), which uses a subset of training sensors during inference. Our method achieves state-of-the-art accuracy and F1-score on UCI-HAR and PAMAP2 benchmark datasets. Implementation is available upon request.


title: Identifying Spurious Correlations using Counterfactual Alignment

abstract: Models driven by spurious correlations often yield poor generalization performance. We propose the counterfactual alignment method to detect and explore spurious correlations of black box classifiers. Counterfactual images generated with respect to one classifier can be input into other classifiers to see if they also induce changes in the outputs of these classifiers. The relationship between these responses can be quantified and used to identify specific instances where a spurious correlation exists as well as compute aggregate statistics over a dataset. Our work demonstrates the ability to detect spurious correlations in face attribute classifiers. This is validated by observing intuitive trends in a face attribute classifier as well as fabricating spurious correlations and detecting their presence, both visually and quantitatively. Further, utilizing the CF alignment method, we demonstrate that we can rectify spurious correlations identified in classifiers.


title: Timing decisions as the next frontier for collective intelligence

abstract: The past decade has witnessed a dramatically growing interest in collective intelligence - the phenomenon of groups having an ability to make more accurate decisions than isolated individuals. However, the vast majority of studies to date have focused, either explicitly or implicitly, on spatial decisions (e.g., potential nest sites, food patches, or migration directions). We highlight the equally important, but severely understudied, realm of temporal collective decision-making, i.e., decisions about when to perform an action. We argue that temporal collective decision making is likely to differ from spatial decision making in several crucial ways and probably involves different mechanisms, model predictions, and experimental outcomes. We anticipate that research focused on temporal decisions should lead to a radically expanded understanding of the adaptiveness and constraints of living in groups.


title: Video Summarization: Towards Entity-Aware Captions

abstract: Existing popular video captioning benchmarks and models deal with generic captions devoid of specific person, place or organization named entities. In contrast, news videos present a challenging setting where the caption requires such named entities for meaningful summarization. As such, we propose the task of summarizing news video directly to entity-aware captions. We also release a large-scale dataset, VIEWS (VIdeo NEWS), to support research on this task. Further, we propose a method that augments visual information from videos with context retrieved from external world knowledge to generate entity-aware captions. We demonstrate the effectiveness of our approach on three video captioning models. We also show that our approach generalizes to existing news image captions dataset. With all the extensive experiments and insights, we believe we establish a solid basis for future research on this challenging task.


title: StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D

abstract: In the realm of text-to-3D generation, utilizing 2D diffusion models through score distillation sampling (SDS) frequently leads to issues such as blurred appearances and multi-faced geometry, primarily due to the intrinsically noisy nature of the SDS loss. Our analysis identifies the core of these challenges as the interaction among noise levels in the 2D diffusion process, the architecture of the diffusion network, and the 3D model representation. To overcome these limitations, we present StableDreamer, a methodology incorporating three advances. First, inspired by InstructNeRF2NeRF, we formalize the equivalence of the SDS generative prior and a simple supervised L2 reconstruction loss. This finding provides a novel tool to debug SDS, which we use to show the impact of time-annealing noise levels on reducing multi-faced geometries. Second, our analysis shows that while image-space diffusion contributes to geometric precision, latent-space diffusion is crucial for vivid color rendition. Based on this observation, StableDreamer introduces a two-stage training strategy that effectively combines these aspects, resulting in high-fidelity 3D models. Third, we adopt an anisotropic 3D Gaussians representation, replacing Neural Radiance Fields (NeRFs), to enhance the overall quality, reduce memory usage during training, and accelerate rendering speeds, and better capture semi-transparent objects. StableDreamer reduces multi-face geometries, generates fine details, and converges stably.


title: Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D

abstract: Diffusion Handles is a novel approach to enabling 3D object edits on diffusion images. We accomplish these edits using existing pre-trained diffusion models, and 2D image depth estimation, without any fine-tuning or 3D object retrieval. The edited results remain plausible, photo-real, and preserve object identity. Diffusion Handles address a critically missing facet of generative image based creative design, and significantly advance the state-of-the-art in generative image editing. Our key insight is to lift diffusion activations for an object to 3D using a proxy depth, 3D-transform the depth and associated activations, and project them back to image space. The diffusion process applied to the manipulated activations with identity control, produces plausible edited images showing complex 3D occlusion and lighting effects. We evaluate Diffusion Handles: quantitatively, on a large synthetic data benchmark; and qualitatively by a user study, showing our output to be more plausible, and better than prior art at both, 3D editing and identity control. Project Webpage: https://diffusionhandles.github.io/


title: Prompt Tuning for Zero-shot Compositional Learning

abstract: Open World Compositional Zero-Shot Learning (OW-CZSL) is known to be an extremely challenging task, which aims to recognize unseen compositions formed from seen attributes and objects without any prior assumption of the output space. In order to achieve this goal, a model has to be "smart" and "knowledgeable". To be smart, a model should be good at reasoning the interactions between attributes and objects from the seen compositions. While "knowledgeable" means the model owns "common sense" to the open world that can "foresee" some features of the unseen compositions. Most previous work focuses on the "smart" part, while few of them provided an effective solution to achieve the "knowledgeable" goal. In this paper, we proposed a framework named Multi-Modal Prompt Tuning (MMPT) to inherit the "knowledgeable" property from the large pre-trained vision-language model. Extensive experiments show that our proposed MMPT obtains new state-of-the-art results in OW-CZSL task. On the UT-Zappos dataset, MMPT pushes the AUC score to $29.8$, while the previous best score is $26.5$. On the more challenging MIT-States dataset, the AUC score of MMPT is 1.5 times better than the current state-of-the-art.


title: DiverseDream: Diverse Text-to-3D Synthesis with Augmented Text Embedding

abstract: Text-to-3D synthesis has recently emerged as a new approach to sampling 3D models by adopting pretrained text-to-image models as guiding visual priors. An intriguing but underexplored problem with existing text-to-3D methods is that 3D models obtained from the sampling-by-optimization procedure tend to have mode collapses, and hence poor diversity in their results. In this paper, we provide an analysis and identify potential causes of such a limited diversity, and then devise a new method that considers the joint generation of different 3D models from the same text prompt, where we propose to use augmented text prompts via textual inversion of reference images to diversify the joint generation. We show that our method leads to improved diversity in text-to-3D synthesis qualitatively and quantitatively.