title: PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own
    Deep Neural Net At Inference
summary: We show that Large Language Model from Power Law Decoder Representations
  (PLDR-LLM) is a foundational model whose deductive outputs are invariant
  tensors up to a small perturbation. PLDR-LLM learns a singularity condition for
  the deductive outputs that enable the once-inferred energy-curvature tensor
  G_{LM} to replace the deep neural network of power law graph
  attention (PLGA) generating the deductive outputs at inference. We demonstrate
  that a cache for G_{LM} (G-cache) and KV-cache can be implemented in
  a straightforward manner to improve the inference time. The invariance and
  generalizable nature of deductive outputs is at a very high fidelity where
  deductive outputs have same RMSE and determinant values up to 15 decimal places
  after caching, and zero-shot benchmark scores remain unchanged. Ablation
  studies show that learned deductive outputs have distinct loss and accuracy
  characteristics from models pretrained with transferred, randomly initialized
  or identity tensors as a constant tensor operator and an LLM with scaled-dot
  product attention (SDPA) is a special case of PLDR-LLM where G_{LM}
  is predefined as identity. The observed invariance characteristic introduces a
  novel asymmetry between training and inference phases with caching. We outline
  observed common characteristics of the deductive outputs for the learned
  singularity condition. We provide an implementation of a training and inference
  framework for PLDR-LLM with KV-cache and G-cache.
paper_authors._id: 67bd1db005a599263a2a684e
paper_authors.name: Burc Gokden
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-25T01:32:33.775Z
paper_authors._id: 671ddb3bf89c9b8208568e73
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-Q77RLuIvzpVU95WGdM7u.png
paper_authors.fullname: Burc Gokden
paper_authors.type: user
paper_authors.user: fromthesky
paper_title: PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own
    Deep Neural Net At Inference
paper_summary: We show that Large Language Model from Power Law Decoder Representations
  (PLDR-LLM) is a foundational model whose deductive outputs are invariant
  tensors up to a small perturbation. PLDR-LLM learns a singularity condition for
  the deductive outputs that enable the once-inferred energy-curvature tensor
  G_{LM} to replace the deep neural network of power law graph
  attention (PLGA) generating the deductive outputs at inference. We demonstrate
  that a cache for G_{LM} (G-cache) and KV-cache can be implemented in
  a straightforward manner to improve the inference time. The invariance and
  generalizable nature of deductive outputs is at a very high fidelity where
  deductive outputs have same RMSE and determinant values up to 15 decimal places
  after caching, and zero-shot benchmark scores remain unchanged. Ablation
  studies show that learned deductive outputs have distinct loss and accuracy
  characteristics from models pretrained with transferred, randomly initialized
  or identity tensors as a constant tensor operator and an LLM with scaled-dot
  product attention (SDPA) is a special case of PLDR-LLM where G_{LM}
  is predefined as identity. The observed invariance characteristic introduces a
  novel asymmetry between training and inference phases with caching. We outline
  observed common characteristics of the deductive outputs for the learned
  singularity condition. We provide an implementation of a training and inference
  framework for PLDR-LLM with KV-cache and G-cache.
paper_ai_summary: PLDR-LLM, a Large Language Model, uses invariant tensors and a learned singularity condition to improve inference efficiency without altering model performance.
paper_ai_keywords: PLDR-LLM deducitve outputs invariant tensors singularity condition energy-curvature tensor $\mathbf{G}_{LM}$ power law graph attention (PLGA) cache G-cache KV-cache inference time RMSE determinant zero-shot benchmark scores ablation studies loss accuracy scaled-dot product attention (SDPA) asymmetric training and inference phases