PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own
  Deep Neural Net At Inference
We show that Large Language Model from Power Law Decoder Representations
(PLDR-LLM) is a foundational model whose deductive outputs are invariant
tensors up to a small perturbation. PLDR-LLM learns a singularity condition for
the deductive outputs that enable the once-inferred energy-curvature tensor
G_{LM} to replace the deep neural network of power law graph
attention (PLGA) generating the deductive outputs at inference. We demonstrate
that a cache for G_{LM} (G-cache) and KV-cache can be implemented in
a straightforward manner to improve the inference time. The invariance and
generalizable nature of deductive outputs is at a very high fidelity where
deductive outputs have same RMSE and determinant values up to 15 decimal places
after caching, and zero-shot benchmark scores remain unchanged. Ablation
studies show that learned deductive outputs have distinct loss and accuracy
characteristics from models pretrained with transferred, randomly initialized
or identity tensors as a constant tensor operator and an LLM with scaled-dot
product attention (SDPA) is a special case of PLDR-LLM where G_{LM}
is predefined as identity. The observed invariance characteristic introduces a
novel asymmetry between training and inference phases with caching. We outline
observed common characteristics of the deductive outputs for the learned
singularity condition. We provide an implementation of a training and inference
framework for PLDR-LLM with KV-cache and G-cache.
67bd1db005a599263a2a684e
Burc Gokden
extracted_pending
2025-02-25T01:32:33.775Z
671ddb3bf89c9b8208568e73
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/-Q77RLuIvzpVU95WGdM7u.png
Burc Gokden
user
fromthesky
PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own
  Deep Neural Net At Inference
We show that Large Language Model from Power Law Decoder Representations
(PLDR-LLM) is a foundational model whose deductive outputs are invariant
tensors up to a small perturbation. PLDR-LLM learns a singularity condition for
the deductive outputs that enable the once-inferred energy-curvature tensor
G_{LM} to replace the deep neural network of power law graph
attention (PLGA) generating the deductive outputs at inference. We demonstrate
that a cache for G_{LM} (G-cache) and KV-cache can be implemented in
a straightforward manner to improve the inference time. The invariance and
generalizable nature of deductive outputs is at a very high fidelity where
deductive outputs have same RMSE and determinant values up to 15 decimal places
after caching, and zero-shot benchmark scores remain unchanged. Ablation
studies show that learned deductive outputs have distinct loss and accuracy
characteristics from models pretrained with transferred, randomly initialized
or identity tensors as a constant tensor operator and an LLM with scaled-dot
product attention (SDPA) is a special case of PLDR-LLM where G_{LM}
is predefined as identity. The observed invariance characteristic introduces a
novel asymmetry between training and inference phases with caching. We outline
observed common characteristics of the deductive outputs for the learned
singularity condition. We provide an implementation of a training and inference
framework for PLDR-LLM with KV-cache and G-cache.
PLDR-LLM, a Large Language Model, uses invariant tensors and a learned singularity condition to improve inference efficiency without altering model performance.
PLDR-LLM deducitve outputs invariant tensors singularity condition energy-curvature tensor $\mathbf{G}_{LM}$ power law graph attention (PLGA) cache G-cache KV-cache inference time RMSE determinant zero-shot benchmark scores ablation studies loss accuracy scaled-dot product attention (SDPA) asymmetric training and inference phases