AbstractPhila PRO

The shunts are available for ComfyUI now. Sorry about the delay, this isn't my only project and I have game dev tasks to handle alongside.

Upgraded version based on t5-flan-base cooking with a more powerful schema, more powerful block catches, a more curated bottleneck integrity, additional loss calculations and roughly 6.7m params.

This variation is based on taking only the t5 in at runtime, and the adapter attempting to approximate the guidance that the vit-l-14 would need by associative loss at training time.
This outcome wasn't robust enough and overall lacked the necessary detail that an adapter of this nature would require, so I recreated the adapter to be trained to accept both the t5-small and vit-l-14 as inputs. The outcomes are substantially more stable. The adapter weights and outcomes are posted as t5-vit-14-v1

It works. I've devised a methodology for token injection using an adapter trained directly between the T5 and CLIP_L to guide the CLIP_L in useful ways.

Try 2 seems to be normalizing around the intentional low loss valuation that I set up around the combination of scaled weights and careful use of alternating special tokens meant to teach models like the T5 how to behave intrinsically according to the paper and the research done on sentencepiece.
Slow bleed should help preserve a large combination of internal structure while slowly attenuating the structure and reshaping the weights of the high/low curve around each batch directly, creating a bit of a cascade 4d bleed effect from one middle-ground high-end topic swapped per one middle-ground low-end topic's valuation for learning and rephrasing.
In the process I've introduced inverse weighting to account for too many of one or too many of another token, while simultaneously improving the power of the lowest without overfitting everything to the lowest on a minimal scale; while simultaneously reducing the overall effect of the highest accountable token. This assists with overfitting everything based on generic flood linear, while allowing training on much smaller captions, without completely obliterating the entire structure of the T5-small in less than a few thousand steps.
Additionally, the highest, and lowest tokens are automatically weighted up or down; and once a token is masked, it automatically rescales the structure around the variant being masked and attended to.

This over time will teach how to predict variant caption middle-curve portions; allowing it to eventually conform to the normalization of the new "middle-ground" prediction purposed to swap elements from one caption to another. This should allow a very high learn rate, without a complete destruction of the T5-small; due to the various regularization techniques engineered for this task today.
This process will tap into similar training methods applied to the T5 small originally, as well as introducing newer methods designed and implemented on much larger scale AI training models - with weighted substructures meant to not rip the arms and legs off of this little model.

It won't do exactly what I wanted to do yet, but that's where the high complexity captions will come into play. They are based entirely on compartmentalized sub-sectioning the critical systems into usable ways - for example;
example prompt; a room of tacos
potential goal; a brightly lit room completely filled with many tacos

Simply put - I'm not that good with sd15. I got a few cool things working, but training a zero'd model isn't my forte. I'm better at enhancing or improving rather than creating entire full structures. My limit here is almost strictly a logistics one. The libraries don't like sd15 and they especially don't like when I start tinkering with internals and recording information from layer activation.
I require additional fundamental tests and foundational documentation before I proceed with my SD15 finetune from baseline to useful - as training tends to be quite vague to me still. Training Sd15 from zero to a useful state, requires additional tests, practices, and experience to train full models from zero to an anchorable enough of a state for Surge to latch onto the necessary points of interest. Those points of interest won't heat up without the necessary level of attachment and consistency within the inference - so they are required to create the powerful web of information required to learn and advance models from point A to B - without simply replacing the neurons themselves and making a flat copy.
There are other potentials here as well, where implanting neurons and finetuning has shown very good results. However, this isn't about that. This is about showcasing the power of the baseline system.

Hence, the other spectrum - where the model already exists, and has some fairly tested power for, I can interpolate two exacting classifier models - teacher/student using direct layer to layer learning and optimizations for time and speed. Classifiers can be small, lightweight, and fully capable of anchorpoint information transfer as well.
So instead of a full 1.5 finetune with various extra improvements - which the plan was to allow interpolation from a much stronger model into a much less robust model; I plan to take a classifier that is fully trained, and remodel one that has never seen the data and attempt to interpolate train this unfinished - imperfect - classifier; in order to introduce the necessary information using surge and anchorpoints into it's already trained using different data neurons, this process must be repeatable and useful for other realms other than simply 1:1 which is the point of an adapter like surge.

I'd say a fair full repeatable notebook is in order using simple processes and a simple classifier layer set. I'll choose well known models that have omitted training intentionally, and introduce that training using another model that is trained in the notebook itself. No tricks, no barriers, nothing special. Just keras, a bit of code, a bit of research, and a bit of elbow grease using the process.
This will showcase the power of Surge while simultaneously introducing a new type of rapid interpolative learning.

Usually anything learning from it's own output is just causing cascade failure given enough time. It's likely that without large amounts of additional information and context along with additional learning weights and parameters in relative sense, it's not going to be particularly effective at remembering new information; even when the conversation is left in context and the context itself is regulated and weighted.
Cascade failure is imminent eventually when the weights get too heavy and it starts to form a natural introspective bias on individual tokens. Looking up the valuation of all English used in literature historically reveals that many words are used over others, and with that same historic bias applied you can likely cut the edge off traditional literature; but modern literature has a completely different spectrum of words with entirely different odds.
Cascade failure is imminent, even with self regularization tokenization; or the process of learning is essentially nonexistent where it simply remembers nothing outside of the context window.
Though... I think I may have an idea to fix it.