AbstractPhila PRO

I know this likely sounds odd or a bit too generic currently. The concept here is straightforward in pragma though. Fixed position, dynamic position, rotary position, global, local, or any other doesn't matter. The model simply needs that learned behavior and the understanding of how to seek the information, with a stable protocol system for acceptance, rejection, and so on. You don't need 600 layers in a model to debate the answer to this question internally, trained through 1000 step differentiated Adam, when the stable answer is something represented somehow in something.

In our case we take a piece of the context window from the overarching global system, say a dynamic allocation <FRAME>, and we snap a positional framed request to this. One of the collective with the necessary compositional addressing behavior intercepts this request, processes the information within it's bounds benefitting from the core model as necessary, sends the compositional result back.

One token done. Say we do this 500 times, each compositional structure intercepted by the necessary arm's behavior, providing that learning as a standard MOE would, except with a format of purpose.

Each of those compositional leaders have their own sub-collectives of solvers. Whatever they distilled or utilized or whatever the model is constructed to use, something like RNN, memory, or whatever. The next token or tokens are predicted and fed out utilizing the necessary structure within those dynamic framings, and once the capacity for that frame is exhausted the frame rises.

Simple in texture really. Something akin to a programming stack, except the addressing for the stack's behavior is a complex mathematic rotary combinatory behavior. We've been doing this for 80 years.

Unknowns are seen and yielded. Problems are spotted and yielded. Everything between is fielded, tested, and internally debated.

This when functional, will not need a byte model. This structure will behave effectively with tokenization models as well. The lookup system needs maturing but the system will comply.

This may seem like 5 steps back 1 step forward from just feeding a large model all the necessary information and getting a multi-debated answer over time. A seemingly difficult and complex idea to understand forming a lesser structure.

This isn't exactly the case though. In this model design's case, you gain the power of yielded multi-stage, multi-step, pre-planned arbitration, in a stable next-token prediction system.

Our hardware is TERRIBLE at this sort of thing, so I have a few pieces to make.

You don't WANT a model like tiny Beatrix to use a system like this, you need something with a bit of intelligence to arbitrate the subsystem. A Qwen with some oomph is a good option.

So... yeah the little models seem to learn most text related tasks on this scale. Individual structures with training runs seem to have very little problems learning them.

9 million params isn't very much, but it's more than enough for this task apparently. I'll be finetuning something much stronger with the dataset sonnet, opus, and fable engineered for the image thinking task. I'm thinking a powerful qwen or something like that. Something that knows images already.

That should be an appropriate target. Something with more brains than an atom.

Castor learned clean chain of thought.

Got a bit overzealous on my fable, opus, and sonnet usage - it's pretty much gone. I'll be setting up something with Opus but the usage is limited so nows a good a time as any to take a break until tomorrow.

Pollux should finish soon, then the dual thinking nanos should be functional.

Pollux finished.

Experimental thought dataset should be pretty beefy now. Not sure how well it'll work unless I train a vit with the information using cross-contrast, so I'll run some finetunes and benchmarks. The nano models don't understand thinking, so I'm not really sure what'll happen if I just raw feed it. I assume overfitting from just the data if I run multi-epoch due to the lack of CoT training, so the model's CoT will likely need to be conditioned ahead of time to be generic CoT as per my assumption. This isn't a guaranteed requirement, it may very well be the opposite - where CoT can be taught using symbolic objective representation rather than needing some sort of CoT dataset to baby-step into the system.

I'll run some experiments and tests, then figure out some good methodologies for the trie structure.

nano-pollux-dual-v1.2 is done, full frame train is in.

Model eats them up. Control is the nano-castor model having never seen frames.

The construction begins on a specific chain of thought dataset built around conditioning for image generation and VLM classification chain of thought. The dataset will be selectively designed for image curation, editing, associative reasoning, classification, and more.

Being text-specific the dataset will contain multiple direct conversions of known popular dataset caption sets into CoT formatting utilizing symbolic associative reasoning. If images do exist, the repo will be directly linked in the readmes, or the image pointed directly at a URL depending the ruling.

The entire purpose of this dataset is to provide a specific set of framed curation datasets, built specifically around populating framed portions of context window information. Think about it like curated thought pathology meant to encourage stable thought structure specific to images. In short, the dataset is being built around multi-parallel thinking, but will function as standard CoT when trained on other models.

There will be multiple prototype trains based on this topic with the entire goal meant to provide strong image control editing and capacity for editing using runtime conditioning, rather than requiring a multi-tuned retrain of models post finetune and post training.

The goal is to enable image editing guidance capacity to every applicable CoT model with the power of a single chain of thought dataset. Image training directly on diffusion models within specific parameters can enable this editing to be participated in with minimal supervision, even when models use entirely different embedding, encoding, lookup, or position systems.

There are no guarantees this will work, and the experimentation will be ongoing while curating and designing. The dataset will be simple at first, then expanding outward heavily as time progresses.

The research shows that stable and curated CoT can be stabilized and made reasonably learnable with simple mitigated finetuning, and the experiments based on this dataset will yield those results publicly.

The frame-centric finetune is in progress, and it broke the readme for the repo so I'll fix it momentarily.

As it stands the full frame training is holding pretty effectively.

The nano framed variants are heavily dependent on the banks, which is both good and bad. The lack of shared structure is a bit of an efficiency problem, as it takes longer to learn and solidify the depth content, but since the model is learning framed depth by structure it's holding either way. The training process is working so far.