Omega Attention v0.1
Omega Attention is a small language model fine-tuned specifically for attention management inside an artificial cognitive architecture.
https://huggingface.co/rekstar/omega-attention-0.5b
Rather than using a general-purpose LLM to repeatedly decide what an agent should focus on, Omega Attention is designed as a lightweight specialist cognitive node.
Given a cognitive workspace containing competing goals, memories, observations, system events, and tool activity, the model selects an attention operation and target.
It is part of the broader Omega artificial cognitive architecture experiment. https://github.com/doctarock/Artificial-Cognitive-Architecture-ACA-
What does it do?
Omega Attention chooses between five attention operations:
ATTEND— bring something into active attentionMAINTAIN— continue attending to the current focusSWITCH— move attention to a more important candidateSUPPRESS— actively suppress a distracting or conflicting candidateIGNORE— take no attentional action
The output contract is deliberately narrow.
{"operation":"SWITCH","target":"observation_4","confidence":0.91,"reason_code":"HIGHER_PRIORITY_INTERRUPT"}
The model is not intended to chat, explain its reasoning, or generate general-purpose text.
Its job is simply:
>
Model
| Property | Value |
|---|---|
| Base model | unsloth/Qwen2.5-0.5B-Instruct-bnb-4bit |
| Base parameters | ~502M |
| Fine-tuning | LoRA |
| LoRA rank | 16 |
| LoRA alpha | 16 |
| Trainable parameters | ~8.8M |
| Trainable percentage | ~1.75% |
| Training epochs | 3 |
| Training examples | 5,000 |
| Evaluation examples | 500 |
| Training platform | AMD Radeon 8060S / ROCm |
| Training framework | Unsloth |
| Output | Compact JSON |
LoRA was applied to:
q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj
Training
The model was trained on 5,000 synthetic cognitive-workspace scenarios.
Each scenario contains competing candidates such as:
- goals
- memories
- observations
- system events
- tool calls
Candidates have properties including activation, salience, confidence, novelty, and age.
Each workspace is paired with the desired attention operation and target.
Training used standard causal language-model fine-tuning with prompt tokens masked from the loss so that training focused on producing the required attention decision.
The final run consisted of 1,875 steps, approximately three epochs.
Training took approximately 54 minutes (3,229 seconds) on an AMD Radeon 8060S using ROCm.
Results
Evaluation was performed against 500 held-out synthetic scenarios.
| Metric | 200 steps | 1,875 steps / 3 epochs |
|---|---|---|
| Valid JSON | 100% | 100% |
| Operation accuracy | 74.2% | 96.0% |
| Target accuracy | 45.2% | 92.2% |
The architecture, dataset, LoRA configuration, and model size were unchanged between these runs.
The primary difference was training duration.
Target accuracy increased from 45.2% to 92.2% without increasing model size or inference cost.
Important evaluation caveat
These results should not be interpreted as demonstrating 96% accuracy on arbitrary real-world attention-management tasks.
The evaluation examples are held out from training, but they are generated by the same synthetic scenario-generation system used to produce the training dataset.
The results therefore demonstrate that a 0.5B model can successfully learn the attention policy represented by this dataset.
Generalisation to different distributions, adversarial scenarios, and real Omega cognitive traces remains to be evaluated.
This is one of the primary goals for future versions.
Why a 0.5B model?
Omega is exploring an alternative to using one large language model for every cognitive operation.
Instead, cognitive functions can potentially be decomposed into specialised nodes:
Cognitive Workspace
|
v
Attention Node
|
"What matters now?"
|
+----------+----------+
| | |
v v v
Memory Reasoning Goals
Attention management is frequent but relatively narrow.
Using a large reasoning model for every attention decision would introduce unnecessary latency and compute cost.
Omega Attention explores whether a very small specialised language model can perform this cognitive function instead.
The v0.1 results suggest that, at least for the synthetic attention policy tested here, increasing model size was unnecessary.
Intended use
Omega Attention is intended for experimentation with:
- artificial cognitive architectures
- autonomous agents
- cognitive workspaces
- attention allocation
- persistent agent loops
- multi-model agent systems
- specialised neural cognitive modules
It is primarily intended to operate as a component inside another system, rather than as a standalone conversational model.
Not intended for
Omega Attention is not intended to be:
- a general-purpose assistant
- a reasoning model
- a chatbot
- a factual knowledge model
- a safety classifier
- a replacement for human decision-making
It has been trained for a deliberately narrow experimental task.
Status
Experimental — v0.1
This is the first public iteration of Omega Attention.
The major question addressed by v0.1 was:
>
The initial result is encouraging.
The next question is more important:
>
Future work will focus on harder evaluation rather than immediately increasing model size.
About Omega
Omega is an experimental artificial cognitive architecture exploring the idea that an artificial mind does not need to be synonymous with a single large language model.
Instead, language models can act as specialised neural components within a larger cognitive system incorporating mechanisms such as attention, working memory, long-term memory, goal management, reasoning, planning, and action.
Omega Attention is the first specialised cognitive model produced as part of that experiment.