Yeah as the title suggests, getting this error although only one TWS session is up and is on a completely different account from main trading account and has its own data sub.
Anyone had this issue, is there an API endpoint to check what sessions are active at any given time?
Polling around 40ish tickers to hydrate my strategy signals yet after just a few requests it gets this error.
Python+ibsync
I do kernel development and need to run multiple theories and hypothesis to find more efficient ways to compute. To this extent I have ablated both models and seeing that Opus does a better job at extremely difficult tasks.
Fable struggles with this and takes longer. I notice opus does it better and also faster to give DETERMINISTIC(eval against compiled kernel) output. Fable struggles and gives up saying there is no more room for optimization.
I was stuck and convinced that if Fable said no more room then no more room then I ran out of the 20x 50% usage so I switched my workflows to opus and suddenly it is testing the right theories and instruction sets that Fable never managed to encounter. Almost as if it was trying to an achieve an impossible goal that it set itself.
Same everything. This was a proper model ablation.
I feel like we are back to old days of 4.6 and OAI 5.x where one codes and other critiques.
Fable seems to do a better job all around. Sol likes smol tasks
Things are expensively healing. Really really expensively. Almost like the same model is being invoked twice where the second one does a full adversarial check which means you pay double.
We hitting a ceiling boys?
with the 5hr removed and limit reset and active fixes for runaway checks.... I only wish for transparency from Anthropic like OpenAI... like just a morsel of transparency :(
I genuinely wonder how anthropic is going to respond. At this point im back to coding manually and just here for the shitshow. popcorn time
Please stfu!
Sincerely, your future schrodinger’s company
Stop it get some help! You not, not making mistakes DARIO.
input: hello
API Error: Repeated 529 Overloaded errors. The API is at capacity
I mean sensional, we are grazing AGI like non other. This advanced intellisense is quite advanced really.
Hello I have been working on creating a LLM from ground up. It is based on deepseek architecture with heavily VRAM footprint reduced optimized(GUM+muon)
Currently this is the json schema I am using which should suffice as to what currently is being pretrained.
I have 2 6000 pro 600W
Testing a 7B parameter model with 64 experts... currently running on single GPU with 100% throughput (hardest part) (~80GB VRAM training) (reduction in expert count will substatially reduce vram footprint.... I am just pushing the limits here!)
My main goal here was simply that open source development will far outpace big firm development. I believe there is someone out there that can use this to build a LLM from group up that can beat all the top 1T parameter model. My goal here to create a large database of trained models that anyone can use. In the future maybe rent models from the open source dev as a support feature. Enough blabbing here is the technical report
since I am using DOLMA/redpajama you can separate the data split and have it train to be good at math, literature, physics... and then ensemble deploy them as agents (This is a todo for now since I don't have a single model to compare against)
This is also following the chinchilla optimal as well! thanks for deepmind!
All bfloat16, can be configured to use fp16 or fp32 if you are from the future and have a GPU that can do fp32 at bf16 speed!
Yes I have lost my mind many times during this, but I got something working!
this is 15000 steps in
======================================================================
[FACTUAL ACCURACY TEST] Step 14000
======================================================================
Prompt: "The capital of France is"
Output: "the city of Nice.
France may also refer to:
France (surname)
France (surname)
France (or Republ..." [CORRECT]
Prompt: "The capital of Japan is"
Output: "the capital of the autonomous prefecture of Hokkaido.
Etymology
The name of Hokkaido is derived fro..." [EXPECTED: Tokyo]
Prompt: "def fibonacci(n):
"""Return the nth Fibonacci ..."
Output: """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""..."
Prompt: "import torch
import torch.nn as nn
class Transfor..."
Output: "// InverterBlock
// s2, s2, s3
// A_1, A_2, A_3, A_4, A_5, A_6
// A1, A2, A3, A4, A5
// A1, A2, A3, ..."
Prompt: "The theory of relativity states that"
Output: "the speed of light varies with the speed of the observer. This is a constant, since the speed of lig..."
Prompt: "In machine learning, gradient descent is used to"
Output: "perform a gradient descent, where the gradient is calculated via a local gradient. The gradient eval..."
Prompt: "Question: What is 2 + 2?
Answer:"
Output: "2 + 2
Author: PCR
Date Submitted: 2nd April 2013
Pp: 200-201
Exercise: Exercise 2.0
2 + 1 = 2 +..." [EXPECTED: 4]
Prompt: "Question: Explain the concept of recursion.
Answer..."
Output: "In programming, a function or sequence of operations is a function that can transform a variable to ..."
FACTUAL ACCURACY: 1/3 = 33.3%
----------------------------------------------------------------------
[SMBench] Step 14000 -- 1/5: Multi-Rule Reasoning
.
.
.
JSON struct defining the arch
"experiment_name": "deepseek_v3_7b_lowvram",
"output_dir": "*******",
"seed": 420,
"model": {
"num_layers": 24,
"vocab_size": 50304,
"norm_type": "rmsnorm",
"norm_eps": 1e-06,
"tie_word_embeddings": false,
"init_method_std": 0.006,
"first_k_dense_replace": 8,
"dense_layer_interval": 1,
"paper_compliant": false,
"mla": {
"d_model": 1408,
"d_latent": 352,
"num_heads": 22,
"num_kv_heads": 2,
"max_context_length": 4096,
"use_flash_mla": false,
.
.
.
},
"moe": {
"num_experts": 64,
"num_experts_per_token": 4,
"expert_intermediate_size": 1536,
"expert_dim": 1536,
"dropout": 0.0,
"num_shared_experts": 1,
.
.
.
.
}
},
"fusions": {
"use_fused_expert_ffn": true,
"use_te_fused_topk": false,
"use_te_fused_permute": false,
"use_fused_softmax": true,
"fused_softmax_in_fp32": true,
"use_group_limited_topk": true,
.
.
.
},
"memory_optimization": {
"use_galore": false,
"galore_rank": 256,
"galore_update_proj_gap": 500,
"galore_scale": 1.0,
.
.
.
},
"training": {
"device": "cuda",
"global_batch_size": 256,
"micro_batch_size": 4,
"gradient_accumulation_steps": 64,
"seq_length": 1024,
"max_batch_seq_multiplier": 1.25,
"tokens_per_parameter_ratio": 40.0,
"total_training_tokens": 280000000000,
"learning_rate": 0.00042,
"min_learning_rate": 4.2e-05,
"lr_preset": "deepseek_v3",
.
.
.
},
"data": {
"use_multi_source": true,
"sources": [
{
"name": "redpajama",
"type": "dolma",
"subset": "dolma_v1_6_redpajama",
"weight": 0.45,
"description": "RedPajama - CommonCrawl-like diverse web/code/books"
},
{
"name": "stack",
"type": "dolma",
"subset": "dolma_v1_6_stack",
"weight": 0.25,
.
.
.
],
"cache_dir": "*******",
"sanitization": {
"enabled": true,
"target_language": "en",
"min_language_confidence": 0.9,
"min_article_length": 100,
.
.
.
},
"preprocessing": {
"num_workers": 8,
"shuffle": true,
"shuffle_seed": 42,
.
.
.
},
"max_articles": null,
"focus_historical": false,
"boost_hiroshima_content": false
},
"distributed": {
"backend": "nccl",
"launcher": "single_gpu",
"tensor_parallel_size": 1,
"pipeline_parallel_size": 1,
"expert_parallel_size": 1,
"data_parallel_size": 1,
"zero_stage": 2,
"zero_offload": true,
"overlap_grad_reduce": true,
"overlap_param_gather": true,
"deepspeed": {
"enabled": false
}
},
"checkpointing": {
"save_interval": 1000,
"save_total_limit": 3,
"resume_from_checkpoint": null,
"checkpoint_format": "pytorch",
"save_optimizer_states": true
},
"logging": {
"log_level": "INFO",
"log_interval": 100,
"tensorboard_dir": "*******",
"wandb": {
"enabled": false
},
"tensorboard": {
"enabled": true
}
},
"validation": {
"enabled": true,
"eval_interval": 1000,
"eval_samples": 500,
"metrics": [
"loss",
"perplexity"
],
"patience": 300,
"early_stopping": false
},
"profiling": {
"trace_nvtx": false
},
"gpu_optimization": {
"cuda_graphs": true,
"torch_compile": true,
"flash_attention": true,
"fused_kernels": true,
"autocast_dtype": "bfloat16"
},
"test_prompts": {
"enabled": true,
So I basically researched and threw every optimization on this planet earth. Even tried to build my own FlashMLA for sm120 blackwell arch and failed miserably although I got inference working... backwards I couldn't due to tiling which ends up being the same if not worse than Aeten torch backend......
But this is working for now, 20seconds a step
eg
Training: 1%|█ | 14609/1000000 [53:18:23<5533:28:53, 23.37s/step, loss=2.1507, mtp=1.9643, ent=4.12, util=100.0%, imbal=0.26, lr=4.20e-04, tok=2.23B]
So in conclusion
I am scared as shit to open source this until I get it working 100% so as to minimize the community hate I will eventually get.
The only point of contention I have is I want all models trained using this to be public I don't want anyone to privatize without open-sourcing for profit so I need to ask around and figure out how to go about this since I want as many models that can be trained using this since I believe there is someone out there with the right configuration already in mind that will beat out the top performing model. This is mainly why I did this, I know I can't create THAT model, but I know for sure as shit there is some genius out there that can train a model that will be SOTA.
There is alot of cleaning up to do before I make it public because scared of the hate and issues I surely cannot fix alone!
If you are interested you can check my account periodically whenever I make a post about making this repo public! or check my github which would be easier I assume lol
https://github.com/IISuperluminaLII
I dont know.. I am open to feedback on how to properly make this public and make it a strict rule to open source all safetensors or checkpoints if using this code... I know there is someone out there given the right tools that can truly build a 10B-50B parameter model ensemble set of models that can achieve near SOTA level performance!! As they always say, divide and conquer
This is getting long already, I have puked my brains out as much as I can. Any input is welcome, even hate! let me know how to fix this so I can deliver the tool the random person who will eventually create the perfect open source model.