What would you demo/build? How would you demo? I might have this opportunity now and would love to build something cool!
Any creative ideas? 🧠🚀
What would you demo/build? How would you demo?
I might have this opportunity now and would love to build something cool!
Any creative ideas? 🧠🚀
Letting an agent optimize a training run without me present usually beats what I would have done by hand, and often it would find more interesting (unfortunately) solutions than I would have tried.
So the question that I am trying to understand now -- what it means to properly write loops (or now graphs lol), and where my value actually is.
I started converging to the workflow where I would spend 1-2 hours carefully designing the optimization objective, goals and constraints, and then just let my Claude Code grind on it for days. I found it very important to have a clear separation of the evaluation code and the optimizable code. So that if I trust the evaluation and I know the agent can not change - I will trust the result the agent produced, so hallucinations is not a problem anymore.
I tried to formalize this philosophy in a skill + CLI library, where Claude helps me build a bulletprrof evaluation environment first (I call it a hill), and then the agent would "climb" it. Hills have a few mechanisms to make sure the agent can not just modify the evals mid-run. You can try it here: https://github.com/autolab-ai/hills (critical feedback is very welcome!).
Curious what everyone's thoughts are, where you see your place in todays workflows, how you design them etc? Do you think this foced separation of the evals and optimiable code is valuable?