u/ha2emnomer

I've been working on an open-source project called UnFlow:

https://github.com/UnFlow-Labs/mlunflow

The idea is pretty simple:

Most ML experiment tracking looks like a list of independent runs usually stored in a table:

run_001
run_002
run_003
run_004
...

But in practice, experiments are usually related.

You change the learning rate, then the number of epochs, then the model, then some preprocessing code. Eventually you have hundreds of runs, but it's surprisingly difficult to answer:

• What actually changed between these two experiments?
• Which experiments are essentially the same computation?
• Have I already run this experiment before?
• How did I get from experiment A to experiment B?
• Can I navigate the history of my experiments rather than just search through runs?

Unflow simply detect code changes in a Python function (limitation that for it is just a single function) and arguments that are passed to this function to build a graph where nodes are "states" and edges are transformations "what has changed", a new state is not added to the graph or executed expect if it has a transformation.

The project is still early, so I'm much more interested in feedback than pretending this is a finished product.

I'm particularly curious about three things:

1. Does the "experiments as a graph" abstraction make sense to you?
2. Do you currently run into problems with duplicated/redundant experiments?
3. If you could see the complete lineage of your ML experiments, what would you want to query or visualize?

Repo: https://github.com/UnFlow-Labs/mlunflow

I'd love to hear how other people currently manage experiment lineage and whether this solves a real problem for you.

I am building a new ML tool as part of my research in ML systems. Currently I am surveying potential users (ML researchers and engineers) to understand how useful this system could be. Would you please respond to this survey ?
https://forms.gle/zGU83uq8r3hcu6gz9

I am happy to answer any questions.
Thanks