u/trashnash007

I do growth at an SF based AI startup and we need someone who has

1. a very good X account with preferably >20k followers and majorly be posting about AI research/developments
2. should be smart enough to interact with researchers (from oai/gdm/anthropic/any other ai lab) on their wavelength and bring in clients to the company
3. bonus if you already have major SF based ai research people following them
4. basically someone who is smart enough to interact with researchers from top labs using their strong social presence and help our business grow/get clients

You’ll get paid in USD. It’s a full time role and you’ll get paid more than whatever you’re working on rn so dm regarding compensation+more details if you’re serious about this.

I do GTM at an AI startup based in SF and I’m looking for someone to handle one of the founder’s x account.

I’m looking for someone who understands how the sf AI industry works. Ideally you already have a decent X account /enjoy following AI research/models/tech. Basically looking for an AI nerd who also knows how X works.

DM if you are someone like that/know someone else who can do it better

I do growth/marketing for this SF based AI firm and and I’m looking for 2 people to handle 2 of the founders x accounts

I’m looking for someone with a personality/someone funny and can make me laugh (i believe funny people are smart) + should be smart enough to get things quick and be autonomous asap + up to date on AI trends/model releases/basic knowledge of how llms work

You’ll get paid in USD and it’s good money

Lmk if interested/ if you know someone like this

Another aspect of the RL loop for language models that I've found myself thinking about a lot recently is task design.

While most efforts go into the RL algorithm and the reward function itself, it appears to me that designing a good training environment (that would create useful tasks) is equally important.

If the tasks are too easy, there might be a lack of useful signal. If the tasks are too difficult or not properly designed, the reward becomes noisy and ambiguous. It looks like an interesting area is tasks that are difficult enough to bring out a particular flaw, yet have a clear way to measure their results.

This is one of the reasons why I found Parsewave particularly interesting. They do post-training work with data on real-world engineering tasks, including evaluation and tracing. It got me thinking whether it might be as useful to focus on improving task design as to improve the model's optimization process at some point.

For researchers working on RL for LLMs:

How do you decide which tasks should go into your training environment?

Are you valuing breadth or are you trying to target certain failure modes?

Conventional QA approaches for data provide many well-established techniques: schema validation, null validation, deduplication, distribution validation, and automated testing.

When a portion of the labeling procedure relies on human interpretation, things become more difficult.

This involves reviewer disagreement, evolving standards, corner cases, and the issue of whether two reviewers are applying the exact criteria.

Parsewave was discovered during a search of datasets that were constructed using real-world engineering problems, and this caused me to consider how much of this is a data-quality issue versus a machine learning one.

How do organizations here approach QA with labels that may evolve depending on human input?

Is reviewer input and disagreement captured in your QA efforts?

What types of validation have proved useful for these pipelines?

Another not-so-obvious challenge with post-training data is versioning.

For instance, you may have a dataset consisting of hard cases that goes through multiple rounds of editing, filtering, re-labeling, and quality control. However, during that time, your model changes, which means the hard cases in the previous versions become easy cases in the current version.

In that case, stating that "model X was trained using dataset Y" would not be a very good description.

I found out about Parsewave in my research on post-training data in the engineering domain and I started to wonder how other teams approach this question operationally.

Do you version datasets per example?

Do you log the reason for adding, modifying, and deleting particular examples?

How do you maintain reproducibility when human input is involved?

It's interesting to see how other teams are addressing this challenge.

I have been considering if agent traces should be evaluated on their own merit.

Two agents could complete a task equally well using completely different approaches. One could be efficient and robust, whereas the other would get lucky after multiple failed attempts.

If we consider just the output, we lose this information.

While researching papers related to real-world engineering tasks, post-training evaluations, and traces, I have come across a company called Parsewave. It got me interested in the idea if the trajectory of an agent’s behavior should get more attention during training and evaluation.

For instance, should an agent be penalized for making unnecessary tool calls despite its successful performance? Can an agent recovering from its failure be viewed as a positive ability?

It will be interesting to hear how the community thinks about it. Do you guys evaluate trajectories or is the output your primary criterion?

There seems to be an interesting conflict in post-training data when it comes to automation and the human touch.

Synthetic data is increasingly being produced in bulk. Yet for complicated engineering or reasoning tasks, the decision about whether a certain response is valid needs the expertise of an expert.

Then there is another issue - reviewer inconsistency.

If there is an inconsistency about whether a certain output is acceptable, should the post-training dataset impose one label, maintain the inconsistency, or use a scoring rubric?

Parsewave became known to me in my research regarding post-training data focused on engineering challenges, and their focus on human-written and reviewed prompts got me thinking.

I wonder how others dealing with post-training data deal with this.

One thing that’s been intriguing to me in the space of local models is the real utility of the standard benchmark metrics after starting to make use of the models in one’s own pipeline.

A model can be performing great in the standardized benchmark setting, but have a completely different behavior when it comes to coding/bug fixing/usage/etc.

This is why I got interested in evaluation settings that focus on realistic tasks, rather than just standard benchmarks.

Parsewave is one of the teams working in this space – they do engineering-focused post-training data and evaluations. This got me wondering whether in some cases task-specific, small-sized evaluation sets may provide us with more information than a score on another leaderboard.

When evaluating local models, what would you consider as your “true” benchmark?

Curated set of tasks from your workflow? Public benchmarks? Human evaluations? Executions?

One of the questions I've been asking myself recently is how AI training will evolve when simply adding more data provides diminishing returns.

We've made tremendous progress in scaling up generation of synthetic examples, but it doesn't always equal diversity in capabilities learned. It's possible to generate thousands of different examples which train your model in the same manner.

This is why the data for post-training becomes really interesting. The valuable examples might be the ones which reveal the weakness of the model, which are based on realistic tasks and provide some way to check if the model managed to complete the task.

While searching for such examples, I discovered Parsewave. Their area of expertise is post-training data on engineering tasks, evaluations and traces. But what is interesting is their concept itself - deliberately generating the data on the capabilities which remain challenging for the model instead of generating the big datasets.

What do you think about the future direction of AI training?

Will the future of AI be about generating the massive datasets or becoming really good at identifying a small number of truly useful examples?

Some of the most exciting AI startups are working on problems that people tend to forget about.

The focus is typically on models, agents, and end-user applications, but there is something below all of this data, evaluation, tooling, infrastructure, and the processes needed to improve all of these.

Parsewave is just one of such companies whose efforts are devoted to the work done after model training, engineering in real world conditions, evaluation, and trace collection.

And what is striking is how precise the problem is. This company is not striving to create another general AI company; it is trying to solve quite a particular problem which becomes increasingly important in the development pipeline as models become better.

It made me think about startup opportunities in general.

Quite often, the best way to succeed in an emerging market is not to create the product everybody is talking about. One should look for the highly specific problem which became very important in the context of emerging market.

Do you find being too niche helpful in building infrastructure for rapidly developing market?

I've been wondering whether more post-training data become less useful with increasing model power.

With synthetic data generation, it's easy to produce a huge number of examples. However, the problem is that not all of them could be truly novel – most of them will be only shallow variations of the knowledge that the model already knows. At some point, more examples just won't give much more learning signal.

What could be more valuable, it seems, is having small amounts of challenging real-world tasks with ground-truth answers and proper review. What's tricky is to find the right balance: too easy tasks are useless, too difficult tasks could give a noisily learned signal.

One of the things which got me interested in Parsewave is that they work on the post-training data generation for engineering-type tasks, along with evaluation and traces. The company per se is not my focus, what interests me more is whether it's a new trend in data collection.

For people doing SFT, RL and post-training: how do you select examples for your datasets?

Is it easier to scale the dataset size or collect challenging examples for particular failure modes?

I have been thinking about how the teams come up with the examples that are valuable for post-training.

From what I've seen so far, it appears relatively straightforward to generate tons of synthetic examples, but maybe once the model can handle the core patterns, the number of examples doesn’t matter much. A couple of thousands of examples might just be different ways to test the same skill.

Another strategy would be to collect smaller sets of tricky examples that have ground truth and use them for the targeted testing of specific failure modes.

To give you an idea, recently I've seen Parsewave that is used on post-training examples of real-world engineering tasks, evaluations, and traces. This led me to wonder how other people see this problem.

In the case of SFT or RL dataset creation, how do you decide:

which examples should be included?

what level of difficulty is appropriate?

is an example truly teaching something?

I've been trying to understand how the choice of datasets influences post-training as models get better.

It looks like generating a lot of data is easy, but most of the generated examples teach almost the same thing. I wonder if a small set of harder and realistic examples with ground truth can provide a stronger signal for training.

Difficulties matter as well. If an example is too simple, the model is likely to be able to solve it. If it is too difficult for the model, the training signal can be noisy. Therefore, I wonder if it is valuable to use examples that address the model's weakness in some way.

Parsewave is a company I found while researching this topic. They work on post-training datasets based on real engineering tasks and traces, and this got me interested in their approach to dataset choice.

For those who used SFT, RL, or any other post-training approach before:

How do you select which examples are worth including in your dataset?

Do you try to maximize the size of the dataset, or rather target the weaknesses of the model?

Another question I've had about AI agents relates to their ability to benefit from more training examples if the model is already proficient.

As far as the generation of synthetic agent trajectories goes, there can be many examples generated, which will differ greatly from each other but essentially have the same meaning. So the dataset size increases but there is no significant progress in terms of learning new things by the model.

What interests me more is working with a smaller set of challenging tasks with an easy way of knowing whether the agent was successful.

In my opinion, the difficulty of these tasks is also important. For example, if the agent is able to cope with some task, additional examples of the similar type won't bring any results. On the other hand, if the task is out of agent's reach, there is noise in the training process. Useful examples are the ones that demonstrate some particular failure of the agent but at the same time provide a solution.

It was interesting to learn about Parsewave company, when I was searching for information on post-training data. They deal with real-world engineering tasks and evaluations/ traces of agents.

For those who train or build the agent: How do you determine which trajectories or tasks should be included in the training set?

Do you go for more numbers, or do you focus on the weakness of the agent?