u/yazidaqel

Every time I've had to build a batch processing pipeline in .NET, I end up duct-taping together the same primitives: a foreach loop, some retry logic, a table to track what ran, manual skip handling, and a prayer. Java devs have had Spring Batch for this since 2007. We have nothing.

So I started Conveyor.Batch — an open-source, production-grade batch processing framework for .NET 8+ that brings chunk-oriented processing, job repositories, restartability, skip/retry policies, and partitioning as first-class citizens to the ecosystem.

The core model should feel familiar if you've done any ETL work:

// Reader streams items asynchronously
public interface IItemReader<out TInput>
{
    IAsyncEnumerable<TInput> ReadAsync(
        StepExecutionContext context,
        CancellationToken cancellationToken);
}

// Processor transforms one item — return null to skip it
public interface IItemProcessor<in TInput, TOutput>
{
    ValueTask<TOutput?> ProcessAsync(
        TInput item,
        StepExecutionContext context,
        CancellationToken cancellationToken);
}

// Writer receives a committed chunk
public interface IItemWriter<in TOutput>
{
    ValueTask WriteAsync(
        IReadOnlyList<TOutput> items,
        StepExecutionContext context,
        CancellationToken cancellationToken);
}

The engine reads in configurable chunks, applies your skip/retry policies per item, flushes committed chunks to the writer, and tracks everything in a job repository so failed runs can restart from where they left off — not from scratch.

What's planned:

• Conveyor.Batch — core chunk engine + abstractions
• Conveyor.Batch.EntityFrameworkCore — persistent job repository (Postgres, SQL Server, SQLite)
• Conveyor.Batch.IO — flat file, JSON, XML readers/writers
• Conveyor.Batch.Http — paginated HTTP reader
• Conveyor.Batch.Hosting — Worker Service integration

Honest status: this is early. The abstractions are defined, the chunk engine design is locked, and we're scaffolding now. Nothing is on NuGet yet.

I'm posting because I want to build this with the community rather than drop a v1.0 announcement six months from now. A few questions I'd genuinely like input on:

1. What batch use cases do you hit most often in .NET that you'd want Conveyor to handle well?
2. Any prior art I should study (beyond Spring Batch itself)?
3. Would you rather see a PostgreSQL-first or SQLite-first job repository for v0?

GitHub: conveyor-batch/conveyor.batch — watch/star if you want to follow along.