Data ingestion, at scale
Hatchet is built to support massively parallel workloads, often seen in document ingestion, image/video processing, or bulk file processing.
Pipelines as code
Every pipeline can be defined as a Hatchet DAG or a durable task. You can run hundreds of thousands of pipelines simultaneously and scale to hundreds of workers.
# Pipelines are DAGs of plain functions
pipeline = hatchet.workflow(
name="ingest-doc", input_validator=Doc
)
@pipeline.task()
async def parse(input: Doc, ctx: Context) -> Parsed:
return parse_document(input.url)
@pipeline.task(parents=[parse])
async def embed(input: Doc, ctx: Context) -> Embedded:
parsed = ctx.task_output(parse)
return embed_chunks(parsed.chunks)
@pipeline.task(parents=[embed])
async def store(input: Doc, ctx: Context) -> Stored:
return upsert_vectors(ctx.task_output(embed))// Pipelines are DAGs of plain functions
const pipeline = hatchet.workflow<Doc>({
name: 'ingest-doc',
});
const parse = pipeline.task({
name: 'parse',
fn: (input) => parseDocument(input.url),
});
const embed = pipeline.task({
name: 'embed',
parents: [parse],
fn: async (_, ctx) => {
const parsed = await ctx.parentOutput(parse);
return embedChunks(parsed.chunks);
},
});// Pipelines are DAGs of plain functions
pipeline := client.NewWorkflow("ingest-doc")
parse := pipeline.NewTask("parse",
func(ctx hatchet.Context, in Doc) (Parsed, error) {
return parseDocument(in.URL)
})
embed := pipeline.NewTask("embed",
func(ctx hatchet.Context, in Doc) (Embedded, error) {
return embedChunks(ctx, in)
}, hatchet.WithParents(parse))
pipeline.NewTask("store",
func(ctx hatchet.Context, in Doc) (Stored, error) {
return upsertVectors(ctx, in)
}, hatchet.WithParents(embed))Fairness and priority for ingestion at scale
Hatchet’s concurrency control and priority features are built to help you run workloads at scale. You can prioritize user-submitted workloads over batch workloads and distribute work fairly between organizations, users, or tenants using Hatchet’s concurrency control features.
Concurrency keys ensure that resources are distributed fairly between users.
Keep data fresh
Hatchet supports event-based triggering and webhooks to ensure your data is always fresh. If data changes upstream, you can quickly sync it back to your data stores using Hatchet’s built-in features.
Run your ingestion pipelines on Hatchet. Highly parallelized, fair scheduling.
Start for free — no credit card required
Processing over 100 million tasks/day on Hatchet Cloud
Open-source and self-hostable