Workflow Engines

Why This Matters

Simple AI assistants usually answer immediately. Enterprise AI systems often execute workflows lasting minutes, hours, days or weeks. Examples include employee onboarding, insurance claim processing, contract approval, financial audits and procurement workflows. These workflows cannot depend on a single AI request.

A workflow engine keeps everything organized.

Everyday Analogy

Imagine a package delivery company. A parcel moves through many stages: Receive Package → Sorting Center → Regional Hub → Delivery Vehicle → Customer. If a truck breaks down, the package isn't lost. The workflow continues after recovery.

Workflow engines provide the same reliability for AI systems.

What Is a Workflow Engine?

A workflow engine controls workflow execution, task sequencing, state management, retry logic, human approvals, error recovery, parallel execution and workflow completion. It ensures every step happens in the correct order.

Workflow vs Orchestration

Orchestration answers "What should happen?" It coordinates agents, models, tools and APIs.

A Workflow Engine answers "How does the work execute reliably?" It manages state, retries, timing, checkpoints and execution.

Orchestration defines the process. The workflow engine executes it.

Workflow Lifecycle

Request → Initialize Workflow → Execute Tasks → Wait for Events → Retry if Needed → Resume → Complete → Archive. Every workflow follows this lifecycle.

Workflow Definition

Defines steps, rules, conditions and dependencies. Acts as the blueprint for the workflow.

Workflow State

Tracks current step, completed tasks, pending work, errors and results. State allows workflows to pause and resume safely.

Task Executor

Responsible for calling AI models, running MCP tools, executing APIs and triggering business logic. Each task executes independently.

Scheduler

Determines when tasks execute, handles delayed execution, recurring jobs and time-based triggers. Useful for long-running business processes.

Event Manager

Listens for events such as approval received, payment completed or document uploaded. Events allow workflows to react dynamically and resume execution.

Sequential Workflow

Task A → Task B → Task C. Simple and predictable.

Parallel Workflow

Task A / Task B / Task C execute simultaneously → Merge Results. Reduces execution time.

Conditional Workflow

If Approved → Continue. If Rejected → Stop. Business rules determine workflow direction.

Loop Workflow

Review → Needs Changes? → Yes → Revise → Review Again. Useful for iterative processes.

State Machines

Many workflow engines use state machines. Example: Draft → Review → Approved → Published. Only valid transitions are allowed. State machines reduce workflow errors.

Checkpoints

Long workflows create checkpoints. Completed steps are recorded. If a failure occurs, execution resumes from the latest checkpoint instead of restarting from the beginning.

Retry Logic

Temporary failures happen. API Timeout → Retry. Network Failure → Retry. Model Unavailable → Switch Model. Permanent Failure → Escalate. Workflow engines distinguish temporary and permanent failures.

Human-in-the-Loop

Not every decision should be automated. Example: Generate Contract → Legal Review → Approved? → Continue. Human approval becomes another workflow step.

Long-Running Workflows

Example: Job Application → Resume Screening → Interview Scheduling → Technical Assessment → Manager Approval → Offer Letter → Employee Onboarding. Execution may span several weeks. Workflow engines manage this complexity.

A Runnable Workflow Skeleton with Checkpointing

The example below implements a durable, resumable workflow using a simple SQLite-backed checkpoint store: the same idea a production system implements with Temporal or a durable-execution framework, just without the distributed-systems machinery. It shows checkpoint persistence, resumption after a simulated crash, and a retry-then-escalate policy.

import sqlite3
import json
import time
import random


class CheckpointStore:
    """Minimal durable checkpoint store backed by SQLite."""

    def __init__(self, db_path=":memory:"):
        self.conn = sqlite3.connect(db_path)
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS checkpoints (
                workflow_id TEXT PRIMARY KEY,
                current_step TEXT,
                state_json TEXT,
                updated_at REAL
            )
        """)

    def save(self, workflow_id: str, step: str, state: dict):
        self.conn.execute(
            "INSERT OR REPLACE INTO checkpoints VALUES (?, ?, ?, ?)",
            (workflow_id, step, json.dumps(state), time.time()),
        )
        self.conn.commit()

    def load(self, workflow_id: str):
        row = self.conn.execute(
            "SELECT current_step, state_json FROM checkpoints WHERE workflow_id = ?",
            (workflow_id,),
        ).fetchone()
        if row is None:
            return None, {}
        step, state_json = row
        return step, json.loads(state_json)


class TransientAPIError(Exception):
    pass


def extract_invoice_fields(pdf_path: str) -> dict:
    if random.random() < 0.3:  # simulated 30% transient failure rate
        raise TransientAPIError("OCR service timeout")
    return {"vendor": "Acme Corp", "amount": 4200.00, "due_date": "2026-08-01"}


def validate_invoice(fields: dict) -> bool:
    return fields["amount"] > 0 and "vendor" in fields


def classify_expense(fields: dict) -> tuple[str, float]:
    return "software_licensing", 0.94


STEPS = ["extract", "validate", "classify", "branch", "post"]


def run_invoice_workflow(workflow_id: str, pdf_path: str, store: CheckpointStore, max_retries=3):
    step, state = store.load(workflow_id)
    start_index = STEPS.index(step) if step else 0

    for i in range(start_index, len(STEPS)):
        current = STEPS[i]

        if current == "extract" and "fields" not in state:
            for attempt in range(1, max_retries + 1):
                try:
                    state["fields"] = extract_invoice_fields(pdf_path)
                    break
                except TransientAPIError:
                    if attempt == max_retries:
                        state["status"] = "escalated"
                        store.save(workflow_id, "extract_failed", state)
                        return state
                    time.sleep(0.2 * attempt)  # backoff

        elif current == "validate":
            if not validate_invoice(state["fields"]):
                state["status"] = "rejected"
                store.save(workflow_id, "validate_failed", state)
                return state

        elif current == "classify":
            category, confidence = classify_expense(state["fields"])
            state["category"], state["confidence"] = category, confidence

        elif current == "branch":
            needs_human = state["confidence"] < 0.9 or state["fields"]["amount"] >= 5000
            state["needs_human_approval"] = needs_human
            if needs_human:
                state["status"] = "waiting_approval"
                store.save(workflow_id, "branch", state)
                return state  # pause here; a separate process resumes after approval

        elif current == "post":
            state["status"] = "posted_to_erp"

        store.save(workflow_id, current, state)  # checkpoint after every step

    return state


if __name__ == "__main__":
    store = CheckpointStore()
    result = run_invoice_workflow("inv-8841", "/invoices/8841.pdf", store)
    print(result)
    # Simulate a crash and resume: a fresh process just re-runs the same
    # function with the same workflow_id, and it picks up from the last
    # saved checkpoint instead of re-extracting the invoice from scratch.
    resumed = run_invoice_workflow("inv-8841", "/invoices/8841.pdf", store)
    print(resumed)

The mechanism that matters here is store.load(workflow_id) at the top: a fresh process invocation, after a real crash, a deploy, or a scheduled restart, looks up exactly where the workflow left off and continues from there, rather than re-running extract_invoice_fields (which costs money and time) a second time. That is the entire value proposition of "checkpointing" made concrete: it turns a crash from "start over" into "resume."

Failure Modes in Workflow Engines

• Checkpoint granularity too coarse. If a workflow only checkpoints at the very end, a crash at step 4 of 5 loses all prior work, equivalent to not checkpointing at all. If it checkpoints after every sub-operation inside a single step, the overhead of constant database writes can dominate execution time. The invoice pipeline above checkpoints once per logical step (extract, validate, classify...), which is usually the right granularity: coarse enough to avoid excessive I/O, fine enough that a crash loses at most one step's work.
• Non-idempotent side effects on retry. If post (posting to the ERP) runs twice because a checkpoint wasn't saved before a crash but the ERP call itself succeeded, the invoice gets posted twice, a duplicate payment. Every step with an external side effect needs to be idempotent (safe to run twice) or protected by a "have I already done this?" check, not just retried blindly.
• Long-running workflows outliving their code. A workflow paused at waiting_approval for three weeks resumes into a codebase where the classify_expense function's signature has changed. Workflow versioning exists specifically so in-flight workflows keep running against the code version they started with, rather than crashing against a newer, incompatible version.
• Silent stalls with no expiry. A workflow enters waiting_approval and the approver never acts: no reminder, no timeout, no escalation. Weeks later, someone discovers a backlog of stalled invoices. Every human-in-the-loop step needs an explicit expected-duration SLA and an alert when it's exceeded.
• State bloat. Storing every intermediate LLM response, full PDF contents, and verbose logs directly in the workflow state (rather than references to externally stored blobs) makes the state object grow until checkpointing itself becomes slow. Store large artifacts externally and keep only references and summaries in the workflow state.

Trade-offs: Choosing a Workflow Engine

The honest guidance: start with a hand-rolled checkpoint store (as in the code above) to prove the workflow logic is correct, then migrate to a managed engine like Temporal once you have more than a handful of long-running workflows or once "we lost a workflow's state during a deploy" has happened at least once in production.

Case study: DoorDash's order-fulfillment workflows on Temporal

DoorDash has published engineering blog posts describing their migration of core order-fulfillment logic (matching orders to dashers, handling cancellations, retrying failed payment captures) onto Temporal, citing the need for workflows that could span minutes to hours, survive service restarts and deploys without losing state, and be paused waiting on external events (a dasher accepting a delivery) without the engineering team hand-rolling retry and state-persistence logic for every new workflow. Their stated motivation mirrors the failure modes above almost exactly: before adopting a durable-execution engine, transient failures in downstream services (payment processors, mapping APIs) could silently drop in-flight orders, and engineers had built ad hoc, inconsistent retry logic per service rather than one reliable, auditable execution model. The broader pattern, companies operating at meaningful scale converging on a dedicated workflow engine once ad hoc retry-and-state code becomes unmaintainable, recurs across DoorDash, Netflix (which built its own Conductor engine for similar reasons), and Uber (Cadence, Temporal's predecessor, originated there).

Enterprise Example

Employee Expense Claim: Employee Uploads Receipt → OCR Extraction → Policy Validation → Budget Verification → Manager Approval → Finance Approval → Payment → Notification → Archive. Every step is tracked and failures are recoverable.

Workflow Monitoring

Track running workflows, failed workflows, average duration, retry count, waiting approvals and completion rate. Monitoring keeps workflows healthy.

A useful monitoring baseline is a small dashboard answering four questions at a glance: how many workflows are currently in each state (running, waiting on human approval, failed, completed), what is the age distribution of anything stuck in a waiting state (flagging outliers that have waited far longer than the typical SLA), what fraction of steps required a retry over the last day (a rising retry rate is often the earliest signal of a downstream API degrading before it fully fails), and how many workflows escalated to a human versus completed automatically (a sudden jump usually means an upstream change, such as a stricter validation rule or a lower confidence threshold, needs review).

Workflow Versioning

Business processes evolve. Existing workflows often continue using their original definition while new workflows use the latest version. This prevents unexpected behavior.

Popular Workflow Engines

Examples include LangGraph, Temporal, Apache Airflow, Prefect, Azure Durable Functions and AWS Step Functions. Each has different strengths depending on the problem being solved.

Best Practices

Keep workflows modular. Use checkpoints. Separate workflow logic from business logic. Design for failure. Monitor execution continuously. Version workflows carefully. Log every major transition.

Common Mistakes

Creating excessively large workflows. Ignoring retries. Losing workflow state. Mixing orchestration and business rules. Forgetting monitoring. Hardcoding workflow logic.

Hands-On Exercise

Design a workflow for an AI-powered recruitment process. Include resume upload, resume analysis, candidate ranking, interview scheduling, human approval, offer generation and employee onboarding. Identify workflow states and checkpoints.

Mini Project

Create a workflow engine design for an enterprise procurement process. Include workflow definition, state machine, retry strategy, human approvals, monitoring and versioning. Draw the complete workflow diagram, and use the engine trade-off table above to justify your specific engine choice.

Worked Example: The Invoice Pipeline

An accounts-payable workflow engine processing 3,000 invoices/month:

1. Extract (LLM): PDF → structured fields (vendor, amount, due date). Deterministic schema check follows (AI-026): malformed output retries once, then routes to a human queue.
2. Validate (code, no AI): vendor exists, PO matches, math adds up. Pure rules; no model needed or wanted (AI-003's "wrong tool" lesson).
3. Classify (LLM): expense category, with confidence score.
4. Branch: confidence > 0.9 and amount < $5,000 → auto-approve; otherwise → human review (AI-053).
5. Post (tool call): approved invoices enter the ERP; every step logged with the workflow run ID (AI-028).

The engine guarantees what agents can't (AI-017): every invoice takes exactly this path, retries are automatic, a crashed step resumes where it stopped, and the auditor can replay any run. AI does two boxes; the engine does the reliability.

Try It Yourself

1. Classify your steps: take any process you run and label each step LLM / code / human. The discipline of asking "does this step really need a model?" halves most AI workflow designs, and their costs (AI-039).
2. Find the resume point: for your workflow, decide what happens if step 3 crashes at 2 a.m. Restart from scratch, or resume from stored state? Durable state is the feature that separates workflow engines from scripts.

Key Takeaways

• Workflow engines reliably execute complex AI processes.
• State management enables pause and resume.
• Checkpoints improve resilience.
• Retries handle temporary failures.
• Workflow engines are essential for enterprise-scale AI platforms.