1. Problem and Goals

We maintain a self-service change system built on “configuration as code”. After a user submits a change request, the system has to complete the following steps in order:

  1. Submit an approval to an external approval system and wait for the result callback;
  2. Using a bot account, create a branch in the Git repository, commit the files, and open a Merge Request (MR);
  3. Wait for the MR’s check pipeline to pass;
  4. Approve and merge the MR as the approval bot, rebasing and retrying if the merge fails;
  5. Wait for the deployment pipeline to roll out the configuration and report back via callback that it is “in effect”.

A single change spans three external systems (the approval system, the Git hosting platform GitLab, and CI) and takes anywhere from a few minutes to a few hours. Handling it as an ordinary synchronous request runs into several problems: an HTTP connection cannot wait that long; a failure at any step can leave the request stuck in an intermediate state nobody knows about; callbacks from external systems may be duplicated, late, or out of order; and a process restart loses whatever step was executing.

We set four goals for this workflow:

  • Observable: at any moment, a request is in an explainable state, and you can tell which step it stopped at.
  • Failures are visible: a failure must end up as an explicit failure state, never as a request that is “processing” forever.
  • Robust to events: duplicate events, late events, and events belonging to an old request must not change the state of the current request.
  • Evolvable: changing the order of steps or adapting to upstream API changes must not interrupt in-flight requests.

2. Constraints and Considerations

External calls are outside the local transaction. After creating the branch, commit and MR on GitLab, the local database write can still fail; conversely, once local state is written, the remote call can still fail. There is no atomicity between the two sides to rely on.

Many event sources, with weak delivery semantics. State changes are driven by user submissions, approval callbacks, the system’s own progression, deployment pipeline callbacks, scheduled reconciliation, and manual recovery by operators. External callbacks are delivered “at least once”; they may arrive late, or even before our own state write lands.

Upstream responses are ambiguous. GitLab returns HTTP 401 when the same account approves twice, which is the same status code as “this account is not allowed to approve”. The response alone cannot distinguish the two.

Deployment topology. The service runs as multiple instances. We did not want to introduce new components such as a workflow engine or distributed locks for this one workflow; the only shared coordination point we have is the relational database.

The workflow must evolve. The step order needed to change from “create the MR, then approve” to “approve, then create the MR”, with approvals already in flight at switchover time. The upstream platform also planned to add a required structured field to its approval API, with a compatibility period.

3. Design

3.1 Two layers of state: step state and external state

We record state at two granularities in two tables:

  • The operation table records step state, down to details such as “pipeline timed out”, “rebase conflict” or “bot approval failed”. Status codes are allocated in ranges by phase (creation from 10001, handling from 10101, closing from 10201), so metrics can be aggregated by prefix.
  • The binding table records external state, expressing only the phases users care about: approval pending, MR being created, merging, merged, in effect, failed, rejected. The table has a unique key on (subject_id, target_key), so each “subject–target” relationship has exactly one row.
enum StepStatus {            // Fine-grained: for troubleshooting and metrics
    CREATE_STARTED(10001), CREATE_SUCCEEDED(10002), CREATE_PIPELINE_FAILED(10004),
    HANDLE_STARTED(10101), HANDLE_APPROVED(10102), HANDLE_REBASE_CONFLICT(10105),
    HANDLE_PIPELINE_PENDING(10501), CALLED_BACK(10301), /* ... */;
}

enum BindingStatus {         // Coarse-grained: for display and "can the user re-apply"
    APPROVAL_SENDING, APPROVAL_SENT, MR_CREATING, MR_CREATED, MERGING, MERGED, VALIDATED,
    CREATE_FAILED, HANDLE_FAILED, REJECTED, CLOSED;

    /** Single source of truth: both the data layer and the API layer use it to decide whether re-applying is allowed. */
    static final Set<BindingStatus> REAPPLIABLE = Set.of(REJECTED, CLOSED);
}

We split state into two layers because the two audiences need different things. Troubleshooting needs to know exactly which step a request stopped at; users, and the “can I re-apply” decision, only care about the broad phase. With a single status field, every refinement of the internal steps would ripple into the frontend display and the re-apply rules.

Every state transition writes both tables in the same local transaction:

void transition(OperationBuilder op, StepStatus step, BindingStatus status) {
    tx.execute(List.of(
        () -> operationRepo.update(op.status(step)),          // Optimistic concurrency check against the state just read
        () -> bindingRepo.updateStatus(op.id(), status)
    ));
}

The diagram below shows the main path of the external state (failure branches omitted):

stateDiagram-v2
    [*] --> APPROVAL_SENT: Request submitted
    APPROVAL_SENT --> REJECTED: Approval rejected or withdrawn
    APPROVAL_SENT --> MR_CREATING: Approval granted
    MR_CREATING --> MR_CREATED: Check pipeline passed
    MR_CREATING --> CREATE_FAILED
    MR_CREATED --> MERGING
    MERGING --> MERGED: Merge succeeded
    MERGING --> HANDLE_FAILED
    MERGED --> VALIDATED: Deploy callback or scheduled reconciliation
    REJECTED --> [*]
    VALIDATED --> [*]

3.2 Asynchronous entry point; every failure is persisted

The submit endpoint performs only synchronous validation (parameters, idempotency, whether a request is already in progress), then hands the long-running workflow to a bounded thread pool and immediately returns HTTP 202 with an operationId. The client polls by operationId.

private void fireAsync(String opId, Task task, StepStatus failStep, BindingStatus failStatus) {
    try {
        executor.execute(() -> {
            try {
                task.run();
            } catch (Exception e) {
                log.error("async step failed", e);        // The failure state has already been written by the aspect; see below
            } catch (Throwable t) {
                log.error("[CRITICAL] JVM-level error", t); // The JVM may be in a bad state; log only, do not write to the DB
            }
        });
    } catch (RejectedExecutionException e) {
        // Pool is full or shutting down: persist the failure so pollers see "failed" rather than "processing"
        tx.execute(List.of(
            () -> operationRepo.updateStatus(opId, failStep),
            () -> bindingRepo.updateStatus(opId, failStatus)));
    }
}

The pool’s core size equals its maximum size, the queue is bounded, and the rejection policy is AbortPolicy. We did not choose the two other common options, for these reasons:

  • CallerRunsPolicy would make a web thread run a task lasting several minutes, propagating overload to every endpoint.
  • An unbounded queue hides overload: tasks may wait in the queue for a long time while users keep seeing “processing”.

AbortPolicy combined with “persist on rejection” turns overload into a failure state that is visible to users and can be counted.

Each step method declares its own failure state through an annotation, and an aspect provides a uniform safety net:

@WorkflowStep(onErrorStep = StepStatus.HANDLE_MERGE_FAILED,
              onErrorStatus = BindingStatus.HANDLE_FAILED)
public Result handleMergeRequest(String opId, long mrIid) { ... }

Failures recognized inside the method (such as pipeline failure or rebase conflict) are written by business code with a more specific state; unexpected exceptions are written by the aspect with the state declared in the annotation. Together they guarantee that every exit path leaves a terminal state behind.

3.3 Callback guard: accept only events for the “current owner” that is “currently waiting”

Approval callbacks are handled in two layers. The first layer deduplicates by the request ID carried in the callback itself, blocking duplicate deliveries. The second layer decides whether the callback is still valid:

Binding row = bindingRepo.findByOperationId(callback.operationId());
if (row == null || !AWAITING_APPROVAL.contains(row.status())) {
    return accepted("IGNORED_STALE");   // Return 2xx so the upstream does not treat it as a failure and keep retrying
}

There are two design points here.

  • Ownership is decided by operationId. When a user re-applies, the binding row is reset in place and given a new operationId (see the database post). A late callback for the old request therefore finds no row and is ignored, instead of pushing the new request into the wrong state.
  • The waiting set includes “sending”. AWAITING_APPROVAL = {APPROVAL_SENDING, APPROVAL_SENT}. We write SENDING before calling the approval API and SENT after the call succeeds. The approval system’s callback can arrive before SENT is written; if the waiting set contained only SENT, such callbacks would be misjudged as stale.

3.4 Idempotency based on remote state

Because local and remote state are not atomic, every step that modifies remote state first reads the remote state before acting:

  • Before handling an MR, if the MR is already merged, we finish directly as “merged” without approving again. Otherwise the re-approval would be rejected by GitLab, the recorded result would be “approval failed”, and troubleshooting would be steered toward a permissions problem.
  • Before closing an MR, if it is already closed or merged, we skip the close call.

The bot approval step needs extra handling for the 401 ambiguity:

void approveAsReviewer(Api reviewer, long mrIid) throws ApiException, InterruptedException {
    int maxRetries = Math.max(0, config.getInt("approval.maxRetries", 3)); // Negative values are treated as 0
    ApiException last = null;
    for (int attempt = 1; attempt <= maxRetries + 1; attempt++) {
        if (hasApprovalFrom(reviewer, mrIid)) {
            return;                                   // Already approved: skip the call
        }
        try {
            approver.approve(reviewer, mrIid);
            return;
        } catch (ApiException e) {
            last = e;
            if (attempt > maxRetries) break;
            Thread.sleep(backoffBaseMs << (attempt - 1)); // Exponential backoff
        }
    }
    // Both duplicate approval and unauthorized approval return 401. Read once more to check whether the approval actually exists.
    if (hasApprovalFrom(reviewer, mrIid)) {
        return;
    }
    throw last;
}
  • Read the approver list before every attempt. If the previous approval request reached GitLab but the response was lost, the next attempt sees that the approval already exists instead of receiving another 401.
  • Read once more after the final failure. After the last attempt there is no next attempt to correct things, and this read covers that case.
  • The read is fail-open. If the approver list cannot be read, we treat it as “not yet approved” and go ahead with the approval. This preserves the behavior from before the check was introduced, so the check itself does not become a new source of failure.
  • The two approval accounts are retried separately. The two approvals are not one atomic operation; retrying them as a pair would re-submit the approval for the first account.

These checks make “re-run from where it stopped” safe, so the operations recovery tool can simply re-trigger a given step.

3.5 Waiting on external state: tiered polling, distinguishing “timeout” from “failure”

Both the rebase result and the pipeline status have to be polled. We abstracted polling into a generic component: short intervals for the first N attempts, longer intervals afterward, and a timeout once the limit is reached. A poll has three possible outcomes: passed, failed, timed out.

  • Pipeline failure is deterministic: retrying with the same MR will fail again, so the request goes into a failure state.
  • Pipeline timeout means the result is still unknown. The external state stays at “merging”, the step state is recorded as “pipeline pending”, and a human confirms the outcome.

The API layer uses this to show users different hints: a deterministic failure says “contact the maintainers”, while an unknown outcome says “you can keep waiting” (see the API post for details).

3.6 Changing the step order online, without draining

When we changed the workflow from “create the MR, then approve” to “approve, then create the MR”, we did not schedule downtime or a drain window. Instead, both orders coexisted for a period of time:

  1. The new workflow is controlled by a configuration switch, off by default, and can be switched back to the old workflow at any time.

  2. Callbacks identify the workflow by payload shape. In the old workflow, the MR already exists when the approval is submitted, and its number is put into an extension field of the approval form; in the new workflow, there is no MR yet when the approval is submitted. Callback handling decides accordingly:

    boolean legacyFlow = StringUtils.isNotEmpty(callback.ext().get("mrId"));
if (approved) {
    if (legacyFlow) workflow.fireHandle(opId, mrId);      // MR already exists: approve and merge
    else            workflow.fireCreateAfterApproval(opId); // Create the MR first
}

    Approvals sent before the switch was flipped but called back afterward are still handled by the old workflow.

  3. The create step queries the database to decide between “insert” and “advance”, rather than relying on a flag passed by the caller:

    if (operationRepo.exists(opId)) {
    transition(op, CREATE_STARTED, MR_CREATING);   // New workflow: the row was created when the approval was submitted
} else {
    insertOperationAndBinding(op);                 // Old workflow: first-time creation
}

    This makes the same step valid for both orders, and keeps it idempotent when operators re-trigger it.

3.7 Scheduled reconciliation for lost callbacks

Callbacks from the deployment pipeline can be lost, leaving the corresponding requests stuck at “merged”. A scheduled job converges such records.

Sliding time window. Each run scans only “merged” records whose last-modified time falls within (now - maxAge, now - minAge). Records newer than minAge may simply have their callback still on the way, so they are skipped; records older than maxAge fall out of the window, which bounds the work per run.

A watermark instead of per-row lookups. The callback table records only MR numbers, with no business fields. The reconciliation job runs just two queries: one IN query that fetches operationId → mrIid in bulk, and one that fetches the largest MR number in the callback table. Then it does one comparison per row:

mrIid <= maxCallbackIid  → advance to "in effect"
mrIid >  maxCallbackIid  → stay "merged", wait for the next run

mrIid == max means this MR’s own callback has arrived; mrIid < max means an MR with a larger number has already been called back, and since MR numbers increase monotonically within a project, we infer that the callback for this earlier MR was lost. This inference holds on the condition that the deployment pipeline runs in merge order, and a later run never skips changes merged before it. The minAge window prevents “late” from being mistaken for “lost”.

Takes effect exactly once when racing with the normal callback path. Both the callback handler and the reconciliation job call the same state-advancing method:

@Transactional
public String markValidated(String opId) {
    Operation op = operationRepo.selectForUpdate(opId);  // Fixed lock order: operation first
    Binding   b  = bindingRepo.selectForUpdate(opId);    // then binding
    if (b == null || (b.status() != MERGED && b.status() != VALIDATED)) {
        return null;                                     // Preconditions not met: idempotent no-op
    }
    if (op.status() != CALLED_BACK) {
        operationRepo.updateStatus(opId, CALLED_BACK);   // Repair a step state that may be lagging
    }
    if (b.status() == MERGED) {
        requireOneRow(bindingRepo.update(
            "SET status = VALIDATED WHERE operation_id = ? AND status = MERGED", opId));
        return b.subjectId();                            // Non-null only when the state actually flipped
    }
    return null;
}

When the two paths race, row locks ensure only one of them completes the flip; the other reads VALIDATED and returns null. Callers count a “validated successfully” metric only when the return value is non-null, and emit it only after the outer transaction commits, so a rollback cannot leave an over-counted metric and concurrency cannot cause double counting.

3.8 Adapting to upstream API changes

The upstream platform planned to add a required field to its approval API: a list of check items that must each be acknowledged. The GitLab client library we use cannot express this field. Our approach:

  • Funnel the approval action into a dedicated component. It has two implementations: the existing one using the client library, and a new one that sends the HTTP request directly with the check items. A configuration switch is read on every call to choose between them, so switching requires no restart.
  • Do not hard-code the check items. The list is read from the upstream configuration API and cached per project.
  • Self-heal using 422 responses. When a project’s check-item configuration changes, a request built from the stale cache receives a 422, and the 422 response body contains the list currently in effect. The component rebuilds the request from that list and retries once, saving an extra read round trip.
  • Keep exception semantics unchanged. Both implementations report failures using the client library’s original exception type, carrying the real HTTP status code. The caller’s retry loop, 401 ambiguity handling and logging all stay unchanged.

3.9 Why we did not choose a workflow engine

A mature workflow engine provides durable execution, automatic retries and crash recovery, which are exactly the parts this design implements by hand. We did not adopt one because the workflow has a limited number of steps; the only shared infrastructure the team already has is the relational database; and the genuinely hard parts (ambiguous external responses, deciding when a callback is stale, idempotency based on remote state) are business semantics that an engine cannot solve for us. The cost is that we have to fill in some durable-execution capabilities ourselves; see Section 5.

4. Results

  • State is queryable. Every request can be looked up by operationId to see its phase and the exact step it stopped at; troubleshooting no longer requires reconstructing the process from logs.
  • Failures are visible. Thread pool overload, step exceptions and pipeline failures all end up as explicit failure states, never as “processing” forever.
  • Robust to events. Duplicate deliveries, late callbacks and callbacks belonging to an old request do not change the current request’s state.
  • Safe re-entry. When operators re-trigger a step from where it stopped, already-merged requests finish directly and already-approved accounts skip approval, without creating new failures.
  • The workflow can evolve. Changing the step order is done through configuration, and in-flight approvals did not need to be drained during the release; upstream API changes are switched by a flag, with no changes for callers and no restart needed.
  • Lost callbacks converge automatically. Scheduled reconciliation advances stuck records to a terminal state and guarantees the success metric is counted exactly once.

5. Known Limitations and Next Steps

  • Valid transitions are not defined centrally. The optimistic concurrency check compares against “the state just read”, which prevents lost updates but not illegal from→to transitions. Validity is currently scattered across entry guards, the re-appliable set and the preconditions of the advancing methods. We plan to introduce a central transition table that every write is validated against.
  • Execution state is not durable. The long-running workflow runs in an in-process thread pool, and waiting is implemented with thread sleeps. If the process crashes mid-step, the record stays “in progress”. The batch path already has a timeout sweep; the single-request path currently relies on the operations tool for recovery. We plan to add timeout sweeps for every “in progress” state, or to make waiting scheduler-driven.
  • Reconciliation relies on inference. The watermark approach depends on monotonic MR numbers and the deployment pipeline’s execution semantics, and since the callback table has no business dimension, the comparison is global. If callbacks could carry a business identifier, we could switch to exact matching.
  • The workflow code lives in one large class. This raises the cost of auditing the state machine as a whole; we plan to split it by phase.