Distributed Locking

Overview

Distributed locking prevents the same critical business operation from running concurrently across multiple application instances.

TeaQL applications commonly use distributed locks for:

• Scheduled jobs
• Order submission
• Payment confirmation
• Inventory deduction
• Idempotent callbacks
• Large data import/export
• Schema or index preparation

CustomUserContext is the recommended place to expose project-specific lock behavior because it knows the user, tenant, request, and infrastructure environment.

---

Local Lock vs Distributed Lock

Lock Type	Scope	Use Case
Local lock	One JVM instance	Protect in-memory work inside one process
Distributed lock	All service instances	Protect business operations in clustered deployments

If your service can run more than one instance, use a distributed lock for business-critical operations.

---

Recommended Design

Keep lock provider details outside business logic:

Business method
  -> userContext.withDistributedLock(...)
  -> lock service
  -> Redis / database / ZooKeeper / etcd / custom provider

Example lock service:

public interface DistributedLockService {

    <T> T executeWithLock(String key, Duration timeout, Supplier<T> task);

    void executeWithLock(String key, Duration timeout, Runnable task);
}

Expose it through CustomUserContext:

public class CustomUserContext extends UserContext {

    public DistributedLockService distributedLockService() {
        return getBean(DistributedLockService.class);
    }

    public void withDistributedLock(String key, Runnable task) {
        distributedLockService().executeWithLock(key, Duration.ofSeconds(30), task);
    }
}

---

Lock Key Design

A lock key must identify the exact business resource being protected.

Good examples:

tenant:order:submit:ORD-2026-000001
tenant:payment:callback:PAY-893020
tenant:inventory:sku:SKU-10086
global:schema:ensure

Bad examples:

lock
order
job

Always include enough scope:

• Tenant
• Operation
• Entity type
• Business ID or internal ID
• Optional region or shard

---

Example: Protect Order Submission

public WebResponse submitOrder(CustomUserContext userContext, String orderId) {
    String lockKey = userContext.currentTenantCode() + ":order:submit:" + orderId;

    userContext.withDistributedLock(lockKey, () -> {
        Order order = Q.orders().filterById(orderId).comment("Query orders").purpose("Load data").executeForOne(userContext);
        order.submit(userContext);
        order.auditAs("Save order").save(userContext);
    });

    return WebResponse.success();
}

This prevents duplicate submission when the user double-clicks, retries, or when two service instances receive the same command.

---

Timeout and Failure Policy

Every distributed lock needs a clear timeout policy.

Common choices:

Policy	Behavior
Fail fast	Return error immediately if lock cannot be acquired
Wait with timeout	Wait for a short period, then fail
Skip	Used for scheduled jobs where another instance is already running
Retry	Retry with backoff for recoverable workflows

Example:

public void runDailySettlement(CustomUserContext userContext) {
    userContext.distributedLockService().executeWithLock(
            "global:settlement:daily",
            Duration.ofMinutes(5),
            () -> SettlementUtil.runDailySettlement(userContext)
    );
}

---

Idempotency

Locks are not a replacement for idempotency.

For critical integrations, use both:

• Distributed lock to prevent concurrent execution
• Idempotency record to prevent duplicate effects after retries
• Transaction boundary to keep state consistent

Example fields:

• External request ID
• Callback ID
• Operation hash
• Processed time
• Result status

---

Best Practices

• Use business-specific lock keys.
• Include tenant scope in lock keys unless the operation is truly global.
• Keep locked sections short.
• Do not call slow external services while holding a lock unless necessary.
• Always release locks with finally behavior inside the provider.
• Use timeout and retry policies intentionally.
• Combine locks with database constraints for critical uniqueness.

---

Testing Checklist

• Two concurrent requests for the same key cannot both execute the critical section.
• Different tenants do not block each other.
• Lock timeout returns the expected error.
• Failed operations release the lock.
• Scheduled jobs run on only one instance.
• Retry behavior does not create duplicate business effects.

---

Summary

Distributed locking is a project-level infrastructure concern. TeaQL keeps it clean by routing lock behavior through CustomUserContext, letting business code express the protected operation while the project controls the actual lock provider and failure policy.