Distributed Scheduler (Part 4) | ShedLock

The Building Manager Analogy

A large apartment building has 20 caretakers on duty at all times, but only one of them is responsible for the weekly boiler inspection. The building manager does not stop the other 19 caretakers and make them wait outside the boiler room — that would be wasteful. Instead, whoever arrives at the boiler room first picks up the "on duty" badge from the hook by the door and does the inspection alone. Anyone else who arrives and sees the badge is missing simply walks away and continues their other duties. The badge is returned after the inspection, or after a maximum permitted time — whichever comes first.

ShedLock is exactly this mechanism for distributed schedulers. It does not lock database rows. It does not block other pods. It grants one instance an exclusive lease on a scheduler job and instructs all others to skip that job entirely.

---

The Problem ShedLock Solves

As covered in the previous chapter, when a Spring Boot application runs as multiple pods, every pod has the same code and the same scheduler. At 1 AM, all 10 pods fire. Previous solutions (pessimistic lock, skip locked) still run all instances simultaneously, just on different data rows.

ShedLock addresses a different requirement: only one pod should run this scheduler job; the other nine should skip it completely and remain free for other work.

Approach	Behaviour	Best For
No deduplication	All pods run simultaneously → duplicates	Single pod only
Pessimistic lock	All pods run, row locks prevent duplication	Small deployments, strict ordering
Skip locked	All pods run in parallel on different rows	Large data, unordered, multiple pods
ShedLock	Only one pod runs; all others skip	Any case where a single executor is needed

---

The ShedLock Table

ShedLock creates and manages one table in your database:

CREATE TABLE shedlock (
    name       VARCHAR(64)  NOT NULL,   -- unique job name (primary key)
    lock_until TIMESTAMP(3) NOT NULL,   -- no other instance may run before this time
    locked_at  TIMESTAMP(3) NOT NULL,   -- when the current holder acquired the lease
    locked_by  VARCHAR(255) NOT NULL,   -- identity of the pod holding the lease
    PRIMARY KEY (name)
);

Critical detail: there is no real pessimistic lock on these rows. ShedLock uses optimistic concurrency — it relies on the atomicity of database INSERT and UPDATE operations to ensure only one pod wins the lease.

---

How ShedLock Works Internally

Proxy Creation

ShedLock intercepts every @Scheduled method annotated with @SchedulerLock and creates a proxy class around it. The proxy:

1. Performs pre‑processing (try to acquire the lease).
2. Calls the actual job if the lease was acquired.
3. Performs post‑processing (update the lock table).

Your class: SchedulerJob
               │
ShedLock creates: SchedulerJobProxy extends SchedulerJob
                       │
                   @Override myJob() {
                       preProcess();   // Try to acquire lease
                       super.myJob();  // Run actual job
                       postProcess();  // Release/update lease
                   }

The proxy job — not the actual job — is what enters the DelayedWorkQueue.

Step‑by‑Step Flow

Step 1 — Application startup

@EnableSchedulerLock triggers a scan for @SchedulerLock annotations and creates proxy jobs. Each proxy job is inserted into the DelayedWorkQueue. All worker threads enter TIMED_WAITING until the scheduled time arrives.

Step 2 — Scheduled time arrives (e.g. 9:00 AM)

Both Pod 1 and Pod 2 have the proxy job due at 9:00 AM. Their worker threads wake up and both attempt to execute the proxy. Pre‑processing begins on both pods simultaneously.

Step 3 — Pre‑processing: try to acquire the lease

First run (ShedLock table is empty):

Both pods try to INSERT into the ShedLock table:

-- Pod 1 attempt
INSERT INTO shedlock (name, lock_until, locked_at, locked_by)
VALUES ('myJob', NOW() + INTERVAL 8 MINUTE, NOW(), 'pod-1');

-- Pod 2 attempt (concurrent)
INSERT INTO shedlock (name, lock_until, locked_at, locked_by)
VALUES ('myJob', NOW() + INTERVAL 8 MINUTE, NOW(), 'pod-2');

Because name is the primary key, only one INSERT succeeds. The other receives a duplicate key error. The pod that fails skips the job entirely — it computes the next runtime, re‑queues the proxy job, and goes back to sleep. No blocking, no waiting.

Subsequent runs (row already exists):

Both pods try to UPDATE the row with a WHERE clause:

UPDATE shedlock
SET lock_until = NOW() + INTERVAL 8 MINUTE,
    locked_at  = NOW(),
    locked_by  = 'pod-X'
WHERE name        = 'myJob'
  AND lock_until <= NOW();   -- The lease must have expired for this pod to win

Again, the database guarantees atomicity — exactly one UPDATE affects one row. The pod whose UPDATE matches zero rows loses and skips the job.

Step 4 — Job execution (winning pod)

The winning pod calls the actual scheduler method. Post‑processing will run when it finishes.

Step 5 — Post‑processing: update lock_until

After the job finishes, the winning pod updates the table:

UPDATE shedlock
SET lock_until = MAX(NOW(), locked_at + INTERVAL 4 MINUTE)
WHERE name      = 'myJob'
  AND locked_by = 'pod-1';

This sets lock_until to the maximum of the current time or locked_at + lockAtLeastFor. The lockAtLeastFor parameter ensures no other pod can immediately pick up the job even if this pod finishes very quickly.

---

The lockAtLeastFor and lockAtMostFor Parameters

Parameter	Purpose	Stored in DB?
lockAtMostFor	Maximum time the lease is held — safety against hung jobs	Yes (as lock_until)
lockAtLeastFor	Minimum time the lease is held — prevents immediate re‑acquisition	No (configuration only)

lockAtMostFor must always be set. A pod that crashes mid‑execution would hold the lease forever without it. ShedLock uses lock_until to determine whether a lease has expired. If the maximum time passes and lock_until is in the past, any pod can acquire the job again.

lockAtLeastFor prevents a race condition where a fast job finishes in 1 second, releases the lease, and the same pod or another pod immediately picks it up on the next cycle before the distributed system has had time to propagate.

Rule of thumb:

• Set lockAtMostFor to at least twice the maximum expected job duration.
• Set lockAtLeastFor to the fixed rate / cron interval to prevent running more often than intended.

---

When Can Two Pods Run Simultaneously with ShedLock?

This is a critical interview question. If lockAtMostFor is set too small and a job runs longer than that value:

• Pod 1 acquires the lease at 9:00, lock_until = 9:08.
• Pod 1 is still running at 9:10 (took longer than 8 minutes).
• Pod 2 checks at 9:10: lock_until = 9:08 < now. Lease expired!
• Pod 2 acquires the lease and starts running.
• Both pods now run the same job concurrently.

Prevention: always set lockAtMostFor to comfortably exceed the maximum realistic job duration.

---

Complete Code Setup

Maven Dependencies

<dependency>
    <groupId>net.javacrumbs.shedlock</groupId>
    <artifactId>shedlock-spring</artifactId>
    <version>5.10.0</version>
</dependency>
<!-- Choose the provider matching your database -->
<dependency>
    <groupId>net.javacrumbs.shedlock</groupId>
    <artifactId>shedlock-provider-jdbc-template</artifactId>
    <version>5.10.0</version>
</dependency>

ShedLock Configuration Bean

import net.javacrumbs.shedlock.core.LockProvider;
import net.javacrumbs.shedlock.provider.jdbctemplate.JdbcTemplateLockProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.core.JdbcTemplate;
import javax.sql.DataSource;

@Configuration
public class ShedLockConfig {

    /**
     * Provides a LockProvider backed by a relational database via JdbcTemplate.
     * For Redis: use RedisLockProvider.
     * For MongoDB: use MongoLockProvider.
     */
    @Bean
    public LockProvider lockProvider(DataSource dataSource) {
        return new JdbcTemplateLockProvider(
            JdbcTemplateLockProvider.Configuration.builder()
                .withJdbcTemplate(new JdbcTemplate(dataSource))
                .usingDbTime()  // Use DB clock, not JVM clock — avoids clock skew
                .build()
        );
    }
}

Application Main Class

import net.javacrumbs.shedlock.spring.annotation.EnableSchedulerLock;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableScheduling;

@SpringBootApplication
@EnableScheduling
@EnableSchedulerLock(defaultLockAtMostFor = "PT10M")  // Default max 10 minutes
public class SchedulerApplication {
    public static void main(String[] args) {
        SpringApplication.run(SchedulerApplication.class, args);
    }
}

Schema SQL (for H2 / PostgreSQL)

-- schema.sql (auto-executed on startup for H2)
CREATE TABLE IF NOT EXISTS shedlock (
    name       VARCHAR(64)  NOT NULL,
    lock_until TIMESTAMP(3) NOT NULL,
    locked_at  TIMESTAMP(3) NOT NULL,
    locked_by  VARCHAR(255) NOT NULL,
    PRIMARY KEY (name)
);

Annotating the Scheduler Job

import net.javacrumbs.shedlock.spring.annotation.SchedulerLock;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.time.LocalDateTime;

@Component
public class SubscriptionExpiryJob {

    /**
     * fixedRate: 5 minutes between runs.
     * name:          unique job identifier (stored as PK in shedlock table).
     * lockAtMostFor: maximum 8 minutes — prevents stuck pods from blocking forever.
     * lockAtLeastFor: minimum 4 minutes — prevents immediate re-acquisition.
     */
    @Scheduled(fixedRate = 300_000)  // 5 minutes in milliseconds
    @SchedulerLock(
        name            = "subscriptionExpiryJob",
        lockAtMostFor   = "PT8M",
        lockAtLeastFor  = "PT4M"
    )
    public void run() {
        System.out.println("Running subscription expiry at: " + LocalDateTime.now());
        // Only ONE pod will reach this line. Others have already skipped.
    }
}

Custom Table and Column Names

If your organisation's schema standards require different names:

@Bean
public LockProvider lockProvider(DataSource dataSource) {
    return new JdbcTemplateLockProvider(
        JdbcTemplateLockProvider.Configuration.builder()
            .withJdbcTemplate(new JdbcTemplate(dataSource))
            .withTableName("distributed_job_locks")
            .withColumnNames(
                new JdbcTemplateLockProvider.ColumnNames(
                    "job_name", "expiry_time", "acquired_at", "acquired_by"
                )
            )
            .usingDbTime()
            .build()
    );
}

---

The Optimistic Concurrency Model Visualised

9:00 AM — both pods wake up

Pod 1: pre-process
  INSERT INTO shedlock VALUES ('myJob', 9:08, 9:00, 'pod-1') ← SUCCESS
Pod 2: pre-process
  INSERT INTO shedlock VALUES ('myJob', 9:08, 9:00, 'pod-2') ← FAIL (PK conflict)
  → Pod 2 skips job, re-queues for 9:05

Pod 1: runs actual job (finishes at 9:03)
  post-process: UPDATE lock_until = MAX(9:03, 9:00+4min) = 9:04

9:05 AM — both pods wake up again

Pod 1 & Pod 2: is lock_until (9:04) ≤ now (9:05)? YES
  Both try: UPDATE shedlock SET lock_until=9:13, locked_at=9:05, locked_by=?
            WHERE name='myJob' AND lock_until <= now
  One UPDATE affects 1 row (atomic), the other affects 0 rows.
  Winner runs the job. Loser skips and re-queues for 9:10.

---

Multiple ShedLock Providers

Database	Provider Dependency
MySQL / PostgreSQL / H2	shedlock-provider-jdbc-template
Redis	shedlock-provider-redis-spring
MongoDB	shedlock-provider-mongo
DynamoDB	shedlock-provider-dynamodb
Cassandra	shedlock-provider-cassandra

All providers implement the LockProvider interface — your scheduler code does not change when switching databases.

---

Interview Questions & Pitfalls

Q1: How does ShedLock prevent multiple pods from running the same job simultaneously?

A: ShedLock uses the atomicity of database operations. For first‑run acquisition it uses INSERT; for subsequent runs it uses UPDATE ... WHERE lock_until <= NOW(). Because both operations are atomic at the database level, exactly one pod succeeds. The losing pod sees no affected rows and skips the job without waiting. This is optimistic concurrency — no real database locks on rows are held.

---

Q2: What happens if the pod holding the ShedLock lease crashes mid‑execution?

A: The lock_until column (set from lockAtMostFor) acts as a safety timeout. Once lock_until passes, any other pod can acquire the lease on the next cycle. Without lockAtMostFor, a crashed pod would hold the lease forever. This is why lockAtMostFor is mandatory — @EnableSchedulerLock has a defaultLockAtMostFor that acts as a fallback for any job that omits it.

---

Q3: Can two instances still run the same ShedLock‑protected job simultaneously?

A: Yes, in one specific scenario: if lockAtMostFor is shorter than the actual job duration. Example — lockAtMostFor = 8 minutes, job takes 10 minutes. Pod 1 acquires at 9:00 with lock_until = 9:08. Pod 1 is still running at 9:10. Pod 2 checks: lock_until = 9:08 < now = 9:10 — Pod 2 acquires the lease and starts. Both pods run simultaneously. Prevention: always set lockAtMostFor to a value safely exceeding the maximum realistic job duration.

---

Q4: What is lockAtLeastFor and is it stored in the database?

A: lockAtLeastFor is a minimum hold time — no other pod can acquire the lease before this duration has passed, even if the job finishes quickly. It is not stored in the database. It is a configuration value used only during post‑processing to compute the lock_until update: lock_until = MAX(now, locked_at + lockAtLeastFor). This prevents a fast job from releasing the lease so quickly that another pod immediately picks it up on the same cycle.

---

Q5: Why use usingDbTime() in the ShedLock configuration?

A: usingDbTime() instructs ShedLock to use the database server's clock for all timestamp comparisons instead of the JVM clock. In a distributed environment, different pods may have clock skew — their local clocks may differ by seconds. Using a single authoritative time source (the database) eliminates clock skew as a source of correctness bugs.

---

Q6: What is the purpose of @EnableSchedulerLock and what happens without it?

A: @EnableSchedulerLock triggers a component scan for @SchedulerLock annotations and creates proxy classes for each annotated job. Without it, the @SchedulerLock annotations are silently ignored — every pod runs every job, exactly as if ShedLock were not present. It also sets defaultLockAtMostFor, which is mandatory because it acts as a safety fallback for jobs that forget to set lockAtMostFor individually.

---

Q7: What is the difference between ShedLock and a pessimistic row lock (SELECT FOR UPDATE)?

A: A pessimistic row lock locks the actual data rows being processed — other pods block and wait for those rows to be released. ShedLock locks the scheduler job itself — other pods do not wait at all; they see the lease is held and immediately skip the job and sleep. ShedLock is therefore much more efficient for the "one instance does this job, others stay free" requirement, while SELECT FOR UPDATE is appropriate when all instances should process different data rows in parallel.