ShedLock: Running a Job on Exactly One Instance

Chapter 22 stopped two instances from processing the same rows, but both still ran the job. Sometimes the job must run on one instance only, while the others skip it entirely. ShedLock does that.

	Without ShedLock	With ShedLock
Who runs the job	every instance	one instance per execution
Duplicates prevented by	row locks inside the job	a lock around the whole job

The idea: a shared lock table

ShedLock keeps one row per job in a shared table, by default named shedlock:

Column	Meaning
name (primary key)	the job's unique name
locked_at	when an instance last took the job
lock_until	until when the job is taken; after this, any instance may take it
locked_by	which instance took it (by default the host name)

ShedLock does not lock database rows in the SELECT ... FOR UPDATE sense. Each instance tries a conditional write, and whichever write succeeds owns the job until lock_until, much like optimistic concurrency. Think of lock_until as the end of a lease.

Where it plugs in

@Scheduled still puts the job in each instance's delayed queue (chapters 20 and 21). ShedLock wraps the job method in a proxy, so what the scheduler runs is, in effect:

// conceptually: generated by ShedLock, not written by you
public void myJob() {
    if (tryToTakeTheLock()) {        // pre-processing
        try {
            realMyJob();             // your method
        } finally {
            releaseTheLock();        // post-processing
        }
    }
    // else: another instance has it; skip this execution
}

Every instance wakes at the scheduled time; only the one that takes the lock runs the job.

Step 1: dependencies

<!-- the ShedLock core and Spring integration -->
<dependency>
    <groupId>net.javacrumbs.shedlock</groupId>
    <artifactId>shedlock-spring</artifactId>
    <version>5.13.0</version>
</dependency>

<!-- where the locks live: here a relational database, via JdbcTemplate -->
<dependency>
    <groupId>net.javacrumbs.shedlock</groupId>
    <artifactId>shedlock-provider-jdbc-template</artifactId>
    <version>5.13.0</version>
</dependency>

plus your database driver and Spring's JDBC support (spring-boot-starter-jdbc or spring-boot-starter-data-jpa). ShedLock has providers for many stores: JDBC databases, Redis, MongoDB, ZooKeeper and more. All instances must use the same store.

Step 2: create the table

ShedLock does not create its table. Create it yourself (for example with Flyway or Liquibase):

CREATE TABLE shedlock (
    name       VARCHAR(64)  NOT NULL,
    lock_until TIMESTAMP    NOT NULL,
    locked_at  TIMESTAMP    NOT NULL,
    locked_by  VARCHAR(255) NOT NULL,
    PRIMARY KEY (name)
);

Step 3: the lock provider

@Configuration
public class ShedLockConfig {

    @Bean
    public LockProvider lockProvider(DataSource dataSource) {   // URL, user, password...
        return new JdbcTemplateLockProvider(
                JdbcTemplateLockProvider.Configuration.builder()
                        .withJdbcTemplate(new JdbcTemplate(dataSource))
                        .usingDbTime()        // use the database clock, not each pod's
                        .build());
    }
}

usingDbTime() makes ShedLock compare times with the database's clock. Pods' clocks drift apart; if pod 1's clock were a minute fast, it could consider a lease expired that pod 2 still holds. One clock for everyone avoids that.

For different table or column names:

JdbcTemplateLockProvider.Configuration.builder()
        .withJdbcTemplate(new JdbcTemplate(dataSource))
        .withTableName("scheduler_lock")
        .withColumnNames(new JdbcTemplateLockProvider.ColumnNames("job_name", "expires_at", "started_at", "instance_id"))
        .usingDbTime()
        .build();

Step 4: enable and annotate

@SpringBootApplication
@EnableScheduling
@EnableSchedulerLock(defaultLockAtMostFor = "10m")
public class SchedulerApplication {

    public static void main(String[] args) {
        SpringApplication.run(SchedulerApplication.class, args);
    }
}

@EnableSchedulerLock loads ShedLock's beans and creates the proxies for @SchedulerLock methods at startup. defaultLockAtMostFor applies to jobs that do not set their own.

@Component
public class SchedulersTest {

    @Scheduled(fixedRate = 300000)                                   // every 5 minutes
    @SchedulerLock(name = "myJob", lockAtMostFor = "8m", lockAtLeastFor = "4m")
    public void myJob() {
        System.out.println("job execution started");
    }
}

name (required): the job's unique name, stored in the name column. It is not generated for you: without it, every execution fails with IllegalArgumentException: lock name can not be empty.

lockAtMostFor: the maximum time the lock is held. When an instance takes the job, lock_until is set to now + lockAtMostFor. If the instance crashes while running the job, it never releases the lock; this value guarantees the lock expires anyway and another instance can take over. Set it well above the job's longest normal run time (see "When a job outlives its lock" below).

lockAtLeastFor: the minimum time the lock is held, even if the job finishes sooner. It is not stored; it is used when releasing. It protects against a fast job running twice for one schedule: if pod 1 finishes in 1 second and pod 2's clock or queue is slightly behind, pod 2 could wake up a moment later, find the lock free, and run the same execution again.

Step 5: a timeline with two pods

Configuration: fixedRate = 5m, lockAtMostFor = 8m, lockAtLeastFor = 4m. The table is empty.

9:00: both pods wake up. Each tries to take the lock. The first time, that is an INSERT; since name is the primary key, only one succeeds:

INSERT INTO shedlock (name, lock_until, locked_at, locked_by)
VALUES ('myJob', now() + 8 min, now(), 'pod1');      -- pod1 wins; pod2's INSERT fails

name	locked_at	lock_until	locked_by
myJob	9:00	9:08	pod1

Pod 2 skips this execution and, as fixedRate dictates, queues its next attempt for 9:05.

9:03: pod 1 finishes early. On release, ShedLock sets:

lock_until = max(now, locked_at + lockAtLeastFor) = max(9:03, 9:04) = 9:04

UPDATE shedlock SET lock_until = '9:04' WHERE name = 'myJob' AND locked_by = 'pod1';

Pod 1 queues its next run for 9:05.

9:05: both wake up again. Now the row exists, so taking the lock is a conditional UPDATE:

UPDATE shedlock
SET lock_until = now() + 8 min, locked_at = now(), locked_by = :me
WHERE name = 'myJob' AND lock_until <= now();

The lease ended at 9:04, so the first UPDATE matches the row and wins (say pod 2); the other's UPDATE then matches nothing (lock_until is now 9:13) and it skips. Pod 2 runs the job.

9:10: pod 2 is still running. Pod 1 tries: lock_until (9:13) is in the future, so its UPDATE changes nothing and it skips again.

And so on: each execution runs on exactly one pod. In a test with two instances, a 5-second rate, a 1-second job and lockAtLeastFor of 4 s, every execution ran on one instance only, and the row read locked_at 21:07:26, lock_until 21:07:30, exactly locked_at + lockAtLeastFor after release.

When a job outlives its lock

lockAtMostFor cuts both ways. If a job runs longer than it, the lease expires while the job is still running, and another instance takes it. Tested with a 12-second job, lockAtMostFor = 8s and a 5-second rate:

instance 8091  started  02:37:35
instance 8092  started  02:37:45      <- 8091 is still running: the lease ran out at :43
instance 8091  finished 02:37:47

For two seconds the job ran twice at the same time, exactly what ShedLock is meant to prevent. So:

• set lockAtMostFor comfortably above the job's worst-case duration;
• still make the job safe to repeat where you can, for example with the row-level guards of chapter 22.

Also note locked_by: it defaults to the host name. Two instances on one machine (as in this test) both show the same name, fedora. In Kubernetes each pod has its own host name, which makes the column useful for finding which pod ran a job.

What ShedLock is not

ShedLock is a simple, practical tool, not a full distributed scheduler:

• It only skips executions that another instance is running; it does not queue them or distribute different jobs across instances.
• It depends on the lock store: if the database is down, nobody can take the lock, and the job does not run.
• Each instance still needs @EnableScheduling and wakes up for every execution; most just skip.

For heavier needs (job history, retries, a cluster-aware queue), look at Quartz in clustered mode, or a dedicated workflow engine.

This completes the book. Starting from two services calling each other, it has covered service discovery, load balancing, resilience, the gateway, central configuration, tracing, logging, and finally scheduled jobs across many instances.

Check yourself

1. How does ShedLock differ from the row locks of chapter 22?
2. What do the shedlock table's four columns hold?
3. Does ShedLock create its table?
4. What does the ShedLock proxy do around your job method?
5. Why use usingDbTime()?
6. Is @SchedulerLock's name optional?
7. What does lockAtMostFor protect against, and what goes wrong if it is shorter than the job?
8. What does lockAtLeastFor protect against? Is it stored?
9. A job takes the lock at 9:00 with lockAtLeastFor = 4m and finishes at 9:03. What is lock_until after release?
10. Two pods try to take a lock that has expired. How does the database ensure only one wins?
11. Name two limitations of ShedLock.

Answers

1. Row locks let every instance run the job but keep them off the same rows; ShedLock lets only one instance run each execution at all.
2. name: the job's unique name (primary key); locked_at: when it was taken; lock_until: when the lease ends; locked_by: which instance took it.
3. No; you create it, e.g. with a migration.
4. Before: try to take the lock (skip the run if another instance holds it); call the method; after: release the lock (respecting lockAtLeastFor).
5. So all instances compare times using one clock, the database's, instead of their own drifting clocks.
6. No. Without it every execution fails with lock name can not be empty.
7. An instance that crashes without releasing: the lock still expires. If the job runs longer than it, the lock expires mid-run and another instance starts the job too (concurrent duplicates).
8. A fast job running twice for one scheduled time because another instance wakes slightly later. No, it is only used when releasing.
9. max(9:03, 9:00 + 4m) = 9:04.
10. The first insert succeeds on the primary key and others fail; afterwards, the conditional UPDATE ... WHERE lock_until <= now() matches the row for only the first instance, since it moves lock_until into the future.
11. Any two of: it only skips executions (no queueing or distribution of different jobs); it depends on the lock store being available; every instance still wakes up for every execution.