Scheduled Jobs: Common Pitfalls and Fixes

A scheduled job looks simple: a method and an annotation. In production it runs unattended, over large amounts of data, on several instances at once. This chapter walks through one realistic job, expiring subscriptions every night, and fixes it one pitfall at a time. The last pitfall leads into chapter 23.

The job

Every night at 1 AM, find subscriptions whose expiry date has passed, mark them EXPIRED, and delete their course enrollments:

@Component
public class SubscriptionScheduler {

    @Scheduled(cron = "0 0 1 * * *")
    public void expireSubscriptions() {
        List<Subscription> expired = subRepo.findByExpiryDateBefore(LocalDate.now());
        for (Subscription sub : expired) {
            sub.setStatus("EXPIRED");                                  // step 1
            subRepo.save(sub);
            enrollmentRepo.deleteBySubscriptionId(sub.getId());       // step 2
        }
    }
}

1. Cron and time zones

@Scheduled(cron = "0 0 9 * * *")
public void myNotificationJob() {
    sendMorningUpdates();
}

"Every day at 9 AM", but in which time zone? Without one, cron uses the server's zone. Servers commonly run in UTC; if your users are in India (IST, UTC+5:30), 9:00 UTC is 2:30 PM for them, and the "morning" update arrives in the afternoon.

Fix: always give the zone the schedule is meant for:

@Scheduled(cron = "0 0 9 * * *", zone = "Asia/Kolkata")

2. Missing transaction

Each subscription takes two writes. Without a transaction, each repository call commits on its own. If step 2 fails for subscription 101, step 1 is already committed: the subscription says EXPIRED, but its enrollments still exist. The data is inconsistent.

Fix: run the two steps in a transaction (@Transactional, from the Spring Boot book) so they commit or roll back together. Where exactly to put the transaction is the subject of pitfalls 4 and 5.

3. Loading everything at once

findByExpiryDateBefore returns every expired subscription. Suppose that is 85,000 rows of about 10 KB each: 850 MB in one list. With a 512 MB heap, the job dies with java.lang.OutOfMemoryError: Java heap space, and does the same every night after.

Fix: process in batches. Fetch a limited number of rows at a time, in a loop:

@Query(value = """
        SELECT * FROM subscription
        WHERE expiry_date < :today
          AND status NOT IN ('EXPIRED')
        ORDER BY expiry_date
        LIMIT 1000
        """, nativeQuery = true)
List<Subscription> fetchTop1000Expired(@Param("today") LocalDate today);

while (true) {
    List<Subscription> expired = subRepo.fetchTop1000Expired(LocalDate.now());
    if (expired.isEmpty()) {
        break;
    }
    for (Subscription sub : expired) { ... }
}

85 batches of 1,000 rows, about 10 MB each.

Watch the loop condition. Inside while (true), the query must stop returning rows you have handled. Here, processing sets status = 'EXPIRED' and the query excludes EXPIRED. If a processed row could still match, the job would fetch it forever.

4. The persistence-context memory leak

Batches alone do not fix memory if the whole loop runs in one transaction, for example with @Transactional on the scheduler class.

JPA keeps every entity loaded in a transaction in the persistence context (the first-level cache) until the transaction ends. With one transaction around all 85 batches, the persistence context holds references to all 85,000 entities, so the garbage collector can free none of them. Memory grows by about 10 MB per batch, and the OutOfMemoryError returns, only later.

A note on scope: for HTTP requests, Spring Boot enables Open Session in View by default, which keeps one persistence context for the whole request (disable it with spring.jpa.open-in-view=false). A scheduled job has no HTTP request, so there the persistence context always lives exactly as long as the transaction.

Fix: one transaction per batch. Move the batch into a separate bean with its own @Transactional method (a separate bean, so the call goes through Spring's proxy):

@Component
public class SubscriptionScheduler {

    @Autowired
    private SubscriptionService subService;

    @Scheduled(cron = "0 0 1 * * *", zone = "Asia/Kolkata")
    public void expireSubscriptions() {
        while (subService.processBatch()) {
            // keep going while there is work
        }
    }
}

@Service
public class SubscriptionService {

    @Transactional
    public boolean processBatch() {
        List<Subscription> expired = subRepo.fetchTop1000Expired(LocalDate.now());
        if (expired.isEmpty()) {
            return false;
        }
        for (Subscription sub : expired) {
            sub.setStatus("EXPIRED");
            enrollmentRepo.deleteBySubscriptionId(sub.getId());
        }
        return true;
    }
}

Each batch commits, its persistence context closes, and its entities can be collected.

5. Exception handling that really isolates failures

Three things can still go wrong:

• one bad record fails its whole batch;
• one failed batch stops the whole job;
• a failure goes unnoticed because nothing is logged or alerted.

Records: one transaction per record

The tempting fix is a try/catch around each record inside the batch transaction, marking the bad one FAILED:

@Transactional
public boolean processBatch() {
    List<Subscription> expired = subRepo.fetchTop1000Expired(LocalDate.now());
    if (expired.isEmpty()) return false;
    for (Subscription sub : expired) {
        try {
            sub.setStatus("EXPIRED");
            subRepo.save(sub);
            enrollmentRepo.deleteBySubscriptionId(sub.getId());
        } catch (Exception e) {                       // does NOT isolate the failure
            log.error("Failed sub {}", sub.getId(), e);
            sub.incrementRetryCount();
            sub.setStatus("FAILED");
            subRepo.save(sub);
        }
    }
    return true;
}

This does not work. Run against PostgreSQL with six subscriptions, where deleting subscription 3's enrollment violates a foreign key:

processed sub 1
processed sub 2
processed sub 3                                  <- no error yet
ERROR: update or delete on table "enrollment" violates foreign key constraint
Failed sub 4: DataIntegrityViolationException    <- blamed on the wrong record
ERROR: current transaction is aborted, commands ignored until end of transaction block
Failed sub 5: JpaSystemException
Failed sub 6: JpaSystemException
-> UnexpectedRollbackException: Transaction silently rolled back because it has been marked as rollback-only

Afterwards, nothing had changed in the database: all six subscriptions were still ACTIVE. Three effects combine:

1. JPA defers SQL. deleteBySubscriptionId only schedules the delete; the SQL runs at the next flush. That flush happened during subscription 4's query, so the error was blamed on subscription 4.
2. A failed statement poisons the transaction. PostgreSQL aborts the transaction after an error, so every later statement fails too, and Spring marks it rollback-only as soon as the exception passes through a transactional repository.
3. Catching does not undo that. At commit, Spring rolls back everything and throws UnexpectedRollbackException. Even the FAILED markers are lost. In the while loop, the next batch fetches the same rows and fails the same way, forever.

Fix: give each record its own transaction, and record failures in a separate one. With a TransactionTemplate using REQUIRES_NEW:

@Service
public class SubscriptionService {

    private final TransactionTemplate perRecordTx;

    public SubscriptionService(PlatformTransactionManager tm) {
        this.perRecordTx = new TransactionTemplate(tm);
        this.perRecordTx.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
    }

    public boolean processBatch() {
        List<Subscription> expired = subRepo.fetchTop1000Expired(LocalDate.now());
        if (expired.isEmpty()) {
            return false;
        }
        for (Subscription s : expired) {
            Long id = s.getId();
            try {
                perRecordTx.executeWithoutResult(status -> {
                    Subscription sub = subRepo.findById(id).orElseThrow();
                    sub.setStatus("EXPIRED");
                    enrollmentRepo.deleteBySubscriptionId(id);
                    subRepo.flush();              // surface errors here, for this record
                });
            } catch (Exception e) {
                log.error("Failed sub {}", id, e);
                perRecordTx.executeWithoutResult(status -> {
                    Subscription sub = subRepo.findById(id).orElseThrow();
                    sub.incrementRetryCount();
                    sub.setStatus("FAILED");      // so the query skips it next time
                });
            }
        }
        return true;
    }
}

The same six subscriptions:

processed sub 1
processed sub 2
Failed sub 3: DataIntegrityViolationException      <- the right record
processed sub 4
processed sub 5
processed sub 6

database: 1 EXPIRED, 2 EXPIRED, 3 FAILED (retry 1), 4 EXPIRED, 5 EXPIRED, 6 EXPIRED

The query must now exclude FAILED too (status NOT IN ('EXPIRED', 'FAILED')), so a bad record is not fetched again and again; a separate process can retry FAILED records later. A transaction per record costs one commit per record, which is fine for most jobs. If that is too slow, keep batch transactions but split a failing batch into single-record retries.

Batches: do not let one failure stop the job, or loop forever

If fetching a batch throws (say, the database is briefly unreachable), the exception leaves the while loop and the remaining batches never run. But catching it and simply continuing is dangerous too: if the database stays down, the loop retries forever, hammering it. Bound the failures:

int consecutiveFailures = 0;
while (true) {
    try {
        if (!subService.processBatch()) {
            break;                                   // no more work
        }
        consecutiveFailures = 0;
    } catch (Exception e) {
        log.error("Batch failed", e);
        if (++consecutiveFailures >= 3) {
            throw e;                                 // give up; the next run will retry
        }
    }
}

The whole job: never fail silently

An exception escaping a @Scheduled method is only logged by Spring's default error handler, and the job just runs again next time. Nobody is told. Wrap the job, log, and alert:

@Scheduled(cron = "0 0 1 * * *", zone = "Asia/Kolkata")
public void expireSubscriptions() {
    try {
        runAllBatches();                             // the loop above
    } catch (Exception e) {
        log.error("Subscription expiry job failed", e);
        alertService.notify(e);
    }
}

So there are three layers: record, batch, job.

6. Too many database round trips

85,000 subscriptions × (1 UPDATE + 1 DELETE) = 170,000 SQL statements, each a network round trip to the database. That is a lot of latency for one job.

The number of statements cannot shrink, since each row needs its update and delete, but the round trips can. Hibernate JDBC batching collects similar statements and sends them together:

spring.jpa.properties.hibernate.jdbc.batch_size=50
spring.jpa.properties.hibernate.order_updates=true

• batch_size=50: send up to 50 similar statements in one JDBC batch call.
• order_updates=true: let Hibernate reorder statements by table. A batch can only hold statements for the same table, so the sequence "update subscription 1, update enrollment 5, update subscription 2, update enrollment 6" would break every batch at each table switch. Reordered ("subscription 1, subscription 2... enrollment 5, enrollment 6..."), they batch well. (order_inserts=true does the same for inserts.)

Fewer round trips mean less CPU and a shorter job.

7. Several instances run the same job

Everything so far assumed one instance. Now order-service runs on two pods. Both have the same @Scheduled method, and at 1 AM both run it:

Step	Pod 1	Pod 2
1	select top 1,000	select the same 1,000
2	update their status	update them again
3	delete enrollments	delete again, or fail

Here the result is wasted work and errors. With a job that sends money or notifications, it means paying or messaging twice.

Option 1: SELECT ... FOR UPDATE

Lock the rows you fetch:

SELECT * FROM subscription
WHERE expiry_date < :today AND status NOT IN ('EXPIRED', 'FAILED')
ORDER BY expiry_date
LIMIT 1000
FOR UPDATE

The first transaction to run this locks the 1,000 rows. The other pod's identical query blocks until the first commits; it then re-evaluates and, since those rows are now EXPIRED, gets the next ones. No duplicates.

In PostgreSQL, a second session running the same FOR UPDATE query while the first held its lock waited until the first committed (2.2 s in the test).

The cost is blocking: all other pods wait on the one holding the lock. If its batch takes an hour, they wait an hour. Use this only with one or two instances, small batches, or when rows must be processed strictly in order.

Option 2: FOR UPDATE SKIP LOCKED

If order does not matter, let each pod skip rows another transaction has locked:

SELECT * FROM subscription
WHERE expiry_date < :today AND status NOT IN ('EXPIRED', 'FAILED')
ORDER BY expiry_date
LIMIT 1000
FOR UPDATE SKIP LOCKED

Two sessions at once, each asking for 3 rows:

session A: 1,2,3
session B: 4,5,6      <- returned immediately, no waiting

No duplicates, no waiting, and the pods share the work in parallel. SKIP LOCKED is supported by PostgreSQL, MySQL 8, Oracle and SQL Server (with different syntax in some); check yours.

Note that both options require the row lock to cover the processing: the select and the updates must happen in the same transaction. With the one-transaction-per-record design of pitfall 5, lock and process each record inside its own transaction.

When only one instance may run the job

Sometimes the requirement is stronger: only one instance runs the job at all, while the others do nothing. Examples include a job that calls an external system once, or one that cannot be split into rows. Row locks do not express that.

Next: Chapter 23 solves it with ShedLock: a shared lock table that lets exactly one instance run each scheduled execution.

Check yourself

1. Which time zone does cron use by default, and how do you fix a job meant for another zone?
2. What goes wrong without a transaction around the two writes per subscription?
3. Why does loading all expired subscriptions at once fail, and what replaces it?
4. What must the batch query guarantee inside while (true)?
5. Why does one transaction around all batches still run out of memory?
6. Why must the batch's @Transactional method be in a separate bean?
7. Why does a try/catch per record inside one batch transaction not isolate failures? Give three reasons.
8. How do you isolate a bad record, and why mark it FAILED?
9. What is the danger of catching batch failures and simply continuing?
10. What happens to an exception that escapes a @Scheduled method, and what should you add?
11. What do hibernate.jdbc.batch_size and order_updates do?
12. What happens when two instances run the same job, and why is it dangerous?
13. How does FOR UPDATE prevent duplicates, and what is its cost?
14. How does SKIP LOCKED differ, and when can you use it?

Answers

1. The server's zone. Set zone = "..." on @Scheduled.
2. If the second write fails, the first is already committed, leaving inconsistent data.
3. All rows load into memory together and can exceed the heap (OutOfMemoryError). Process in batches with a LIMIT.
4. That processed rows no longer match (e.g. it excludes EXPIRED), otherwise the loop never ends.
5. The persistence context keeps every entity loaded in the transaction, so none can be garbage-collected until it ends.
6. Calls within the same bean bypass Spring's proxy, so @Transactional would not apply.
7. JPA defers SQL, so the error appears later, on another record; the database (PostgreSQL) aborts the transaction, so every following statement fails; Spring marks the transaction rollback-only, so at commit everything, including the FAILED markers, rolls back (UnexpectedRollbackException).
8. Process each record in its own transaction (e.g. TransactionTemplate with REQUIRES_NEW) and record the failure in a separate transaction. FAILED keeps the query from fetching it again in every batch.
9. If the cause persists (database down), the loop retries forever. Limit consecutive failures and stop.
10. Spring's default error handler logs it and the job runs again next time; nobody is alerted. Wrap the job, log and alert.
11. batch_size groups up to N similar statements into one JDBC call; order_updates lets Hibernate group statements by table so batches are not broken.
12. Both process the same rows; for jobs that pay or notify, users are paid or messaged twice.
13. The first transaction locks the selected rows; others block until it commits and then see them processed. Cost: other instances wait, which does not scale.
14. It skips rows locked by others instead of waiting, so instances take different rows in parallel. Use it when processing order does not matter.