Distributed Schedulers (Part 3) | 8 Common Pitfalls and Their Solutions

The Night Shift Crew Analogy

A hospital runs a nightly cleanup crew that sanitises every room. If the crew is new and untrained, things go wrong: they sanitise the same room twice because two teams did not coordinate, they flood the building's drains by running all water simultaneously, one spill is not noticed until the whole wing is contaminated, and the morning shift finds the work unfinished because an error in room 47 caused the whole team to stop. Each failure is avoidable with the right protocol.

Schedulers in production suffer the same class of problems. This chapter walks through eight real pitfalls — from time zone misconfigurations to memory leaks to duplicate execution across multiple instances — and provides production‑grade solutions for each.

Pitfall 1 — Cron Job Time Zone Mismatch

The Problem

java

@Scheduled(cron = "0 0 9 * * ?")  // Intended: "9 AM every day"
public void morningNotification() {
    // Send morning push notification to users
}

No time zone is specified. The scheduler picks the JVM default, which is typically UTC on a cloud server. If users are in IST (UTC+5:30), 9 AM UTC is 2:30 PM IST — the "morning notification" arrives in the afternoon.

The Solution

Always specify zone explicitly:

java

@Scheduled(cron = "0 0 9 * * ?", zone = "Asia/Kolkata")
public void morningNotification() {
    // Now fires at 9 AM IST regardless of server time zone
}

Make the zone configurable:

java

@Scheduled(cron = "${scheduler.morning.cron:0 0 9 * * ?}",
           zone  = "${scheduler.morning.zone:Asia/Kolkata}")
public void morningNotification() { }

Pitfall 2 — Missing `@Transactional`

The Problem

java

@Scheduled(cron = "0 0 1 * * ?", zone = "UTC")
public void processSubscriptions() {
    List<Subscription> expired = subscriptionRepo.findAllExpired();
    for (Subscription s : expired) {
        subscriptionRepo.updateStatus(s.getId(), "EXPIRED"); // Table A
        enrollmentRepo.deleteBySubscriptionId(s.getId());   // Table B — may fail
    }
}

If the deletion from Table B throws an exception, Table A already has updated rows but Table B still has stale data. The database is now inconsistent.

The Solution

Annotate the scheduler or the inner service method with @Transactional:

java

import org.springframework.transaction.annotation.Transactional;

@Scheduled(cron = "0 0 1 * * ?", zone = "UTC")
@Transactional
public void processSubscriptions() {
    List<Subscription> expired = subscriptionRepo.findAllExpired();
    for (Subscription s : expired) {
        subscriptionRepo.updateStatus(s.getId(), "EXPIRED");
        enrollmentRepo.deleteBySubscriptionId(s.getId());
        // If this throws, both operations roll back atomically
    }
}

Or delegate to a separate @Service method that carries @Transactional, which is cleaner for testing.

Pitfall 3 — Fetching Huge Data in One Shot (Memory Overflow)

The Problem

java

@Scheduled(cron = "0 0 1 * * ?", zone = "UTC")
@Transactional
public void processSubscriptions() {
    // Fetches ALL 85,000 expired records at once
    List<Subscription> expired = subscriptionRepo.findAllExpired();
    // Each record is 10 KB → 850 MB → OutOfMemoryError: Java heap space
    for (Subscription s : expired) { /* ... */ }
}

If the JVM heap is 512 MB and 85,000 records at 10 KB each are loaded in one query, the process crashes with an OutOfMemoryError.

The Solution — Batch Processing

java

@Scheduled(cron = "0 0 1 * * ?", zone = "UTC")
public void processSubscriptions() {
    boolean hasMore = true;
    while (hasMore) {
        // Fetch top 1,000 records that are not yet expired or failed
        List<Subscription> batch = subscriptionRepo
            .findTop1000ByStatusNotIn(List.of("EXPIRED", "FAILED"));
        hasMore = !batch.isEmpty();
        if (hasMore) {
            subscriptionBatchService.processBatch(batch);
        }
    }
}

Each batch is 10 MB (1,000 × 10 KB). After each batch completes, the garbage collector can reclaim that memory before the next batch starts.

Critical: ensure your query excludes already‑processed records to avoid an infinite loop:

java

// Repository method
@Query("SELECT s FROM Subscription s WHERE s.expiryDate < :now " +
       "AND s.status NOT IN :statuses ORDER BY s.id ASC LIMIT 1000")
List<Subscription> findTop1000ByStatusNotIn(
    @Param("now") LocalDateTime now,
    @Param("statuses") List<String> statuses);

Pitfall 4 — Memory Leak Through First‑Level Cache (Persistence Context)

The Problem

Even with batch processing, placing a single @Transactional over the entire scheduler job causes a memory leak:

java

@Scheduled(cron = "0 0 1 * * ?", zone = "UTC")
@Transactional  // ← One giant transaction wrapping ALL 85 batches
public void processSubscriptions() {
    // Batch 1: loads 1,000 entities → persistence context grows
    // Batch 2: loads 1,000 more entities → persistence context grows more
    // ...
    // Batch 85: persistence context holds references to ALL 85,000 entities
    //            GC cannot reclaim any of them → effective memory leak
}

JPA's first‑level cache (persistence context) holds a reference to every entity loaded within a transaction. With one big transaction across all 85 batches, 85,000 entities accumulate in memory and cannot be garbage collected.

Note: in schedulers there is no HTTP request, so spring.jpa.open-in-view (which scopes the persistence context to the HTTP request) does not apply. In schedulers the persistence context always lives for the duration of the transaction.

The Solution — Transaction Per Batch

java

@Scheduled(cron = "0 0 1 * * ?", zone = "UTC")
public void processSubscriptions() {  // No @Transactional here
    boolean hasMore = true;
    while (hasMore) {
        List<Subscription> batch = subscriptionRepo
            .findTop1000ByStatusNotIn(List.of("EXPIRED", "FAILED"));
        hasMore = !batch.isEmpty();
        if (hasMore) {
            subscriptionBatchService.processBatch(batch); // @Transactional here
        }
        // Each batch has its own transaction — persistence context is destroyed
        // after each batch, allowing GC to reclaim those entity references
    }
}

@Service
public class SubscriptionBatchService {

    @Transactional  // One transaction PER BATCH
    public void processBatch(List<Subscription> batch) {
        for (Subscription s : batch) {
            subscriptionRepo.updateStatus(s.getId(), "EXPIRED");
            enrollmentRepo.deleteBySubscriptionId(s.getId());
        }
    }
}

Pitfall 5 — Silent Exception Handling (Three‑Layer Safety)

The Problem

An unhandled exception in a scheduler exits silently — the scheduler logs nothing and stops all remaining batches.

The Solution — Record Level, Batch Level, Job Level

java

@Scheduled(cron = "0 0 1 * * ?", zone = "UTC")
public void processSubscriptions() {
    // Layer 3: overall job safety net
    try {
        boolean hasMore = true;
        while (hasMore) {
            List<Subscription> batch = subscriptionRepo
                .findTop1000ByStatusNotIn(List.of("EXPIRED", "FAILED"));
            hasMore = !batch.isEmpty();
            if (hasMore) {
                // Layer 2: batch level
                try {
                    subscriptionBatchService.processBatch(batch);
                } catch (Exception batchEx) {
                    log.error("Batch processing failed, continuing", batchEx);
                    // Continue to next batch rather than aborting the whole job
                }
            }
        }
    } catch (Exception jobEx) {
        log.error("Subscription expiry job failed at top level", jobEx);
        // Optionally: alert via PagerDuty, Slack, etc.
    }
}

@Service
public class SubscriptionBatchService {

    @Transactional
    public void processBatch(List<Subscription> batch) {
        for (Subscription s : batch) {
            // Layer 1: record level
            try {
                subscriptionRepo.updateStatus(s.getId(), "EXPIRED");
                enrollmentRepo.deleteBySubscriptionId(s.getId());
            } catch (Exception recordEx) {
                log.error("Failed to process subscription {}", s.getId(), recordEx);
                // Mark this record as FAILED so it is excluded from future batches
                subscriptionRepo.updateStatus(s.getId(), "FAILED");
            }
        }
    }
}

The three layers ensure:

One bad record does not fail the entire batch.
One bad batch does not abort the entire job.
Any unexpected job‑level failure is logged and alerted.

Pitfall 6 — Excessive SQL Round Trips

The Problem

For each of the 85,000 records, the code issues two SQL statements (one update, one delete) — 170,000 round trips total. Each round trip has network latency. Total job duration can be unacceptably long.

The Solution — Hibernate JDBC Batching

properties

# application.properties
spring.jpa.properties.hibernate.jdbc.batch_size=50
spring.jpa.properties.hibernate.order_updates=true
spring.jpa.properties.hibernate.order_inserts=true

batch_size=50 instructs Hibernate to collect 50 statements and send them in a single JDBC call. order_updates=true allows Hibernate to reorder SQL operations so that all updates to the same table are grouped together — this is necessary for the driver to form correct batches.

Without batching:  UPDATE subs → DELETE enroll → UPDATE subs → DELETE enroll → ... (170,000 round trips)
With batching:     [UPDATE×50 batch] → [DELETE×50 batch] → ... (≈3,400 round trips)

The actual SQL execution count does not change, but the number of network round trips drops dramatically.

Pitfall 7 — Duplicate Execution in a Multi‑Instance Environment

The Problem

When your service runs as multiple pods (e.g., in Kubernetes), each pod has the same code and its own instance of the scheduler. At 1 AM, all 10 pods fire simultaneously and process the same records multiple times — potentially double‑charging customers or sending duplicate emails.

Solution A — Pessimistic Lock (`SELECT FOR UPDATE`)

java

@Query("SELECT s FROM Subscription s WHERE s.status = 'PENDING' " +
       "AND s.expiryDate < :now ORDER BY s.id LIMIT 1000 FOR UPDATE")
List<Subscription> findTop1000ForUpdate(@Param("now") LocalDateTime now);

Only one transaction can hold the row locks at a time; other instances block.

Con: blocking is wasteful. If you have 100 instances, 99 are idle, waiting for the first to release locks. Suitable only for small deployments with strict ordering requirements.

Solution B — Pessimistic Lock with Skip Locked (Parallel Processing)

java

@Query("SELECT s FROM Subscription s WHERE s.status = 'PENDING' " +
       "AND s.expiryDate < :now ORDER BY s.id LIMIT 1000 FOR UPDATE SKIP LOCKED")
List<Subscription> findTop1000SkipLocked(@Param("now") LocalDateTime now);

SKIP LOCKED allows each instance to skip rows already locked by another instance and take the next available set. Instance 1 locks rows 1–1,000; Instance 2 automatically skips to rows 1,001–2,000. No waiting, no duplicates, parallel processing.

Supported by MySQL 8+, PostgreSQL 9.5+, Oracle 12c+.

Solution C — ShedLock (Only One Instance Runs)

If your requirement is that exactly one instance runs the scheduler (the others skip it entirely), use ShedLock — covered in depth in the next chapter.

Pitfall 8 — Summary Table

Pitfall	Root Cause	Solution
Time zone mismatch	Default JVM time zone used	Always set `zone` in `@Scheduled`
Missing `@Transactional`	Partial updates on failure	Annotate with `@Transactional`
Memory overflow	Entire dataset loaded at once	Batch processing
First‑level cache leak	Single transaction across batches	One `@Transactional` per batch
Silent exceptions	No try‑catch around scheduler	Three‑layer exception handling
Excessive SQL round trips	Per‑record SQL calls	Hibernate JDBC batching
Duplicate execution (blocking)	Multiple instances, same code	Pessimistic lock
Duplicate execution (parallel)	Multiple instances, same code	Skip locked or ShedLock

Production‑Grade Combined Example

java

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
public class SubscriptionExpiryJob {

    private static final Logger log = LoggerFactory.getLogger(SubscriptionExpiryJob.class);

    private final SubscriptionBatchService batchService;
    private final SubscriptionRepository   subscriptionRepo;

    public SubscriptionExpiryJob(SubscriptionBatchService batchService,
                                  SubscriptionRepository subscriptionRepo) {
        this.batchService     = batchService;
        this.subscriptionRepo = subscriptionRepo;
    }

    // Pitfall 1 fix: explicit time zone
    @Scheduled(cron = "0 0 1 * * ?", zone = "Asia/Kolkata")
    public void run() {
        log.info("Subscription expiry job started");
        // Pitfall 5 fix: job-level safety net
        try {
            boolean hasMore = true;
            while (hasMore) {
                // Pitfall 7 fix: skip locked for parallel multi-instance processing
                var batch = subscriptionRepo.findTop1000SkipLocked();
                hasMore = !batch.isEmpty();
                if (hasMore) {
                    // Pitfall 5 fix: batch-level safety net
                    try {
                        batchService.processBatch(batch);
                    } catch (Exception e) {
                        log.error("Batch failed, moving to next batch", e);
                    }
                }
            }
        } catch (Exception e) {
            log.error("Top-level job failure", e);
        }
        log.info("Subscription expiry job finished");
    }
}

@Service
class SubscriptionBatchService {

    private static final Logger log = LoggerFactory.getLogger(SubscriptionBatchService.class);

    private final SubscriptionRepository subscriptionRepo;
    private final EnrollmentRepository   enrollmentRepo;

    SubscriptionBatchService(SubscriptionRepository subscriptionRepo,
                              EnrollmentRepository enrollmentRepo) {
        this.subscriptionRepo = subscriptionRepo;
        this.enrollmentRepo   = enrollmentRepo;
    }

    // Pitfall 2 fix: @Transactional on the batch
    // Pitfall 4 fix: one transaction per batch (not wrapping the whole job)
    @Transactional
    public void processBatch(List<Subscription> batch) {
        for (Subscription s : batch) {
            // Pitfall 5 fix: record-level safety net
            try {
                subscriptionRepo.updateStatus(s.getId(), "EXPIRED");
                enrollmentRepo.deleteBySubscriptionId(s.getId());
            } catch (Exception e) {
                log.error("Failed subscription {}: {}", s.getId(), e.getMessage());
                // Pitfall 3 fix: mark FAILED so record is excluded from future batches
                subscriptionRepo.updateStatus(s.getId(), "FAILED");
            }
        }
    }
}

Interview Questions & Pitfalls

Q1: An interviewer asks: "Your scheduler runs at 1 AM and processes 85,000 records. After a few runs you notice the pod restarts with an OutOfMemoryError. What do you check first?"

A: Check whether the scheduler loads all records in a single query inside a single transaction. If yes, two problems compound: the initial memory spike from loading all records, and the JPA first‑level cache accumulating references to all entities for the duration of the transaction. The fix is batch processing (1,000 records at a time) combined with a separate @Transactional per batch so the persistence context is destroyed after each batch, allowing GC to reclaim memory.

Q2: What is open-in-view and why is it irrelevant to schedulers?

A: spring.jpa.open-in-view=true (the default) creates one persistence context per HTTP request, shared across all transactions within that request. In schedulers, there is no HTTP request — so this setting has no effect. The persistence context lifecycle in a scheduler is determined entirely by @Transactional boundaries. This is why naive use of a single @Transactional over the entire scheduler job causes the first‑level cache to grow unboundedly.

Q3: What are the three levels of exception handling in a well‑designed scheduler?

A: Record level (try‑catch around each record, marks the record as failed so it is skipped next run), batch level (try‑catch around each batch, logs and continues to the next batch), and job level (try‑catch around the entire job for unexpected failures with alerting). Without all three levels, a single bad record can silently abort thousands of subsequent records.

Q4: What is SKIP LOCKED and how does it enable parallel scheduler processing?

A: SKIP LOCKED is a database directive that tells the query to bypass any rows already locked by another transaction and return the next available rows. When multiple pods run the same scheduler simultaneously, each pod locks a different set of rows. Pod 1 locks rows 1–1,000; Pod 2 skips those and locks 1,001–2,000; and so on. No blocking, no duplicate processing, linear throughput scaling.

Q5: When is pessimistic lock without SKIP LOCKED appropriate for scheduler deduplication?

A: When strict ordering is required — you must process batch 1 before batch 2, batch 2 before batch 3. In that case, only one instance processes at a time, which is acceptable given the ordering requirement. For small deployments (one or two pods) with small batches, the blocking overhead is negligible. For large deployments or unordered processing, use SKIP LOCKED or ShedLock.

Q6: What happens if Hibernate order_updates is not set when using JDBC batching?

A: Without order_updates=true, Hibernate may interleave operations across tables: update subscription table, delete from enrollment table, update subscription table, delete from enrollment table. The JDBC driver cannot form batches across different tables, so the batch size setting has no effect and all statements are sent individually. Setting order_updates=true allows Hibernate to reorder statements so all updates to the same table are grouped together, enabling the driver to form proper JDBC batches.

Q7: A cron job sends morning push notifications. Users complain they receive them in the afternoon. What is the cause and fix?

A: The cause is a missing zone attribute in @Scheduled(cron = ...). Without it, the scheduler uses the JVM default time zone, typically UTC on cloud servers. If users are in IST (UTC+5:30), 9 AM UTC is 2:30 PM IST. The fix is to add zone = "Asia/Kolkata" (or the appropriate zone) and ideally externalise it to application.properties so it can be changed without code changes.

Distributed Schedulers (Part 3) | 8 Common Pitfalls and Their Solutions ​

The Night Shift Crew Analogy ​

Pitfall 1 — Cron Job Time Zone Mismatch ​

The Problem ​

The Solution ​

Pitfall 2 — Missing @Transactional ​

The Problem ​

The Solution ​

Pitfall 3 — Fetching Huge Data in One Shot (Memory Overflow) ​

The Problem ​

The Solution — Batch Processing ​

Pitfall 4 — Memory Leak Through First‑Level Cache (Persistence Context) ​

The Problem ​

The Solution — Transaction Per Batch ​

Pitfall 5 — Silent Exception Handling (Three‑Layer Safety) ​

The Problem ​

The Solution — Record Level, Batch Level, Job Level ​

Pitfall 6 — Excessive SQL Round Trips ​

The Problem ​

The Solution — Hibernate JDBC Batching ​

Pitfall 7 — Duplicate Execution in a Multi‑Instance Environment ​

The Problem ​

Solution A — Pessimistic Lock (SELECT FOR UPDATE) ​

Solution B — Pessimistic Lock with Skip Locked (Parallel Processing) ​

Solution C — ShedLock (Only One Instance Runs) ​

Pitfall 8 — Summary Table ​

Production‑Grade Combined Example ​

Interview Questions & Pitfalls ​

Distributed Schedulers (Part 3) | 8 Common Pitfalls and Their Solutions

The Night Shift Crew Analogy

Pitfall 1 — Cron Job Time Zone Mismatch

The Problem

The Solution

Pitfall 2 — Missing `@Transactional`

The Problem

The Solution

Pitfall 3 — Fetching Huge Data in One Shot (Memory Overflow)

The Problem

The Solution — Batch Processing

Pitfall 4 — Memory Leak Through First‑Level Cache (Persistence Context)

The Problem

The Solution — Transaction Per Batch

Pitfall 5 — Silent Exception Handling (Three‑Layer Safety)

The Problem

The Solution — Record Level, Batch Level, Job Level

Pitfall 6 — Excessive SQL Round Trips

The Problem

The Solution — Hibernate JDBC Batching

Pitfall 7 — Duplicate Execution in a Multi‑Instance Environment

The Problem

Solution A — Pessimistic Lock (`SELECT FOR UPDATE`)

Solution B — Pessimistic Lock with Skip Locked (Parallel Processing)

Solution C — ShedLock (Only One Instance Runs)

Pitfall 8 — Summary Table

Production‑Grade Combined Example

Interview Questions & Pitfalls