Background Jobs, Queues and Scheduled Tasks Explained

By Grocito17 October 2026

Background jobs are tasks an application runs outside the user's request, such as sending emails, building reports or processing images. A queue holds the waiting jobs and workers pick them up one by one. Scheduled tasks run at set times, while event-driven jobs start when something happens. Done well, they keep pages fast and work reliable, provided you add retries, idempotency and good monitoring.

Key takeaways

Move slow, unreliable or non-urgent work off the request path so that users get a quick response.
A queue and workers decouple the moment work is requested from the moment it is done, and let you scale each side separately.
Scheduled jobs run on a clock, event-driven jobs run in response to something, and many systems need both.
Reliability needs retries with backoff, idempotent jobs, timeouts, a dead-letter place for failures and visible alerts.
Duplicate sends, silent failures and missing retries are the most common and most damaging mistakes.
Time zones and daylight saving cause surprising schedule bugs, so decide on a clear convention.

Why do applications move work off the request path?

When someone clicks a button, they expect an answer in a moment. Some tasks simply take longer or can fail for reasons outside your control. Typical candidates are:

Emails and notifications: the mail provider may be slow or temporarily down.
Reports and exports: heavy queries that take minutes.
Imports: processing a large spreadsheet row by row.
Image and video processing: resizing, converting and scanning uploads.
Webhooks: delivering events to customers' systems, some of which will be offline. See webhooks explained for how these deliveries are normally handled.

Handling these in the background means the user sees a fast confirmation, and the system can retry safely if something fails. It also protects your site's speed during busy periods.

What are queues and workers?

A queue is an ordered list of jobs waiting to be done. A worker is a process that takes a job from the queue, performs it and marks it complete. The part of the application that creates the job, often called the producer, does not wait for the result.

This arrangement has useful consequences.

Smoothing: a sudden burst of requests becomes a longer queue rather than an overloaded server.
Independent scaling: add more workers when the queue grows, without touching the web servers.

For a broader view of how these parts fit together, see our guide to web application architecture basics.

What is the difference between scheduled and event-driven jobs?

Scheduled tasks

Scheduled tasks, often called cron-style jobs after the classic Unix scheduler, run at set times: every night, every hour, on the first day of the month. Examples include nightly backups, daily summary emails, subscription renewals and clearing expired data.

Event-driven jobs

Event-driven jobs start because something happened: an order was placed, a file was uploaded, a form was submitted. They respond immediately, with no waiting for the next clock tick.

Most systems need both. A common pattern is a scheduled task that checks for work and puts jobs on a queue, so that scheduling and processing remain separate.

How do you make background jobs reliable?

Background work fails quietly if you do not design for failure. Build in these protections.

Retries with backoff

Temporary failures are normal: a network blip, a provider timeout. Retry automatically, with growing delays between attempts and a small random variation so that retries do not all arrive together. Set a maximum number of attempts.

Idempotency

A job is idempotent if running it twice has the same effect as running it once. Because retries and crashes mean a job may run more than once, this matters enormously. For example, before sending a confirmation email, record that it has been sent, and check that record first. Use unique identifiers for payments and operations so duplicates can be recognised and ignored.

Timeouts

Give every job and every external call a time limit.

Dead-letter handling

After the maximum retries, move the job to a separate holding area, often called a dead-letter queue, instead of dropping it or retrying forever. Someone can then inspect, fix the cause and replay it.

Visibility into failures

Failures should create alerts and be easy to see. A dashboard showing queue length, age of the oldest job, failure counts and recent errors is as important as the jobs themselves.

What about ordering and concurrency?

If order matters, for example applying a series of updates to the same account, process one account's jobs one at a time or include version information so out-of-date updates are ignored. Use database transactions and sensible locking, keep jobs small and independent, and limit simultaneous heavy jobs so that a flood of imports cannot starve urgent work.

How do you handle long-running tasks and tell users what is happening?

If a task takes more than a few seconds, acknowledge quickly with a status such as queued or in progress, show progress where possible and let people leave and be notified when the work is done. Keep generated reports available to download for a defined time, and report failure honestly with the next step the user can take. Splitting a large job into smaller ones also makes progress easier to measure and failures easier to retry.

How do you respect third-party rate limits?

Workers can generate traffic far faster than outside services allow. If a provider limits calls per second, control the worker pace to match, using a throttle at the queue or worker level. Handle refusal responses by waiting for the time the provider states, and by backing off. Providers usually publish their limits and ask callers to back off when refused, so follow those rules.

What should you monitor and alert on?

Queue length and the age of the oldest waiting job.
Failure and retry counts, grouped by job type.
Whether scheduled jobs actually ran. A job that silently stops running is the hardest to notice, so alert on a missing run, not just on errors.

How do you handle security and secrets in workers?

Workers often hold powerful credentials, so give each only the permissions it needs, keep secrets in a managed store rather than in code and out of logs, and put identifiers rather than full personal data in job payloads. Validate job input as you would a web request.

How can you scale workers?

Start with more worker processes reading the same queue, scale on queue length or age, and separate queues by job type so a slow category does not block a quick one. Watch your database as you scale, since many workers can overload it.

What are the options: serverless functions and managed queues?

You do not have to run your own servers for this. Broad categories are library-based queues inside your application, managed queue services from cloud providers, serverless functions triggered by messages, schedules or uploads and workflow engines for multi-step processes with waiting or approvals.

For a comparison of the hosting styles behind these, see serverless vs containers vs virtual machines.

What are the time zone and daylight saving gotchas?

Schedules go wrong in predictable ways.

Store and compute in a single standard time, and convert only for display.
Daylight saving changes can make a local time occur twice or not at all, so a job might run twice or be skipped. Avoid scheduling important work in the hours around clock changes, or use a standard time for the schedule.
Long months and leap days break "on the 31st" rules, so define what should happen.

How do you test background jobs?

Run each job directly with sample data and twice in a row to confirm it is idempotent. Simulate failures such as a provider timeout and check retry and dead-letter behaviour. Test with a large queue, test schedules around clock changes and month ends with a controllable clock, and rehearse replaying failed jobs.

Common mistakes to avoid

No retries, so a short outage permanently loses work.
Retries without idempotency, causing duplicate emails, charges or shipments.
Silent failures that nobody sees until a customer complains.
One queue for everything, so a huge import delays password reset emails.

A worked example

Imagine a subscription business that emails invoices each month from a nightly web request. When the email provider slows, the job times out half way and reruns, so some customers get two invoices. The team moves to a queue: a scheduled job creates one task per customer, each task checks a sent flag, retries with backoff and moves to a dead-letter area after several failures, and an alert fires if the monthly run does not start. Customers receive one invoice each.

FAQ

What is a background job in web development?

A background job is a task that runs outside the normal request and response cycle, so the user does not wait for it. Common examples are sending emails, generating reports, processing uploads and delivering webhooks. A queue and worker processes usually handle them.

What is the difference between a cron job and a queue?

A cron job is a task that runs on a schedule, such as every night. A queue is a list of jobs waiting to be processed by workers, usually triggered by events. They work well together: a scheduled task can add many jobs to a queue.

How do I stop background jobs from running twice?

Make jobs idempotent so that a repeat has no extra effect. Record completed work, use unique identifiers and check them before acting. You should assume that any job may run more than once, because retries and crashes make duplicates unavoidable.

What happens when a background job fails?

A well-designed system retries it several times with growing delays, then moves it to a dead-letter queue and raises an alert. Someone then investigates and can replay the job once the cause is fixed. Without that design, failures are often silent.

Next steps

List the slow or fragile things your application does inside a page request, such as emails, exports and calls to outside services, and move the most troublesome one onto a queue with retries and an alert. If you are planning new features or want a reliability review of an existing platform, see our app development services or contact us.