Background Jobs
Frappe ships with a system for running jobs in the background. It is implemented by using the schedule package and a simple long-running infinite while loop .
You can enqueue a python method to run in the background by using the
frappe.enqueue
method:
def long_running_job(param1, param2):
# expensive tasks
pass
# directly pass the function
frappe.enqueue(long_running_job, queue='short', param1='A', param2='B')
# or pass the full module path as string
frappe.enqueue('app.module.folder.long_running_job', queue='short', param1='A', param2='B')
Here are all the possible arguments you can pass to the
enqueue
:
frappe.enqueue(
method, # python function or a module path as string
queue="default", # one of short, default, long
timeout=None, # pass timeout manually
is_async=True, # if this is True, method is run in worker
now=False, # if this is True, method is run directly (not in a worker)
job_name=None, # specify a job name
enqueue_after_commit=False, # enqueue the job after the database commit is done at the end of the request
at_front=False, # put the job at the front of the queue
**kwargs, # kwargs are passed to the method as arguments
)
You can also enqueue a
Document
method by using
frappe.enqueue_doc
:
frappe.enqueue_doc(
doctype,
name,
"do_something", # name of the controller method
queue="long",
timeout=4000,
param="value"
)
Queue
There are 3 default queues that are configured with the framework:
short
,
default
, and
long
. Each queue has a default timeout as follows:
-
short: 300 seconds
-
default: 300 seconds
-
long: 1500 seconds
You can also pass a custom timeout to the
enqueue
method.
Custom Queues
You can add custom queues by configuring them in
[common_site_config.json](/frappe/en/basics/site_config#common-site-config)
:
{
...
"workers": {
"myqueue": {
"timeout": 5000, # queue timeout
"background_workers": 4, # number of workers for this queue
}
}
}
Workers
By default Frappe sets up 3 worker types for consuming from each queue. The default configuration looks like this:
bench worker --queue short
bench worker --queue default
bench worker --queue long
In production these 3 worker processes are replicated to configured number of background workers to handle higher workloads.
NOTE: This way of mapping workers to single queue is just a convention and it's not necessary to follow it.
Multi-queue consumption
You can specify more than one queue for workers to consume from by specifying a comma separate string of queue names.
Example: If you wanted to combine short and default workers and only use two types of workers instead of default configuration then you can modify your worker configuration like this:
bench worker --queue short,default
bench worker --queue long
NOTE: The examples shown here are for Procfile format but they can be applied to supervisor or systemd configurations easily too.
Burst Mode using
--burst
bench worker --queue short --burst
This command will spawn a tempoary worker that will start consuming short queue and quit once queue is empty. If you periodically need higher amount of workers then you can use your OS's crontab to setup burst workers at specific times.
Scheduler Events
You can use Scheduler Events for running tasks periodically in the background using the
scheduler_events
hook.
app/hooks.py
scheduler_events = {
"hourly": [
# will run hourly
"app.scheduled_tasks.update_database_usage"
],
}
app/scheduled_tasks.py
def update_database_usage():
pass
After changing any scheduled events in hooks.py, you need to run bench migrate for changes to take effect.
Available Events
-
hourly
,daily
,weekly
, andmonthly
These events will trigger every hour, day, week, and month respectively.
-
hourly_long
,daily_long
,weekly_long
,monthly_long
Same as above but these jobs are run in the long worker suitable for long-running jobs.
-
all
The
all
event is triggered every 4 minutes. This can be configured via thescheduler_interval
key in common_site_config.json -
cron
A valid cron string that can be parsed by croniter .
Usage Examples:
scheduler_events = {
"daily": [
"app.scheduled_tasks.manage_recurring_invoices"
],
"daily_long": [
"app.scheduled_tasks.take_backups_daily"
],
"cron": {
"15 18 * * *": [
"app.scheduled_tasks.delete_all_barcodes_for_users"
],
"*/6 * * * *": [
"app.scheduled_tasks.collect_error_snapshots"
],
"annual": [
"app.scheduled_tasks.collect_error_snapshots"
]
}
}
The jobs triggered by scheduler are run by Administrator user. This also means any docs you create via scheduled job will be owned by the Administrator user unless specified otherwise.