10. Concurrent executions

disk_watch.py is scheduled every 5 minutes. One day, a hung network filesystem makes an execution last longer than 5 minutes, and executions start piling up. In this last step, Scrippy prevents simultaneous executions of disk_watch.py.

The code

#!/usr/bin/env python3
"""
--------------------------------------------------------------------------------
  @author         : Luiggi Vercotti
  @date           : 2026-10-08
  @version        : 1.0.0
  @description    : Monitor filesystem usage

--------------------------------------------------------------------------------
  Update:
  0.1.0  2026-10-08 - Luiggi Vercotti - cre: Script creation
  0.2.0  2026-10-08 - Luiggi Vercotti - evo: Several mount points, log levels
  0.3.0  2026-10-08 - Luiggi Vercotti - evo: Monitoring exit status
  0.4.0  2026-10-08 - Luiggi Vercotti - evo: Configuration file
  0.5.0  2026-10-08 - Luiggi Vercotti - evo: Command line options
  0.6.0  2026-10-08 - Luiggi Vercotti - evo: Alerting API token
  0.7.0  2026-10-08 - Luiggi Vercotti - evo: Report file
  0.8.0  2026-10-08 - Luiggi Vercotti - evo: Error handling
  0.9.0  2026-10-08 - Luiggi Vercotti - evo: Access control
  1.0.0  2026-10-08 - Luiggi Vercotti - evo: Concurrent executions

--------------------------------------------------------------------------------
  List of authorized users or group:
    @group:ops

--------------------------------------------------------------------------------
  Concurrent executions:
    @max_instance: 1
    @exit_on_wait: true

--------------------------------------------------------------------------------
  List of mandatory configuration parameters:
    @conf:filesystems|mount_points|str|false
    @conf:thresholds|warning|int|false
    @conf:thresholds|critical|int|false
    @conf:alerting|token|str|true

--------------------------------------------------------------------------------
  List of execution options and argument parameters:
    @args:mount-point|str|Mount points to check (default: from configuration)|+||exclude||false|false
    @args:exclude|str|Mount points to skip|+||||false|false
    @args:warning|int|Warning threshold in percent (default: from configuration)|1||||false|false
    @args:critical|int|Critical threshold in percent (default: from configuration)|1||||false|false
    @args:token|str|Alerting API token (default: from configuration)|1||||false|true
    @args:report|bool|Write a report into the report directory||false|||false|false
    @args:format|choice|Report format (default: text)|1|text,csv||report|false|false

--------------------------------------------------------------------------------
  Functioning:
  ---------------
    Checks the usage of each mount point against a warning and a critical
    threshold and exits with a monitoring compatible exit status:
      0: OK, 1: WARNING, 2: CRITICAL, 3: UNKNOWN (a mount point can not be checked)

    Configuration:
      - filesystems::mount_points: Comma separated list of mount points
      - thresholds::warning: Warning threshold in percent
      - thresholds::critical: Critical threshold in percent
      - alerting::token: Alerting API token (secret)
"""
# ------------------------------------------------------------------------------
#  Initialization of the environment
# ------------------------------------------------------------------------------
import os
import sys
import shutil
import datetime
import scrippy_core
from scrippy_core import logger, ScrippyCoreError

OK, WARNING, CRITICAL, UNKNOWN = 0, 1, 2, 3
STATUS = {OK: "OK", WARNING: "WARNING", CRITICAL: "CRITICAL", UNKNOWN: "UNKNOWN"}
EXTENSIONS = {"text": "txt", "csv": "csv"}

# ------------------------------------------------------------------------------
#  Definition of functions and classes
# ------------------------------------------------------------------------------
def get_mount_points(config, args):
  if args.mount_point:
    return args.mount_point
  mount_points = [mp.strip() for mp in config.get("filesystems", "mount_points").split(",")]
  return [mp for mp in mount_points if mp not in (args.exclude or [])]

def get_thresholds(config, args):
  warning = args.warning if args.warning is not None else config.get("thresholds", "warning", "int")
  critical = args.critical if args.critical is not None else config.get("thresholds", "critical", "int")
  if not 0 < warning < critical <= 100:
    raise ScrippyCoreError(f"Invalid thresholds: warning={warning}, critical={critical}")
  return warning, critical

def get_usage(mount_point):
  logger.debug(f"[+] Checking {mount_point}")
  usage = shutil.disk_usage(mount_point)
  return round(usage.used * 100 / usage.total, 1)

def check(mount_points, warning, critical):
  results = []
  for mount_point in mount_points:
    try:
      usage = get_usage(mount_point)
    except OSError as err:
      logger.error(f" '-> {mount_point}: {err.strerror}")
      results.append((mount_point, None, UNKNOWN))
      continue
    if usage >= critical:
      logger.error(f" '-> {mount_point}: {usage}% [CRITICAL]")
      status = CRITICAL
    elif usage >= warning:
      logger.warning(f" '-> {mount_point}: {usage}% [WARNING]")
      status = WARNING
    else:
      logger.info(f" '-> {mount_point}: {usage}% [OK]")
      status = OK
    results.append((mount_point, usage, status))
  return results

def write_report(results, report_format, workspace_path, reportdir):
  name = f"disk_watch_{datetime.datetime.now():%Y%m%d_%H%M%S}.{EXTENSIONS[report_format]}"
  tmp_report = os.path.join(workspace_path, name)
  logger.info(f"[+] Writing report: {tmp_report}")
  with open(tmp_report, mode="w", encoding="utf-8") as report:
    for mount_point, usage, status in results:
      if report_format == "csv":
        report.write(f"{mount_point};{usage};{STATUS[status]}\n")
      else:
        report.write(f"{mount_point:<20} {usage}% {STATUS[status]}\n")
  report_dir = os.path.join(reportdir, "disk_watch")
  os.makedirs(report_dir, exist_ok=True)
  logger.info(f"[+] Publishing report into {report_dir}")
  shutil.move(tmp_report, os.path.join(report_dir, name))

# ------------------------------------------------------------------------------
#  Main processing
# ------------------------------------------------------------------------------
def main():
  with scrippy_core.ScriptContext(retention=100) as _context:
    config = _context.config
    args = _context.args
    token = args.token or config.get("alerting", "token")
    logger.debug(f"[+] Alerting API token: {token}")
    warning, critical = get_thresholds(config, args)
    logger.log_section("disk_watch", f"Thresholds: warning={warning}% critical={critical}%")
    results = check(get_mount_points(config, args), warning, critical)
    if args.report:
      write_report(results, args.format or "text", _context.workspace_path, _context.env["reportdir"])
    status = max(status for _, _, status in results)
    logger.info(f"[+] Status: {STATUS[status]}")
    if status != OK:
      sys.exit(status)

# ------------------------------------------------------------------------------
#  Entry point
# ------------------------------------------------------------------------------
if __name__ == '__main__':
  main()

What changed: The Concurrent executions section declares that only one execution is allowed at a time (@max_instance: 1), and that an execution started while another one is running must exit immediately (@exit_on_wait: true).

While an execution is running, a second one stops before running any of your code:

[...]
[20261008 19:04:40] [INFO    ] [+] Waiting for an execution slot: 2/1 [0s]
[20261008 19:04:40] [CRITICAL] [ScrippyCoreError]: EagernessError: `exit_on_wait` is set to True

Declaring the concurrent executions policy

Declaration Type Purpose Default
@max_instance Integer The maximum number of simultaneous executions. 0 means unlimited 0
@timeout Integer The maximum waiting time in seconds for an execution slot. Only used when @exit_on_timeout is true 0
@exit_on_timeout Boolean Exit with an error when the waiting time is over false
@exit_on_wait Boolean Exit with an error immediately instead of waiting false

The boolean values true, 1 and on (case insensitive) are true. Any other value is false.

When the maximum number of executions is reached, a new execution waits for a slot. The waiting executions are queued: They get a slot in the order they started.

Wait for a slot, without limit

@max_instance: 1

The new execution waits until the running one ends, however long it takes.

Wait for a slot, with a time limit

@max_instance: 1
@timeout: 30
@exit_on_timeout: true

The new execution waits up to 30 seconds:

[20261008 19:16:19] [INFO    ] [+] Waiting for an execution slot: 2/1 [30s]
[20261008 19:16:25] [INFO    ] [+] Opening history (retention=100)
[...]

If no slot is released in time, the execution exits with an error:

[20261008 19:15:43] [INFO    ] [+] Waiting for an execution slot: 2/1 [7s]
[20261008 19:15:49] [CRITICAL] [ScrippyCoreError]: TimeoutError: Timeout expired

Note: The availability of a slot is checked every 3 seconds: The actual waiting time is the largest multiple of 3 lower than @timeout (6 seconds for @timeout: 7, no waiting at all for a @timeout lower than or equal to 3).

Do not wait

@max_instance: 1
@exit_on_wait: true

The new execution exits immediately with an error, as disk_watch.py does.

Good to know

  • The executions are counted per script name and per tmpdir: The running executions are tracked in the <tmpdir>/<script name>.pids file. Two users with different Scrippy installations do not see each other's executions, while two scripts with the same name sharing the same tmpdir share the same limit.
  • The terminated executions, even killed ones, are automatically removed from the tracking file.
  • A waiting or refused execution is not recorded in the history: Only the executions that got a slot are.
  • The concurrency check is done after the configuration check and the access control.

The complete script

Congratulations, disk_watch.py is complete! Starting from a few lines, it now has:

  • a validated configuration file, overridable by command line options,
  • an auto-generated help, a version and its source code at hand,
  • colored logs, a log file per execution and an execution history,
  • a protected API token,
  • a report built in a temporary workspace and published in the report directory,
  • a fine error handling and monitoring compatible exit status,
  • an access control and a concurrent executions policy.

Most of these features did not require any code: Only declarations in the header.

The Reference chapter describes every Scrippy feature in details. The Modules chapter goes on extending disk_watch.py with the Scrippy modules.