Resource quantity calculations

The Buffer Manager will periodically calculate the following:

  • Contingency Reserve (C) calculated from the percentage set with the environment variable from Helm

  • Total Capacity (T) of the volume mounted, from the environment variable SDP_BUFFERMANAGER_TOTAL_CAPACITY or from the disk itself if this variable is unset.

  • Orphaned Data (OD)

    OD = sum(size of data that does not have a corresponding allocation in the configuration database)

The Buffer Manager publishes the resource quantity as:

Q = max(0, T - OD - C)

where T is total capacity, OD is orphaned data, and C is contingency reserve.

The Processing Controller then uses this published quantity when allocating requests. In the Processing Controller resource management documentation this is described in more detail.

Startup sequence

On startup, the Buffer Manager keeps the published allocatable quantity at 0 until startup reconciliation succeeds:

  1. Publish resource state INITIALISING and set the allocatable quantity of the resource to 0.

  2. Run one full orphan discovery scan and create allocations for discoverable orphan data.

  3. Recompute orphaned data and capacity.

  4. Publish state READY.

If startup reconciliation fails (for example allocation creation failure), the Buffer Manager sets the resource status to DEGRADED and leaves the allocatable quantity at 0. In this case, the resource state will include a reason string describing the failure.

Orphan data handling

The Buffer Manager treats data under the PVC as orphaned when it finds content with no existing data-product flow entry in the configuration database.

The scan covers three areas under the PVC mount path:

  • Top-level directories or files, except:

    • the product directory

    • the recycle_bin directory

    • any directory listed in SDP_BUFFERMANAGER_CUSTOM_UNTRACKED_DIRS (empty by default)

    • any directory listed in SDP_BUFFERMANAGER_DIR_SCAN_LEVELS (empty by default)

    • any symlinks

  • Content under the product directory — the scan walks product/{eb_id}/{subsystem}/ and reports any direct child of a subsystem directory that is not already tracked by a known PB allocation. Subsystem directories themselves are never treated as orphans regardless of their contents.

  • Content under directories with a scan level — the scan walks each directory listed in SDP_BUFFERMANAGER_DIR_SCAN_LEVELS down to its configured level and reports every entry at that level as a separate orphan.

For every untracked entry the Buffer Manager attempts to detect whether it follows the standard product/{eb_id}/{subsystem}/{pb_id} layout by checking whether the outer directory matches a valid EB ID and the inner directory matches a valid PB ID. If so, the existing EB and PB identifiers are used directly. Otherwise a unique, hashed EB and PB ID are generated for the entry.

For each untracked entry the Buffer Manager creates configuration-database entries so that the data is represented explicitly instead of remaining orphaned:

  1. It creates an ExecutionBlock entity if one does not already exist.

  2. It creates a ProcessingBlock entity if one does not already exist, using a placeholder realtime script, no dependencies, and PB state FINISHED.

  3. It creates a data-product flow named orphaned-data-product with the expiry time configured by SDP_BUFFERMANAGER_ORPHAN_EXPIRATION_TIME (default: -1, no expiry) and flow state COMPLETED.

  4. It records the measured directory size as final_data_size in the flow state.

  5. It creates a capacity-buffer-storage allocation named orphan-buffer with the measured directory size as quantity, a link to the current PVC resource, and no request link.

This reconciliation runs:

  1. Once at startup, before the Buffer Manager publishes any non-zero quantity.

  2. Periodically in a background worker, using SDP_BUFFERMANAGER_LOOP_INTERVAL (default: 60 seconds).

Each reconciliation cycle also re-measures the disk usage of all existing orphan flows and updates final_data_size in the flow state if the size has changed. This keeps the recorded size accurate as data continues to grow or shrink after initial registration. If the measured path no longer exists, the size is set to 0. If du fails, the flow is skipped for that cycle and retried on the next.

If the Buffer Manager cannot determine the directory size, it records -1 for the size-based fields so the orphaned entry is still visible in the configuration database.

After this reconciliation, valid PB directories are expected to be represented by flow and allocation entries. In steady state, orphaned data is therefore content that is still not represented by allocations on the current capacity-buffer resource.

The recycle_bin directory is treated separately. Data moved there by the deletion flow still occupies space on the PVC, but it is intentionally excluded from orphaned-data calculation.

Directory scan levels

Some top-level directories, such as users, hold many independent datasets owned by different users or groups. The Buffer Manager walks these directories down to a configured level and registers each entry at that level as its own orphan.

The directories are configured with SDP_BUFFERMANAGER_DIR_SCAN_LEVELS as a comma-separated list of <dir>:<level> entries. The listed directory itself counts as level 1. The level is required: an entry without a valid level (for example users or users:0) is a configuration error and the Buffer Manager fails at startup. The default value is empty, so no directories are scanned this way unless configured.

For example, with users:3, each entry at users/<username>/<entry> is registered as one orphan:

users/                            level 1
├── <username>/                   level 2
│   ├── notes.txt                 level 3 -> one orphan
│   └── my_test_set/              level 3 -> one orphan
├── shared/
│   └── dataset/                  level 3 -> one orphan
└── bench/
    └── 2026-07-22-ical-1node/    level 3 -> one orphan

Setting users:3,example:2 also registers each direct child of example/ as one orphan.

The following rules also apply:

  • Directories above the configured level, such as users/<username>/, users/shared/ and users/bench/, are never registered as orphans, so they are never expired or removed. Only their contents are.

  • Files found above the configured level, such as users/readme.txt with level 3, are ignored: they are not registered as orphans, and their space is not counted.

  • Symlinks are not followed.

  • Entries can be any file or directory; they do not follow the product/ layout. Each entry is registered with a generated EB and PB ID.

  • Sizes are re-measured during each reconciliation cycle whenever the contents have changed.

  • product and recycle_bin cannot be given a scan level.

Removed Orphans

If the directory of an orphan is removed then the flow will be marked as deleted during the next reconciliation run. This will also mark the available space as being usable again.

Timeout configuration

All du subprocess calls use a timeout controlled by:

  • SDP_BUFFERMANAGER_DU_TIMEOUT (seconds, default: 1800)

When a du call times out (or otherwise fails), behavior depends on the path:

  • Untracked orphaned-data scan: the specific entry is skipped for that cycle (treated as 0 contribution) and a warning is logged.

  • Orphan registration flow/allocation creation: the directory size is recorded as -1.

  • Periodic flow size refresh (final_data_size): the flow is skipped for that cycle

    and retried on the next reconciliation. The previously recorded size is preserved.

Configuration database resource

Other SDP components can query the configuration database to retrieve the current capacity and free space of the capacity buffers. The information is periodically updated and can be accessed from the resource entry in the configuration database, as shown in the example below:

/resource/capacity-buffer-storage:test-pvc = {
    "information": {
        "contingency-reserve": 10,
        "orphaned-data": 0,
        "total-capacity": 25600
    },
    "instances": [],
    "key": {
        "kind": "capacity-buffer-storage",
        "name": "test-pvc"
    },
    "quantity": 23040,
    "tags": []
}

/resource/capacity-buffer-storage:test-pvc/state = {
    "status": "READY"
}

/resource/capacity-buffer-storage:test-pvc/state = {
    "status": "DEGRADED",
    "reason": "Startup orphan reconciliation failed: RuntimeError: ..."
}