SKA SDP Buffer Manager

The Buffer Manager is a component responsible for monitoring and managing the storage space allocated for the Science Data Processor (SDP) within the SKA telescope system.

The Buffer Manager performs the following key functions:

It periodically checks the total capacity of the persistent volume claim (PVC) dedicated to SDP data (see also: Resource quantity calculations).
It connects to the SKA SDP Configuration Database and updates the status of the capacity buffer storage resource with the latest capacity information.
It periodically checks if data product flow entries can be marked for deletion (see also Moving data to recycle bin), which happens if:
- All the dependencies on a flow are in FINISHED status
- Flow’s current status is either COMPLETED, INCOMPLETE, or FAILED
- Flow has expired
It deletes data products when their associated flow entries are in state of DELETING (see also Moving data to recycle bin).
It periodically checks for data product flows in PENDING state, and creates the data directory if required. And then sets the status to INITIALISED.
It scans for orphaned data-product directories under the PVC mount path and reconciles them into configuration-database entities and allocations (see also: Resource quantity calculations).
It records the final data size (MB) in the flow state for data-product flows in COMPLETED, INCOMPLETE, or FAILED status.
- A value of -1 indicates the size could not be determined, the buffer manager will keep the entry eligible for retry on subsequent runs so the size can be re-attempted later.
- A value of 0 indicates the directory was missing

Prerequisites

Before running the Buffer Manager, ensure the following prerequisites are met:

SKA SDP Configuration Database: An accessible instance of the SKA SDP configuration database must be running.
Persistent Volume Claim (PVC): A PVC dedicated for SDP data buffers must be provisioned and accessible. The name of this PVC and its mount path are required.
Environment Variables: Several environment variables need to be set to configure the Buffer Manager. You must set the following variables:

Environment Variable	Default Value	Description
`SDP_LOG_LEVEL`	`INFO`	The logging level for the Buffer Manager.
`SDP_CONFIG_HOST`	`None`	The hostname or IP address of the SKA SDP Configuration Database.
`SDP_DATA_PVC_MOUNT_PATH`	`/mnt/data`	The mount path of the SDP data PVC within the Buffer Manager container.
`SDP_BUFFERMANAGER_LOOP_INTERVAL`	`60`	The interval (in seconds) at which the Buffer Manager will check the buffer resources and update the configuration database.
`SDP_DATA_PVC_NAME`	`test-pvc`	The name of the Kubernetes Persistent Volume Claim (PVC) used for SDP data buffers.
`SDP_BUFFERMANAGER_LIVENESS_FILE`	`/tmp/alive`	The path to the file that the Buffer Manager will periodically touch to indicate its liveness to Kubernetes.
`SDP_BUFFERMANAGER_CONTINGENCY_RESERVE`	`10`	The amount of space (in %) to reserve as a contingency within the buffer. This helps prevent the buffer from becoming completely full.
`SDP_BUFFERMANAGER_BATCH_SIZE`	`100`	The batch size to use for database inserts of orphaned data. Must be between 1 and 1000 (inclusive). If set outside those limits the value will be silently forced between those values.
`SDP_BUFFERMANAGER_DU_TIMEOUT`	`1800`	Timeout (in seconds) for each `du` subprocess call used to measure directory sizes.
`SDP_BUFFERMANAGER_ORPHAN_EXPIRATION_TIME`	`-1`	Expiration time (in seconds) for orphaned data-product flows. A value of `-1` disables automatic expiry so orphaned data is never automatically deleted.
`SDP_BUFFERMANAGER_CUSTOM_UNTRACKED_DIRS`	`""`	Comma-separated list of additional top-level directory names under the PVC mount path to exclude from untracked-entity detection. The built-in exclusions (`product` and `recycle_bin`) always apply regardless of this setting.
`SDP_BUFFERMANAGER_DIR_SCAN_LEVELS`	`""`	Comma-separated list of top-level directories under the PVC mount path whose contents are tracked per entry, given as `<dir>:<level>`. The level is required and the directory itself is level 1, so for example `users:3` tracks each `users/<user>/<entry>` and `users:3,example:2` tracks each `example/<entry>` as well. An entry without a valid level stops the Buffer Manager at startup. An empty value (the default) disables this behaviour.

Contents: