SKA SDP Buffer Manager

The Buffer Manager is a component responsible for monitoring and managing the storage space allocated for the Science Data Processor (SDP) within the SKA telescope system.

The Buffer Manager performs the following key functions:

It periodically checks the total capacity and available free space of the persistent volume claim (PVC) dedicated to SDP data (see also: Resource quantity calculations).
It connects to the SKA SDP configuration database and updates the status of the capacity buffer storage resource with the latest capacity and free space information.
It periodically checks if data product Flow entries can be marked for deletion (see also Moving data to recycle bin), which happens if:
- All the dependencies on a flow are in FINISHED status
- Flow’s current status is either COMPLETED, INCOMPLETE, or FAILED
- Flow has expired
It deletes data products when their associated Flow entries are in state of DELETING (see also Moving data to recycle bin).
It periodically checks for data product flows in PENDING state, and creates the data directory if required. And then sets the status to INITIALISED.
It scans for orphaned data-product under the PVC mount path (e.g. product/{eb_id}/ska-sdp/{pb_id}) and for each directory without an existing flow:
- creates an ExecutionBlock entity
- creates a ProcessingBlock entity and marks its state as FINISHED
- creates a data-product Flow with expiry_time=-1 (never expires) and sets its state to COMPLETED
- creates an allocation with the current size of the pb_id directory as the quantity, a resource link and no request link.
It records the final data size (MB) in the flow state for data-product flows in COMPLETED, INCOMPLETE, or FAILED status.
- A value of -1 indicates the size could not be determined, the buffer manager will keep the entry eligible for retry on subsequent runs so the size can be re-attempted later.
- A value of 0 indicates the directory was missing

Prerequisites

Before running the Buffer Manager, ensure the following prerequisites are met:

SKA SDP Configuration Database: An accessible instance of the SKA SDP configuration database must be running.
Persistent Volume Claim (PVC): A PVC dedicated for SDP data buffers must be provisioned and accessible. The name of this PVC and its mount path are required.
Environment Variables: Several environment variables need to be set to configure the Buffer Manager. You must set the following variables:

Environment Variable	Default Value	Description
`SDP_LOG_LEVEL`	`INFO`	The logging level for the Buffer Manager.
`SDP_CONFIG_HOST`	`None`	The hostname or IP address of the SKA SDP configuration database.
`sdp_data_pvc_mount_path`	`/mnt/data`	The mount path of the SDP data PVC within the Buffer Manager container.
`SDP_BUFFERMANAGER_LOOP_INTERVAL`	`60`	The interval (in seconds) at which the Buffer Manager will check the buffer resources and update the configuration database.
`SDP_DATA_PVC_NAME`	`test-pvc`	The name of the Kubernetes Persistent Volume Claim (PVC) used for SDP data buffers.
`SDP_BUFFERMANAGER_LIVENESS_FILE`	`/tmp/alive`	The path to the file that the Buffer Manager will periodically touch to indicate its liveness to Kubernetes.
`SDP_BUFFERMANAGER_CONTINGENCY_RESERVE`	`10`	The amount of space (in %) to reserve as a contingency within the buffer. This helps prevent the buffer from becoming completely full.
`SDP_BUFFERMANAGER_BATCH_SIZE`	`100`	The batch size to use for database inserts of orphaned data. Must be between 1 and 1000 (inclusive). If set outside those limits the value will be silently forced between those values.

Contents: