SKA SDP Buffer Manager
======================

The Buffer Manager is a component responsible for monitoring and managing the
storage space allocated for the Science Data Processor (SDP) within the SKA
telescope system.

The Buffer Manager performs the following key functions:

* It periodically checks the total capacity and available free space of the
  persistent volume claim (PVC) dedicated to SDP data (see also: :ref:`calculations`).
* It connects to the SKA SDP configuration database and updates the status of
  the capacity buffer storage resource with the latest capacity and free space information.
* It periodically checks if data product Flow entries can be marked for deletion (see also :ref:`deleting`), which happens if:

    * All the dependencies on a flow are in ``FINISHED`` status
    * Flow's current status is either ``COMPLETED``, ``INCOMPLETE``, or ``FAILED``
    * Flow has expired

* It deletes data products when their associated Flow entries are in state of ``DELETING`` (see also :ref:`deleting`).
* It periodically checks for data product flows in ``PENDING`` state, and creates the data directory if required. And then sets the status to ``INITIALISED``.
* It scans for orphaned data-product under the PVC mount path (e.g. ``product/{eb_id}/ska-sdp/{pb_id}``) and for each directory without an existing flow:

    * creates an ExecutionBlock entity
    * creates a ProcessingBlock entity and marks its state as ``FINISHED``
    * creates a data-product Flow with ``expiry_time=-1`` (never expires) and sets its state to ``COMPLETED``
    * creates an allocation with the current size of the pb_id directory as the ``quantity``, a resource link and no request link.

* It records the final data size (MB) in the flow state for data-product flows in ``COMPLETED``, ``INCOMPLETE``, or ``FAILED`` status.

    - A value of ``-1`` indicates the size could not be determined, the buffer
      manager will keep the entry eligible for retry on subsequent runs so the
      size can be re-attempted later.
    - A value of ``0``  indicates the directory was missing


Prerequisites
-------------

Before running the Buffer Manager, ensure the following prerequisites are met:

* SKA SDP Configuration Database: An accessible instance of the SKA SDP
  configuration database must be running.
* Persistent Volume Claim (PVC): A PVC dedicated for SDP data buffers must be
  provisioned and accessible. The name of this PVC and its mount path are required.
* Environment Variables: Several environment variables need to be set to
  configure the Buffer Manager. You must set the following variables:

.. list-table::
    :widths: 20, 20, 60
    :header-rows: 1

    * - Environment Variable
      - Default Value
      - Description
    * - ``SDP_LOG_LEVEL``
      - ``INFO``
      - The logging level for the Buffer Manager.
    * - ``SDP_CONFIG_HOST``
      - ``None``
      - The hostname or IP address of the SKA SDP configuration database.
    * - ``sdp_data_pvc_mount_path``
      - ``/mnt/data``
      - The mount path of the SDP data PVC within the Buffer Manager container.
    * - ``SDP_BUFFERMANAGER_LOOP_INTERVAL``
      - ``60``
      - The interval (in seconds) at which the Buffer Manager will check the
        buffer resources and update the configuration database.
    * - ``SDP_DATA_PVC_NAME``
      - ``test-pvc``
      - The name of the Kubernetes Persistent Volume Claim (PVC) used for SDP
        data buffers.
    * - ``SDP_BUFFERMANAGER_LIVENESS_FILE``
      - ``/tmp/alive``
      - The path to the file that the Buffer Manager will periodically touch to
        indicate its liveness to Kubernetes.
    * - ``SDP_BUFFERMANAGER_CONTINGENCY_RESERVE``
      - ``10``
      - The amount of space (in %) to reserve as a contingency within the
        buffer. This helps prevent the buffer from becoming completely full.
    * - ``SDP_BUFFERMANAGER_BATCH_SIZE``
      - ``100``
      - The batch size to use for database inserts of orphaned data. Must be
        between 1 and 1000 (inclusive). If set outside those limits the value
        will be silently forced between those values.


.. toctree::
  :maxdepth: 2
  :caption: Contents:

  calculation
  deletion
  releases/changelog