Changelog

0.10.0

  • [Added] Monitor top-level directories per entry at a configurable directory level via SDP_BUFFERMANAGER_DIR_SCAN_LEVELS (<dir>:<level>, e.g. users:3). Disabled when empty (the default).

  • [Fixed] Symlinks under product/ are no longer followed when looking for untracked data.

0.9.0

  • (MR58)

    • Move to python3.14

    • Upgrade ska-sdp-config to 1.6.6

0.8.0

  • [Changed] Empty EB and subsystem directories under product/ are now removed automatically after the last data product inside them is moved to the recycle bin. (MR54)

  • [Changed] Touch liveness file before each main loop function to prevent pod from failing if any of them takes too long to run. (MR55)

  • [Changed] Minor refactor to introduce better txn use. (MR55)

  • [Changed] Use batches for main watcher, and switch du command to use proper bytes. (MR57)

0.7.0

  • [Changed] The periodic orphan size refresh now runs a cheaper find pass first and only falls back to the expensive du when something under the orphan path has been created, modified or deleted since the last measurement. (MR49)

  • [Updated] Orphan flows that no longer exists on disk will now be marked as deleted. (MR50)

0.6.0

  • [Fixed] Duplicate recycle bin destination is now given a Unix timestamp suffix (e.g. product_1754221441) to avoid the collision. (MR51)

  • [Changed] Orphan data product size are now recalculated periodically in the orphan worker thread, and the flow final_size attribute is updated with the new size. (MR45)

  • [Fixed] Improved untracked data product detection to cover top-level PVC directories and all subsystem layouts uniformly. The scan walks product/{eb_id}/{subsystem}/ and reports untracked direct children; subsystem directories themselves are never treated as orphan entities. (MR43)

  • [Added] Non-standard orphan directories are assigned unique hashed EB/PB identifiers to avoid ID collisions. (MR43)

  • [Added] Add a new env variable to set expiration time for orphaned data products, allowing them to be automatically deleted after a specified duration. (MR43)

  • [Added] Add SDP_BUFFERMANAGER_CUSTOM_UNTRACKED_DIRS environment variable to configure additional top-level PVC directories that should be excluded from untracked-entity detection. (MR43)

  • [Updated] Orphan-related logic has been moved into a dedicated orphanwatcher module and data models into a datamodels module, improving code organisation. (MR43)

  • [Updated] Total configured capacity can be parsed from “Gi” and “GiB”-like units. (MR42)

  • [Added] Add startup sequence that sets the resource quantity to 0 until the first complete run of the orphan reconciliation. (MR40)

  • [Added] Add configurable sizing timeouts to improve robustness during orphan and capacity scans. (MR40)

  • [Fixed] Changed the orphan data calculation to be based on the sum of untracked file size rather than disc use calculations, resolving inaccuracies on NFS storage volumes. (MR40)

  • [Updated] Upgrade ska-sdp-config to 1.6.4 (MR48)

0.5.1

  • [Updated] Upgrade ska-sdp-config to 1.6.1 (MR39)

0.5.0

  • [Updated] Upgrade ska-sdp-config to 1.5.1 (MR37)

  • [Added] Use batches to add orphaned data, and seperate disk i/o tasks from DB tasks. (MR33)

0.4.0

  • [Updated] Use glob.glob to find all data directories in one go for processing orphaned data. (MR29)

  • [Updated] When allocatable space is determined, do not read size of directory from disk if allocation already contains final size. (MR29)

  • [Fixed] Generate orphaned data for more than one pb for a given eb if exists. (MR29)

  • [Updated] Create allocations for orphaned data. (MR31)

  • [Updated] Added functionality to update flow sizes of finished data products. (MR30)

  • [Fixed] Calculation of orphaned data value. (MR27)

  • [Fixed] etcd transaction failures in the Buffer Manager caused by exceeding the maximum number of operations per transaction (“too many operations in txn request”). (MR23, MR25)

  • [Fixed] Setting permissions on data product directories to work on any file system. (MR22)

  • [Updated] To create execution block along with processing block and flows for orphaned data products. (MR21)

  • [Updated] Update Available Quantity calculation. (MR24)

0.3.0

  • [Updated] Total storage capacity is configured via the SDP_BUFFERMANAGER_TOTAL_CAPACITY environment varialble, when set (MR19)

  • [Updated] Dependency status is checked to see if flow can be deleted (MR18)

  • [Added] Data Product Flow directories are now created for flows that are in PENDING state. (MR20)

0.2.0

  • [Updated] Upgraded ska-sdp-config to 1.3.0 (MR17)

  • [Updated] When removing data product directories, if path is not specified in flow, construct path using eb_id (MR16)

  • [Updated] Allow flow directory to be missing when deleting. (MR11)

  • [Updated] Upgraded ska-sdp-config to 1.2.0 (MR8)

  • [Updated] Update base images to ska-build-python 0.3.1 and ska-python 0.2.3 (MR4)

  • [Added] Add feature to move data products associated with data flows in DELETING to a recycle bin. (MR5)

  • [Added] Mark flow entries for deletion that have expired and are no longer referenced. (MR6, MR7)

0.1.0

  • [Changed] Buffer Manager now works with capacity buffer storage, not performance buffer storage (MR3)

  • [Changed] Renamed PerformanceBufferStorageResource to CapacityBufferStorageResource (MR3)

  • [Added] Update main loop to alight with structure in ska-sdp-proccontrol. (MR2)

  • [Added] Included changelog in documentation. (MR2)

  • [Added] Created the PerformanceBufferStorageResource class to create and update resource entries in the configuration database. (MR2)