Moving data to recycle bin

On the same frequency as the storage usage checks, the Buffer Manager will also look for any flow entries that match the following criteria and mark them for “DELETING”.

• All the dependencies on a flow are in FINISHED state.

• The state is in COMPLETED, INCOMPLETE, or FAILED.

• The flow’s expiry_time entry is set (and is positive).

• And that the last_updated + expiry_time is in the past.

In addition, it also periodically checks data-product flows and moves data associated with the ones in DELETING state to a “recycle bin”, which is a reserved area in the data PVC. Once a data product is in the “recycle bin”, the state of its associated flow is set to DELETED.

Users can then go to the recycle_bin directory and delete the data products manually if they are no longer needed.

If a previous destination directory already exists, the Buffer Manager appends a Unix timestamp suffix to the destination name (e.g. test_orphan2_1754221441) so that the original recycle bin entry is preserved and the new move always succeeds.

Once a data product has been moved, the Buffer Manager also removes any ancestor directories that are now empty (e.g. product/<eb_id>/ska-sdp/ and product/<eb_id>/), keeping the product/ directory clean.

For directories with a scan level (see Directory scan levels), only the entry itself is moved. The directories above it are kept, even if they are now empty. For example, removing users/<username>/my_test_set keeps users/<username>/.