Here we have a list of known issues when using  Python on the Atos HPCF:

shutil.copy* functions not reliable on Lustre

Versions affected: Any python >=3.8, including the available installations in modules or through conda, tykky, or containers.

There is a bug in the implementation of the shutil.copy* functions in Python that may cause files copied to be incomplete when in Lustre filesystems such as SCRATCH or HPCPERM.

Workaround:

When copying files which are on Lustre-based filesystems from python scripts, the advise would be to avoid using any of the shutil.copy* functions and use the Linux commands directly using the cp command from the Operating system:

import subprocess

source_file = "path/to/source/file.txt"
destination_file = "path/to/destination/file.txt"

subprocess.run(["cp", source_file, destination_file], check=True)


  • No labels