Pyxis container import fails with "could not remove squashfs" error despite correct runtime_path configuration

[Tanya1515]

I have configured Slurm with Pyxis extension and Enroot containers. In /etc/slurm/plugstack.conf/pyxis.conf, I set the following configuration:

required /usr/local/lib/slurm/spank_pyxis.so runtime_path=/tmp/enroot_data

The directory /tmp/enroot_data exists and is readable/writable, however, when importing Docker images, it fails with the error:

[2025-11-06T20:36:19.163] [19.0] error: pyxis: child 70956 failed with error code: 1
[2025-11-06T20:36:19.163] [19.0] error: pyxis: failed to import docker image
[2025-11-06T20:36:19.163] [19.0] error: pyxis: printing enroot log file:
[2025-11-06T20:36:19.163] [19.0] pyxis: could not remove squashfs /tmp/enroot_data/1000/19.0.squashfs: No such file or directory
[2025-11-06T20:36:19.163] [19.0] error: pyxis: couldn't start container
[2025-11-06T20:36:19.163] [19.0] error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
[2025-11-06T20:36:19.163] [19.0] error: Failed to invoke spank plugin stack

Steps to Reproduce:

1. Configure Pyxis with runtime_path=/tmp/enroot_data in plugstack.conf

2. Submit a Slurm job with container support:

bash
#SBATCH --container-image=ubuntu

3. Job fails during container image import phase

What I've Tried:

1. Confirmed /tmp/enroot_data directory exists with proper read/write permissions

2. Verified Slurm user can access the directory

3. The directory structure shows /tmp/enroot_data/1000/ but the specific squashfs file mentioned in the error doesn't exist.

Could you please tell me what the problem might be? Thanks in advance.