Nomad Agent "Too many open files"

[mpass99]

Related to #612

Investigate the Linux Error Failed to allocate directory watch: Too many open files that appears when having a high number of runners on a single agent.

• In Handle permanently dead Nomad jobs #612 the high number was created by migrations between the agents on restarts. Check if this issue can be reproduced with just one agent and without a restart.

[mpass99]

The warning does not happen for Nomad itself, but for systemctl restart commands.
It is not triggered by systemctl status nomad.service or systemctl start nomad.service. The error can also be triggered when just restarting Docker, instead of Nomad.

In our staging environment, the error happens not with 160 Runner aka 108 Nomad subprocesses, but with 180 Runners.
Maybe the fs.inotify.max_user_instances=128 limit is exceeded.

I would consider this as low priority as it seems to be an error just with using systemctl. Also investigating this issue follows a small path next to crashing the Nomad agents for multiple restarts.

[MrSerth]

Thanks for digging further. Based on your discoveries I agree about the relatively lower priority of this issue 👍. Shall we set it to pending?