hq configuration for HPC clusters

[samuelcajahuaringa]

Hello Hyperqueue team,
I have doubts about hq. To use hq to create a server in login node and create workers but SLURM run jobs on compute nodes. How can I check if this connection is possible? because I have the following problem:

2025-06-09T08:47:42Z INFO Detected 99830218752B of memory (92.97 GiB)
2025-06-09T08:47:42Z INFO SLURM environment detected
2025-06-09T08:47:42Z INFO Starting hyperqueue worker v0.22.0
2025-06-09T08:47:42Z INFO Connecting to: sorgan:46751
2025-06-09T08:47:42Z INFO Listening on port 35679
2025-06-09T08:47:42Z INFO Connecting to server (candidate addresses = [10.68.107.100:46751])
2025-06-09T08:47:42Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:47:44Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:47:46Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:47:48Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:47:50Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:47:52Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:47:54Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:47:56Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:47:58Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:00Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:02Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:04Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:06Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:08Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:10Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:12Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:14Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:16Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:18Z ERROR Could not connect to server, error: No route to host (os error 113)
2025-06-09T08:48:20Z ERROR Could not connect to server, error: No route to host (os error 113)
Error: Server could not be connected

Thanks,
Samuel

[spirali]

Hi,

Is IP address: 10.68.107.100 the place where the server is running?

You can also try to start HQ server inside the SLURM job and check if a worker is able to connect to the server, just to exclude other problems.

[samuelcajahuaringa]

Hi Ada,
Thanks for you soon reply.

Is IP address: 10.68.107.100 the place where the server is running?
How do I check that?

In my case I tried to aboard the task farming problem using hyperqueue, where the tasks are independently but the execution time are totally different, basically is heterogeneous workload problem.
I want to perform 10000 tasks and use only 100 nodes (workers) and assign the tasks dynamically for each available node. I managed to carry out a small test (1 server node and 2 workers node), but for the server was necessary reserve a node instead to use the login node to managed the workers node.

how could I modify my scripts to use the node login for the server?
Below are my results and scripts at the end of this message.

1. reserve a node for hq server

oscar.macollunco@sorgan:/scratch/oscar.macollunco$ sbatch hq_server.slurm
Submitted batch job 3425
oscar.macollunco@sorgan:/scratch/oscar.macollunco$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3345 all remyexp rodrigo. R 3-14:31:20 1 sorgan-cpu1
3425 cpu hq-serve oscar.ma R 0:03 1 sorgan-cpu6
3415 gpu FNO ronaldo. R 12:01:13 1 sorgan-gpu4

2. reserves 2 nodes for hq workers

worker 1:
oscar.macollunco@sorgan:/scratch/oscar.macollunco$ sbatch hq_worker.slurm
Submitted batch job 3426
oscar.macollunco@sorgan:/scratch/oscar.macollunco$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3345 all remyexp rodrigo. R 3-14:32:06 1 sorgan-cpu1
3426 cpu hq-worke oscar.ma R 0:01 1 sorgan-cpu2
3425 cpu hq-serve oscar.ma R 0:49 1 sorgan-cpu6
3415 gpu FNO ronaldo. R 12:01:59 1 sorgan-gpu4

worker 2:
oscar.macollunco@sorgan:/scratch/oscar.macollunco$ sbatch hq_worker.slurm
Submitted batch job 3427
oscar.macollunco@sorgan:/scratch/oscar.macollunco$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
3345 all remyexp rodrigo. R 3-14:35:40 1 sorgan-cpu1
3427 cpu hq-worke oscar.ma R 0:03 1 sorgan-cpu3
3426 cpu hq-worke oscar.ma R 3:35 1 sorgan-cpu2
3425 cpu hq-serve oscar.ma R 4:23 1 sorgan-cpu6
3415 gpu FNO ronaldo. R 12:05:33 1 sorgan-gpu4

checking the workers

oscar.macollunco@sorgan:/scratch/oscar.macollunco$ ./hq worker list --server-dir /scratch/oscar.macollunco/hq-server-dir
+----+---------+---------------------+------------------------+---------+----------------+
| ID | State | Hostname | Resources | Manager | Manager Job ID |
+----+---------+---------------------+------------------------+---------+----------------+
| 1 | RUNNING | sorgan-cpu2.cluster | cpus 24; mem 92.97 GiB | SLURM | 3426 |
| 2 | RUNNING | sorgan-cpu3.cluster | cpus 24; mem 92.97 GiB | SLURM | 3427 |
+----+---------+---------------------+------------------------+---------+----------------+

3. submit the tasks:
   oscar.macollunco@sorgan:/scratch/oscar.macollunco$ ./hq submit --server-dir /scratch/oscar.macollunco/hq-server-dir --array 0-20 --cpus=24 ./run_lammps.sh
   Job submitted successfully, job ID: 1

4. job id that correspond to 21 tasks
   oscar.macollunco@sorgan:/scratch/oscar.macollunco$ ./hq job list --server-dir /scratch/oscar.macollunco/hq-server-dir
   +----+------+---------+-------+
   | ID | Name | State | Tasks |
   +----+------+---------+-------+
   | 1 | bash | RUNNING | 21 |
   +----+------+---------+-------+

5. job info
   oscar.macollunco@sorgan:/scratch/oscar.macollunco$ ./hq job info 1 --server-dir /scratch/oscar.macollunco/hq-server-dir
   +----------------------+---------------------------------------------------+
   | ID | 1 |
   | Name | bash |
   | State | [####################....................] |
   | | RUNNING (2) |
   | | FINISHED (8) |
   | | WAITING (11) |
   | Session | closed |
   | Tasks | 21; Ids: 0-20 |
   | Workers | sorgan-cpu2.cluster,sorgan-cpu3.cluster |
   | Resources | cpus: 24 compact |
   | Priority | 0 |
   | Command | /bin/bash |
   | | ./run_lammps.sh |
   | Stdout | /scratch/oscar.macollunco/job-1/%{TASK_ID}.stdout |
   | Stderr | /scratch/oscar.macollunco/job-1/%{TASK_ID}.stderr |
   | Environment | |
   | Working directory | /scratch/oscar.macollunco |
   | Task time limit | None |
   | Crash limit | 5 |
   | Submission date | 2025-06-09 09:51:03 UTC |
   | Submission directory | /scratch/oscar.macollunco |
   | Makespan | 3m 50s |
   +----------------------+---------------------------------------------------+

scripts:

script hq_server.slurm

#!/bin/bash
#SBATCH --job-name=hq-server
#SBATCH --partition=cpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=48:00:00
#SBATCH --output=slurm-hq-server.%j.out
#SBATCH --error=slurm-hq-server.%j.err

./hq server start --server-dir /scratch/oscar.macollunco/hq-server-dir

script hq_worker.slurm

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=24 # adjust if needed
#SBATCH --exclusive
#SBATCH --partition=cpu
#SBATCH --time=01:00:00
#SBATCH --job-name=hq-worker
#SBATCH --output=slurm-worker-%j.out
#SBATCH --error=slurm-worker-%j.err

export HQ_SERVER_DIR=/scratch/oscar.macollunco/hq-server-dir

cd $SLURM_SUBMIT_DIR

#Start HQ worker (all CPUs on node)
./hq worker start --server-dir $HQ_SERVER_DIR --cpus=24

script run_lammps.sh

#!/bin/bash
mpirun -np 24 /home/oscar.macollunco/lammps-2Apr2025/build/lmp -in in.airebo -var seed $HQ_TASK_ID -l log_$HQ_TASK_ID.lammps

Thanks,
Samuel

[spirali]

You can check the IP address of the current machine by running "ip a" (or "ipconfig" on older systems).

Your configuration looks good at the first glance.

The HQ worker needs to be able to connect to the server. If the network is not configured that way on your cluster, then we cannot do anything about it. In that case, running the server inside a SLURM job is the only option.

[samuelcajahuaringa]

Hi Ada,
Thanks for you reply.
How I check that, to be honest I'm physicists and I know a little about computer science,
the results of ip a in the cluster that I use is:

oscar.macollunco@sorgan:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether ac:1f:6b:5a:15:e8 brd ff:ff:ff:ff:ff:ff
altname enp28s0f0
inet 10.68.107.100/16 brd 10.68.255.255 scope global dynamic noprefixroute eno1
valid_lft 6881sec preferred_lft 6881sec
inet6 fe80::ae1f:6bff:fe5a:15e8/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether ac:1f:6b:5a:15:e9 brd ff:ff:ff:ff:ff:ff
altname enp28s0f1
4: ibp94s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc fq_codel state UP group default qlen 256
link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:b8:59:9f:03:00:89:86:f1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
5: sbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 8e:1d:24:49:55:b1 brd ff:ff:ff:ff:ff:ff
inet 10.22.0.1/16 brd 10.22.255.255 scope global sbr0
valid_lft forever preferred_lft forever
inet6 fe80::8c1d:24ff:fe49:55b1/64 scope link
valid_lft forever preferred_lft forever
6: vethad937c6a@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master sbr0 state UP group default
link/ether e2:c4:0f:8a:13:41 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::e0c4:fff:fe8a:1341/64 scope link
valid_lft forever preferred_lft forever
7: veth0fd7a8e2@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master sbr0 state UP group default
link/ether 9e:64:da:11:de:14 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::9c64:daff:fe11:de14/64 scope link
valid_lft forever preferred_lft forever
8: veth9d02e8be@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master sbr0 state UP group default
link/ether 16:04:1b:a4:42:25 brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::1404:1bff:fea4:4225/64 scope link
valid_lft forever preferred_lft forever

Thanks,
Samuel

[spirali]

It seems that the machine has IP adress: 10.68.107.100.
It is this section:

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether ac:1f:6b:5a:15:e8 brd ff:ff:ff:ff:ff:ff
altname enp28s0f0
inet 10.68.107.100/16 brd 10.68.255.255 scope global dynamic noprefixroute eno1

So it seems that the login node is not allowed to accept connections from worker nodes. So the only option is to start HQ server from a worker node or ask admins to allow this connection.