Connmgr closing connections when running on Docker container

[MauDagos]

Currently, we are experiencing an issue with Pushpin version 1.40.1 running as a Docker container.

When subscribers are consuming a feed of information from our Pushpin server, we have observed the following pattern:

• The feed seems to freeze.
• The subscriber is not disconnected.
• We have observed on connmgr that Pushpin seems to disconnect from our backed application recording: [pushpin::connmgr::connection] closing idle connection to 10.0.3.244:8080 for puspin-server
• Then, almost 1 second later we have found: server-conn 0-0-0: process error: CoreHttp(Io(Kind(UnexpectedEof)))

We have tried many configurations with different host SO and different Docker image base, but we always get the error. Even when running the container using the host network (--network=host).

Finally, when we run Pushpin 1.40.1 as a service instead of as a Docker container, we haven’t experienced any of this issue.

Is there a specific configuration needed to run Pushpin 1.40.1 in a Docker container?

Please find below a snippet of the logs, in case it's useful:

[DEBUG] 2025-06-16 09:10:00.620 [pushpin::connmgr::connection] closing idle connection to 10.0.3.244:8080 for puspin-server
[DEBUG] 2025-06-16 09:10:16.703 [pushpin::connmgr::server] server-worker 0: sending keep alives for 2 sessions
[TRACE] 2025-06-16 09:10:16.703 [pushpin::connmgr::zhttpsocket] OUT stream to { "from": "connmgr", "id": [ { "id": "0-0-0", "seq": 2931 }, { "id": "0-1-1", "seq": 1191 } ], "ext": { "multi": true }, "type": "keep-alive" }
[DEBUG] 2025-06-16 09:10:16.713 [pushpin::connmgr::server] server-worker 1: sending keep alives for 1 sessions
[TRACE] 2025-06-16 09:10:16.713 [pushpin::connmgr::zhttpsocket] OUT stream to { "from": "connmgr", "id": "1-0-0", "seq": 987, "ext": { "multi": true }, "type": "keep-alive" }
[TRACE] 2025-06-16 09:10:20.406 [pushpin::connmgr::zhttpsocket] IN stream connmgr { "from": "handler_10", "id": "0-0-0", "seq": 2935, "type": "keep-alive" }
[DEBUG] 2025-06-16 09:10:20.406 [pushpin::connmgr::server] server-worker 0: queued zmq message for 1 conns
[DEBUG] 2025-06-16 09:10:20.406 [pushpin::connmgr::connection] server-conn 0-0-0: handle packet: keep-alive
[TRACE] 2025-06-16 09:10:21.421 [pushpin::connmgr::zhttpsocket] IN stream connmgr { "from": "handler_10", "id": "1-0-0", "seq": 992, "type": "keep-alive" }
[DEBUG] 2025-06-16 09:10:21.421 [pushpin::connmgr::server] server-worker 1: queued zmq message for 1 conns
[DEBUG] 2025-06-16 09:10:21.421 [pushpin::connmgr::connection] server-conn 1-0-0: handle packet: keep-alive
[TRACE] 2025-06-16 09:10:22.430 [pushpin::connmgr::zhttpsocket] IN stream connmgr { "from": "handler_10", "id": "0-1-1", "seq": 1194, "type": "keep-alive" }
[DEBUG] 2025-06-16 09:10:22.431 [pushpin::connmgr::server] server-worker 0: queued zmq message for 1 conns
[DEBUG] 2025-06-16 09:10:22.431 [pushpin::connmgr::connection] server-conn 0-1-1: handle packet: keep-alive
[DEBUG] 2025-06-16 09:11:01.699 [pushpin::connmgr::server] server-worker 0: sending keep alives for 2 sessions
[TRACE] 2025-06-16 09:11:01.699 [pushpin::connmgr::zhttpsocket] OUT stream to { "from": "connmgr", "id": [ { "id": "0-0-0", "seq": 2932 }, { "id": "0-1-1", "seq": 1192 } ], "ext": { "multi": true }, "type": "keep-alive" }
[DEBUG] 2025-06-16 09:11:01.717 [pushpin::connmgr::server] server-worker 1: sending keep alives for 1 sessions
[TRACE] 2025-06-16 09:11:01.717 [pushpin::connmgr::zhttpsocket] OUT stream to { "from": "connmgr", "id": "1-0-0", "seq": 988, "ext": { "multi": true }, "type": "keep-alive" }
[TRACE] 2025-06-16 09:11:05.996 [pushpin::connmgr::zhttpsocket] IN stream connmgr { "from": "handler_10", "id": "0-0-0", "seq": 2936, "type": "keep-alive" }
[DEBUG] 2025-06-16 09:11:05.996 [pushpin::connmgr::server] server-worker 0: queued zmq message for 1 conns
[DEBUG] 2025-06-16 09:11:05.996 [pushpin::connmgr::connection] server-conn 0-0-0: handle packet: keep-alive
[TRACE] 2025-06-16 09:11:07.021 [pushpin::connmgr::zhttpsocket] IN stream connmgr { "from": "handler_10", "id": "1-0-0", "seq": 993, "type": "keep-alive" }
[DEBUG] 2025-06-16 09:11:07.021 [pushpin::connmgr::server] server-worker 1: queued zmq message for 1 conns
[DEBUG] 2025-06-16 09:11:07.021 [pushpin::connmgr::connection] server-conn 1-0-0: handle packet: keep-alive
[TRACE] 2025-06-16 09:11:08.031 [pushpin::connmgr::zhttpsocket] IN stream connmgr { "from": "handler_10", "id": "0-1-1", "seq": 1195, "type": "keep-alive" }
[DEBUG] 2025-06-16 09:11:08.032 [pushpin::connmgr::server] server-worker 0: queued zmq message for 1 conns
[DEBUG] 2025-06-16 09:11:08.032 [pushpin::connmgr::connection] server-conn 0-1-1: handle packet: keep-alive
[TRACE] 2025-06-16 09:11:16.752 [mio::poll] deregistering event source from poller
[DEBUG] 2025-06-16 09:11:16.752 [pushpin::connmgr::connection] server-conn 0-0-0: process error: CoreHttp(Io(Kind(UnexpectedEof)))

[jkarneges]

This feels like a timeout issue at the network routing layer. I'm not sure, but maybe Docker adds a NAT-like behavior.

One thing I notice in the logs you provided is nothing is sent to the client for a whole minute. I suggest sending data periodically to the client to ensure all the routing layers keep respecting the connection. You can use GRIP keep alives so Pushpin does this automatically. Note that the "keep-alive" packets you see in the logs are between internal components and don't actually go to the client.

The "closing idle connection" is probably not related. That happens normally if enough time passes without requests to the backend.