hostd IO skyrocketed and webui unreachable

[kocoten1992]

Current Behavior

Hi team, I have a strange error:

I notice sometime hostd behave like this (frequently on 1 popular host), it would run for days, I have no choice but to restart hostd so I don't know when (if ever) it will finish.

When it happen, access localhost:9980 is not available (browser keep spinning), and I check my host appear to be offline with both hostscore and siagraph, eventually browser will show this:

{"level":"info","ts":"2026-01-05T22:22:56+07:00","caller":"hostd/main.go:493","msg":"hostd","version":"v2.6.0-beta.3","network":"mainnet","commit":"42c5b17","buildDate":"2025-12-21T07:20:09+07:00"}
{"level":"info","ts":"2026-01-05T22:34:49+07:00","caller":"hostd/run.go:509","msg":"started RHP4 listener","address":"[::]:9984"}
{"level":"info","ts":"2026-01-05T22:34:49+07:00","caller":"hostd/run.go:535","msg":"started RHP4 QUIC listener","address":"[::]:9984"}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"91825ff1","rpc":"Settings","elapsed":19.273593577}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"b007c10c","rpc":"Settings","elapsed":17.92759261}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"3a5f6716","rpc":"Settings","elapsed":29.210799631}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"c72ab314","rpc":"Settings","elapsed":28.714210601}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"dc4d98a4","rpc":"Settings","elapsed":28.303433526}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"dbf51e5c","rpc":"Settings","elapsed":28.294828939}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"2ad0dd94","rpc":"Settings","elapsed":29.37744183}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"163.172.106.39:57530","streamID":"f9f3c422","rpc":"Settings","elapsed":5.220830983}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"6e244975","rpc":"Settings","elapsed":29.378670764}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"48f02569","rpc":"Settings","elapsed":19.271505196}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"a1cbe5a8","rpc":"Settings","elapsed":29.379998492}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.107.134:35884","streamID":"57968aa2","rpc":"Settings","elapsed":19.877203642}
{"level":"info","ts":"2026-01-05T22:35:53+07:00","caller":"hostd/run.go:599","msg":"node started","network":"mainnet","hostKey":"ed25519:e493cfd2246f135a23a304c4aebb1dfd47c84f0ff63af8fb07c4298d1f1a3ce4","http":"[::]:9980","p2p":"[::]:9981"}
{"level":"info","ts":"2026-01-05T22:35:54+07:00","logger":"pin","caller":"pin/pin.go:215","msg":"updated prices","currency":"usd","threshold":"0.01","current":"0.001687","average":"0.001687","last":"0","storage":"219543788008 H","ingress":"94.842916419679 pS","egress":"1.422643746295198 nS"}
{"level":"error","ts":"2026-01-05T22:36:23+07:00","logger":"connectivity","caller":"connectivity/connectivity.go:138","msg":"failed to test connection","error":"failed to test connection: failed to send request: Post \"https://api.siascan.com/troubleshoot\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","consecutiveFailures":0}
{"level":"info","ts":"2026-01-05T22:36:34+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"4e842493","rpc":"Settings","elapsed":0.000442383}
{"level":"info","ts":"2026-01-05T22:36:38+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"b2f8c4c1","rpc":"WriteSector","elapsed":3.20616477}
{"level":"info","ts":"2026-01-05T22:36:42+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"d43c1d95","rpc":"WriteSector","elapsed":1.111556909}
{"level":"info","ts":"2026-01-05T22:36:46+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"212.51.148.7:48911","protocol":"sia/rhp4","streamID":"d4249eb4","rpc":"Settings","elapsed":0.000352556}
{"level":"info","ts":"2026-01-05T22:36:48+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"212.51.148.7:48911","protocol":"sia/rhp4","streamID":"2fa6af61","rpc":"WriteSector","elapsed":1.329216606}
{"level":"info","ts":"2026-01-05T22:37:01+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"2880133a","rpc":"WriteSector","elapsed":1.368322987}
{"level":"info","ts":"2026-01-05T22:37:08+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"c974924f","rpc":"WriteSector","elapsed":1.059577449}
{"level":"info","ts":"2026-01-05T22:37:14+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"984c94e1","rpc":"WriteSector","elapsed":1.075041801}
{"level":"info","ts":"2026-01-05T22:37:22+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"8e319223","rpc":"WriteSector","elapsed":1.386558514}
{"level":"info","ts":"2026-01-05T22:37:35+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"47.186.117.129:59501","streamID":"3dbe61ae","rpc":"Settings","elapsed":0.000429791}
{"level":"info","ts":"2026-01-05T22:37:36+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"b9fcb950","rpc":"WriteSector","elapsed":1.380651812}
{"level":"info","ts":"2026-01-05T22:37:39+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"66.23.193.244:48794","streamID":"8f5584cf","rpc":"Settings","elapsed":0.000501893}
{"level":"info","ts":"2026-01-05T22:37:39+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"66.23.193.244:38535","protocol":"sia/rhp4","streamID":"0902704f","rpc":"Settings","elapsed":0.000409457}
{"level":"info","ts":"2026-01-05T22:37:40+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"66.23.193.244:48806","streamID":"675bc497","rpc":"ReplAccounts","elapsed":0.042057888}
{"level":"info","ts":"2026-01-05T22:37:40+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"66.23.193.244:48806","streamID":"01d97309","rpc":"ReplAccounts","elapsed":0.01324736}
{"level":"info","ts":"2026-01-05T22:37:42+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"2ebd759a","rpc":"WriteSector","elapsed":1.108742458}
{"level":"info","ts":"2026-01-05T22:37:47+07:00","logger":"syncer","caller":"syncer/syncer.go:710","msg":"syncing blocks","peer":"->185.89.253.123:9981","start":"556689::315acc07","n":13}
{"level":"info","ts":"2026-01-05T22:37:53+07:00","logger":"syncer","caller":"syncer/parallel_sync.go:220","msg":"finished sync batch","blocks":13,"elapsed":5.798,"blocks/sec":2.2420355259245426}
{"level":"info","ts":"2026-01-05T22:37:54+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"34db21ef","rpc":"WriteSector","elapsed":1.367303432}
{"level":"info","ts":"2026-01-05T22:38:03+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"161.35.175.162:50452","streamID":"ad9b3cf8","rpc":"Settings","elapsed":0.000472734}
{"level":"info","ts":"2026-01-05T22:38:10+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"c2c3c125","rpc":"WriteSector","elapsed":1.791151363}
{"level":"info","ts":"2026-01-05T22:38:17+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"45407dd2","rpc":"WriteSector","elapsed":2.250209626}
{"level":"info","ts":"2026-01-05T22:38:36+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"163.172.106.39:39554","streamID":"3c066a07","rpc":"LatestRevision","elapsed":0.000492033}
{"level":"info","ts":"2026-01-05T22:38:39+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"1119a0dc","rpc":"WriteSector","elapsed":3.137736133}
{"level":"info","ts":"2026-01-05T22:38:41+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"149.88.104.11:33300","streamID":"6b7bdd01","rpc":"Settings","elapsed":0.000427032}
{"level":"info","ts":"2026-01-05T22:38:48+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"410634db","rpc":"WriteSector","elapsed":2.168910901}
{"level":"error","ts":"2026-01-05T22:38:53+07:00","logger":"connectivity","caller":"connectivity/connectivity.go:138","msg":"failed to test connection","error":"failed to test connection: failed to send request: Post \"https://api.siascan.com/troubleshoot\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","consecutiveFailures":1}
{"level":"info","ts":"2026-01-05T22:39:01+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"ffe9d832","rpc":"WriteSector","elapsed":2.197748102}
{"level":"info","ts":"2026-01-05T22:39:08+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"a98db532","rpc":"WriteSector","elapsed":1.786021432}
{"level":"info","ts":"2026-01-05T22:39:18+07:00","logger":"rhp4.quic","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"204.8.15.207:56734","protocol":"sia/rhp4","streamID":"e455dbaa","rpc":"WriteSector","elapsed":1.896141639}
{"level":"info","ts":"2026-01-05T22:39:42+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"141.147.2.195:52190","streamID":"fe1c5765","rpc":"Settings","elapsed":0.000379384}
{"level":"info","ts":"2026-01-05T22:39:44+07:00","logger":"syncer","caller":"syncer/syncer.go:710","msg":"syncing blocks","peer":"<-146.70.134.252:53586","start":"523927::dd01bb4b","n":8593}
{"level":"info","ts":"2026-01-05T22:39:56+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"a7984855","rpc":"Settings","elapsed":0.000458644}
{"level":"info","ts":"2026-01-05T22:39:56+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"f5750bce","rpc":"Settings","elapsed":0.000595799}
{"level":"info","ts":"2026-01-05T22:39:59+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"97d927be","rpc":"WriteSector","elapsed":2.2980455}
{"level":"info","ts":"2026-01-05T22:39:59+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"bcbf3156","rpc":"WriteSector","elapsed":0.796652897}
{"level":"info","ts":"2026-01-05T22:40:01+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"9bb59770","rpc":"WriteSector","elapsed":0.850298546}
{"level":"info","ts":"2026-01-05T22:40:01+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"fcdcba01","rpc":"WriteSector","elapsed":0.824171765}
{"level":"info","ts":"2026-01-05T22:40:05+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"73f82a8e","rpc":"WriteSector","elapsed":1.563011046}
{"level":"info","ts":"2026-01-05T22:40:13+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"4873dff0","rpc":"WriteSector","elapsed":2.042101944}
{"level":"info","ts":"2026-01-05T22:40:17+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"142.254.17.97:39442","streamID":"b4d3de1a","rpc":"Settings","elapsed":7.616008123}
{"level":"info","ts":"2026-01-05T22:40:23+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"8abf5022","rpc":"WriteSector","elapsed":2.529633614}
{"level":"info","ts":"2026-01-05T22:40:24+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"9779c441","rpc":"WriteSector","elapsed":1.785647229}
{"level":"info","ts":"2026-01-05T22:40:24+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"e7877a09","rpc":"WriteSector","elapsed":1.816213374}
{"level":"error","ts":"2026-01-05T22:40:31+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1202","msg":"RPC failed","peerAddress":"15.204.160.10:44380","streamID":"7f1b9866","rpc":"LatestRevision","error":"failed to lock contract: failed to get contract: transaction failed (attempt 1): contract not found","elapsed":0.000465527}
{"level":"info","ts":"2026-01-05T22:40:34+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.79.230.109:43116","streamID":"558bcafc","rpc":"Settings","elapsed":4.492858086}
{"level":"info","ts":"2026-01-05T22:40:35+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"c8516abd","rpc":"WriteSector","elapsed":2.073971707}
{"level":"info","ts":"2026-01-05T22:40:45+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"4b6cb3cc","rpc":"WriteSector","elapsed":2.024898172}
{"level":"info","ts":"2026-01-05T22:40:46+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"1b3c5270","rpc":"WriteSector","elapsed":1.522181452}
{"level":"info","ts":"2026-01-05T22:40:46+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"34ea8feb","rpc":"WriteSector","elapsed":1.291567298}
{"level":"info","ts":"2026-01-05T22:40:53+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"163.172.106.39:34290","streamID":"f1d425ae","rpc":"Settings","elapsed":7.101359975}
{"level":"info","ts":"2026-01-05T22:40:56+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"6cc260a5","rpc":"WriteSector","elapsed":3.802336698}
{"level":"info","ts":"2026-01-05T22:41:00+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"98eca592","rpc":"WriteSector","elapsed":4.065497836}
{"level":"info","ts":"2026-01-05T22:41:01+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"41433360","rpc":"WriteSector","elapsed":2.574020133}
{"level":"info","ts":"2026-01-05T22:41:01+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"9d3d8dc4","rpc":"WriteSector","elapsed":2.589973618}
{"level":"info","ts":"2026-01-05T22:41:08+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"c18cf481","rpc":"WriteSector","elapsed":2.796621503}
{"level":"info","ts":"2026-01-05T22:41:10+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"66.23.193.244:38450","streamID":"f521d49e","rpc":"AppendSectors","elapsed":21.450628973}
{"level":"info","ts":"2026-01-05T22:41:11+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"574105b6","rpc":"WriteSector","elapsed":3.057617384}
{"level":"info","ts":"2026-01-05T22:41:11+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"5ab7a3b3","rpc":"WriteSector","elapsed":3.257131753}
{"level":"info","ts":"2026-01-05T22:41:14+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"518567fb","rpc":"WriteSector","elapsed":1.590359227}
{"level":"info","ts":"2026-01-05T22:41:18+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"fc5fae20","rpc":"WriteSector","elapsed":2.057171178}
{"level":"info","ts":"2026-01-05T22:41:23+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"ecc3402f","rpc":"WriteSector","elapsed":2.561214795}
{"level":"info","ts":"2026-01-05T22:41:26+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"993f68b6","rpc":"WriteSector","elapsed":1.524091081}
{"level":"info","ts":"2026-01-05T22:41:33+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"4bbe6f37","rpc":"WriteSector","elapsed":2.493563552}
{"level":"info","ts":"2026-01-05T22:41:46+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"51.81.56.133:45622","streamID":"82d6c6fa","rpc":"WriteSector","elapsed":10.213959946}

I suspect some error in an infinite loop but don't know where to start.

Expected Behavior

Should always be able to access webui and some indicator what is going on..

Steps to Reproduce

Sorry, no step to reproduce, possible I've run into a rare bug.

Version

v2.6.0-beta.3

Anything else?

v2.6.0-beta.3 is not the reason, I've been hit with v2.5.1 and many version in the past - could it be databas/blockchain cause this?

[ChrisSchinnerl]

How much data are you storing and when did you update to the beta? The beta contains a fix which is meant to reduce disk I/o but to do so it needs to build a cache. So at the very least every sector you store needs to be read in full once before that cache is done building for all your data. Then the I/O should go down significantly.

Also are you running hostd from an HDD by chance? Since your settings rpc is taking so long to return.

[kocoten1992]

Hi, I upgrade to v2.6.0-beta.3 when it released (2025-12-21) (this release was specify for me because of a bug), the host currently have 7.7TB (was 9TB). I'm using HDD with zfs mirror:

	tank                        ONLINE       0     0     0
	  mirror-0                  ONLINE       0     0     0
	    wwn-0x5000c500e64a2a2a  ONLINE       0     0     0
	    wwn-0x5000cca26fedcfe4  ONLINE       0     0     0
	  mirror-1                  ONLINE       0     0     0
	    wwn-0x5000c500e5cfa83d  ONLINE       0     0     0
	    wwn-0x5000c500e748bb8c  ONLINE       0     0     0
	  mirror-2                  ONLINE       0     0     0
	    wwn-0x5000cca258c30106  ONLINE       0     0     0
	    wwn-0x5000cca290c69218  ONLINE       0     0     0

I could run a specific version to debug this if you need me to.

[n8mgr]

Is hostd.sqlite3 also stored on this ZFS cluster?

[kocoten1992]

yes, it is.

[ChrisSchinnerl]

If you have an SSD in your machine as well we recommend moving hostd on it and only keeping the volumes on your hdds. Keeping the database on HDDs can be a huge bottleneck.

Regarding the I/O, it should subside after a while. You might see random bursts for about 2 weeks but then the cache should be built.

[kocoten1992]

I been thinking about that, but then I need a zfs cluster for ssd as well (that what stop me), just to confirm, nothing is unexpected happen here?

[ChrisSchinnerl]

No, clients will check if sectors are still available on approximately 2 week intervals. Before the fix verifying a sector took a 4MiB read. After the cache is built it will go down to 4kib. After at most 7.7TB of total reads, you should see reduced disk I/O.

[kocoten1992]

Hi @ChrisSchinnerl, is there a way to know if the cache is being built - a sql query or a command line or anything? May need to verify it is actually the issue..

[ChrisSchinnerl]

Yeah you can check the stored_sectors table for rows where cached_subtree_roots is NULL.

e.g.

SELECT COUNT(*) FROM stored_sectors WHERE cached_subtree_roots IS NULL;

That number should go down over time.

[kocoten1992]

The two sql was run ~1h between each other while io rampage was going on, so it is something else I think.

[n8mgr]

Can you provide the full log lines from around that time period?

[kocoten1992]

I check again, it did go down now:

But IO was normal when it go down, will wait until IO go off again and give you the logs while IO frenzy. So far, the cleanest instance of logs is in the first post (#904 (comment)), it was captured from hostd start until it went off.

[kocoten1992]

Hi @n8mgr, there is absolutely no log when IO issue happen:

As you can see, the last log is on:

{"level":"info","ts":"2026-01-07T11:59:55+07:00","logger":"rhp4.siamux","caller":"v4/server.go:1206","msg":"RPC success","peerAddress":"66.23.193.244:41808","streamID":"c48b9ef1","rpc":"ReadSector","elapsed":0.020020606}

But date show Wed Jan 7 02:41:47 PM +07 2026 (~3h after that), furthermore, it look like it does not necessary to go high IO (in the picture just 10MB/s), webui are not accessible during this time and cache built stop.

[kocoten1992]

Also everytime I restart hostd, it take a long time to start up (the check use tcp to check port 9984):

Not sure this is relate to this problem (other host don't have this issue), if it not, will create another issue later.