Handle blocked cron jobs

[joaovictornsv]

Blocked cron jobs occur when the finishedAt field is missing in the cron-history collection, often due to unexpected application termination.

Possible solutions:

• Add event listeners for application termination to handle cleanup, e.g., process.on('SIGTERM', () => { ... }).
  • Note: Not every uncaught exception leads to application termination, so ensure cleanup doesn’t prematurely mark jobs as finished.
• Implement a mechanism to mark jobs as finished if they exceed a configurable timeout (e.g., X minutes).
  • Optionally, add a new field indicating the job was forcefully finished.

[joaovictornsv]

Implement a mechanism to mark jobs as finished if they exceed a configurable timeout (e.g., X minutes)

Regarding the second point, the current option to allow parallel execution already handles this issue.

[JorgenVatle]

Appreciate the effort to maintain the package. 🙌

Just wanted to point out that Meteor doesn't play nicely with process event listeners, specifically SIGTERM for some odd reason. Safest option would be to also include something similar to the parallel timeout option as a failsafe.

Pulling the plug on a server or running process.exit() under certain conditions could also prevent cleanup handlers from firing.

[joaovictornsv]

Appreciate the effort to maintain the package. 🙌

Just wanted to point out that Meteor doesn't play nicely with process event listeners, specifically SIGTERM for some odd reason. Safest option would be to also include something similar to the parallel timeout option as a failsafe.

Pulling the plug on a server or running process.exit() under certain conditions could also prevent cleanup handlers from firing.

Hi Jorgen!

Thank you for sharing this. I was not aware of these limitations. I ran some tests locally, stopping the server in different ways, and the cleaning worked well.

Do you know a specific case when the clean cannot be triggered? I would like to test it as well.

[JorgenVatle]

For SIGTERM, there is an issue where Meteor doesn't fully shut down in development. Hooks are triggered, but each instance of Meteor created every time you make a change to your code is persisted within the current process. So it won't prevent the actual shutdown hooks, but can severely impact performance in development. Production should remain unaffected. In other words, you end up with one new Meteor server every time Meteor restarts.

• Timeouts and intervals not clearing up on live server reload in development environment JorgenVatle/meteor-vite#179
• Registering a SIGTERM event listener causes Meteor dev server instances to remain indefinitely meteor/meteor#13490

Generally this type of issue is pretty minor and shouldn't be of concern since it should be fixed by upstream. However, all modern Meteor versions are affected and it's likely not to be addressed with the next release.

[JorgenVatle]

Apart from that, the hooks are good. I'm just concerned jobs can get blocked if a server is terminated abruptly. So something like a job timeout to handle edge cases like this is important. But I'm in no place to say what should be done, just voicing concern here. 😇