Auto-scaling #47
Description
Is your feature request related to a problem? Please describe.
Solves #46: Large queries can utilize all the resources and pending queries may starve.
Describe the solution you'd like
Auto-scaling: The resource manager spins up and down workers depending on resource demands. Auto-scaling can be achieved easily by having a Kubernetes-compliant deployment.
Describe alternatives you've considered
Limiting the number of cores required by query execution via the num_partitions argument in the execute call.
The standalone cluster resource manager fairness scheduler is buggy and not reliable.