Presto Cluster Autoscaling
Autoscaling helps you manage your Presto clusters by automatically adjusting the number of worker nodes in an Ahana-managed Presto cluster.
Overview
When you create a new Presto cluster it can be challenging to know in advance the number of worker nodes that you need, and more so to predict these requirements for the future. The number of worker nodes ideally should be sized to both ensure efficient performance and to avoid excess costs. Autoscaling can help achieve this balance by adjusting the number of worker nodes automatically as loads change over time, reducing the need for manual intervention.
There are currently two types of scaling strategy you can use with an Ahana-managed Presto cluster:
- Static
- Scale Out only (CPU)
A Static scaling strategy means that the number of worker nodes is constant while the cluster is being used.
A Scale Out only (CPU) scaling strategy means that the number of worker nodes begins at a user-defined minimum number of worker nodes and can increase to a user-defined maximum number, based on the average CPU utilization of the existing worker nodes.
Both scaling strategies have the ability to scale in to a single or a user-set minimum number of worker nodes when the cluster is idle for a user-specified amount of time.
Both scaling strategies can set a Query Termination Grace Period for existing query tasks to complete on a worker node before that worker node is deleted.
Static scaling strategy
The Static scaling strategy is the default strategy when creating a Presto cluster.
In a Static scaling strategy, the defined number of worker nodes is constant when the cluster is running. If Scale to a single worker node when idle is selected, the cluster resizes to a single worker node after no queries have been received by the cluster for a user-specified amount of time. Set the time to wait before scaling in in Time window before scaling to a single worker node.
Scale Out only scaling strategy
The Scale Out only (CPU) scaling strategy gives the ability to define an auto-scaling policy for the Presto cluster. The Presto cluster starts with the Minimum worker node count. If the average CPU utilization of the worker nodes goes above 75%, then the number of worker nodes increases gradually to the Maximum worker node count by adding N number of worker nodes as defined in the Scale Out Step Size.
If no queries are received by the cluster for the period of time set in Time window before scaling to minimum worker node count, the cluster moves to Idle state, which triggers a resize to its Minimum Worker Node Count.
When does autoscaling occur?
Autoscaling occurs when the average CPU utilization of the worker nodes goes above 75% for a period of 15 minutes.
The worker node count will increase until it reaches the Maximum Worker Node Count.
When does idle time occur?
Idle time occurs when there is no queries received by the cluster for a user-defined period of time. The default Time window before scaling to minimum worker node count is 30 minutes.
For a Static cluster the Idle time behaviour must be enabled, however for a Scale Out only (CPU) the Idle time scale in behaviour is enabled by default.
Restrictions and limitations
Currently, a Presto cluster can only scale back to its Minimum Worker Node Count for a Scale out only (CPU) scaling strategy, or Scale to a single worker node when idle for a Static scaling strategy.
The Idle Time Window can be set to between 10 minutes and 60 minutes.
The time window for CPU utilization is not user configurable.