Documentation

Processing State and Scaling

Each node of the cluster can be in one of three processing states:

ACCEPTING

The node is accepting new jobs.

DRAINING

The node is not accepting new jobs, but it is still processing existing jobs.

STOPPED

The node is not accepting new jobs and no jobs are running.

A node always starts in the ACCEPTING state. If you need to perform maintenance on a node, or if you want the node to leave the cluster in a clean state for other reasons, the system administrator can issue the stop command:

> grbcluster --server=server1:61000 stop

If jobs are currently running, the state will change to DRAINING until the jobs finish, at which point it will change to STOPPED. If no jobs are running, the state will change to STOPPED immediately. In the DRAINING or STOPPED states, new jobs are rejected on that node. However, the node is still a member of the cluster and all of the other features, commands, and APIs are still active.

Once a node is in the STOPPED state, you can safely remove it from the cluster (to perform maintenance, shut it down, etc.). To return it to the cluster and resume job processing, run the start command:

> grbcluster --server=server1:61000 start

The flag --server is used to target a specific node in the cluster. Adding the --all flag requests that the command (e.g., start or stop) be applied to all nodes in the cluster.

By using the start and stop with a cluster of Compute Servers, you can effectively scale your cluster up or down, depending on the current cluster workload:

• You can scale down the cluster by stopping the processing on some nodes.
• You can scale up the cluster by starting new nodes or resuming processing on some nodes. As soon as a node starts or resumes processing, it will pick up jobs from the current queue or wait for new jobs to be submitted.