Try our new documentation site (beta).
Connecting Nodes
Every Remote Services cluster starts with a single node. The steps for starting Remote Services on a single node, either as a standard process or as a service, were covered in earlier sections.
Before adding nodes into your cluster, you first need to make sure that the cluster token (property CLUSTER_TOKEN in the configuration file) has the same value in each node and in the Cluster Manager. For better security, we recommend that you change the predefined value of the token by generating a new one and pasting the same value into each node configuration file. You can generate a new token with the following command:
> grb_rs token GRBTK-6o4xujs59WJO5508nmaNwc1TtjZJAL1UcwN4vTD4qK4nata8oLr9GnubyXrLTkggc/aw2A==
Adding nodes with a Cluster Manager
If you have started a Cluster Manager, you add additional nodes using the exact same command you used to add the first node. You do this by providing the Cluster Manager address. The Cluster Manager acts as a registry of nodes of your cluster, and the nodes will then connect between themselves.
> grb_rs --manager=http://mymanager:61080 --port=61000
The MANAGER property can also be set through the configuration file:
MANAGER=http://mymanager:61080 PORT=61000You won't have the opportunity to provide command-line options when starting grb_rs as a service, so your only option in this case is to provide this information through the configuration file.
If you wish to start multiple grb_rs processes on the same machine for testing purposes (this is not recommended for production use), you will need to make sure each instance of grb_rs is started on a different port and using a different data directory. The command grb_rs init will help you by copying the default configuration and the data directory into a current directory.
For example, to start two nodes on the same machine with a hostname of myserver:
- In a first terminal window, create a new directory node1,
- Change your current directory to node1 and run grb_rs init
- Start the first node:
grb_rs --manager=http://mymanager:61080 --port=61000
- In a second terminal window, create a new directory node2,
- Change your current directory to node2 and run grb_rs init
- Start the second node on a different port and connect to the Cluster Manager:
grb_rs --manager=http://mymanager:61080 --port=61001
Adding nodes to a Self-Managed Cluster
If you have not started a Cluster Manager, nodes must be connected to each other. Once you've started a single-node cluster, you can add nodes using the --join flag to grb_rs or the JOIN configuration property. For example, if you've already started a cluster on the default port of server1, you would run the following command on the new node (call it server2) to create a two-node cluster:
> grb_rs --join=server1
In the log output for server2, you should see the result of the handshake between the servers:
info : Node server1, transition from JOINING to ALIVE
Similarly, the log output of server1 will include the line:
info : Node server2, added to the cluster
If you are using a non-default port, you can specify the target node port as part of the node URL in the --join flag. You can specify the port of the current node using the --port flag. You can use different ports on different machines, but it is a good practice to use the same one (port 61000 is typically a good choice). The command would look like this:
> grb_rs --join=server1:61000 --port=61000
The JOIN property can also be set through the configuration file:
JOIN=server1:61000 PORT=61000Again, you won't have the opportunity to provide command-line options when starting grb_rs as a service, so your only option in this case is to provide this information through the configuration file.
Once you've created a multi-node cluster, you can add additional nodes by doing a JOIN with any member node. Furthermore, the --join flag or the JOIN property can take a comma-separated list of node names, so a node can still join a cluster even if one of the member nodes is unavailable. Note that when a list of nodes is specified, the joining node will try to join with all of the specified nodes at the same time. Joining is an asynchronous process, so if some target nodes are not reachable, the joining node will retry before giving up on joining. If all of the nodes are reachable, they will all join and form a single cluster.
If you wish to start multiple grb_rs processes on the same machine for testing purposes (this is not recommended for production use), you will need to make sure each instance of grb_rs is started on a different port and using a different data directory. The command grb_rs init will help you by copying the default configuration and the data directory into a current directory.
For example, to start two nodes on the same machine with a hostname of myserver:
- In a first terminal window, create a new directory node1,
- Change your current directory to node1 and run grb_rs init
- Start the first node:
grb_rs --port=61000
- In a second terminal window, create a new directory node2,
- Change your current directory to node2 and run grb_rs init
- Start the second node on a different port and join the first node:
grb_rs --port=61001 --join=myserver:61000
Checking the status of your cluster
You can use grbcluster to check the status of the cluster:
> grbcluster nodes ID ADDRESS STATUS TYPE LICENSE PROCESSING #Q #R JL IDLE %MEM %CPU b7d037db server1:61000 ALIVE COMPUTE VALID ACCEPTING 0 0 10 <1s 61.42 9.72 eb07fe16 server2:61000 ALIVE COMPUTE VALID ACCEPTING 0 0 8 <1s 61.42 8.82The nodes of the cluster constantly share information about their status. Each node can be in one of the following states:
- ALIVE
- The node is up and running.
- DEGRADED
- The node failed to respond to recent communications.
The node could return to the ALIVE state if it becomes
reachable again. The node will stay in this state until a timeout
(controlled by the configuration property DEGRADED_TIMEOUT), at which point it is considered as FAILED
- FAILED
- The node has been in DEGRADED state for too long,
and has been flagged as FAILED.
A node will remain in the FAILED state for a short time, and it
will eventually be removed from the cluster. If the node comes back
online, it will not re-join the cluster automatically.
- JOINING
- The node is in the process of joining the cluster.
- LEAVING
- The node left the cluster. It will stay in that state for a short time period before being removed from the cluster.
