Adding and removing IBFT validators
Over the lifetime of an IBFT network, validators will need to be added and removed as authorities change. Here we will showcase adding a new validator to an IBFT network and removing an existing one.
Adding a node to the validator set
Adding a node to the IBFT validator set is relatively easy once a node is part of the network. It does not matter whether the node is already online or not, as the process to add the new node as a validator only needs the existing validators.
Warning
If you are adding multiple validators before they are brought online, make sure you don’t go over the BFT limit and cause the chain to stop progressing.
Adding a new validator requires that a majority of existing validators propose the new node to be added. This is
achieved by calling the propose
RPC method with the value true
and replacing the address to your required one:
geth attach /qdata/dd/geth.ipc
In Geth console, run:
istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);
Returns null
.
This indicates that the current node wishes to add address 0xb131288f355bc27090e542ae0be213c20350b767
as a new
validator.
Example
You can find the resources required to run the examples in the quorum-examples repository.
-
The examples use
docker-compose
for the container definitions. If you are following along by copying the commands described, then it is important to set the project name for Docker Compose, or to remember to change the prefix for your directory. See Docker documentation for more details.To set the project name, run the following:
export COMPOSE_PROJECT_NAME=addnode
-
Bring up the network, which contains 7 nodes, of which 6 are validators.
docker-compose -f ibft-6-validators.yml up
We will be adding the 7th node as a validator. You may notice in the logs of node 7 messages along the lines of
node7_1 | WARN [01-20|10:37:16.034] Block sealing failed err=unauthorized
. This is because the node was started up with minting enabled, but doesn’t have the authority to create blocks, and so throws this error. -
Now we need to propose node 7 as a new proposer from the existing nodes.
Note
Remember, you could do this stage before starting node 7 in your network
We need a majority of existing validators to propose the new node before the changes will take effect.
Lets start with node 1 and see what happens:
Propose node 7 from node 1:
docker exec -it addnode_node1_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc
Returns
null
.Wait about 5 seconds, and then run:
docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
Result is:
{ epoch: 30000, hash: "0xf814863d809ce3a683ee0a2197b15a8152d2696fc9c4e47cd82d0bd5cdaa3e45", number: 269, policy: 0, tally: { 0xb131288f355bc27090e542ae0be213c20350b767: { authorize: true, votes: 1 } }, validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], votes: [{ address: "0xb131288f355bc27090e542ae0be213c20350b767", authorize: true, block: 268, validator: "0xd8dba507e85f116b1f7e231ca8525fc9008a6966" }] }
Let’s break this down. Firstly, we proposed the address
0xb131288f355bc27090e542ae0be213c20350b767
to be added; that is what thetrue
parameter is for. If we had set it tofalse
, that means we want to remove an existing validator with that address.Secondly, we fetched the current snapshot, which gives us an insight into the current running state of the voting. We can see that the new address has 1 vote under the
tally
section, and that one vote is described under thevotes
section. Your vote was registered! -
Let’s run this from node 2 and see similar results:
docker exec -it addnode_node2_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc
Returns
null
.Again, you may have to wait 5 - 10 seconds for the snapshot to show the vote
docker exec -it addnode_node2_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
Result is:
{ epoch: 30000, hash: "0x93efcd458f3b875902a4532bb77d5e7ebb701791ea95486ecd58baf682312d74", number: 391, policy: 0, tally: { 0xb131288f355bc27090e542ae0be213c20350b767: { authorize: true, votes: 2 } }, validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], votes: [{ address: "0xb131288f355bc27090e542ae0be213c20350b767", authorize: true, block: 388, validator: "0xd8dba507e85f116b1f7e231ca8525fc9008a6966" }, { address: "0xb131288f355bc27090e542ae0be213c20350b767", authorize: true, block: 390, validator: "0x6571d97f340c8495b661a823f2c2145ca47d63c2" }] }
True to form, you have the second vote registered!
-
Ok, let’s finally vote on nodes 3 and 4.
docker exec -it addnode_node3_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc
Returns
null
.docker exec -it addnode_node4_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc
Returns
null
. -
Now you have a majority of votes, let’s check the snapshot again:
docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
Result is:
{ epoch: 30000, hash: "0xd4234184538297f71f5b7024a2e11f51f06b4f569ebd9e3644abd391b8c66101", number: 656, policy: 0, tally: {}, validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb131288f355bc27090e542ae0be213c20350b767", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], votes: [] }
You can see that the votes have now been wiped clean, ready for a new round. Additionally, the address you were adding,
0xb131288f355bc27090e542ae0be213c20350b767
now exists within thevalidators
list! Lastly, theunauthorized
messages that node 7 was giving before has stopped, as it now has the authority to mint blocks.
Removing a node from the validator set
Removing a validator is similar to adding a node, but this time you want to propose nodes with the value false
,
to indicate you are deauthorising them. It does not matter whether the node is still online or not, as it doesn’t
require any input from the node being removed.
Warning
Be aware when removing nodes that cross the BFT boundary, that is, going from 10 validators to 9, as this may impact the chains ability to progress if other nodes are offline
Removing a new validator requires that a majority of existing validators propose the new node to be removed. This is
achieved by calling the propose
RPC method with the value false
and replacing the address to your required one:
geth attach /qdata/dd/geth.ipc
In Geth console, run:
istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", false);
Returns null
.
Example
You can find the resources required to run the examples in the quorum-examples repository.
-
The examples use
docker-compose
for the container definitions. If you are following along by copying the commands described, then it is important to set the project name for Docker Compose, or to remember to change the prefix for your directory. See Docker documentation for more details.To set the project name, run the following:
export COMPOSE_PROJECT_NAME=addnode
-
Bring up the network, which contains 7 nodes, of which 6 are validators.
Set the environment variable for docker-compose:
export COMPOSE_PROJECT_NAME=addnode
Start the 7 node network, of which 6 are validators:
docker-compose -f ibft-6-validators.yml up
-
Now you need to propose node 6 as the node to remove.
Note
You need a majority of existing validators to propose the new node before the changes will take effect.
Lets start with node 1 and see what happens:
Propose node 7 from node 1:
docker exec -it addnode_node1_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc
Returns
null
.Wait about 5 seconds, and then run:
docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
Result is:
{ epoch: 30000, hash: "0xba9f9b72cad90ae8aee39f352b45f21d5ed5535b4479743e3f39b231fd717792", number: 140, policy: 0, tally: { 0x8157d4437104e3b8df4451a85f7b2438ef6699ff: { authorize: false, votes: 1 } }, validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], votes: [{ address: "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", authorize: false, block: 136, validator: "0xd8dba507e85f116b1f7e231ca8525fc9008a6966" }] }
Let’s break this down.
Firstly, you proposed the address 0x8157d4437104e3b8df4451a85f7b2438ef6699ff
to be removed; that is what the
false
parameter is for.
Secondly, you fetched the current snapshot, which gives us an insight into the current running state of the voting.
You can see that the proposed address has 1 vote under the tally
section, and that one vote is described under the
votes
section. Here, the authorize
section is set to false
, which is inline with our proposal to remove the
validator.
-
You need to get a majority, so let’s run the proposal on 3 more nodes:
docker exec -it addnode_node2_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc
Returns
null
.docker exec -it addnode_node3_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc
Returns
null
.docker exec -it addnode_node4_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc
Returns
null
. -
Let’s check the snapshot now all the required votes are in:
docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
Result is:
{ epoch: 30000, hash: "0x25815a32b086926875ea2c44686e4b20effabc731b2b121ebf0e0f395101eea5", number: 470, policy: 0, tally: {}, validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"], votes: [] }
The validator has been removed from the
validators
list, and you are left with the other 5 still present. You will also see in the logs of node 6 a message likenode6_1 | WARN [01-20|11:35:52.044] Block sealing failed err=unauthorized
. This is because it is still minting blocks, but realises it does not have the authority to push them to any of the other nodes on the network (you will also see this message for node 7, which was never authorised but still set up to mine).