Adding and removing IBFT validators

Over the lifetime of an IBFT network, validators will need to be added and removed as authorities change. Here we will showcase adding a new validator to an IBFT network and removing an existing one.

Adding a node to the validator set

Adding a node to the IBFT validator set is relatively easy once a node is part of the network. It does not matter whether the node is already online or not, as the process to add the new node as a validator only needs the existing validators.

Warning

If you are adding multiple validators before they are brought online, make sure you don’t go over the BFT limit and cause the chain to stop progressing.

Adding a new validator requires that a majority of existing validators propose the new node to be added. This is achieved by calling the propose RPC method with the value true and replacing the address to your required one:

geth attach /qdata/dd/geth.ipc

In Geth console, run:

istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);

Returns null.

This indicates that the current node wishes to add address 0xb131288f355bc27090e542ae0be213c20350b767 as a new validator.

Example

You can find the resources required to run the examples in the quorum-examples repository.

1. The examples use docker-compose for the container definitions. If you are following along by copying the commands described, then it is important to set the project name for Docker Compose, or to remember to change the prefix for your directory. See Docker documentation for more details.

   To set the project name, run the following:

   export COMPOSE_PROJECT_NAME=addnode
   

2. Bring up the network, which contains 7 nodes, of which 6 are validators.

   docker-compose -f ibft-6-validators.yml up
   

   We will be adding the 7^th node as a validator. You may notice in the logs of node 7 messages along the lines of node7_1 | WARN [01-20|10:37:16.034] Block sealing failed err=unauthorized. This is because the node was started up with minting enabled, but doesn’t have the authority to create blocks, and so throws this error.

3. Now we need to propose node 7 as a new proposer from the existing nodes.

   Note

   Remember, you could do this stage before starting node 7 in your network

   We need a majority of existing validators to propose the new node before the changes will take effect.

   Lets start with node 1 and see what happens:

   Propose node 7 from node 1:

   docker exec -it addnode_node1_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc
   

   Returns null.

   Wait about 5 seconds, and then run:

   docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
   

   Result is:

   {
     epoch: 30000,
     hash: "0xf814863d809ce3a683ee0a2197b15a8152d2696fc9c4e47cd82d0bd5cdaa3e45",
     number: 269,
     policy: 0,
     tally: {
       0xb131288f355bc27090e542ae0be213c20350b767: {
         authorize: true,
         votes: 1
       }
     },
     validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"],
     votes: [{
         address: "0xb131288f355bc27090e542ae0be213c20350b767",
         authorize: true,
         block: 268,
         validator: "0xd8dba507e85f116b1f7e231ca8525fc9008a6966"
     }]
   }
   

   Let’s break this down. Firstly, we proposed the address 0xb131288f355bc27090e542ae0be213c20350b767 to be added; that is what the true parameter is for. If we had set it to false, that means we want to remove an existing validator with that address.

   Secondly, we fetched the current snapshot, which gives us an insight into the current running state of the voting. We can see that the new address has 1 vote under the tally section, and that one vote is described under the votes section. Your vote was registered!

4. Let’s run this from node 2 and see similar results:

    docker exec -it addnode_node2_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc
   

   Returns null.

   Again, you may have to wait 5 - 10 seconds for the snapshot to show the vote

   docker exec -it addnode_node2_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
   

   Result is:

   {
     epoch: 30000,
     hash: "0x93efcd458f3b875902a4532bb77d5e7ebb701791ea95486ecd58baf682312d74",
     number: 391,
     policy: 0,
     tally: {
       0xb131288f355bc27090e542ae0be213c20350b767: {
         authorize: true,
         votes: 2
       }
     },
     validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"],
     votes: [{
         address: "0xb131288f355bc27090e542ae0be213c20350b767",
         authorize: true,
         block: 388,
         validator: "0xd8dba507e85f116b1f7e231ca8525fc9008a6966"
     }, {
         address: "0xb131288f355bc27090e542ae0be213c20350b767",
         authorize: true,
         block: 390,
         validator: "0x6571d97f340c8495b661a823f2c2145ca47d63c2"
     }]
   }
   

   True to form, you have the second vote registered!

5. Ok, let’s finally vote on nodes 3 and 4.

   docker exec -it addnode_node3_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc
   

   Returns null.

   docker exec -it addnode_node4_1 geth --exec 'istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", true);' attach /qdata/dd/geth.ipc
   

   Returns null.

6. Now you have a majority of votes, let’s check the snapshot again:

   docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
   

   Result is:

   {
     epoch: 30000,
     hash: "0xd4234184538297f71f5b7024a2e11f51f06b4f569ebd9e3644abd391b8c66101",
     number: 656,
     policy: 0,
     tally: {},
     validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb131288f355bc27090e542ae0be213c20350b767", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"],
     votes: []
   }
   

   You can see that the votes have now been wiped clean, ready for a new round. Additionally, the address you were adding, 0xb131288f355bc27090e542ae0be213c20350b767 now exists within the validators list! Lastly, the unauthorized messages that node 7 was giving before has stopped, as it now has the authority to mint blocks.

Removing a node from the validator set

Removing a validator is similar to adding a node, but this time you want to propose nodes with the value false, to indicate you are deauthorising them. It does not matter whether the node is still online or not, as it doesn’t require any input from the node being removed.

Warning

Be aware when removing nodes that cross the BFT boundary, that is, going from 10 validators to 9, as this may impact the chains ability to progress if other nodes are offline

Removing a new validator requires that a majority of existing validators propose the new node to be removed. This is achieved by calling the propose RPC method with the value false and replacing the address to your required one:

geth attach /qdata/dd/geth.ipc

In Geth console, run:

istanbul.propose("0xb131288f355bc27090e542ae0be213c20350b767", false);

Returns null.

Example

You can find the resources required to run the examples in the quorum-examples repository.

1. The examples use docker-compose for the container definitions. If you are following along by copying the commands described, then it is important to set the project name for Docker Compose, or to remember to change the prefix for your directory. See Docker documentation for more details.

   To set the project name, run the following:

   export COMPOSE_PROJECT_NAME=addnode
   

2. Bring up the network, which contains 7 nodes, of which 6 are validators.

   Set the environment variable for docker-compose:

   export COMPOSE_PROJECT_NAME=addnode
   

   Start the 7 node network, of which 6 are validators:

   docker-compose -f ibft-6-validators.yml up
   

3. Now you need to propose node 6 as the node to remove.

   Note

   You need a majority of existing validators to propose the new node before the changes will take effect.

   Lets start with node 1 and see what happens:

   Propose node 7 from node 1:

   docker exec -it addnode_node1_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc
   

   Returns null.

   Wait about 5 seconds, and then run:

   docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
   

   Result is:

   {
     epoch: 30000,
     hash: "0xba9f9b72cad90ae8aee39f352b45f21d5ed5535b4479743e3f39b231fd717792",
     number: 140,
     policy: 0,
     tally: {
       0x8157d4437104e3b8df4451a85f7b2438ef6699ff: {
         authorize: false,
         votes: 1
       }
     },
     validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0x8157d4437104e3b8df4451a85f7b2438ef6699ff", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"],
     votes: [{
         address: "0x8157d4437104e3b8df4451a85f7b2438ef6699ff",
         authorize: false,
         block: 136,
         validator: "0xd8dba507e85f116b1f7e231ca8525fc9008a6966"
     }]
   }
   

Let’s break this down. Firstly, you proposed the address 0x8157d4437104e3b8df4451a85f7b2438ef6699ff to be removed; that is what the false parameter is for.

Secondly, you fetched the current snapshot, which gives us an insight into the current running state of the voting. You can see that the proposed address has 1 vote under the tally section, and that one vote is described under the votes section. Here, the authorize section is set to false, which is inline with our proposal to remove the validator.

1. You need to get a majority, so let’s run the proposal on 3 more nodes:

   docker exec -it addnode_node2_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc
   

   Returns null.

   docker exec -it addnode_node3_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc
   

   Returns null.

   docker exec -it addnode_node4_1 geth --exec 'istanbul.propose("0x8157d4437104e3b8df4451a85f7b2438ef6699ff", false);' attach /qdata/dd/geth.ipc
   

   Returns null.

2. Let’s check the snapshot now all the required votes are in:

   docker exec -it addnode_node1_1 geth --exec 'istanbul.getSnapshot();' attach /qdata/dd/geth.ipc
   

   Result is:

   {
     epoch: 30000,
     hash: "0x25815a32b086926875ea2c44686e4b20effabc731b2b121ebf0e0f395101eea5",
     number: 470,
     policy: 0,
     tally: {},
     validators: ["0x6571d97f340c8495b661a823f2c2145ca47d63c2", "0xb912de287f9b047b4228436e94b5b78e3ee16171", "0xd8dba507e85f116b1f7e231ca8525fc9008a6966", "0xe36cbeb565b061217930767886474e3cde903ac5", "0xf512a992f3fb749857d758ffda1330e590fa915e"],
     votes: []
   }
   

   The validator has been removed from the validators list, and you are left with the other 5 still present. You will also see in the logs of node 6 a message like node6_1 | WARN [01-20|11:35:52.044] Block sealing failed err=unauthorized. This is because it is still minting blocks, but realises it does not have the authority to push them to any of the other nodes on the network (you will also see this message for node 7, which was never authorised but still set up to mine).

See also

• Adding a new node to the network