Skip to content
You are reading GoQuorum development version documentation and some displayed features may not be available in the stable release. You can switch to stable version using the version box at screen bottom.

Logs & Errors Consolidation

Although there is sufficient logging in Quorum (geth), the error messages are not always straightforward to decipher. This is our take on cataloguing of error messages along with possible cause and remediation actions, to serve as first point of reference before reaching out to the support team.

Note

The log level “ERROR” is written out in uppercase as part of the log message and can be used for alert monitoring.

Message & Parameters Cause Action
"Could not search for pattern", "pattern", pattern, "contract", contracts[types[i]], "err", err Generated by abigen due to an internal error when generating ABI for a contract Raise an issue ticket
"account listing failed", "error", err Call to Clef API method account_list failed Depends on the error detail in the log message
"operation SelfDerive not supported on external signers" An unsupported operation was performed by an external wallet Contact the third party provider of the wallet
"Failed to enumerate smart card readers", "err", err A failure occurred when scanning for smart card wallets The logged message should contain further error details and determine the action to take
"Failed to enumerate USB devices", "hub", hub.scheme, "vendor", hub.vendorID, "failcount", failcount, "err", err A failure occurred when scanning for USB wallets. The logged message should contain further error details Depends on the error detail in the log message, if not using USB wallets then add --nousb to command line
"Import error", "err", err Message is generated when using geth import and indicates an issue with the import file You need to use a valid import file
"Import error", "file", arg, "err", err Message is generated when using geth import and indicates an issue with the named import file You need to use a valid import file
"Failed to retrieve signer address", "err", err Message is generated when using puppeth and indicates an issue with signer JSON key file Ensure you have a valid key file
"Bootstrap URL invalid", "enode", url, "err", err One or more enode values specified on the command line for a bootstrap node, is not a valid url. You need to provide a correct url
"Invalid smartcard daemon path", "path", path, "type", fi.Mode().String() This message is generated if the socket file specified for the smartcard daemon (pcscd) is not actually a socket file Ensure that the daemon is running and the correct socket file is specified on the command line
"Failed to get signer address", "err", err This indicates that the public address could not be obtained for the signature on a message in an IBFT network. This is potentially an internal error or an issue with the crypto package. Depends on the root cause in the log message - an issue ticket may need to be raised
"Failed to serialize JavaScript exception", "exception", msg, "err", err This is an internal error, occurring if the given message could not be serialized into a javascript message by the Otto JS parser. Raise an issue ticket
"Non contiguous block insert", "number", block.Number(), "hash", block.Hash() Message is generated if an ‘out of sequence’ block is received for insertion into the chain. This usually occurs if node is out of sync or holds a corrupt chain. See instructions under the section on Resolution of database corruption issues
"Non contiguous receipt insert", "number", blockChain[i].Number(), "hash", blockChain[i].Hash(), "parent", blockChain[i].ParentHash() Message is generated if an ‘out of sequence’ receipt is received for insertion. This usually occurs if node is out of sync or holds corrupt data. See instructions under the section on Resolution of database corruption issues
"Found bad hash, rewinding chain", "number", header.Number, "hash", header.ParentHash Message is generated on startup if a block is found in the database with one of a set ‘bad hash’ values predefined in core/blocks.go. The node will rewind the chain to prior to the bad hash and resync from that point No action should be necessary as node will rewind and recover
"Chain rewind was successful, resuming normal operation" This message is generated after the chain has been rewound following a “Found bad hash, rewinding chain” message No action is necessary
"Impossible reorg, please file an issue", "oldnum", oldBlock.Number(), "oldhash", oldBlock.Hash(), "newnum", newBlock.Number(), "newhash", newBlock.Hash() This can occur if there was an issue during sync See section on Impossible reorg for actions
"########## BAD BLOCK #########" This can occur if there was an issue inserting a new block into the chain See section on Bad block for actions
"Failed to commit recent state trie", "err", err This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. See instructions under the section on Resolution of database corruption issues
"Dangling trie nodes after full cleanup" This occurs if the in-memory cache was not fully flushed to the underlying database during shutdown No action is possible
"Failed writing private state root", "err", err This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. See instructions under the section on Resolution of database corruption issues
"Non contiguous header insert", "number", chain[i].Number, "hash", chain[i].Hash() This message is generated if blocks are received with non-contiguous block numbers or block hashes This is most likely a coding error - raise an issue ticket
"Pricing query for empty pool" This message is generated to catch a coding error scenario Raise an issue ticket
"Demoting invalidated transaction", "hash", hash This occurs if a nonce gap appears in the pending queue after housekeeping (which should not happen) No action should be necessary as the transactions are automatically moved to the non-executable queue
"Unrooted old chain seen by tx pool", "block", oldHead.Number, "hash", oldHead.Hash() This can happen on a chain reorg and indicates that the block at the head of the ‘old’ chain did not have a valid parent hash. This may cause issues with the contents of the transaction pool. In order to ensure integrity of the transaction pool, it may be necessary to follow the instructions under the section on Resolution of database corruption issues
"Unrooted new chain seen by tx pool", "block", newHead.Number, "hash", newHead.Hash() This can happen on a chain reorg and indicates that the block at the head of the ‘new’ chain did not have a valid parent hash. This may cause issues with the contents of the transaction pool. In order to ensure integrity of the transaction pool, it may be necessary to follow the instructions under the section on Resolution of database corruption issues
"Failed to reset txpool state", "err", err This indicates that the transaction pool state could not be retrieved after a reorg; the err message will provide more detail This can be down to mis-configuration, or a corrupt database. See instructions under the section on Resolution of database corruption issues
"Impossible fork ID validation", "id", id This indicates that an internal error occurred when validating the fork ID for chain compatibility checks as per EIP-2124 Raise an issue ticket
"Invalid block body RLP", "hash", hash, "err", err This occurs if a block retrieved from the database could not be decode. This implies an internal error or an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Invalid block header RLP", "hash", hash, "err", err This occurs if a block header retrieved from the database could not be decoded. This implies an internal error or an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Invalid receipt array RLP", "hash", hash, "err", err This occurs if the transaction receipts retrieved from the database for a block, could not be decoded. This implies an internal error or an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Missing body but have receipt", "hash", hash, "number", number This occurs if the block body relating to a transaction receipt could not be retrieved from the database. This implies there is an issue with the database or an internal error occurred. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Failed to derive block receipts fields", "hash", hash, "number", number, "err", err This occurs if the transaction information relating to a receipt could not be retrieved from the database. This implies there is an issue with the database or an internal error occurred. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Invalid block total difficulty RLP", "hash", hash, "err", err This occurs if the block difficulty was retrieved from the database but could not be decoded. This implies an internal error or an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Receipt not found", "number", blockNumber, "hash", blockHash, "txhash", hash This occurs if a transaction receipt could not be retrieved from the database for the block which contains it. This implies there is an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Transaction referenced missing", "number", blockNumber, "hash", blockHash This occurs if a transaction could not be retrieved from the database for the block which contains it. This implies there is an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Transaction not found", "number", blockNumber, "hash", blockHash, "txhash", hash This occurs if transaction data could not be retrieved from the database for a known transaction. This implies there is an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Invalid transaction lookup entry RLP", "hash", hash, "blob", data, "err", err This occurs if a transaction retrieved from the database could not be decoded. This implies an internal error or an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Invalid chain config JSON", "hash", hash, "err", err This occurs if the chain configuration JSON read from the database could not be un-marshalled. This implies an internal error or an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Database contains unaccounted data", "size", unaccounted This occurs if unrecognised/unaccounted data is found in the database. This implies there is an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Failed to append ancient hash", "number", f.frozen, "hash", hash, "err", err This occurs if ‘ancient’ data could not be appended to the tables in the freezer database. This implies there is an issue with the database. Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node
"Failed to append ancient header", "number", f.frozen, "hash", hash, "err", err This occurs if ‘ancient’ data could not be appended to the tables in the freezer database. This implies there is an issue with the database. Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node
"Failed to append ancient body", "number", f.frozen, "hash", hash, "err", err This occurs if ‘ancient’ data could not be appended to the tables in the freezer database. This implies there is an issue with the database. Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node
"Failed to append ancient receipts", "number", f.frozen, "hash", hash, "err", err This occurs if ‘ancient’ data could not be appended to the tables in the freezer database. This implies there is an issue with the database. Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node
"Failed to append ancient difficulty", "number", f.frozen, "hash", hash, "err", err This occurs if ‘ancient’ data could not be appended to the tables in the freezer database. This implies there is an issue with the database. Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node
"Current full block number unavailable", "hash", hash This occurs if the block number for the current head block could not be read from the database when checking whether any blocks can be moved to the freezer. This implies there is an issue with the chain database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Current full block unavailable", "number", *number, "hash", hash This occurs if the data for the current head block could not be read from the database when checking whether any blocks can be moved to the freezer. This implies there is an issue with the chain database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"Canonical hash missing, can't freeze", "number", f.frozen This occurs if the hash for a block could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node
"Block header missing, can't freeze", "number", f.frozen, "hash", hash This occurs if a block header could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node
"Block body missing, can't freeze", "number", f.frozen, "hash", hash This occurs if a block could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node
"Block receipts missing, can't freeze", "number", f.frozen, "hash", hash This occurs if the receipts for a block could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node
"Total difficulty missing, can't freeze", "number", f.frozen, "hash", hash This occurs if the block difficuly could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node
"Failed to decode the value returned by iterator", "error", err This is most likely due to an internal error. Raise an issue ticket
"Failed to decode state object", "addr", addr, "err", err This occurs if state data retrieved from the database could not be decoded. This implies an internal error or an issue with the database. Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues
"EIP activation failed", "eip", eip, "error", err This is most likely due to an internal error. Raise an issue ticket
"Unknown metered peer event type", "type", event.Type This occurs if an unknown event message type was received from a peer. This implies an internal error or some incompatibility between nodes. Raise an issue ticket
"Cannot set etherbase in Istanbul consensus" This occurs if an attempt is made to perform the miner_setEtherbase() API call - this is not allowed if Istanbul consensus is in use Action is not possible
"Cannot start mining without etherbase", "err", err This occurs if the etherbase account is not set. Note that for some consensus mechanisms, this is automatically set to the signing account. Ensure node has an etherbase account set up
"Etherbase account unavailable locally", "err", err This occurs if the clique consensus mechanism is used and the configured signing account (etherbase) could not be found in the wallet. Check the configuration - in particular that the signing account defined in the genesis extradata matches the value in the keystore and is set up as a local account
"Propagating dangling block", "number", block.Number(), "hash", hash This occurs when propagating a block to peers, if the block does not have a parent. Raise an issue ticket (this scenario should never occur in normal operation)
"Failed to encode receipt", "err", err The list of receipts for a block could not be RLP encoded. The error detail should give more information and will determine the action to be taken
"Peer removal failed", "peer", id, "err", err A request to remove a peer failed. The error detail should give more information and will determine the action to be taken, but it is most likely caused by attempting to remove a peer that was already removed
"Unknown downloader chain/mode combo", "light", d.lightchain != nil, "full", d.blockchain != nil, "mode", d.mode This occurs if the node was unable to report sync progress due to an invalid combination of downloader type and sync mode. Raise an issue ticket (this scenario should never occur in normal operation)
"Failure in running pprof server", "err", err This occurs if the pprof HTTP server could not be started. The error detail should give more information and will determine the action to be taken
"Database compaction failed", "err", err A database error occurred during DB compaction. The error detail should give more information and will determine the action to be taken. It may be necessary to follow instructions under the section on Resolution of database corruption issues
"Oracle contract binding failed", "err", err This occurs when using a checkpoint oracle contract and means that the node could not bind to the oracle contract. The error detail should give more information and will determine the action to be taken
"Already bound and listening to registrar" This occurs when using a checkpoint oracle contract and means that the there is already a binding in place for the oracle contract. No action is necessary
"Refusing to mine without etherbase" This indicates that the node is set up to mint blocks, but the coinbase account is not set. Check the configuration is correct for the consensus mechanism in use
"Failed to prepare header for mining", "err", err This occurs when a node is attempting to create a block, but fails during the preparation stage when creating the header. The error detail should give more information and will determine the action to be taken
"Failed to create mining context", "err", err This means an error has occurred when setting up the state after creation of a new block. Raise an issue ticket
"Failed to fetch pending transactions", "err", err This error should never occur. Raise an issue ticket
"Block found but no relative pending task", "number", block.Number(), "sealhash", sealhash, "hash", hash This indicates that an internal error has occurred. Raise an issue ticket
"Failed writing block to chain", "err", err This indicates that a failure occurred when committing a block and associated state to the database. The error detail should give more information and will determine the action to be taken
"Failed writing private block bloom", "err", err This indicates that a failure occurred when writing a bloom filter to the database for private transaction receipts. The error detail should give more information and will determine the action to be taken
"Failed to persist node key: %v", err This occurs if the node is unable to save a newly generated node key into the file specified by the configuration. The error detail should give more information and will determine the action to be taken
"Can't load node list file: %v", err This occurs if the node is unable to read static node information from the static-nodes.json (or trusted-nodes.json) file. Check that the file exists in the expected location and that it contains valid JSON
"Node URL %s: %v", url, err This indicates that there is a mis-configured URL in the static-nodes.json (or trusted-nodes.json) file. Check and correct the offending URL in the JSON file
"Read Error for permissioned-nodes.json file. This is because 'permissioned' flag is specified but no permissioned-nodes.json file is present.", "err", err This is self-explanatory Either remove the --permissioned flag or add permissioned-nodes.json file
"parsePermissionedNodes: Failed to access nodes", "err", err This indicates that the file permissioned-nodes.json could not be read. The error detail should give more information and will determine the action to be taken
"parsePermissionedNodes: Failed to load nodes", "err", err This indicates that the node information in permissioned-nodes.json could not be unmarshalled. Check that the contents of the file are well-formed
"parsePermissionedNodes: Node URL blank" This indicates that an entry in permissioned-nodes.json has an empty node URL. Either remove the node details or specify a valid URL
"parsePermissionedNodes: Node URL", "url", url, "err", err This indicates that an entry in permissioned-nodes.json is not valid. Check that the entry with this URL is correctly formatted
"Failed to expire nodedb items: %v", err This message should never be seen Raise an issue ticket
"error encoding packet:", err A message could not be RLP encoded prior to sending to a peer node. The error detail should give more information and will determine the action to be taken
"could not sign packet:", err A message could not be signed prior to sending to a peer node. The error detail should give more information and will determine the action to be taken
"Failed to execute permission action", "action", action, "err", err This indicates a problem when invoking the specified action on the permissions contract. The error detail should give more information and will determine the action to be taken
"bootupNetwork SetPolicy failed", "err", err This indicates an issue occurred when invoking setPolicy() on the permissions contract. The error detail should give more information and will determine the action to be taken, but it is most likely due to misconfiguration of the permissions contract(s)
"bootupNetwork init failed", "err", err This indicates an issue occurred when invoking init() on the permissions contract. The error detail should give more information and will determine the action to be taken, but it is most likely due to misconfiguration of the permissions contract(s)
"failed to updated network boot status", "error", err This indicates an issue occurred when invoking updateNetworkBootStatus() on the permissions contract. The error detail should give more information and will determine the action to be taken, but it is most likely due to misconfiguration of the permissions contract(s)
"failed to get raft id", "err", err, "enodeId", enodeId This occurs with raft consensus, if the permission contract generates a NodeDeactivated event, but a raft id could not be found for the enodeId that is being removed. Check whether the node has already been removed, if not then this may be an internal error and a ticket should be raised
"failed parse node id", "err", err, "enodeId", enodeId This occurs with clique/istanbul consensus, if the permission contract generates a NodeDeactivated event, but the enodeId could not be correctly parsed for the node that is being removed. Raise an issue ticket
"Revoke role - cache is missing role", "org", evtRoleRevoked.OrgId, "role", evtRoleRevoked.RoleId This occurs if the permission contract generates a RoleRevoked event, but the specified role could not be found in the cache. Raise an issue ticket
"can't open file", "file", fullPath, "error", err This indicates that the specified file could not be opened. Check that the file exists at the given path and has read permission
"error reading file", "err", err, "file", fullPath This indicates that the specified file could not be read. The error detail should give more information and will determine the action to be taken
"error unmarshalling the file", "err", err, "file", fullPath This indicates that the contents of the specified file could not be unmarshalled. Check that the file contains valid JSON
"Read Error for disallowed-nodes.json file", "err", err This indicates that the disallowed-nodes.json file could not be read. This message is normally logged the first time a node is being added to the file, in which case no action is needed
"Failed to create disallowed-nodes.json file", "err", err This indicates that the disallowed-nodes.json file could not be created. Check that the data directory location for this file is writeable
"Failed to access the file", "fileName", fileName, "err", err This indicates that the specified file could not be read. Check that the file exists at the given path and has read permission
"Failed to load nodes list from file", "fileName", fileName, "err", err This indicates that the contents of the specified file could not be unmarshalled. Check that the file contains valid JSON
"Error writing new node info to file", "fileName", fileName, "err", err This indicates that the specified file could not be created. Check that the location for this file is writeable
"unable to delegate RPC API calls to plugin", "provider", interfaceName, "error", err This message is generated by the plugin service if it fails to delegate an RPC call to the specified provider. The error detail should give more information and will determine the action to be taken, however it is most likely due to misconfiguration
"error decoding pub key from enodeId", "enodeId", address.NodeId.String(), "err", err This occurs when adding a node for raft consensus, and the enodId could not be decoded into a public key. Ensure the correct enodeId is specified
"failed to extend chain: %s", err.Error() This occurs if the node was unable to add a new block to the chain when using raft consensus. The error detail should give more information and will determine the action to be taken. Note that it may occur during node shutdown with error detail “abort during blocks processing”, in which case it can be ignored.
"error decoding block", "err", err This can occur on a raft node when restoring a snapshot if a block could not be decoded. Most likely the raft snapshot is corrupt. Raft may recover, or a restart may be required; if not, the data directory to be deleted and the node resynced.
"error inserting the block into the chain", "number", block.NumberU64(), "hash", block.Hash(), "err", err This can occur on a raft node when restoring a snapshot if a block could not be added to the chain. The node may be out of sync or hold a corrupt chain. The error detail should give more information. Raft may recover, or a restart may be required; if not, the data directory to be deleted and the node resynced.
"RPC method " + method + " crashed: " + err) This indicates that an issue occurred when responding to an RPC call. The error detail should give more information, however check that the arguments for the RPC call are correct
"Invalid smartcard socket file type", "path", scpath, "type", fi.Mode().String() This message is generated by clef if the socket file specified for the smartcard daemon (pcscd) is not actually a socket file. Ensure that the daemon is running and the correct socket file is specified on the clef command line
"Failed to commit preimage from trie database", "err", err This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. See instructions under the section on Resolution of database corruption issues
"Failed to write flush list to disk", "err", err This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. See instructions under the section on Resolution of database corruption issues
"Failed to commit trie from trie database", "err", err This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. See instructions under the section on Resolution of database corruption issues
"Failed to write trie to disk", "err", err This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. See instructions under the section on Resolution of database corruption issues
"Attempted to dereference the trie cache meta root" This occurs if an error occurred during garbage collection. The node may need a restart
"Unhandled trie error: %v", err This indicates that an error occurred when retrieving data from a trie. The error detail should give more information; it may be neccessary to follow the instructions under the section on Resolution of database corruption issues

Impossible reorg issue

This can occur if there was an issue during sync. There are a number of potential causes:

  • Node was not cleanly shutdown previously, causing database corruption
  • An edge case where two sealers generate blocks at the exact same time (depends on consensus type in use)
  • Insufficient resources (disk or memory)
  • Some (unknown) issue during resync

The action to be taken will depend on the cause. Note that more recent versions of GoQuorum may already include fixes to prevent some of these issues.

In the event of database corruption, see instructions below for Resolution of database corruption issues.

Bad block issue

This can occur if there was an issue when inserting a new block into the chain and is logged in the form:

########## BAD BLOCK #########
Chain config: %v
Number: %v
Hash: 0x%x
'transaction receipt'
Error: %v
##############################

The error detail gives more information as to the root cause. Here are a few possibilities and suggested actions:

  • "invalid gas used": This occurs if the local node calculates a different gas usage for transactions than the node which created the block. This may be due to a configuration issue.
  • "insufficient balance to pay for gas": This is similar to "invalid gas used", but occurs when there is insufficient gas to complete a transaction.
  • "invalid merkle root": Cause is usually a corrupted database or failed import. See instructions under the section on Resolution of database corruption issues.
  • "unknown ancestor": This is usually due to an issue with fast sync. Restarting the node may resolve it.
  • "blacklisted hash": The cause and resolution of this is similar to the "Found bad hash, rewinding chain" error.

Note that more recent versions of GoQuorum may already include fixes to prevent some of these issues.

Resolution of database corruption issues

Corruption of the database (chain data) can occur due to lack of resources, or after a ‘forced’ shutdown. In some cases, GoQuorum will recover automatically or require a restart of the failing node.

If the node does not recover then it may require the chaindata to be deleted and resynced.

The chain data can be removed by shutting down the node and running geth removedb --datadir /path/to/data/directory.

If raft consensus is in use, then the raft logs must also be removed before restarting the node, these consist of all directories containing raft under the data directory.

ConsenSys has acquired Quorum from J.P. Morgan. Please read the FAQ.
Questions or feedback? You can discuss issues and obtain free support on GoQuorum Slack channel.
For paid professional support by ConsenSys, contact us at quorum@consensys.net