Nud freezes when booting up

[Hyena]

I have now full access to the Pi for the next 4-5 days. I hope to get this matter resolved during this time. I can give you all sorts of information that is needed for debugging. The debug.log file is about 200 MiB large and I can upload it. However, before I do that, I’d like you to confirm that the debug.log file does not contain any sensitive information such as private keys or even public keys. In case the debug.log file could contain sensitive information, please let me know so I can think how to remove it before uploading it.

Here’s a piece from the debug file that has been repeating since 30th of August:

2015-08-30 19:15:37 UTC Nu version v2.0.0-beta (2015-08-07 14:14:37 +0200)
2015-08-30 19:15:37 UTC Default data directory /home/pi/.nu
2015-08-30 19:15:37 UTC Loading addresses…
2015-08-30 19:15:37 UTC dbenv.open LogDir=/media/TOURO/nu/data/database ErrorFile=/media/TOURO/nu/data/db.log
2015-08-30 19:15:38 UTC Loaded 6818 addresses
2015-08-30 19:15:38 UTC addresses 1085ms
2015-08-30 19:15:38 UTC Loading block index…
2015-08-30 19:15:38 UTC PPCoin Network: genesis=0x000003cc2da5a0a289ad nBitsLimit=0x1e0fffff nBitsInitial=0x1e00ffff nStakeMinAge=604800 nCoinbaseMaturity=100 nCoinstakeMaturity=5000 nModifierInterval=14400
2015-08-30 19:16:05 UTC ERROR: FetchInputs() : 0634d8808e prev tx 095e5afa7c index entry not found
2015-08-30 19:16:05 UTC ERROR: CTxDB::LoadBlockIndex() : Rebuilding money supply: Failed to load tx inputs
2015-08-30 19:16:05 UTC block index 27050ms
2015-08-30 19:16:05 UTC Loading portfolio…
2015-08-30 19:16:05 UTC Loading wallet for unit S…
2015-08-30 19:16:07 UTC nFileVersion = 2000000
2015-08-30 19:16:08 UTC Error loading blkindex.dat
wallet 2649ms
2015-08-30 19:16:08 UTC Done loading
2015-08-30 19:16:08 UTC mapBlockIndex.size() = 68369
2015-08-30 19:16:08 UTC nBestHeight = -1
2015-08-30 19:16:08 UTC setKeyPool.size() = 101
2015-08-30 19:16:08 UTC mapWallet.size() = 3072
2015-08-30 19:16:08 UTC mapAddressBook.size() = 5
2015-08-30 19:16:08 UTC Loading wallet for unit B…
2015-08-30 19:16:09 UTC nFileVersion = 2000000
2015-08-30 19:16:09 UTC Error loading blkindex.dat
wallet 1335ms
2015-08-30 19:16:09 UTC Done loading
2015-08-30 19:16:09 UTC mapBlockIndex.size() = 68369
2015-08-30 19:16:09 UTC nBestHeight = -1
2015-08-30 19:16:09 UTC setKeyPool.size() = 101
2015-08-30 19:16:09 UTC mapWallet.size() = 0
2015-08-30 19:16:09 UTC mapAddressBook.size() = 1
2015-08-30 19:16:09 UTC Nu: Error loading blkindex.dat

2015-08-30 19:16:09 UTC DBFlush(false)
2015-08-30 19:16:09 UTC addr.dat refcount=0
2015-08-30 19:16:09 UTC addr.dat checkpoint
2015-08-30 19:16:09 UTC addr.dat closed
2015-08-30 19:16:09 UTC blkindex.dat refcount=0
2015-08-30 19:16:09 UTC blkindex.dat checkpoint
2015-08-30 19:16:09 UTC blkindex.dat closed
2015-08-30 19:16:09 UTC walletB.dat refcount=0
2015-08-30 19:16:09 UTC walletB.dat checkpoint
2015-08-30 19:16:09 UTC walletB.dat detach
2015-08-30 19:16:10 UTC walletB.dat closed
2015-08-30 19:16:10 UTC walletS.dat refcount=0
2015-08-30 19:16:10 UTC walletS.dat checkpoint
2015-08-30 19:16:10 UTC walletS.dat detach
2015-08-30 19:16:11 UTC walletS.dat closed
2015-08-30 19:16:11 UTC StopNode()
2015-08-30 19:16:12 UTC DBFlush(true)
2015-08-30 19:16:12 UTC addr.dat refcount=0
2015-08-30 19:16:12 UTC addr.dat checkpoint
2015-08-30 19:16:12 UTC addr.dat closed
2015-08-30 19:16:12 UTC Nu exiting

2015-08-30 19:25:37 UTC

This is just before the previous log spam started:

2015-08-30 19:07:12 UTC sending getdata: block 1d1690f3da77f555dfcc
2015-08-30 19:07:12 UTC sending getdata: block a5d937fcf110734f48a1
2015-08-30 19:07:12 UTC 2015-08-30 19:07:12 UTC sending: getdata (17787 bytes)
2015-08-30 19:07:13 UTC 2015-08-30 19:07:13 UTC received: addr (30003 bytes)
2015-08-30 19:07:13 UTC Added 36 addresses from 114.215.118.148: 23 tried, 6832 new
2015-08-30 19:07:14 UTC 2015-08-30 19:07:14 UTC received: addr (18183 bytes)
2015-08-30 19:07:14 UTC Added 16 addresses from 114.215.118.148: 23 tried, 6832 new
2015-08-30 19:07:15 UTC 2015-08-30 19:07:15 UTC received: block (608 bytes)
2015-08-30 19:07:15 UTC received block ee6778ded3d4aaaca98d3542ece4f02a7397a695bfd87367b8604fd32ab57a00
2015-08-30 19:07:15 UTC CheckStakeKernelHash() : using modifier 0x5e91a95ccceabccf at height=41420 timestamp=2014-10-10 00:01:04 UTC for block from height=33194 timestamp=2014-10-04 02:24:04 UTC
2015-08-30 19:07:15 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x5e91a95ccceabccf nTimeBlockFrom=1412389444 nTxPrevOffset=160 nTimeTxPrev=1412389444 nPrevout=1 nTimeTx=1414549654 hashProof=0000016b8bc3beb8676165f0af7bfd0b2ec5ee6b3da560f3a657ac771ec1f266
2015-08-30 19:07:15 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:15 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68361 nTime=1414549650
2015-08-30 19:07:15 UTC Postponing 6 reconnects
2015-08-30 19:07:15 UTC REORGANIZE
2015-08-30 19:07:15 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:15 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:15 UTC REORGANIZE: done
2015-08-30 19:07:16 UTC 

************************
EXCEPTION: 11DbException       
Db::put: Invalid argument       
nu in ProcessMessages()       

2015-08-30 19:07:16 UTC ProcessMessage(block, 608 bytes) FAILED
2015-08-30 19:07:16 UTC 2015-08-30 19:07:16 UTC received: block (696 bytes)
2015-08-30 19:07:16 UTC received block 124e948bd4f54d6a099a8f8575a78c0c7e769aedf714fbd22d3923e605de6769
2015-08-30 19:07:17 UTC CheckStakeKernelHash() : using modifier 0xe5eaf1afc04c51d5 at height=37836 timestamp=2014-10-07 12:00:51 UTC for block from height=30418 timestamp=2014-10-01 12:31:05 UTC
2015-08-30 19:07:17 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xe5eaf1afc04c51d5 nTimeBlockFrom=1412166665 nTxPrevOffset=160 nTimeTxPrev=1412166665 nPrevout=1 nTimeTx=1414549665 hashProof=00000b61bcaf56bd75494e194511ed3c1c2763795c256f79d32bced28ff0c92a
2015-08-30 19:07:17 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:17 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68362 nTime=1414549654
2015-08-30 19:07:17 UTC connection timeout
2015-08-30 19:07:17 UTC Postponing 7 reconnects
2015-08-30 19:07:17 UTC REORGANIZE
2015-08-30 19:07:17 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:17 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:17 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:17 UTC InvalidChainFound: invalid block=124e948bd4f54d6a099a  height=68363  trust=72612969755
2015-08-30 19:07:17 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:17 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:17 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:17 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:17 UTC 2015-08-30 19:07:17 UTC received: block (5253 bytes)
2015-08-30 19:07:17 UTC received block f07332987aa5bdb27eb48ca1308547fefa442743ed757554a134b2f98bfae92e
2015-08-30 19:07:17 UTC CheckStakeKernelHash() : using modifier 0xb78ae95ceaebd4ce at height=62075 timestamp=2014-10-24 20:03:10 UTC for block from height=53765 timestamp=2014-10-18 21:32:53 UTC
2015-08-30 19:07:17 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xb78ae95ceaebd4ce nTimeBlockFrom=1413667973 nTxPrevOffset=161 nTimeTxPrev=1413667973 nPrevout=1 nTimeTx=1414549767 hashProof=000007478b45cf668a15675cf7f877bd5b01f6451f276c9e22be632735cc91f0
2015-08-30 19:07:17 UTC trying connection 178.143.106.115:7890 lastseen=-2.2hrs
2015-08-30 19:07:17 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:17 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68363 nTime=1414549665
2015-08-30 19:07:17 UTC Postponing 8 reconnects
2015-08-30 19:07:17 UTC REORGANIZE
2015-08-30 19:07:17 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:17 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:17 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:17 UTC InvalidChainFound: invalid block=f07332987aa5bdb27eb4  height=68364  trust=72614074553
2015-08-30 19:07:17 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:17 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:17 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:17 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:18 UTC connected 178.143.106.115:7890
2015-08-30 19:07:18 UTC 2015-08-30 19:07:18 UTC sending: version (129 bytes)
2015-08-30 19:07:18 UTC 2015-08-30 19:07:18 UTC received: block (23918 bytes)
2015-08-30 19:07:18 UTC received block f01c6afe58ae91137377c9230d917711641b7cb852f11db182e6a2cd7f7d6b87
2015-08-30 19:07:18 UTC CheckStakeKernelHash() : using modifier 0xe496ede532bd1f07 at height=13706 timestamp=2014-09-16 16:02:57 UTC for block from height=9666 timestamp=2014-09-10 17:21:47 UTC
2015-08-30 19:07:18 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xe496ede532bd1f07 nTimeBlockFrom=1410369707 nTxPrevOffset=161 nTimeTxPrev=1410369707 nPrevout=1 nTimeTx=1414550164 hashProof=00000ade6f98b1171b1362a99ece7cf9331c7b0c4687ffb7df7d73ce5c7959c4
2015-08-30 19:07:18 UTC trying connection 45.55.43.33:7890 lastseen=-16.5hrs
2015-08-30 19:07:18 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:18 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68364 nTime=1414549767
2015-08-30 19:07:18 UTC Postponing 9 reconnects
2015-08-30 19:07:18 UTC REORGANIZE
2015-08-30 19:07:18 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:18 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:18 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:18 UTC InvalidChainFound: invalid block=f01c6afe58ae91137377  height=68365  trust=72615179197
2015-08-30 19:07:18 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:18 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:18 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:18 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:18 UTC 2015-08-30 19:07:18 UTC received: block (608 bytes)
2015-08-30 19:07:18 UTC received block 491314addc1845be6443acda31828cf8755d1c1512f48c53f5e2d00faba00f25
2015-08-30 19:07:18 UTC CheckStakeKernelHash() : using modifier 0x846be59bcd2c00e3 at height=53084 timestamp=2014-10-18 12:01:09 UTC for block from height=45013 timestamp=2014-10-12 13:19:12 UTC
2015-08-30 19:07:18 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x846be59bcd2c00e3 nTimeBlockFrom=1413119952 nTxPrevOffset=160 nTimeTxPrev=1413119952 nPrevout=1 nTimeTx=1414550192 hashProof=0000032f22dff415238df3253ccd6c6969d6543ed8a59255e55e28a2c729f2ad
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68365 nTime=1414550164
2015-08-30 19:07:19 UTC Postponing 10 reconnects
2015-08-30 19:07:19 UTC REORGANIZE
2015-08-30 19:07:19 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:19 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:19 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:19 UTC InvalidChainFound: invalid block=491314addc1845be6443  height=68366  trust=72616282612
2015-08-30 19:07:19 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:19 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:19 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:19 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:19 UTC 2015-08-30 19:07:19 UTC received: block (523 bytes)
2015-08-30 19:07:19 UTC received block 005f0513d00e90a93b60cf4d6319f9c1ef3bf61027b40afd58969d530566b21c
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : using modifier 0xf37f167cccbac433 at height=51967 timestamp=2014-10-17 20:00:45 UTC for block from height=43886 timestamp=2014-10-11 21:29:08 UTC
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xf37f167cccbac433 nTimeBlockFrom=1413062948 nTxPrevOffset=161 nTimeTxPrev=1413062948 nPrevout=1 nTimeTx=1414550258 hashProof=0000022f00afa50c3b723318a703d1a78f86a6d21fa197e8bd51135d346cbecf
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68366 nTime=1414550192
2015-08-30 19:07:19 UTC Postponing 11 reconnects
2015-08-30 19:07:19 UTC REORGANIZE
2015-08-30 19:07:19 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:19 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:19 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:19 UTC InvalidChainFound: invalid block=005f0513d00e90a93b60  height=68367  trust=72617386144
2015-08-30 19:07:19 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:19 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:19 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:19 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:19 UTC 2015-08-30 19:07:19 UTC received: block (560 bytes)
2015-08-30 19:07:19 UTC received block ad25bce380c4dab8161e1a0e98336bea25e6cc89af7293a65a842c0b9fec184d
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : using modifier 0xc87a550b8e46fbb1 at height=59669 timestamp=2014-10-23 04:00:51 UTC for block from height=51332 timestamp=2014-10-17 05:14:43 UTC
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xc87a550b8e46fbb1 nTimeBlockFrom=1413522883 nTxPrevOffset=160 nTimeTxPrev=1413522883 nPrevout=1 nTimeTx=1414550275 hashProof=00000a9a21191f0059fe185b9138ed3c64305a6a3b8ba22d04ca625ddc21e97b
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68367 nTime=1414550258
2015-08-30 19:07:19 UTC Postponing 12 reconnects
2015-08-30 19:07:19 UTC REORGANIZE
2015-08-30 19:07:19 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:19 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:19 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:19 UTC InvalidChainFound: invalid block=ad25bce380c4dab8161e  height=68368  trust=72618489655
2015-08-30 19:07:19 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:19 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:19 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:19 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:19 UTC 2015-08-30 19:07:19 UTC received: block (609 bytes)
2015-08-30 19:07:19 UTC received block 9e1b962419f98cd2d3d98be69a30bba67badbe0eefdb9e9da04a2777431d4a74
2015-08-30 19:07:19 UTC 2015-08-30 19:07:19 UTC Flushing wallet.dat
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : using modifier 0x102cb2515bee81a8 at height=65101 timestamp=2014-10-26 16:03:18 UTC for block from height=56469 timestamp=2014-10-20 18:26:42 UTC
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x102cb2515bee81a8 nTimeBlockFrom=1413829602 nTxPrevOffset=161 nTimeTxPrev=1413829602 nPrevout=1 nTimeTx=1414550349 hashProof=00000eac292afb1192922f228faac1aa46e9f00488df2188d69f2df1cc993a2e
2015-08-30 19:15:37 UTC 
.
. (empty lines)
.
2015-08-30 19:15:37 UTC Nu version v2.0.0-beta (2015-08-07 14:14:37 +0200)
2015-08-30 19:15:37 UTC Default data directory /home/pi/.nu
2015-08-30 19:15:37 UTC Loading addresses...
...(previous log block spam starts here)

[sigmike]

I’d like you to confirm that the debug.log file does not contain any sensitive information such as private keys or even public keys

I doesn’t contain private keys but it may reveal that you own some addresses (for example if you find a block or if you send or receive transactions). It may also reveal your IP, username, hostname, etc. With some investigation people connected to your node may also identify your node among their peers.

2015-08-30 19:16:05 UTC ERROR: FetchInputs() : 0634d8808e prev tx 095e5afa7c index entry not found2015-08-30 19:16:05 UTC ERROR: CTxDB::LoadBlockIndex() : Rebuilding money supply: Failed to load tx inputs

Db::put: Invalid argument

These are unusual errors. Do you have anything in /media/TOURO/nu/data/db.log?

[mhps]

Which R-pi binary do you use? Did you compile it yourself? I see this error and suspect your blk*.dat are not compatible with your nud

2015-08-30 19:16:09 UTC Error loading blkindex.dat

An efficient way to debug is keeping an uptodate blockchain on a laptop/PC via a different network. Make sure it can mint, then turn off minting on it but keep the wallet online. Then you can copy over teh working block chain any time you want to the Pi (blk* database/* walletS.dat, with 600 permission)

The reason to have them on different networks is that I suspect have two identical wallet behihnd the same IP might get the IP banned by other nodes in some situation.

[Hyena]

This has been spammed many times, but not too many times (only 454 KiB)

BDB2506 file walletB.dat has LSN 2008/4971684, past end of log at 92/7190103
BDB2507 Commonly caused by moving a database from one database environment
BDB2508 to another without clearing the database LSNs, or by removing all of
BDB2509 the log files from a database environment
BDB2516 DB_ENV->log_flush: LSN of 2008/4971684 past current end-of-log of 92/7203245
BDB2517 Database environment corrupt; the wrong log files may have been removed or incompatible database files imported from another environment
BDB0061 PANIC: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
BDB3027 walletB.dat: unable to flush page: 18
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 0
BDB3027 blkindex.dat: unable to flush page: 0
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 773
BDB3027 blkindex.dat: unable to flush page: 773
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 2846
BDB3027 blkindex.dat: unable to flush page: 2846
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 3316
BDB3027 blkindex.dat: unable to flush page: 3316
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 4108
BDB3027 blkindex.dat: unable to flush page: 4108
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 5176
BDB3027 blkindex.dat: unable to flush page: 5176
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 5721
BDB3027 blkindex.dat: unable to flush page: 5721
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 6550
BDB3027 blkindex.dat: unable to flush page: 6550
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 6635
BDB3027 blkindex.dat: unable to flush page: 6635
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 8719
BDB3027 blkindex.dat: unable to flush page: 8719
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 9256
BDB3027 blkindex.dat: unable to flush page: 9256
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 17189
BDB3027 blkindex.dat: unable to flush page: 17189
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 17352
BDB3027 blkindex.dat: unable to flush page: 17352
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 17353
BDB3027 blkindex.dat: unable to flush page: 17353
BDB4519 txn_checkpoint: failed to flush the buffer cache: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

By the way, are the following log lines normal?

2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 35a5ebbabc9ad1441248a31a23c64ec6a01b91eba4601f1080d64b5dbc9748bf
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (534 bytes)
2015-09-15 10:36:01 UTC received block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: liquidity (137 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (537 bytes)
2015-09-15 10:36:01 UTC received block 67209539cbc613659ae6d22b5189aec1962ee26d1f43d6011e0dafdbd2619d9a
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 67209539cbc613659ae6d22b5189aec1962ee26d1f43d6011e0dafdbd2619d9a
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (536 bytes)
2015-09-15 10:36:01 UTC received block df33dacd7a975b30aee5996a4b66b28dbc1f35f3e5a8221f9b152034f901de7c
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block df33dacd7a975b30aee5996a4b66b28dbc1f35f3e5a8221f9b152034f901de7c
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (595 bytes)
2015-09-15 10:36:01 UTC received block b344065bbbd5fbecbfcdae163db622673b9ff2b1dd94816c2ad12405c9b1a21d
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block b344065bbbd5fbecbfcdae163db622673b9ff2b1dd94816c2ad12405c9b1a21d
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (534 bytes)
2015-09-15 10:36:01 UTC received block 8c4893bce24a421866cf1dd9a3b272e8d7fef660e03a907d87966bccb07353a5
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 8c4893bce24a421866cf1dd9a3b272e8d7fef660e03a907d87966bccb07353a5
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (535 bytes)
2015-09-15 10:36:01 UTC received block 35a5ebbabc9ad1441248a31a23c64ec6a01b91eba4601f1080d64b5dbc9748bf
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 35a5ebbabc9ad1441248a31a23c64ec6a01b91eba4601f1080d64b5dbc9748bf
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (534 bytes)
2015-09-15 10:36:01 UTC received block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: liquidity (137 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (537 bytes)
2015-09-15 10:36:01 UTC received block 67209539cbc613659ae6d22b5189aec1962ee26d1f43d6011e0dafdbd2619d9a
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 67209539cbc613659ae6d22b5189aec1962ee26d1f43d6011e0dafdbd2619d9a
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (536 bytes)
2015-09-15 10:36:01 UTC received block df33dacd7a975b30aee5996a4b66b28dbc1f35f3e5a8221f9b152034f901de7c
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block df33dacd7a975b30aee5996a4b66b28dbc1f35f3e5a8221f9b152034f901de7c
2015-09-15 10:36:02 UTC 2015-09-15 10:36:02 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:02 UTC 2015-09-15 10:36:02 UTC received: block (595 bytes)
2015-09-15 10:36:02 UTC received block b344065bbbd5fbecbfcdae163db622673b9ff2b1dd94816c2ad12405c9b1a21d
2015-09-15 10:36:02 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed

Some errors and failures appear periodically in the debug.log file. I started to download everything from scratch again.

[Hyena]

Yes I compiled it myself. Could it be that the external USB drive I use being an NTFS drive could cause any problems?

/dev/sda1: LABEL="TOURO" UUID="EC1EA3241EA2E734" TYPE="ntfs" PARTUUID="f7340f82-01"

When I bought it I forgot to format it with ext4 so turns out I’ve been using it as ntfs ever since not sure how important it is to change it now.

[mhps]

By the way, are the following log lines normal?

Things in this log all happened in a second. It’s not receiving new blocks as a node normally does. You are showing a segment when the client is not yet sync’ed. You should see things like this

received block 4df9a64d998f484c25e6cf23885c399511f56c950766276cf3d1178b11f32267
SetBestChain: new best=4df9a64d998f484c25e6 height=527968 trust=765451674720
moneysupply(S)=877032065.7799 moneysupply(B)=4607984.5713
ProcessBlock: ACCEPTED
accepted liquidity info from B8BxkGL8kK6X9fBxbeEBHwaSTCLuvvHqAM
2015-09-15 11:03:40 UTC Flushing wallet.dat
Flushed wallet.dat 74ms

[Hyena]

Yes I mostly see this stuff in the log that you pointed out. However, sometimes I see those failures.

[sigmike]

2015-09-15 10:36:01 UTC received block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed

That means you received a block but could not find the Proof of Stake input in your transaction database. It may happen, even repeatedly (because you received a chain of blocks but you still miss a large part of the chain) but it should solve itself later. If it doesn’t, it’s likely your database is missing important parts.

The other errors suggest your database is corrupted. You may have a problem with your storing device. I think NTFS should work, but it’s probably more risky than a well tested ext4. You may also want to check your system and kernel logs for errors (if the device or the filesystem failed it should have been reported there when it happened).

[Hyena]

Ok thanks for the info. Let’s see how the downloading goes this time. I’ve been downloading the block chain from scratch for the past 36 hours and it hasn’t got stuck yet. Isn’t it possible to save the block chain in a way that it would not become corrupted so easily? For example, bad blocks could be downloaded again and replaced. The system should be more elastic so that it would overcome such problems. The bigger the block chain the more likely it will become that it becomes corrupted. I suspect that power failure may have caused it. Perhaps nud got killed in the middle of writing something and later it was unable understand that last block or whatever.

[Hyena]

Ok I finished downloading the block chain and it seems that nud is attempting to mint now although I haven’t yet found any blocks. Here’s the screenshot of top now that the block chain has been downloaded.

edit:
any ideas if these errors are normal:

2015-09-17 10:45:41 UTC ThreadDumpAddress exiting
2015-09-17 10:47:49 UTC GetVoteFromDataFeed error: Data feed failed: SSL connect error
2015-09-17 10:47:49 UTC UpdateFromDataFeed failed: Data feed failed: SSL connect error
2015-09-17 10:47:50 UTC ThreadUpdateFromDataFeed exited

also I should point out that I am currently having a situation where nud and bcexchanged are both totally unresponsive. Sent kill -TERM <pid> to them and they just won’t die. This is important to get fixed since I suspect that killing the daemon might get the block chain corrupted.

[mhps]

That “72,6 wa” indicates that your CPU is spending lots of time waiting for IO. That IO is from swapping. Thrashing actually. The 6+ load shows that y our system is very stressed. I bet the debug.log shows wallet flushing times in several to several tens seconds, which normally should be tens of millisec…

I suggest get another pi for bced.

[masterOfDisaster]

Why not trying “ramdrive-nud”?
I mean, it takes some MB of the RAM, but as flushing the wallets might cause a lot of the IO, it could still be better with less RAM left, but flushing wallets at RAM speed (will only take msecs!).

Compare

 tail ~/.nu/debug.log -n 1000 | grep Flushed

with the wallets on disk and the wallets on ramdrive to find out whether it helps to move the wallets to RAM.

[Hyena]

I’m not going to get another pi because of that because I think a pi should handle multiple PoS daemons with its 1 GiB of RAM. So I expect some optimizations to be done. Before that happens I might just stop minting nushares, since they are cheaper. Alternatively I could make a script that would make sure one day nud gets to run and the other day bcexcanged gets to run. Ideally a good PoS coin should not give a penalty for that since I should accumulate coin days and get a higher chance of minting a block the day I run the daemon.

[Hyena]

pi@pi-desktop:~$ tail /media/TOURO/nu/17-09-2015-data/debug.log -n 1000 | grep Flushed
2015-09-17 10:26:47 UTC Flushed wallet.dat 397452ms
2015-09-17 10:54:10 UTC Flushed wallet.dat 886614ms

is this too much?

[masterOfDisaster]

Try to come to your own conclusion:

Raspberry Pi spending all its time flushing wallets?

Here’s the status of the first flushings after starting nud with the wallets on ram drive; faster than before and not written on SD card!

tail ~/.nu/debug.log  -n 500 | grep Flush
DBFlush(false)
DBFlush(true)
2015-09-11 16:35:40 UTC Flushing wallet.dat
Flushed wallet.dat 7ms
2015-09-11 16:36:57 UTC Flushing wallet.dat
Flushed wallet.dat 7ms
2015-09-11 16:37:42 UTC Flushing wallet.dat
Flushed wallet.dat 6ms

Remark: this wallet is empty, but it’s an order of magnitude less time than without wallets on ramdrive:

Raspberry Pi spending all its time flushing wallets?

I just had a look at the debug.log, because another SD card "decided" to stop working properly. The card basically is broken, but I could retrieve the debug.log… The flushing seems to be more often than just when new blocks arrive, right? […] 2015-09-09 13:23:00 UTC Flushing wallet.dat Flushed wallet.dat 351ms 2015-09-09 13:23:25 UTC Flushing wallet.dat Flushed wallet.dat 55ms 2015-09-09 13:24:08 UTC Flushing wallet.dat Flushed wallet.dat 48ms 2015-09-09 13:24:28 UTC Flushing wallet.dat…

[mhps]

I think a pi should handle multiple PoS daemons with its 1 GiB of RAM.

Maybe the next release of nud will do.

I should accumulate coin days and get a higher chance of minting

I think nu doesn’t use coinday when looking for kernels. it only uses coin amount. There is some security reason for it.

886614ms

OMG. Your pi is miserable.

[Hyena]

False alarm about having downloaded the block chain. Turned out that for some reason nud stopped downloading new blocks. It periodically downloaded voting information but no new blocks. I falsely assumed that it got finished. I restarted the system and it started to download again. The blockchain height is currently 483912 but according to blockexplorer.nu the height is 530732 so it will take a while.

[masterOfDisaster]

If ramdrive-nud helps getting the flushing of the wallets in a better condition, you might try ramdrive-bcexchanged as well

[Hyena]

Again it stopped downloading new blocks. This time I see the following failures being spammed in the debug.log file:

pi@pi-desktop:~$ tail -F /media/TOURO/nu/data/debug.log
2015-09-18 07:50:01 UTC connection timeout
2015-09-18 07:50:01 UTC ComputeNextStakeModifier: prev modifier=0x05f184d670d94edd time=2015-09-07 00:01:31 UTC epoch=1441584091
2015-09-18 07:50:01 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=516553 nTime=1441592413
2015-09-18 07:50:02 UTC trying connection 14.192.213.121:7890 lastseen=-559.4hrs
2015-09-18 07:50:02 UTC SetBestChain: new best=f7d533250bb1198dc2f0  height=516554  trust=752130327424  moneysupply(S)=876575780.4714 moneysupply(B)=4602036.0149
2015-09-18 07:50:02 UTC ProcessBlock: ACCEPTED
2015-09-18 07:50:02 UTC 2015-09-18 07:50:02 UTC received: block (633 bytes)
2015-09-18 07:50:02 UTC received block bdc96d88ef830f8a17f29798d18618366cc6f90d5ff21a233ebfa53f3b9c56a4
2015-09-18 07:50:02 UTC CheckStakeKernelHash() : using modifier 0x06e9d201b5234f61 at height=514192 timestamp=2015-09-05 12:01:50 UTC for block from height=505958 timestamp=2015-08-30 12:54:19 UTC
2015-09-18 07:50:02 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x06e9d201b5234f61 nTimeBlockFrom=1440939259 nTxPrevOffset=160 nTimeTxPrev=1440939259 nPrevout=1 nTimeTx=1441592456 hashProof=000000354c3b1e83df5da00988a10883938e0b0ea2bdc8494aa18404384d5dde
2015-09-18 07:50:02 UTC ComputeNextStakeModifier: prev modifier=0x05f184d670d94edd time=2015-09-07 00:01:31 UTC epoch=1441584091
2015-09-18 07:50:02 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=516554 nTime=1441592446
2015-09-18 07:50:03 UTC SetBestChain: new best=bdc96d88ef830f8a17f2  height=516555  trust=752131547484  moneysupply(S)=876575820.4714 moneysupply(B)=4602036.0149
2015-09-18 07:50:03 UTC ProcessBlock: ACCEPTED
2015-09-18 07:50:03 UTC 2015-09-18 07:50:03 UTC received: block (668 bytes)
2015-09-18 07:50:03 UTC received block d739f753ab2715ff6e34615e4f3a263233bf73dc251b4a50418466b5d32c5057
2015-09-18 07:50:04 UTC CheckStakeKernelHash() : using modifier 0x3070c383ddce04de at height=0 timestamp=1970-01-01 00:00:00 UTC for block from height=466553 timestamp=2015-08-03 02:20:04 UTC
2015-09-18 07:50:04 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x3070c383ddce04de nTimeBlockFrom=1438568404 nTxPrevOffset=422 nTimeTxPrev=1438567899 nPrevout=322 nTimeTx=1441592472 hashProof=00000134bf0a07882f9ed9b4714612a3337577f2775e4dae6a5f0b1217b2aaa2
2015-09-18 07:50:04 UTC ComputeNextStakeModifier: prev modifier=0x05f184d670d94edd time=2015-09-07 00:01:31 UTC epoch=1441584091
2015-09-18 07:50:04 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=516555 nTime=1441592456
2015-09-18 07:50:07 UTC connection timeout
2015-09-18 07:50:07 UTC trying connection 77.128.146.141:7890 lastseen=-8842.8hrs
2015-09-18 07:50:12 UTC connection timeout
2015-09-18 07:50:13 UTC trying connection 2.38.211.181:7890 lastseen=-254.3hrs
2015-09-18 07:50:15 UTC Updating from data feed https://raw.githubusercontent.com/cryptog/nu_data_feed/master/cryptog_nu_data_feed.json
2015-09-18 07:50:18 UTC connection timeout
2015-09-18 07:50:18 UTC trying connection 86.135.208.10:7890 lastseen=-2826.6hrs
2015-09-18 07:50:23 UTC connection timeout
2015-09-18 07:50:24 UTC trying connection 46.109.129.53:7890 lastseen=-42.8hrs
2015-09-18 07:50:29 UTC connection timeout
2015-09-18 07:50:29 UTC trying connection 95.188.131.121:7890 lastseen=-3762.9hrs
2015-09-18 07:50:34 UTC connection timeout
2015-09-18 07:50:35 UTC trying connection 85.212.85.147:7890 lastseen=-320.6hrs
2015-09-18 07:50:35 UTC connect() failed after select(): No route to host
2015-09-18 07:50:35 UTC trying connection 210.174.40.53:7890 lastseen=-5165.4hrs
2015-09-18 07:50:40 UTC connect() failed after select(): Connection refused
2015-09-18 07:50:40 UTC trying connection 193.138.219.233:7890 lastseen=-18.8hrs
2015-09-18 07:50:40 UTC connect() failed after select(): Connection refused
2015-09-18 07:50:41 UTC trying connection 217.71.44.124:7890 lastseen=-587.7hrs
2015-09-18 07:50:46 UTC connection timeout

Any ideas why the lastseen parameter is negative?

[Hyena]

Finished downloading the block chain. Again, nud started to freeze at startup and bcexchanged also stopped minting blocks. I stopped the bcexchanged and this time only started nud. It started to work properly and I’ve been minting nushares for some 24 hours now. I think it’s now confirmed that the issue is RAM. Nud consuming more than 700 MiB cannot be ran in parallel with other block chains. Restarting it after every 4 hours did not help either. It froze at startup. I might make a script that would run nud and bcexchange in turns. So 50% of the time nud gets to run and 50% of time bcexchanged gets to run. I guess it is better than running only one of them.