Nud freezes when booting up

[Hyena]

Just reporting the status. Nud hasn’t still started to mint blocks even though I started to download the whole block chain from scratch 4 days ago. I am sure that the block chains should be downloaded by now but I cannot verify since I’m far away from the Pi and I haven’t set up a RAT. However, bcexchanged has been successfully minting blocks ever since I started nud and bcexchanged processes from the nice command and made them restart once in every 10 hours. From here I can deduce that there is something seriously wrong with nud while bcexchanged is working properly. Any chance that nunet is being attacked? Perhaps my node keeps constantly minting on the forked chain or something? I am almost tempted to switch back to nud 1.0.x branch if only it was compatible. Or alternatively, would it be possible to broadcast CPU intense messages to the NuNet that would cripple Raspberry Pi devices somehow? I suspect that my nud is minting but all the blocks it finds are stale orphans for some reason.

[mhps]

I suggest you temporarily turn on port forward 22 to the pi on your router so you can remotely access the pi.

Note that accessing the pi also disturbs it. nud getinfo accesses the database so is more intrusive, and slow, than less debug.log.

At least in the pre-2.0 version my pi would for unknown reason stop finding blocks. No orphans were found either. So it was really not able to find kernels. It would be nice if there is a debug switch to show that every second the wallet is able to finish looping through all utxos searching for kernels, as I suspect the wallet was too overloaded to go through many utxos.

[Hyena]

My router? The Pi itself is the router . I think I’m pretty firewalled by the ISP. The wallet might be overloaded of course. Could it be that the number of UTXOs in the wallet would decapitate the CPU? If the algorithmic complexity is n^2 and I have really really many UTXOs all under the same address, there may be some bottlenecks that cripple weaker devices first but won’t do shit to the developers and their high speed systems. I’m a programmer myself and I know that having a powerful machine for development makes it easier for me not to optimize the code for speed while writing it.

[mhps]

Could it be that the number of UTXOs in the wallet would decapitate the CPU?

It’s highly likely. The config parameter reserve might be useful to debug by excluding part of the coins from minting and letting one see if the CPU load descreases. But I seem to remember that it excludes the whole address, not part of its utxos. @sigmike

[sigmike]

Could it be that the number of UTXOs in the wallet would decapitate the CPU?

Each UTXO with at least 7 days and 10,000 NSR implies generating a hash and checking it every second. So it’s O(n) but one step is quite CPU intensive. If you have lots of UTXOs it could be too much for your CPU. If checking all your UTXOs takes more than one second then the client will spend all its time doing that.

We could probably check that in the client: for example if 0.9s have passed while processing UTXOs we would stop the process and warn the user.

We could also make the process use multiple cores, but that’s more work.

The config parameter reserve might be useful

Indeed that would limit the amount of UTXOs checked each second. You could try different values and check the CPU usage to reach an amount that do not use all your CPU.

But I seem to remember that it excludes the whole address, not part of its utxos. @sigmike

No, it works on utxos. Basically at the beginning of each minting step all your balance is requested exactly in the same way it would if you tried to send your whole balance, and each UTXO found is checked. When there’s a reserve only the balance minus the reserve is requested. The address concept and the associated balance is only used in human interfaces. The code almost always works with UTXOs.

[mhps]

Indeed that would limit the amount of UTXOs checked each second. You could try different values and check the CPU usage to reach an amount that do not use all your CPU.

Thanks! Now I am looking forward to the next occurance of not being able to find blocks so that I can do some tests.

[sportscliche]

I am experiencing similar issues with the 32-bit 2.0.1 nu-qt client on Ubuntu. I had been minting PPC, BKS, and NSR quite nicely for some time on a low-spec laptop (1 Gb RAM). A couple of days ago, I noticed that the nu blockchain wasn’t synching. I updated the OS for routine maintenance, re-booted, started all three clients and got the following error from nu:

EXCEPTION: 11dbException
Db::put: Cannot allocate memory
nu in ProcessMessages()

ProcessMessage(block, 513 bytes) FAILED
2015-09-05 16:47:45 UTC Flushing wallet.dat

I have minted 4 blocks since then, but they are all orphaned.

To troubleshoot, I decided to do a fresh install of nu on a higher spec computer (4 Gb RAM) with the same OS. The blockchain took 14 hours to download and synch. The wallet is not minting and there are no errors, but conky tells me that nu is consuming ~18% of the total available RAM. On the laptop, nu is using almost 50% of the RAM and appears to have bogged down to the point that it is no longer working. Peerunity and B&C, in contrast, are running/minting just fine, with ~10X less demand for RAM.

[Hyena]

Just telling that I haven’t been able to mint any nushares since 20th of August on the same Pi that is also running bcexchanged and I have minted blockshares every single day. Both of the processes get killed once in every 10 hours to overcome possible freezes/deadlocks. I can soon get access to the debug.log file and I will upload it so that the devs could take a look. I’d really like to get this thing working again on my Pi. Otherwise it doesn’t make any sense to have a PoS ASIC if it doesn’t work.

[mhps]

Make sure to make a screen copy of top then switch to Raspbian and do it again.

[Ben]

@Hyena - I just picked up a new RPi v2b. Hoping to have time this weekend to get it set up and do a little testing to help out the cause.

[masterOfDisaster]

I’m not absolutely sure, but I think the unofficial nud build might be made with libdb5.1.
If you don’t want to have your wallet files converted, you should make your own build with libdb4.8.
The documentation is no up-to-date with regard to compiling nud with libdb4.8.
If you are not familiar with it, you might want to have a look here:

Issue: [Suggestion] "Minting on a Raspberry Pi"

[Hyena]

I have now full access to the Pi for the next 4-5 days. I hope to get this matter resolved during this time. I can give you all sorts of information that is needed for debugging. The debug.log file is about 200 MiB large and I can upload it. However, before I do that, I’d like you to confirm that the debug.log file does not contain any sensitive information such as private keys or even public keys. In case the debug.log file could contain sensitive information, please let me know so I can think how to remove it before uploading it.

Here’s a piece from the debug file that has been repeating since 30th of August:

2015-08-30 19:15:37 UTC Nu version v2.0.0-beta (2015-08-07 14:14:37 +0200)
2015-08-30 19:15:37 UTC Default data directory /home/pi/.nu
2015-08-30 19:15:37 UTC Loading addresses…
2015-08-30 19:15:37 UTC dbenv.open LogDir=/media/TOURO/nu/data/database ErrorFile=/media/TOURO/nu/data/db.log
2015-08-30 19:15:38 UTC Loaded 6818 addresses
2015-08-30 19:15:38 UTC addresses 1085ms
2015-08-30 19:15:38 UTC Loading block index…
2015-08-30 19:15:38 UTC PPCoin Network: genesis=0x000003cc2da5a0a289ad nBitsLimit=0x1e0fffff nBitsInitial=0x1e00ffff nStakeMinAge=604800 nCoinbaseMaturity=100 nCoinstakeMaturity=5000 nModifierInterval=14400
2015-08-30 19:16:05 UTC ERROR: FetchInputs() : 0634d8808e prev tx 095e5afa7c index entry not found
2015-08-30 19:16:05 UTC ERROR: CTxDB::LoadBlockIndex() : Rebuilding money supply: Failed to load tx inputs
2015-08-30 19:16:05 UTC block index 27050ms
2015-08-30 19:16:05 UTC Loading portfolio…
2015-08-30 19:16:05 UTC Loading wallet for unit S…
2015-08-30 19:16:07 UTC nFileVersion = 2000000
2015-08-30 19:16:08 UTC Error loading blkindex.dat
wallet 2649ms
2015-08-30 19:16:08 UTC Done loading
2015-08-30 19:16:08 UTC mapBlockIndex.size() = 68369
2015-08-30 19:16:08 UTC nBestHeight = -1
2015-08-30 19:16:08 UTC setKeyPool.size() = 101
2015-08-30 19:16:08 UTC mapWallet.size() = 3072
2015-08-30 19:16:08 UTC mapAddressBook.size() = 5
2015-08-30 19:16:08 UTC Loading wallet for unit B…
2015-08-30 19:16:09 UTC nFileVersion = 2000000
2015-08-30 19:16:09 UTC Error loading blkindex.dat
wallet 1335ms
2015-08-30 19:16:09 UTC Done loading
2015-08-30 19:16:09 UTC mapBlockIndex.size() = 68369
2015-08-30 19:16:09 UTC nBestHeight = -1
2015-08-30 19:16:09 UTC setKeyPool.size() = 101
2015-08-30 19:16:09 UTC mapWallet.size() = 0
2015-08-30 19:16:09 UTC mapAddressBook.size() = 1
2015-08-30 19:16:09 UTC Nu: Error loading blkindex.dat

2015-08-30 19:16:09 UTC DBFlush(false)
2015-08-30 19:16:09 UTC addr.dat refcount=0
2015-08-30 19:16:09 UTC addr.dat checkpoint
2015-08-30 19:16:09 UTC addr.dat closed
2015-08-30 19:16:09 UTC blkindex.dat refcount=0
2015-08-30 19:16:09 UTC blkindex.dat checkpoint
2015-08-30 19:16:09 UTC blkindex.dat closed
2015-08-30 19:16:09 UTC walletB.dat refcount=0
2015-08-30 19:16:09 UTC walletB.dat checkpoint
2015-08-30 19:16:09 UTC walletB.dat detach
2015-08-30 19:16:10 UTC walletB.dat closed
2015-08-30 19:16:10 UTC walletS.dat refcount=0
2015-08-30 19:16:10 UTC walletS.dat checkpoint
2015-08-30 19:16:10 UTC walletS.dat detach
2015-08-30 19:16:11 UTC walletS.dat closed
2015-08-30 19:16:11 UTC StopNode()
2015-08-30 19:16:12 UTC DBFlush(true)
2015-08-30 19:16:12 UTC addr.dat refcount=0
2015-08-30 19:16:12 UTC addr.dat checkpoint
2015-08-30 19:16:12 UTC addr.dat closed
2015-08-30 19:16:12 UTC Nu exiting

2015-08-30 19:25:37 UTC

This is just before the previous log spam started:

2015-08-30 19:07:12 UTC sending getdata: block 1d1690f3da77f555dfcc
2015-08-30 19:07:12 UTC sending getdata: block a5d937fcf110734f48a1
2015-08-30 19:07:12 UTC 2015-08-30 19:07:12 UTC sending: getdata (17787 bytes)
2015-08-30 19:07:13 UTC 2015-08-30 19:07:13 UTC received: addr (30003 bytes)
2015-08-30 19:07:13 UTC Added 36 addresses from 114.215.118.148: 23 tried, 6832 new
2015-08-30 19:07:14 UTC 2015-08-30 19:07:14 UTC received: addr (18183 bytes)
2015-08-30 19:07:14 UTC Added 16 addresses from 114.215.118.148: 23 tried, 6832 new
2015-08-30 19:07:15 UTC 2015-08-30 19:07:15 UTC received: block (608 bytes)
2015-08-30 19:07:15 UTC received block ee6778ded3d4aaaca98d3542ece4f02a7397a695bfd87367b8604fd32ab57a00
2015-08-30 19:07:15 UTC CheckStakeKernelHash() : using modifier 0x5e91a95ccceabccf at height=41420 timestamp=2014-10-10 00:01:04 UTC for block from height=33194 timestamp=2014-10-04 02:24:04 UTC
2015-08-30 19:07:15 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x5e91a95ccceabccf nTimeBlockFrom=1412389444 nTxPrevOffset=160 nTimeTxPrev=1412389444 nPrevout=1 nTimeTx=1414549654 hashProof=0000016b8bc3beb8676165f0af7bfd0b2ec5ee6b3da560f3a657ac771ec1f266
2015-08-30 19:07:15 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:15 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68361 nTime=1414549650
2015-08-30 19:07:15 UTC Postponing 6 reconnects
2015-08-30 19:07:15 UTC REORGANIZE
2015-08-30 19:07:15 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:15 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:15 UTC REORGANIZE: done
2015-08-30 19:07:16 UTC 

************************
EXCEPTION: 11DbException       
Db::put: Invalid argument       
nu in ProcessMessages()       

2015-08-30 19:07:16 UTC ProcessMessage(block, 608 bytes) FAILED
2015-08-30 19:07:16 UTC 2015-08-30 19:07:16 UTC received: block (696 bytes)
2015-08-30 19:07:16 UTC received block 124e948bd4f54d6a099a8f8575a78c0c7e769aedf714fbd22d3923e605de6769
2015-08-30 19:07:17 UTC CheckStakeKernelHash() : using modifier 0xe5eaf1afc04c51d5 at height=37836 timestamp=2014-10-07 12:00:51 UTC for block from height=30418 timestamp=2014-10-01 12:31:05 UTC
2015-08-30 19:07:17 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xe5eaf1afc04c51d5 nTimeBlockFrom=1412166665 nTxPrevOffset=160 nTimeTxPrev=1412166665 nPrevout=1 nTimeTx=1414549665 hashProof=00000b61bcaf56bd75494e194511ed3c1c2763795c256f79d32bced28ff0c92a
2015-08-30 19:07:17 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:17 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68362 nTime=1414549654
2015-08-30 19:07:17 UTC connection timeout
2015-08-30 19:07:17 UTC Postponing 7 reconnects
2015-08-30 19:07:17 UTC REORGANIZE
2015-08-30 19:07:17 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:17 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:17 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:17 UTC InvalidChainFound: invalid block=124e948bd4f54d6a099a  height=68363  trust=72612969755
2015-08-30 19:07:17 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:17 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:17 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:17 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:17 UTC 2015-08-30 19:07:17 UTC received: block (5253 bytes)
2015-08-30 19:07:17 UTC received block f07332987aa5bdb27eb48ca1308547fefa442743ed757554a134b2f98bfae92e
2015-08-30 19:07:17 UTC CheckStakeKernelHash() : using modifier 0xb78ae95ceaebd4ce at height=62075 timestamp=2014-10-24 20:03:10 UTC for block from height=53765 timestamp=2014-10-18 21:32:53 UTC
2015-08-30 19:07:17 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xb78ae95ceaebd4ce nTimeBlockFrom=1413667973 nTxPrevOffset=161 nTimeTxPrev=1413667973 nPrevout=1 nTimeTx=1414549767 hashProof=000007478b45cf668a15675cf7f877bd5b01f6451f276c9e22be632735cc91f0
2015-08-30 19:07:17 UTC trying connection 178.143.106.115:7890 lastseen=-2.2hrs
2015-08-30 19:07:17 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:17 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68363 nTime=1414549665
2015-08-30 19:07:17 UTC Postponing 8 reconnects
2015-08-30 19:07:17 UTC REORGANIZE
2015-08-30 19:07:17 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:17 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:17 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:17 UTC InvalidChainFound: invalid block=f07332987aa5bdb27eb4  height=68364  trust=72614074553
2015-08-30 19:07:17 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:17 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:17 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:17 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:18 UTC connected 178.143.106.115:7890
2015-08-30 19:07:18 UTC 2015-08-30 19:07:18 UTC sending: version (129 bytes)
2015-08-30 19:07:18 UTC 2015-08-30 19:07:18 UTC received: block (23918 bytes)
2015-08-30 19:07:18 UTC received block f01c6afe58ae91137377c9230d917711641b7cb852f11db182e6a2cd7f7d6b87
2015-08-30 19:07:18 UTC CheckStakeKernelHash() : using modifier 0xe496ede532bd1f07 at height=13706 timestamp=2014-09-16 16:02:57 UTC for block from height=9666 timestamp=2014-09-10 17:21:47 UTC
2015-08-30 19:07:18 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xe496ede532bd1f07 nTimeBlockFrom=1410369707 nTxPrevOffset=161 nTimeTxPrev=1410369707 nPrevout=1 nTimeTx=1414550164 hashProof=00000ade6f98b1171b1362a99ece7cf9331c7b0c4687ffb7df7d73ce5c7959c4
2015-08-30 19:07:18 UTC trying connection 45.55.43.33:7890 lastseen=-16.5hrs
2015-08-30 19:07:18 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:18 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68364 nTime=1414549767
2015-08-30 19:07:18 UTC Postponing 9 reconnects
2015-08-30 19:07:18 UTC REORGANIZE
2015-08-30 19:07:18 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:18 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:18 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:18 UTC InvalidChainFound: invalid block=f01c6afe58ae91137377  height=68365  trust=72615179197
2015-08-30 19:07:18 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:18 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:18 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:18 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:18 UTC 2015-08-30 19:07:18 UTC received: block (608 bytes)
2015-08-30 19:07:18 UTC received block 491314addc1845be6443acda31828cf8755d1c1512f48c53f5e2d00faba00f25
2015-08-30 19:07:18 UTC CheckStakeKernelHash() : using modifier 0x846be59bcd2c00e3 at height=53084 timestamp=2014-10-18 12:01:09 UTC for block from height=45013 timestamp=2014-10-12 13:19:12 UTC
2015-08-30 19:07:18 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x846be59bcd2c00e3 nTimeBlockFrom=1413119952 nTxPrevOffset=160 nTimeTxPrev=1413119952 nPrevout=1 nTimeTx=1414550192 hashProof=0000032f22dff415238df3253ccd6c6969d6543ed8a59255e55e28a2c729f2ad
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68365 nTime=1414550164
2015-08-30 19:07:19 UTC Postponing 10 reconnects
2015-08-30 19:07:19 UTC REORGANIZE
2015-08-30 19:07:19 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:19 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:19 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:19 UTC InvalidChainFound: invalid block=491314addc1845be6443  height=68366  trust=72616282612
2015-08-30 19:07:19 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:19 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:19 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:19 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:19 UTC 2015-08-30 19:07:19 UTC received: block (523 bytes)
2015-08-30 19:07:19 UTC received block 005f0513d00e90a93b60cf4d6319f9c1ef3bf61027b40afd58969d530566b21c
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : using modifier 0xf37f167cccbac433 at height=51967 timestamp=2014-10-17 20:00:45 UTC for block from height=43886 timestamp=2014-10-11 21:29:08 UTC
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xf37f167cccbac433 nTimeBlockFrom=1413062948 nTxPrevOffset=161 nTimeTxPrev=1413062948 nPrevout=1 nTimeTx=1414550258 hashProof=0000022f00afa50c3b723318a703d1a78f86a6d21fa197e8bd51135d346cbecf
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68366 nTime=1414550192
2015-08-30 19:07:19 UTC Postponing 11 reconnects
2015-08-30 19:07:19 UTC REORGANIZE
2015-08-30 19:07:19 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:19 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:19 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:19 UTC InvalidChainFound: invalid block=005f0513d00e90a93b60  height=68367  trust=72617386144
2015-08-30 19:07:19 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:19 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:19 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:19 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:19 UTC 2015-08-30 19:07:19 UTC received: block (560 bytes)
2015-08-30 19:07:19 UTC received block ad25bce380c4dab8161e1a0e98336bea25e6cc89af7293a65a842c0b9fec184d
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : using modifier 0xc87a550b8e46fbb1 at height=59669 timestamp=2014-10-23 04:00:51 UTC for block from height=51332 timestamp=2014-10-17 05:14:43 UTC
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0xc87a550b8e46fbb1 nTimeBlockFrom=1413522883 nTxPrevOffset=160 nTimeTxPrev=1413522883 nPrevout=1 nTimeTx=1414550275 hashProof=00000a9a21191f0059fe185b9138ed3c64305a6a3b8ba22d04ca625ddc21e97b
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: prev modifier=0x64b2645d54a70220 time=2014-10-29 00:05:02 UTC epoch=1414541102
2015-08-30 19:07:19 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=68367 nTime=1414550258
2015-08-30 19:07:19 UTC Postponing 12 reconnects
2015-08-30 19:07:19 UTC REORGANIZE
2015-08-30 19:07:19 UTC REORGANIZE: Disconnect 0 blocks; 2aec402a4b37149f2313..2aec402a4b37149f2313
2015-08-30 19:07:19 UTC REORGANIZE: Connect 1 blocks; 2aec402a4b37149f2313..f87671f3455c255606f9
2015-08-30 19:07:19 UTC ERROR: Reorganize() : ConnectBlock f87671f3455c255606f9 failed
2015-08-30 19:07:19 UTC InvalidChainFound: invalid block=ad25bce380c4dab8161e  height=68368  trust=72618489655
2015-08-30 19:07:19 UTC InvalidChainFound:  current best=2aec402a4b37149f2313  height=68355  trust=72604136079
2015-08-30 19:07:19 UTC ERROR: SetBestChain() : Reorganize failed
2015-08-30 19:07:19 UTC ERROR: AcceptBlock() : AddToBlockIndex failed
2015-08-30 19:07:19 UTC ERROR: ProcessBlock() : AcceptBlock FAILED
2015-08-30 19:07:19 UTC 2015-08-30 19:07:19 UTC received: block (609 bytes)
2015-08-30 19:07:19 UTC received block 9e1b962419f98cd2d3d98be69a30bba67badbe0eefdb9e9da04a2777431d4a74
2015-08-30 19:07:19 UTC 2015-08-30 19:07:19 UTC Flushing wallet.dat
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : using modifier 0x102cb2515bee81a8 at height=65101 timestamp=2014-10-26 16:03:18 UTC for block from height=56469 timestamp=2014-10-20 18:26:42 UTC
2015-08-30 19:07:19 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x102cb2515bee81a8 nTimeBlockFrom=1413829602 nTxPrevOffset=161 nTimeTxPrev=1413829602 nPrevout=1 nTimeTx=1414550349 hashProof=00000eac292afb1192922f228faac1aa46e9f00488df2188d69f2df1cc993a2e
2015-08-30 19:15:37 UTC 
.
. (empty lines)
.
2015-08-30 19:15:37 UTC Nu version v2.0.0-beta (2015-08-07 14:14:37 +0200)
2015-08-30 19:15:37 UTC Default data directory /home/pi/.nu
2015-08-30 19:15:37 UTC Loading addresses...
...(previous log block spam starts here)

[sigmike]

I’d like you to confirm that the debug.log file does not contain any sensitive information such as private keys or even public keys

I doesn’t contain private keys but it may reveal that you own some addresses (for example if you find a block or if you send or receive transactions). It may also reveal your IP, username, hostname, etc. With some investigation people connected to your node may also identify your node among their peers.

2015-08-30 19:16:05 UTC ERROR: FetchInputs() : 0634d8808e prev tx 095e5afa7c index entry not found2015-08-30 19:16:05 UTC ERROR: CTxDB::LoadBlockIndex() : Rebuilding money supply: Failed to load tx inputs

Db::put: Invalid argument

These are unusual errors. Do you have anything in /media/TOURO/nu/data/db.log?

[mhps]

Which R-pi binary do you use? Did you compile it yourself? I see this error and suspect your blk*.dat are not compatible with your nud

2015-08-30 19:16:09 UTC Error loading blkindex.dat

An efficient way to debug is keeping an uptodate blockchain on a laptop/PC via a different network. Make sure it can mint, then turn off minting on it but keep the wallet online. Then you can copy over teh working block chain any time you want to the Pi (blk* database/* walletS.dat, with 600 permission)

The reason to have them on different networks is that I suspect have two identical wallet behihnd the same IP might get the IP banned by other nodes in some situation.

[Hyena]

This has been spammed many times, but not too many times (only 454 KiB)

BDB2506 file walletB.dat has LSN 2008/4971684, past end of log at 92/7190103
BDB2507 Commonly caused by moving a database from one database environment
BDB2508 to another without clearing the database LSNs, or by removing all of
BDB2509 the log files from a database environment
BDB2516 DB_ENV->log_flush: LSN of 2008/4971684 past current end-of-log of 92/7203245
BDB2517 Database environment corrupt; the wrong log files may have been removed or incompatible database files imported from another environment
BDB0061 PANIC: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
BDB3027 walletB.dat: unable to flush page: 18
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 0
BDB3027 blkindex.dat: unable to flush page: 0
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 773
BDB3027 blkindex.dat: unable to flush page: 773
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 2846
BDB3027 blkindex.dat: unable to flush page: 2846
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 3316
BDB3027 blkindex.dat: unable to flush page: 3316
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 4108
BDB3027 blkindex.dat: unable to flush page: 4108
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 5176
BDB3027 blkindex.dat: unable to flush page: 5176
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 5721
BDB3027 blkindex.dat: unable to flush page: 5721
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 6550
BDB3027 blkindex.dat: unable to flush page: 6550
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 6635
BDB3027 blkindex.dat: unable to flush page: 6635
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 8719
BDB3027 blkindex.dat: unable to flush page: 8719
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 9256
BDB3027 blkindex.dat: unable to flush page: 9256
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 17189
BDB3027 blkindex.dat: unable to flush page: 17189
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 17352
BDB3027 blkindex.dat: unable to flush page: 17352
BDB0060 PANIC: fatal region error detected; run recovery
BDB3015 blkindex.dat: write failed for page 17353
BDB3027 blkindex.dat: unable to flush page: 17353
BDB4519 txn_checkpoint: failed to flush the buffer cache: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

By the way, are the following log lines normal?

2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 35a5ebbabc9ad1441248a31a23c64ec6a01b91eba4601f1080d64b5dbc9748bf
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (534 bytes)
2015-09-15 10:36:01 UTC received block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: liquidity (137 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (537 bytes)
2015-09-15 10:36:01 UTC received block 67209539cbc613659ae6d22b5189aec1962ee26d1f43d6011e0dafdbd2619d9a
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 67209539cbc613659ae6d22b5189aec1962ee26d1f43d6011e0dafdbd2619d9a
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (536 bytes)
2015-09-15 10:36:01 UTC received block df33dacd7a975b30aee5996a4b66b28dbc1f35f3e5a8221f9b152034f901de7c
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block df33dacd7a975b30aee5996a4b66b28dbc1f35f3e5a8221f9b152034f901de7c
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (595 bytes)
2015-09-15 10:36:01 UTC received block b344065bbbd5fbecbfcdae163db622673b9ff2b1dd94816c2ad12405c9b1a21d
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block b344065bbbd5fbecbfcdae163db622673b9ff2b1dd94816c2ad12405c9b1a21d
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (534 bytes)
2015-09-15 10:36:01 UTC received block 8c4893bce24a421866cf1dd9a3b272e8d7fef660e03a907d87966bccb07353a5
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 8c4893bce24a421866cf1dd9a3b272e8d7fef660e03a907d87966bccb07353a5
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (535 bytes)
2015-09-15 10:36:01 UTC received block 35a5ebbabc9ad1441248a31a23c64ec6a01b91eba4601f1080d64b5dbc9748bf
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 35a5ebbabc9ad1441248a31a23c64ec6a01b91eba4601f1080d64b5dbc9748bf
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (534 bytes)
2015-09-15 10:36:01 UTC received block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: liquidity (137 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (537 bytes)
2015-09-15 10:36:01 UTC received block 67209539cbc613659ae6d22b5189aec1962ee26d1f43d6011e0dafdbd2619d9a
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block 67209539cbc613659ae6d22b5189aec1962ee26d1f43d6011e0dafdbd2619d9a
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:01 UTC 2015-09-15 10:36:01 UTC received: block (536 bytes)
2015-09-15 10:36:01 UTC received block df33dacd7a975b30aee5996a4b66b28dbc1f35f3e5a8221f9b152034f901de7c
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed
2015-09-15 10:36:01 UTC WARNING: ProcessBlock(): check proof-of-stake failed for block df33dacd7a975b30aee5996a4b66b28dbc1f35f3e5a8221f9b152034f901de7c
2015-09-15 10:36:02 UTC 2015-09-15 10:36:02 UTC sending: getblocks (965 bytes)
2015-09-15 10:36:02 UTC 2015-09-15 10:36:02 UTC received: block (595 bytes)
2015-09-15 10:36:02 UTC received block b344065bbbd5fbecbfcdae163db622673b9ff2b1dd94816c2ad12405c9b1a21d
2015-09-15 10:36:02 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed

Some errors and failures appear periodically in the debug.log file. I started to download everything from scratch again.

[Hyena]

Yes I compiled it myself. Could it be that the external USB drive I use being an NTFS drive could cause any problems?

/dev/sda1: LABEL="TOURO" UUID="EC1EA3241EA2E734" TYPE="ntfs" PARTUUID="f7340f82-01"

When I bought it I forgot to format it with ext4 so turns out I’ve been using it as ntfs ever since not sure how important it is to change it now.

[mhps]

By the way, are the following log lines normal?

Things in this log all happened in a second. It’s not receiving new blocks as a node normally does. You are showing a segment when the client is not yet sync’ed. You should see things like this

received block 4df9a64d998f484c25e6cf23885c399511f56c950766276cf3d1178b11f32267
SetBestChain: new best=4df9a64d998f484c25e6 height=527968 trust=765451674720
moneysupply(S)=877032065.7799 moneysupply(B)=4607984.5713
ProcessBlock: ACCEPTED
accepted liquidity info from B8BxkGL8kK6X9fBxbeEBHwaSTCLuvvHqAM
2015-09-15 11:03:40 UTC Flushing wallet.dat
Flushed wallet.dat 74ms

[Hyena]

Yes I mostly see this stuff in the log that you pointed out. However, sometimes I see those failures.

[sigmike]

2015-09-15 10:36:01 UTC received block 6259114e8c9fcf88ace02d8f9d22506a644cfb85b967ccc3af4be70d6e75a24d
2015-09-15 10:36:01 UTC ERROR: CheckProofOfStake() : INFO: read txPrev failed

That means you received a block but could not find the Proof of Stake input in your transaction database. It may happen, even repeatedly (because you received a chain of blocks but you still miss a large part of the chain) but it should solve itself later. If it doesn’t, it’s likely your database is missing important parts.

The other errors suggest your database is corrupted. You may have a problem with your storing device. I think NTFS should work, but it’s probably more risky than a well tested ext4. You may also want to check your system and kernel logs for errors (if the device or the filesystem failed it should have been reported there when it happened).

[Hyena]

Ok thanks for the info. Let’s see how the downloading goes this time. I’ve been downloading the block chain from scratch for the past 36 hours and it hasn’t got stuck yet. Isn’t it possible to save the block chain in a way that it would not become corrupted so easily? For example, bad blocks could be downloaded again and replaced. The system should be more elastic so that it would overcome such problems. The bigger the block chain the more likely it will become that it becomes corrupted. I suspect that power failure may have caused it. Perhaps nud got killed in the middle of writing something and later it was unable understand that last block or whatever.