Nud freezes when booting up

[Hyena]

Ok I finished downloading the block chain and it seems that nud is attempting to mint now although I haven’t yet found any blocks. Here’s the screenshot of top now that the block chain has been downloaded.

edit:
any ideas if these errors are normal:

2015-09-17 10:45:41 UTC ThreadDumpAddress exiting
2015-09-17 10:47:49 UTC GetVoteFromDataFeed error: Data feed failed: SSL connect error
2015-09-17 10:47:49 UTC UpdateFromDataFeed failed: Data feed failed: SSL connect error
2015-09-17 10:47:50 UTC ThreadUpdateFromDataFeed exited

also I should point out that I am currently having a situation where nud and bcexchanged are both totally unresponsive. Sent kill -TERM <pid> to them and they just won’t die. This is important to get fixed since I suspect that killing the daemon might get the block chain corrupted.

[mhps]

That “72,6 wa” indicates that your CPU is spending lots of time waiting for IO. That IO is from swapping. Thrashing actually. The 6+ load shows that y our system is very stressed. I bet the debug.log shows wallet flushing times in several to several tens seconds, which normally should be tens of millisec…

I suggest get another pi for bced.

[masterOfDisaster]

Why not trying “ramdrive-nud”?
I mean, it takes some MB of the RAM, but as flushing the wallets might cause a lot of the IO, it could still be better with less RAM left, but flushing wallets at RAM speed (will only take msecs!).

Compare

 tail ~/.nu/debug.log -n 1000 | grep Flushed

with the wallets on disk and the wallets on ramdrive to find out whether it helps to move the wallets to RAM.

[Hyena]

I’m not going to get another pi because of that because I think a pi should handle multiple PoS daemons with its 1 GiB of RAM. So I expect some optimizations to be done. Before that happens I might just stop minting nushares, since they are cheaper. Alternatively I could make a script that would make sure one day nud gets to run and the other day bcexcanged gets to run. Ideally a good PoS coin should not give a penalty for that since I should accumulate coin days and get a higher chance of minting a block the day I run the daemon.

[Hyena]

pi@pi-desktop:~$ tail /media/TOURO/nu/17-09-2015-data/debug.log -n 1000 | grep Flushed
2015-09-17 10:26:47 UTC Flushed wallet.dat 397452ms
2015-09-17 10:54:10 UTC Flushed wallet.dat 886614ms

is this too much?

[masterOfDisaster]

Try to come to your own conclusion:

Raspberry Pi spending all its time flushing wallets?

Here’s the status of the first flushings after starting nud with the wallets on ram drive; faster than before and not written on SD card!

tail ~/.nu/debug.log  -n 500 | grep Flush
DBFlush(false)
DBFlush(true)
2015-09-11 16:35:40 UTC Flushing wallet.dat
Flushed wallet.dat 7ms
2015-09-11 16:36:57 UTC Flushing wallet.dat
Flushed wallet.dat 7ms
2015-09-11 16:37:42 UTC Flushing wallet.dat
Flushed wallet.dat 6ms

Remark: this wallet is empty, but it’s an order of magnitude less time than without wallets on ramdrive:

Raspberry Pi spending all its time flushing wallets?

I just had a look at the debug.log, because another SD card "decided" to stop working properly. The card basically is broken, but I could retrieve the debug.log… The flushing seems to be more often than just when new blocks arrive, right? […] 2015-09-09 13:23:00 UTC Flushing wallet.dat Flushed wallet.dat 351ms 2015-09-09 13:23:25 UTC Flushing wallet.dat Flushed wallet.dat 55ms 2015-09-09 13:24:08 UTC Flushing wallet.dat Flushed wallet.dat 48ms 2015-09-09 13:24:28 UTC Flushing wallet.dat…

[mhps]

I think a pi should handle multiple PoS daemons with its 1 GiB of RAM.

Maybe the next release of nud will do.

I should accumulate coin days and get a higher chance of minting

I think nu doesn’t use coinday when looking for kernels. it only uses coin amount. There is some security reason for it.

886614ms

OMG. Your pi is miserable.

[Hyena]

False alarm about having downloaded the block chain. Turned out that for some reason nud stopped downloading new blocks. It periodically downloaded voting information but no new blocks. I falsely assumed that it got finished. I restarted the system and it started to download again. The blockchain height is currently 483912 but according to blockexplorer.nu the height is 530732 so it will take a while.

[masterOfDisaster]

If ramdrive-nud helps getting the flushing of the wallets in a better condition, you might try ramdrive-bcexchanged as well

[Hyena]

Again it stopped downloading new blocks. This time I see the following failures being spammed in the debug.log file:

pi@pi-desktop:~$ tail -F /media/TOURO/nu/data/debug.log
2015-09-18 07:50:01 UTC connection timeout
2015-09-18 07:50:01 UTC ComputeNextStakeModifier: prev modifier=0x05f184d670d94edd time=2015-09-07 00:01:31 UTC epoch=1441584091
2015-09-18 07:50:01 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=516553 nTime=1441592413
2015-09-18 07:50:02 UTC trying connection 14.192.213.121:7890 lastseen=-559.4hrs
2015-09-18 07:50:02 UTC SetBestChain: new best=f7d533250bb1198dc2f0  height=516554  trust=752130327424  moneysupply(S)=876575780.4714 moneysupply(B)=4602036.0149
2015-09-18 07:50:02 UTC ProcessBlock: ACCEPTED
2015-09-18 07:50:02 UTC 2015-09-18 07:50:02 UTC received: block (633 bytes)
2015-09-18 07:50:02 UTC received block bdc96d88ef830f8a17f29798d18618366cc6f90d5ff21a233ebfa53f3b9c56a4
2015-09-18 07:50:02 UTC CheckStakeKernelHash() : using modifier 0x06e9d201b5234f61 at height=514192 timestamp=2015-09-05 12:01:50 UTC for block from height=505958 timestamp=2015-08-30 12:54:19 UTC
2015-09-18 07:50:02 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x06e9d201b5234f61 nTimeBlockFrom=1440939259 nTxPrevOffset=160 nTimeTxPrev=1440939259 nPrevout=1 nTimeTx=1441592456 hashProof=000000354c3b1e83df5da00988a10883938e0b0ea2bdc8494aa18404384d5dde
2015-09-18 07:50:02 UTC ComputeNextStakeModifier: prev modifier=0x05f184d670d94edd time=2015-09-07 00:01:31 UTC epoch=1441584091
2015-09-18 07:50:02 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=516554 nTime=1441592446
2015-09-18 07:50:03 UTC SetBestChain: new best=bdc96d88ef830f8a17f2  height=516555  trust=752131547484  moneysupply(S)=876575820.4714 moneysupply(B)=4602036.0149
2015-09-18 07:50:03 UTC ProcessBlock: ACCEPTED
2015-09-18 07:50:03 UTC 2015-09-18 07:50:03 UTC received: block (668 bytes)
2015-09-18 07:50:03 UTC received block d739f753ab2715ff6e34615e4f3a263233bf73dc251b4a50418466b5d32c5057
2015-09-18 07:50:04 UTC CheckStakeKernelHash() : using modifier 0x3070c383ddce04de at height=0 timestamp=1970-01-01 00:00:00 UTC for block from height=466553 timestamp=2015-08-03 02:20:04 UTC
2015-09-18 07:50:04 UTC CheckStakeKernelHash() : check protocol=0.3 modifier=0x3070c383ddce04de nTimeBlockFrom=1438568404 nTxPrevOffset=422 nTimeTxPrev=1438567899 nPrevout=322 nTimeTx=1441592472 hashProof=00000134bf0a07882f9ed9b4714612a3337577f2775e4dae6a5f0b1217b2aaa2
2015-09-18 07:50:04 UTC ComputeNextStakeModifier: prev modifier=0x05f184d670d94edd time=2015-09-07 00:01:31 UTC epoch=1441584091
2015-09-18 07:50:04 UTC ComputeNextStakeModifier: no new interval keep current modifier: pindexPrev nHeight=516555 nTime=1441592456
2015-09-18 07:50:07 UTC connection timeout
2015-09-18 07:50:07 UTC trying connection 77.128.146.141:7890 lastseen=-8842.8hrs
2015-09-18 07:50:12 UTC connection timeout
2015-09-18 07:50:13 UTC trying connection 2.38.211.181:7890 lastseen=-254.3hrs
2015-09-18 07:50:15 UTC Updating from data feed https://raw.githubusercontent.com/cryptog/nu_data_feed/master/cryptog_nu_data_feed.json
2015-09-18 07:50:18 UTC connection timeout
2015-09-18 07:50:18 UTC trying connection 86.135.208.10:7890 lastseen=-2826.6hrs
2015-09-18 07:50:23 UTC connection timeout
2015-09-18 07:50:24 UTC trying connection 46.109.129.53:7890 lastseen=-42.8hrs
2015-09-18 07:50:29 UTC connection timeout
2015-09-18 07:50:29 UTC trying connection 95.188.131.121:7890 lastseen=-3762.9hrs
2015-09-18 07:50:34 UTC connection timeout
2015-09-18 07:50:35 UTC trying connection 85.212.85.147:7890 lastseen=-320.6hrs
2015-09-18 07:50:35 UTC connect() failed after select(): No route to host
2015-09-18 07:50:35 UTC trying connection 210.174.40.53:7890 lastseen=-5165.4hrs
2015-09-18 07:50:40 UTC connect() failed after select(): Connection refused
2015-09-18 07:50:40 UTC trying connection 193.138.219.233:7890 lastseen=-18.8hrs
2015-09-18 07:50:40 UTC connect() failed after select(): Connection refused
2015-09-18 07:50:41 UTC trying connection 217.71.44.124:7890 lastseen=-587.7hrs
2015-09-18 07:50:46 UTC connection timeout

Any ideas why the lastseen parameter is negative?

[Hyena]

Finished downloading the block chain. Again, nud started to freeze at startup and bcexchanged also stopped minting blocks. I stopped the bcexchanged and this time only started nud. It started to work properly and I’ve been minting nushares for some 24 hours now. I think it’s now confirmed that the issue is RAM. Nud consuming more than 700 MiB cannot be ran in parallel with other block chains. Restarting it after every 4 hours did not help either. It froze at startup. I might make a script that would run nud and bcexchange in turns. So 50% of the time nud gets to run and 50% of time bcexchanged gets to run. I guess it is better than running only one of them.

[mhps]

I might make a script that would run nud and bcexchange in turns. So 50% of the time nud gets to run and 50% of time bcexchanged gets to run. I guess it is better than running only one of them.

If I understand it right every second the minting part of the client wakes up and loops through all utxos looking for kernels. If you let nud wakes up 0.5 seconds after the “round second” moment on the Pi’s clock, at which point bcexchanged starts its loop, then the two minters might take turns using the Pi’s resource within a second period, assuming they each till take less than 0.5 sec most of the time.
If my assumptions are right and you compile your code, the change in the source could be a trivial one-liner. @sigmike

[sigmike]

Yes the code could be changed to run for a maximum amount of time every second as I said earlier.

The code starts here: https://bitbucket.org/JordanLeePeershares/nubit/src/6fe88883f316b1ff3448d52438e9d122f808da51/src/wallet.cpp?at=master&fileviewer=file-view-default#wallet.cpp-1533
For each output it searches a kernel for each nSearchInterval seconds (which is usually 1, unless there are seconds to catch up). You could easily get the current time at the beginning (with GetTimeMillis()) and return error("something") in the loop whenever the time elapsed since the start is above a threshold (it would not be precise, but probably good enough).

You may also want to force nSearchInterval to always be 1, as having to catch up probably happens rarely in normal situation and is aggravating the situation when there’s not enough resources.

[mhps]

Thanks. I will first try measuring how long it takes to loop through when no kernel is found.