Nud freezes when booting up

[Hyena]

These are the last lines of my debug.log file. It seems that nud started yesterday and got stuck when initially booting up. I have Raspberry Pi 2 and it is running bcexchanged and nud in parallel. What could be the cause of such a freeze? Could it be that there is not enough RAM so nud just freezes infinitely? If that’s the case then perhaps it could be possible to have something written in the debug.log file so that the users would know for sure what the problem is. Right now I have no idea what really causes this problem. I can only guess that it’s caused by either upgrading to 2.0.0 or by adding bcexchanged on the same system.

2015-08-25 17:36:35 UTC Nu version v2.0.0-beta (2015-08-07 14:14:37 +0200)
2015-08-25 17:36:35 UTC Default data directory /home/pi/.nu
2015-08-25 17:36:35 UTC Loading addresses...
2015-08-25 17:36:35 UTC dbenv.open LogDir=/media/TOURO/nu/data/database ErrorFile=/media/TOURO/nu/data/db.log
2015-08-25 17:36:38 UTC Loaded 6897 addresses
2015-08-25 17:36:38 UTC  addresses              3337ms
2015-08-25 17:36:38 UTC Loading block index...
2015-08-25 17:36:38 UTC PPCoin Network: genesis=0x000003cc2da5a0a289ad nBitsLimit=0x1e0fffff nBitsInitial=0x1e00ffff nStakeMinAge=604800 nCoinbaseMaturity=100 nCoinstakeMaturity=5000 nModifierInterval=14400

[masterOfDisaster]

I still have bcexchanged and nud running in parallel on the same RPI2.
They’ve been running for days.
No problems so far.

[sigmike]

Nu has to load a lot of data at startup (just after the last line of the log you showed). We will try to improve that in the future. In the meantime I suggest you just wait for it to load the block index, or try to improve the file reading speed somehow.

[Hyena]

Yes it typically loads this stuff in 15 minutes but this time it had done it 24 hours and was still loading so it obviously got stuck somewhere. I am using an external USB drive and I use some bandwidth limiter, but those don’t seem a very probable cause of the freeze. The internet connection is also not 100% reliable since I use a mobile broadband connection and a script to automatically reconnect when the connection has died. Since there is no going back to nud 1.x branch I could only try to shut down bcexchanged or add a swapfile just in case. I was also thinking of running nud from a timeout process so that it would restart nud periodically to overcome any freezes that might randomly happen.

[sigmike]

Network should not matter at this stage.

Indeed 24h is too long. Being too low on memory could explain that. Can you monitor your memory usage during startup? With top for example.

Some I/O error could also explain the problem. Is there any error in dmesg?

[mhps]

Could it be that there is not enough RAM so nud just freezes infinitely?

Yes. I have experienced this, with the same debug.log, after i disabled swap. The system started thrashing when RAM use reached 90+% ( as shown by hop).
it was a R-pi B+ and old Nud, though.

[Hyena]

I will do it but I can get my own hands on the Pi no earlier than on the 13th of September. In the meantime I can only let my brother execute some critical commands for me when needed since he’s physically present at the Pi.

[Hyena]

Yes I have seen similar behaviour multiple times already. What OS did/do you run on the Pi? I’m using ubuntu-mate. Perhaps my OS consumes more RAM and that’s why I’m having these problems and you are not.

[mhps]

What OS did/do you run on the Pi

I am using raspbian.

[masterOfDisaster]

I have raspbian on my rpi as well

[Hyena]

Here’s the top 5 memory using processes on my Pi right after nud was restarted:

$ ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -5

38.9  0.5 370900   526 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
17.8  6.5 182076 14905 ../GIT/nubit/src/nud -logtimestamps -debug -server -datadir=/media/TOURO/nu/data/
10.4  7.6 241644  1572 ../GIT/bcexchange/src/bcexchanged -logtimestamps -debug -server -datadir=/media/TOURO/bcexchange/data/
%MEM %CPU    VSZ   PID CMD
 0.6  0.0  63176   454 /usr/bin/whoopsie -f

Any ideas what the hell is irqbalance and why does it consume so much resources? Is it normal? Do other Pi users have that same process with same resource consumption?

After 10 minutes when nud booted up, I have this:

39.0  0.5 371356   526 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
26.2  3.4 261408 14905 ../GIT/nubit/src/nud -logtimestamps -debug -server -datadir=/media/TOURO/nu/data/
 9.6  7.6 241644  1572 ../GIT/bcexchange/src/bcexchanged -logtimestamps -debug -server -datadir=/media/TOURO/bcexchange/data/
%MEM %CPU    VSZ   PID CMD
 0.6  0.0  63176   454 /usr/bin/whoopsie -f

and this:

$ cat /proc/swaps
Filename				Type		Size	Used	Priority
/media/TOURO/swapfile                   file		1048572	3492	-1

nud is currently running as it should be, writing stuff into debug.log file at regular basis. let’s see how long it lasts.

[Hyena]

I had a chance to debug the situation a bit. The nud was totally unresponsive and didn’t even react to kill -TERM <pid>. It kept 1 core totally busy (50% CPU). Later I checked the debug.log file and discovered that even though nud was minting and finding blocks all blocks it found were rejected by the network (orphan / stale). RPC calls to nud took forever or never finished. By the way, I had checkblocks=1 in the nu.conf file to make it boot up as fast as possible. I then discovered that my nud never got past some certain block height which was a couple of hundred blocks less than the real block count. I also saw a lot of protocol version errors in the debug.log file. It said something like “your block protol version of 0 is less than 4” or something, I don’t remember exactly. Yes, I was running nud 2.0.0.0 per getinfo.

I was getting pretty desperate so I deleted the whole block chain and started to download it from scratch this morning. This was the fourth time I had to download nushares block chain from scratch on my Pi to resolve the problem of not minting blocks. This part of nu is badly designed since in the perfect world scenario the block chain should never become corrupt. I cannot afford to download the whole chain so often because my internet resource is limited by maximum bandwidth per month (25 GiB).

I am now starting the nud and bcexchanged daemons with the nice command to make them less important than other processes:
$ nice ../GIT/nubit/src/nud

I just cannot tolerate the fact that nud consumes pretty much all the CPU of my Pi2. Also, I now automatically call mkswap and swapon on system startup to make sure I never run out of RAM. To automatically overcome situations where nud has frozen, I am now sending SIGTERM and SIGKILL to nud once in every 10 hours to force it restart.

Let’s see how it works out now. Ever since Nu 2.0.0 came out and I started to run bcexchanged on the same Pi2 problems started. Hopefully downloading the Nu’s block chain from scratch solves the problem for now. However, in future releases I hope to see some improvements in these sectors.

[mhps]

I have read that Raspbian is particularly efficient in memory use. If you are using another OS, maybe that is the problem.

[masterOfDisaster]

That would explain why I have no memory issues running nud and bcexchanged in parallel on my RPI2.
Minting with them wouldn’t change the memory usage, right?

Here’s how top on my RPI2 looks like

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
25561 pi        20   0  654m 527m  40m S   2,6 54,2   1267:27 nud
20943 pi        20   0  235m 120m  37m S   2,3 12,4 596:14.35 bcexchanged

[Hyena]

I have mostly only used Ubuntu based Linuxes (Mint personally and Ubuntu at work). That’s why I chose Ubuntu for Pi. Would Rasbian be very different from Ubuntu? I mean the essential commands, network-manager and the stuff a programmer typically uses?

[masterOfDisaster]

Raspbian is based on Debian just like Ubuntu is.
If you are familiar with Ubuntu, you will have no big trouble using Raspbian.

sudo apt-get update

[Hyena]

I should have done my homework before choosing the OS. But hey, I’m probably going to buy another Pi some time in the future. It’s going to be a PoS Mining Farm full of Raspberry Pis I would also like to try MintBox Mini some time.

[mhps]

Would Rasbian be very different from Ubuntu?

If you have a spare SD card (>4GB) you can try raspbian. Takes less than an hour after downloading t he image.

[Hyena]

That’s a good idea. The great thing about those images is that you can duplicate the preconfigured OS for many Pi devices easily. I believe you can also backup the system that way.

[mhps]

right. You could copy *.conf blk* addr* wallet* database/* to a USB stick, switch card with a different OS, and copy things back, so you don’t need to redownload the b-chain.
For experimenting on many OSes, maybe one could run everything from a USD stick.