Raspberry Pi spending all its time flushing wallets?

I see processing liquidity info is actually useful. If there will be hundrends LPCs reporting liquidity then every node will get quite busy.

I just had a look at the debug.log, because another SD card “decided” to stop working properly. The card basically is broken, but I could retrieve the debug.log…

The flushing seems to be more often than just when new blocks arrive, right?

[...]
2015-09-09 13:23:00 UTC Flushing wallet.dat
Flushed wallet.dat 351ms
2015-09-09 13:23:25 UTC Flushing wallet.dat
Flushed wallet.dat 55ms
2015-09-09 13:24:08 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:24:28 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 13:24:49 UTC Flushing wallet.dat
Flushed wallet.dat 52ms
2015-09-09 13:25:14 UTC Flushing wallet.dat
Flushed wallet.dat 409ms
2015-09-09 13:26:58 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 13:27:20 UTC Flushing wallet.dat
Flushed wallet.dat 349ms
2015-09-09 13:29:01 UTC Flushing wallet.dat
Flushed wallet.dat 66ms
2015-09-09 13:29:45 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 13:30:13 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 13:30:34 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 13:31:27 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:31:56 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 13:32:56 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 13:33:12 UTC Flushing wallet.dat
Flushed wallet.dat 28ms
2015-09-09 13:33:37 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 13:34:38 UTC Flushing wallet.dat
Flushed wallet.dat 417ms
2015-09-09 13:35:06 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 13:35:18 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:36:47 UTC Flushing wallet.dat
Flushed wallet.dat 51ms
2015-09-09 13:38:20 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 13:40:03 UTC Flushing wallet.dat
Flushed wallet.dat 53ms
2015-09-09 13:40:09 UTC Flushing wallet.dat
Flushed wallet.dat 209ms
2015-09-09 13:40:27 UTC Flushing wallet.dat
Flushed wallet.dat 401ms
2015-09-09 13:40:49 UTC Flushing wallet.dat
Flushed wallet.dat 33ms
2015-09-09 13:41:30 UTC Flushing wallet.dat
Flushed wallet.dat 28ms
2015-09-09 13:42:03 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 13:42:45 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 13:43:26 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 13:44:39 UTC Flushing wallet.dat
Flushed wallet.dat 309ms
2015-09-09 13:45:32 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 13:47:53 UTC Flushing wallet.dat
Flushed wallet.dat 52ms
2015-09-09 13:48:54 UTC Flushing wallet.dat
Flushed wallet.dat 78ms
2015-09-09 13:49:52 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 13:52:49 UTC Flushing wallet.dat
Flushed wallet.dat 265ms
2015-09-09 13:54:23 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:55:18 UTC Flushing wallet.dat
Flushed wallet.dat 69ms
2015-09-09 13:56:22 UTC Flushing wallet.dat
Flushed wallet.dat 35ms
2015-09-09 13:56:36 UTC Flushing wallet.dat
Flushed wallet.dat 382ms
2015-09-09 13:57:11 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:57:22 UTC Flushing wallet.dat
Flushed wallet.dat 68ms
2015-09-09 13:59:38 UTC Flushing wallet.dat
Flushed wallet.dat 1458ms
2015-09-09 14:01:12 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:03:10 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:03:36 UTC Flushing wallet.dat
Flushed wallet.dat 67ms
2015-09-09 14:04:46 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:05:49 UTC Flushing wallet.dat
Flushed wallet.dat 65ms
2015-09-09 14:05:55 UTC Flushing wallet.dat
Flushed wallet.dat 261ms
2015-09-09 14:09:55 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:10:10 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 14:12:21 UTC Flushing wallet.dat
Flushed wallet.dat 30ms
2015-09-09 14:12:57 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:13:28 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:15:59 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:17:12 UTC Flushing wallet.dat
Flushed wallet.dat 68ms
2015-09-09 14:18:33 UTC Flushing wallet.dat
Flushed wallet.dat 469ms
2015-09-09 14:19:02 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:22:48 UTC Flushing wallet.dat
Flushed wallet.dat 40ms
2015-09-09 14:23:23 UTC Flushing wallet.dat
Flushed wallet.dat 149ms
2015-09-09 14:23:56 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:24:14 UTC Flushing wallet.dat
Flushed wallet.dat 903ms
2015-09-09 14:24:43 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:25:58 UTC Flushing wallet.dat
Flushed wallet.dat 32ms
2015-09-09 14:26:18 UTC Flushing wallet.dat
Flushed wallet.dat 183ms
2015-09-09 14:27:20 UTC Flushing wallet.dat
Flushed wallet.dat 52ms
2015-09-09 14:28:10 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:28:41 UTC Flushing wallet.dat
Flushed wallet.dat 44ms
2015-09-09 14:31:24 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 14:32:00 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:32:18 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:33:13 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:33:47 UTC Flushing wallet.dat
Flushed wallet.dat 70ms
2015-09-09 14:34:42 UTC Flushing wallet.dat
Flushed wallet.dat 388ms
2015-09-09 14:34:48 UTC Flushing wallet.dat
Flushed wallet.dat 44ms
2015-09-09 14:35:56 UTC Flushing wallet.dat
Flushed wallet.dat 55ms
2015-09-09 14:38:37 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:40:58 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 14:41:28 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:43:45 UTC Flushing wallet.dat
Flushed wallet.dat 52ms
2015-09-09 14:46:06 UTC Flushing wallet.dat
Flushed wallet.dat 57ms
2015-09-09 14:46:42 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:47:28 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:49:19 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 14:49:27 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:50:37 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:51:31 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:55:08 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 14:55:34 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:56:14 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:56:27 UTC Flushing wallet.dat
Flushed wallet.dat 44ms
2015-09-09 14:57:06 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:59:11 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 14:59:30 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 15:02:18 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 15:04:12 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:04:17 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 15:05:06 UTC Flushing wallet.dat
Flushed wallet.dat 63ms
2015-09-09 15:07:31 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:09:15 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:09:26 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:12:20 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 15:14:04 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:15:24 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 15:16:07 UTC Flushing wallet.dat
Flushed wallet.dat 71ms
2015-09-09 15:17:11 UTC Flushing wallet.dat
Flushed wallet.dat 313ms
2015-09-09 15:17:36 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 15:18:10 UTC Flushing wallet.dat
Flushed wallet.dat 38ms
2015-09-09 15:18:27 UTC Flushing wallet.dat
Flushed wallet.dat 66ms
2015-09-09 15:19:22 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
[...]

It’s a typical R-pi 2 flushing time example.

Flushing the wallet (walletS.dat and walleB.dat?) means that there is write access to the SD card, right?
If it happens that frequently plus blocks get written on the SD card, etc. it’s no wonder that an SD card sooner or later gives up - I fear it has just not as good wear level mechanisms as e.g. SSDs do have.
Using the RPI with an external hard drive would not be my favourite solution…
Maybe I need to try NAS :slight_smile:

So ssd will be broken after several months minting?

I wonder if it is possible to put wallet.dat on a RAM drive and backuo it to the SD every hour.

The SD card was broken. Not by minting. Just by letting nud run 24x7 with empty wallets.

That would be great!
In terms of performance and hopefully in terms of SD card life time.
And even for those who mint it would be no big deal if the RPI would suffer data loss from a power outage because the minting transactions cause no new keys to be generated - they are part of the already existing keypool.

One can use a symbolic link (pointing to the wallet on the RAM drive) to substitute the wallet in ~/.nu . Backup using the RPC command.

But what is the typical time of access before a sector on a SD card fail? 100,000 times?

Honestly, I don’t know. Maybe I have crappy SD cards.

What I know is that I have an RPI running for years (it’s a model with only 256 MB RAM) with the same SD card that mints Peercoins for … some time.
And I have an RPI2 (speedo model with 1 GB RAM) that just tries to keep the Nu blockchain up-to-date which has had a failed SD card more than once in this one year.

Either the (micro) SD cards (Samsung EVO) are of bad quality or nud kills them.

It seems that many SD card use wear leveling and can be written 1 million times.

I found different numbers, but still very big numbers in this thread: http://superuser.com/questions/17350/whats-the-life-expectancy-of-an-sd-card
What I found there, too:

Having ext4 on /dev/mmcblk0p2 might make the situation worse for the SD card that it were without journaling.
…but my Peercoin minter has ext4 as well.
Maybe it’s just a set of crappy micro SD cards.
Next time I’ll try something like this: www.amazon.com/gp/product/B00D6XSRL2/

Extra feature might be lost when you write an image to the card according to the “Risks of reformatting” section here

Also see http://www.ideaheap.com/2013/07/stopping-sd-card-corruption-on-a-raspberry-pi/

Now Raspbian already has some measures implemented to reduce SD card wear https://www.raspberrypi.org/forums/viewtopic.php?f=78&t=62235

Risks of reformatting applies to SD cards being used on a RPI - thank you for that hint. I was already expecting ext4 to have to do with the problems :wink:

This deals with overclocked RPIs and I read is as this is a special type of corruption only happening with (extremely) overclocked RPIs - I’m not sure whether it’s rather the file system that gets corrupted by overclocked RPIs or the SD cards.
I’m not overclocking my RPI at all.

Thank you very much for the links, especially this one. I was looking for information about how to protect the SD card in a RPI from being used too much and found similar tips.

Unfortunately the best protection seems to be turning off the journaling and I’m not sure I’m going to do that.

I think I try changing the location of the wallet files. Loading them from a tmpfs would be great - for the performance of the flushing process and for the SD card!

edit (edited again, because after restarting the RPI I found out that the folder /tmp/ram got deleted, the entry in /etc/fstab was not working and I was in fact using the manually mounted version of /tmp/ram; creating /ramdisk and entering that in fstab solved that):
I’ve created /tmp/ram /ramdisk, added

tmpfs           /ramdisk        tmpfs   defaults,noatime,nosuid,nodev,noexec,mode=1777,size=16M 0 0

to /etc/fstab, renamed the wallet files, copied them to /ramdisk, created symlinks for walletB.dat and walletS.dat and started nud.
It’s working!

edit2:
Here’s the status of the first flushings after starting nud with the wallets on ram drive; faster than before and not written on SD card!

tail ~/.nu/debug.log  -n 500 | grep Flush
DBFlush(false)
DBFlush(true)
2015-09-11 16:35:40 UTC Flushing wallet.dat
Flushed wallet.dat 7ms
2015-09-11 16:36:57 UTC Flushing wallet.dat
Flushed wallet.dat 7ms
2015-09-11 16:37:42 UTC Flushing wallet.dat
Flushed wallet.dat 6ms

Looks great! Are you going to auto backup the live wallet to the sd card? You could run the R-pi for many days and don’t have an update wallet on persistent storage.
Maybe put this after something llike

while true; do ./nud backupwallet `date +%F%T`; sleep 86400; done &

That’s a good idea. I was thinking the same. Another thought was - will I remember what steps to do to make nud run form the ramdisk? Would be nice to automate it a little bit.

I chose to create a tiny project (https://github.com/Lamz0rNewb/ramdrive-nud):
https://github.com/Lamz0rNewb/ramdrive-nud/blob/master/README.md

edit: terminating nud leads to this error message (every time):

 terminate called after throwing an instance of boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
  what():  boost::lock_error

Is nud terminating to fast being run with wallets on a ramdisk?

edit2: and the repo has now an experimental branch.
I decided to make the ugly scripts a little bit more generic. I admit that I thought of running bcexchanged from ramdisk as well :wink:

edit3:
from the last version of the experimental branch a new little brother was created - if ramdrive-nud helps getting the flushing of the wallets under control and you want to do the same with the bcexchange client:

A solution to the flushing problem is here