Raspberry Pi spending all its time flushing wallets?

[sigmike]

Is this fix to boost Yacoin minting performance applicable to Nu?

We have already implemented a similar improvement in Nu. The stake modifier calculation is indeed slow and the problem was much more severe in Nu because the outputs are split by default. So we added a stake modifier cache: https://bitbucket.org/JordanLeePeershares/nubit/commits/1915c897e047814ed371655bfaf53f5e25ccca47

Minting is still a computationally intensive job when there are many outputs. There is certainly room for more improvements, but a Raspberry Pi may not be enough when there is a lot of outputs to handle. A solution could be to split the shares on multiple nodes.

[mhps]

the outputs are split by default

If I send shares that have generated a lot of transactions over the last 10 months to a new address in an empty wallet, will minting on the new wallet be computationally less intensive?

Another solution for now is getting a Raspberry Pi 2, which has a fast 4 core CPU and 1G RAM – for the same price and less power.

[sigmike]

If I send shares that have generated a lot of transactions over the last 10 months to a new address in an empty wallet, will minting on the new wallet be computationally less intensive?

If you use the same split amount then the difference will be negligible because you’ll have the same amount of unspent outputs. The only difference will be that the new outputs will all have the same stake modifier so at start up you’ll have only one cache entry to calculate. But once the cache is filled for all the outputs there should be no difference.

[mhps]

Will the wallet be easier to flush because there are less tx history associated with the new address?
The performance of the Pi seems to be degrading so I am suspecting the tx history stored in the wallet makes updating/flushing it increasingly slow. According to top the CPU is only 40-60% busy.

[sigmike]

Will the wallet be easier to flush because there are less tx history associated with the new address?

I’m not sure. As I understand it, flushing should only take time if there was many changes made to the database. I suspect the long time reported in the log is actually spent waiting for some locks, and not doing actual calculation or I/O. You can try disabling minting to see whether it improves the flushing delay. If it does, then changing your wallet history will probably not change much.

[mhps]

From the links in my post above the flushing is unlikely just waiting. It severely degrade performance. The flushing strategy was made for small bitcoin wallets and block time is10 min.

Anyway I have moved my shares to two new addresses in a new wallet. The flushing time has been dramatically reduced from 50-80sec to 200-600ms.

But once the cache is filled for all the outputs there should be no difference.

I hope it takes several months as the outputs find new blocks.

[mhps]

There is certainly room for more improvements

I see a lot of activities receiving liquidity info from debug.log
Liquidity info seems useless for a headless minting node on Ras-pi. Do you think ignoring liquidity info would reduce CPU and memory useage much? I could tinker with the code. Where would you suggest to look?

[sigmike]

Liquidity info seems useless for a headless minting node on Ras-pi.

It may be for you, but not for the health of the network. If many people stop propagating liquidity info it will be less reliable.

Do you think ignoring liquidity info would reduce CPU and memory useage much?

CPU: probably a little as these messages are signed and verifying a signature is costly. A client could probably skip the signature verification though.
Memory: very little, unless we start having lots custodians with lots of long identifiers. A liquidity info is less than 1 kB.

I could tinker with the code. Where would you suggest to look?

It’s processed here: https://bitbucket.org/JordanLeePeershares/nubit/src/master/src/liquidityinfo.cpp#liquidityinfo.cpp-55

Returning true validates the propagation to the other nodes.

[mhps]

If many people stop propagating liquidity info it will be less reliable.

I see processing liquidity info is actually useful. If there will be hundrends LPCs reporting liquidity then every node will get quite busy.

[masterOfDisaster]

I just had a look at the debug.log, because another SD card “decided” to stop working properly. The card basically is broken, but I could retrieve the debug.log…

The flushing seems to be more often than just when new blocks arrive, right?

[...]
2015-09-09 13:23:00 UTC Flushing wallet.dat
Flushed wallet.dat 351ms
2015-09-09 13:23:25 UTC Flushing wallet.dat
Flushed wallet.dat 55ms
2015-09-09 13:24:08 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:24:28 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 13:24:49 UTC Flushing wallet.dat
Flushed wallet.dat 52ms
2015-09-09 13:25:14 UTC Flushing wallet.dat
Flushed wallet.dat 409ms
2015-09-09 13:26:58 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 13:27:20 UTC Flushing wallet.dat
Flushed wallet.dat 349ms
2015-09-09 13:29:01 UTC Flushing wallet.dat
Flushed wallet.dat 66ms
2015-09-09 13:29:45 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 13:30:13 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 13:30:34 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 13:31:27 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:31:56 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 13:32:56 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 13:33:12 UTC Flushing wallet.dat
Flushed wallet.dat 28ms
2015-09-09 13:33:37 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 13:34:38 UTC Flushing wallet.dat
Flushed wallet.dat 417ms
2015-09-09 13:35:06 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 13:35:18 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:36:47 UTC Flushing wallet.dat
Flushed wallet.dat 51ms
2015-09-09 13:38:20 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 13:40:03 UTC Flushing wallet.dat
Flushed wallet.dat 53ms
2015-09-09 13:40:09 UTC Flushing wallet.dat
Flushed wallet.dat 209ms
2015-09-09 13:40:27 UTC Flushing wallet.dat
Flushed wallet.dat 401ms
2015-09-09 13:40:49 UTC Flushing wallet.dat
Flushed wallet.dat 33ms
2015-09-09 13:41:30 UTC Flushing wallet.dat
Flushed wallet.dat 28ms
2015-09-09 13:42:03 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 13:42:45 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 13:43:26 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 13:44:39 UTC Flushing wallet.dat
Flushed wallet.dat 309ms
2015-09-09 13:45:32 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 13:47:53 UTC Flushing wallet.dat
Flushed wallet.dat 52ms
2015-09-09 13:48:54 UTC Flushing wallet.dat
Flushed wallet.dat 78ms
2015-09-09 13:49:52 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 13:52:49 UTC Flushing wallet.dat
Flushed wallet.dat 265ms
2015-09-09 13:54:23 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:55:18 UTC Flushing wallet.dat
Flushed wallet.dat 69ms
2015-09-09 13:56:22 UTC Flushing wallet.dat
Flushed wallet.dat 35ms
2015-09-09 13:56:36 UTC Flushing wallet.dat
Flushed wallet.dat 382ms
2015-09-09 13:57:11 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 13:57:22 UTC Flushing wallet.dat
Flushed wallet.dat 68ms
2015-09-09 13:59:38 UTC Flushing wallet.dat
Flushed wallet.dat 1458ms
2015-09-09 14:01:12 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:03:10 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:03:36 UTC Flushing wallet.dat
Flushed wallet.dat 67ms
2015-09-09 14:04:46 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:05:49 UTC Flushing wallet.dat
Flushed wallet.dat 65ms
2015-09-09 14:05:55 UTC Flushing wallet.dat
Flushed wallet.dat 261ms
2015-09-09 14:09:55 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:10:10 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 14:12:21 UTC Flushing wallet.dat
Flushed wallet.dat 30ms
2015-09-09 14:12:57 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:13:28 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:15:59 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:17:12 UTC Flushing wallet.dat
Flushed wallet.dat 68ms
2015-09-09 14:18:33 UTC Flushing wallet.dat
Flushed wallet.dat 469ms
2015-09-09 14:19:02 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:22:48 UTC Flushing wallet.dat
Flushed wallet.dat 40ms
2015-09-09 14:23:23 UTC Flushing wallet.dat
Flushed wallet.dat 149ms
2015-09-09 14:23:56 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:24:14 UTC Flushing wallet.dat
Flushed wallet.dat 903ms
2015-09-09 14:24:43 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:25:58 UTC Flushing wallet.dat
Flushed wallet.dat 32ms
2015-09-09 14:26:18 UTC Flushing wallet.dat
Flushed wallet.dat 183ms
2015-09-09 14:27:20 UTC Flushing wallet.dat
Flushed wallet.dat 52ms
2015-09-09 14:28:10 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:28:41 UTC Flushing wallet.dat
Flushed wallet.dat 44ms
2015-09-09 14:31:24 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 14:32:00 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:32:18 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:33:13 UTC Flushing wallet.dat
Flushed wallet.dat 49ms
2015-09-09 14:33:47 UTC Flushing wallet.dat
Flushed wallet.dat 70ms
2015-09-09 14:34:42 UTC Flushing wallet.dat
Flushed wallet.dat 388ms
2015-09-09 14:34:48 UTC Flushing wallet.dat
Flushed wallet.dat 44ms
2015-09-09 14:35:56 UTC Flushing wallet.dat
Flushed wallet.dat 55ms
2015-09-09 14:38:37 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:40:58 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 14:41:28 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:43:45 UTC Flushing wallet.dat
Flushed wallet.dat 52ms
2015-09-09 14:46:06 UTC Flushing wallet.dat
Flushed wallet.dat 57ms
2015-09-09 14:46:42 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:47:28 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:49:19 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 14:49:27 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:50:37 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 14:51:31 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:55:08 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 14:55:34 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 14:56:14 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:56:27 UTC Flushing wallet.dat
Flushed wallet.dat 44ms
2015-09-09 14:57:06 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 14:59:11 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 14:59:30 UTC Flushing wallet.dat
Flushed wallet.dat 47ms
2015-09-09 15:02:18 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
2015-09-09 15:04:12 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:04:17 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 15:05:06 UTC Flushing wallet.dat
Flushed wallet.dat 63ms
2015-09-09 15:07:31 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:09:15 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:09:26 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:12:20 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 15:14:04 UTC Flushing wallet.dat
Flushed wallet.dat 48ms
2015-09-09 15:15:24 UTC Flushing wallet.dat
Flushed wallet.dat 45ms
2015-09-09 15:16:07 UTC Flushing wallet.dat
Flushed wallet.dat 71ms
2015-09-09 15:17:11 UTC Flushing wallet.dat
Flushed wallet.dat 313ms
2015-09-09 15:17:36 UTC Flushing wallet.dat
Flushed wallet.dat 46ms
2015-09-09 15:18:10 UTC Flushing wallet.dat
Flushed wallet.dat 38ms
2015-09-09 15:18:27 UTC Flushing wallet.dat
Flushed wallet.dat 66ms
2015-09-09 15:19:22 UTC Flushing wallet.dat
Flushed wallet.dat 50ms
[...]

[mhps]

It’s a typical R-pi 2 flushing time example.

[masterOfDisaster]

Flushing the wallet (walletS.dat and walleB.dat?) means that there is write access to the SD card, right?
If it happens that frequently plus blocks get written on the SD card, etc. it’s no wonder that an SD card sooner or later gives up - I fear it has just not as good wear level mechanisms as e.g. SSDs do have.
Using the RPI with an external hard drive would not be my favourite solution…
Maybe I need to try NAS

[Sabreiib]

So ssd will be broken after several months minting?

[mhps]

I fear it has just not as good wear level mechanisms as e.g. SSDs do have.

I wonder if it is possible to put wallet.dat on a RAM drive and backuo it to the SD every hour.

[masterOfDisaster]

So ssd will be broken after several months minting?

The SD card was broken. Not by minting. Just by letting nud run 24x7 with empty wallets.

I wonder if it is possible to put wallet.dat on a RAM drive and backuo it to the SD every hour.

That would be great!
In terms of performance and hopefully in terms of SD card life time.
And even for those who mint it would be no big deal if the RPI would suffer data loss from a power outage because the minting transactions cause no new keys to be generated - they are part of the already existing keypool.

[mhps]

One can use a symbolic link (pointing to the wallet on the RAM drive) to substitute the wallet in ~/.nu . Backup using the RPC command.

But what is the typical time of access before a sector on a SD card fail? 100,000 times?

[masterOfDisaster]

what is the typical time of access before a sector on a SD card fail? 100,000 times?

Honestly, I don’t know. Maybe I have crappy SD cards.

What I know is that I have an RPI running for years (it’s a model with only 256 MB RAM) with the same SD card that mints Peercoins for … some time.
And I have an RPI2 (speedo model with 1 GB RAM) that just tries to keep the Nu blockchain up-to-date which has had a failed SD card more than once in this one year.

Either the (micro) SD cards (Samsung EVO) are of bad quality or nud kills them.

[mhps]

what is the typical time of access before a sector on a SD card fail?

It seems that many SD card use wear leveling and can be written 1 million times.

[masterOfDisaster]

I found different numbers, but still very big numbers in this thread: http://superuser.com/questions/17350/whats-the-life-expectancy-of-an-sd-card
What I found there, too:

Tips

• Do not defrag a memory card. This consumes write/erase cycles and shortens the MTBF.
• Use FAT32 instead of a journaling file system (like NTFS), which will write more often.
• SD cards are rated to hold data at something like 10 years sitting idle. I recall reading (not sure where) about re-energizing cards by occasionally inserting into a reader.

Having ext4 on /dev/mmcblk0p2 might make the situation worse for the SD card that it were without journaling.
…but my Peercoin minter has ext4 as well.
Maybe it’s just a set of crappy micro SD cards.
Next time I’ll try something like this: www.amazon.com/gp/product/B00D6XSRL2/

[mhps]

Next time I’ll try something like this: www.amazon.com/gp/product/B00D6XSRL2/

Extra feature might be lost when you write an image to the card according to the “Risks of reformatting” section here

Also see http://www.ideaheap.com/2013/07/stopping-sd-card-corruption-on-a-raspberry-pi/

Now Raspbian already has some measures implemented to reduce SD card wear https://www.raspberrypi.org/forums/viewtopic.php?f=78&t=62235