[CRITICAL] Nu Network fork experienced at block 407,740

#1

A few hours ago the network experienced a fork when two competing chains on the Nu network failed to reach consensus. This has caused some of the active Nu clients to proceed down one chain while the rest went another direction. The cause of this fork was that the network could not recover a fork that happened at a block generating a custodian grant. Forked blocks in this case are not accepted by the rest of the network during the chain reorganization because the custodian is considered already elected (because it is in the current chain).

This is not a fatal condition but it is important that we all work together to minimize the danger of transactions being processed on the secondary chain.

The Nu Development team is actively working on a software patch that will allow the network to reconcile situations like this in the future. We do not have an ETA on when the new version of the wallet client will be released but we’re hoping it is within a day or two once the fix has been written and changes have been sufficiently tested.

What Do I Need To Do?

First, you’ll need to check to see if you are on the “main” blockchain or the “secondary” one. The easiest way to do this is to check to see if you wallet client or Nu daemon process recognizes the hash of the block that caused the fork. You will need to use the debug window (located under the Help option in the application’s file menu) in your wallet to verify the current block height. Once you’ve opened the dialog, select the Console tab and in the text input type in the following text and then hit ENTER:

getblockhash 407740

If you see this response you ARE on the main blockchain and do not need to do anything else.

bc23d7c56bec2c4b41ca67ff6cf2fbc6bd7fd995686e7f74e5cc675e225abfa0

If you see anything else you ARE NOT on the main blockchain and will need to take additional steps.

A test on a client that was tracking the secondary chain returned db861b0be97a5d64104ac2b991341bd9d05841c1e7ffbc26f7322a851a54beef instead of the bc23d7c... string that the main chain responds with.

Additional steps for clients on the SECONDARY blockchain

The most important thing you can do to support the network is to take your client offline. Continuing to propagate the secondary blockchain is not in the best interests of the network because it could mean that people who are synchronizing their clients with the network could be inadvertently directed towards the secondary blockchain.

Once you’ve taken your client offline you can do one of two things to get back on the main blockchain:

  • Wait for the updated client to be released with the software fix. Once the fix is in place and you’ve installed the new client the network will correctly handle the reorganization and your client should automatically make its way back to the main chain without manual intervention. At the time that this topic was written we do not have an ETA for when the new client will be available other than to say “as soon as possible”.

  • Fully re-synchronize the blockchain. This is a manual (and more invasive) fix for the issue, but perfectly safe if you follow the included instructions. The challenge right now is dealing with the potential for re-synching the chain only to find yourself back on the secondary chain because others who had are also on the wrong chain continue to broadcast. This is a time consuming process, unfortunately.

Steps to re-synchronize your blockchain:

  1. From your client’s File menu option select Backup wallet. Select a location that you’ll easily be able to find your wallet backup and save it there. This ensures you won’t lose anything important (private keys for addresses with your NBT and NSR).

###IMPORTANT!
You will need to do this for BOTH your NSR and NBT wallet. Make sure you switch between wallet views and run the operation twice. If you are using the Nu daemon and have questions about backing up your wallets, please let @Ben, @CoinGame, @sigmike, @pennybreaker, @erasmospunk, or @JordanLee know and we’ll help you get it sorted out based on how you’ve set up your daemons.

  1. Once you’ve backed up your wallets, shut down your Nu client completely.

  2. You will need to delete the block index files (blkindex.dat, blk000*.dat) in your Nu application directory. There will only be one blkindex.dat file but could be multiple blk0001.dat, blk0002.dat, etc.; you’ll need to remove all of them. If you need assistance locating your application directory, please review the Nu Documentation site for instructions for your computer’s operating system.

  3. After you’ve deleted the files you can restart your client to begin to re-synchronize the blockchain. This is not a fast process and depending on the unique conditions of your internet connection speeds, the number of network nodes you connect to, and the speed of your computer’s processor, it may take over 12 hours. You are not doing anything wrong if it takes a long time, it is just the way that it currently works. You’ll know the synchronization has been successful if you get to the end of the process (green checkmark icon in the lower right corner of the client) and are able to see the correct response from the getblockhash 407740 console command.

Also, if you could please help us spread the message to other users (especially exchanges or commercial services that use Nu) we’d really appreciate it. Making sure that they are on the correct blockchain is paramount.

Please don’t hesitate to ask if you have questions about the process used to correct this issue or about the issue itself. Thank you for your patience while we work to resolve this issue.

#2
#3

Wow - thanks for your prompt response

#4

What happens if you make transactions (send, receive NBT/NSR) while being on the wrong chain?

#5

This is very important

#6

The transactions will be recorded in both chains.

1405 blocks have been found in the last 24 hours on the main chain (1440 is normal and average) which suggests few if any are on the smaller chain at this point.

sigmike has a fix coded that is currently being tested. We expect it will be released very soon.

4 Likes
#7

Thanks for the explanation and action. Just some data points – my wallet had all found blocks orphaned starting after about 9:30 GMT. At about 15 GMT I restarted it. Looking at the debug.log i could see I was on the right chain all the time.

I think in situation like this the network could help nodes reciver quickly by publishing a few nodes that is known to be on the right branch, and letting those who are on the wrong one addnode these correct ones. (edit: I assumed this would help the wrong nodes recover without downloading the whole chain. But now I am not sure) Downloading the whole chain takes a long time. If many are downloading network security is lowered because less nodes are minting for a long time.

edit: I see this in the log after restarting

received block db861b0be97a5d64104ac2b991341bd9d05841c1e7ffbc26f7322a851a54beef
ERROR: AcceptBlock() : unexpected number of expansion transaction
ERROR: ProcessBlock() : AcceptBlock FAILED
disconnecting node 198.52.199.75:7890
disconnecting node 198.52.199.75:7890
Disconnected 198.52.199.75:7890 for misbehavior (score=100)

Will this disconnecting reaction isolate 198.52.199.75 from the correct nodes such that it will have a hard time to recovering to / downloading the correct branch?

Will this disconnecting reaction aggravate network split because the nodes in the wrong branch are also actively disconnecting from those in the right one?

1 Like
#8

What if we change blocktime to 2 or 5 minutes?

#9

If this is so important then why hasn’t it been posted on any of Nu’s social media? There is currently nothing on Twitter, Facebook or Reddit. Are you just waiting for the fix first?

#10

That’s why we need message service on Nu. Where is emeth?

#11

I’ve been monitoring the situation all day and watching @sigmike, @Ben, @Coingame, @Pennybreaker and @JordanLee discuss it in an internal chat. It appears the problem has been addressed and a fix is coming very soon. As Jordan pointed out, very few if any individuals are on a separate chain, indicating that @Ben’s original post here was sufficient to notify shareholders. If anyone on the development team had requested a full social media blast I would have done so of course.

1 Like
#12

I am on the secondary chain :stuck_out_tongue:

#13

me too. It seems we are all in asia.

#14

I am worried about exchanges - If I have to withdraw NBT from exchanges like poloniex, is there any issue right at the moment?

#15

Nice to hear.

But I would have a few questions to clarify my mind:

  • what caused the fork in the first place?
  • is there any mechanism in the protocol that would get the clients which are on the wrong chain to be back on the right chain (like get the longest chain or the chain with the highest minting power, akin to the longest chain in bitcoin) ?
  • if both chain records the same transactions as explained by JL above, what is the difference between both?

One might remember the bitcoin blockchain fork back in mar 2013 in comparison: https://bitcoinmagazine.com/3668/bitcoin-network-shaken-by-blockchain-fork/,
which was caused by a database system version mismatch.

In any case, such a fork showing up and being contained rapidly and efficiently without any lost funds would only strengthen the network.

I made 2 tweets about the fork in en and jp btw, just in case.

1 Like
#16

Since both chains use the same protocol shouldn’t they all have 1440 blocks per day on average in their own universe?

#17

New transactions are mirrored on both block chains, you shouldn’t have an issue.

PoS uses most destroyed share days to determine longest chain. I’m not going to try to explain what happened here, though, cause I don’t know.

The difference is block 407740, which was the block in which my nupond proposal passed. At the very least, the txid’s are different. Most likely, a few shareholders minted also some blocks only on the alternate blockchain and not also on the main one, so there would be other differences. I’m kinda talking out my ass though.

#18

So at the time of the fork, both chains had the same destroyed share days?

The one with 1NBT requested?

#19

So, there’s no fork? what’s the fork?

#20

it happened on my 781 nbt proposal. I don’t know why, but that triggered it somehow. The devs know why and are in the process of releasing a fix. I’m guessing some minters ended up on the wrong chain and things went weird, but then the nodes distributed both chains and now we’re all in a ‘nothing at stake’ situation where we’re minting on both chains. But once everyone is on the main chain, the wrong one basically vanishes because no one is recording it.

2 Likes