Liquidbits server down?

masterOfDisaster 2015-09-14 21:23:14 UTC #1

After 15:05 UTC liquidbits seems to have run into trouble.
My BTC ALP bot and my USD ALP bot report

2015/09/14-15:05:09 WARNING: too many rejected requests for btc on ccedk, adjusting nonce shift to 1
2015/09/14-15:05:09 WARNING: too many missing requests for btc on ccedk, increasing sampling to 146
2015/09/14-15:05:35 DEBUG: /liquidity: server response invalid
2015/09/14-15:05:35 ERROR: submit: server response invalid

2015/09/14-15:05:01 WARNING: too many missing requests for usd on ccedk, increasing sampling to 145
2015/09/14-15:05:35 DEBUG: /liquidity: server response invalid
2015/09/14-15:05:35 ERROR: submit: server response invalid
2015/09/14-15:05:36 DEBUG: /liquidity: server response invalid
2015/09/14-15:05:36 ERROR: submit: server response invalid

and haven’t been able to reach the server ever since.

edit: @Cybnate is there something wrong on my end or is it really the server?

[Resolved] Temporary NuBot outage on CCEDK

Nagalim 2015-09-14 21:32:10 UTC #2

Definitely down
http://nbt.liquidbits.net/status

zoro 2015-09-15 07:48:54 UTC #3

i wonder if Cybnate knows about it!
i also wonder why CCEDK has so bad API response!

Cybnate 2015-09-15 09:06:23 UTC #4

Sorry, stuck at work and only noticed now it is down. Looking into what is going on…

Cybnate 2015-09-15 09:15:18 UTC #5

Doesn’t look good. Significant increase in bandwidth consumption in last 12h. Unable to access to server.

Please cancel your orders manually especially on the NBT/BTC pair if you haven’t done already. Will need to log a call with my provider and possible have to restore a backup.

zoro 2015-09-15 09:16:42 UTC #6

the bot managed to cancel the orders when it lost the server!
excellent bot feature

nmei 2015-09-15 09:38:56 UTC #7

Agreed. One stuck, but I contacted CCEDK and they cancelled it.

Cybnate 2015-09-15 09:42:25 UTC #8

Unfortunately I have to wait until a response from my cloud provider as even the access to the droplet is not available (beyond OS). So this is most likely a provider delivery issue. This means I cannot proceed restoring until they have made the droplet available to me again. This may or may not take a while. Will update you as soon as I have a response and I’m able to restart or restore.

masterOfDisaster 2015-09-15 09:48:41 UTC #9

Yes, it’s a very sophisticated bot

That’s strange; the bot should have done that - unless it was terminated not gracefully.
Why did you contact CCEDK? You could just have logged into your account and cancelled the order…

I’ve started my bots again after I verified that all orders have been removed automatically. This way the bots will resume operation as soon as the ALP server is restored.
I wish we already had fixed payout - I’d become rich after the server will available and I were the only LP for some time

Thank you for the update.

Cybnate 2015-09-15 09:49:56 UTC #10

Just regained access. Verifying a few things…

masterOfDisaster 2015-09-15 09:57:15 UTC #11

ALP server resumed activity:

2015/09/15-09:47:23 INFO: starting PyBot for btc on ccedk
2015/09/15-09:47:23 INFO: successfully deleted all orders for btc on ccedk
2015/09/15-09:47:23 INFO: waiting 6.10 seconds to synchronize with other trading bots for btc on ccedk
2015/09/15-09:47:30 INFO: successfully placed bid btc order of

2015/09/15-09:47:26 INFO: starting PyBot for usd on ccedk
2015/09/15-09:47:26 INFO: successfully deleted all orders for usd on ccedk
2015/09/15-09:47:27 INFO: waiting 2.96 seconds to synchronize with other trading bots for usd on ccedk
2015/09/15-09:47:32 INFO: successfully placed bid usd order of

Cybnate 2015-09-15 10:07:17 UTC #12

Can confirm that services are up and running again. The syslog just ended about 12h ago, so it appears that droplet went down not just the OS or service.

I think it is safe for now to resume bot operations, but there is a risk that this may happen again until I understand what happened. Still waiting for a response from cloud provider.

Cybnate 2015-09-15 10:30:20 UTC #13

Provider just responded, they didn’t see any issues from their side which didn’t help.
Traffic, CPU and memory usage have been stable in last 30 minutes and I haven’t seen anything weird on a first scan, so it might just have been a glitch. No non-standard behaviour.

Will need to dive a bit deeper into the logs later and will keep an eye on the stats.
Will try to get some sleep for now…

seki 2015-09-15 11:58:33 UTC #14

what are the server specs? i mean cpu,mem,bandwidth
also it would be very careful for the VPS to alert you somehow in such cases
reject and missing requests continue as usual
Moreover i would like to be able to select how many of my available funds to be used by the bot in the specific pair

willy 2015-09-15 12:20:20 UTC #15

I recommend

http://monitority.com/

for monitoring. It’s free.

Just in case.

Nagalim 2015-09-15 12:45:10 UTC #16

Make a new post, talk up the benefits and what implementation would look like, and maybe someone will come along and see how easily the code can be modified to achieve that. The issues I’ve always had with such a topic are mostly conceptual, but this is not the thread to discuss it in.

Cybnate 2015-09-16 08:49:03 UTC #17

The server is running on a 2Gb, 2 CPU machine with SSD (upgraded from 1Gb after nud 2.00 proofed memory hungry).
With around 40% CPU utilisation and 45% memory utilisation on average things look healthy.

Re API I’ve run some shadow bots on other exchanges and I can achieve e.g. close to 100% efficiency on Bittrex. CCEDK changed something in the API or on their side which created issues, they fixed it to some extend (from 70% to 90% efficiency) but still remaining issues and we don’t have the expertise available to fix this (missing Creon!). I’m hoping that the NuBot integration solves this problem. So practically the LiquidBits ALP underdelivers up to 10% to the advertised rates.

Anyway, the server has been stable for almost 24h by now. Will to a manual pay run this weekend lacking time at the moment. This pay would only be covering 12h of the 24h yesterday as the server has been down for about 12h yesterday, so no logs are available. Can’t make that any better.

Was yesterday in workshops and writing papers and basically ignored all emails and phone calls till I came home. There is no affordable service which can fixed that But appreciate your recommendation. The other issue is that it is hard, but not impossible, to gain proper access through my 3g phone at work, all company computers are behind many firewalls so I wouldn’t be able to remote in easily. The LiquidBits ALP is a best effort service after all and doesn’t have failover. Regular backups are in place, but that doesn’t fix cloud provider delivery issues. However I will do what I can within my means and availability to keep it stable and running and I’m open to suggestions for improvements.

My apologies for the inconvenience.

Cybnate 2015-09-16 09:59:21 UTC #18

Ok, server.py just went down before payout while I was looking at it. It looks like that the latest changes may have a bug, analysing the logs.
This time I have the credit file over the last 24h, so will do manual payout later when I’m on top of this.

Edit: Looks like the nud didn’t initialise properly after restart causing the server.py to halt on payout just 15 minutes ago. Can now confirm that Nud is reporting liquidity properly after restart server.py according to the logs. Will keep a very close eye on logs over next 24h until payout is successful and logs don’t show issues. It looks like it was just a bad restart yesterday.

Message to self: don’t tell that software is stable, just leave it alone

Cybnate 2015-09-17 09:57:34 UTC #19

Good news, the server just did an automatic payout and ran for another 24h without problems. Will process the backpay over the weekend and will continue monitoring it closely till then.

The issue after the restart was related to Nu deamon not ready while the server.py already started. On short outages or restarts the Nu daemon catches up quickly. This time the blockchain was 12h behind and the script didn’t take that into account when starting the server.py. Still slightly puzzled by what really caused the initial problem. I’m afraid I have to keep it on a cloud glitch as I couldn’t find any meaningful syslog notifications other than that it just went offline until I restarted it.

Cybnate 2015-09-20 05:44:24 UTC #20

Apologies this took a while but the backpayments are underway. The first batch is only about 1/3 of what you expect, I couldn’t extract more out of the logs as proof of liquidity provisioning. Server.py had been down for more than 12h without any logging and the pay.py software didn’t like the .log file causing errors. Took me a little while to make that work.

The second backpay batch to follow soon is from the day after when Nud was not talking to server.py and therefore no payment was made (see above). From that event I have the full log of liquidity provision, so that is a normal payment for that 24h window. After the 17th the payments resumed the normal way. Please advise if you think you missed out.

Again my apologies for this.