[Passed] NuPool 8 ALPv2

[woolly_sammoth]

I was basing the extra liquidity on the output of https://eu.nupool.net/status
Are you submitting liquidity to NuPool from PPyBot? I didn;t think that would work but interesting if it is

[masterOfDisaster]

I didn;t think that would work but interesting if it is

Interesting is not the adjective I’d be using in this case
And it’s one of the reasons why Nu funded gateway should always operate outside the spread for which ALP are receiving compensation.

edit:
I was trying to run NuBot on bittrex to see whether I can help in any way with log messages.
This one is unrelated to the issues, I guess, but it’s not good, either - right?

08:37:52.413 [main] WARN  - Streamer appear to be offline @ stream.tradingbot.nu [c.n.n.s.Subscriber:611]
08:37:52.432 [main] ERROR - Streamer is offline, using local price monitor instead. [c.n.n.s.S.NuBotSecondary:148]

[woolly_sammoth]

it’s not a great error but is not so bad. Both Nubot and the ALP software will fall back to using a secondary price feed.
@desrever, could you look at restarting the Price Feed server please?

I don’t really think that @cybnate is submitting liquidity to NuPool. The endpoint signatures have changed so I don;t think PyBot would be able to. It’s only that sense of security that allowed me to use the term interesting

[masterOfDisaster]

Interesting!

Spock might have said fascinating!

[Cybnate]

I don’t really think that @cybnate is submitting liquidity to NuPool.

Apologies, I misunderstood. Thought you were referring to Bittrex liquidity in general, not just Nupool related.

[huafei]

price of ALP server still in broken , please have a check

@woolly_sammoth

[huafei]

OMG , the price feed server use broken again and again

[zoro]

by “broken” you mean that the price cannot be calculated at all thus no rewards are sent?
Or that we have to set a very low offset (like 0.002) in order to see rewards?

[woolly_sammoth]

It is that the service that fetches the price crashes so the price remains static instead of updating.
I’ve been tracking and fixing issues as they occur. This latest one is a little more in depth. I’m hitting an ‘open files’ hard limit. I’ve increased the limit but I’m not convinced the change will take effect without a reboot (which I would like to avoid if possible)
The error only seems to affect every now and again and restarting the pool software fixes it immediately. I will keep investigating and see if I can determine the situations in which the error occurs. I could also put a checker thread in place to restart the price_fetcher thread if it isn’t running.
Either way, I will be looking to get this fixed ASAP as havig the price incorrect ofr any amount of time, negates the purpose of the pool.

[Nagalim]

Pybot always did a check between client price and server price. If they didnt match, dont put up orders. Placing orders at the wrong price is pretty much game over for people trusting the software.

[woolly_sammoth]

I agree, that’s why I’m concentrating on fixing this.
NuBot is much more intelligent than Pybot and places orders completely independently of the price that the server gives. So the price orders are placed at won’t be wrong in dollars, they just may not get credited correctly by the pool.
The original intention was to use the price streamer service for both Bots and ALP. That has unfortunately been offline for the duration of this run so far.
Its also worth mentioning that this ‘too many open files’ error didn’t occur during testing. We do have more connections open now though.
I have pushed some new code to try and mitigate this and that is running currently in the alp server.
I will continue to monitor and provide fixes if this happens again.

[huafei]

you mean that the price cannot be calculated at all thus no rewards are sent

the price at status page always not change

[huafei]

“poloniex”: {“btc”: {“ask”: {“rank_1”: 24052.0875039, “rank_2”: 0.0, “total”: 24052.0875039}, “bid”: {“rank_1”: 0.0, “rank_2”: 22.40239479, “total”: 22.40239479}, “total”: 24074.489898689997}}}}, “server_time”: 1461292660, “server_up_time”: 63740, “status”: true}

“prices”: {“btc”: 444.1},

[huafei]

OMG…
what is price feed alp server used?

[Nagalim]

https://github.com/inuitwallet/ALP-Server/blob/master/src/price_fetcher.py#L278

    # Bitcoin
            if self.unit == 'btc':
                return self.fetch_price(['bitfinex', 'blockchain', 'bitcoin_average',
                                         'coinbase', 'bitstamp', 'yahoo', 'google_official'])

Now, I’m not so good at understanding NuBot. I think the feed order is chosen as the first entry of the list is the main feed and everything else gets cross checked, but I’m not sure.
https://bitbucket.org/JordanLeePeershares/nubottrading/src/a23c13bfe5c7ba681d459898256aea23eae81e28/src/main/java/com/nubits/nubot/pricefeeds/FeedFacade.java?at=master&fileviewer=file-view-default

[woolly_sammoth]

The issue isn’t the order in which the feeds are fetched, it’s that the price feed checker is still hitting the open files limit and crashing. Once that happens it doesn’t fetch the price at all.
I would like to reboot the server in order to raise these limits so that the issue doesn’t keep occuring.

The exchange price feeds were only intended to be backup feeds in case the price streamer failed. As that has been the case for the duration of this operation, It would make sense for the price checker to do a simple comparison as described by Nagalim. I will implement that later after I have rebooted the server.

The reboot will take place at or close to 20:00 UTC this evening (23rd April 2016). The software should restart and NuBots should reconnect. It would be worth checking your Nubots after the Reboot though just to check that they are connected.

[huafei]

the price always frozen , that will throw rank_1 to rank_2 after market change

[woolly_sammoth]

I’ve just checked the server and the price fetch routine is active. It updates the price every 60 seconds. I just watched it update the price.
Are you refreshing the https://eu.nupool.net/status page? Do you not see the price update at all?

Edit: just checked the status page that isn’t reflecting the price on the server correctly. I will investigate that when I look at the server this evening.

[huafei]

yes, alway see the price from status page

[woolly_sammoth]

Apologies for this. My availability for this evening has changed and I won’t be able to work then on the ALP software.
I have a bit of time now so will investigate the price not updating on the front end, work on averaging the price feeds as suggested by Nagalim and then reboot the server to raise the open files limits.
I will edit this post when I’m just about to reboot and will post again when the server is operational again.

Apologies for changing the timing like this.

Edit (14:13 UTC). I am about to stop the alp software and reboot the server. I will update here when it is back.
Edit2 (14:16 UTC) Server is back after an update. I will continue to monitor and tweak where necessary. I have started a new branch which features a price_feed that isn’t so reliant on the price feed server and will be able to handle errors in a saner way