Without a server to connect to, the client is useless
edit: is it only my bot that is affected or is the server on liquidbitsā ccedk btc and usd pairs not crediting orders?
Without a server to connect to, the client is useless
edit: is it only my bot that is affected or is the server on liquidbitsā ccedk btc and usd pairs not crediting orders?
not properly accepting orders. ask side liquidity not fully accepted and credited althout the total ask side is far below target.
Not sure. there was a payout about 8h ago but less than usual. My own testbots on all 3 pairs seems to work fine and at least two others also appear to have a working bot. I didnāt change anything, so maybe your latest bot version has a bug? Not sure, canāt find anything else.
Edit: both USD and BTC ask side are on or above target, so that might explain what you are seeing.
It is not the VPS, the server is running including the Nu daemon. The problem is with the server.py which becomes non-responsive. It provides me with this error:
2015/10/12-20:26:33 ERROR: exception caught in main loop: dictionary changed size during iteration
And I have no clue what I can do about this error. Will continue to trial and error and tweak the config files to a situation before last week. Will keep my fingers crossed. My apologies but there is no support available to investigate the root cause.
Will restart in a few minutesā¦
there is no way of an automatic restart after error-crash?
Not for the server. The manual payout and logs are too complicated to automate a restart.
It is even more complicated as the server.py process is still working in a way. It will require an external trigger like feedback from the client.py from a different pc to determine that it is down. There might be a smart way to built something in the software itself when it triggers those exception errors it automatically restarts. Will need some help to code for that. Edit: it would still require manual payouts every time that happens.
Letās see how we go in the next 48h. Iāve reverted everything back to weeks ago and excluded the cryptsy and bter wrappers being loaded in the first place. They have been there for 8 weeks or so just for testing purposes, so unlikely to resolve this, but it is worth a try. At least it helps narrowing down the problem.
Iām considering some generic compensation for those who have been active in the last 2 weeks on LiquidBits. This instead of spending a lot of time on calculating all the manual payouts. Will focus on trying to stabilise the situation first though.
Same error again, deleted more config and restarted. Not much left now. If it goes down again, Iām afraid Iām out of options. I can only restart again until someone is available to help (@woolly_sammoth).
Need some sleep now, will check later againā¦
The server is down again. It really appears we have a new problem and not a fluke. I donāt have the knowledge to fix this in the server software. For now my next steps are until recommended otherwise:
Step 1:
When problem continue in the next few days:
Step 2:
When the problem stil continues:
Step 3:
In the mean time I have to advise people to assess their exposure/risks on the exchanges based on the current instability and my inability to restart the server several times a day, let alone do manual payouts. Iām basically already defaulting on the last grant and I have to think about how to compensate any liquidity provision during this period lacking the proof of liquidity provisioning during the outages.
these problems started when you test extra configuration for southX?
lets hope that step1 or step 2 will solve the issue
True, but that code hasnāt only been made obsolete in the config.py, but also physically removed from the exchanges.py. It might just have been a coincidenceā¦
Edit 7.50 UTC: server updated and restarted as per step 1, submitting liquidity
Are we sure that the problem is ccedk-specific? Is the server-side software the same as those running on other pools?
take it easy.
Havenāt seen reports from the other pools, so it seems to be server specific. We havenāt been trying running CCEDK exchange on other pools as far as Iām aware. Would be a bit of trial and error anyway. Rather have some debugging to find the root cause of this. It might be the custom made bots which make the server software not behave or some changes on CCEDK we are not aware off. Anyway it shouldnāt crash the software as it does.
Not being a coder, Iām staying away from changing code myself. Just updating the config files and following the updates on the nu-pool github. If there is some code difference we will learn that in step 2, but unless I made a stupid mistake syncing code in Github, it is unlikely to be the case. At the same time Woolly-sammoth will help me with some code to debug to identify what is causing this.
Right I think the best thing to do is outputing a lot of debugging messages to get a clue exactly where the server crashes, then log the context of that part in the next run.
Agree. Iāve already provided some detailed logging, but it still doesnāt give away what is happening. The event is also very random in time and there are two types of exceptions causing the crash. The cause might be the same though. If you are interested in more detail I just shared the latest log here: https://gitter.im/inuitwallet/nu-pool. Any thoughts are welcome.
That error seems to say to me that your config dictionary changed during iteration. Iād put a print statement for maybe the size or type of config before line 800 in server.py. Do you know what steps to take to reproduce the errors, or is it just run it and it errors eventually? Is it at 24 hour intervals or anything?
It is very random, not reproducible at this stage. Iāve done a lot of log analyses to find a trend, but with only 5 examples and seemingly random occurrences it is hard to identify trends.
How would you feel about a proposal for running NuBot at CCEDK to provide liquidity at the NBT/BTC pair?
Would you consider that supportive (for Nu) or a slap in the face (for your ALP operation)?
I could imagine running NuBot at the CCEDK NBT/BTC pair with the funds I used for the ALP bot so far.
Although Iām a proponent of supporting the NBT/USD pair, Iād rather not support that pair in my operation-to-be, because I might end up with USD and could not balance the liquidity or withdraw USD.
I can try setting up an ALP for the NBT/BTC pair on ccedk and see if it errors.