Reliability analysis of bot paper vs bot real results
SETTINGS
Both bots run virtually simultaneously, identical entry and exit conditions
Bybit exchange
Working on an interval of 15 min
Running time +2D, over 20 tx
Multi mode
Observed 4 currencies
simultaneous transactions on 2
according to backtest provides >99% tx coverage
Entry conditions:
price increase by a certain % + RSI
Exit conditions:
Take profit: 5%
Stop Loss: when EMA 8 cuts downward through SMA 6
Move Stoploss 4%->2%
SAFEGUARDS
To ensure maximum bot compliance:
tx amount only $5 to eliminate own influence on the market
selected currencies with a minimum move of $1000 for +2/-2 price (currently Omnicat does not meet this condition)
15 min interval instead of 1 min to eliminate random price inconsistencies
no Trailing Stop
Move SL set at 4% to minimize the risk of movement inside the 1 min candle
setting 2/4 tx allows to open tx at the same time
exit determined by indicator, not % to averaging the exit conditions
RESULTS
Unsatisfactory. There are differences of 2 types:
bot paper opens many more tx than bot real
tx opened and closed at the same time achieve different results
Bot paper generally opens and closes tx 1-2 sec faster than bot real
STATISTICS
bot paper performed 27 tx
Bot real made 16 tx
difference: 11 tx
Out of 16 tx performed
5 tx differed by less than 0.5%
2 tx differed by 0.5%-1%
9 tx differed by more than 1%
FINAL EFFECT
bot paper: +3.8 daily
bot real: -2.6% daily
Currently bot real has a large unrealized gain on INSP, which will probably change the results to the positive. However, this in no way increases the reliability of the bot.
CONCLUSION
Despite my attempts to achieve reliable results, the results of the paper bot are only marginally related to the results of the real bot. BIG PROBLEM. This significantly undermines the usefulness of any testing.
I DON’T UNDERSTAND
Although the analysis of tx times indicates an average time of over 1h, the Stats specify both Win and Lose tx at 0 min
REQUEST
If you see errors in my settings please comment. Gainium is a very cool tool, but I’m slowly starting to doubt its reliability
You have chosen alphabetical pair selection. If deals behave similar in real and paper trading despite of real world effects that aren’t their in paper trading, I would also expect more similar results indeed.
I discovered something that may explain some of the missing transactions. A few log entries like:
Error: Not enough balance Order id: D-TP-qzjh8DLfEGG986jBCY3ydPVFiAbJeE, side: sell, price - 0.05243 , order type: deal close, balance total - 0 INSP, free - 0 INSP, required - 129.66 INSP
As many as 6 of the 11 missing trades involve INSP, so that may be the reason.
Where does this error come from and how to avoid it? It is the system that should allocate the right amount of funds
Same question for:
Warning: order value exceeded lower limit.
First of all, thanks for thoroughly analyzing this. It’s a useful opportunity to discuss the usefulness of paper trading.
The paper bot behaves like a real would. If it runs out of funds, it’s not going to automatically add more to complete the order. That would undermine the reliability of the comparison. So both paper and real exchange should have the same balances at the start.
This error is telling you that the bot could not open the trade because the size is lower than the exchange minimum. This can happen when you use % of USDT to open orders. The paper bot doesn’t have this issue. We cannot possibly keep up to date with the exchange minimums for every pair, which differ quite a lot. The paper bot will open even a 0.01 USDT base order if you set it.
And if you overcome those hurdles, there are still more. The paper exchange doesn’t have an order book, it’s going to execute the trade at whatever the price was. A real exchange will place your order in the order book and you can get filled at a different price than paper. In big cap tokens the difference is minimal, but on smaller cap like you have tested it can be considerable.
Then you have partially filled orders, which again has to do with the orderbook. Also slippage, the paper bot doesn’t have any but real do, again the orderbook.
Paper bot is just another way to test a strategy, but it will never be exactly like a live bot. The best way to test strategies in order of reliability of the test is:
Backtest > Paper trading > Trial run (small capital) > Real run
