STATS: reliability analysis of paper bot vs real bot results

Reliability analysis of bot paper vs bot real results

SETTINGS
Both bots run virtually simultaneously, identical entry and exit conditions
Bybit exchange
Working on an interval of 15 min
Running time +2D, over 20 tx

Multi mode
Observed 4 currencies
simultaneous transactions on 2
according to backtest provides >99% tx coverage

Entry conditions:

  • price increase by a certain % + RSI

Exit conditions:

  • Take profit: 5%
  • Stop Loss: when EMA 8 cuts downward through SMA 6
  • Move Stoploss 4%->2%

SAFEGUARDS
To ensure maximum bot compliance:

  • tx amount only $5 to eliminate own influence on the market
  • selected currencies with a minimum move of $1000 for +2/-2 price (currently Omnicat does not meet this condition)
  • 15 min interval instead of 1 min to eliminate random price inconsistencies
  • no Trailing Stop
  • Move SL set at 4% to minimize the risk of movement inside the 1 min candle
  • setting 2/4 tx allows to open tx at the same time
  • exit determined by indicator, not % to averaging the exit conditions

RESULTS
Unsatisfactory. There are differences of 2 types:

  • bot paper opens many more tx than bot real
  • tx opened and closed at the same time achieve different results
    Bot paper generally opens and closes tx 1-2 sec faster than bot real

STATISTICS

  • bot paper performed 27 tx
  • Bot real made 16 tx
  • difference: 11 tx

Out of 16 tx performed

  • 5 tx differed by less than 0.5%
  • 2 tx differed by 0.5%-1%
  • 9 tx differed by more than 1%

FINAL EFFECT

  • bot paper: +3.8 daily
  • bot real: -2.6% daily
    Currently bot real has a large unrealized gain on INSP, which will probably change the results to the positive. However, this in no way increases the reliability of the bot.

CONCLUSION
Despite my attempts to achieve reliable results, the results of the paper bot are only marginally related to the results of the real bot. BIG PROBLEM. This significantly undermines the usefulness of any testing.

I DON’T UNDERSTAND
Although the analysis of tx times indicates an average time of over 1h, the Stats specify both Win and Lose tx at 0 min

REQUEST
If you see errors in my settings please comment. Gainium is a very cool tool, but I’m slowly starting to doubt its reliability

DATA
Tx analysis sheet: https://i.ibb.co/5464rVt/TEST-07-11-2024-14-26-17.jpg
Bot paper: Gainium app
Bot real: Gainium app
Previous post about bot incompatibility: Test bot and real bot perform different transactions

You have chosen alphabetical pair selection. If deals behave similar in real and paper trading despite of real world effects that aren’t their in paper trading, I would also expect more similar results indeed.

Alphabetical applies to both Real and Paper, so it doesn’t explain the differences IMHO

I discovered something that may explain some of the missing transactions. A few log entries like:
Error: Not enough balance Order id: D-TP-qzjh8DLfEGG986jBCY3ydPVFiAbJeE, side: sell, price - 0.05243 , order type: deal close, balance total - 0 INSP, free - 0 INSP, required - 129.66 INSP

As many as 6 of the 11 missing trades involve INSP, so that may be the reason.
Where does this error come from and how to avoid it? It is the system that should allocate the right amount of funds

Same question for:
Warning: order value exceeded lower limit.

Lower limit possibly refers to minimum order size? Different funds could explain the differences. Also there may be partial executions.

1 Like

Better quality tx sheet

First of all, thanks for thoroughly analyzing this. It’s a useful opportunity to discuss the usefulness of paper trading.

The paper bot behaves like a real would. If it runs out of funds, it’s not going to automatically add more to complete the order. That would undermine the reliability of the comparison. So both paper and real exchange should have the same balances at the start.

This error is telling you that the bot could not open the trade because the size is lower than the exchange minimum. This can happen when you use % of USDT to open orders. The paper bot doesn’t have this issue. We cannot possibly keep up to date with the exchange minimums for every pair, which differ quite a lot. The paper bot will open even a 0.01 USDT base order if you set it.


And if you overcome those hurdles, there are still more. The paper exchange doesn’t have an order book, it’s going to execute the trade at whatever the price was. A real exchange will place your order in the order book and you can get filled at a different price than paper. In big cap tokens the difference is minimal, but on smaller cap like you have tested it can be considerable.

Then you have partially filled orders, which again has to do with the orderbook. Also slippage, the paper bot doesn’t have any but real do, again the orderbook.

Paper bot is just another way to test a strategy, but it will never be exactly like a live bot. The best way to test strategies in order of reliability of the test is:

Backtest > Paper trading > Trial run (small capital) > Real run

2 Likes

Two first steps done, two next are under testing :wink:

Thanks for clarification

Perhaps a good idea would be to create a list of some guidelines on how to use paper bots correctly.

For example:
1 Remember that tokens with a threshold of +2/-2% below $1k will give false results due to low liquidity

2 Take care that your orders are at most 1/1000th of the liquidity of a token on a given exchange

3 The longer the interval (1h vs 1 min), the more correct the results in general.

4 Trailing stoploss and Move stoplos below 2% give high risk of false readings. Use higher or none at all.

5 When backtesting, use slippage in the range of 0.2-0.5 to get reliable results
etc

I tried to put together my own safety list, but apparently it was not sufficient. One arranged by you would be a big help not only for me.