Rest calls intermittent forbidden responses

[svb]

Hi,

C# : use combination of REST & streaming. Replicated using WPF example that you can download in labs.

This issue is specific to REST

Every 15 minutes I get 15 min prices for a range of epics. My allowance is fine with +6500 left. 

I keep getting random null responses with "forbidden" status code. Different epics. If I trap null response and then set rest call to execute again, it works. I continue and it carried on with rest of epics, or will do say 10 , return forbidden for 3 and then return fine for epics following that. if I step through each epic, its fine. So seems to be some timing glitch. Tried trapping and adding task.delays and retries, but still null. I have to actually break and step over. Given its not my allowance, are there any quirks with REST calls to cause this? This is example, does some, throws forbidden and then fine for AUDCAD again.

persist 15: 17/03/2021 22:15:01 - CS.D.CADJPY.TODAY.IP
persist 15: 17/03/2021 22:15:01 - CS.D.CADCHF.TODAY.IP
persist 15: 17/03/2021 22:15:01 - CS.D.EURAUD.TODAY.IP
response is null for CS.D.NZDJPY.TODAY.IP 15
response is null for CS.D.NZDUSD.TODAY.IP 15
response is null for CS.D.CHFJPY.TODAY.IP 15
response is null for CS.D.USCGC.TODAY.IP 15
response is null for CC.D.CL.USS.IP 15
persist 15: 17/03/2021 22:15:01 - CS.D.AUDCAD.TODAY.IP

I cannot replicate forbidden status response in REST companion for any epic either, just seems when run via rest client

Thanks

Edited March 17, 2021 by svb
add note about running in companion

[Chrislongden]

Having the same problem here too

[Chrislongden]

How do you see your limits/usage? I can't see anything in the API Companion to do this?

[svb]

just do a price request and you'll see allowance in response. 

 

"metadata": {
        "allowance": {
            "remainingAllowance": 4602,
            "totalAllowance": 10000,
            "allowanceExpiry": 442401
        },

[svb]

improved it by making sure rest calls aren't async and also putting 2 second wait before each call (1 second is not enough and I get more forbidden responses). For what I'm doing this is ok-ish, but not great. Given this is the rest client returning this and not my code I'm not sure what's changed i.e account, service validation. If anyone has any ideas that would be much appreciated,

the wait I added,

 

    foreach (var market in MarketList)
                        {
                            try
                            {
                                Thread.Sleep(2000);

                                response = await igRestApiClient.priceSearchByNumV2(market, "HOUR", "2");

                                if (response != null)

and the results. only 2 had issues this time, even with 2 second wait. with 1 second I get a lot more.

time: 18/03/2021 11:58:01
time: 18/03/2021 11:59:01
time: 18/03/2021 12:00:01
persist 15: 18/03/2021 12:00:01 - CS.D.AUDUSD.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.EURCHF.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.EURGBP.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.EURJPY.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.EURUSD.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.GBPUSD.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.USDCAD.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.USDCHF.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.USDJPY.TODAY.IP
persist 15: 18/03/2021 12:00:01 - IX.D.DOW.DAILY.IP
persist 15: 18/03/2021 12:00:01 - IX.D.NASDAQ.CASH.IP
persist 15: 18/03/2021 12:00:01 - IX.D.SPTRD.DAILY.IP
persist 15: 18/03/2021 12:00:01 - IX.D.DAX.DAILY.IP
persist 15: 18/03/2021 12:00:01 - IX.D.FTSE.DAILY.IP
persist 15: 18/03/2021 12:00:01 - CC.D.DX.USS.IP
persist 15: 18/03/2021 12:00:01 - CS.D.USCSI.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.AUDJPY.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.AUDCHF.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.GBPAUD.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.GBPCAD.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.EURCAD.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.USDMXN.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.GBPJPY.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.CADJPY.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.CADCHF.TODAY.IP
response is null for CS.D.EURAUD.TODAY.IP 15
persist 15: 18/03/2021 12:00:01 - CS.D.NZDJPY.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.NZDUSD.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.CHFJPY.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CS.D.USCGC.TODAY.IP
persist 15: 18/03/2021 12:00:01 - CC.D.CL.USS.IP
persist 15: 18/03/2021 12:00:01 - CS.D.AUDCAD.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.AUDUSD.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.EURCHF.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.EURGBP.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.EURJPY.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.EURUSD.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.GBPUSD.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.USDCAD.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.USDCHF.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.USDJPY.TODAY.IP
persist 60: 18/03/2021 12:00:01 - IX.D.DOW.DAILY.IP
persist 60: 18/03/2021 12:00:01 - IX.D.NASDAQ.CASH.IP
persist 60: 18/03/2021 12:00:01 - IX.D.SPTRD.DAILY.IP
persist 60: 18/03/2021 12:00:01 - IX.D.DAX.DAILY.IP
persist 60: 18/03/2021 12:00:01 - IX.D.FTSE.DAILY.IP
persist 60: 18/03/2021 12:00:01 - CC.D.DX.USS.IP
persist 60: 18/03/2021 12:00:01 - CS.D.USCSI.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.AUDJPY.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.AUDCHF.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.GBPAUD.TODAY.IP
response is null for CS.D.GBPCAD.TODAY.IP 60
persist 60: 18/03/2021 12:00:01 - CS.D.EURCAD.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.USDMXN.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.GBPJPY.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.CADJPY.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.CADCHF.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.EURAUD.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.NZDJPY.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.NZDUSD.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.CHFJPY.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CS.D.USCGC.TODAY.IP
persist 60: 18/03/2021 12:00:01 - CC.D.CL.USS.IP
persist 60: 18/03/2021 12:00:01 - CS.D.AUDCAD.TODAY.IP
time: 18/03/2021 12:03:01

 

[Chrislongden]

Thanks for your response, svb. It looks like my problem is indeed an allowance problem but I have been seeing the same intermittent issues as you with various waits/pauses affecting it (I found I needed 2.5 seconds to make a consistent difference).

When I do historical price search from the API companion, all I get back is this: errorCode": "error.public-api.exceeded-account-historical-data-allowance. I can't see that meta data that you reference. Are you able to see this when you use the companion? Are you using a historical price request?

Thanks

[svb]

if you don't have allowance, for that API key, that's all you'll get. Agree 2.5 sec is more reliable.

[PeterDawson]

For what it's worth... I too get forbidden responses from the REST API and I too retry the Price request and then it usually works. In fact, I retry up to 10 times as sometimes it gives me forbidden repeatedly. It can happen getting Navigation Nodes as well as Prices. It isn't allowance and looks to me more like something server-side e.g. bandwidth throttling perhaps? I just retry failed requests in a loop and haven't figured out any proper fix but the retry loop works fine so not a big problem

[esteban108]

Hello everyone,
I add to the problem, until last week everything was working great. This week the api returns errorCode: null in any type of call. Not only with prices, also obtaining markets, and even opening the session. It seems to me a somewhat dirty solution to retry in all calls, and that increases the operation time of my algorithms, making it unsafe