[API] Historical Data Allowance

[Nexuscrawler]

I programmed a bot to automatically create and close orders for me based on several moving averages and trade momentum. I've tried to find a broker with an API for quiet a while now and I think IG might be the best choice for me, but I'm really having trouble testing my trade strategies mainly due to the low data point allowance on historical data.

The advantage of using a bot is partially its high ability for number-crunshing, but the incredibly low limit of 10,000 data points per week (~60 points per hour) is just not enough. My bot is iterating through the markets, checking for positions with positive trends. Once it finds them, I want to backtest my strategy for this position to see if the bot would have made or lost money with it. The resolution I'd need for that would be in hours, but for just one year worth of data that's 8,760 data points already and then I'd have to wait a week to get more. It's just not feasible.

I know that IG probably has these data limits to prevent people from abusing the API, but the main culprit would not be the amount of data points, but the amount of request. Is there a specific reason why historic data requests are not handled like any other request going to the API (Limited to 60 requests per minute) and instead has this weird way of handling it with data points? The current amount of data could easily be interpreted by a human looking at a chart, which deafeats the whole purpose of having a bot.

My suggestion would be to simply set the allowed amount of requests for historical data to 60 per minute, if that limit is not applied already anyway. Set the amount of data points per page to 250 (or 100, or 500, whatever you feel comfortable with) instead of letting the user decide and the just return a link to the next page like it's already being done. This way we could comfortably get all the data we need while still being throttled.

It would be great if you could consider it.

[DSchenk]

Maybe you can apply for a corporate account? Not sure if they charge you then though.

Email: newaccounts.uk@ig.com

Other option/workaround (although not sure if IG wants to see this ) : Work with multiple API keys. Once one has reached its limit, swap over to the next one. Rotate like 5 keys through, at least that way you have 50,000 historical data points

Or: Look if you can optimise your query. Sometimes it's better to scan on a broad level, e.g. daily or monthly timeframe, then we you find something interesting, zoom in to a hourly or 5-minute chart or whatever it is you need for your strategy.

 

In regards to backtesting, try the backtest functionality of ProRealTime. I think it's pretty powerful. No real need to use the API imho.

[Nexuscrawler]

36 minutes ago, DSchenk said:

Maybe you can apply for a corporate account? Not sure if they charge you then though.

Would a corporation account remove that limit? I imagine I'd actually need a corporation to be able to sign up for one?

36 minutes ago, DSchenk said:

Other option/workaround (although not sure if IG wants to see this ) : Work with multiple API keys.

I do know that this is not allowed for real-time subscriptions for the lightstreamer interface, not sure about the REST API. Even with 5 accounts though, 50.000 data points per week are still rather low. It's just pretty outdated considering today's technology, but on the other hand, IG is one of very few European brokers who even have an API. It just feels a bit abandoned, because requests to raise the historical data limits are made since 2014 and my IG-Labs account is still not activated after a week, hence me giving feedback here. 😃

36 minutes ago, DSchenk said:

Look if you can optimise your query.

I'm already querying data for positions only once and then store them in my own database, so once I have the necessary historical data for one position, I'm pretty much set for it. My problem are the 40 other positions and I don't want to wait 40 weeks. A hourly resolution is the minimum I need for my strategy unfortunately. i could maybe go with 6 months for backtesting, but I'd prefer a year since I'd like to hold positions over a long period.

36 minutes ago, DSchenk said:

In regards to backtesting, try the backtest functionality of ProRealTime. I think it's pretty powerful. No real need to use the API imho.

I'm using the API to be able to completely automize the lookup of trades, strategy testing and trading. The goal would be for me to not have to do anything at all. 😃 Not sure how I would include ProRealTime into that workflow, unless they also have an API I could use for backtesting. I just need to be able to connect to stuff somehow.

Edited January 10, 2020 by Nexuscrawler

[Caseynotes]

2 hours ago, Nexuscrawler said:

I programmed a bot to automatically create and close orders for me based on several moving averages and trade momentum. I've tried to find a broker with an API for quiet a while now and I think IG might be the best choice for me, but I'm really having trouble testing my trade strategies mainly due to the low data point allowance on historical data.

Hi, this question has come up a number of times in the past, I don't use the API but have you searched the forum for previous threads on the subject? If you get nothing on the forum search box use google with 'api data allowance' and add 'IG community'.

[Nexuscrawler]

4 hours ago, Caseynotes said:

Hi, this question has come up a number of times in the past, I don't use the API but have you searched the forum for previous threads on the subject?

I did, yes, but most of these questions are about the error message you get when you exceed your data allowance, and not actually asking to raise it. The most recent question about it I could find was asked in 2014 and ended with IG saying that they'll look into it. Since it's been more than 5 years I wanted to bring it up again, because it's currently the biggest limitation you have when developing automatic trade programs.

[Caseynotes]

1 minute ago, Nexuscrawler said:

I did, yes, but most of these questions are about the error message you get when you exceed your data allowance, and not actually asking to raise it. The most recent question about it I could find was asked in 2014 and ended with IG saying that they'll look into it. Since it's been more than 5 years I wanted to bring it up again, because it's currently the biggest limitation you have when developing automatic trade programs.

ok, I vaguely remember peeps discussing it here in the last few years and thought someone had a get round for the problem but I may be wrong, anyway there are a few api'ers check in on this forum so maybe over the weekend someone might offer up some news. Reply to this post tomorrow sometime to keep the thread up on the noticeboard so others might see it and respond.

[Nexuscrawler]

Since it's been nearly 5 months, bump!

I'm getting the feeling the API is just no longer being worked on? The streaming client is deprecated for 4 years now.

[Bopperz]

If you email them they will increase your allowance temporarily. I was using a demo account too, they might be more friendly if they can see trading/income potential.

I used to hit the limit regularly, but now I keep a database of prices, and I only need to update these daily. So I am using much less data.

edit: What assets are you looking at? Can you get data from another source?

Regarding the IG API in general, its not the best. I really like the web interface, and I used PRT for a while and liked it. But I do have a Oanda account purely because their API is much nicer!

 

Edited July 31, 2020 by Bopperz