Access historical data

[Michael1964]

Can anyone advise me on how to obtain all the pricing data (1 min resolution) on IG for the past 10 years? I'm doing a research paper and need to run simulations which I'm happy to share with anyone who can help. I cannot seem to retrieve much via the API companion, given the 10,000 data points limit. The IP companion also doesn't appear to provide 1 min resolutions. I can only get 10 min resolutions. 
Is there anyone who can help me with the harvesting of the data? 

[Michael1964]

16 hours ago, bug-or-feature said:

My advice would be to not attempt to do anything like this. "All the pricing data at 1min resolution" is an absurd amount of data. The IG platform is not a good place to try anything like this, due to the number of markets on offer, and the rate limitations.

There are plenty of other platforms where you can obtain financial pricing data, either paid or free

Thank you for your response. Rate limitations indeed.
Unfortunately my research is based on a spread hypothesis of all the indices on IG. 
I've looked at other platforms but cannot find anything that covers all the indices. Which platform would you recommend as being the most comprehensive? 

[bug-or-feature]

Ah, well firstly "all the indices on IG" is a lot less than "all the pricing data". It's still a lot. Spread bets, CFDs, something else?

A lot depends what your hypothesis is. Are you investigating spread values? If so, you could get the prices from somewhere else, and sample the IG spread using the Streaming API. With that API you can be subscribed to 40 instruments concurrently. See here

Platform choice is a big subject, probably not a question for this forum - I'm sure you have great Google skills being a researcher. But you could start with Yahoo Finance, Google Finance or AlphaVantage

[TheGuru12]

You can't get this kind of data from IG Index. Try this but even this has limits

 

https://www.dukascopy.com/swiss/english/marketwatch/historical/

[bug-or-feature]

18 hours ago, SmitBret said:

First, you can use the IG API with pagination and aggregation. Although the IG API has a limit of 10,000 data points per request, you can paginate through the data and aggregate it yourself. This will involve writing a script that requests data in chunks and then stitches them together. To do this, you need to determine the date range by calculating the total number of minutes in 10 years, set up pagination by making API requests in a loop, each time requesting the next chunk of 10,000 data points until you have covered the entire date range, and handle API limits by incorporating necessary delays between requests to avoid getting blocked.

Unhelpful - the IG API for historic data has a limit of 10,000 data points per week, not per request. See

https://labs.ig.com/faq.html

under "Limits"

[bug-or-feature]

My advice would be to not attempt to do anything like this. "All the pricing data at 1min resolution" is an absurd amount of data. The IG platform is not a good place to try anything like this, due to the number of markets on offer, and the rate limitations.

There are plenty of other platforms where you can obtain financial pricing data, either paid or free

[SmitBret]

First, you can use the IG API with pagination and aggregation. Although the IG API has a limit of 10,000 data points per request, you can paginate through the data and aggregate it yourself. This will involve writing a script that requests data in chunks and then stitches them together. To do this, you need to determine the date range by calculating the total number of minutes in 10 years, set up pagination by making API requests in a loop, each time requesting the next chunk of 10,000 data points until you have covered the entire date range, and handle API limits by incorporating necessary delays between requests to avoid getting blocked.

If the IG API is not feasible, consider alternative data sources. Services like Quandl, EOD Historical Data, or Alpha Vantage may offer historical minute-by-minute data, though some of these services might require a subscription. Brokerage platforms or brokers like Interactive Brokers or MetaTrader might also provide extensive historical data to their customers.