Daily histrorical data - interpretation of UTC (?) date stamp and daylight savings

[smx01]

Hi everyone,

I am working to download historical daily data via the REST API (crypto EPICs mainly - e.g. CS.D.BCHUSD.CFD.IP). 

I believe the timestamp that comes with the data us UTC. I am finding that for the period where AEST is in daylight savings time (DST), e.g. Oct-6-2019 - Apr-5-2020, the date stamp changes. Here are two corresponding transitions:

Here's what the REST API looks like around Oct-6

Quote

 

Response from IG REST API

{'prices':                         bid                             ask                         spread                  last                         

                       Open    High     Low   Close    Open    High     Low   Close   Open High  Low Close  Open  High   Low Close Volume

DateTime                                                                                                                                 

2019:10:04-00:00:00  221.72  223.35  215.64  222.62  223.72  225.35  217.64  224.62    2.0  2.0  2.0   2.0  None  None  None  None  24155

2019:10:05-00:00:00  222.63  224.51  217.35  220.59  224.63  226.51  219.35  222.59    2.0  2.0  2.0   2.0  None  None  None  None  15863

2019:10:06-00:00:00  220.58  222.86  217.54  219.96  222.58  224.86  219.54  221.96    2.0  2.0  2.0   2.0  None  None  None  None  19415

2019:10:06-23:00:00  219.94  229.79  214.72  228.05  221.94  231.79  216.72  230.05    2.0  2.0  2.0   2.0  None  None  None  None  33125

2019:10:07-23:00:00  228.06  238.31  228.05  231.62  230.06  240.31  230.05  233.62    2.0  2.0  2.0   2.0  None  None  None  None  32501, 'instrumentType': 'CURRENCIES', 'allowance': {'remainingAllowance': 9992, 'totalAllowance': 10000, 'allowanceExpiry': 602836}}

 

And here's what it looks like around April-5 2020

Quote

 

Response from IG REST API

{'prices':                         bid                             ask                         spread                  last                          

                       Open    High     Low   Close    Open    High     Low   Close   Open High  Low Close  Open  High   Low Close  Volume

DateTime                                                                                                                                  

2020:04:02-23:00:00  225.78  248.33  224.26  236.75  227.78  250.33  226.26  238.75    2.0  2.0  2.0   2.0  None  None  None  None  106597

2020:04:03-23:00:00  236.81  238.76  229.59  234.14  238.81  240.76  231.59  236.14    2.0  2.0  2.0   2.0  None  None  None  None   47901

2020:04:04-23:00:00  234.15  241.81  233.12  235.39  236.15  243.81  235.12  237.39    2.0  2.0  2.0   2.0  None  None  None  None   59462

2020:04:06-00:00:00  235.37  245.53  224.56  243.58  237.37  247.53  226.56  245.58    2.0  2.0  2.0   2.0  None  None  None  None   65707

2020:04:07-00:00:00  243.57  264.96  241.44  258.13  245.57  266.96  243.44  260.13    2.0  2.0  2.0   2.0  None  None  None  None   84988

2020:04:08-00:00:00  258.14  279.18  245.70  259.45  260.14  281.18  247.70  261.45    2.0  2.0  2.0   2.0  None  None  None  None  103281

2020:04:09-00:00:00  259.48  273.91  252.52  261.94  261.48  275.91  254.52  263.94    2.0  2.0  2.0   2.0  None  None  None  None   95054

2020:04:10-00:00:00  261.95  262.39  231.19  235.09  263.95  264.39  233.19  237.09    2.0  2.0  2.0   2.0  None  None  None  None   97915, 'instrumentType': 'CURRENCIES', 'allowance': {'remainingAllowance': 5197, 'totalAllowance': 10000, 'allowanceExpiry': 253266}}

 

So two questions:

1. Why does this happen? UCT is UCT - 00:00:00 corresponds to the start of the day for a 24 hr period. I don't know why it would shift just because we're in DST in AEST.  Is it that the actual reference point for a 24 hr period is 10am -10am AEST, and when DST starts, the UTC adjusts? And then the actual daily/24hr data then 23:00:00 UCT to 23:00:00 UCT? 

2. So you end up with dates within DST shifted by 1 day, which can be fixed post-REST-API - but is there another way to deal with this within the API itself?

Thanks for any advice...

Steven.

[majicktrader]

Hi Steven, I just found this because I was looking up about daylight saving. 

The English have British Summer Time BST which is UTC+1, so the market adjusts. It also reverts back to UTC+0 when it goes back in time. 

Daylight saving also occurs in NY, so they too move forward in Summer by 1 hour and back in Winter. 

Personally it does my head in, especially when my timezone doesn't change. 

[jlz]

30 minutes ago, majicktrader said:

Hi Steven, I just found this because I was looking up about daylight saving. 

The English have British Summer Time BST which is UTC+1, so the market adjusts. It also reverts back to UTC+0 when it goes back in time. 

Daylight saving also occurs in NY, so they too move forward in Summer by 1 hour and back in Winter. 

Personally it does my head in, especially when my timezone doesn't change. 

Every programming language has a core method to convert datetimes to local times.

Python: https://stackoverflow.com/questions/4770297/convert-utc-datetime-string-to-local-datetime

c#: https://docs.microsoft.com/en-us/dotnet/api/system.timezoneinfo.converttimefromutc?view=netcore-3.1

You don't have to deal with time zones yourself. 

[smx01]

On 14/10/2020 at 17:50, majicktrader said:

The English have British Summer Time BST which is UTC+1, so the market adjusts. It also reverts back to UTC+0 when it goes back in time. 

Daylight saving also occurs in NY, so they too move forward in Summer by 1 hour and back in Winter. 

Personally it does my head in, especially when my timezone doesn't change. 

Thanks.

But why does IG do this? UTC is universal, its a fixed notion of time. UTC itself shouldn't/doesn't change because of daylight savings etc.

Is it the case that IG 24 windows are timed according to a fixed local time in London? Or Sydney? And therefore the UCT changes?

Still confused

[smx01]

On 14/10/2020 at 18:24, jlz said:

Every programming language has a core method to convert datetimes to local times.

Python: https://stackoverflow.com/questions/4770297/convert-utc-datetime-string-to-local-datetime

c#: https://docs.microsoft.com/en-us/dotnet/api/system.timezoneinfo.converttimefromutc?view=netcore-3.1

You don't have to deal with time zones yourself. 

Thanks Jlz. I've used the arrow lib, it's excellent.

My assumption is that the timestamps coming out of IG are UCT. So I translate the UCT to my local time zone (Melbourne, Aus). 

My confusion is more around my previous response, why does the UCT time shift change, if it's UTC. It must be because ultimately that UCT is based on some other timezone. 

Edited May 13, 2021 by smx01

[smx01]

Is anyone from IG available to answer this query definitively, once and for all?