Lightstreamer Connect fail on DEMO account

[swingwin]

I can't connect to the DEMO stream ("https://demo-apd.marketdatasystems.com") since December 2021 with a C# .NET application using the IG API
On the other hand the connection to a LIVE stream ("https://apd.marketdatasystems.com") works

It seems that the problem comes from the "DotNetClient_N2.dll" library.
Has anyone encountered the same problem ?

 

Edited January 22, 2022 by swingwin

[mf2]

@charlie303

I used IG's Streaming Companion: https://labs.ig.com/sample-apps/streaming-companion/index.html

You can put your credentials there, subscribe to markets etc. and check the requests in the browser console. There I noticed that at least the companion is using version 6.

Regarding C# I can’t provide any help, sadly, as I am using C++ and libcurl.

[charlie303]

@mf2

Many Thanks for your help you have steered me in the right direction. I understand now the libraries you are referring to are probably the javascript client libraries v6 which the IG Companions use. I downloaded Fiddler and did some investigation and have revisited code samples.

Here's what I have gleaned so far and what I hope to do next.
It seems to me the issue is to do with logging in to the REST API using OAuth - which is detailed here as POST v3.0
https://labs.ig.com/rest-trading-api-reference/service-detail?id=600

Once this is achieved I then need to retrieve the session tokens (CST and XST) which will allow me to connect to the LightStreamer server. If any one wishes to follow this it is in the IG API companion, first Login (OAuth) then just below in Session Details set 'Retrieve session tokens to true'. On the right hand side you will see the CST and XST headers retrieved).

This explains why @jlz code did not work for me because it is still using REST API Authenticate V2 and not the V3 OAuth method. @jlz code has the new .Net Standard library (v5.0.5) but this, apparently, is not enough.
I owe @TeeBeast a lot more credit because in his fork (listed in this thread) he does use the V3 OAuth authentication code. I initially struggled with his code because I am developing in Visual Studio 2017 and his fork, I believe is for Visual Studio 2019. The Windows Presentation Forms project failed to load for me as well but after a lot of deep diving (especially with Notepad) I was able to understand most of it (or so I like to thnk!). Some of the REST API code seems a little too tightly coupled with WPF which is not ideal for me as I still prefer good old chunky Windows Forms. His implementation of V3 OAuth seems 'indepth' but this is not intended as a criticism as some research shows OAuth code and libraries for C# / .Net are very complicated and overblown. I'm kind of stripping his code to make it work in Visual Studio 2017 but he has pointed me in the right direction too. I had to change some project build options too so as to use C# v7.x.

The V3 OAuth login explains why @Tartempion was having trouble as they too were probably using Authentication V2 within LabView.

I downloaded the java code sample (and Eclipse IDE) as the reports are this works. It seems to me the IG guys are very much java centric and have allowed the .Net guys to sink or swim. In the code (I'm not a java programmer) it seems to use V3 OAuth and then kinda downgrades and uses V2 to get the necessary CST and XST tokens from the HTTP headers for LightStreamer login).

So, I am currently trying to get my head around OAuth for C# - it is not documented on IG Labs website which version they use so I'm going to assume for now V2. If I can get some code to login (I managed to do this with a heavily hacked version of @TeeBeast) then I will need to write some code to obtain the CST and XST tokens for LightStreamer login). Then I should be able to connect and get live prices, etc.

This current thinking explains to me the issues reported here. Why the legacy .Net client does / does not work. Why the newer one also has intermittent results. Why java works. Why Visual Studio 2017 users have trouble. Why developers think the problem is focused around the LightSreamer STREAM API end when it actuality it could be the REST API login method. Why the IG Companions work but C# code fails.

Incidentally, I took the bold move of disabling my DEMO API Key on the My IG website and was then able to create a new one. I don't think IG Markets have any block lists as such, I just think they are more java centric and have other priorities.

I will keep this thread updated and if I can get it working I will post some code. It might be that you do not necessarily have to move to LightStreamer .Net Standard v5.0.5 libraries and the legacy ones will work once you can get round the AuthenticateV3 OAuth login hurdle. It's on my list because as of the time of writing I'm not blown away by .Net standard though I understand LightStreamer for .Net Standard is easier to use.

Or, I'm wrong and the problem is elsewhere. Fingers crossed.

Feedback appreciated, especially from @TeeBeast, who it looks had to hand crank the java code into C#, no easy task. I see from some of his GitHub issues people were having the CST / XST issue, for example at the end of
https://github.com/IG-Group/ig-webapi-dotnet-sample/issues/18
He and @jlz seem to be the main drivers of keeping C# / .Net code relevant. Maybe there should be one thread / github page to hold C# /.Net code / VB.Net, etc.

Charlie

 

[mf2]

FWIW, the programming language does not really matter and you don’t need any special libraries to access the LightStreamer API. You can send standard HTTP requests and it works, as long as you stick to the protocol. Personally, I prefer this approach as this makes my program independent from any third-party libraries.

Regarding version 6.0, I meant that the LightStreamer API Companion adds `LS_client_version=6.0` to the request. Note that you don’t need any JavaScript client libraries to access the API via JavaScript, you could use AJAX and/or the .txt endpoint instead of the .js endpoint.

The issue is just that some details in the specification are left to speculation or interpretation, therefore it requires some experimenting. Furthermore, you need to make sure that you process Chunked requests properly if you don’t use polling.

Session V3 authentication can be used in your program, you just have to issue another Session V1 REST Api call afterwards to retrieve the CST and XST tokens, as you observed. Also, the documentation is here, you just need to scroll down to V3: https://labs.ig.com/rest-trading-api-reference/service-detail?id=600

(Note that IG calls this "oauth", but its really not)

So here is what I am doing:

1. Authenticate with the REST api via a V3 session call
2. Issue a V1 session call to get the XST and CST token
3. Send the following HTTP POST request to the lightstreamer server, at `/lightstreamer/create_session.txt`:

"LS_cause=new.api&LS_polling=true&LS_polling_millis=0&LS_idle_millis=0&LS_client_version=6.0&LS_user=<IG Account ID>&LS_password=CST-<CST-Token>|XST-<XST-Token>&LS_container=lsc&"

Some notes:

1. I use polling, which is not recommended, but its fine for my case.
2. The `|` in the password needs to be escaped properly.

Edited May 16, 2022 by mf2

[TommyT]

On 14/05/2022 at 19:27, charlie303 said:

@mf2
Thanks for the reply. I've been going crazy as this weekend (Sat 14 May 2022) my LIVE LightStreamer API Key stopped working, though the API Companion website is still working.
 

@chalie303,

LIVE LightStreamer: same thing has happened to be since 14 May. I cant now connect, though I have been connecting fine, everyday for the last 2 months. 

This is too much of a coincidence that it happens to both of us at the same time. 

Regards TommyT 

[charlie303]

@TommyT

I'm guessing during a 'scheduled maintenance' IG implemented the same changes to their LIVE environment that have been running on their DEMO environment for a few months now. I'm guessing the main change that is affecting posters on this thread is that you will now be required to login to the REST API with a Version 3 Authenticate session call. $10 says your code is still using Version 2!

@mf2

Many, many thanks for your help and showing the way. I think the problem is we need to go to Version 3 authenticate on the REST API and then all else will fall into place. Sorry, if I confuse the issue about going on about 'libraries'. Time to just code and make it work. Appreciate the tip about it not been a 100% OAuth implementation.

I will post some code if/when I get it working.

It would be nice if IG could post somewhere updates to their REST / STREAM implementation just to give us a heads up in future.

Charlie

[TommyT]

2 hours ago, charlie303 said:

I'm guessing during a 'scheduled maintenance' IG implemented the same changes to their LIVE environment that have been running on their DEMO environment for a few months now. I'm guessing the main change that is affecting posters on this thread is that you will now be required to login to the REST API with a Version 3 Authenticate session call. $10 says your code is still using Version 2!

@charlie303,

I would agree that some Friday night maintenance has implemented some breaking changes to LIVE lightstreamer, but I am not convinced is a V3 issue. My feeling that is may be some setting of a connection option....possibly inaccessible or unsupported in the LSClient and ConnectionInfo  props

Is this c# conversion to V3 authentication something you have got working for lightstreamer connection.?

I note that (by using browser dev tools) the streaming companion uses V1 authentication and passes a basic CST/XST password token. This fits with mf2's opening of the connection 

"

1. Authenticate with the REST api via a V3 session call
2. Issue a V1 session call to get the XST and CST token
3. Send the following HTTP POST request to the lightstreamer server, at `/lightstreamer/create_session.txt`:

"

regards TommyT

[teebeast]

Hi,

I looked at my fork and found a bug, that the GET request /session?fetchSessionTokens=true was not called. So there was no correct authentication against lightstreamer. The sample application now works with my demo account.

Thanks to thompsont1962 at Github. For more information see https://github.com/teebeast/ig-webapi-dotnet-sample.

[swingwin]

@teebeast
I tested your latest update this morning.
For my demo and live accounts, it seems that the lightstreamer connection is working correctly, but then I have no stream that is done and this is true for all the tabs

 

[charlie303]

I'm having similar issues.
I was wrong (again!) about requiring OAuth login Authentication.

I've created a very simple Windows Form app where I copy and paste the CST and XST tokens over from the IG API Companion page. I hard code the lightstreamer endpoint and my client account ID.
I have stripped some very light code from both @jlz / algotrader and @teeBeast projects just to see if I can prove the functionality.

I can get a 'true' result when I connect to the lightstreamer endpoint but after that I can not stream any data at all.
I do get a lot of HTTP 204 No Content
The IG Stream Companion still works.

I'm on Visual Studio 2017 and have upgraded to .Net Framework 4.8. When I get the Lighstreamer .Net Standard 5.0.5 package from NuGet I also download .Net Standard libraries themselves. Sometimes I have had to restart the IDE and update again through NuGet to ensure I have the latest versions of the .Net Standard libraries. I am still on Lighstreamer 5.0.5 and now the latest version of Newtonsoft JSON.

It tells me I connect but nothing else.

I looked at @mf code to see if there are any hidden properties in the connection options (for example I will explore setting polling to 0) but I find the whole issue somewhat frustrating, especially when my previous Lightstreamer code worked a charm. Have IG Labs tested for .Net? I will look at sending a create session.txt file to see if I can get that working. @mf2 could you post a create_session.txt file and the complete URL you send it to?

Are you @TeeBeast actually getting live streaming data of market prices?
I use CS.D.AUDUSD.MINI.IP in the IG Stream Companion and no problems.

What am I missing?
 

[teebeast]

@swingwin

I fixed it short before i wrote my post. So please try it again.

[charlie303]

I saw that post and saw you added the Refresh-tokens call. Thanks
Are you actually getting live market prices? on CS.D.AUDUSD.MINI.IP  for example?

Is @swingwin still having issues?

Seems we are not the only ones

 

[teebeast]

@charlie303

When I fixed it, I had no problem with refreshing values.

When you create a test project at Github, I can look at your code. I implemented my own sample api including lightstreamer and its working fine. But currently I cannot publish it to Github, because I have to do some refactorings and removing secure information from source code.

[teebeast]

@charlie303

I just tested CS.D.AUDUSD.MINI.IP with my own implementation and I got values. Generally its the same implementation but i try to give it a cleaner structure of code. So it should also work with IG sample app.

[TommyT]

@charlie303,

in case your interested in keeping any c# code that uses v2 authentication:

I took teebeast's new project with v3 session auth and the 5.0.5 LS and I swapped out the v3 session auth and replaced it with a v2 session auth: Lo and behold the LS connect worked fine. Seems the 5.0.5 LS is the key to the broken link.

Regards TommyT

[mf2]

1 hour ago, charlie303 said:

@mf2 could you post a create_session.txt file and the complete URL you send it to?

Fascinating how complicated something supposedly simple as posting code on the IG messageboard is, just because support for proper formatting is missing.

Anyway, here is  a very minimal example for acessing a LightStreamer API: https://dpaste.com/BGNARUXSS

Notes:

• It does not use IG's server, but a public demo server. However, the code is essential the same as I am using in my application.
• I wrote it under Linux, it might be of limited use to a Windows User. Maybe under WSL it runs.
• Linux users can compile it via: gcc -lcurl main.c

 

[swingwin]

charlie303 wrote : "Is @swingwin still having issues?"
Yes I have exactly the same problems as you, since also in my case I develop a C# windows forms application.
The only difference with you is that I am using Visual Studio 2019 IDE and windows x64 configuration

Everything was OK until December 2021, when the DEMO lightstreamer stopped working.
And now since Sunday 15 May 2022, the lightstreamer LIVE does not work anymore.

@teebeast: I tested your update this afternoon, but still no data is coming in through the DEMO and LIVE lghtstreamer

 

[charlie303]

I guess I'm having a bad day ;-(

I've just installed Visual Studio 2019. I can load and run @teebeast code but no streaming data!
I'm on Windows 7. 64 bit machine.
Time to take a break.
I will look @mf2 stuff tomorrow.
There are posts in other threads about IG knowing there is some problem at their end, fingers crossed, maybe it will just work in the morning.

Many thanks for all your help though.

Charlie

 

[swingwin]

This evening I checked that the example and the java library were still working correctly.
With the java app everything is operational with the DEMO server. The Lightstreamer connection is OK and the streams are returning data.

And for this JAVA solution, the API session is created in version V2 (and not in version V3)

In my opinion this confirms more and more that our problems with .NET are in the .NET lightstreamer DLL library.

[charlie303]

@swingwin

That wouldn't explain why some people can get @teebeastor @jlzcode to work as we are all using the same libraries.

Why aren't they blocked?

My IG Stream Companion still works. I was going to try and get the java code to work so I appreciate your feedback on that. Other threads talk about regional IP address settings and where the account residency is located. Others about being on a blocklist because of over use in the past. I've seen people mention the problem with what appears to be Excel spreadsheets and I don't think they use .Net libraries as they are COM based, though I haven't investigated.

The truth is I don't know and I am pretty much out of ideas, especially after I went down the OAuth V3 Authentication rabbit hole and I am wary about erroneous speculation. I'm tending to think there is an issue at IGs end that they won't acknowledge because they are more java focused.

I'm scratching around in the dark at the moment but will continue to push the helpdesk to address the issue. Why haven't any IG admins posted on this thread? They do so on others when they have an answer.

Charlie

 

[swingwin]

@charlie303

Another test that is increasingly pointing to problems with the lightstreamer .NET library.

I just tested an old Python application, which I had archived in a corner.
Well, the Python lightstreamer works fine in both demo and live version.

For excel, I just checked and the excel solution provided by IG in the past is built on the "DotNetClient_N2.dll" library, and I do have the problem with Excel as the person who reported it a few days ago

[charlie303]

@swingwin

Thanks for the update.

That still doesn't explain why some people can get @teebeastand @jlz code to work.
Were all tests conducted with the same account username and password details? Are you able to test with someone else's credentials?

My suspicions point to the problem being isolated to .Net libraries and the IG LightStreamer server setup for certain users. Though what those users have in common that is causing the problem I don't know. This would explain why the IG team are reluctant to investigate if they are not experiencing the problems from a java / IG STREAM Companion point of view. They don't want to fiddle with settings.

IG support for Microsoft users is woeful.

I've phoned the helpdesk again and will persist.

Edited May 18, 2022 by charlie303
Too many carriage returns in original post.

[charlie303]

I'm tending to think that there is not a bug in the LightStreamer code as their libraries seem to be used extensively around the internet. IG are keen to point out to stick to LightStreamer .Net Standard client libraries v5.0.5 when the latest stable build is v5.1.3. Why the reluctance to move to the latest library? I've never had any problems with upgrading Newtonsoft.Json. Or have I got this wrong too. I may test against other versions of the library but feel this will be a wild goose chase.

Diagnosing the problem from the client end is difficult because in Fiddler I am not seeing any detailed HTTP Headers. If someone at IG's end could look at the LightStreamer server logs when one of us was trying to connect I'm sure useful information would be gleaned.

I guess a possible solution would be to roll our own libraries built from the socket level up but I am not fancying that. Isn't that what the provider libraries are supposed to do?

 

[teebeast]

@charlie303

You cannot update lightstreamer .net standard client higher than 5.0.5, because they changed some things that are not compatible with the server version IG is using. Here is the changeset at Github which shows you the incompatible change. I got the latest version of Lightstreamer-lib-client-dotnet to run, when I change the parameter LS_cid which I cannot recommend.

I also looked at the chart data with my demo account and has no problems. A problem could be the demo app itself. I only reworked the parts that were necessary to obviously run against the latest nuget packages. Maybe there are some bugs which I cannot reproduce.

Edited May 18, 2022 by teebeast
Grammatik

[charlie303]

Progress! but not yet a solution.

I was able to get my DotNetClient_N2.dll code working on my second machine! It is a Windows 10 Home machine and is normally used for music production, I don't want to clog it up with a development environment. I will copy over a .exe of @TeeBeast app to test later on.

My new line of investigation is the HTTPS SSL stack. I had this problem a long time ago when I was on XP and IG updated their HTTPS SSL stack. XP couldn't handle it and there was no way to patch as it is hard coded into the OS. That forced me to upgrade to Windows 7. SOAP UI still worked on Windows XP then because it uses its own SSL implementation written in java of course whereas I believe all .Net code goes through the Windows HTTP SSL stack.

I'm using
https://www.howsmyssl.com/
to initially compare the stacks between both machines and see if it is possible to patch my Windows 7 machine to the same stack as my Windows 10.

@swingwin
Are you on Windows 10? Is it fully updated?

I don't know if it is possible to find out what SSL stack and in what order IG are using. It would be helpful in future if IG could post to a sticky thread the changes they make to their servers.

Charlie

 

[swingwin]

@Charlie

No windows10 which prevents me from installing VS2022.
My configuration : windows 7 64 bits

Seeing that the Lightstreamer Python library seems to work well, and with good performances, I'll try to interface it with .NET.
It should be much less complicated than interfacing the Lightstreamer Java library

[Tartempion]

Hello guys !

I had the same problem as you since Monday morning although I don't use the dotnet library from IG. I use two constructors from the dotnet framework in my LabVIEW program. The first is WebClient (it's a bit barbaric as a way of doing things, but it's been working well since 2015 even if Microsoft recommends using a more current solution: System.Net.Http.HttpClient ).

Its path is System (4.0.0.0)>System.Net>Webclient().

This constructor has a method named OpenRead which generates a Stream instance. Then I use the StreamReader constructor (the second constructor).

Its path is mscorlib(4.0.0.0)>System.IO>StreamReader(Stream,Stream).

This constructor takes as argument the Stream instance provided by the OpenRead method of the WebClient. Then just call the "ReadLine" method of the StreamReader in a loop.

I tell you my life to make it clear that with this very simple structure I had the same problem as you since this weekend. The stream server was rejecting me without even returning a response. But today I added in my program a call to the ServicePointManager property. The call to this property is done before the call to the WebClient constructor.

The System.Net.ServicePointManager class has the path :

System(4.0.0.0)>System.Net>ServicePointManager

and I assign it the value Tls12 (among the other values which are: SystemDefault, Ssl3, Tls, Tls11,Tls12, Tls13 and " others"). And the flow starts again. I don't use version 3 for the IG API  and it still works. Hope this can help you!

Thank you for your various messages wich allowed me to find the way out of my problem. I hope this will also solve yours which seems to me to be of the same nature.

Thanks to Firefox and its Web Development tool in the IG Companion webpage (https://labs.ig.com/sample-apps/streaming-companion/index.html) .

 

[Tartempion]

Sorry I didn’t name the property : The property of the ServicePointManager class  is  SecurityProtocol. This property must be set to "Tls12"

[charlie303]

@TartempionWell Done!

May I ask what machine your code is running on? Is it Windows 7?

My research says Windows 7 OS can't install the TLS Cipher Suite for HTTPS SSL that you are using, though independent apps running on Windows 7 can build their own, eg Brave Browser, Firefox, SOAPUI which is written in java and I guess the Python environment too, must all build their own HTTPS SSL stack independent of Microsoft.

Maybe there is a way of adding it. It seems Microsoft programmers are out of luck (again) as we are hard wired to the underlying OS despite all the marketing hype about how 'open source' the framework supposedly is. Forced into upgrades for little benefit.

My solution for now is to use my second machine for development as my code works - no changes whatsoever, still uses the DotNetClient_N2.dll.

I'm impressed with your solution as I was with @mf2for rolling your own libraries. When I get time I will do the same for my code. Their is information out there about the HTTP Clients in .Net and libraries like CurlThin for C#.

Would you care to share the few lines of Webclient / StreamReader you use? 

It is frustrating that needless small changes to libraries / server setups break existing apps. Open Source is the future. I say needless because IG have different HTTPS Cipher Suites on their servers for the REST API and the STREAM API. If it was a technical or security decision then why the differences? Scroll down and compare Configuration -> Cipher Suites if you have time.

https://www.ssllabs.com/ssltest/analyze.html?d=api.ig.com

https://www.ssllabs.com/ssltest/analyze.html?d=demo-apd.marketdatasystems.com

 

[Tartempion]

@charlie303

Windows 10.

Unfortunately I am completely unable to translate this into c# code. My knowledge of C# is too rudimentary. I am learning C# but at a really beginner level. It's easier to read than to write. I program in Labview. I wrote a very synthetic labview code that summarizes my use of the two dotnet constructors. This code is not executable for someone who wants to run it with labview. It's just a fairly accurate representation of what you were asking me. I can't go any further, I don't have the skills in C#. But Swingwin knows a little labview, maybe he can translate my code for you.

Labview programmation is data flow programmation. The data flow is going from left to right.

Hope the image is readable enough.

[Tartempion]

The dotnet methods used are NOT asynchronous. But their async versions exist .
 (For labview , not async dotnet methods are better because alll the rest of program is asynchronous and parallelized.
It would be useless and it would consume more resources just for the StreamReader.

Useless because he is alone for the dotnet engine. )