Windows proxies/certificates using WinHTTP

[WebFreak001]

Maybe dlang-requests should additionally wrap WinHTTP on Windows for not needing any dependencies like SSL.

It can take the system proxy (if http_proxy is not set) and has a lot of additional internal capabilities like kerberos authentication which I think would be pretty cool to have. Also it's recognized for example by anti-virus tools and other utilities to have all HTTP requests tracked using a central API. I think there could be a good wrapper around it in the library.

It would be cool if it would be compatible with vibe.d tasks using the async methods which integrate in the rest of Win32 events.

https://docs.microsoft.com/en-us/windows/desktop/WinHttp/about-winhttp

[TransientResponse]

I've not been able to get the OpenSSL stuff working with this library, so I'm going to fork it and give a WinHTTP wrapper a shot. Shouldn't be too hard (famous last words).

[WebFreak001]

I wrote some WinHTTP code to download a file if you want to look at it for reference: (though there might be better ways, it's a way that worked and is stitched together from the examples and documentation) https://github.com/Pure-D/serve-d/blob/master/http/source/served/http.d#L38

[TransientResponse]

I've already got working code using WinHTTP in D by copy-pasting MS example code, D-ifying it, and then lots of trial and error :).

Your example looks like Wininet though. Well, wininet will let me handle FTP as well. I'll stick to WinHTTP for the HTTP side as I've already figured out how to do headers and such on that.

[ikod]

Hello,
Is is possible to adapt windows ssl library (SecureChannel?) to solve this problem? It would be nice to add this functionality. I'm very weak with windows programming, so I afraid to start this job.

[TransientResponse]

So for WinHTTP SSL is handled very easily. The function WinHttpOpenRequest is called with an SSL flag, like so: WinHttpOpenRequest(hConnect, "METHOD"w.toUTF16z, "/route"w.toUTF16z, null, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, WINHTTP_FLAG_SECURE);

That WINHTTP_FLAG_SECURE is literally it, assuming you gave the correct port in the earlier call to WinHttpConnect. Windows does the whole SSL connection thing itself. Totally transparent from the point of view of the API. For non-HTTPS calls, passing a flag value of 0 works.

[ikod]

So for WinHTTP SSL is handled very easily. The function WinHttpOpenRequest is called with an SSL flag, like so: WinHttpOpenRequest(hConnect, "METHOD"w.toUTF16z, "/route"w.toUTF16z, null, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, WINHTTP_FLAG_SECURE);

That WINHTTP_FLAG_SECURE is literally it, assuming you gave the correct port in the earlier call to WinHttpConnect. Windows does the whole SSL connection thing itself. Totally transparent from the point of view of the API. For non-HTTPS calls, passing a flag value of 0 works.

But with WinHTTP we will lose some dlang-requests functionality like receiveAsRange or not? If not - I'm ok with WinHTTP for windows builds.

[TransientResponse]

`ReceiveAsRange` is actually kind of the default of how WinHTTP handles response data—you read it in chunks. My example code stitched these chunks together with ~= on a dynamic array, but we could easily make a wrapper to return the original chunks instead. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

…

________________________________ From: ikod <[email protected]> Sent: Saturday, July 27, 2019 10:03:52 AM To: ikod/dlang-requests <[email protected]> Cc: TransientResponse <[email protected]>; Comment <[email protected]> Subject: Re: [ikod/dlang-requests] Windows proxies/certificates using WinHTTP (#97) So for WinHTTP SSL is handled very easily. The function WinHttpOpenRequest is called with an SSL flag, like so: WinHttpOpenRequest(hConnect, "METHOD"w.toUTF16z, "/route"w.toUTF16z, null, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, WINHTTP_FLAG_SECURE); That WINHTTP_FLAG_SECURE is literally it, assuming you gave the correct port in the earlier call to WinHttpConnect. Windows does the whole SSL connection thing itself. Totally transparent from the point of view of the API. For non-HTTPS calls, passing a flag value of 0 works. But with WinHTTP we will lose some dlang-requests functionality like receiveAsRange or not? If not - I'm ok with WinHTTP for windows builds. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fikod%2Fdlang-requests%2Fissues%2F97%3Femail_source%3Dnotifications%26email_token%3DABJLBRAI5MJGK6PY2JFUHKLQBRIURA5CNFSM4HCADZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26MBJQ%23issuecomment-515686566&data=02%7C01%7C%7C342479bf62084bc88a5608d7129b4225%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636998330340953219&sdata=0%2FshOGRVvCkXoCixudWCYE58huhykRm%2F%2BekUXmzmr4k%3D&reserved=0>, or mute the thread<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABJLBRAB2YNK5XDIBS6GEVDQBRIURANCNFSM4HCADZDA&data=02%7C01%7C%7C342479bf62084bc88a5608d7129b4225%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636998330340963224&sdata=8QwxytNhcF%2FCaTE1REYcagCAWTN4SxVrA8H8RcNu3bs%3D&reserved=0>.

[ikod]

Thanks @TransientResponse !

Then I think it is worth to try. Did you forked this repo in order to implement this feature?

[TransientResponse]

It’s forked here: https://github.com/TransientResponse/dlang-requests First small, non-functional commit coming shortly. I’m heading out of the house soon. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

…

________________________________ From: ikod <[email protected]> Sent: Saturday, July 27, 2019 10:18:23 AM To: ikod/dlang-requests <[email protected]> Cc: TransientResponse <[email protected]>; Mention <[email protected]> Subject: Re: [ikod/dlang-requests] Windows proxies/certificates using WinHTTP (#97) Thanks @TransientResponse<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FTransientResponse&data=02%7C01%7C%7Ce0ab41ce3a21472ab49908d7129d492c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636998339049299769&sdata=gWLLnPM9kkZQ01NUtOYzbPMY6b20FoleN%2BJ9UCnvp58%3D&reserved=0> ! Then I think it is worth to try. Did you forked this repo in order to implement this feature? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fikod%2Fdlang-requests%2Fissues%2F97%3Femail_source%3Dnotifications%26email_token%3DABJLBRH36YSXPPJREM6NWD3QBRKK7A5CNFSM4HCADZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26MI4Q%23issuecomment-515687538&data=02%7C01%7C%7Ce0ab41ce3a21472ab49908d7129d492c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636998339049309773&sdata=6uJbfu%2B3lx4LkQqMpuQi5Agd66IuwVbIRGIwxQ0Z3Fo%3D&reserved=0>, or mute the thread<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABJLBRA5TQMTHJBVGDHIQ7TQBRKK7ANCNFSM4HCADZDA&data=02%7C01%7C%7Ce0ab41ce3a21472ab49908d7129d492c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636998339049319778&sdata=4alyzw4C49Pa4ubryCMjujhvGM8pE7o776WjbAwOlsE%3D&reserved=0>.

[TransientResponse]

Well, my first implementation works and 11/12 unittests pass (unless dub is tricking me). The test that's failing is package.d:80:__unittest_L29_C9 Check handling incomplete status line, and the reason seems to be that a GET to /incomplete on httpbin.org (local or remote) is returning 404 and not 600 or 502 as the assert wants. Not sure why this is.

[ikod]

Well, my first implementation works and 11/12 unittests pass (unless dub is tricking me). The test that's failing is package.d:80:__unittest_L29_C9 Check handling incomplete status line, and the reason seems to be that a GET to /incomplete on httpbin.org (local or remote) is returning 404 and not 600 or 502 as the assert wants. Not sure why this is.

Very good!

You can enable full debug on particularly this test with globalLogLevel = LogLevel.trace and Request.verbosity = 2 or 3; You can place debug output here, I'll check

[ikod]

hello @TransientResponse

any news on this?

[TransientResponse]

I'm so sorry for the long delay on this, life got hectic for me at that time and I got pulled away by other things. I'll try to get back into it tomorrow and update you then. On Friday, November 12, 2021 6:40 AM (-05:00), ikod wrote: hello @TransientResponse any news on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

…

-- Sent with Vivaldi Mail. Download Vivaldi for free at vivaldi.com

[TransientResponse]

I set loglevel to trace and verbosity on the failing request to 2, and I only get a little more info:

2021-11-13T08:57:35.574 [info] source\requests\package.d:82:__unittest_L29_C9 Check handling incomplete status line
2021-11-13T08:57:35.575 [trace] source\requests\http.d:2616:execute serving Request(GET, http://127.0.0.1:8081/incomplete)
Doing connection to 127.0.0.1 with scheme http
Doing GET for /incomplete
2021-11-13T08:57:35.580 [trace] source\requests\streams.d:1000:connect Create connection to 127.0.0.1:8081
2021-11-13T08:57:35.580 [trace] source\requests\streams.d:1009:connect Trying 127.0.0.1:8081
2021-11-13T08:57:35.581 [trace] source\requests\streams.d:1019:connect Connected to 127.0.0.1:8081
2021-11-13T08:57:35.581 [trace] source\requests\streams.d:981:close Close socket

core.exception.AssertError@source\requests\package.d(86): unittest failure

The trace lines from streams.d look as though it's not calling the WinHttp API at all in that test, but I know it is compiling and testing. I can get errors out of WinAPI if I don't have an httpbin server running locally (like in docker). Unless the unittest build overrides version(Windows) checks?