[Caml-list] Re: Multithreaded https requests in ocamlnet netclient

[Mike Lin <mlin@mlin.net>]

[-- Attachment #1: Type: text/plain, Size: 3221 bytes --]

Thanks, Gerd!

FWIW I could not reproduce the crash by using ocaml-ssl's blocking
operations directly.

https://gist.github.com/4152047#file_ssl_threads.ml

This works fine- so, perhaps something nasty arises from using nonblocking
I/O on ssl sockets from multiple threads. I'm sure if there is any other
critical difference with how netclient/equeue-ssl and my example use
ocaml-ssl.

I also don't have time to pursue this much further, so I will try to put
all of my http operations on one thread as your example suggests.

Best,
Mike


On Tue, Nov 27, 2012 at 5:03 AM, Gerd Stolpmann <info@gerd-stolpmann.de>wrote:

> Hi Mike,
>
> I can confirm the problem, but I'm not able to track it down.
>
> First of all, I'm quite sure that you cannot share SSL contexts between
> threads. But even if I fix this, I get crashes - usually early during
> SSL_connect.
>
> I'm wondering whether ocaml-ssl initializes openssl correctly for
> multi-threading. In my version of openssl the prototype is
>
> int CRYPTO_THREADID_set_callback(**void (*threadid_func)(CRYPTO_**THREADID
> *));
>
> but in ocaml-ssl the CRYPTO_THREADID pointer is ignored (it assumes void
> here). So, maybe this is the reason. I currently don't have time to follow
> this idea.
>
> If everything fails, there is a way to use Netclient in threaded
> applications so that only one thread is doing the networking stuff, see
> this example:
>
> https://godirepo.camlcity.org/**svn/lib-ocamlnet2/trunk/code/**
> examples/netclient/simple/**http_mt.ml<https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/examples/netclient/simple/http_mt.ml>
>
> As Netclient is event-driven, the requests are processed in parallel (with
> one exception, though: DNS lookups are synchronous, but this can be worked
> around, as the DNS lookup function is settable in Netclient).
>
> Gerd
>
> Am 27.11.2012 03:54:16 schrieb(en) Mike Lin:
>
>  Hello Gerd, all,
>>
>> I'm trying to find the right incantations to make parallel https requests
>> from multiple threads using Http_client. (I know the library is capable of
>> multiple requests from one thread -- I have other strong reasons to use
>> multithreading.)
>>
>> https://gist.github.com/**4152047 <https://gist.github.com/4152047>
>>
>> This program generally segfaults for me at some random point during the 10
>> requests.  Compiled as in the comment in line 2 (ubuntu quantal,
>> libocamlnet-ocaml-dev 3.5.1, ocaml 3.12.1).
>>
>> Permutations attempted:
>>
>> 1) Change https to http in google URL -- works as expected
>> 2) Serialize requests by uncommenting line 35 -- works as expected
>> 3) Move invocations of Ssl.init, Ssl.create_context,
>> HTTPS.https_transport_channel_**type in and out of fresh_pipeline (which
>> runs
>> in the worker threads) -- no help
>>
>> Any suggestions? Thanks!
>>
>> Mike Lin
>>
>>
>
>
> --
> ------------------------------**------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> Creator of GODI and camlcity.org.
> Contact details:        http://www.camlcity.org/**contact.html<http://www.camlcity.org/contact.html>
> Company homepage:       http://www.gerd-stolpmann.de
> ------------------------------**------------------------------

[-- Attachment #2: Type: text/html, Size: 4427 bytes --]