[Caml-list] AW: Multithreaded https requests in ocamlnet netclient

[Gerd Stolpmann <info@gerd-stolpmann.de>]

Hi Mike,

I can confirm the problem, but I'm not able to track it down.

First of all, I'm quite sure that you cannot share SSL contexts between  
threads. But even if I fix this, I get crashes - usually early during  
SSL_connect.

I'm wondering whether ocaml-ssl initializes openssl correctly for  
multi-threading. In my version of openssl the prototype is

int CRYPTO_THREADID_set_callback(void (*threadid_func)(CRYPTO_THREADID  
*));

but in ocaml-ssl the CRYPTO_THREADID pointer is ignored (it assumes  
void here). So, maybe this is the reason. I currently don't have time  
to follow this idea.

If everything fails, there is a way to use Netclient in threaded  
applications so that only one thread is doing the networking stuff, see  
this example:

https://godirepo.camlcity.org/svn/lib-ocamlnet2/trunk/code/examples/netclient/simple/http_mt.ml

As Netclient is event-driven, the requests are processed in parallel  
(with one exception, though: DNS lookups are synchronous, but this can  
be worked around, as the DNS lookup function is settable in Netclient).

Gerd

Am 27.11.2012 03:54:16 schrieb(en) Mike Lin:
> Hello Gerd, all,
> 
> I'm trying to find the right incantations to make parallel https  
> requests
> from multiple threads using Http_client. (I know the library is  
> capable of
> multiple requests from one thread -- I have other strong reasons to  
> use
> multithreading.)
> 
> https://gist.github.com/4152047
> 
> This program generally segfaults for me at some random point during  
> the 10
> requests.  Compiled as in the comment in line 2 (ubuntu quantal,
> libocamlnet-ocaml-dev 3.5.1, ocaml 3.12.1).
> 
> Permutations attempted:
> 
> 1) Change https to http in google URL -- works as expected
> 2) Serialize requests by uncommenting line 35 -- works as expected
> 3) Move invocations of Ssl.init, Ssl.create_context,
> HTTPS.https_transport_channel_type in and out of fresh_pipeline  
> (which runs
> in the worker threads) -- no help
> 
> Any suggestions? Thanks!
> 
> Mike Lin
> 



-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------