ocamlnet mini tutorial: netclient and cookie based authentication

[Hendrik Tews <tews@cs.ru.nl>]

Hi,

for ocamlnet novices I document here what you need to access web
sites that use cookies for authentication (eg, wikis or mailman).
With this you can let ocaml perform your mailing-list
administrator tasks or write a wiki bot.

Cookie authentication works as follows: When you first visit the
site you have to authenticate with username and password that are
usually sent (in cleartext) in a post request to the server. The
server responses with a cookie that will be used for
authentication in all following requests. 

Because you have to retrieve and set cookies in the headers you
cannot use the Http_client.Convenience module. You really have to
dive into ocamlnet.

Here comes the solution:

Netstring extension:

- You need get_set_cookie, which will retrieve cookies from a
  response header but which is not in ocamlnets netstring library
  currently. Download from [1] or with two additional helper
  functions from [2].


Prerequisites:

- Create a pipeline with "new Http_client.pipeline", it will
  process all http requests. (Don't ask me about timeouts and
  retries, the defaults work fine for me.)


Get going (download some page without authentication):

- Create a get call "new Http_client.get url"

- Process the call: Add it to the pipeline (method #add) and run
  the pipeline (#run). By this the call will be changed and
  contain the response. 

- Check the response status (#response_status), which should
  usually be `OK and get the page (#response_body#value).

- In case of error convert the status into something readable
  with Nethttp.string_of_http_status or consult the response
  status message (#response_status_text).


Login:

- Create post call with 
  "new Http_client.post url parameters", the parameters are a
  string * string list, eq 
    [("wpName", "mywikiname");
     ("wpPassword", "mywikipasswd");
     ("wpLoginattempt", "Log in")]
  for a MediaWiki.

  [In order to get the parameters save the login page, change the
  method of the right from into "GET", load the page into a
  browser, press the login button and parse the new URL you get.]

- Process the call.

- Check the status (#response_status): For MediaWiki it should be
  `Found, for mailman and others `Ok (don't ask me why).

- Retrieve the response header (#response_header) and extract the
  cookies with get_set_cookies from [2].


Make an authenticated request:

- Create your call "new get url" for getting a page or 
  "new post url" for performing an action.

- Access the request header (#request_header `Base) and add your
  cookies there (set_cookies from [2]). This is all done via
  side-effects, no need to write the modified header back into
  the call.

- Process the call and get the response.


Compilation:

- with findlib and -package netclient -linkpkg:

  ocamlfind {ocamlc,ocamlopt} -package netclient -linkpkg your files ...


Acknowledgements:

- Praise Gerd Stolpmann for ocamlnet and findlib.



That's it. For an example go to [3], which contains a MediaWiki
bot that updates structural information in the Mozilla wiki [4].
The needed utility functions are in wiki_http: wiki_login to
login, download to download a web page, get_page to get a wiki page
and write_wiki_page to write one.


Resources:

[1] https://godirepo.camlcity.org/wwwsvn/trunk/code/get-set-cookie.ml?rev=1145&root=lib-ocamlnet2&view=auto

[2] http://www.sos.cs.ru.nl/cgi-bin/~tews/olmar/viewvc-patch.cgi/elsa/olmar/wiki_bot/netstring_ext.ml?revision=HEAD&view=markup

[3] http://www.sos.cs.ru.nl/cgi-bin/~tews/olmar/viewvc-patch.cgi/elsa/olmar/wiki_bot/

[4] http://wiki.mozilla.org/Elsa_ast_nodes


Bye,

Hendrik