From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <andrej.bauer@andrej.com>
X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr
X-Spam-Level: 
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=disabled 
	version=3.1.3
Received: from mail1-relais-roc.national.inria.fr (mail1-relais-roc.national.inria.fr [192.134.164.82])
	by yquem.inria.fr (Postfix) with ESMTP id 59B79BC37
	for <caml-list@yquem.inria.fr>; Fri, 12 Jun 2009 10:20:14 +0200 (CEST)
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApYCABatMUrRVdvXmGdsb2JhbACYAj8BAQEBAQgJDAcStgiECwU
X-IronPort-AV: E=Sophos;i="4.42,208,1243807200"; 
   d="scan'208";a="31087076"
Received: from mail-ew0-f215.google.com ([209.85.219.215])
  by mail1-smtp-roc.national.inria.fr with ESMTP; 12 Jun 2009 10:20:14 +0200
Received: by ewy11 with SMTP id 11so2617210ewy.9
        for <caml-list@inria.fr>; Fri, 12 Jun 2009 01:20:14 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.9.81 with SMTP id 59mr1197706wes.181.1244794813714; Fri, 
	12 Jun 2009 01:20:13 -0700 (PDT)
In-Reply-To: <4A310A5B.9010404@ens-lyon.org>
References: <7d8707de0906110557n6a1511a2k9f4f00827f954cb6@mail.gmail.com>
	 <4A310A5B.9010404@ens-lyon.org>
Date: Fri, 12 Jun 2009 10:20:13 +0200
Message-ID: <7d8707de0906120120x10cc8fe0p54adbd189003f3da@mail.gmail.com>
Subject: Re: [Caml-list] ocamllex and python-style indentation
From: Andrej Bauer <andrej.bauer@andrej.com>
To: caml-list@inria.fr
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Spam: no; 0.00; ocamllex:01 andrej:01 andrej:01 inserting:01 tokens:01 separators:01 separators:01 preprocess:01 ocaml:01 lexer:01 lexeme:01 buf:01 token:01 andreas:01 caml-list:01 

Thanks to Andreas, I'll have a look at the "old" code.

I think I understand the general idea of inserting "virtual" tokens,
but the details confuse me still. So starting with

> if True:
>     x = 3
>     y = (2 +
>       4 + 5)
> else:
>     x = 5
>     if False:
>         x = 8
>         z = 2

Martin suggests the following:

> {
> if True:
> ;
>    {
>    x = 3
>    ;
>    y = (2 +
>    ;
>      {
>      4 + 5)
>      }
>    }
> ;
> else:
> ;
>    {
>    x = 5
>    ;
>    if False:
>    ;
>        {
>        x = 8
>        ;
>        z = 2
>        }
>    }
> }

I have two questions. Notice that the { ... } and ( ... ) need not be
correctly nested (in the top half), so how are we going to deal with
this? The second question is, why are there the separators after and
just before "else:". I would expect separators inside { .... }, but
not around "else".

Presumably the intermediate stage that I would preprocess the token
stream would have to know about indentation levels. I have not tried
this, but ocaml lexer will correctly match things like

| '\n' [' ' '\t']* -> { INDENTATION (compute_indentation (lexeme buf)) }

Yes?

With kind regards,

Andrej