Testing lexers and parsers

Testing lexers and parsers

[Joel Reymont]

Folks,

I'm trying to incrementally develop a translator using a test-driven  
approach. I figure I can write a bunch of tests for small chunks of  
functionality to structurally test expected AST against produced.

Once I get to testing the parsing of whole files I think I can print  
the AST I produce into a file and test this against a target file  
with the correct printed AST. I can probably run 'cmp' to make sure  
my files match. This would not be a structural comparison, though,  
and would be highly dependent on white space which does not strike me  
as ideal.

Is there a better way to accomplish this while still using structural  
comparison of ASTs?

Should I try to read my target AST into OCaml, for example? Should I  
create a bunch of "target AST" modules with functions that return my  
target AST?

How does everyone else approach testing of lexers and parsers?

	Thanks, Joel

--
http://wagerlabs.com/

Re: [Caml-list] Testing lexers and parsers

[Denis Bueno]

On 2/27/07, Joel Reymont <joelr1@gmail.com> wrote:
> Is there a better way to accomplish this while still using structural
> comparison of ASTs?
>
> Should I try to read my target AST into OCaml, for example? Should I
> create a bunch of "target AST" modules with functions that return my
> target AST?

I recently had to deal with this problem. I ended up testing the
parsing/printing by parsing a file, printing it out, and parsing the
print-out, and then making sure the ASTs built up are structurally
identical (modulo any position fields you might have lurking in your
ASTs).

I have to say, ever since I did this testing, I have had no problems
with my parser.

-Denis

Re: [Caml-list] Testing lexers and parsers

[Joel Reymont]

On Feb 27, 2007, at 3:42 PM, Denis Bueno wrote:

> I recently had to deal with this problem. I ended up testing the
> parsing/printing by parsing a file, printing it out, and parsing the
> print-out,

Are you printing out the AST to make sure you can reproduce the  
original file you parsed?

	Thanks, Joel

--
http://wagerlabs.com/

Re: [Caml-list] Testing lexers and parsers

[Denis Bueno]

[Meant to send to the list the first time. Sorry for the dupe.]

On 2/27/07, Denis Bueno <dbueno@gmail.com> wrote:
> On 2/27/07, Joel Reymont <joelr1@gmail.com> wrote:
> > Are you printing out the AST to make sure you can reproduce the
> > original file you parsed?

Yes. I wanted to make sure my AST-printing functions would print
correct source code in the source language. This makes it very easy to
inspect AST fragments.

-Denis

Re: [Caml-list] Testing lexers and parsers

[konrad]

I actually attended a project in university on writing a C compiler (written in Java) and we printed out XML code for debugging, that represented the AST. That was a real benefit cause you can still see the structure of the AST in the XML tree. It is even easy to do that - one tree traversal is enough.

chris

On Tue, 27 Feb 2007 11:11:09 -0500, "Denis Bueno" <dbueno@gmail.com> wrote:
> [Meant to send to the list the first time. Sorry for the dupe.]
> 
> On 2/27/07, Denis Bueno <dbueno@gmail.com> wrote:
>> On 2/27/07, Joel Reymont <joelr1@gmail.com> wrote:
>> > Are you printing out the AST to make sure you can reproduce the
>> > original file you parsed?
> 
> Yes. I wanted to make sure my AST-printing functions would print
> correct source code in the source language. This makes it very easy to
> inspect AST fragments.
> 
> -Denis
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs