Re: [Caml-list] Unicode support?

[Gerd Stolpmann <info@gerd-stolpmann.de>]

On Wed, 12 Sep 2001, you wrote:
>Hi!
>
>I'm very interesed to use Caml for educational purpose
>in multibyte environment such as CJK.
>So I'd like to know whether Caml supoprt Unicode or not?
>(Unicode (ISO 10646) is the character coding (or character set) standard.)
>Many people forsee that Unicode is going to replace all older lagcy
>encodings in a few years
>and many languages such as C, Java, Perl, Tcl, C# and Python support it
>already.

People have also forseen that PL/I replaces Cobol. So take that with care.

>If Caml doesn't support unicode, could you give me any clue
>from which part in caml source I should start in order to support unicode?

In the mailing list archives you find a long debate, beginning here and never
ended:

http://caml.inria.fr/archives/199910/msg00132.html

Further threads:

http://caml.inria.fr/archives/199910/msg00189.html
http://caml.inria.fr/archives/200101/msg00006.html

>More question,
>Which form is better for processing unicode in Caml, UTF-8 or UCS-2 (or
>UCS-4)?

There is currently not very much unicode support in caml. But there are some
software packages and projects aiming at improving this (besides other goals).
For my XML parser "PXP" I needed some unicode functionality, and that was the
reason to become active regarding this issue.

Caml strings can contain arbitrary byte sequences, and the null byte is
explicitly allowed. So this would permit all of UTF-8, UCS-2, and UCS-4. For
PXP, I needed especially a lexical analyzer that supports unicode, and that
lead to two solutions:

- PXP itself contains a tool that converts .mll files containing unicode
  characters into their UTF-8 representation. The idea is that UTF-8 regular
  expressions can be handled by every regular expression engine.

  This approach worked for the limited usage of unicode in the XML definition,
  but the resulting lexers are huge.

  This tool was a contribution by Claudio Sacerdoti Coen.

  Link: http://www.ocaml-programming.de/packages/documentation/pxp
  (look into the directory tools/src/ucs2_to_utf8)

- Alain Frisch developed a patch for the ocamllex tool: wlex. He first used it
  for his XPath package, but it is now also used in PXP. wlex reads an
  arbitrary encoding, maps this encoding to character classes, and allows it to
  form regular expressions using these classes.

  The lexers are usually much smaller, but you need to patch ocamllex.

  Link: http://www.eleves.ens.fr:8080/home/frisch/soft

Another problem is to convert between various character encodings. The
netstring package contains a routine that can map between UTF-8, UCS-2, UCS-4,
and many 8-bit encodings (but no other multibyte encodings).

Link: http://www.ocaml-programming.de/packages/documentation/netstring/

The netstring development is now done by the ocamlnet project
(http://ocamlnet.sourceforge.net).

I hope this helps,

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 45             
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany                     
----------------------------------------------------------------------------
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr