From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail1-relais-roc.national.inria.fr (mail1-relais-roc.national.inria.fr [192.134.164.82]) by walapai.inria.fr (8.13.6/8.13.6) with ESMTP id p8U00sJI020561 for ; Fri, 30 Sep 2011 02:00:54 +0200 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjkBAAcGhU7RVaC2imdsb2JhbABBhGWjLwgUAQEBCgkNBxIGIYFsAg8dATkDDQUQDwImAiQSAQUBIhsaojuCWQqLCYMghRmJKQIEBoEmhFWBEgSTVo0QPYFIgkE X-IronPort-AV: E=Sophos;i="4.68,464,1312149600"; d="scan'208";a="122212838" Received: from mail-gy0-f182.google.com ([209.85.160.182]) by mail1-smtp-roc.national.inria.fr with ESMTP/TLS/RC4-SHA; 30 Sep 2011 02:00:48 +0200 Received: by gyf2 with SMTP id 2so1612656gyf.27 for ; Thu, 29 Sep 2011 17:00:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=l2qFMvvQ86gQescB0x4UBI3ilajpZQIDgLt74Etn6AI=; b=kzybZt7epqj2MD2/4eUJ3hFLmUts8Gs1DTNMVXD4xrXyEDVDDnAFh7uEyYx7b731Ym 2iR06jxCATI83Vc3NdvrsLLrUon4hxkR4YtB18np8x5r14hXElTXgSlIWd8fuYwbAsoC 9xC9DXckwoCTRJ8oHQzPdGWLe/z6TCsnalxOM= MIME-Version: 1.0 Received: by 10.42.74.195 with SMTP id x3mr1763281icj.2.1317340847391; Thu, 29 Sep 2011 17:00:47 -0700 (PDT) Sender: tcvena@gmail.com Received: by 10.42.227.138 with HTTP; Thu, 29 Sep 2011 17:00:47 -0700 (PDT) Date: Thu, 29 Sep 2011 20:00:47 -0400 X-Google-Sender-Auth: Y-scB6AUav0-w4TAvEDrdlpPXd4 Message-ID: From: Taylor Venable To: caml-list Content-Type: text/plain; charset=UTF-8 Subject: [Caml-list] Problem with un-flushed output getting mangled Hi there, I'm having a problem with some code I've written. The gist of my program (whose code I unfortunately cannot share at this time, I'll have to get approval first) is this: 1. Read elements from an XML file and turn them into objects. 2. Place these objects into hash tables. 3. Do some stuff with those objects. 4. Print them out to various files (depending on the type of the object). My trouble is this: if I fail to call flush on the output channels in step #4, I get mangled output. By mangled I mean that in the middle of one line, suddenly the data from another line appears. The other line exists elsewhere in the output. Sometimes lines are simply duplicated. I found this highly strange and thought the problem was in my code at first. But I couldn't find anything, so I decided to make a call to flush after every line I wrote to the output. Suddenly my problem disappeared! My understanding, however, is that flush shouldn't be required to do this correctly. After all, I simply open the output channel, write to it a bunch, and then finally close it. I haven't yet been able to come up with a simple case that exhibits the problem. I can't share the code with you yet, and I can't share the data either, so I'll try to give as much information as I can. A. I only call open_out, output_string, output_char, and close_out. B. Although I link to Batteries (version 1.4.1) I don't use its IO layer. I just call the functions that I need directly (e.g. BatString.join) C. There are two files that exhibit the problem. D. The problems in the output file occur in exactly the same position every time, even if the data itself changes! D1. In one file, it's position 2883585. At that location, it duplicates text from position 794139. venatc01 01 Clinton William clinwj01@some.domain 1234567 J 1600 Pennsylvania Ave Washington DC 12345 US 1 Y This is a sanitized example of what the output looks like. It's supposed to be the information for user venatc01, but suddenly in the middle of the line the information for a certain Bill Clinton is injected. The row describing Bill Clinton appears earlier in the file. This particular file is quite long, and there are several duplicate lines: position 2610356 is duplicated at 2883693, position 2435496 is duplicated at 2883819. D2. In the other file, it's position 20481. At that location, it duplicates text from either position 6434 or 10494. (You can't tell because it's the same data in both spots.) line 232: 11667.201210 venatc01 S Y Y line 378: 14900.201210 venatc01 S Y Y line 737: 1241210 venatc01 S Y Y Everything after the 124 above is copied either from line 232 or 378. E. Flushing the output of one file after every line printed fixes that one file, but does not affect the position of the problem in the other file, which remains the same. F. Adjusting the heap size using OCAMLRUNPARAM=s=4M,i=32M,o=150 had no effect. G. The problem exists both with byte compilation and native compilation. H. I'm using OCaml 3.12.1 on Linux x86_64. I've assumed that you don't need to call flush periodically to avoid problems like this, but maybe that's not the case? Should one expect any problems or difficulties if one doesn't explicitly flush every so often? If anybody has any ideas on how to debug this, I will be greatly appreciative. I don't know that much about OCaml internals and how to debug things like this. If I can provide some more information, let me know. If it will help to have the code, I'll speak with my boss. In the mean time, I'll keep trying to reproduce with a much simpler program. Thanks for any thoughts. -- Taylor C. Venable http://metasyntax.net/