From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by yquem.inria.fr (Postfix) with ESMTP id 0200ABC57 for ; Fri, 17 Dec 2010 01:36:36 +0100 (CET) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArUBALc/Ck3ZSMDjk2dsb2JhbACkPxUBAQEBCQkKCREDIcMrhUoEjhQ X-IronPort-AV: E=Sophos;i="4.59,358,1288566000"; d="scan'208";a="84189577" Received: from fmmailgate02.web.de ([217.72.192.227]) by mail2-smtp-roc.national.inria.fr with ESMTP; 17 Dec 2010 01:36:36 +0100 Received: from smtp01.web.de ( [172.20.0.243]) by fmmailgate02.web.de (Postfix) with ESMTP id 2C13C18E949D4; Fri, 17 Dec 2010 01:36:36 +0100 (CET) Received: from [78.43.204.177] (helo=frosties.localdomain) by smtp01.web.de with asmtp (TLSv1:AES256-SHA:256) (WEB.DE 4.110 #24) id 1PTOJI-000057-00; Fri, 17 Dec 2010 01:36:36 +0100 Received: from mrvn by frosties.localdomain with local (Exim 4.72) (envelope-from ) id 1PTOJH-0000gT-Kp; Fri, 17 Dec 2010 01:36:35 +0100 From: Goswin von Brederlow To: "Richard W.M. Jones" Cc: Joel Reymont , caml-list@yquem.inria.fr Subject: Re: [Caml-list] appending data to a mmap-ed file References: <8D1E9D5B-188C-4AF3-980B-EF229BA98FB4@gmail.com> <20101216171617.GB3647@annexia.org> Date: Fri, 17 Dec 2010 01:36:35 +0100 In-Reply-To: <20101216171617.GB3647@annexia.org> (Richard W. M. Jones's message of "Thu, 16 Dec 2010 17:16:17 +0000") Message-ID: <87aak5jkvg.fsf@frosties.localnet> User-Agent: Gnus/5.110009 (No Gnus v0.9) XEmacs/21.4.22 (linux, no MULE) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: goswin-v-b@web.de X-Sender: goswin-v-b@web.de X-Provags-ID: V01U2FsdGVkX1+OzG4cZVBpL5H5lz5Flom69vxyp/+7FEGbXlrU RvxWRsVqyHotea26Ya4GIOnKz2J+SLvEj5+uOP1q0wFzGulhem PNRQjCCAo= X-Spam: no; 0.00; appending:01 appending:01 allocating:01 mfg:98 wrote:01 ints:01 caml-list:01 writes:01 appended:02 data:02 data:02 append:02 structures:02 parse:02 string:02 "Richard W.M. Jones" writes: > On Thu, Dec 16, 2010 at 11:31:16AM +0000, Joel Reymont wrote: >> I'm constantly appending to a file of stock quotes (ints, longs, >> doubles, etc.). I have this file mapped into memory with mmap. >> >> What's the most efficient way to make newly appended data available >> as part of the memory mapping? > > Unfortunately it's hard to reliably extend an mmap'd area. The reason > is not that you can't do it, but that you might overrun another memory > mapping after it, where that other mapping could be something > important like your program or a shared library. The other mapping > might not even be present at the time you initially map your file, but > might appear as the result of an innocuous operation such as printing > a string or allocating memory. > > Now you can, with a bunch of work, avoid this: parse /proc/self/maps, > select a suitable base address for your mapping, move the mapping if > it gets too large for the selected area or if another library is > mapped in above it, etc. but this quickly gets very difficult. > > I would suggest a simpler way to solve your problem is simply to open > the data file and append to it. If you need to reference the values, > keep them in memory structures. > > Rich. Or avoid the whole issue and make the file large enough to begin with. Thanks to sparse files you can create a huge file that only uses 1 block on disk. Then you can mmap that and it will use up more disk space as you fill in data automatically. MfG Goswin