* Ask for a more efficient way to deallocate memory (full version) @ 2007-12-09 21:39 Fabrice.Pardo 2007-12-09 21:55 ` [Caml-list] " Olivier Andrieu ` (3 more replies) 0 siblings, 4 replies; 19+ messages in thread From: Fabrice.Pardo @ 2007-12-09 21:39 UTC (permalink / raw) To: caml-list Hello, and sorry for my previous unfinished message. As the function Unix.opendir returns a value, we can expect that Unix.closedir will be automatically called when the dir_handle value is no more referenced. That's not the case, and calling opendir too many times as in for i = 1 to 2000 do let d = Unix.opendir "/tmp" in () done;; raises Exception: Unix.Unix_error (Unix.EMFILE, "opendir", "/tmp"). Replacing Unix.opendir by safe_opendir: let safe_opendir path = let _ = Gc.major () and d = Unix.opendir path in Gc.finalise Unix.closedir d seems a solution, but the efficiency is very low due to high cost of garbage collector calling function. My question is how to write a code as efficient as it would be possible using a reference-counted language. Thanks. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-09 21:39 Ask for a more efficient way to deallocate memory (full version) Fabrice.Pardo @ 2007-12-09 21:55 ` Olivier Andrieu 2007-12-10 11:25 ` Fabrice Pardo 2007-12-09 21:57 ` Oliver Bandel ` (2 subsequent siblings) 3 siblings, 1 reply; 19+ messages in thread From: Olivier Andrieu @ 2007-12-09 21:55 UTC (permalink / raw) To: Fabrice.Pardo; +Cc: caml-list On Dec 9, 2007 10:39 PM, <Fabrice.Pardo@lpn.cnrs.fr> wrote: > Hello, and sorry for my previous unfinished message. > > As the function Unix.opendir returns a value, > we can expect that Unix.closedir will be automatically > called when the dir_handle value is no more referenced. Well, not really: a GC manages memory, not OS resources like file descriptors. I wouldn't recommend relying on the GC to close those kind of resources. Write your code like this: let with_dir fname f = let d = Unix.opendir f in try let r = f d in Unix.closedir d ; r with exn -> Unix.closedir d ; raise exn once the "f" function returns, you directory handle is closed. -- Olivier ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-09 21:55 ` [Caml-list] " Olivier Andrieu @ 2007-12-10 11:25 ` Fabrice Pardo 2007-12-10 12:03 ` Michaël Le Barbier 2007-12-10 16:33 ` Oliver Bandel 0 siblings, 2 replies; 19+ messages in thread From: Fabrice Pardo @ 2007-12-10 11:25 UTC (permalink / raw) To: caml-list Olivier Andrieu wrote: > > Well, not really: a GC manages memory, not OS resources like file descriptors. > I wouldn't recommend relying on the GC to close those kind of resources. > > Write your code like this: > > let with_dir fname f = > let d = Unix.opendir f in > try let r = f d in Unix.closedir d ; r > with exn -> Unix.closedir d ; raise exn > > once the "f" function returns, you directory handle is closed. > > Thanks, this solution is clear and compact. I think that all functions allocating resources out of the OCaml heap should be documented as it, warning the programmer that he should not be lazy, relying on automatic deallocation of resources. The fact that Unix.opendir should be better documented in that sense is marginal for me. It was only an example of external C-allocated resource. When using these kind of external C functions, OCaml seems then less comfortable to the programmer than reference counted languages. The price to pay is not too high as functional code permit to encapsulate the problem as shown by Olivier. I have a new question now: What is the drawback if we keep hidden unsecure external functions (allocating out of the heap resources, as Unix.opendir), only publishing secure functions as "with_dir" ? Of course, I mean other drawback than changing the API. -- Fabrice ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 11:25 ` Fabrice Pardo @ 2007-12-10 12:03 ` Michaël Le Barbier 2007-12-10 16:33 ` Oliver Bandel 1 sibling, 0 replies; 19+ messages in thread From: Michaël Le Barbier @ 2007-12-10 12:03 UTC (permalink / raw) To: Fabrice Pardo; +Cc: caml-list Fabrice Pardo <Fabrice.Pardo@Lpn.cnrs.fr> writes: > I have a new question now: > What is the drawback if we keep hidden > unsecure external functions (allocating out of the heap > resources, as Unix.opendir), > only publishing secure functions as "with_dir" ? > Of course, I mean other drawback than changing the API. A drawback is that careful management of these ressources is no more possible. If the choices made by module designer to ensure security do not fit well your problem, you are left with no option other than writing poorly behaving applications. In contrast, given a module that delegates ressource management to its users, it is fairly trivial to wrap modules's functionnalitites into a module that automatizes ressource management in a way that will fit many uses. One would write an `EasyLazyUnix', `CookedUnix' or `UnixChest' module with automatic deallocation of ressources. -- Cordialement, Michaël LB ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 11:25 ` Fabrice Pardo 2007-12-10 12:03 ` Michaël Le Barbier @ 2007-12-10 16:33 ` Oliver Bandel 2007-12-10 20:27 ` Richard Jones 1 sibling, 1 reply; 19+ messages in thread From: Oliver Bandel @ 2007-12-10 16:33 UTC (permalink / raw) To: caml-list Zitat von Fabrice Pardo <Fabrice.Pardo@Lpn.cnrs.fr>: [...] > When using these kind of external C functions, > OCaml seems then less comfortable to the programmer > than reference counted languages. I doubt that reference-count is the reason here. perl also uses reference count, but Filehandles and Dirhandles have to be closed with close / closedir. So, a language that uses reference counting does not necessarily handle file-/dir-handles. There are different things you mix up. BTW: The Unix-module it's name is "Unix"-module, and it is not part of the so-called OCaml standard-lib. And because it is used to access the Unix-API (hence it's name), it should behave like a OCaml-binding to the Unix-API. And this means: you have a opendir, a ´readdir and a closedir function. And they behave like the Unix-API, as mentioned above. When you come from C, use Unix-functions, and then go to OCaml, you have the aequivalents there too. If you want "safer" functions, write them (or use already existing wrapper-libs, if they are there). But the need to have access to the Unix-API is there, because it's system-near programming, and too much High-level programmign around it, is not what one needs, when using the system's API. The discussion I mentioned in one of my ealrier mails has brought in some arguments to the reason, why it is good, not to make all the deallocation-things automatically. I think one reason is overhead. But one reason more was to have a separation of system-level and non-system level, but on that point I'm quite unsure to get the arguments right from my memory. If I have some time, I can look for the thread. Possibly someone else has the link at hand. [...] > > I have a new question now: > What is the drawback if we keep hidden > unsecure external functions Unsecure? No coredump, so it's secure. If you need more and more and more RAM in your application and your applications ays to you: "Sorry, all memory ehausted!", is this an unsecure application? Unsecure for me would be if the program throws an exception that can't be caught ans is from a different level of programming. If OCaml gives you a Unix-exception, when you wasted all your filedescriptors, this IMHO is not unsecure. Unsecure in this case would be, if you get a segmentation fault/ coredump, instead a clean OCaml exception. > (allocating out of the heap > resources, as Unix.opendir), > only publishing secure functions as "with_dir" ? The Unix-module has the reason, that it gives you access to the system's API. And the systems API is the systems API is the systems API. If you need access to it, use Unix-module. If you use a higher-level library, and do not have access to the system itself, you get more and more bloat. I would think, this is one of the problems, Java-programmers have... at least I remember people who had problems with their super-super highleve network-API (including automatically encrypted network connections) and nothing worked because of that bloat. What a good cure would it have been, to open a socket. ;-) Ciao, Oliver P.S.: Imagine a programming language that handles all the things you want to have in the way you look for it... then people would say: Well, what a bad language, that does not give me clear access to the system! => Then one possibly would have a problem to write system-near drivers, where you need direct, clear access to the system, without many layers between your language and the system! Each thing has two (or more) sides. For me the things that OCaml offers are the right things. OCaml is not perfect, but very very good. And as I have done some Unix-programming, I find it not only ok, I find it necessary to have clear access to the system, as I have also in C. And I don't want to switch back to C, only because I want to have access to the Unix-API! (And there is no limitation in writing coe around that unix-module. So: one can add wrapping code around, but you can't scratch code off, if it is always there and bloats your libs! It's not like in color-systems: you can use additive and subtractive ways of mixing colors, but normally you only can add code, not throw it away if it's built-in! ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 16:33 ` Oliver Bandel @ 2007-12-10 20:27 ` Richard Jones 2007-12-10 21:05 ` Oliver Bandel 2007-12-10 21:15 ` Gordon Henriksen 0 siblings, 2 replies; 19+ messages in thread From: Richard Jones @ 2007-12-10 20:27 UTC (permalink / raw) To: Oliver Bandel; +Cc: caml-list On Mon, Dec 10, 2007 at 05:33:21PM +0100, Oliver Bandel wrote: > Zitat von Fabrice Pardo <Fabrice.Pardo@Lpn.cnrs.fr>: > > [...] > > When using these kind of external C functions, > > OCaml seems then less comfortable to the programmer > > than reference counted languages. > > I doubt that reference-count is the reason here. > perl also uses reference count, but Filehandles > and Dirhandles have to be closed with close / closedir. This isn't true. In Perl file handles are closed at the end of a scope if they are no longer used. In other words a Perl-equivalent to this loop will never use more than a single file descriptor: while (true) { open "foo" } Even in a GC'd language which could finalize the file handle you could never be sure when the GC is going to be called so it could use an indefinite number of file descriptors. There is a really good paper on this subject -- how ref counting and garbage collection are orthogonal language concepts -- but I can't find it right now. Rich. -- Richard Jones Red Hat ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 20:27 ` Richard Jones @ 2007-12-10 21:05 ` Oliver Bandel 2007-12-10 21:15 ` Gordon Henriksen 1 sibling, 0 replies; 19+ messages in thread From: Oliver Bandel @ 2007-12-10 21:05 UTC (permalink / raw) To: caml-list Zitat von Richard Jones <rich@annexia.org>: > On Mon, Dec 10, 2007 at 05:33:21PM +0100, Oliver Bandel wrote: > > Zitat von Fabrice Pardo <Fabrice.Pardo@Lpn.cnrs.fr>: > > > > [...] > > > When using these kind of external C functions, > > > OCaml seems then less comfortable to the programmer > > > than reference counted languages. > > > > I doubt that reference-count is the reason here. > > perl also uses reference count, but Filehandles > > and Dirhandles have to be closed with close / closedir. > > This isn't true. In Perl file handles are closed at the end of a > scope if they are no longer used. In other words a Perl-equivalent > to > this loop will never use more than a single file descriptor: > > while (true) { > open "foo" > } Oh, so I was wrong here. :( Perl handles Filehandles in a special way (not as other things in perl), and I thought it also handles Filehandles different, in respect to reference counting. Well, some details I've forgotten during the time I rather used OCaml instead of Perl ;-( Thanks for the correction. [...] > There is a really good paper on this subject -- how ref counting and > garbage collection are orthogonal language concepts [...] That was, what I wanted to express, but my example was wrong. Ciao, Oliver ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 20:27 ` Richard Jones 2007-12-10 21:05 ` Oliver Bandel @ 2007-12-10 21:15 ` Gordon Henriksen 2007-12-10 22:13 ` Oliver Bandel 2007-12-10 23:24 ` Jon Harrop 1 sibling, 2 replies; 19+ messages in thread From: Gordon Henriksen @ 2007-12-10 21:15 UTC (permalink / raw) To: caml-list [-- Attachment #1: Type: text/plain, Size: 2606 bytes --] On Dec 10, 2007, at 15:27, Richard Jones wrote: > On Mon, Dec 10, 2007 at 05:33:21PM +0100, Oliver Bandel wrote: > >> Zitat von Fabrice Pardo <Fabrice.Pardo@Lpn.cnrs.fr>: >> >>> When using these kind of external C functions, OCaml seems then >>> less comfortable to the programmer than reference counted languages. >> >> I doubt that reference-count is the reason here. perl also uses >> reference count, but Filehandles and Dirhandles have to be closed >> with close / closedir. > > This isn't true. In Perl file handles are closed at the end of a > scope if they are no longer used. In other words a Perl-equivalent > to this loop will never use more than a single file descriptor: > > while (true) { > open "foo" > } > > Even in a GC'd language which could finalize the file handle you > could never be sure when the GC is going to be called so it could > use an indefinite number of file descriptors. > > There is a really good paper on this subject -- how ref counting and > garbage collection are orthogonal language concepts -- but I can't > find it right now. True. To address the original point, however: This is a significant problem that crops up frequently in language/ runtime designs, especially when migrating to static, garbage collected runtimes from scripting runtimes. parrot tackled this problem; they call it timely collection and went far out of their way to implement it, performing miniature collections when a resource variable goes out of scope. This is rather expensive. The practical reality is that, short of re-engineering memory management, developers migrating from C++ (RTTI) and scripting languages may need to retrain themselves to recognize that: finalization does not provide a time bound on destruction garbage collection manages memory and no other resources C# provides a convenient using construct to ease the pain, but Java code is littered with finally blocks for this reason. Ocaml is hampered in that its try block has no native finally construct, which requires repetition in the normal flow-of-control and exceptional cases. Implementing reusable using constructs with filters and closures for each type of resource is the best option I've seen, although a generic finally construct is also possible also using closures: let finally expr cleanup = try let result = expr () in cleanup (); result with x -> cleanup (); raise x Unfortunately, this is not at all attractive at the call site. — Gordon [-- Attachment #2: Type: text/html, Size: 10331 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 21:15 ` Gordon Henriksen @ 2007-12-10 22:13 ` Oliver Bandel 2007-12-10 22:59 ` Jon Harrop 2007-12-10 23:24 ` Jon Harrop 1 sibling, 1 reply; 19+ messages in thread From: Oliver Bandel @ 2007-12-10 22:13 UTC (permalink / raw) To: caml-list Zitat von Gordon Henriksen <gordonhenriksen@mac.com>: [...] > Ocaml is hampered in that its try block has no native finally > construct, which requires repetition in the normal flow-of-control > and > exceptional cases. [...] I have not looked at it in detail and did not used it, but there is an implementation of a language extension by Martin Jambon, that offers a try...finally. Because I don't know how it's implemented, I don't know if this adds a performance lack. But possibly it would make sense to look at it closer to explore it's advantages and disadvantages... ...but I've no experience in OCaml extension, so I hope someone here can say some wprds about it. Look here: http://martin.jambon.free.fr/ocaml.html Look for "TryFinally". Ciao, Oliver ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 22:13 ` Oliver Bandel @ 2007-12-10 22:59 ` Jon Harrop 2007-12-10 23:29 ` Jon Harrop ` (2 more replies) 0 siblings, 3 replies; 19+ messages in thread From: Jon Harrop @ 2007-12-10 22:59 UTC (permalink / raw) To: caml-list On Monday 10 December 2007 22:13, Oliver Bandel wrote: > Because I don't know how it's implemented, I don't know if this adds a > performance lack. But possibly it would make sense to look at it closer > to explore it's advantages and disadvantages... > ...but I've no experience in OCaml extension, so I hope someone here can > say some wprds about it. OCaml has a great macro system called camlp4 that lets you add new syntactic constructs and rewrite OCaml code at will. So Martin's code adds a new syntactic construct and automatically injects the code for the compiler that we must currently write by hand. Consequently, its performance will be superb: as good as hand-written OCaml code. However, using any camlp4 macros requires using the camlp4 replacement for the front-end of the compiler. That uses a different parsing technology (top-down recursive descent LL rather than bottom-up LALR) so the error messages from the compiler are completely different. -- Dr Jon D Harrop, Flying Frog Consultancy Ltd. http://www.ffconsultancy.com/products/?e ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 22:59 ` Jon Harrop @ 2007-12-10 23:29 ` Jon Harrop 2007-12-11 2:03 ` Yitzhak Mandelbaum 2007-12-15 21:33 ` Oliver Bandel 2 siblings, 0 replies; 19+ messages in thread From: Jon Harrop @ 2007-12-10 23:29 UTC (permalink / raw) To: caml-list On Monday 10 December 2007 22:59, Jon Harrop wrote: > On Monday 10 December 2007 22:13, Oliver Bandel wrote: > > Because I don't know how it's implemented, I don't know if this adds a > > performance lack. But possibly it would make sense to look at it closer > > to explore it's advantages and disadvantages... > > ...but I've no experience in OCaml extension, so I hope someone here can > > say some wprds about it. > > OCaml has a great macro system called camlp4 that lets you add new > syntactic constructs and rewrite OCaml code at will. So Martin's code adds > a new syntactic construct and automatically injects the code for the > compiler that we must currently write by hand. Consequently, its > performance will be superb: as good as hand-written OCaml code. > > However, using any camlp4 macros requires using the camlp4 replacement for > the front-end of the compiler. That uses a different parsing technology > (top-down recursive descent LL rather than bottom-up LALR) so the error > messages from the compiler are completely different. Incidentally, the single biggest disadvantage of adding syntactic constructs like this using macros is that your code will no longer autoindent properly. -- Dr Jon D Harrop, Flying Frog Consultancy Ltd. http://www.ffconsultancy.com/products/?e ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 22:59 ` Jon Harrop 2007-12-10 23:29 ` Jon Harrop @ 2007-12-11 2:03 ` Yitzhak Mandelbaum 2007-12-15 21:33 ` Oliver Bandel 2 siblings, 0 replies; 19+ messages in thread From: Yitzhak Mandelbaum @ 2007-12-11 2:03 UTC (permalink / raw) To: caml-list [-- Attachment #1: Type: text/plain, Size: 706 bytes --] On Dec 10, 2007, at 5:59 PM, Jon Harrop wrote: > > However, using any camlp4 macros requires using the camlp4 > replacement for the > front-end of the compiler. That uses a different parsing technology > (top-down > recursive descent LL rather than bottom-up LALR) so the error > messages from > the compiler are completely different. Just to clarify : the error messages from the *parser* are completely different. Everything after the parser is the same with or without camlp4. Most importantly, you don't have to learn new type-checker messages. Yitzhak -------------------------------------------------- Yitzhak Mandelbaum AT&T Labs - Research http://www.research.att.com/~yitzhak [-- Attachment #2: Type: text/html, Size: 3755 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 22:59 ` Jon Harrop 2007-12-10 23:29 ` Jon Harrop 2007-12-11 2:03 ` Yitzhak Mandelbaum @ 2007-12-15 21:33 ` Oliver Bandel 2007-12-16 15:14 ` Jon Harrop 2 siblings, 1 reply; 19+ messages in thread From: Oliver Bandel @ 2007-12-15 21:33 UTC (permalink / raw) To: caml-list Zitat von Jon Harrop <jon@ffconsultancy.com>: > On Monday 10 December 2007 22:13, Oliver Bandel wrote: > > Because I don't know how it's implemented, I don't know if this > adds a > > performance lack. But possibly it would make sense to look at it > closer > > to explore it's advantages and disadvantages... > > ...but I've no experience in OCaml extension, so I hope someone > here can > > say some wprds about it. > > OCaml has a great macro system called camlp4 that lets you add new > syntactic > constructs and rewrite OCaml code at will. Yes, so far I know it. > So Martin's code adds a > new > syntactic construct and automatically injects the code for the > compiler that > we must currently write by hand. Consequently, its performance will > be > superb: as good as hand-written OCaml code. So, is it just a kind of pre-compilation? The core-OCaml compiler will be the same? Is the TryFinally-code slight enough, so that it's behaving like handwritten OCaml? Did you look at the TryFinally in detail? > > However, using any camlp4 macros requires using the camlp4 > replacement for the > front-end of the compiler. I have not looked at the OCaml-compiler in detail, but from what you wrote I would assume, there are at least two parts. > That uses a different parsing technology > (top-down > recursive descent LL rather than bottom-up LALR) so the error > messages from > the compiler are completely different. Oh, well.... that's new to me. I had not needed camlp4 so far. And I don't know if it makes sense to look at it now, because there will be (or already is?) a replacement (from 3.10 on?). I hope the camlp4-tutorial will be updated to the camlp4-replacement. Possibly later I might use it. Then it would be good to have introductional material. Ciao, Oliver ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-15 21:33 ` Oliver Bandel @ 2007-12-16 15:14 ` Jon Harrop 0 siblings, 0 replies; 19+ messages in thread From: Jon Harrop @ 2007-12-16 15:14 UTC (permalink / raw) To: caml-list On Saturday 15 December 2007 21:33, Oliver Bandel wrote: > Zitat von Jon Harrop <jon@ffconsultancy.com>: > > So Martin's code adds a > > new > > syntactic construct and automatically injects the code for the > > compiler that > > we must currently write by hand. Consequently, its performance will > > be > > superb: as good as hand-written OCaml code. > > So, is it just a kind of pre-compilation? You could say that, yes. It is pre-compiling code written in your own syntax into ordinary OCaml code. > The core-OCaml compiler will be the same? Yes. > Is the TryFinally-code slight enough, so that > it's behaving like handwritten OCaml? Exactly, yes. If you're worried about performance then there are cases where you can actually use camlp4 macros to improve performance. > Did you look at the TryFinally in detail? The article I wrote for the OCaml Journal actually walks through the creation of a camlp4 macro that augments OCaml's syntax with a try..finally construct. > > However, using any camlp4 macros requires using the camlp4 > > replacement for the > > front-end of the compiler. > > I have not looked at the OCaml-compiler in detail, > but from what you wrote I would assume, there > are at least two parts. You don't really see that there are two parts to the compiler. The only thing that intrudes on the programmer (besides different error messages) is the new compilation line. > I had not needed camlp4 so far. And I don't know if it makes sense to > look at it now, because there will be (or already is?) a replacement > (from 3.10 on?). I've only been talking about the new implementation in OCaml 3.10 and beyond. There was a previous implementation as well (now renamed camlp5). > I hope the camlp4-tutorial will be updated to the camlp4-replacement. > Possibly later I might use it. Then it would be good to have > introductional material. Yes. There is certainly a shortage of information on this subject. Eventually the camlp5 will be phased out but I can't see that happening until its new replacement camlp4 is properly documented. -- Dr Jon D Harrop, Flying Frog Consultancy Ltd. http://www.ffconsultancy.com/products/?e ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-10 21:15 ` Gordon Henriksen 2007-12-10 22:13 ` Oliver Bandel @ 2007-12-10 23:24 ` Jon Harrop 1 sibling, 0 replies; 19+ messages in thread From: Jon Harrop @ 2007-12-10 23:24 UTC (permalink / raw) To: caml-list On Monday 10 December 2007 21:15, Gordon Henriksen wrote: > let finally expr cleanup = > try > let result = expr () in > cleanup (); > result > with x -> > cleanup (); > raise x I was asking Pierre Weis about getting "try finally" into OCaml recently. I think the best work-around is to use a combinator, i.e.: let try_finally x f g = ... where "x" gets passed to "f" and "g", so you typically do: try_finally (open_in file) input_line close_in which looks nicer. However, I still think this construct (and others) should be added to the language. -- Dr Jon D Harrop, Flying Frog Consultancy Ltd. http://www.ffconsultancy.com/products/?e ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-09 21:39 Ask for a more efficient way to deallocate memory (full version) Fabrice.Pardo 2007-12-09 21:55 ` [Caml-list] " Olivier Andrieu @ 2007-12-09 21:57 ` Oliver Bandel 2007-12-09 22:12 ` Jon Harrop 2007-12-09 22:16 ` Oliver Bandel 3 siblings, 0 replies; 19+ messages in thread From: Oliver Bandel @ 2007-12-09 21:57 UTC (permalink / raw) To: caml-list Hello, Zitat von Fabrice.Pardo@lpn.cnrs.fr: > Hello, and sorry for my previous unfinished message. > > As the function Unix.opendir returns a value, > we can expect that Unix.closedir will be automatically > called when the dir_handle value is no more referenced. > That's not the case, Yes, and that's a operating system issue, not OCaml's. There btw. a long while ago was already a discussion on that. opendir does only give back a handle, and that does not need much memory. So it is not a problem for that you should use the Gc-module. the error you got says: there are not more filedescriptors available to you, and the system gives you an error because of that and OCaml then throsws you an exception. You have to open and to close by yourself. There are many possiblities to do that work. You can code it directly in a function. So, you could open the directory, read the conetnst to alist, then close the directory and give back the dircontens (list of entries) as a return value. The calling function then can collect the items. Or give a list of dierctories to the function and let it fo all opendir/readdir/closedir things, collecting all dir-contents. So, this all is possible without Gc-module, and using the Gc-module shows me that you don't understand the rpoblem at hand: it is not the garbage collector that must be used to throw away allocated data, it is the Operating-System, which needs the files to be closed. As it is not the Garbage Collector, it makes no sense to use Gc-functions here. Just open the dir, read the contents and then close the dir. That's all. > and calling opendir too many times as in > > for i = 1 to 2000 do > let d = Unix.opendir "/tmp" in () > done;; This makles no sense. Why to open directories and not use them? Do you go to the bakery, buy 2000 breads and then put them in your garage, not eating one of them? [...] > > Replacing Unix.opendir by safe_opendir: > > let safe_opendir path = > let _ = Gc.major () > and d = Unix.opendir path > in Gc.finalise Unix.closedir d > > seems a solution, but the efficiency is very low due to high > cost of garbage collector calling function. So, don't use it. > > My question is how to write a code as efficient > as it would be possible using a reference-counted language. [...] So, it seems you think OCaml is very inefficient and so you try to make coding "as efficient as possible" before you write any line of code. This makes no sense. OCaml is astouningly efficient, and you should better start to write your code and only if you come to a limit, where you really think you have to optimize, then you should. So, do you really want to open some-thousand directories? Then you can write a function, that gets a list of directories as argument and then in a loop opens --- reads (and daves data) --- closes No Gc-stuff needed. Ciao, Oliver ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-09 21:39 Ask for a more efficient way to deallocate memory (full version) Fabrice.Pardo 2007-12-09 21:55 ` [Caml-list] " Olivier Andrieu 2007-12-09 21:57 ` Oliver Bandel @ 2007-12-09 22:12 ` Jon Harrop 2007-12-09 22:34 ` Oliver Bandel 2007-12-09 22:16 ` Oliver Bandel 3 siblings, 1 reply; 19+ messages in thread From: Jon Harrop @ 2007-12-09 22:12 UTC (permalink / raw) To: caml-list On Sunday 09 December 2007 21:39, Fabrice.Pardo@lpn.cnrs.fr wrote: > Hello, and sorry for my previous unfinished message. > > As the function Unix.opendir returns a value, > we can expect that Unix.closedir will be automatically > called when the dir_handle value is no more referenced. You cannot expect that. > ... > My question is how to write a code as efficient > as it would be possible using a reference-counted language. RAII wrapped in a higher-order function as Olivier described is the idiomatic solution in OCaml. -- Dr Jon D Harrop, Flying Frog Consultancy Ltd. http://www.ffconsultancy.com/products/?e ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-09 22:12 ` Jon Harrop @ 2007-12-09 22:34 ` Oliver Bandel 0 siblings, 0 replies; 19+ messages in thread From: Oliver Bandel @ 2007-12-09 22:34 UTC (permalink / raw) To: caml-list Zitat von Jon Harrop <jon@ffconsultancy.com>: > On Sunday 09 December 2007 21:39, Fabrice.Pardo@lpn.cnrs.fr wrote: > > Hello, and sorry for my previous unfinished message. > > > > As the function Unix.opendir returns a value, > > we can expect that Unix.closedir will be automatically > > called when the dir_handle value is no more referenced. > > You cannot expect that. > > > ... > > My question is how to write a code as efficient > > as it would be possible using a reference-counted language. > > RAII wrapped in a higher-order function as Olivier described is the > idiomatic > solution in OCaml. 1st: it has a little bug ;-) 2nd: why using a higher-level function here? I would directly code the reading of the dir, because the typical application of opendir would be to read the contents of the dir. But possibly you have some more ideas on how to use the contents of a dir, different than reading ot's contents? ;-) Or perhaps it's a matter of taste, what one preferes. Higher-order functions are a fine thing; I would use them, if necessary, but not obessively every time and everywhere. Ciao, Oliver P.S.: One possibly also could use CPS for that open-read-close stuff. But I'm not sure if this would make sense. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Caml-list] Ask for a more efficient way to deallocate memory (full version) 2007-12-09 21:39 Ask for a more efficient way to deallocate memory (full version) Fabrice.Pardo ` (2 preceding siblings ...) 2007-12-09 22:12 ` Jon Harrop @ 2007-12-09 22:16 ` Oliver Bandel 3 siblings, 0 replies; 19+ messages in thread From: Oliver Bandel @ 2007-12-09 22:16 UTC (permalink / raw) To: caml-list Zitat von Fabrice.Pardo@lpn.cnrs.fr: [...] > > My question is how to write a code as efficient > as it would be possible using a reference-counted language. [...] I don't now, from which languages you come, but possibly Python or Perl. In a discussion on using data-structures (e.g. lists) compared to do it "with iterators", I talked witha Python-programmer. He insisted to use iterators, because that's so much more effifient. Yes, when using Python you possibly should use it, because it's not as efficient like OCaml is. I prefer to have the data in a data structure, so when changing something I easily can insert a map-function to recalculate the results, without ugly hacking. And always I was astouned on OCaml's efficiency. But when I started with OCaml I also had same ideas in my head: I couldn't believe that it's true how efficient OCaml is. But you have to do very heavy things to come to the limits. I doubt that you are at that point. :-) Ciao, Oliver ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2007-12-16 15:57 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2007-12-09 21:39 Ask for a more efficient way to deallocate memory (full version) Fabrice.Pardo 2007-12-09 21:55 ` [Caml-list] " Olivier Andrieu 2007-12-10 11:25 ` Fabrice Pardo 2007-12-10 12:03 ` Michaël Le Barbier 2007-12-10 16:33 ` Oliver Bandel 2007-12-10 20:27 ` Richard Jones 2007-12-10 21:05 ` Oliver Bandel 2007-12-10 21:15 ` Gordon Henriksen 2007-12-10 22:13 ` Oliver Bandel 2007-12-10 22:59 ` Jon Harrop 2007-12-10 23:29 ` Jon Harrop 2007-12-11 2:03 ` Yitzhak Mandelbaum 2007-12-15 21:33 ` Oliver Bandel 2007-12-16 15:14 ` Jon Harrop 2007-12-10 23:24 ` Jon Harrop 2007-12-09 21:57 ` Oliver Bandel 2007-12-09 22:12 ` Jon Harrop 2007-12-09 22:34 ` Oliver Bandel 2007-12-09 22:16 ` Oliver Bandel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox