[Caml-list] Flush behavior of baseic I/O class type

[Caml-list] Flush behavior of baseic I/O class type

[Yamagata Yoriyuki]

Hi,

In http://www.ocaml-programming.de/rec/IO-Classes.html, flush method
is defined as "The implementation may choose that put does not write
directly to the underlying resource, but into a buffer. In this case,
the call of flush writes the contents of the buffer to the
resource. When there is no such buffer, the call does nothing."

However if flush cannot output the whole buffer, what should it do?
The underlying output channel could be blocked, or the channel is code
converter and flush is called in the middle of the multibyte
character.

I think three possibility.
  1) Output as far as possible, and leave the rest.
  2) raise an exception
  3) undefined.

What is your opinion?

--
Yamagata Yoriyuki


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Flush behavior of baseic I/O class type

[David MENTRE]

Hello,

Yamagata Yoriyuki <yoriyuki@mbg.ocn.ne.jp> writes:

>   1) Output as far as possible, and leave the rest.
>   2) raise an exception
>   3) undefined.

Or  4) call blocked until the whole buffer can be flushed.

My personnal choice: 4, 1, 2, 3

Yours,
d.

PS: I've never used IO-Classes.
-- 
pub  1024D/A3AD7A2A 2004-10-03 David MENTRE <dmentre@linux-france.org>
 5996 CC46 4612 9CA4 3562  D7AC 6C67 9E96 A3AD 7A2A

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Flush behavior of baseic I/O class type

[Yamagata Yoriyuki]

From: David MENTRE <dmentre@linux-france.org>
Subject: Re: [Caml-list] Flush behavior of baseic I/O class type
Date: Sat, 16 Oct 2004 18:26:24 +0200

> Yamagata Yoriyuki <yoriyuki@mbg.ocn.ne.jp> writes:
> 
> >   1) Output as far as possible, and leave the rest.
> >   2) raise an exception
> >   3) undefined.
> 
> Or  4) call blocked until the whole buffer can be flushed.

It may be ok for non-blocking I/O, but it will cause "busy-waiting" ,
so perhaps a user does not like it.  Moreover, for example, if the
channel is actually a filter, and it needs more inputs to determine
the next output, then it will stuck forever.  (See the case of the
multibyte charcter code converter in the previous mail)

The easiest way (from the implementer's point of the view) would be
1), or 1') do the best effort to output, but nothing is guaranteed.
However I'm afraid that it will cause a subtle I/O bug.

--
Yamagata Yoriyuki

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Flush behavior of baseic I/O class type

[Gerd Stolpmann]

On Sam, 2004-10-16 at 19:51, Yamagata Yoriyuki wrote:
> From: David MENTRE <dmentre@linux-france.org>
> Subject: Re: [Caml-list] Flush behavior of baseic I/O class type
> Date: Sat, 16 Oct 2004 18:26:24 +0200
> 
> > Yamagata Yoriyuki <yoriyuki@mbg.ocn.ne.jp> writes:
> > 
> > >   1) Output as far as possible, and leave the rest.
> > >   2) raise an exception
> > >   3) undefined.
> > 
> > Or  4) call blocked until the whole buffer can be flushed.
> 
> It may be ok for non-blocking I/O, but it will cause "busy-waiting" ,
> so perhaps a user does not like it.  Moreover, for example, if the
> channel is actually a filter, and it needs more inputs to determine
> the next output, then it will stuck forever.  (See the case of the
> multibyte charcter code converter in the previous mail)
> 
> The easiest way (from the implementer's point of the view) would be
> 1), or 1') do the best effort to output, but nothing is guaranteed.
> However I'm afraid that it will cause a subtle I/O bug.

Which one?

We need "flush" to ensure data is actually communicated to the other
endpoint of the channel, i.e. it is a means of synchronisation. When the
data are sent in a way this synchronisation cannot be done, this feature
is used in the wrong way, and I think raising an exception would be
justified.

Well, this is why we _need_ "flush" sometimes. There may be other
situations where softer requirements suffice, e.g. when "flush" is
called for debugging purposes such that as much as data as possible are
printed.

So I think there is no general rule, it depends on the purpose "flush"
is used for.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Flush behavior of baseic I/O class type

[John Prevost]

> It may be ok for non-blocking I/O, but it will cause "busy-waiting" ,
> so perhaps a user does not like it.  Moreover, for example, if the
> channel is actually a filter, and it needs more inputs to determine
> the next output, then it will stuck forever.  (See the case of the
> multibyte charcter code converter in the previous mail)
> 
> The easiest way (from the implementer's point of the view) would be
> 1), or 1') do the best effort to output, but nothing is guaranteed.
> However I'm afraid that it will cause a subtle I/O bug.

If there is an explicit "flush" call, then the only reasonable thing
to do is to block until everything that may be written has been
written.  In fact, the flush call on a filter should write everything,
and then tell the next level down to also flush.

For the MBCS implementation that you describe, you're right that
things are slighlty more complicated.  In that case, the MBCS
implementation could choose to *either* "break" the incomplete
multi-byte character somehow (in a similar way to how it would have to
somehow deal with an incomplete multi-byte character when the file is
closed), or it might choose to flush everything up to that incomplete
multi-byte character, but keep the incomplete character in the buffer.
 My choice would probably be to "break" it.  Either way, the
implementation should describe what it does.

Finally, you mention difficulties with blocking vs. non-blocking I/O
here.  You are correct that there is a problem with flush in this
case--and the problem (to me) seems to be in the API design for these
I/O classes.  If you look at the design for Java I/O, you'll notice
that the java.io output classes contain flush methods but do not
support non-blocking I/O, and the java.nio classes support
non-blocking I/O but do not support flush methods.  In C, the fflush
call "may also fail and set errno for any of the errors specified for
the routine write(2)."  And write may return EAGAIN in order to
indicate that the write would block.  And finally, it's worth noting
that the "write" family of calls make a lot more sense in a
non-blocking situation than the "fwrite" family of calls.

So I would suggest that the solution for non-block I/O here is that
the API should be changed.  One choice is that either flush must be
able to return a result or it must be able to raise an exception that
indicates that not all of the output has yet been written (and it's
important that callers don't see the "broken MBCS character" above as
something they can re-try).

The other choice is that there should be a slightly different set of
operations on blocking and non-blocking streams.  In that case:

Non-blocking output operations should always be attempted immediately,
and return a result indicating that there is a problem if they cannot
write everything.  Non-blocking I/O should therefore not have a flush
operation. (If I am writing an application that uses non-blocking I/O,
this is the API I want: I am doing my own buffering, and need to know
exactly what has been written and what still needs to be written.)

Blocking output should never return until every character has either
been written completely, or has been placed in an intermediate buffer.
 And the flush operation should block until all output has been
written (and the MBCS implementor must make the choice I described
above.)  (If I am writing an application that uses blocking I/O, this
is again the API I want: I do not want to worry about buffering, and
either don't care about blocking or am handling it with multiple
threads.  When I want to make sure that everything has been written, I
am happy to call flush.)

Finally, one might argue that if an *output* channel of multi-byte
characters has received a partial character, then a type error has
occurred.  I understand, however, that there may be issues of
expediency that prevent you from making each character atomic.

John.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Flush behavior of baseic I/O class type

[skaller]

On Sun, 2004-10-17 at 01:52, Yamagata Yoriyuki wrote:

> I think three possibility.
>   1) Output as far as possible, and leave the rest.
>   2) raise an exception
>   3) undefined.
> 
> What is your opinion?

Hmmm, well having read all the other replies so far,
perhaps I can contribute to the confusion with
a slighly different approach.

Your specification says '... the call of flush writes .. '
but that's an imperative description.

Instead, you could use a declarative description,
that is, specify the pre- and post- conditions.

For example:

pre-condition: the buffer contains whole characters
post-condition: the buffer is empty and its data is
scheduled for transmission

With this specification:

(1) if you find yourself in the middle of a multi-byte
character, the post condition is not guarranteed
because the pre-condition is not met. The behaviour
may vary depending on the kind of device, communication
path, time of day, and version of the program.

(2) The words 'scheduled for transmission' imply
flush can return before actual transmission.
You have to say that, because you cannot possibly
ensure the transmission on a high level operating
system. On Linux for example there is no way
to ensure disk writes end up on the media.
Just try writing to a floppy. On RH9 the only
way to physically write the floppy is to
unmount it.

It's quite possible that flushing a buffer works
by unlinking the memory block from the IO object,
and adding it to a queue 'to be written', and then
sometime later another thread writes it.

It's *also* possible you have a block device that
cannot support partial writing: you have to write
256 bytes, period, end of story. In this case,
flush will not be able to write everything, period,
end of story .. it isn't acceptable to write a padded
block out except at the end of the file. This may
be the case for a tape drive, for example.

In this case the write is still scheduled, meaning
the level of abstraction the current class deals
with has lost control of the situation and delegated
it to another layer .. but no assurance that the
physical transmission will occur is made, unless
subsequent operations on those lower levels
are done correctly. However, the class managing
the buffer has still carried out its job by delegating
to a lower level.

As an example -- you have an object designed to
physically write, and a program that flushes
at the end of each line. But the problem is you 
are getting a huge bill from your ISP, by sending
twice the number of TCP/IP packets you would need to,
because most are only half full.

So you just add a buffering class into the chain,
which captures physical write from its owner
and buffers them. The upchain class is still satisfying
its post-condition. The downchain class doesn't write,
even when flush is called. It's *designed* that way.
As far as it is concerned, once stuff is given
to it that stuff is *already* scheduled for transmission
by definition, and so flush on it does nothing -- the post
condition is already satisfied.[Caveat: its buffer
must be 'empty' after flush. But what that means is
also an abstraction.. it probably doesn't have floating
buffers, but uses a single fixed one .. in which case
the 'floating buffer it would have had' is deemed
flushed at all times anyhow .. :]

(3) When you have blocking IO, nothing changes.
Flush should block, if it is necessary to satisfy
the post-condition.

The bottom line is that 'scheduled for transmission'
is itself an abstract phrase. What it means depends
on the class. Each class should define this phrase,
so the programmer can choose an appropriate class
for their needs.

Finally, I'd like to give a 'real world' and critical
example. C++ has three kinds of device:

(1) ostream
(2) FILE*
(3) Unix file handle

Typically, ostreams and FILE* both write via Unix file handle.
So what if you use the ostream 'cout' and the FILE* 'stdout'
and the Unix file handle '1' all in the same program??

Well, the ISO Standard in *this* case requires that
interleaved calls to cout and stdout (but not 1) work
'properly' (So they're required to share the same buffer).
[AFAIC remember :] 

This is not so in general for other objects such
as a disk file or communication port. It's not even
required for streams on the same disk file --
they can use different buffers "just don't do that" :)


-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners