[Caml-list] Protobuf for ocaml

[Caml-list] Protobuf for ocaml

[picflo_2]

Hi, 

I want to use protobuf for an microservice environment where some Services are weiten in ocaml.
Looking at the different available solutions, I was wondering whether there is a single one considered the most mature and stable? 

Thanks a lot, 
Florian

Re: [Caml-list] Protobuf for ocaml

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

I think it's probably the one that is a type annotation using [@@deriving ]
on your type declaration. You get functions derived by the compiler for
you.

**Dont use piqi!** It's really terrible code.

On Sat, Feb 10, 2018 at 1:05 PM, <picflo_2@web.de> wrote:

> Hi,
>
> I want to use protobuf for an microservice environment where some Services
> are weiten in ocaml.
> Looking at the different available solutions, I was wondering whether
> there is a single one considered the most mature and stable?
>
> Thanks a lot,
> Florian
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 1452 bytes --]

Re: [Caml-list] Protobuf for ocaml

[Louis Roché]

Can you give more details please ?

> On 11 Feb 2018, at 04:56, Kenneth Adam Miller <kennethadammiller@gmail.com> wrote:
> 
> **Dont use piqi!** It's really terrible code.

Re: [Caml-list] Protobuf for ocaml

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

The space complexity is ridiculous (at least X^2, possibly 2^n), it isn't
as convenient as any of the others in allowing you to derive your own
specs, it's horribly written code that uses ref and doesn't conform to
OCaml conventions, it caused a lot of crashing on our end, only works for a
particular compile target until I went to update it, the Makefile is an
awful unmaintainable and unreadable mess, the language it defines is
unnecessary and it's hard to get anything to change.

We (me and some team mates) had this forced on us for the features we
required, which was to be able to get data in and out of ocaml to other
languages, and wanted to use it for a large project. It ended up causing a
lot of unnecessary and unwanted slow down in development, both for our
cases and for CMU's BAP. They had the same problems, and everyone that has
touched it had to do unwanted work trying to get it to reliably do what it
advertises.

On Sat, Feb 10, 2018 at 7:43 PM, Louis Roché <mail@louisroche.net> wrote:

> Can you give more details please ?
>
> > On 11 Feb 2018, at 04:56, Kenneth Adam Miller <
> kennethadammiller@gmail.com> wrote:
> >
> > **Dont use piqi!** It's really terrible code.
>

[-- Attachment #2: Type: text/html, Size: 1647 bytes --]

Re: [Caml-list] Protobuf for ocaml

[Maxime Ransan]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

The difference between ppx_deriving_protobuf and ocaml-protoc is that the
former generates a protobuf schema from an OCaml type while the later
generates the OCaml type from a protobuf schema. They are both using the
same runtime library for the binary encoding.

Although I have written ocaml-protoc as a personal project, it has been
used successfully in a professional context to interface with apache mesos.


On Sat, Feb 10, 2018 at 8:10 PM Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> The space complexity is ridiculous (at least X^2, possibly 2^n), it isn't
> as convenient as any of the others in allowing you to derive your own
> specs, it's horribly written code that uses ref and doesn't conform to
> OCaml conventions, it caused a lot of crashing on our end, only works for a
> particular compile target until I went to update it, the Makefile is an
> awful unmaintainable and unreadable mess, the language it defines is
> unnecessary and it's hard to get anything to change.
>
> We (me and some team mates) had this forced on us for the features we
> required, which was to be able to get data in and out of ocaml to other
> languages, and wanted to use it for a large project. It ended up causing a
> lot of unnecessary and unwanted slow down in development, both for our
> cases and for CMU's BAP. They had the same problems, and everyone that has
> touched it had to do unwanted work trying to get it to reliably do what it
> advertises.
>
> On Sat, Feb 10, 2018 at 7:43 PM, Louis Roché <mail@louisroche.net> wrote:
>
>> Can you give more details please ?
>
>
>>
>> > On 11 Feb 2018, at 04:56, Kenneth Adam Miller <
>> kennethadammiller@gmail.com> wrote:
>> >
>> > **Dont use piqi!** It's really terrible code.
>>
>

[-- Attachment #2: Type: text/html, Size: 2684 bytes --]

Aw: [Caml-list] Protobuf for ocaml

[picflo_2]

[-- Attachment #1: Type: text/html, Size: 4285 bytes --]

Re: [Caml-list] Protobuf for ocaml

[Daniil Baturin]

[-- Attachment #1.1: Type: text/plain, Size: 889 bytes --]

Hi Florian,

It took me some time to get over the fact that Protobuf is just a
payload encoding protocol: no less, but no more. You can think of it as
OSI layer 6 (presentation).
Everything else, including reliable (or unreliable) delivery (layer 4),
message routing (layer 3), or even the wire protocol are separate concerns.

You may want to look into existing message transport solutions such as
ZMQ if the situation is more complicated than point to point stateless
communication.

On 02/11/2018 09:15 PM, picflo_2@web.de wrote:
> Hi,
>  
> thanks for all the answers, I will look deeper into ocaml-protoc.
>  
> As a kind of follow-up question. Do you add some kind of unique
> identifier to every message to allow resending the message in case
> some service goes down?
>  
> Thanks again for the help here I highly appreciate it,
> Florian
>  
>  



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 862 bytes --]

Re: [Caml-list] Protobuf for ocaml

[Maxime Ransan]

[-- Attachment #1: Type: text/plain, Size: 2746 bytes --]

Both solution focuses purely on data serialization. The protocol part
(choice of transport protocol, service discovery, retries, async vs
async..) are expected to be done outside on those library.

On Sun, Feb 11, 2018 at 9:15 AM <picflo_2@web.de> wrote:

> Hi,
>
> thanks for all the answers, I will look deeper into ocaml-protoc.
>
> As a kind of follow-up question. Do you add some kind of unique identifier
> to every message to allow resending the message in case some service goes
> down?
>
> Thanks again for the help here I highly appreciate it,
> Florian
>
>
> *Gesendet:* Sonntag, 11. Februar 2018 um 03:15 Uhr
> *Von:* "Maxime Ransan" <maxime.ransan@gmail.com>
> *An:* "Kenneth Adam Miller" <kennethadammiller@gmail.com>
> *Cc:* "Louis Roché" <mail@louisroche.net>, picflo_2@web.de, "caml users" <
> caml-list@inria.fr>
> *Betreff:* Re: [Caml-list] Protobuf for ocaml
> The difference between ppx_deriving_protobuf and ocaml-protoc is that the
> former generates a protobuf schema from an OCaml type while the later
> generates the OCaml type from a protobuf schema. They are both using the
> same runtime library for the binary encoding.
>
> Although I have written ocaml-protoc as a personal project, it has been
> used successfully in a professional context to interface with apache mesos.
>
>
> On Sat, Feb 10, 2018 at 8:10 PM Kenneth Adam Miller <
> kennethadammiller@gmail.com> wrote:
>
>> The space complexity is ridiculous (at least X^2, possibly 2^n), it isn't
>> as convenient as any of the others in allowing you to derive your own
>> specs, it's horribly written code that uses ref and doesn't conform to
>> OCaml conventions, it caused a lot of crashing on our end, only works for a
>> particular compile target until I went to update it, the Makefile is an
>> awful unmaintainable and unreadable mess, the language it defines is
>> unnecessary and it's hard to get anything to change.
>>
>> We (me and some team mates) had this forced on us for the features we
>> required, which was to be able to get data in and out of ocaml to other
>> languages, and wanted to use it for a large project. It ended up causing a
>> lot of unnecessary and unwanted slow down in development, both for our
>> cases and for CMU's BAP. They had the same problems, and everyone that has
>> touched it had to do unwanted work trying to get it to reliably do what it
>> advertises.
>>
>> On Sat, Feb 10, 2018 at 7:43 PM, Louis Roché <mail@louisroche.net>
>> wrote:
>>>
>>> Can you give more details please ?
>>
>>
>>>
>>> > On 11 Feb 2018, at 04:56, Kenneth Adam Miller <
>>> kennethadammiller@gmail.com> wrote:
>>> >
>>> > **Dont use piqi!** It's really terrible code.
>>>
>>

[-- Attachment #2: Type: text/html, Size: 5006 bytes --]

Aw: [Caml-list] Protobuf for ocaml

[picflo_2]

[-- Attachment #1: Type: text/html, Size: 6825 bytes --]

Re: [Caml-list] Protobuf for ocaml

[Daniil Baturin]

[-- Attachment #1.1: Type: text/plain, Size: 543 bytes --]

I'm using https://opam.ocaml.org/packages/ocaml-protoc/

The schema is intentionally handwritten, but ppx_deriving_protobuf can
produce input files for protoc if you want to start from that point instead.

On 02/11/2018 01:05 AM, picflo_2@web.de wrote:
> Hi, 
>
> I want to use protobuf for an microservice environment where some Services are weiten in ocaml.
> Looking at the different available solutions, I was wondering whether there is a single one considered the most mature and stable? 
>
> Thanks a lot, 
> Florian
>



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 862 bytes --]

Re: [Caml-list] Protobuf for ocaml

[SP]

Just on the odd chance that protobuf isn't a requirement, you may want
to consider MessagePack (with https://github.com/vbmithr/ocaml-msgpck).
Used it in production and worked well for us.  Ignore this message if
this of no interest.

-- 
	SP