From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id DE4167FA5F for ; Tue, 17 Jan 2017 12:52:44 +0100 (CET) Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=jesper.louis.andersen@gmail.com; spf=Pass smtp.mailfrom=jesper.louis.andersen@gmail.com; spf=None smtp.helo=postmaster@mail-wm0-f41.google.com Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of jesper.louis.andersen@gmail.com) identity=pra; client-ip=74.125.82.41; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jesper.louis.andersen@gmail.com"; x-sender="jesper.louis.andersen@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of jesper.louis.andersen@gmail.com designates 74.125.82.41 as permitted sender) identity=mailfrom; client-ip=74.125.82.41; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jesper.louis.andersen@gmail.com"; x-sender="jesper.louis.andersen@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-wm0-f41.google.com) identity=helo; client-ip=74.125.82.41; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jesper.louis.andersen@gmail.com"; x-sender="postmaster@mail-wm0-f41.google.com"; x-conformance=sidf_compatible IronPort-PHdr: =?us-ascii?q?9a23=3ATHgZMRJSQ10bZUvLrdmcpTZWNBhigK39O0sv0rFi?= =?us-ascii?q?tYgUIv/xwZ3uMQTl6Ol3ixeRBMOAuq4C0LWd6PioGTRZp83e4DZaKN0EfiRGoP?= =?us-ascii?q?tVtjRoONSCB0z/IayiRA0BN+MGamVY+WqmO1NeAsf0ag6aiHSz6TkPBke3blIt?= =?us-ascii?q?dazdU7TfhMWv1u2054abI0AR3GL8MoVJMQ6uoA7Nms4TiIpkYuZtm1qa6kdPLs?= =?us-ascii?q?1QxGcgAFufnx/i79+98JJyu3BZvfMl39RNWqL7e+I/V7MOSHwNM20prOj2rwXD?= =?us-ascii?q?XEPb7XsRTn4VgzJHBgHE6FfxWZKn4QXgse8o+iSBJcDsBZQzRDW5p45tRBLyky?= =?us-ascii?q?oBf2o7/XrPh9Y2iKVGoQnnrhpzzpTPbYe9O/93f6ebdtQfEzkSFv1NXjBMV9vv?= =?us-ascii?q?J7AECPAMaKMF9oQ=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0D2AABMBH5YhilSfUpdGQEBAQEBAQEBA?= =?us-ascii?q?QEBBwEBAQEBFAEBAQEBAQEBAQEBBwEBAQEBgw4BAQEBAX6BCYNRnB+CNiqNIYU?= =?us-ascii?q?rggsqgkKDNgKCEUAXAQEBAQEBAQEBAQESAQEBCAsLCh0wgjMEARYFghYBAQEDA?= =?us-ascii?q?SMdARsSCwEDAQsGBQsNDR0CAiEBAREBBQEKEgYBEhKIVQEDEAgOolY/jAOCAwU?= =?us-ascii?q?BHIMJBYNbChknAwpVgXcBAQEBAQEEAQEBAQEBAQEBARYCBhKLGoJQgiWCWYJeB?= =?us-ascii?q?YckDI1/hVM4hl2GfoQEgXdRiAqGG4oYhw4ygRQPEQGBSxIdTxSEFCCBaj41hwm?= =?us-ascii?q?BTwEBAQ?= X-IPAS-Result: =?us-ascii?q?A0D2AABMBH5YhilSfUpdGQEBAQEBAQEBAQEBBwEBAQEBFAE?= =?us-ascii?q?BAQEBAQEBAQEBBwEBAQEBgw4BAQEBAX6BCYNRnB+CNiqNIYUrggsqgkKDNgKCE?= =?us-ascii?q?UAXAQEBAQEBAQEBAQESAQEBCAsLCh0wgjMEARYFghYBAQEDASMdARsSCwEDAQs?= =?us-ascii?q?GBQsNDR0CAiEBAREBBQEKEgYBEhKIVQEDEAgOolY/jAOCAwUBHIMJBYNbChknA?= =?us-ascii?q?wpVgXcBAQEBAQEEAQEBAQEBAQEBARYCBhKLGoJQgiWCWYJeBYckDI1/hVM4hl2?= =?us-ascii?q?GfoQEgXdRiAqGG4oYhw4ygRQPEQGBSxIdTxSEFCCBaj41hwmBTwEBAQ?= X-IronPort-AV: E=Sophos;i="5.33,244,1477954800"; d="scan'208,217";a="255749993" Received: from mail-wm0-f41.google.com ([74.125.82.41]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/AES128-GCM-SHA256; 17 Jan 2017 12:52:43 +0100 Received: by mail-wm0-f41.google.com with SMTP id c85so196624540wmi.1 for ; Tue, 17 Jan 2017 03:52:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lzzz+PzTUSfzk3fgZgGlndlKG6lMRPMFagNiXGcinL8=; b=GwiNmtpvBCHoINgbiN6EwsVE3+aRQNdSNTwcOD3nBC3/QcUR3nnyxDvmuXX6EKD+B3 iTbtx6Jky8gVH6reVAKEJG8ahdfgMp+WIdYMjGBM8k0fKxrHxZX/h7BzW7JqZ9Ae9a8t 3jzmornK4OnClH0RRGcaC09KUSFeB81zF9jH9zsYTwHiyJ4r4s2pq6y/RxJiSd5EUH0z FeyA+aJa4pleztXhxuSjHVotiYtRLuFI0euS3a6c5K6z4PsW/D6kUwJDpcCDnHorEMBF ofCGF/W9TboLPt0dhqPlzu2t7p31asJdEglhh9i855TOnepDz1A31Zxos3SE71+4pGfC 58Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lzzz+PzTUSfzk3fgZgGlndlKG6lMRPMFagNiXGcinL8=; b=SHE4hl0UormHygx79EAg8pwdEHPr2PJJ2fiiIXdoab74/nXdWfvL3aSiLKkvL/gwZi CXRrByCGQXRLCHSPGIqSavFa94dUfBle8ARO2OtAqJGhGb9SWj7smk2V9JZGF0CLGPAH hjbssa20QvY1sX9o0KcKThJ0UPdNHWPuJdKgR5IkEZOFPKBrJovl3mEIBFGLVhnHV0jn Ax6i0oes6gkQ/vgEIs3+BfN1zIHviPHyZ6Zro5iddRp/YQrBLSPg8ENmT2jz52UkhpBw eNwSj3EFnEUyCTCR7TNKAy8DS5AainD25H7CQ1+9DTspVSiMHXsO7IlnCHL/qzvvc1lz gl4w== X-Gm-Message-State: AIkVDXIrDvktvwn2Zsao/d3lXKwmAPFyW0awhdek1jayyGy/I9FsfyIS0Swu9uASm61YEx1rt4KZHl7HzZhmbQ== X-Received: by 10.28.50.135 with SMTP id y129mr10183748wmy.2.1484653963613; Tue, 17 Jan 2017 03:52:43 -0800 (PST) MIME-Version: 1.0 References: <2383491.giY4FN1y6Q@twitter> <6099A589-2D9B-4394-B727-E7F8B26067FD@recoil.org> In-Reply-To: <6099A589-2D9B-4394-B727-E7F8B26067FD@recoil.org> From: Jesper Louis Andersen Date: Tue, 17 Jan 2017 11:52:32 +0000 Message-ID: To: Anil Madhavapeddy , Chet Murthy Cc: "caml-list@inria.fr users" Content-Type: multipart/alternative; boundary=001a1140df221d9a30054648f0ed Subject: Re: [Caml-list] A question about Ocaml logging --001a1140df221d9a30054648f0ed Content-Type: text/plain; charset=UTF-8 Another point from the trenches of the real world: I've had great success with metrics over logging in many situations. Logs are really nice for knowing what went wrong in a system. And if the logs have some kind of structure (JSON, S-exps, Erlang Terms, ...) you can also dump all context information relevant to the log entry. Modern systems such as Kibana can read such structure and build a search index over them. It is very useful when trying to figure out what went wrong. In a distributed system, you keep a unique request id in the structural log so you can join log entries from multiple subsystems easily in the centralized logging platform. Likewise, whenever a given event occurs, you bump a counter or add an entry to a histogram (HdrHistogram comes to mind). This allows you to export counts to a foreign system for plotting of how the system fares internally. It is much cheaper than rendering a log line in the system, and it is almost as informative if you don't need the log line, but rather its occurrence. Live timing histograms also has the advantage that problems tend to show up in the small before a catastrophe. So you can alter the operating point of the system dynamically long before the catastrophe occur in the real world. I almost never have debug-style logging lines in my code anymore. I much prefer adding a live tracing to the system instead (Erlang has tracing facilities, DTrace is useful on Illumos and FreeBSD, etc). Granted, you can't do this posthumously of an error, but on the other hand, you can tailor it to the situation you have. It also takes care of the need to recompile and redeploy with debug logging (in which case you can't posthumously handle an error). The right way to handle system failure is to take a snapshot of the systems state (or of part of the system to which the problem pertains). Store this snapshot somewhere (in the cloud) and index them, so you can go back with a debugger, attach to the core dump, and inspect what went wrong. Post-mortem debugging is needed because many of the errors which occur lies outside the imagination of the programmers: people abuse systems in ways nobody thought possible. On Mon, Jan 16, 2017 at 4:15 PM Anil Madhavapeddy wrote: > > On 9 Jan 2017, at 18:52, Chet Murthy wrote: > > > > All, > > > > I hope this is the right place to ask this question. I've been > > writing a nontrivial distributed system (well, a number of them over > > the last few years) and have had need of a robust and flexible logging > > framework. Specifically, I've been using "bolt" and its descendant, > > "volt", which provide camlp4 syntax extensions. These extensions make > > the syntax of the logging statements significantly less verbose, and > > that in itself ia a valuable thing. > > > > With the arrival of ppx rewriters, I realize that the camlp4/camlp5 > > method of adding syntax to ocaml is deprecated. So I wonder: is there > > some really good logging toolkit out there, that I've overlooked. > > > > I'm aware of a number of different packages, but only bolt/volt have > > syntax extensions, and it's my belief that they're essential to making > > effortless pervasive log-line instrumentation. > > > > But perhaps I just haven't looked hard enough .... So .... before I > > go write my own, I figured I'd ask the list if there were such a > > thing. > > Dear Chet, > > In MirageOS, we've been moving away from syntax extensions and > towards placing the logging directives as closures directly within the > code. This is slightly slower in the case of debug logging, but in > practise for distributed systems we are finding that having "permanent" > logging at different levels is more valuable than the "recompile with > debug logging" that we used to use. > > The basis library for this that our libraries are mostly using now in the > forthcoming MirageOS3 is Daniel Buenzli's Logs library: > http://erratique.ch/software/logs > > It does not have a syntax extension out of the box, but it does provide > flexible support for multiple backends, and has an Lwt module included. > > Hope that helps! > > regards, > Anil > > > > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs > --001a1140df221d9a30054648f0ed Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Another point from the trenches of the= real world:

I've had great success with metrics over logg= ing in many situations.

Logs are really nice for knowing what went w= rong in a system. And if the logs have some kind of structure (JSON, S-exps= , Erlang Terms, ...) you can also dump all context information relevant to = the log entry. Modern systems such as Kibana can read such structure and bu= ild a search index over them. It is very useful when trying to figure out w= hat went wrong. In a distributed system, you keep a unique request id in th= e structural log so you can join log entries from multiple subsystems easil= y in the centralized logging platform.

Likewise, whenever a gi= ven event occurs, you bump a counter or add an entry to a histogram (HdrHis= togram comes to mind). This allows you to export counts to a foreign system= for plotting of how the system fares internally. It is much cheaper than r= endering a log line in the system, and it is almost as informative if you d= on't need the log line, but rather its occurrence. Live timing histogra= ms also has the advantage that problems tend to show up in the small before= a catastrophe. So you can alter the operating point of the system dynamica= lly long before the catastrophe occur in the real world.

I alm= ost never have debug-style logging lines in my code anymore. I much prefer = adding a live tracing to the system instead (Erlang has tracing facilities,= DTrace is useful on Illumos and FreeBSD, etc). Granted, you can't do t= his posthumously of an error, but on the other hand, you can tailor it to t= he situation you have. It also takes care of the need to recompile and rede= ploy with debug logging (in which case you can't posthumously handle an= error).

The right way to handle system failure is to take a s= napshot of the systems state (or of part of the system to which the problem= pertains). Store this snapshot somewhere (in the cloud) and index them, so= you can go back with a debugger, attach to the core dump, and inspect what= went wrong. Post-mortem debugging is needed because many of the errors whi= ch occur lies outside the imagination of the programmers: people abuse syst= ems in ways nobody thought possible.


On Mon, Jan 16, 2017 at 4:15 PM Anil Madhavapeddy &l= t;anil@recoil.org> wrote:
> On 9 Jan 2017, at 18:52, Chet Murthy = <chetsky@gmail.com> wrote:
>
> All,
>
> I hope this is the right place to ask this question.=C2=A0 I've be= en
> writing a nontrivial distributed system (well, a number of them over > the last few years) and have had need of a robust and flexible logging=
> framework.=C2=A0 Specifically, I've been using "bolt" an= d its descendant,
> "volt", which provide camlp4 syntax extensions.=C2=A0 These = extensions make
> the syntax of the logging statements significantly less verbose, and > that in itself ia a valuable thing.
>
> With the arrival of ppx rewriters, I realize that the camlp4/camlp5 > method of adding syntax to ocaml is deprecated.=C2=A0 So I wonder: is = there
> some really good logging toolkit out there, that I've overlooked.<= br class=3D"gmail_msg"> >
> I'm aware of a number of different packages, but only bolt/volt ha= ve
> syntax extensions, and it's my belief that they're essential t= o making
> effortless pervasive log-line instrumentation.
>
> But perhaps I just haven't looked hard enough ....=C2=A0 So .... b= efore I
> go write my own, I figured I'd ask the list if there were such a > thing.

Dear Chet,

In MirageOS, we've been moving away from syntax extensions and
towards placing the logging directives as closures directly within the
code.=C2=A0 This is slightly slower in the case of debug logging, but in practise for distributed systems we are finding that having "permanent= "
logging at different levels is more valuable than the "recompile with<= br class=3D"gmail_msg"> debug logging" that we used to use.

The basis library for this that our libraries are mostly using now in the forthcoming MirageOS3 is Daniel Buenzli's Logs library:
=C2=A0 http://erratique.ch/software/logs
It does not have a syntax extension out of the box, but it does provide
flexible support for multiple backends, and has an Lwt module included.

Hope that helps!

regards,
Anil




--
Caml-list mailing list.=C2=A0 Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-= list
Beginner's list: http://groups.= yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bug= s
--001a1140df221d9a30054648f0ed--