Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2430

Make qserv server-side log messages more standard

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Qserv server-side Python logging appears to mostly use a common format: "%(asctime)s %(name)s %(levelname)s: %(message)s". It also mostly uses a common date format: "%m/%d/%Y %I:%M:%S". But I see instances of:

      • "%(asctime)s %(levelname)s %(message)s"
      • "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
      • "%(asctime)s {%(pathname)s:%(lineno)d} %(levelname)s %(message)s"
      • and now, after DM-2176, "%(asctime)s [PID:%(process)d] [%(levelname)s] (%(funcName)s() at %(filename)s:%(lineno)d) %(name)s: %(message)s"

      Unless these are used in very different contexts, it will aid automated log processing for them to be more standardized.

      In addition, the date format is unacceptable as it does not use RFC 3339 (ISO8601) format and does not include a timezone indicator (which means the default datefmt is insufficient). This must be fixed. See also DM-1203.

        Attachments

          Issue Links

            Activity

            Hide
            jbecla Jacek Becla added a comment - - edited

            Per discussion at the qserv hangout 4/1:

            • different applications might want different format, e.g., multithreaded app need PID
            • conclusion: keep all common parts on the left side
            • proposed:
              • common part: "%(date)s %(levelname)s"
              • followed by optional: "[PID:%(process)d]
              • followed by optional: "%(filename)s:%(lineno)d)"
              • followed by: "%(message)s"
            • date format:
              • YYYY-MM-DD hh:mm:ss.ssssss Z
            Show
            jbecla Jacek Becla added a comment - - edited Per discussion at the qserv hangout 4/1: different applications might want different format, e.g., multithreaded app need PID conclusion: keep all common parts on the left side proposed: common part: " %(date)s %(levelname)s " followed by optional: " [PID:%(process)d] followed by optional: " %(filename)s:%(lineno)d) " followed by: " %(message)s " date format: YYYY-MM-DD hh:mm:ss.ssssss Z
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Jacek Becla,

            log4cxx doesn't provide a built-in parameter to log pid. So I need to get pid of calling program in log library code and then send it to log4cxx. Do you want me to do this in a new ticket?

            On the other hand, log4cxx can easily log thread name (using %t), would it be enough for now?

            Thanks

            Show
            jammes Fabrice Jammes added a comment - Hi Jacek Becla , log4cxx doesn't provide a built-in parameter to log pid. So I need to get pid of calling program in log library code and then send it to log4cxx. Do you want me to do this in a new ticket? On the other hand, log4cxx can easily log thread name (using %t), would it be enough for now? Thanks
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Jacek Becla, Kian-Tat Lim,

            Do you know how to handle multi-line log messages (FYI czar log it's config file and also SQL schemas)?

            Thx

            Show
            jammes Fabrice Jammes added a comment - Hi Jacek Becla , Kian-Tat Lim , Do you know how to handle multi-line log messages (FYI czar log it's config file and also SQL schemas)? Thx
            Hide
            ktl Kian-Tat Lim added a comment -

            I think you can capture the process id as a NDC (named debugging context) item and log it that way.

            log messages should (for now) never, ever be multi-line. If you need to log a message that contains newlines (including JSON if we ever have it), you should escape them.

            Show
            ktl Kian-Tat Lim added a comment - I think you can capture the process id as a NDC (named debugging context) item and log it that way. log messages should (for now) never, ever be multi-line. If you need to log a message that contains newlines (including JSON if we ever have it), you should escape them.
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Kian-Tat Lim,

            I think the multi-line escaper should be implemented into lsst/log library for better code factorization, isn't it?
            Furthermore, I would try to also implement the pid retrieval in lsst/log, ins't it?

            Show
            jammes Fabrice Jammes added a comment - Hi Kian-Tat Lim , I think the multi-line escaper should be implemented into lsst/log library for better code factorization, isn't it? Furthermore, I would try to also implement the pid retrieval in lsst/log, ins't it?
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Andy Salnikov,

            Thanks for your review, I've done some updates, can I merge please?

            Have a nice day,

            Show
            jammes Fabrice Jammes added a comment - Hi Andy Salnikov , Thanks for your review, I've done some updates, can I merge please? Have a nice day,
            Hide
            salnikov Andy Salnikov added a comment -

            Fabrice, sorry for delay, I have not finished review yet, I hope I can can do it later today.

            Show
            salnikov Andy Salnikov added a comment - Fabrice, sorry for delay, I have not finished review yet, I hope I can can do it later today.
            Hide
            salnikov Andy Salnikov added a comment -

            Fabrice, I'm done with review, comments are in PR, the most important is do not do def str(), this is dangerous.

            Show
            salnikov Andy Salnikov added a comment - Fabrice, I'm done with review, comments are in PR, the most important is do not do def str() , this is dangerous.

              People

              • Assignee:
                jammes Fabrice Jammes
                Reporter:
                ktl Kian-Tat Lim
                Reviewers:
                Andy Salnikov, Jacek Becla
                Watchers:
                Andy Salnikov, Fabrice Jammes, Jacek Becla, Kian-Tat Lim
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel