Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-171

Add pipe_drivers to lsst_distrib

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      pipe_drivers provides some useful scripts for processing data on a cluster using ctrl_pool. I propose it be bundled with lsst_distrib so users can use it conveniently.

      pipe_drivers will also pull in ctrl_pool and therefore mpich and mpi4py. mpi4py had some trouble building on OSX recently, but this has been resolved. Based on recent lsstsw runs, pipe_drivers builds fine and should not break lsst_distrib when included.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Trying to build pipe_drivers on my Mac and mpich fails because libxml/parser.h can not be found. It's strange because it's on one machine in /usr/include/libxml2/ (that was upgraded from Yosemite) but not in a machine that has always been El Cap and never Yosemite. In the latter case it's in the Xcode SDK location and not in /usr/include. I need to see what mpich is trying to do.

            Show
            tjenness Tim Jenness added a comment - Trying to build pipe_drivers on my Mac and mpich fails because libxml/parser.h can not be found. It's strange because it's on one machine in /usr/include/libxml2/ (that was upgraded from Yosemite) but not in a machine that has always been El Cap and never Yosemite. In the latter case it's in the Xcode SDK location and not in /usr/include . I need to see what mpich is trying to do.
            Hide
            nlust Nate Lust added a comment -

            In my case, I had to reinstall xcode command line tools as my /usr/include was empty, and that populated it. I checked pkg-info to verify the location that libxml2 was installed.

            Show
            nlust Nate Lust added a comment - In my case, I had to reinstall xcode command line tools as my /usr/include was empty, and that populated it. I checked pkg-info to verify the location that libxml2 was installed.
            Hide
            tjenness Tim Jenness added a comment -

            I probably should do this on a proper ticket... It's not clear to me how it works when /usr/include has files though given:

            • clang is clever enough to find the correct SDK /usr/include without you having to tell it where to look (use clang -v to see that happening.
            • The include directive is for libxml/parser.h yet the include files on OS X are in /usr/include/libxml2/libxml/ so there is a libxml2 missing from the path.
            • mpich correctly determines that it should not use libxml as HWLOC_HAVE_LIBXML2 is not defined.
            • topology-xml-libxml.c is not actually built on a system that works but is built on one that fails (there is a conditional make step).

            So there is something about the version of OSX that lacks files in /usr/include that is convincing the build that libxml can be used.

            Show
            tjenness Tim Jenness added a comment - I probably should do this on a proper ticket... It's not clear to me how it works when /usr/include has files though given: clang is clever enough to find the correct SDK /usr/include without you having to tell it where to look (use clang -v to see that happening. The include directive is for libxml/parser.h yet the include files on OS X are in /usr/include/libxml2/libxml/ so there is a libxml2 missing from the path. mpich correctly determines that it should not use libxml as HWLOC_HAVE_LIBXML2 is not defined. topology-xml-libxml.c is not actually built on a system that works but is built on one that fails (there is a conditional make step). So there is something about the version of OSX that lacks files in /usr/include that is convincing the build that libxml can be used.
            Hide
            tjenness Tim Jenness added a comment -

            See DM-5847 for the mpich discussion.

            Show
            tjenness Tim Jenness added a comment - See DM-5847 for the mpich discussion.
            Hide
            tjenness Tim Jenness added a comment -

            Adding pipe_drivers to lsst_distrib gets a from me now that DM-5847 is closed. Can I ask that documenting pipe_drivers also be included (DM-5837) in this RFC as a requirement before we mark this RFC as Implemented?

            Show
            tjenness Tim Jenness added a comment - Adding pipe_drivers to lsst_distrib gets a from me now that DM-5847 is closed. Can I ask that documenting pipe_drivers also be included ( DM-5837 ) in this RFC as a requirement before we mark this RFC as Implemented?
            Hide
            price Paul Price added a comment -

            Adopted: We will provide some minimal documentation and then put pipe_drivers in lsst_distrib.

            Show
            price Paul Price added a comment - Adopted: We will provide some minimal documentation and then put pipe_drivers in lsst_distrib.

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Watchers:
                John Swinbank, Kian-Tat Lim, Nate Lust, Paul Price, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel