Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14364

Gather package metadata from a 'newinstall.sh' based installation in dispatch_verify.py


    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: verify
    • Labels:


      In DM-14328 we start the discussion about publishing verification jobs produced by the weekly HSC reprocessing to SQuaSH.

      This happens at LDF using a newinstall.sh based installation of the LSST stack.

      Unlike an lsstsw-based installation we don't have repos.yaml and manifest.txt, available only in the build enviroment, from where we get the git_branch, git_url and git_sha package metadata added by dispatch_verify.py in the verification jobs.

      For this use case (the HSC reprocessing) from discussions with Simon Krughoff and Frossie Economou it seems that the reprocessing happens on a long enough cadence that code changes may not be as useful as they are in the CI daily builds, but we still can send to SQuaSH the package name and eups_versions used on a given processing.

      The CmdLineTask framework stores a packages.pickle file with that information which can be obtained from the butler using the packages dataset type , e.g:

      >>> from lsst.daf.persistence import Butler
      >>> butler = Butler('/datasets/hsc/repo/rerun/RC/w_2018_18/DM-14243/')
      >>> packages = butler.get('packages')

      which seems the right interface for obtaining the provenance information.

      In this ticket we propose to create an additional interface in lsst.verify to gather the package metadata from LSST stack installations that use the newinstall.sh method and add that information into the verification job sent to SQuaSH.



          There are no comments yet on this issue.


            • Assignee:
              jsick Jonathan Sick
              afausti Angelo Fausti
              Angelo Fausti, Frossie Economou, Jonathan Sick, Joshua Hoblitt, Simon Krughoff
            • Votes:
              0 Vote for this issue
              5 Start watching this issue


              • Created:

                Summary Panel