Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5147

Provide usable repos in {{validation_data_*}} packages.

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Validation
    • Labels:
      None

      Description

      Re-interpreted ticket:
      1. Provide already-initialized repositories in the `validation_data_cfht`, `validation_data_decam`, and `validation_data_hsc` packages alongside the raw data. The goal is to allow both easy quick-start analyses as well as comparisons of output steps from processCcd.py and friends at each step of the processing.
      2. Add (Cfht,Decam,HSC).list files to provide for easy processing of the available dataIds in the example data.
      3. Update README files to explain available data.

      [Original request:]
      In validation_drp when I run examples/runXTest.sh I find that any data I had saved in CFHT or DECam is lost, even if I have carefully renamed it. This is very dangerous and I lost a lot of work due to it. At a bare minimum please do NOT touch any directories not named "input" or "output".

      Lower priority requests that I hope you will consider:

      • Have the the input repo be entirely contained in the validation_data_X packages, ready to use "as is". That would simplify the use of those packages by other code. It would also simplify validate_drp, and it would just leave the output repo to generate (which already has a link back to the input repo).
      • Have runXTest.sh accept a single argument: the path to the output. (The path to the input is not necessary if you implement the first suggestion).

        Attachments

          Activity

          Hide
          nlust Nate Lust added a comment -

          if there will be a nightly/weekly/monthly build running (ci_hsc descendent) Would it make sense to have that get piped into these to be checked out?

          And no there was no indication that there was an issue with the astrometry, I just didn't have a great way to check for the decam and cfht data sets, and wanted to make sure that it was thought about, and that it was the minimum amount necessary for the repository.

          Show
          nlust Nate Lust added a comment - if there will be a nightly/weekly/monthly build running (ci_hsc descendent) Would it make sense to have that get piped into these to be checked out? And no there was no indication that there was an issue with the astrometry, I just didn't have a great way to check for the decam and cfht data sets, and wanted to make sure that it was thought about, and that it was the minimum amount necessary for the repository.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          lsst_qa will be the night/weekly/monthly package and is under development as DM-5381. It will exercise each of these validation_data_* packages through ci_hsc and related scripts to be developed for cfht and decam.

          And yes, not automatically, but on a regular, and curated, basis, the output repos in these validation_data_* repos will be updated based on the output of lsst_qa.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - lsst_qa will be the night/weekly/monthly package and is under development as DM-5381 . It will exercise each of these validation_data_* packages through ci_hsc and related scripts to be developed for cfht and decam . And yes, not automatically, but on a regular, and curated, basis, the output repos in these validation_data_* repos will be updated based on the output of lsst_qa .
          Hide
          nlust Nate Lust added a comment - - edited

          This all sounds good to me, and the structure looks fine, with the caveat that the procedure for updating these be worked out ahead of time, and documented. Preferably in a way that a human will not have to have too much input. Hopefully this will not be a chore for anyone.

          Show
          nlust Nate Lust added a comment - - edited This all sounds good to me, and the structure looks fine, with the caveat that the procedure for updating these be worked out ahead of time, and documented. Preferably in a way that a human will not have to have too much input. Hopefully this will not be a chore for anyone.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Recreating/updating instructions are in the respective README.md files under the Recreating the repository section. It's not meant to completely trivial as my vision still involves a human thinking about any updates to the results and making sure they are understood. It's possible that creating an updating script might reduce the chances of mistakes when doing such updates – future experience will inform this.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Recreating/updating instructions are in the respective README.md files under the Recreating the repository section. It's not meant to completely trivial as my vision still involves a human thinking about any updates to the results and making sure they are understood. It's possible that creating an updating script might reduce the chances of mistakes when doing such updates – future experience will inform this.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Merged to master.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Merged to master.

            People

            • Assignee:
              wmwood-vasey Michael Wood-Vasey
              Reporter:
              rowen Russell Owen
              Reviewers:
              Nate Lust
              Watchers:
              John Swinbank, Michael Wood-Vasey, Nate Lust, Russell Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: