Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32373

Run Gen 3 single frame measurement on on validation_data_cfht

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Convert the validation_data_hsc pipeline to work on the CFHT dataset, and run them. The output of this will be the input for DM-29535.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment - - edited

            Notes from talking with Simon Krughoff:

            • validation_data_hsc should be deprecated, as it is not used by the verification team for anything.
            • faro is run on rc2_subset via a script in the bin directory there. This just runs pipetask on the obs_subaru DRP.yaml.

            I should reorganize the validation_data datasets (assuming we're still going to use them, which might not be feasible!) to have the necessary scripts/configs in them.

            A guess at the necessary steps to run on the decam/cfht validation datasets:

            • Find out if anyone has a working DECam/CFHT DRP.yaml, and how much does it look like the obs_subaru one?
            • Make a gen3 repo that contains the raws/calibs/refcat/skymap.
            • Decide whether to always assume the repo is clean (i.e. a new Jenkins checkout every time) or whether to export/import the repo to be able to run repeatedly. The latter would be more future-proofed (like how testdata_jointcal is used for testing), but might be harder to actually do (e.g. calibrations).
            • Run `pipetask REPO obs_subaru/DPR.yaml#step1` and see what happens. We'll almost certainly have to override that for DECam.
            Show
            Parejkoj John Parejko added a comment - - edited Notes from talking with Simon Krughoff : validation_data_hsc should be deprecated, as it is not used by the verification team for anything. faro is run on rc2_subset via a script in the bin directory there. This just runs pipetask on the obs_subaru DRP.yaml. I should reorganize the validation_data datasets (assuming we're still going to use them, which might not be feasible!) to have the necessary scripts/configs in them. A guess at the necessary steps to run on the decam/cfht validation datasets: Find out if anyone has a working DECam/CFHT DRP.yaml, and how much does it look like the obs_subaru one? Make a gen3 repo that contains the raws/calibs/refcat/skymap. Decide whether to always assume the repo is clean (i.e. a new Jenkins checkout every time) or whether to export/import the repo to be able to run repeatedly. The latter would be more future-proofed (like how testdata_jointcal is used for testing), but might be harder to actually do (e.g. calibrations). Run `pipetask REPO obs_subaru/DPR.yaml#step1` and see what happens. We'll almost certainly have to override that for DECam.
            Hide
            Parejkoj John Parejko added a comment -

            As part of this, we might want to move the CFHT defects out of obs_cfht, and into an obs_cfht_data, to match our convention on that.

            Show
            Parejkoj John Parejko added a comment - As part of this, we might want to move the CFHT defects out of obs_cfht, and into an obs_cfht_data, to match our convention on that.
            Hide
            Parejkoj John Parejko added a comment -

            Simon Krughoff: do you mind reviewing these two tickets? It's about 80 lines of changes in obs_cfht to get it to work in gen3, and a bunch of changes and removals in validation_data_cfht demonstrating that we can run a gen3 singleframe pipeline on it. I did the minimum I thought was necessary to get jointcal to run on this data, so if you want to run faro on this output, you'll need to make more changes.

            The commits should all be atomic, which should make the validation_data_cfht changes easier to view, since I changed lfs files.

            https://github.com/lsst/obs_cfht/pull/94
            https://github.com/lsst/validation_data_cfht/pull/13

            Show
            Parejkoj John Parejko added a comment - Simon Krughoff : do you mind reviewing these two tickets? It's about 80 lines of changes in obs_cfht to get it to work in gen3, and a bunch of changes and removals in validation_data_cfht demonstrating that we can run a gen3 singleframe pipeline on it. I did the minimum I thought was necessary to get jointcal to run on this data, so if you want to run faro on this output, you'll need to make more changes. The commits should all be atomic, which should make the validation_data_cfht changes easier to view, since I changed lfs files. https://github.com/lsst/obs_cfht/pull/94 https://github.com/lsst/validation_data_cfht/pull/13
            Hide
            Parejkoj John Parejko added a comment - - edited

            Jenkins (though v_d_cfht isn't included and doesn't have anything to run anyway): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35449/pipeline

            Show
            Parejkoj John Parejko added a comment - - edited Jenkins (though v_d_cfht isn't included and doesn't have anything to run anyway): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35449/pipeline
            Hide
            krughoff Simon Krughoff added a comment -

            Very sorry for taking so long to get to this. It all looks good. Tim has a comment on obs_cfht that I think you should consider

            Show
            krughoff Simon Krughoff added a comment - Very sorry for taking so long to get to this. It all looks good. Tim has a comment on obs_cfht that I think you should consider
            Hide
            sullivan Ian Sullivan added a comment -

            Note that this ticket was completed using an unmerged ticket branch of DM-28862

            Show
            sullivan Ian Sullivan added a comment - Note that this ticket was completed using an unmerged ticket branch of DM-28862

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              Simon Krughoff
              Watchers:
              Ian Sullivan, John Parejko, Leanne Guy, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.