Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24300

Create Gen3 repo from testdata_jointcal

    Details

    • Story Points:
      6
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      When planning PipelineTask conversion of FGCM, we recognized a Gen3-converted testdata_jointcal repo as a useful prerequisite (presumably the same is also true for a PipelineTask conversion of jointcal).

      I'm not familiar enough with the structure of that package to have a plan in mind for how to organize things (or whether to try to convert the HSC data separately and before trying to convert the DECam data - I assume we'll defer the CFHT regardless).  Given that there is data from multiple instruments here it could also be an interesting test of making multi-instrument Gen3 repos, but we shouldn't let that block getting something usable for PipelineTask testing.  I'm hoping John Parejko (who now knows both about this data and gen2to3) might have some thoughts on those points.

      Tim Jenness, if we can get this scheduled to be done by someone more familiar with the conversion code this month, that will open up Eli Rykoff to work on the actual FGCM conversion next month.  I don't think it should be a lot of work, but again, I'm less knowledgeable about this package than others.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Looks okay. Minor comments only. Presumably you can run this as part of a setUpClass in a gen3 test in fgcmcal? Or else add it as a scons target there so that it gets converted before the tests run.

            Show
            tjenness Tim Jenness added a comment - Looks okay. Minor comments only. Presumably you can run this as part of a setUpClass in a gen3 test in fgcmcal? Or else add it as a scons target there so that it gets converted before the tests run.
            Hide
            erykoff Eli Rykoff added a comment -

            Thanks for the quick turnaround! I'm trying to think of ways to cache this usefully (in the future).  Is it possible to have a scons target in testdata_jointcal (separate ticket!) so that it can be "installed".  Would it then sit there on Jenkins until testdata_jointcal was updated (or a node was wiped, or something like that)?

            Show
            erykoff Eli Rykoff added a comment - Thanks for the quick turnaround! I'm trying to think of ways to cache this usefully (in the future).  Is it possible to have a scons target in testdata_jointcal (separate ticket!) so that it can be "installed".  Would it then sit there on Jenkins until testdata_jointcal was updated (or a node was wiped, or something like that)?
            Hide
            tjenness Tim Jenness added a comment -

            It is possible to build and install testdata_jointcal and run the conversion dynamically. We have to balance that with the annoyance of having to reinstall the entire dataset every single time a dependency changes (obs_base and daf_butler). An alternative to that is to have a package that depends on testdata_jointcal and does the installation itself – that could soft link to testdata_jointcal so wouldn't take a huge amount of space when a dependency changes.

            Show
            tjenness Tim Jenness added a comment - It is possible to build and install testdata_jointcal and run the conversion dynamically. We have to balance that with the annoyance of having to reinstall the entire dataset every single time a dependency changes (obs_base and daf_butler). An alternative to that is to have a package that depends on testdata_jointcal and does the installation itself – that could soft link to testdata_jointcal so wouldn't take a huge amount of space when a dependency changes.
            Hide
            erykoff Eli Rykoff added a comment -

            Oh, good point, I wasn't thinking about the fact that daf_butler changes would trigger the rebuild (which kind of is the point, though, isn't it?).  Anyway, at the moment this will unblock a ton of work, and I'm not sure how we'll be using it in a month, so let's punt this.  Thanks!

            Show
            erykoff Eli Rykoff added a comment - Oh, good point, I wasn't thinking about the fact that daf_butler changes would trigger the rebuild (which kind of is the point, though, isn't it?).  Anyway, at the moment this will unblock a ton of work, and I'm not sure how we'll be using it in a month, so let's punt this.  Thanks!
            Hide
            tjenness Tim Jenness added a comment -

            Yes, daf_butler triggering the build is the point, but if the testdata package is multi GB you really don't want to be installing that each time daf_butler changes. You want to redo the conversion and install a soft link tree – hence the suggestion for a second package that does that.

            Show
            tjenness Tim Jenness added a comment - Yes, daf_butler triggering the build is the point, but if the testdata package is multi GB you really don't want to be installing that each time daf_butler changes. You want to redo the conversion and install a soft link tree – hence the suggestion for a second package that does that.

              People

              • Assignee:
                erykoff Eli Rykoff
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Tim Jenness
                Watchers:
                Eli Rykoff, Jim Bosch, John Parejko, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: