Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29615

Fix connections problems in running Gen3 jointcal on HSC RC2

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • jointcal
    • 2
    • Data Release Production
    • No

    Description

      I've brought some of these up on Slack in response to madamow's problem reports, but I figured it'd be good to capture them here with my proposed fixes:

      • The fact that jointcal marks its reference catalogs as regular Inputs instead of PrerequisiteInputs currently makes it impossible to run jointcal in the same pipeline as later tasks (some of which declare the same reference catalog as a PrerequisiteInput). That's not really jointcal's fault, but I also think it's good practice to use PrerequisiteInput for reference catalogs anyway, because they're never something we'd produce in the same pipeline as something that consumes them, and when that's the case then PrerequisiteInput has some advantages, like better diagnostics when no datasets can be found.
      • The inputCamera connection definitely needs to be turned into a PrerequisiteInput (validity-range calibrations only work as prerequisites), and it also needs a special lookup function like this one in FGCM: https://github.com/lsst/fgcmcal/blob/d78879ea4eb43453c9955831f4d697073a1b463f/python/lsst/fgcmcal/fgcmMakeLut.py#L62. That works around the fact that jointcal wants a camera without saying what period in time it should be valid for. I'm kind of surprised this hasn't caused problems in CI already, and I'd guess it's because the test repo is set up with simplifer-than-real calibration content.

      I think we can work around this in the w14 processing, but we'll see how it goes.

      Attachments

        Issue Links

          Activity

            erykoff Eli Rykoff added a comment - - edited

            I'm not sure how jointcal got around the camera issue in the tests; I expected that to be needed when Parejkoj was putting that together, but then somebody convinced me that the lookup worked because jointcal is given explicit visits while fgcmcal runs the look-up table creation ab initio? In any case, I'd love to understand more in which instances this works and doesn't work.

            erykoff Eli Rykoff added a comment - - edited I'm not sure how jointcal got around the camera issue in the tests; I expected that to be needed when Parejkoj was putting that together, but then somebody convinced me that the lookup worked because jointcal is given explicit visits while fgcmcal runs the look-up table creation ab initio? In any case, I'd love to understand more in which instances this works and doesn't work.
            jbosch Jim Bosch added a comment -

            One more problem; not a connections one, but I figure we might as well tackle it on this ticket: we need to add jointcal to the obs_subaru DRP.yaml, and include the config override file at https://github.com/lsst/jointcal/blob/master/tests/config/config-gen3.py there (or whatever it should be for HSC RC2; we're still testing whether it'll be sufficient to make jointcal run in Gen3, but it may be a different question whether those are all the right configs).

            jbosch Jim Bosch added a comment - One more problem; not a connections one, but I figure we might as well tackle it on this ticket: we need to add jointcal to the obs_subaru DRP.yaml, and include the config override file at https://github.com/lsst/jointcal/blob/master/tests/config/config-gen3.py there (or whatever it should be for HSC RC2; we're still testing whether it'll be sufficient to make jointcal run in Gen3, but it may be a different question whether those are all the right configs).
            jbosch Jim Bosch added a comment -

            I think this is ready review; I've successfully finished jointcal runs in Gen3 for tract=9615 of RC2 (the similarity of that to this ticket number was a useful mnemonic for remembering what I was doing), as well as all downstream-of-jointcal processing, and tract=9813 is underway and seems to be proceeding well.

            This grew a tiny bit to include a non-jointcal change (in pipe_tasks).

            PRs are:

            jbosch Jim Bosch added a comment - I think this is ready review; I've successfully finished jointcal runs in Gen3 for tract=9615 of RC2 (the similarity of that to this ticket number was a useful mnemonic for remembering what I was doing), as well as all downstream-of-jointcal processing, and tract=9813 is underway and seems to be proceeding well. This grew a tiny bit to include a non-jointcal change (in pipe_tasks). PRs are: https://github.com/lsst/jointcal/pull/177 https://github.com/lsst/pipe_tasks/pull/496 https://github.com/lsst/obs_subaru/pull/356
            jbosch Jim Bosch added a comment -

            Parejkoj, I think I've addressed all review comments. Everything in jointcal itself was straightforward, but I've got a question for you on how best to proceed on the obs_subaru PR.

            jbosch Jim Bosch added a comment - Parejkoj , I think I've addressed all review comments. Everything in jointcal itself was straightforward, but I've got a question for you on how best to proceed on the obs_subaru PR.
            jbosch Jim Bosch added a comment -

            Ah, except that I hadn't noticed until just now that an earlier Jenkins run failed in ci_hsc, because of course the obs_subaru changes tell it to run jointcal and we don't want that. So I'll have a ci_hsc_gen3 branch for you to review up shortly.

            jbosch Jim Bosch added a comment - Ah, except that I hadn't noticed until just now that an earlier Jenkins run failed in ci_hsc, because of course the obs_subaru changes tell it to run jointcal and we don't want that. So I'll have a ci_hsc_gen3 branch for you to review up shortly.
            jbosch Jim Bosch added a comment -

            Ok, I've added a ci_hsc_gen3 PR, and just reverted the change that disabled jointcal photometry entirely; doing the right thing there seems trickier than I'd like to deal with on this ticket, which otherwise should by pretty easy to get into the weekly. Jenkins is running (https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34009/pipeline), but with my local ci_hsc_gen3 already past the point where I'd expect failures, I think it'll succeed.

            jbosch Jim Bosch added a comment - Ok, I've added a ci_hsc_gen3 PR, and just reverted the change that disabled jointcal photometry entirely; doing the right thing there seems trickier than I'd like to deal with on this ticket, which otherwise should by pretty easy to get into the weekly. Jenkins is running ( https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34009/pipeline ), but with my local ci_hsc_gen3 already past the point where I'd expect failures, I think it'll succeed.
            Parejkoj John Parejko added a comment - - edited

            I left a response on the obs_subaru PR about a way around the jointcal HSC test failures. The various broader testing questions remain unsolved, but DM-28863 may help us answer those.

            Thanks for fixing these things for the run.

            Parejkoj John Parejko added a comment - - edited I left a response on the obs_subaru PR about a way around the jointcal HSC test failures. The various broader testing questions remain unsolved, but DM-28863 may help us answer those. Thanks for fixing these things for the run.

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              John Parejko
              Eli Rykoff, Jim Bosch, John Parejko, Monika Adamow (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.