Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29615

Fix connections problems in running Gen3 jointcal on HSC RC2

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • jointcal
    • 2
    • Data Release Production
    • No

    Description

      I've brought some of these up on Slack in response to madamow's problem reports, but I figured it'd be good to capture them here with my proposed fixes:

      • The fact that jointcal marks its reference catalogs as regular Inputs instead of PrerequisiteInputs currently makes it impossible to run jointcal in the same pipeline as later tasks (some of which declare the same reference catalog as a PrerequisiteInput). That's not really jointcal's fault, but I also think it's good practice to use PrerequisiteInput for reference catalogs anyway, because they're never something we'd produce in the same pipeline as something that consumes them, and when that's the case then PrerequisiteInput has some advantages, like better diagnostics when no datasets can be found.
      • The inputCamera connection definitely needs to be turned into a PrerequisiteInput (validity-range calibrations only work as prerequisites), and it also needs a special lookup function like this one in FGCM: https://github.com/lsst/fgcmcal/blob/d78879ea4eb43453c9955831f4d697073a1b463f/python/lsst/fgcmcal/fgcmMakeLut.py#L62. That works around the fact that jointcal wants a camera without saying what period in time it should be valid for. I'm kind of surprised this hasn't caused problems in CI already, and I'd guess it's because the test repo is set up with simplifer-than-real calibration content.

      I think we can work around this in the w14 processing, but we'll see how it goes.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch created issue -
            madamow Monika Adamow (Inactive) made changes -
            Field Original Value New Value
            Link This issue relates to DM-29528 [ DM-29528 ]
            jbosch Jim Bosch made changes -
            Assignee John Parejko [ parejkoj ] Jim Bosch [ jbosch ]
            jbosch Jim Bosch made changes -
            Team Alert Production [ 10300 ] Data Release Production [ 10301 ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-28991 [ DM-28991 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-29670 [ DM-29670 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-29670 [ DM-29670 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-29670 [ DM-29670 ]
            jbosch Jim Bosch made changes -
            Reviewers John Parejko [ parejkoj ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-29008 [ DM-29008 ]
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-29149 [ 458506 ]
            yusra Yusra AlSayyad made changes -
            Story Points 2

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              John Parejko
              Eli Rykoff, Jim Bosch, John Parejko, Monika Adamow (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.