Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: jointcal
-
Labels:
-
Story Points:2
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
I've brought some of these up on Slack in response to Monika Adamow's problem reports, but I figured it'd be good to capture them here with my proposed fixes:
- The fact that jointcal marks its reference catalogs as regular Inputs instead of PrerequisiteInputs currently makes it impossible to run jointcal in the same pipeline as later tasks (some of which declare the same reference catalog as a PrerequisiteInput). That's not really jointcal's fault, but I also think it's good practice to use PrerequisiteInput for reference catalogs anyway, because they're never something we'd produce in the same pipeline as something that consumes them, and when that's the case then PrerequisiteInput has some advantages, like better diagnostics when no datasets can be found.
- The inputCamera connection definitely needs to be turned into a PrerequisiteInput (validity-range calibrations only work as prerequisites), and it also needs a special lookup function like this one in FGCM: https://github.com/lsst/fgcmcal/blob/d78879ea4eb43453c9955831f4d697073a1b463f/python/lsst/fgcmcal/fgcmMakeLut.py#L62. That works around the fact that jointcal wants a camera without saying what period in time it should be valid for. I'm kind of surprised this hasn't caused problems in CI already, and I'd guess it's because the test repo is set up with simplifer-than-real calibration content.
I think we can work around this in the w14 processing, but we'll see how it goes.
Attachments
Issue Links
Activity
One more problem; not a connections one, but I figure we might as well tackle it on this ticket: we need to add jointcal to the obs_subaru DRP.yaml, and include the config override file at https://github.com/lsst/jointcal/blob/master/tests/config/config-gen3.py there (or whatever it should be for HSC RC2; we're still testing whether it'll be sufficient to make jointcal run in Gen3, but it may be a different question whether those are all the right configs).
I think this is ready review; I've successfully finished jointcal runs in Gen3 for tract=9615 of RC2 (the similarity of that to this ticket number was a useful mnemonic for remembering what I was doing), as well as all downstream-of-jointcal processing, and tract=9813 is underway and seems to be proceeding well.
This grew a tiny bit to include a non-jointcal change (in pipe_tasks).
PRs are:
John Parejko, I think I've addressed all review comments. Everything in jointcal itself was straightforward, but I've got a question for you on how best to proceed on the obs_subaru PR.
Ah, except that I hadn't noticed until just now that an earlier Jenkins run failed in ci_hsc, because of course the obs_subaru changes tell it to run jointcal and we don't want that. So I'll have a ci_hsc_gen3 branch for you to review up shortly.
Ok, I've added a ci_hsc_gen3 PR, and just reverted the change that disabled jointcal photometry entirely; doing the right thing there seems trickier than I'd like to deal with on this ticket, which otherwise should by pretty easy to get into the weekly. Jenkins is running (https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34009/pipeline), but with my local ci_hsc_gen3 already past the point where I'd expect failures, I think it'll succeed.
I left a response on the obs_subaru PR about a way around the jointcal HSC test failures. The various broader testing questions remain unsolved, but DM-28863 may help us answer those.
Thanks for fixing these things for the run.
I'm not sure how jointcal got around the camera issue in the tests; I expected that to be needed when John Parejko was putting that together, but then somebody convinced me that the lookup worked because jointcal is given explicit visits while fgcmcal runs the look-up table creation ab initio? In any case, I'd love to understand more in which instances this works and doesn't work.