Details

    • Type: Technical task
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw, meas_algorithms
    • Labels:
    • Story Points:
      2
    • Team:
      External

      Description

      Make sure we have spatially-varying transmission curves attached to Exposures by the time we write the calexp dataset.

      Will need to decide how to store filter curves - best bet is probably as a butler dataset keyed only by filter for now.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            Paul Price, I have on the branches for this ticket some new butler datasets ("transmission_optics", "transmission_filter", and "transmission_sensor"), and a script to add the HSC version of these to a data repository (along with some code to combine them and attach them to Exposures in ISR, but that's not relevant right now).

            The problem with all of this is that these are regular butler datasets, not calibration datasets, and they really ought to be calibration datasets - even if we only have one validity period for all of them right now, we'll need another one for at least the post-recoating mirror reflectivities soon. I know next to nothing about how the calibration-generation side of things works; could you take a quick look at my code and give me a quick walkthrough on how to convert it to create calibration products? If it's easier, you're also welcome to just do it yourself (on this branch, in fact) - I'm using tickets/DM-12366 for integrating all of this and I have a copy of all of these commits there that I can use to work on the coaddition part of this in the meantime.

            Merlin Fisher-Levine and Robert Lupton may have thoughts on this as well, though I expect the scripts for creating transmission curve data products in LSST will look nothing like the approach taken here (which can be characterized as, "Robert writes a notebook to understand Kawanomoto-san's numbers, Eli copies dicts full of numbers from Robert's notebook into FGCM and figures out to interpolate them, Jim copies Eli's Python module into obs_subaru, writes a script to call Butler.put, and asks Paul for help).

            Show
            jbosch Jim Bosch added a comment - Paul Price , I have on the branches for this ticket some new butler datasets ("transmission_optics", "transmission_filter", and "transmission_sensor"), and a script to add the HSC version of these to a data repository (along with some code to combine them and attach them to Exposures in ISR, but that's not relevant right now). The problem with all of this is that these are regular butler datasets, not calibration datasets, and they really ought to be calibration datasets - even if we only have one validity period for all of them right now, we'll need another one for at least the post-recoating mirror reflectivities soon. I know next to nothing about how the calibration-generation side of things works; could you take a quick look at my code and give me a quick walkthrough on how to convert it to create calibration products? If it's easier, you're also welcome to just do it yourself (on this branch, in fact) - I'm using tickets/ DM-12366 for integrating all of this and I have a copy of all of these commits there that I can use to work on the coaddition part of this in the meantime. Merlin Fisher-Levine and Robert Lupton may have thoughts on this as well, though I expect the scripts for creating transmission curve data products in LSST will look nothing like the approach taken here (which can be characterized as, "Robert writes a notebook to understand Kawanomoto-san's numbers, Eli copies dicts full of numbers from Robert's notebook into FGCM and figures out to interpolate them, Jim copies Eli's Python module into obs_subaru, writes a script to call Butler.put, and asks Paul for help).
            Hide
            jbosch Jim Bosch added a comment -

            As we discussed, this is now mostly ready for review (though it now has tests, including ci_hsc). I still would would like to try to make these calibration datasets, but what I have so far works.

            Here are the PRs:

            Show
            jbosch Jim Bosch added a comment - As we discussed, this is now mostly ready for review (though it now has tests, including ci_hsc). I still would would like to try to make these calibration datasets, but what I have so far works. Here are the PRs: https://github.com/lsst/ip_isr/pull/44 https://github.com/lsst/obs_base/pull/79 https://github.com/lsst/obs_subaru/pull/108 https://github.com/lsst/ci_hsc/pull/34
            Hide
            jbosch Jim Bosch added a comment -

            Actually, after taking a look at what it would take to make these calibration datasets, I don't think I should try to do it right now. We wouldn't actually be using any of that functionality yet (since we only have one curve of each type for all of the science data we have so far), we're a bit pressed for time getting this all working, and I think anything I could do now to get them in would be a bit of a hack. I'm not sure we'll be able to wait for Gen3, but we can at least wait until we're prepping for the next HSC data release. I'm going to put my effort into writing the code to coadd them instead.

            Show
            jbosch Jim Bosch added a comment - Actually, after taking a look at what it would take to make these calibration datasets, I don't think I should try to do it right now. We wouldn't actually be using any of that functionality yet (since we only have one curve of each type for all of the science data we have so far), we're a bit pressed for time getting this all working, and I think anything I could do now to get them in would be a bit of a hack. I'm not sure we'll be able to wait for Gen3, but we can at least wait until we're prepping for the next HSC data release. I'm going to put my effort into writing the code to coadd them instead.
            Hide
            price Paul Price added a comment -

            This is great. I put minor comments on the GitHub PRs.

            Show
            price Paul Price added a comment - This is great. I put minor comments on the GitHub PRs.
            Hide
            jbosch Jim Bosch added a comment -

            In the course of writing RFC-440, I decided it would be best to disable this by default until the necessary datasets are present on lsst-dev.  I've removed that commit and will create a new ticket for enabling it triggered by that RFC.

            Show
            jbosch Jim Bosch added a comment - In the course of writing RFC-440 , I decided it would be best to disable this by default until the necessary datasets are present on lsst-dev.  I've removed that commit and will create a new ticket for enabling it triggered by that RFC.
            Hide
            jbosch Jim Bosch added a comment -

            Merged to master.

            Show
            jbosch Jim Bosch added a comment - Merged to master.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Paul Price
              Watchers:
              Jim Bosch, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: