Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20763

Add initial support for Gen3 Butler to obs_decam

    XMLWordPrintable

    Details

      Description

      In order to best support AP PipelineTask conversion work at PCW2019, we need to have a Gen3 data repository for ap_verify_hits2015 (or maybe ap_verify_ci_hits2015, or both?).

      Anyhow, first step towards that is Gen3 Butler support for obs_decam, which is what this ticket is about.  That includes:

      • Writing an Instrument subclass for DECam.  The docs for the base class should tell you most of what you need to do (though I've just discovered they're not appearing in the Sphinx HTML for some reason), but I'm sure a concrete example will be even more useful, which HSC provides.  Note that because this will be only the second Instrument subclass, it's probable we'll find we want to adjust the ABC-subclass boundary a bit.
      • Writing a Formatter for raw DECam data (or if necessary, different formatters for different CCDs), probably by inheriting from FitsRawFormatterBase.  Once again, you can use HSC as an example.  This should do the same thing as the Gen2 CameraMapper's std_raw method.  A test that shows the kind of consistency we expect (for HSC) can be found in ci_hsc_gen2 - note that the Gen3 version generally tries much harder to clean things up and make things are components are internally consistent, so it may not be possible to make them entirely consistent with the Gen2 version.  We may not have a good place to put such a test for DECam until we're further along converting an actual test dataset, but it'd be great to write one for one-off testing now and attach it to this ticket, with the expectation that we'll put it in a CI'd package in the future.

      After those steps (and after running the Instrument class's register method on a Gen3 repo), I think it should be possible to run the Gen3 raw ingest task on DECam, and I consider that the end point for this ticket as long as any unexpected problems aren't huge.   This will rely a lot on the DECam translator in astro_metadata_translator as well as the above, but I think that's already working.

      Note that there's no command-line driver script for that task yet, but it's pretty straightforward to run from an interactive Python prompt.  Given that we know that DECam has buggy boresight information in the headers, I don't expect it to populate the database with good spatial regions yet, and I'm okay with fixing that on another ticket (probably by using astro_metadata_translator's on-the-fly header-patching system to just fix the boresight values to something we've fit for the test data we care about).

      Please ping #dm-middleware on Slack with any questions.

        Attachments

          Issue Links

            Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Description In order to best support AP PipelineTask conversion work at PCW2019, we need to have a Gen3 data repository for {{ap_verify_hits2015}} (or (maybe {{ap_verify_ci_hits2015}}, or both?).

            The first step towards that is Gen3 Butler support for obs_decam, which is what this ticket is about.  That includes:
             * Writing an [Instrument|https://github.com/lsst/daf_butler/blob/7220349561d794e99e523aff29bdb5c3bd80f379/python/lsst/daf/butler/instrument.py#L36] subclass for DECam.  The docs for the base class should tell you most of what you need to do (though I've just discovered they're not appearing in the Sphinx HTML for some reason), but I'm sure a concrete example will be even more useful, which [HSC|https://github.com/lsst/obs_subaru/blob/e6ea06fa94726c5b0f3c7e020223bbc1a58a2d70/python/lsst/obs/subaru/gen3/hsc/instrument.py#L50] provides.  Note that because this will be only the second {{Instrument}} subclass, it's probable we'll find we want to adjust the ABC-subclass boundary a bit.
             * Writing a {{Formatter}} for raw DECam data (or if necessary, different formatters for different CCDs), probably by inheriting from [FitsRawFormatterBase|https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/fitsRawFormatterBase.py#L29].  Once again, you can use [HSC|https://github.com/lsst/obs_subaru/blob/e6ea06fa94726c5b0f3c7e020223bbc1a58a2d70/python/lsst/obs/subaru/gen3/hsc/rawFormatter.py#L38] as an example.  This should do the same thing as the Gen2 CameraMapper's {{std_raw}} method.  A test that shows the kind of consistency we expect (for HSC) can be found in [ci_hsc_gen2 |https://github.com/lsst/ci_hsc_gen2/blob/tickets/DM-19817/tests/test_butlerShims.py#L81]- note that the Gen3 version generally tries much harder to clean things up and make things are components are internally consistent, so it may not be possible to make them entirely consistent with the Gen2 version.  We may not have a good place to put such a test for DECam until we're further along converting an actual test dataset, but it'd be great to write one for one-off testing now and attach it to this ticket, with the expectation that we'll put it in a CI'd package in the future.

            After those steps (and after running the {{Instrument}}} class's {{register}} method on a Gen3 repo), I think it should be possible to run the [Gen3 raw ingest task|https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/gen3/ingest.py] on DECam, and I consider that the end point for this ticket as long as any unexpected problems aren't huge.   This will rely a lot on the DECam translator in {{astro_metadata_translator}} as well as the above, but I think that's already working.

            Note that there's no command-line driver script for that task yet, but it's pretty straightforward to run from an interactive Python prompt.  Given that we know that DECam has buggy boresight information in the headers, I don't expect it to populate the database with good spatial regions yet, and I'm okay with fixing that on another ticket (probably by using {{astro_metadata_translator's}} on-the-fly header-patching system to just fix the boresight values to something we've fit for the test data we care about).

            Please ping #dm-middleware on Slack with any questions.
            In order to best support AP PipelineTask conversion work at PCW2019, we need to have a Gen3 data repository for {{ap_verify_hits2015}} (or maybe {{ap_verify_ci_hits2015}}, or both?).

            Anyhow, first step towards that is Gen3 Butler support for obs_decam, which is what this ticket is about.  That includes:
             * Writing an [Instrument|https://github.com/lsst/daf_butler/blob/7220349561d794e99e523aff29bdb5c3bd80f379/python/lsst/daf/butler/instrument.py#L36] subclass for DECam.  The docs for the base class should tell you most of what you need to do (though I've just discovered they're not appearing in the Sphinx HTML for some reason), but I'm sure a concrete example will be even more useful, which [HSC|https://github.com/lsst/obs_subaru/blob/e6ea06fa94726c5b0f3c7e020223bbc1a58a2d70/python/lsst/obs/subaru/gen3/hsc/instrument.py#L50] provides.  Note that because this will be only the second {{Instrument}} subclass, it's probable we'll find we want to adjust the ABC-subclass boundary a bit.
             * Writing a {{Formatter}} for raw DECam data (or if necessary, different formatters for different CCDs), probably by inheriting from [FitsRawFormatterBase|https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/fitsRawFormatterBase.py#L29].  Once again, you can use [HSC|https://github.com/lsst/obs_subaru/blob/e6ea06fa94726c5b0f3c7e020223bbc1a58a2d70/python/lsst/obs/subaru/gen3/hsc/rawFormatter.py#L38] as an example.  This should do the same thing as the Gen2 CameraMapper's {{std_raw}} method.  A test that shows the kind of consistency we expect (for HSC) can be found in [ci_hsc_gen2 |https://github.com/lsst/ci_hsc_gen2/blob/tickets/DM-19817/tests/test_butlerShims.py#L81]- note that the Gen3 version generally tries much harder to clean things up and make things are components are internally consistent, so it may not be possible to make them entirely consistent with the Gen2 version.  We may not have a good place to put such a test for DECam until we're further along converting an actual test dataset, but it'd be great to write one for one-off testing now and attach it to this ticket, with the expectation that we'll put it in a CI'd package in the future.

            After those steps (and after running the {{Instrument}}} class's {{register}} method on a Gen3 repo), I think it should be possible to run the [Gen3 raw ingest task|https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/gen3/ingest.py] on DECam, and I consider that the end point for this ticket as long as any unexpected problems aren't huge.   This will rely a lot on the DECam translator in {{astro_metadata_translator}} as well as the above, but I think that's already working.

            Note that there's no command-line driver script for that task yet, but it's pretty straightforward to run from an interactive Python prompt.  Given that we know that DECam has buggy boresight information in the headers, I don't expect it to populate the database with good spatial regions yet, and I'm okay with fixing that on another ticket (probably by using {{astro_metadata_translator's}} on-the-fly header-patching system to just fix the boresight values to something we've fit for the test data we care about).

            Please ping #dm-middleware on Slack with any questions.
            swinbank John Swinbank made changes -
            Description In order to best support AP PipelineTask conversion work at PCW2019, we need to have a Gen3 data repository for {{ap_verify_hits2015}} (or maybe {{ap_verify_ci_hits2015}}, or both?).

            Anyhow, first step towards that is Gen3 Butler support for obs_decam, which is what this ticket is about.  That includes:
             * Writing an [Instrument|https://github.com/lsst/daf_butler/blob/7220349561d794e99e523aff29bdb5c3bd80f379/python/lsst/daf/butler/instrument.py#L36] subclass for DECam.  The docs for the base class should tell you most of what you need to do (though I've just discovered they're not appearing in the Sphinx HTML for some reason), but I'm sure a concrete example will be even more useful, which [HSC|https://github.com/lsst/obs_subaru/blob/e6ea06fa94726c5b0f3c7e020223bbc1a58a2d70/python/lsst/obs/subaru/gen3/hsc/instrument.py#L50] provides.  Note that because this will be only the second {{Instrument}} subclass, it's probable we'll find we want to adjust the ABC-subclass boundary a bit.
             * Writing a {{Formatter}} for raw DECam data (or if necessary, different formatters for different CCDs), probably by inheriting from [FitsRawFormatterBase|https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/fitsRawFormatterBase.py#L29].  Once again, you can use [HSC|https://github.com/lsst/obs_subaru/blob/e6ea06fa94726c5b0f3c7e020223bbc1a58a2d70/python/lsst/obs/subaru/gen3/hsc/rawFormatter.py#L38] as an example.  This should do the same thing as the Gen2 CameraMapper's {{std_raw}} method.  A test that shows the kind of consistency we expect (for HSC) can be found in [ci_hsc_gen2 |https://github.com/lsst/ci_hsc_gen2/blob/tickets/DM-19817/tests/test_butlerShims.py#L81]- note that the Gen3 version generally tries much harder to clean things up and make things are components are internally consistent, so it may not be possible to make them entirely consistent with the Gen2 version.  We may not have a good place to put such a test for DECam until we're further along converting an actual test dataset, but it'd be great to write one for one-off testing now and attach it to this ticket, with the expectation that we'll put it in a CI'd package in the future.

            After those steps (and after running the {{Instrument}}} class's {{register}} method on a Gen3 repo), I think it should be possible to run the [Gen3 raw ingest task|https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/gen3/ingest.py] on DECam, and I consider that the end point for this ticket as long as any unexpected problems aren't huge.   This will rely a lot on the DECam translator in {{astro_metadata_translator}} as well as the above, but I think that's already working.

            Note that there's no command-line driver script for that task yet, but it's pretty straightforward to run from an interactive Python prompt.  Given that we know that DECam has buggy boresight information in the headers, I don't expect it to populate the database with good spatial regions yet, and I'm okay with fixing that on another ticket (probably by using {{astro_metadata_translator's}} on-the-fly header-patching system to just fix the boresight values to something we've fit for the test data we care about).

            Please ping #dm-middleware on Slack with any questions.
            In order to best support AP PipelineTask conversion work at PCW2019, we need to have a Gen3 data repository for {{ap_verify_hits2015}} (or maybe {{ap_verify_ci_hits2015}}, or both?).

            Anyhow, first step towards that is Gen3 Butler support for obs_decam, which is what this ticket is about.  That includes:
             * Writing an [Instrument|https://github.com/lsst/daf_butler/blob/7220349561d794e99e523aff29bdb5c3bd80f379/python/lsst/daf/butler/instrument.py#L36] subclass for DECam.  The docs for the base class should tell you most of what you need to do (though I've just discovered they're not appearing in the Sphinx HTML for some reason), but I'm sure a concrete example will be even more useful, which [HSC|https://github.com/lsst/obs_subaru/blob/e6ea06fa94726c5b0f3c7e020223bbc1a58a2d70/python/lsst/obs/subaru/gen3/hsc/instrument.py#L50] provides.  Note that because this will be only the second {{Instrument}} subclass, it's probable we'll find we want to adjust the ABC-subclass boundary a bit.
             * Writing a {{Formatter}} for raw DECam data (or if necessary, different formatters for different CCDs), probably by inheriting from [FitsRawFormatterBase|https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/fitsRawFormatterBase.py#L29].  Once again, you can use [HSC|https://github.com/lsst/obs_subaru/blob/e6ea06fa94726c5b0f3c7e020223bbc1a58a2d70/python/lsst/obs/subaru/gen3/hsc/rawFormatter.py#L38] as an example.  This should do the same thing as the Gen2 CameraMapper's {{std_raw}} method.  A test that shows the kind of consistency we expect (for HSC) can be found in [ci_hsc_gen2 |https://github.com/lsst/ci_hsc_gen2/blob/tickets/DM-19817/tests/test_butlerShims.py#L81]- note that the Gen3 version generally tries much harder to clean things up and make things are components are internally consistent, so it may not be possible to make them entirely consistent with the Gen2 version.  We may not have a good place to put such a test for DECam until we're further along converting an actual test dataset, but it'd be great to write one for one-off testing now and attach it to this ticket, with the expectation that we'll put it in a CI'd package in the future.

            After those steps (and after running the {{Instrument}} class's {{register}} method on a Gen3 repo), I think it should be possible to run the [Gen3 raw ingest task|https://github.com/lsst/obs_base/blob/master/python/lsst/obs/base/gen3/ingest.py] on DECam, and I consider that the end point for this ticket as long as any unexpected problems aren't huge.   This will rely a lot on the DECam translator in {{astro_metadata_translator}} as well as the above, but I think that's already working.

            Note that there's no command-line driver script for that task yet, but it's pretty straightforward to run from an interactive Python prompt.  Given that we know that DECam has buggy boresight information in the headers, I don't expect it to populate the database with good spatial regions yet, and I'm okay with fixing that on another ticket (probably by using {{astro_metadata_translator's}} on-the-fly header-patching system to just fix the boresight values to something we've fit for the test data we care about).

            Please ping #dm-middleware on Slack with any questions.
            swinbank John Swinbank made changes -
            Assignee John Parejko [ parejkoj ]
            swinbank John Swinbank made changes -
            Epic Link DM-20336 [ 337097 ]
            swinbank John Swinbank made changes -
            Sprint AP F19-2 [ 940 ]
            Parejkoj John Parejko made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-20842 [ DM-20842 ]
            Hide
            Parejkoj John Parejko added a comment -

            Notes from my discussion with Tim Jenness:

            Suggested making a Formatter.fileDescriptor property, and all subsequent things that get read from the fileDescriptor (e.g. metadata) can be made cached properties (e.g. fitsRawFormatterBase.metadata). Filed as DM-20842.

            Make FilterDefinition a dataclass in obs_base, and define the decam and hsc filters on it.

            What to do about detector, visit, and exposure max ids? They are currently defined in each obs package, but astro_metadata_translator should be able to know (and it should be the only location to compute the actual ids as well). We should at least be able to add tests of these to test_register().

            Show
            Parejkoj John Parejko added a comment - Notes from my discussion with Tim Jenness : Suggested making a Formatter.fileDescriptor property, and all subsequent things that get read from the fileDescriptor (e.g. metadata) can be made cached properties (e.g. fitsRawFormatterBase.metadata ). Filed as DM-20842 . Make FilterDefinition a dataclass in obs_base, and define the decam and hsc filters on it. What to do about detector, visit, and exposure max ids? They are currently defined in each obs package, but astro_metadata_translator should be able to know (and it should be the only location to compute the actual ids as well). We should at least be able to add tests of these to test_register() .
            Hide
            tjenness Tim Jenness added a comment -

            astro_metadata_translator knows how to calculate the exposure_id and detector_exposure_id but it currently duplicates logic from gen 2 butler to do it except for obs_lsst (where both gen2 and gen3 use astro_metadata_translator). This does need a bit of thought. It may be that we decide that exposure_id should be in astro_metadata_translator but detector_exposure_id is only ever useful for butler and therefore should only be in butler.

            Show
            tjenness Tim Jenness added a comment - astro_metadata_translator knows how to calculate the exposure_id and detector_exposure_id but it currently duplicates logic from gen 2 butler to do it except for obs_lsst (where both gen2 and gen3 use astro_metadata_translator). This does need a bit of thought. It may be that we decide that exposure_id should be in astro_metadata_translator but detector_exposure_id is only ever useful for butler and therefore should only be in butler.
            tjenness Tim Jenness made changes -
            Link This issue blocks DM-16297 [ DM-16297 ]
            Hide
            Parejkoj John Parejko added a comment -

            Also need to move the guts of test_ingest:HscIngestTestCase to obs_base and specialize it for DECam/HSC in setUp. The tempdir needs to be dealt with in IngestTestBase setUp/tearDown, and called via super.

            Show
            Parejkoj John Parejko added a comment - Also need to move the guts of test_ingest:HscIngestTestCase to obs_base and specialize it for DECam/HSC in setUp . The tempdir needs to be dealt with in IngestTestBase setUp/tearDown, and called via super.
            Hide
            Parejkoj John Parejko added a comment - - edited
            Show
            Parejkoj John Parejko added a comment - - edited First jenkins run on this ticket plus ci_hsc_gen2/ci_hsc_gen3: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30300/pipeline
            Hide
            Parejkoj John Parejko added a comment -

            Jim Bosch, Tim Jenness: I think it is time to have the two of you give an "official" look at this. There are still a few open questions left hanging in docstrings/comments, and I have not yet implemented writeCuratedCalibrations, but we should get a look at the overall design now. There are 4 PRs: daf_butler, obs_base, obs_decam, obs_subaru.

            I believe I've fixed the HSC filter bug I introduced yesterday: we can talk about what's going on there later, now that I've sorted it out. It gets to some questions as to how we name the filters between gen2 and gen3.

            Show
            Parejkoj John Parejko added a comment - Jim Bosch , Tim Jenness : I think it is time to have the two of you give an "official" look at this. There are still a few open questions left hanging in docstrings/comments, and I have not yet implemented writeCuratedCalibrations , but we should get a look at the overall design now. There are 4 PRs: daf_butler, obs_base, obs_decam, obs_subaru. I believe I've fixed the HSC filter bug I introduced yesterday: we can talk about what's going on there later, now that I've sorted it out. It gets to some questions as to how we name the filters between gen2 and gen3.
            Parejkoj John Parejko made changes -
            Reviewers Jim Bosch, Tim Jenness [ jbosch, tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-20994 [ DM-20994 ]
            Hide
            jbosch Jim Bosch added a comment -

            Comments on various PRs.  I would recommend looking at the comments for all packages before finishing work on any of them, because there were a few places where seeing an implementation led to comments about the interface.

            Show
            jbosch Jim Bosch added a comment - Comments on various PRs.  I would recommend looking at the comments for all packages before finishing work on any of them, because there were a few places where seeing an implementation led to comments about the interface.
            jbosch Jim Bosch made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-21016 [ DM-21016 ]
            Hide
            Parejkoj John Parejko added a comment - - edited
            Show
            Parejkoj John Parejko added a comment - - edited Standard jenkins run (without new ci_hsc_gen3): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30326/pipeline
            Hide
            Parejkoj John Parejko added a comment -

            A standard Jenkins run (without ci_hsc_gen3) passed with this version: I've made a tweak to jointcal's tests so that it behaves better with the FilterDefinitions. I think any better solution will have to wait on the implementation of RFC-624.

            Could someone please look at the jointcal PR? https://github.com/lsst/jointcal/pull/147

            We'll have to decide whether getting ci_hsc_gen3 to pass is a requirement for merging this or not.

            Show
            Parejkoj John Parejko added a comment - A standard Jenkins run (without ci_hsc_gen3) passed with this version: I've made a tweak to jointcal's tests so that it behaves better with the FilterDefinitions. I think any better solution will have to wait on the implementation of RFC-624 . Could someone please look at the jointcal PR? https://github.com/lsst/jointcal/pull/147 We'll have to decide whether getting ci_hsc_gen3 to pass is a requirement for merging this or not.
            Hide
            jbosch Jim Bosch added a comment -

            We'll have to decide whether getting ci_hsc_gen3 to pass is a requirement for merging this or not.

            It is.  It's not that we can actually live with having ci_hsc_gen3 broken; it's just that Nate Lust hasn't had a chance to announce it broadly yet, and we've gotten by okay anyway because the set of people likely to commit things that break it is pretty much the set of people who already know about it.  I'll announce it next week if he doesn't.

            Show
            jbosch Jim Bosch added a comment - We'll have to decide whether getting ci_hsc_gen3 to pass is a requirement for merging this or not. It is.  It's not that we can actually live with having ci_hsc_gen3 broken; it's just that Nate Lust hasn't had a chance to announce it broadly yet, and we've gotten by okay anyway because the set of people likely to commit things that break it is pretty much the set of people who already know about it.  I'll announce it next week if he doesn't.
            Hide
            jbosch Jim Bosch added a comment -

            jointcal branch has been reviewed.  Can you point me at the current problem in ci_hsc_gen3?  I may have some time today to help out with that, and I suspect my time would be better spent doing that than trying to juggle a bunch of branches on top of yours for the main ticket I'm working on.

            Show
            jbosch Jim Bosch added a comment - jointcal branch has been reviewed.  Can you point me at the current problem in ci_hsc_gen3?  I may have some time today to help out with that, and I suspect my time would be better spent doing that than trying to juggle a bunch of branches on top of yours for the main ticket I'm working on.
            Hide
            Parejkoj John Parejko added a comment - - edited
            Show
            Parejkoj John Parejko added a comment - - edited post-rebase, post Jim Bosch cleanup Jenkins run with both ci_hsc versions: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/30338/pipeline
            Hide
            Parejkoj John Parejko added a comment -

            A thank you to Jim Bosch and Tim Jenness for all their review and pair coding work, and extra thanks to Jim Bosch for doing the final grunt work of sorting out the colorterms and filterMap issues.

            Merged and done.

            Show
            Parejkoj John Parejko added a comment - A thank you to Jim Bosch and Tim Jenness for all their review and pair coding work, and extra thanks to Jim Bosch for doing the final grunt work of sorting out the colorterms and filterMap issues. Merged and done.
            Parejkoj John Parejko made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            swinbank John Swinbank made changes -
            Story Points 8 12
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-22708 [ DM-22708 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-28460 [ DM-28460 ]

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Jim Bosch, Tim Jenness
              Watchers:
              Eli Rykoff, Jim Bosch, John Parejko, John Swinbank, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.