Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29313

FilterLabel is not properly filled in for CFHT raws

    XMLWordPrintable

    Details

    • Story Points:
      4
    • Epic Link:
    • Sprint:
      AP S21-4 (March)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      There is a bug somewhere in how the CFHT filterLabels are filled out for raw files. To demonstrate this, go to lsst-devl:/scratch/parejkoj/cfht-gen2/lsst_ci and setup the following:

      setup -kr /project/shared/data/validation_data/validation_data_cfht/
      		 
      setup -kr /project/shared/data/validation_data/validation_data_decam/
      		 
      setup -kr /project/shared/data/test_data/testdata_subaru/
      		 
      setup -kr /project/shared/data/test_data/testdata_cfht/
      		 
      setup -kr /project/shared/data/test_data/testdata_decam/
      		 
      setup -kr .

      You can then run python check_filterLabel.py (I'll attach a copy to this post for future reference) in that directory to get the following output:

      $ python check_filterLabel.py 
       
      		 
      ExposureF.readFits before butler import:
      		 
      afw.image.MaskedImageFitsReader WARN: Expected extension type not found: IMAGE
      		 
      FilterLabel(physical="r.MP9601")
      		 
       
      		 
      CameraMapper INFO: Loading exposure registry from /scratch/parejkoj/cfht-gen2/lsst_ci/CfhtQuick/input/registry.sqlite3
      		 
       
      		 
      ExposureF.readFits after butler creation:
      		 
      afw.image.MaskedImageFitsReader WARN: Expected extension type not found: IMAGE
      		 
      FilterLabel(band="r", physical="r.MP9601")
      		 
       
      		 
      gen2 butler.get raw:
      		 
      CameraMapper WARN: Multiple matches for filter None with data ID 'r'.
      		 
      FilterLabel(band="r")
      		 
       
      		 
      gen2 butler.get calexp:
      		 
      CameraMapper WARN: Multiple matches for filter FilterLabel(band="r") with data ID 'r'.
      		 
      FilterLabel(band="r")

      In particular, that middle line is interesting to me: it looks like something correct is happening when the CFHT CameraMapper registers its filters (afw readFits goes from just having physical, to having both), but then the butler isn't catching them somehow.

      I don't know if we can create gen3 CFHT repos yet, so I don't know how to test that aspect.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-25629 Do release 22.0.0 of science pipelines

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-28597 Fix colorterm/photoCal filterLabel confusion

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-29335 FilterLabel of DC2 exposures are not populated with bandLabel

          • Invalid

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29446 DC2 singleFrameDriver Failure: FilterLabel(physical="z_sim_1.4") has no band.

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Parejkoj John Parejko created issue -
            Parejkoj John Parejko made changes -
            Field Original Value New Value
            Link This issue blocks DM-28597 [ DM-28597 ]
            Parejkoj John Parejko made changes -
            Description There is a bug somewhere in how the CFHT filterLabels are filled out for raw files. To demonstrate this, go to {{lsst-devl:/scratch/parejkoj/cfht-gen2/lsst_ci}} and setup the following:

            {code}
            setup -kr /project/shared/data/validation_data/validation_data_cfht/
            setup -kr /project/shared/data/validation_data/validation_data_decam/
            setup -kr /project/shared/data/test_data/testdata_subaru/
            setup -kr /project/shared/data/test_data/testdata_cfht/
            setup -kr /project/shared/data/test_data/testdata_decam/
            setup -kr .
            {code}

            You can then run {{python check_filterLabel.py}} (I'll attach a copy to this post for future reference) in that directory to get the following output:

            {code}
            $ python check_filterLabel.py

            ExposureF.readFits before butler import:
            afw.image.MaskedImageFitsReader WARN: Expected extension type not found: IMAGE
            FilterLabel(physical="r.MP9601")

            CameraMapper INFO: Loading exposure registry from /scratch/parejkoj/cfht-gen2/lsst_ci/CfhtQuick/input/registry.sqlite3

            ExposureF.readFits after butler creation:
            afw.image.MaskedImageFitsReader WARN: Expected extension type not found: IMAGE
            FilterLabel(band="r", physical="r.MP9601")

            gen2 butler.get raw:
            CameraMapper WARN: Multiple matches for filter None with data ID 'r'.
            FilterLabel(band="r")

            gen2 butler.get calexp:
            CameraMapper WARN: Multiple matches for filter FilterLabel(band="r") with data ID 'r'.
            FilterLabel(band="r")
            {code}

            In particular, that middle line is interesting to me: it looks like something correct is happening when the CFHT CameraMapper registers its filters (afw readFits goes from just having physical, to having both), but then the butler isn't catching them somehow.             		There is a bug somewhere in how the CFHT filterLabels are filled out for raw files. To demonstrate this, go to {{lsst-devl:/scratch/parejkoj/cfht-gen2/lsst_ci}} and setup the following:

            {code}
            setup -kr /project/shared/data/validation_data/validation_data_cfht/
            setup -kr /project/shared/data/validation_data/validation_data_decam/
            setup -kr /project/shared/data/test_data/testdata_subaru/
            setup -kr /project/shared/data/test_data/testdata_cfht/
            setup -kr /project/shared/data/test_data/testdata_decam/
            setup -kr .
            {code}

            You can then run {{python check_filterLabel.py}} (I'll attach a copy to this post for future reference) in that directory to get the following output:

            {code}
            $ python check_filterLabel.py

            ExposureF.readFits before butler import:
            afw.image.MaskedImageFitsReader WARN: Expected extension type not found: IMAGE
            FilterLabel(physical="r.MP9601")

            CameraMapper INFO: Loading exposure registry from /scratch/parejkoj/cfht-gen2/lsst_ci/CfhtQuick/input/registry.sqlite3

            ExposureF.readFits after butler creation:
            afw.image.MaskedImageFitsReader WARN: Expected extension type not found: IMAGE
            FilterLabel(band="r", physical="r.MP9601")

            gen2 butler.get raw:
            CameraMapper WARN: Multiple matches for filter None with data ID 'r'.
            FilterLabel(band="r")

            gen2 butler.get calexp:
            CameraMapper WARN: Multiple matches for filter FilterLabel(band="r") with data ID 'r'.
            FilterLabel(band="r")
            {code}

            In particular, that middle line is interesting to me: it looks like something correct is happening when the CFHT CameraMapper registers its filters (afw readFits goes from just having physical, to having both), but then the butler isn't catching them somehow.

            I don't know if we can create gen3 CFHT repos yet, so I don't know how to test that aspect.
            Parejkoj John Parejko made changes -
            Attachment check_filterLabel.py [ 48496 ]
            sullivan Ian Sullivan made changes -
            Labels SciencePipelines
            krzys Krzysztof Findeisen made changes -
            Assignee Krzysztof Findeisen [ krzys ]
            krzys Krzysztof Findeisen made changes -
            Epic Link DM-29214 [ 459218 ]
            Sprint AP S21-5 (April) [ 1084 ]
            Team Alert Production [ 10300 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-29335 [ DM-29335 ]
            krzys Krzysztof Findeisen made changes -
            Issue Type Story [ 10001 ] Bug [ 1 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue blocks DM-25629 [ DM-25629 ]
            krzys Krzysztof Findeisen made changes -
            Rank Ranked higher
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Sprint AP S21-5 (April) [ 1084 ] AP S21-4 (March) [ 1079 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-29446 [ DM-29446 ]
            krzys Krzysztof Findeisen made changes -
            Reviewers John Parejko [ parejkoj ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            krzys Krzysztof Findeisen made changes -
            Story Points 4
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              John Parejko
              Watchers:
              Ian Sullivan, John Parejko, Krzysztof Findeisen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.