Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28046

Update PhosimEimgParseTask to use LsstCam

    XMLWordPrintable

    Details

      Description

      To allow ingest of phosim-simulated e-images, `obs_lsst/config/phosim/ingestEimg.py` was used as config for ingestImages.py , but with an update to the phosim repackager to emulate LsstCam headers,  a change is needed to use LsstCamMapper for header translation  (PhosimParseTask to LsstCamParseTask in  https://github.com/lsst/obs_lsst/blob/61239df17fc0cad6f2250682df420affea96f534/python/lsst/obs/lsst/phosim.py#L62  )

      Otherwise an incorrect set of header translators is used : 

      ingestImages.py /project/scichris/repackagerTest/input_w48/ /project/scichris/repackagerTest/intraRpkgd/*.fits  --configfile /project/scichris/dev/obs_lsst/config/phosim/ingestEimg.py
      

      yields 

      root INFO: Loading config overrride file '/project/scichris/dev/obs_lsst_dev/config/ingest.py'
      root INFO: Loading config overrride file '/project/scichris/dev/obs_lsst_dev/config/lsstCam/ingest.py'
      LsstCamMapper WARN: Unable to find valid calib root directory
      CameraMapper INFO: Loading Posix exposure registry from /project/scichris/repackagerTest/input_w48
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_unique_name' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['SENSNAME'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_group' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['RAFTNAME'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_num' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['RAFTNAME'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_name' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['SENSNAME'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'boresight_rotation_angle' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['ROTANGZ', 'ROTANGLE'] in header"
      astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_exposure_id' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['RAFTNAME'] in header"
      ingest WARN: Failed to ingest file /project/scichris/repackagerTest/intraRpkgd/MC_H_20000217_000031_R00_SW0.fits: 'detector'
      Traceback (most recent call last):
        File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 625, in run
          self.runFile(infile, registry, args, pos)
        File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 598, in runFile
          if registry is not None and self.register.check(registry, fileInfo):
        File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 352, in check
          values = [self.typemap[self.config.columns[col]](info[col]) for col in self.config.unique]
        File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 352, in <listcomp>
          values = [self.typemap[self.config.columns[col]](info[col]) for col in self.config.unique]
      KeyError: 'detector'The above exception was the direct cause of the following exception:Traceback (most recent call last):
        File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/bin/ingestImages.py", line 3, in <module>
          IngestTask.parseAndRun()
        File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 431, in parseAndRun
          task.run(args)
        File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 629, in run
          raise IngestError(f"Failed to ingest file {infile}", infile, pos) from exc
      lsst.pipe.tasks.ingest.IngestError: Failed to ingest file /project/scichris/repackagerTest/intraRpkgd/MC_H_20000217_000031_R00_SW0.fits
      

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Looks fine although I'm worried about old data now being broken and that there is no test that new data header parsing works.

            Show
            tjenness Tim Jenness added a comment - Looks fine although I'm worried about old data now being broken and that there is no test that new data header parsing works.
            Hide
            ksuberlak Krzysztof Suberlak added a comment - - edited

            I replied in the github thread: as long as we keep the name of the ingestEimg.py config file  https://github.com/lsst/obs_lsst/blob/master/config/phosim/ingestEimg.py for ingest of e-images, then no changes are needed in ts_wep ( in https://github.com/lsst-ts/ts_wep/blob/08d0c2c4c0f6ed37fcc86af9ac09386469ae8824/python/lsst/ts/wep/CamDataCollector.py#L105 ) .

            Updated ingestEimg.py could contain eg. 

            from lsst.obs.lsst.phosim import LsstCamEimgParseTask
            config.parse.retarget(LsstCamEimgParseTask)
            

            Then instead of changing PhosimEImgParseTask in phosim.py https://github.com/lsst/obs_lsst/blob/ecc98f4876efebeaa34fb9ca4440296b4a120763/python/lsst/obs/lsst/phosim.py#L62  and deleting (or needing to change) test_parsetask_phosim_eimg_translator in test_parsetask.py https://github.com/lsst/obs_lsst/blob/ecc98f4876efebeaa34fb9ca4440296b4a120763/tests/test_parsetask.py#L317 one could add there a test_parsetask_lsstCam_eimg_translator . That way nothing that uses PhosimEImgParseTask would break.

            Would the preferred location of LsstCamEImgParseTask be in obs/lsst/ingest.py

            https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/ingest.py (so it's in the same place as LsstCamParseTask, and we are importing above

            from lsst.obs.lsst.ingest import LsstCamEimgParseTask) , or in obs/lsst/phosim.py

            https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/phosim.py (since it pertains to data created by phosim)?

            I think the former (since in the updated repackager we are making phosim data look like lsstCam data), but I am open for other suggestions. It is understood to be temporary, while the ts_aos still uses Gen2 Butler in  ts_wep,  as the next major move will be to update to Gen3.

            Show
            ksuberlak Krzysztof Suberlak added a comment - - edited I replied in the github thread: as long as we keep the name of the ingestEimg.py config file  https://github.com/lsst/obs_lsst/blob/master/config/phosim/ingestEimg.py  for ingest of e-images, then no changes are needed in ts_wep ( in https://github.com/lsst-ts/ts_wep/blob/08d0c2c4c0f6ed37fcc86af9ac09386469ae8824/python/lsst/ts/wep/CamDataCollector.py#L105  ) . Updated ingestEimg.py could contain eg.  from lsst.obs.lsst.phosim import LsstCamEimgParseTask config.parse.retarget(LsstCamEimgParseTask) Then instead of changing PhosimEImgParseTask in phosim.py https://github.com/lsst/obs_lsst/blob/ecc98f4876efebeaa34fb9ca4440296b4a120763/python/lsst/obs/lsst/phosim.py#L62   and deleting (or needing to change) test_parsetask_phosim_eimg_translator in test_parsetask.py   https://github.com/lsst/obs_lsst/blob/ecc98f4876efebeaa34fb9ca4440296b4a120763/tests/test_parsetask.py#L317  one could add there a test_parsetask_lsstCam_eimg_translator . That way nothing that uses PhosimEImgParseTask would break. Would the preferred location of LsstCamEImgParseTask be in  obs/lsst/ingest.py https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/ingest.py (so it's in the same place as LsstCamParseTask , and we are importing above from lsst.obs.lsst.ingest import LsstCamEimgParseTask ) , or in  obs/lsst/phosim.py https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/phosim.py (since it pertains to data created by phosim)? I think the former (since in the updated repackager we are making phosim data look like lsstCam data), but I am open for other suggestions. It is understood to be temporary, while the ts_aos  still uses Gen2 Butler in  ts_wep ,  as the next major move will be to update to Gen3.
            Hide
            ksuberlak Krzysztof Suberlak added a comment -

            Considering that we will need to simulate phosim-LsstComCam-like images that also make e-images, there will be a place for LsstComCamEimgParseTask. Thus I made LsstCamEimgParseTask, adding the testfile in config, and a test for header translation.

            Show
            ksuberlak Krzysztof Suberlak added a comment - Considering that we will need to simulate phosim-LsstComCam-like images that also make e-images, there will be a place for LsstComCamEimgParseTask . Thus I made LsstCamEimgParseTask , adding the testfile in config, and a test for header translation.
            Hide
            ksuberlak Krzysztof Suberlak added a comment -

            I close it since the PR277 got merged

            Show
            ksuberlak Krzysztof Suberlak added a comment - I close it since the PR277 got merged

              People

              Assignee:
              ksuberlak Krzysztof Suberlak
              Reporter:
              ksuberlak Krzysztof Suberlak
              Reviewers:
              Tim Jenness
              Watchers:
              Kian-Tat Lim, Krzysztof Suberlak, Robert Lupton, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.