[DM-28046] Update PhosimEimgParseTask to use LsstCam - Jira

XML

Word

Printable

Details

Type: Improvement
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: obs_lsst
Labels:
- SciencePipelines

Story Points:
6
Epic Link:
Integrate WEP into DM stack
Team:
Telescope and Site

Description

To allow ingest of phosim-simulated e-images, `obs_lsst/config/phosim/ingestEimg.py` was used as config for ingestImages.py , but with an update to the phosim repackager to emulate LsstCam headers, a change is needed to use LsstCamMapper for header translation (PhosimParseTask to LsstCamParseTask in https://github.com/lsst/obs_lsst/blob/61239df17fc0cad6f2250682df420affea96f534/python/lsst/obs/lsst/phosim.py#L62 )

Otherwise an incorrect set of header translators is used :

ingestImages.py /project/scichris/repackagerTest/input_w48/ /project/scichris/repackagerTest/intraRpkgd/*.fits  --configfile /project/scichris/dev/obs_lsst/config/phosim/ingestEimg.py

yields

root INFO: Loading config overrride file '/project/scichris/dev/obs_lsst_dev/config/ingest.py'

root INFO: Loading config overrride file '/project/scichris/dev/obs_lsst_dev/config/lsstCam/ingest.py'

LsstCamMapper WARN: Unable to find valid calib root directory

CameraMapper INFO: Loading Posix exposure registry from /project/scichris/repackagerTest/input_w48

astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_unique_name' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['SENSNAME'] in header"

astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_group' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['RAFTNAME'] in header"

astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_num' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['RAFTNAME'] in header"

astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_name' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['SENSNAME'] in header"

astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'boresight_rotation_angle' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['ROTANGZ', 'ROTANGLE'] in header"

astro_metadata_translator.observationInfo WARN: Ignoring Error calculating property 'detector_exposure_id' using translator <class 'lsst.obs.lsst.translators.phosim.LsstCamPhoSimTranslator'>: "Could not find ['RAFTNAME'] in header"

ingest WARN: Failed to ingest file /project/scichris/repackagerTest/intraRpkgd/MC_H_20000217_000031_R00_SW0.fits: 'detector'

Traceback (most recent call last):

  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 625, in run

    self.runFile(infile, registry, args, pos)

  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 598, in runFile

    if registry is not None and self.register.check(registry, fileInfo):

  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 352, in check

    values = [self.typemap[self.config.columns[col]](info[col]) for col in self.config.unique]

  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 352, in <listcomp>

    values = [self.typemap[self.config.columns[col]](info[col]) for col in self.config.unique]

KeyError: 'detector'The above exception was the direct cause of the following exception:Traceback (most recent call last):

  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/bin/ingestImages.py", line 3, in <module>

    IngestTask.parseAndRun()

  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 431, in parseAndRun

    task.run(args)

  File "/opt/lsst/software/stack/stack/miniconda3-py37_4.8.2-cb4e2dc/Linux64/pipe_tasks/21.0.0-12-gb58d4cc3+0fdaeca7f2/python/lsst/pipe/tasks/ingest.py", line 629, in run

    raise IngestError(f"Failed to ingest file {infile}", infile, pos) from exc

lsst.pipe.tasks.ingest.IngestError: Failed to ingest file /project/scichris/repackagerTest/intraRpkgd/MC_H_20000217_000031_R00_SW0.fits

Attachments

Issue Links

is child task of

DM-28556 Adapt the Latest File Format of Corner WFS

Done

Activity

Ascending order - Click to sort in descending order

No builds found.

Krzysztof Suberlak created issue - 16/Dec/20 4:58 AM

Krzysztof Suberlak made changes - 16/Dec/20 4:58 AM

Field	Original Value	New Value
Link		This issue has to be done before ~~DM-26836~~ [ ~~DM-26836~~ ]

Krzysztof Suberlak made changes - 22/Dec/20 10:40 PM

Status

To Do [ 10001 ]

In Progress [ 3 ]

Krzysztof Suberlak made changes - 22/Dec/20 10:58 PM

Reviewers		Tim Jenness [ tjenness ]
Status	In Progress [ 3 ]	In Review [ 10004 ]

Hide

Permalink

Tim Jenness added a comment - 05/Jan/21 9:35 PM

Looks fine although I'm worried about old data now being broken and that there is no test that new data header parsing works.

Show

Tim Jenness added a comment - 05/Jan/21 9:35 PM Looks fine although I'm worried about old data now being broken and that there is no test that new data header parsing works.

Tim Jenness made changes - 05/Jan/21 9:35 PM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Krzysztof Suberlak made changes - 29/Jan/21 11:20 PM

Link

This issue is child task of ~~DM-28556~~ [ ~~DM-28556~~ ]

Krzysztof Suberlak made changes - 29/Jan/21 11:20 PM

Epic Link

~~DM-28335~~ [ 445220 ]

Krzysztof Suberlak made changes - 29/Jan/21 11:29 PM

Link

This issue has to be done before ~~DM-26836~~ [ ~~DM-26836~~ ]

Hide

Permalink

Krzysztof Suberlak added a comment - 29/Jan/21 11:56 PM - edited

I replied in the github thread: as long as we keep the name of the ingestEimg.py config file https://github.com/lsst/obs_lsst/blob/master/config/phosim/ingestEimg.py for ingest of e-images, then no changes are needed in ts_wep ( in https://github.com/lsst-ts/ts_wep/blob/08d0c2c4c0f6ed37fcc86af9ac09386469ae8824/python/lsst/ts/wep/CamDataCollector.py#L105 ) .

Updated ingestEimg.py could contain eg.

from lsst.obs.lsst.phosim import LsstCamEimgParseTask

config.parse.retarget(LsstCamEimgParseTask)

Then instead of changing PhosimEImgParseTask in phosim.py https://github.com/lsst/obs_lsst/blob/ecc98f4876efebeaa34fb9ca4440296b4a120763/python/lsst/obs/lsst/phosim.py#L62 and deleting (or needing to change) test_parsetask_phosim_eimg_translator in test_parsetask.py https://github.com/lsst/obs_lsst/blob/ecc98f4876efebeaa34fb9ca4440296b4a120763/tests/test_parsetask.py#L317 one could add there a test_parsetask_lsstCam_eimg_translator . That way nothing that uses PhosimEImgParseTask would break.

Would the preferred location of LsstCamEImgParseTask be in obs/lsst/ingest.py

https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/ingest.py (so it's in the same place as LsstCamParseTask, and we are importing above

from lsst.obs.lsst.ingest import LsstCamEimgParseTask) , or in obs/lsst/phosim.py

https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/phosim.py (since it pertains to data created by phosim)?

I think the former (since in the updated repackager we are making phosim data look like lsstCam data), but I am open for other suggestions. It is understood to be temporary, while the ts_aos still uses Gen2 Butler in ts_wep, as the next major move will be to update to Gen3.

Show

Krzysztof Suberlak added a comment - 29/Jan/21 11:56 PM - edited I replied in the github thread: as long as we keep the name of the ingestEimg.py config file https://github.com/lsst/obs_lsst/blob/master/config/phosim/ingestEimg.py for ingest of e-images, then no changes are needed in ts_wep ( in https://github.com/lsst-ts/ts_wep/blob/08d0c2c4c0f6ed37fcc86af9ac09386469ae8824/python/lsst/ts/wep/CamDataCollector.py#L105 ) . Updated ingestEimg.py could contain eg. from lsst.obs.lsst.phosim import LsstCamEimgParseTask config.parse.retarget(LsstCamEimgParseTask) Then instead of changing PhosimEImgParseTask in phosim.py https://github.com/lsst/obs_lsst/blob/ecc98f4876efebeaa34fb9ca4440296b4a120763/python/lsst/obs/lsst/phosim.py#L62 and deleting (or needing to change) test_parsetask_phosim_eimg_translator in test_parsetask.py https://github.com/lsst/obs_lsst/blob/ecc98f4876efebeaa34fb9ca4440296b4a120763/tests/test_parsetask.py#L317 one could add there a test_parsetask_lsstCam_eimg_translator . That way nothing that uses PhosimEImgParseTask would break. Would the preferred location of LsstCamEImgParseTask be in obs/lsst/ingest.py https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/ingest.py (so it's in the same place as LsstCamParseTask , and we are importing above from lsst.obs.lsst.ingest import LsstCamEimgParseTask ) , or in obs/lsst/phosim.py https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/phosim.py (since it pertains to data created by phosim)? I think the former (since in the updated repackager we are making phosim data look like lsstCam data), but I am open for other suggestions. It is understood to be temporary, while the ts_aos still uses Gen2 Butler in ts_wep , as the next major move will be to update to Gen3.

Hide

Permalink

Krzysztof Suberlak added a comment - 25/Feb/21 3:23 AM

Considering that we will need to simulate phosim-LsstComCam-like images that also make e-images, there will be a place for LsstComCamEimgParseTask. Thus I made LsstCamEimgParseTask, adding the testfile in config, and a test for header translation.

Show

Krzysztof Suberlak added a comment - 25/Feb/21 3:23 AM Considering that we will need to simulate phosim-LsstComCam-like images that also make e-images, there will be a place for LsstComCamEimgParseTask . Thus I made LsstCamEimgParseTask , adding the testfile in config, and a test for header translation.

Krzysztof Suberlak made changes - 01/Mar/21 6:56 PM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

Hide

Permalink

Krzysztof Suberlak added a comment - 01/Mar/21 6:57 PM

I close it since the PR277 got merged

Show

Krzysztof Suberlak added a comment - 01/Mar/21 6:57 PM I close it since the PR277 got merged

Andrew Connolly made changes - 29/Sep/21 9:01 PM

Story Points

People

Assignee:: Krzysztof Suberlak

Reporter:: Krzysztof Suberlak

Reviewers:: Tim Jenness

Watchers:: Kian-Tat Lim, Krzysztof Suberlak, Robert Lupton, Tim Jenness

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 16/Dec/20 4:58 AM

Updated:: 29/Sep/21 9:01 PM

Resolved:: 01/Mar/21 6:56 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Issue Links

Activity

People

Dates

Jenkins