# Run meas_mosaic on acceptance data

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
4
• Sprint:
DRP F17-6, DRP S18-1
• Team:
Data Release Production

## Description

Run meas_mosaic on the acceptance data, and produce catalogs that are ready to be validated via validate_drp.

## Attachments

1. WIDE_9698_34642.png
69 kB
2. WIDE_9698_allVisits.png
310 kB

## Activity

Hide
Lauren MacArthur added a comment -

I think you are getting these messages because you are looking for things that shouldn't exist. For example, in your

 /project/parejkoj/DM-11783/logs/validate-meas_mosaic-SSP_WIDE_9698_HSC-R-105878.1.log

I see

 No locations for get: datasetType:photoCalib dataId:DataId(initialdata={'ccd': 0, 'filter': 'HSC-R', 'tract':  9698, 'field': 'SSP_WIDE', 'visit': 34642, 'dateObs': '2015-07-15', 'taiObs': '2015-07-15', 'expTime': 30.0, ' pointing': 1291}, tag=set()) 

Indeed, it's not there:

 lauren@lsst-dev01:/datasets/hsc/repo/rerun/private/lauren/DM-11786/jointcal-results/9698 $ls photoCalib-0034642* photoCalib-0034642-070.fits photoCalib-0034642-081.fits photoCalib-0034642-092.fits photoCalib-0034642-071.fits photoCalib-0034642-082.fits photoCalib-0034642-093.fits photoCalib-0034642-072.fits photoCalib-0034642-084.fits photoCalib-0034642-094.fits photoCalib-0034642-073.fits photoCalib-0034642-085.fits photoCalib-0034642-096.fits photoCalib-0034642-074.fits photoCalib-0034642-086.fits photoCalib-0034642-097.fits photoCalib-0034642-075.fits photoCalib-0034642-087.fits photoCalib-0034642-098.fits photoCalib-0034642-078.fits photoCalib-0034642-088.fits photoCalib-0034642-099.fits photoCalib-0034642-079.fits photoCalib-0034642-090.fits photoCalib-0034642-102.fits photoCalib-0034642-080.fits photoCalib-0034642-091.fits  HOWEVER, this is as it should be since ccd 0 of visit 34642 doesn't overlap tract 9698: so meas_mosaic does not produce an output for it. To get an idea for numbers, the following shows the visit/tract overlap for all visits in the tract 9698 list: So, indeed, if you are searching through the full 0..8^10..103 list of ccds for all visits, you will get A LOT of those "no matching visits" messages! Show Lauren MacArthur added a comment - I think you are getting these messages because you are looking for things that shouldn't exist. For example, in your /project/parejkoj/DM-11783/logs/validate-meas_mosaic-SSP_WIDE_9698_HSC-R-105878.1.log I see No locations for get: datasetType:photoCalib dataId:DataId(initialdata={'ccd': 0, 'filter': 'HSC-R', 'tract': 9698, 'field': 'SSP_WIDE', 'visit': 34642, 'dateObs': '2015-07-15', 'taiObs': '2015-07-15', 'expTime': 30.0, ' pointing': 1291}, tag=set()) Indeed, it's not there: lauren@lsst-dev01:/datasets/hsc/repo/rerun/private/lauren/DM-11786/jointcal-results/9698$ ls photoCalib-0034642* photoCalib-0034642-070.fits photoCalib-0034642-081.fits photoCalib-0034642-092.fits photoCalib-0034642-071.fits photoCalib-0034642-082.fits photoCalib-0034642-093.fits photoCalib-0034642-072.fits photoCalib-0034642-084.fits photoCalib-0034642-094.fits photoCalib-0034642-073.fits photoCalib-0034642-085.fits photoCalib-0034642-096.fits photoCalib-0034642-074.fits photoCalib-0034642-086.fits photoCalib-0034642-097.fits photoCalib-0034642-075.fits photoCalib-0034642-087.fits photoCalib-0034642-098.fits photoCalib-0034642-078.fits photoCalib-0034642-088.fits photoCalib-0034642-099.fits photoCalib-0034642-079.fits photoCalib-0034642-090.fits photoCalib-0034642-102.fits photoCalib-0034642-080.fits photoCalib-0034642-091.fits HOWEVER, this is as it should be since ccd 0 of visit 34642 doesn't overlap tract 9698: so meas_mosaic does not produce an output for it. To get an idea for numbers, the following shows the visit/tract overlap for all visits in the tract 9698 list: So, indeed, if you are searching through the full 0..8^10..103 list of ccds for all visits, you will get A LOT of those "no matching visits" messages!
Hide
John Parejko added a comment -

Interesting. meas_mosaic behaves differently from jointcal in this regard: jointcal will process and produce output for all ccds in any visit that is included in the tract, whether each particular ccd overlaps the tract or not, since it's using them as part of the fit anyway.

Am I going to have to just live with this, or is there a better approach? I'm currently using Hsin-Fang Chiang's sqlite files to build the visit lists per tract, which is how the script files I listed above are created.

Show
John Parejko added a comment - Interesting. meas_mosaic behaves differently from jointcal in this regard: jointcal will process and produce output for all ccds in any visit that is included in the tract, whether each particular ccd overlaps the tract or not, since it's using them as part of the fit anyway. Am I going to have to just live with this, or is there a better approach? I'm currently using Hsin-Fang Chiang 's sqlite files to build the visit lists per tract, which is how the script files I listed above are created.
Hide
Lauren MacArthur added a comment -

I think the philosophy is along the lines of "why persist them if they will not be used?" You do need the full visit list per tract, but you could sub-select on the ccds for each visit that actually overlap the tract/patch of interest. I believe the coadd assembly does this using:
which gets called here:

Show
Lauren MacArthur added a comment - I think the philosophy is along the lines of "why persist them if they will not be used?" You do need the full visit list per tract, but you could sub-select on the ccds for each visit that actually overlap the tract/patch of interest. I believe the coadd assembly does this using: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/coaddBase.py#L103 which gets called here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeCoaddTempExp.py#L277
Hide
John Parejko added a comment -

I have successfully run all of this data through validate_drp's matchedVisitsMetricsTask, so I think we're good here.

The one thing I would recommend is that you move any relevant files (datasets, processing scripts, etc.) from /scratch to /project so we don't lose them in a deletion event on scratch.

Show
John Parejko added a comment - I have successfully run all of this data through validate_drp's matchedVisitsMetricsTask, so I think we're good here. The one thing I would recommend is that you move any relevant files (datasets, processing scripts, etc.) from /scratch to /project so we don't lose them in a deletion event on scratch.
Hide
Lauren MacArthur added a comment -

Great, thanks John. I will certainly heed your advice on the file shuffling!

Show
Lauren MacArthur added a comment - Great, thanks John. I will certainly heed your advice on the file shuffling!

## People

• Assignee:
Lauren MacArthur
Reporter:
John Parejko
Reviewers:
John Parejko
Watchers:
Hsin-Fang Chiang, Jim Bosch, John Parejko, John Swinbank, Lauren MacArthur