Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11786

Run meas_mosaic on acceptance data

    Details

    • Story Points:
      4
    • Sprint:
      DRP F17-6, DRP S18-1
    • Team:
      Data Release Production

      Description

      Run meas_mosaic on the acceptance data, and produce catalogs that are ready to be validated via validate_drp.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            I think you are getting these messages because you are looking for things that shouldn't exist. For example, in your

            /project/parejkoj/DM-11783/logs/validate-meas_mosaic-SSP_WIDE_9698_HSC-R-105878.1.log 

            I see

            No locations for get: datasetType:photoCalib dataId:DataId(initialdata={'ccd': 0, 'filter': 'HSC-R', 'tract': 
            9698, 'field': 'SSP_WIDE', 'visit': 34642, 'dateObs': '2015-07-15', 'taiObs': '2015-07-15', 'expTime': 30.0, '
            pointing': 1291}, tag=set())
            

            Indeed, it's not there:

            lauren@lsst-dev01:/datasets/hsc/repo/rerun/private/lauren/DM-11786/jointcal-results/9698 $ ls photoCalib-0034642*
            photoCalib-0034642-070.fits  photoCalib-0034642-081.fits  photoCalib-0034642-092.fits
            photoCalib-0034642-071.fits  photoCalib-0034642-082.fits  photoCalib-0034642-093.fits
            photoCalib-0034642-072.fits  photoCalib-0034642-084.fits  photoCalib-0034642-094.fits
            photoCalib-0034642-073.fits  photoCalib-0034642-085.fits  photoCalib-0034642-096.fits
            photoCalib-0034642-074.fits  photoCalib-0034642-086.fits  photoCalib-0034642-097.fits
            photoCalib-0034642-075.fits  photoCalib-0034642-087.fits  photoCalib-0034642-098.fits
            photoCalib-0034642-078.fits  photoCalib-0034642-088.fits  photoCalib-0034642-099.fits
            photoCalib-0034642-079.fits  photoCalib-0034642-090.fits  photoCalib-0034642-102.fits
            photoCalib-0034642-080.fits  photoCalib-0034642-091.fits
            

            HOWEVER, this is as it should be since ccd 0 of visit 34642 doesn't overlap tract 9698:

            so meas_mosaic does not produce an output for it.

            To get an idea for numbers, the following shows the visit/tract overlap for all visits in the tract 9698 list:

            So, indeed, if you are searching through the full 0..8^10..103 list of ccds for all visits, you will get A LOT of those "no matching visits" messages!

            Show
            lauren Lauren MacArthur added a comment - I think you are getting these messages because you are looking for things that shouldn't exist. For example, in your /project/parejkoj/DM-11783/logs/validate-meas_mosaic-SSP_WIDE_9698_HSC-R-105878.1.log I see No locations for get: datasetType:photoCalib dataId:DataId(initialdata={'ccd': 0, 'filter': 'HSC-R', 'tract': 9698, 'field': 'SSP_WIDE', 'visit': 34642, 'dateObs': '2015-07-15', 'taiObs': '2015-07-15', 'expTime': 30.0, ' pointing': 1291}, tag=set()) Indeed, it's not there: lauren@lsst-dev01:/datasets/hsc/repo/rerun/private/lauren/DM-11786/jointcal-results/9698 $ ls photoCalib-0034642* photoCalib-0034642-070.fits photoCalib-0034642-081.fits photoCalib-0034642-092.fits photoCalib-0034642-071.fits photoCalib-0034642-082.fits photoCalib-0034642-093.fits photoCalib-0034642-072.fits photoCalib-0034642-084.fits photoCalib-0034642-094.fits photoCalib-0034642-073.fits photoCalib-0034642-085.fits photoCalib-0034642-096.fits photoCalib-0034642-074.fits photoCalib-0034642-086.fits photoCalib-0034642-097.fits photoCalib-0034642-075.fits photoCalib-0034642-087.fits photoCalib-0034642-098.fits photoCalib-0034642-078.fits photoCalib-0034642-088.fits photoCalib-0034642-099.fits photoCalib-0034642-079.fits photoCalib-0034642-090.fits photoCalib-0034642-102.fits photoCalib-0034642-080.fits photoCalib-0034642-091.fits HOWEVER, this is as it should be since ccd 0 of visit 34642 doesn't overlap tract 9698: so meas_mosaic does not produce an output for it. To get an idea for numbers, the following shows the visit/tract overlap for all visits in the tract 9698 list: So, indeed, if you are searching through the full 0..8^10..103 list of ccds for all visits, you will get A LOT of those "no matching visits" messages!
            Hide
            Parejkoj John Parejko added a comment -

            Interesting. meas_mosaic behaves differently from jointcal in this regard: jointcal will process and produce output for all ccds in any visit that is included in the tract, whether each particular ccd overlaps the tract or not, since it's using them as part of the fit anyway.

            Am I going to have to just live with this, or is there a better approach? I'm currently using Hsin-Fang Chiang's sqlite files to build the visit lists per tract, which is how the script files I listed above are created.

            Show
            Parejkoj John Parejko added a comment - Interesting. meas_mosaic behaves differently from jointcal in this regard: jointcal will process and produce output for all ccds in any visit that is included in the tract, whether each particular ccd overlaps the tract or not, since it's using them as part of the fit anyway. Am I going to have to just live with this, or is there a better approach? I'm currently using Hsin-Fang Chiang 's sqlite files to build the visit lists per tract, which is how the script files I listed above are created.
            Hide
            lauren Lauren MacArthur added a comment -

            I think the philosophy is along the lines of "why persist them if they will not be used?" You do need the full visit list per tract, but you could sub-select on the ccds for each visit that actually overlap the tract/patch of interest. I believe the coadd assembly does this using:
            https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/coaddBase.py#L103
            which gets called here:
            https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeCoaddTempExp.py#L277

            Show
            lauren Lauren MacArthur added a comment - I think the philosophy is along the lines of "why persist them if they will not be used?" You do need the full visit list per tract, but you could sub-select on the ccds for each visit that actually overlap the tract/patch of interest. I believe the coadd assembly does this using: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/coaddBase.py#L103 which gets called here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeCoaddTempExp.py#L277
            Hide
            Parejkoj John Parejko added a comment -

            I have successfully run all of this data through validate_drp's matchedVisitsMetricsTask, so I think we're good here.

            The one thing I would recommend is that you move any relevant files (datasets, processing scripts, etc.) from /scratch to /project so we don't lose them in a deletion event on scratch.

            Show
            Parejkoj John Parejko added a comment - I have successfully run all of this data through validate_drp's matchedVisitsMetricsTask, so I think we're good here. The one thing I would recommend is that you move any relevant files (datasets, processing scripts, etc.) from /scratch to /project so we don't lose them in a deletion event on scratch.
            Hide
            lauren Lauren MacArthur added a comment -

            Great, thanks John. I will certainly heed your advice on the file shuffling!

            Show
            lauren Lauren MacArthur added a comment - Great, thanks John. I will certainly heed your advice on the file shuffling!

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                John Parejko
                Watchers:
                Hsin-Fang Chiang, Jim Bosch, John Parejko, John Swinbank, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel