Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10976

Get HiTS 2014 data

    XMLWordPrintable

    Details

    • Story Points:
      5
    • Sprint:
      Alert Production F17 - 7, Alert Production F17 - 8
    • Team:
      Alert Production

      Description

      • Get HiTS data for 2014 fields matching 2015 fields for the prototype AP system (DM-7925), to be used for template generation
      • Once the dataset exists, make a git-lfs repo for it and/or add it to /datasets on lsst-dev (along with the 2015 field data to be used as "science"). This should adhere to the layout defined in DM-11116.

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            The filenames are different for the 2014 data compared to 2015, and there are fewer MasterCals than in 2015. It looks like the MasterCal flats and biases are the tu*.fits.fz files. In 2015, there is a full set of biases and flats for 9 different dates. In 2014, there are only 5 different dates (thus fewer files). I have organized the 2014 HiTS data into raw and calib subdirectories as in ap_verify_hits2015. They are located on lsst-dev at /project/mrawls/ap_verify_2014hits. I am running the prototype pipeline on them (ingestion only for starters) and the output is being written to /project/mrawls/output_2014hits.

            Show
            mrawls Meredith Rawls added a comment - The filenames are different for the 2014 data compared to 2015, and there are fewer MasterCals than in 2015. It looks like the MasterCal flats and biases are the tu*.fits.fz files. In 2015, there is a full set of biases and flats for 9 different dates. In 2014, there are only 5 different dates (thus fewer files). I have organized the 2014 HiTS data into raw and calib subdirectories as in ap_verify_hits2015 . They are located on lsst-dev at /project/mrawls/ap_verify_2014hits . I am running the prototype pipeline on them (ingestion only for starters) and the output is being written to /project/mrawls/output_2014hits .
            Hide
            mrawls Meredith Rawls added a comment -

            OK Eric Bellm, the 2014 HiTS data is now ingested on lsst-dev in /project/mrawls/prototype_ap/output_2014hits. I have ingested the raw images and all the calibration files you should need to run processCcd. I recommend proceeding by using the doProcessCcd function from decam_process.py - you just need to feed it the visit number you want to process, one at a time.

            To do this one visit at a time from the command line (not necessarily the most efficient option, just an example!) with the DM-11021 version of decam_process.py would look like this

            1) edit the visit as defined in main() to be one of the visits you care about
            2) edit the ccdnum as defined in main() to be 1..62 to process all the CCDs
            3) comment out the calls to doIngest, doIngestCalibs, the second doProcessCcd, and doDiffIm so only the one call to doProcessCcd remains
            4) run it: python decam_hits.py -d ap_verify_2014hits -o output_2014hits

            For your reference, these are all 69 visits for the three 2014 HiTS fields

            288929
            288934
            288935
            288970
            288975
            288976
            289010
            289015
            289016
            289050
            289055
            289056
            289155
            289160
            289161
            289196
            289201
            289202
            289237
            289242
            289243
            289278
            289283
            289284
            289362
            289367
            289368
            289403
            289408
            289409
            289444
            289449
            289450
            289486
            289492
            289493
            289567
            289572
            289573
            289608
            289613
            289614
            289650
            289655
            289656
            289691
            289696
            289697
            289777
            289782
            289783
            289818
            289820
            289823
            289828
            289850
            289851
            289865
            289870
            289871
            289892
            289893
            289907
            289912
            289913
            289931
            292404
            292405
            292413

            Show
            mrawls Meredith Rawls added a comment - OK Eric Bellm , the 2014 HiTS data is now ingested on lsst-dev in /project/mrawls/prototype_ap/output_2014hits . I have ingested the raw images and all the calibration files you should need to run processCcd . I recommend proceeding by using the doProcessCcd function from decam_process.py - you just need to feed it the visit number you want to process, one at a time. To do this one visit at a time from the command line (not necessarily the most efficient option, just an example!) with the DM-11021 version of decam_process.py would look like this 1) edit the visit as defined in main() to be one of the visits you care about 2) edit the ccdnum as defined in main() to be 1..62 to process all the CCDs 3) comment out the calls to doIngest , doIngestCalibs , the second doProcessCcd , and doDiffIm so only the one call to doProcessCcd remains 4) run it: python decam_hits.py -d ap_verify_2014hits -o output_2014hits For your reference, these are all 69 visits for the three 2014 HiTS fields 288929 288934 288935 288970 288975 288976 289010 289015 289016 289050 289055 289056 289155 289160 289161 289196 289201 289202 289237 289242 289243 289278 289283 289284 289362 289367 289368 289403 289408 289409 289444 289449 289450 289486 289492 289493 289567 289572 289573 289608 289613 289614 289650 289655 289656 289691 289696 289697 289777 289782 289783 289818 289820 289823 289828 289850 289851 289865 289870 289871 289892 289893 289907 289912 289913 289931 292404 292405 292413
            Hide
            ebellm Eric Bellm added a comment -

            It looks like you ingested the defects, but processCcd is barfing when it can't find defects:

            NoResults: No locations for get: datasetType:defects dataId:DataId(initialdata=

            {'filter': 'g', 'hdu': 42, 'ccdnum': 29, 'date': '2014-03-01', 'object': 'Blind14A_09', 'visit': 288934}

            , tag=set([]))

            for example (and many more). Any chance you could do a quick spot check on one visit?

            Show
            ebellm Eric Bellm added a comment - It looks like you ingested the defects, but processCcd is barfing when it can't find defects: NoResults: No locations for get: datasetType:defects dataId:DataId(initialdata= {'filter': 'g', 'hdu': 42, 'ccdnum': 29, 'date': '2014-03-01', 'object': 'Blind14A_09', 'visit': 288934} , tag=set([])) for example (and many more). Any chance you could do a quick spot check on one visit?
            Hide
            mrawls Meredith Rawls added a comment -

            Should be good to go now - I've rerun the pipeline from ingestion through process. The problem was the validity date range on the defects, which I learned is set by the start date onward to very far in the future. (So for 2014-12-05 defects, they weren't valid for dates prior to then - I had to go grab a set of defects dated 2013-11-29 instead.)

            You can find find one sample processed calexp on lsst-dev in /project/mrawls/prototype_ap/output_2014hits/processed. Let me know if you have further issues!

            Show
            mrawls Meredith Rawls added a comment - Should be good to go now - I've rerun the pipeline from ingestion through process. The problem was the validity date range on the defects, which I learned is set by the start date onward to very far in the future. (So for 2014-12-05 defects, they weren't valid for dates prior to then - I had to go grab a set of defects dated 2013-11-29 instead.) You can find find one sample processed calexp on lsst-dev in /project/mrawls/prototype_ap/output_2014hits/processed . Let me know if you have further issues!
            Hide
            ebellm Eric Bellm added a comment -

            Seems to be processing correctly now...

            Show
            ebellm Eric Bellm added a comment - Seems to be processing correctly now...

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              krughoff Simon Krughoff
              Reviewers:
              Eric Bellm
              Watchers:
              David Reiss, Eric Bellm, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: