Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11371

Put HiTS datasets in /datasets

    XMLWordPrintable

    Details

      Description

      Decide what to include (probably just the overlap fields being used in verify_ap?), issue RFC, and package according to https://developer.lsst.io/services/datasets.html

        Attachments

          Issue Links

            Activity

            No builds found.
            ebellm Eric Bellm created issue -
            ebellm Eric Bellm made changes -
            Field Original Value New Value
            Epic Link DM-10773 [ 32739 ]
            ebellm Eric Bellm made changes -
            Epic Link DM-10773 [ 32739 ] DM-10771 [ 32737 ]
            ebellm Eric Bellm made changes -
            Component/s Alert Production [ 13900 ]
            ebellm Eric Bellm made changes -
            Description Decide what to include (probably just the overlap fields being used in `verify_ap`?), issue RFC, and package according to https://developer.lsst.io/services/datasets.html Decide what to include (probably just the overlap fields being used in {{verify_ap}}?), issue RFC, and package according to https://developer.lsst.io/services/datasets.html
            Hide
            swinbank John Swinbank added a comment -

            This ticket is the current sprint, but nobody is assigned to it. Eric Bellm, am I right in guessing that you're going to take this on?

            Show
            swinbank John Swinbank added a comment - This ticket is the current sprint, but nobody is assigned to it. Eric Bellm , am I right in guessing that you're going to take this on?
            ebellm Eric Bellm made changes -
            Assignee David Reiss [ reiss ]
            Hide
            reiss David Reiss added a comment -

            Need to know a bit more to proceed – are we talking the 3 `ap_verify` HiTS fields from both 2014 and 2015?
            Also, what is the use/context? Is this to be used directly by `ap_verify` or are we going to continue just using the `ap_verify_hits2015` repo for that?

            Show
            reiss David Reiss added a comment - Need to know a bit more to proceed – are we talking the 3 `ap_verify` HiTS fields from both 2014 and 2015? Also, what is the use/context? Is this to be used directly by `ap_verify` or are we going to continue just using the `ap_verify_hits2015` repo for that?
            Hide
            ebellm Eric Bellm added a comment -

            Yes, I think the raw data from the 3 `ap_verify` HiTS fields for both 2014 and 2015.

            We need the `ap_verify` data to be portable, which I think imposes some differences relative to the structure of the Common Datasets. Should we try to update our data formats to harmonize the layouts a little more?

            Show
            ebellm Eric Bellm added a comment - Yes, I think the raw data from the 3 `ap_verify` HiTS fields for both 2014 and 2015. We need the `ap_verify` data to be portable, which I think imposes some differences relative to the structure of the Common Datasets. Should we try to update our data formats to harmonize the layouts a little more?
            reiss David Reiss made changes -
            Link This issue relates to DM-11575 [ DM-11575 ]
            Hide
            reiss David Reiss added a comment - - edited
            Show
            reiss David Reiss added a comment - - edited RFC-372
            reiss David Reiss made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            reiss David Reiss made changes -
            Link This issue relates to RFC-372 [ RFC-372 ]
            Hide
            reiss David Reiss added a comment - - edited

            Implementation: getting these warnings for every raw image ingested:

            ingest.parse WARN: Unable to find value for ccdnum (derived from CCDNUM)
            ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM)

            and for every calib:

            ingestCalibs.parse WARN: translate_ccdnum failed to translate ccdnum: CALIB_ID not found

            Also, the tu* calib files seem to be duplicates of others which are named c4d*. I will skip ingesting all tu* files.

            Show
            reiss David Reiss added a comment - - edited Implementation: getting these warnings for every raw image ingested: ingest.parse WARN: Unable to find value for ccdnum (derived from CCDNUM) ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM) and for every calib: ingestCalibs.parse WARN: translate_ccdnum failed to translate ccdnum: CALIB_ID not found Also, the tu* calib files seem to be duplicates of others which are named c4d* . I will skip ingesting all tu* files.
            Hide
            reiss David Reiss added a comment -

            Colin Slater, I'm not sure how much of this needs to be reviewed, but would you mind checking off on this ticket? Let me know if there are any obvious issues with the ingested HiTS data.
            Thanks a lot!

            Show
            reiss David Reiss added a comment - Colin Slater , I'm not sure how much of this needs to be reviewed, but would you mind checking off on this ticket? Let me know if there are any obvious issues with the ingested HiTS data. Thanks a lot!
            reiss David Reiss made changes -
            Reviewers Colin Slater [ ctslater ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            reiss David Reiss made changes -
            Reviewers Colin Slater [ ctslater ] Hsin-Fang Chiang [ hchiang2 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            reiss David Reiss added a comment -

            Thanks, Hsin-Fang Chiang for reviewing the repo – I have marked RFC-372 as implemented and am now closing this ticket.

            Show
            reiss David Reiss added a comment - Thanks, Hsin-Fang Chiang for reviewing the repo – I have marked RFC-372 as implemented and am now closing this ticket.
            reiss David Reiss made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Labels sciencepipelines datasets sciencepipelines
            Hide
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            Greg Daues may you please apply immutability to:

            • /datasets/decam/_internal/raw/hits
            • /datasets/decam/_internal/preprocessed/masterCal/hits/
            • /datasets/decam/_internal/calib/cpHits
            • /datasets/decam/20*
            • /datasets/decam/registry.sqlite3
            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited Greg Daues may you please apply immutability to: /datasets/decam/_internal/raw/hits /datasets/decam/_internal/preprocessed/masterCal/hits/ /datasets/decam/_internal/calib/cpHits /datasets/decam/20* /datasets/decam/registry.sqlite3
            Hide
            gdaues Greg Daues added a comment -

            These files & directories are now immutable and ready for use.

            Show
            gdaues Greg Daues added a comment - These files & directories are now immutable and ready for use.

              People

              Assignee:
              reiss David Reiss
              Reporter:
              ebellm Eric Bellm
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Colin Slater, David Reiss, Eric Bellm, Greg Daues, Hsin-Fang Chiang, John Swinbank
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.