# Put HiTS datasets in /datasets

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
4
• Sprint:
• Team:

#### Description

Decide what to include (probably just the overlap fields being used in verify_ap?), issue RFC, and package according to https://developer.lsst.io/services/datasets.html

#### Activity

No builds found.
Eric Bellm created issue -
Field Original Value New Value
Epic Link DM-10773 [ 32739 ]
 Epic Link DM-10773 [ 32739 ] DM-10771 [ 32737 ]
 Component/s Alert Production [ 13900 ]
 Description Decide what to include (probably just the overlap fields being used in verify_ap?), issue RFC, and package according to https://developer.lsst.io/services/datasets.html Decide what to include (probably just the overlap fields being used in {{verify_ap}}?), issue RFC, and package according to https://developer.lsst.io/services/datasets.html
Hide
John Swinbank added a comment -

This ticket is the current sprint, but nobody is assigned to it. Eric Bellm, am I right in guessing that you're going to take this on?

Show
John Swinbank added a comment - This ticket is the current sprint, but nobody is assigned to it. Eric Bellm , am I right in guessing that you're going to take this on?
 Assignee David Reiss [ reiss ]
Hide
David Reiss added a comment -

Need to know a bit more to proceed – are we talking the 3 ap_verify HiTS fields from both 2014 and 2015?
Also, what is the use/context? Is this to be used directly by ap_verify or are we going to continue just using the ap_verify_hits2015 repo for that?

Show
David Reiss added a comment - Need to know a bit more to proceed – are we talking the 3 ap_verify HiTS fields from both 2014 and 2015? Also, what is the use/context? Is this to be used directly by ap_verify or are we going to continue just using the ap_verify_hits2015 repo for that?
Hide
Eric Bellm added a comment -

Yes, I think the raw data from the 3 ap_verify HiTS fields for both 2014 and 2015.

We need the ap_verify data to be portable, which I think imposes some differences relative to the structure of the Common Datasets. Should we try to update our data formats to harmonize the layouts a little more?

Show
Eric Bellm added a comment - Yes, I think the raw data from the 3 ap_verify HiTS fields for both 2014 and 2015. We need the ap_verify data to be portable, which I think imposes some differences relative to the structure of the Common Datasets. Should we try to update our data formats to harmonize the layouts a little more?
 Link This issue relates to DM-11575 [ DM-11575 ]
Hide
David Reiss added a comment - - edited
Show
David Reiss added a comment - - edited RFC-372
 Status To Do [ 10001 ] In Progress [ 3 ]
 Link This issue relates to RFC-372 [ RFC-372 ]
Hide
David Reiss added a comment - - edited

Implementation: getting these warnings for every raw image ingested:

ingest.parse WARN: Unable to find value for ccdnum (derived from CCDNUM)
ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM)

and for every calib:

Also, the tu* calib files seem to be duplicates of others which are named c4d*. I will skip ingesting all tu* files.

Show
David Reiss added a comment - - edited Implementation: getting these warnings for every raw image ingested: ingest.parse WARN: Unable to find value for ccdnum (derived from CCDNUM) ingest.parse WARN: Unable to find value for ccd (derived from CCDNUM) and for every calib: ingestCalibs.parse WARN: translate_ccdnum failed to translate ccdnum: CALIB_ID not found Also, the tu* calib files seem to be duplicates of others which are named c4d* . I will skip ingesting all tu* files.
Hide
David Reiss added a comment -

Colin Slater, I'm not sure how much of this needs to be reviewed, but would you mind checking off on this ticket? Let me know if there are any obvious issues with the ingested HiTS data.
Thanks a lot!

Show
David Reiss added a comment - Colin Slater , I'm not sure how much of this needs to be reviewed, but would you mind checking off on this ticket? Let me know if there are any obvious issues with the ingested HiTS data. Thanks a lot!
 Reviewers Colin Slater [ ctslater ] Status In Progress [ 3 ] In Review [ 10004 ]
 Reviewers Colin Slater [ ctslater ] Hsin-Fang Chiang [ hchiang2 ]
 Status In Review [ 10004 ] Reviewed [ 10101 ]
Hide
David Reiss added a comment -

Thanks, Hsin-Fang Chiang for reviewing the repo – I have marked RFC-372 as implemented and am now closing this ticket.

Show
David Reiss added a comment - Thanks, Hsin-Fang Chiang for reviewing the repo – I have marked RFC-372 as implemented and am now closing this ticket.
 Resolution Done [ 10000 ] Status Reviewed [ 10101 ] Done [ 10002 ]
 Labels sciencepipelines datasets sciencepipelines
Hide
Hsin-Fang Chiang added a comment - - edited

Greg Daues may you please apply immutability to:

• /datasets/decam/_internal/raw/hits
• /datasets/decam/_internal/preprocessed/masterCal/hits/
• /datasets/decam/_internal/calib/cpHits
• /datasets/decam/20*
• /datasets/decam/registry.sqlite3
Show
Hsin-Fang Chiang added a comment - - edited Greg Daues may you please apply immutability to: /datasets/decam/_internal/raw/hits /datasets/decam/_internal/preprocessed/masterCal/hits/ /datasets/decam/_internal/calib/cpHits /datasets/decam/20* /datasets/decam/registry.sqlite3
Hide
Greg Daues added a comment -

These files & directories are now immutable and ready for use.

Show
Greg Daues added a comment - These files & directories are now immutable and ready for use.

#### People

Assignee:
David Reiss
Reporter:
Eric Bellm
Reviewers:
Hsin-Fang Chiang
Watchers:
Colin Slater, David Reiss, Eric Bellm, Greg Daues, Hsin-Fang Chiang, John Swinbank