Add quantities useful for QA to ccdVisitTable

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
3
• Sprint:
DRP S22B
• Team:
Data Release Production
• Urgent?:
No

Description

The per-collection consolidated ccdVisitTable dataset is required for the plots to be made in DM-34625. However, they are currently missing a number of the columns most useful for QA. The current column listing is:

 ['visitId', 'physical_filter', 'band', 'ra', 'decl', 'zenithDistance', 'zeroPoint', 'psfSigma', 'skyBg', 'skyNoise', 'detector', 'seeing', 'skyRotation', 'expMidpt', 'expTime', 'obsStart', 'darkTime', 'xSize', 'ySize', 'llcra', 'llcdec', 'ulcra', 'ulcdec', 'urcra', 'urcdec', 'lrcra', 'lrcdec'] 

To this list I would like to add the following columns which come directly from the visitSummary tables (they were added after the ccdVisitTable dataset was defined, hence they are not yet included):

 ['astromOffsetMean', 'astromOffsetStd', 'nPsfStar', 'psfStarDeltaE1Median', 'psfStarDeltaE2Median', 'psfStarDeltaE1Scatter', 'psfStarDeltaE2Scatter', 'psfStarDeltaSizeMedian', 'psfStarDeltaSizeScatter', 'psfStarScaledDeltaSizeScatter'] 

Activity

Hide
Lauren MacArthur added a comment -

I ran this through ci_hsc_gen3 and looked at the results. The additional columns are indeed now included in the ccdVisitTable (with the correct values as compared with those in the visitSummary tables).

In case it's important, the size difference between the ccdVisitTable table created on main vs. this ticket branch is: 25K vs. 33K, so a factor of 1.32 larger.

Show
Lauren MacArthur added a comment - I ran this through ci_hsc_gen3 and looked at the results. The additional columns are indeed now included in the ccdVisitTable (with the correct values as compared with those in the visitSummary tables). In case it's important, the size difference between the ccdVisitTable table created on main vs. this ticket branch is: 25K vs. 33K, so a factor of 1.32 larger.
Hide
Lauren MacArthur added a comment - - edited

Would you mind giving this a look?  One thing I'm not sure about is if there are any schemas that need updating for this change.

Jenkins + ci_hsc + ci_imsim is running green.

Show
Lauren MacArthur added a comment - - edited Would you mind giving this a look?  One thing I'm not sure about is if there are any schemas that need updating for this change. Jenkins + ci_hsc + ci_imsim is  running green .
Hide
Lauren MacArthur added a comment -

This fell behind main while awaiting review, so I rebased and ran another jenkins.

Show
Lauren MacArthur added a comment - This fell behind main while awaiting review, so I rebased and ran another  jenkins .
Hide
Lauren MacArthur added a comment -

And another rebase & green Jenkins after the great merge yesterday.

Show
Lauren MacArthur added a comment - And another rebase & green Jenkins  after the great merge yesterday.
Hide

Green light to add to the existing CcdVisit Table.

The CcdVisit schema need updates in:

Show
Yusra AlSayyad added a comment - Green light to add to the existing CcdVisit Table. The CcdVisit schema need updates in: https://github.com/lsst/sdm_schemas/blob/main/yml/imsim.yaml https://github.com/lsst/sdm_schemas/blob/main/yml/hsc.yaml
Hide
Lauren MacArthur added a comment -

Awesome, thanks Yusra.  One question:  while adding the entries into the schema, I see some entries there include the fields:

 mysql:datatype: fits:tunit: 

while others do not. It is not clear to me where the distinction lies (the existing entries for the CcdVisit Table do not include them, but I'm not sure if this was an oversight). Can you enlighten me as to when they are indeed required?

Show
Lauren MacArthur added a comment - Awesome, thanks Yusra.  One question:  while adding the entries into the schema, I see some entries there include the fields: mysql:datatype: fits:tunit: while others do not. It is not clear to me where the distinction lies (the existing entries for the CcdVisit Table do not include them, but I'm not sure if this was an oversight). Can you enlighten me as to when they are indeed required?
Hide
Lauren MacArthur added a comment -

As per our discussion, it is not clear at present if those fields are indeed required and, if they are, they can all be updated in bulk on a subsequent ticket.

Final Jenkins is happy.

Show
Lauren MacArthur added a comment - As per our discussion, it is not clear at present if those fields are indeed required and, if they are, they can all be updated in bulk on a subsequent ticket. Final Jenkins is happy .

People

Assignee:
Lauren MacArthur
Reporter:
Lauren MacArthur
Reviewers:
Watchers:
Eli Rykoff, Lauren MacArthur, Yusra AlSayyad