Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34630

Add quantities useful for QA to ccdVisitTable

    XMLWordPrintable

    Details

    • Story Points:
      3
    • Epic Link:
    • Sprint:
      DRP S22B
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The per-collection consolidated ccdVisitTable dataset is required for the plots to be made in DM-34625. However, they are currently missing a number of the columns most useful for QA. The current column listing is:

      ['visitId', 'physical_filter', 'band', 'ra', 'decl', 'zenithDistance', 'zeroPoint', 'psfSigma', 'skyBg', 'skyNoise', 'detector', 'seeing', 'skyRotation', 'expMidpt', 'expTime', 'obsStart', 'darkTime', 'xSize', 'ySize', 'llcra', 'llcdec', 'ulcra', 'ulcdec', 'urcra', 'urcdec', 'lrcra', 'lrcdec']
      

      To this list I would like to add the following columns which come directly from the visitSummary tables (they were added after the ccdVisitTable dataset was defined, hence they are not yet included):

      ['astromOffsetMean', 'astromOffsetStd', 'nPsfStar', 'psfStarDeltaE1Median', 'psfStarDeltaE2Median', 'psfStarDeltaE1Scatter', 'psfStarDeltaE2Scatter', 'psfStarDeltaSizeMedian', 'psfStarDeltaSizeScatter', 'psfStarScaledDeltaSizeScatter']
      

        Attachments

          Issue Links

            Activity

            No builds found.
            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Link This issue blocks DM-34625 [ DM-34625 ]
            lauren Lauren MacArthur made changes -
            Epic Link DM-32157 [ 779840 ] DM-30555 [ 511221 ]
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            lauren Lauren MacArthur added a comment -

            I ran this through ci_hsc_gen3 and looked at the results. The additional columns are indeed now included in the ccdVisitTable (with the correct values as compared with those in the visitSummary tables).

            In case it's important, the size difference between the ccdVisitTable table created on main vs. this ticket branch is: 25K vs. 33K, so a factor of 1.32 larger.

            Show
            lauren Lauren MacArthur added a comment - I ran this through ci_hsc_gen3 and looked at the results. The additional columns are indeed now included in the ccdVisitTable (with the correct values as compared with those in the visitSummary tables). In case it's important, the size difference between the ccdVisitTable table created on main vs. this ticket branch is: 25K vs. 33K, so a factor of 1.32 larger.
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Would you mind giving this a look?  One thing I'm not sure about is if there are any schemas that need updating for this change.

            Jenkins + ci_hsc + ci_imsim is running green.

            Show
            lauren Lauren MacArthur added a comment - - edited Would you mind giving this a look?  One thing I'm not sure about is if there are any schemas that need updating for this change. Jenkins + ci_hsc + ci_imsim is  running green .
            lauren Lauren MacArthur made changes -
            Reviewers Yusra AlSayyad [ yusra ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            lauren Lauren MacArthur added a comment -

            This fell behind main while awaiting review, so I rebased and ran another jenkins.

            Show
            lauren Lauren MacArthur added a comment - This fell behind main while awaiting review, so I rebased and ran another  jenkins .
            Hide
            lauren Lauren MacArthur added a comment -

            And another rebase & green Jenkins after the great merge yesterday.

            Show
            lauren Lauren MacArthur added a comment - And another rebase & green Jenkins  after the great merge yesterday.
            Hide
            yusra Yusra AlSayyad added a comment -

            Green light to add to the existing CcdVisit Table.

            The CcdVisit schema need updates in:

            Show
            yusra Yusra AlSayyad added a comment - Green light to add to the existing CcdVisit Table. The CcdVisit schema need updates in: https://github.com/lsst/sdm_schemas/blob/main/yml/imsim.yaml https://github.com/lsst/sdm_schemas/blob/main/yml/hsc.yaml
            yusra Yusra AlSayyad made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            lauren Lauren MacArthur added a comment -

            Awesome, thanks Yusra.  One question:  while adding the entries into the schema, I see some entries there include the fields:

            mysql:datatype:
            fits:tunit:
            

            while others do not. It is not clear to me where the distinction lies (the existing entries for the CcdVisit Table do not include them, but I'm not sure if this was an oversight). Can you enlighten me as to when they are indeed required?

            Show
            lauren Lauren MacArthur added a comment - Awesome, thanks Yusra.  One question:  while adding the entries into the schema, I see some entries there include the fields: mysql:datatype: fits:tunit: while others do not. It is not clear to me where the distinction lies (the existing entries for the CcdVisit Table do not include them, but I'm not sure if this was an oversight). Can you enlighten me as to when they are indeed required?
            Hide
            lauren Lauren MacArthur added a comment -

            As per our discussion, it is not clear at present if those fields are indeed required and, if they are, they can all be updated in bulk on a subsequent ticket.

            Final Jenkins is happy.

            Show
            lauren Lauren MacArthur added a comment - As per our discussion, it is not clear at present if those fields are indeed required and, if they are, they can all be updated in bulk on a subsequent ticket. Final Jenkins is happy .
            lauren Lauren MacArthur made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            lauren Lauren MacArthur made changes -
            Story Points 3
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-36200 [ DM-36200 ]
            lauren Lauren MacArthur made changes -
            Remote Link This issue links to "Page (Confluence)" [ 34432 ]

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Eli Rykoff, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.