Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29407

visitSummary files should not persist the detector

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
    • Story Points:
      1
    • Epic Link:
    • Sprint:
      DRP S21a (Dec Jan)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The visitSummary files currently persist the detector for each row, but it turns out that this makes the files 140x larger than otherwise, and take 50x longer to read than otherwise. Persisting the detector in here is definitely not necessary; all we need is the id and that is persisted as the row index.

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          No builds found.
          erykoff Eli Rykoff created issue -
          erykoff Eli Rykoff made changes -
          Field Original Value New Value
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          Permalink
          erykoff Eli Rykoff added a comment -

          PRs are:
          https://github.com/lsst/jointcal/pull/175
          https://github.com/lsst/pipe_tasks/pull/488
          https://github.com/lsst/testdata_jointcal/pull/34
          Jenkins (including ci_hsc):
          https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33886/pipeline

          Show
          erykoff Eli Rykoff added a comment - PRs are: https://github.com/lsst/jointcal/pull/175 https://github.com/lsst/pipe_tasks/pull/488 https://github.com/lsst/testdata_jointcal/pull/34 Jenkins (including ci_hsc): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33886/pipeline
          erykoff Eli Rykoff made changes -
          Reviewers John Parejko [ parejkoj ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          Permalink
          Parejkoj John Parejko added a comment -

          Thanks for digging into this. You jointcal change looks good, baring one minor doc typo.

          Show
          Parejkoj John Parejko added a comment - Thanks for digging into this. You jointcal change looks good, baring one minor doc typo.
          Parejkoj John Parejko made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          erykoff Eli Rykoff made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]

            People

            Assignee:
            erykoff Eli Rykoff
            Reporter:
            erykoff Eli Rykoff
            Reviewers:
            John Parejko
            Watchers:
            Eli Rykoff, John Parejko
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.