Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29916

Single-shot, multi-cycle FGCM is memory-inefficient

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: fgcmcal
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Running the multiple fit-cycle fgcmFitCycle PipelineTask under BPS requires giving it 50 Gb of memory for RC2's tract=9813. The memory usage (as reported by fgcmcal logs) goes up with each fit cycle (I'll attach a log after the ticket is created).

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29776 Attempt complete two-QG HSC RC2 run on w16 on one tract

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29918 Investigate bad cgroup/HTCondor memory analysis in BPS vs. fgcmcal

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            Not as bad as I had feared, just tedious and I had to be careful to only release the shared memory explicitly. I tested the full RC2 calibration on lsst-devl01 and the peak memory usage was 20Gb, and never rose from cycle to cycle. For future reference, I know how to get the peak memory down to less than 15Gb but that will require temporary files.
            fgcmcal PR: https://github.com/lsst/fgcmcal/pull/60
            fgcm PR: https://github.com/lsst/fgcm/pull/22 (needs approval but not review)
            Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34086/pipeline

            Show
            erykoff Eli Rykoff added a comment - Not as bad as I had feared, just tedious and I had to be careful to only release the shared memory explicitly. I tested the full RC2 calibration on lsst-devl01 and the peak memory usage was 20Gb, and never rose from cycle to cycle. For future reference, I know how to get the peak memory down to less than 15Gb but that will require temporary files. fgcmcal PR: https://github.com/lsst/fgcmcal/pull/60 fgcm PR: https://github.com/lsst/fgcm/pull/22 (needs approval but not review) Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34086/pipeline
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Looks good, glad it wasn't hard, and thanks for taking care of it so quickly!

            Show
            jbosch Jim Bosch added a comment - Looks good, glad it wasn't hard, and thanks for taking care of it so quickly!

              People

              Assignee:
              erykoff Eli Rykoff
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Jim Bosch
              Watchers:
              Eli Rykoff, Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.