Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26662

ap_verify import slow in Gen 3

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      AP F20-4 (September), AP F20-5 (October)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Ingesting HiTS2015 takes a very long time in Gen 3; after 30 minutes on lsst-devl01, it had not yet finished the preliminary import used to create an independent repository. This is unexpected, since of the affected datasets (calibs, crosstalk, defects, refcats, templates), only templates should scale up in size with the overall dataset size, and only defects and refcats should require file copies. For comparison, the entire import of CI-HiTS2015 takes 30 seconds.

      Run tests comparing the current Gen 2 and Gen 3 behavior to confirm whether the latter is much slower. If so, track down the cause.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26937 Profile raw data ingest

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-26526 Can't run RawIngestTask with processes != 1

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-25806 Support parallel ap_verify ingestion in Gen 3

          • Done

            Activity

            Ascending order - Click to sort in descending order
            krzys Krzysztof Findeisen created issue -
            krzys Krzysztof Findeisen made changes -
            Field Original Value New Value
            Epic Link DM-25145 [ 435263 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-26526 [ DM-26526 ]
            krzys Krzysztof Findeisen made changes -
            Description Ingesting {{ap_verify_hits2015}} takes a very long time in Gen 3; after 30 minutes on {{lsst-devl01}}, it had not yet finished the preliminary import of calibs, templates, etc. This is unexpected, since of the affected datasets (calibs, crosstalk, defects, refcats, templates), only templates should scale up in size with the overall dataset size, and only defects and refcats should require file copies.

            Run tests comparing the current Gen 2 and Gen 3 behavior to confirm whether the latter is much slower. If so, track down the cause.             		Ingesting {{ap_verify_hits2015}} takes a very long time in Gen 3; after 30 minutes on {{lsst-devl01}}, it had not yet finished the preliminary import used to create an independent repository. This is unexpected, since of the affected datasets (calibs, crosstalk, defects, refcats, templates), only templates should scale up in size with the overall dataset size, and only defects and refcats should require file copies.

            Run tests comparing the current Gen 2 and Gen 3 behavior to confirm whether the latter is much slower. If so, track down the cause.
            krzys Krzysztof Findeisen made changes -
            Description Ingesting {{ap_verify_hits2015}} takes a very long time in Gen 3; after 30 minutes on {{lsst-devl01}}, it had not yet finished the preliminary import used to create an independent repository. This is unexpected, since of the affected datasets (calibs, crosstalk, defects, refcats, templates), only templates should scale up in size with the overall dataset size, and only defects and refcats should require file copies.

            Run tests comparing the current Gen 2 and Gen 3 behavior to confirm whether the latter is much slower. If so, track down the cause.             		Ingesting {{HiTS2015}} takes a very long time in Gen 3; after 30 minutes on {{lsst-devl01}}, it had not yet finished the preliminary import used to create an independent repository. This is unexpected, since of the affected datasets (calibs, crosstalk, defects, refcats, templates), only templates should scale up in size with the overall dataset size, and only defects and refcats should require file copies. For comparison, the entire import of {{CI-HiTS2015}} takes 30 seconds.

            Run tests comparing the current Gen 2 and Gen 3 behavior to confirm whether the latter is much slower. If so, track down the cause.
            tjenness Tim Jenness made changes -
            Labels gen3-middleware
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-25806 [ DM-25806 ]
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            swinbank John Swinbank made changes -
            Epic Link DM-25145 [ 435263 ] DM-26803 [ 439755 ]
            sullivan Ian Sullivan made changes -
            Story Points 4 6
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-26937 [ DM-26937 ]
            sullivan Ian Sullivan made changes -
            Sprint AP F20-4 (September) [ 1039 ] AP F20-4 (September), AP F20-5 (October) [ 1039, 1046 ]
            krzys Krzysztof Findeisen made changes -
            Attachment cprofile-checksum.dat [ 45651 ]
            Attachment cprofile-ingest.dat [ 45652 ]
            krzys Krzysztof Findeisen made changes -
            Attachment cprofile-ingest.dat [ 45652 ]
            krzys Krzysztof Findeisen made changes -
            Attachment cprofile-nocheck.dat [ 45653 ]
            krzys Krzysztof Findeisen made changes -
            Attachment cprofile-nocheck.dat [ 45653 ]
            krzys Krzysztof Findeisen made changes -
            Attachment cprofile-checksum.dat [ 45651 ]
            krzys Krzysztof Findeisen made changes -
            Attachment cprofile-checksum.dat [ 45654 ]
            Attachment cprofile-nocheck.dat [ 45655 ]
            krzys Krzysztof Findeisen made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Tim Jenness
              Watchers:
              Eric Bellm, Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.