Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31653

Reprocess a subset of DC2 with profiling

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • ap_pipe
    • None
    • 2
    • AP F21-4 (September), AP F21-5 (October)
    • Alert Production
    • No

    Description

      After a full ap_pipe run has completed and the results verified, reprocess that same subset of DC2 with code profiling turned on. Analyze the output and evaluate whether the information is sufficient to undertake future performance tuning.

      Attachments

        1. combined.prof
          1.32 MB
        2. detector_55_profile.png
          detector_55_profile.png
          258 kB
        3. prof_detector_100.out
          1.03 MB
        4. prof_detector_101.out
          1.03 MB
        5. prof_detector_102.out
          1.02 MB
        6. prof_detector_103.out
          1.03 MB
        7. prof_detector_104.out
          1.03 MB
        8. prof_detector_108.out
          1.03 MB
        9. prof_detector_109.out
          1.03 MB
        10. prof_detector_110.out
          1.03 MB
        11. prof_detector_111.out
          1.03 MB
        12. prof_detector_112.out
          1.02 MB
        13. prof_detector_113.out
          1.03 MB
        14. prof_detector_21.out
          1.03 MB
        15. prof_detector_22.out
          1.03 MB
        16. prof_detector_23.out
          1.03 MB
        17. prof_detector_24.out
          1.03 MB
        18. prof_detector_25.out
          1.03 MB
        19. prof_detector_26.out
          1.03 MB
        20. prof_detector_54.out
          1.03 MB
        21. prof_detector_55.out
          1.03 MB
        22. prof_detector_56.out
          1.03 MB
        23. prof_detector_57.out
          1.03 MB
        24. prof_detector_58.out
          1.03 MB
        25. prof_detector_59.out
          1.03 MB
        26. prof_detector_60.out
          1.02 MB
        27. prof_detector_61.out
          1.03 MB
        28. prof_detector_62.out
          1.03 MB
        29. prof_detector_63.out
          1.03 MB
        30. prof_detector_64.out
          1.03 MB
        31. prof_detector_65.out
          1.03 MB
        32. prof_detector_66.out
          1.03 MB
        33. prof_detector_67.out
          1.03 MB
        34. prof_detector_68.out
          1.03 MB
        35. prof_detector_69.out
          1.03 MB
        36. prof_detector_70.out
          1.03 MB
        37. prof_detector_71.out
          1.03 MB
        38. prof_detector_99.out
          1.02 MB
        39. screenshot-1.png
          screenshot-1.png
          337 kB

        Issue Links

          Activity

            I've generated a profile and am sifting through the results.

            swnelson Spencer Nelson added a comment - I've generated a profile and am sifting through the results.

            I've attached profiles for each detector, as well as a profile which combines them all. These are from doing the same run as described in DM-31652, but with `--profile` each for each detector.

            swnelson Spencer Nelson added a comment - I've attached profiles for each detector, as well as a profile which combines them all. These are from doing the same run as described in DM-31652 , but with `--profile` each for each detector.

            Here's the full snakeviz for detector 55.

            swnelson Spencer Nelson added a comment - Here's the full snakeviz for detector 55.

            And here's just `imageDifference.py`'s subtree:

            swnelson Spencer Nelson added a comment - And here's just `imageDifference.py`'s subtree:

            Total timings per CCD (we need "time" to be <60s here) :

            {{
            ccd time peaks sources template coverage
            ----- ------- ------- --------- -------------------
            24 208.685 3964 85 73.5%
            25 222.415 3352 145 100.0%
            26 220.112 3043 109 100.0%
            54 206.47 4012 104 64.8%
            55 213.181 3495 112 100.0%
            56 210.468 3346 98 100.0%
            57 208.791 3751 75 56.4%
            58 227.843 3884 81 100.0%
            59 228.365 3433 139 100.0%
            60 219.037 4022 44 47.8%
            61 232.226 3958 77 100.0%
            62 216.831 3872 83 100.0%
            63 216.16 3508 145 100.0%
            64 209.726 3360 131 100.0%
            65 230.07 3660 139 100.0%
            66 219.15 3485 119 100.0%
            67 220.535 3256 151 100.0%
            68 221.017 3494 120 100.0%
            69 216.132 3329 65 100.0%
            70 225.792 3702 107 100.0%
            71 228.86 3592 111 100.0%
            99 200.749 3865 20 39.1%
            100 214.815 3939 78 100.0%
            101 218.667 4108 133 100.0%
            102 198.854 4267 20 27.7%
            103 229.007 4029 70 82.1%
            104 207.243 3575 95 73.4%
            108 220.909 3803 87 100.0%
            109 208.137 4008 79 100.0%
            110 214.698 3812 175 99.8%
            111 192.182 3580 80 64.4%
            112 198.977 3753 37 55.7%
            113 198.513 3486 59 47.0%}}

            swnelson Spencer Nelson added a comment - Total timings per CCD (we need "time" to be <60s here) : {{ ccd time peaks sources template coverage ----- ------- ------- --------- ------------------- 24 208.685 3964 85 73.5% 25 222.415 3352 145 100.0% 26 220.112 3043 109 100.0% 54 206.47 4012 104 64.8% 55 213.181 3495 112 100.0% 56 210.468 3346 98 100.0% 57 208.791 3751 75 56.4% 58 227.843 3884 81 100.0% 59 228.365 3433 139 100.0% 60 219.037 4022 44 47.8% 61 232.226 3958 77 100.0% 62 216.831 3872 83 100.0% 63 216.16 3508 145 100.0% 64 209.726 3360 131 100.0% 65 230.07 3660 139 100.0% 66 219.15 3485 119 100.0% 67 220.535 3256 151 100.0% 68 221.017 3494 120 100.0% 69 216.132 3329 65 100.0% 70 225.792 3702 107 100.0% 71 228.86 3592 111 100.0% 99 200.749 3865 20 39.1% 100 214.815 3939 78 100.0% 101 218.667 4108 133 100.0% 102 198.854 4267 20 27.7% 103 229.007 4029 70 82.1% 104 207.243 3575 95 73.4% 108 220.909 3803 87 100.0% 109 208.137 4008 79 100.0% 110 214.698 3812 175 99.8% 111 192.182 3580 80 64.4% 112 198.977 3753 37 55.7% 113 198.513 3486 59 47.0%}}

            Could you please post the scripts you used to generate this output? I'll be following up on this with some attempts at trimming some of that time.

            Parejkoj John Parejko added a comment - Could you please post the scripts you used to generate this output? I'll be following up on this with some attempts at trimming some of that time.

            Parejkoj Sure, here they are: https://gist.github.com/spenczar/fbd3e8cd4a1d3a748ebbe7e59b3cd6d2

            The first file in the gist is what I did to run each CCD sequentially, generating a profile for each. These are the CCDs for exposure 254379.

            The second file is the pipeline.yaml I used.

            The third file is a big gnarly script for pulling timings out of log files, which I used for that "total timings" table.

            https://github.com/lsst-dm/ap_pipe-notebooks/blob/main/notebooks/DM-31652-process_one_exposure_of_DC2.ipynb shows what I did to make templates ahead of time.

            swnelson Spencer Nelson added a comment - Parejkoj Sure, here they are: https://gist.github.com/spenczar/fbd3e8cd4a1d3a748ebbe7e59b3cd6d2 The first file in the gist is what I did to run each CCD sequentially, generating a profile for each. These are the CCDs for exposure 254379. The second file is the pipeline.yaml I used. The third file is a big gnarly script for pulling timings out of log files, which I used for that "total timings" table. https://github.com/lsst-dm/ap_pipe-notebooks/blob/main/notebooks/DM-31652-process_one_exposure_of_DC2.ipynb shows what I did to make templates ahead of time.

            People

              swnelson Spencer Nelson
              sullivan Ian Sullivan
              Ian Sullivan, John Parejko, Spencer Nelson
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.