Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5017

Make validate_drp examples/runDecamTest.sh take

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Validation
    • Labels:
      None

      Description

      On my Mac examples/runDecamTest.sh master took an hour. It would be very helpful if it was much faster.

        Attachments

          Activity

          Hide
          rowen Russell Owen added a comment -

          Here is a log of how far my Mac has gotten in 50 minutes. I am guessing that the huge number of detected sources is relevant: Measuring 3189 sources (1761 parents, 1428 children)

          Show
          rowen Russell Owen added a comment - Here is a log of how far my Mac has gotten in 50 minutes. I am guessing that the huge number of detected sources is relevant: Measuring 3189 sources (1761 parents, 1428 children)
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          I'd like to interpret this issue in two different parts:
          a) Create an examples/runDecamTest_quick.sh that just runs one CCD.
          b) Figure out if it really should be taking as long as it does, and if not, what can be changed in the configuration settings or algorithms to reduce the running time to what we "feel" is more reasonable (e.g., 10 minutes).

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - I'd like to interpret this issue in two different parts: a) Create an examples/runDecamTest_quick.sh that just runs one CCD. b) Figure out if it really should be taking as long as it does, and if not, what can be changed in the configuration settings or algorithms to reduce the running time to what we "feel" is more reasonable (e.g., 10 minutes).
          Hide
          rowen Russell Owen added a comment -

          I agree that 10 minutes would be a big improvement. If you can also usefully offer a significantly faster quick version, I would not turn that down (perhaps as an option), but the first is more important. If you are not seeing such long run times then perhaps I have a system problem. fc-list can definitely take far too long on my computer, but it only ran 4 times (according to my log) so I don't think it accounts for the majority of the time.

          Show
          rowen Russell Owen added a comment - I agree that 10 minutes would be a big improvement. If you can also usefully offer a significantly faster quick version, I would not turn that down (perhaps as an option), but the first is more important. If you are not seeing such long run times then perhaps I have a system problem. fc-list can definitely take far too long on my computer, but it only ran 4 times (according to my log) so I don't think it accounts for the majority of the time.
          Hide
          rowen Russell Owen added a comment -

          For the record: simply processing --id visit=176837 ccdnum=10 took me 13 minutes on my Mac with an SSD drive using master. This may be relevant: Measuring 5327 sources (3205 parents, 2122 children)

          Show
          rowen Russell Owen added a comment - For the record: simply processing --id visit=176837 ccdnum=10 took me 13 minutes on my Mac with an SSD drive using master. This may be relevant: Measuring 5327 sources (3205 parents, 2122 children)
          Hide
          rowen Russell Owen added a comment -

          With Paul Price's speed fix I'm finding that examples/runDecamTest.sh takes about 10 minutes on my Mac. A shorter test that results in some QA graphs would still be appreciated, but I think the pressure is off. If you want to just close this ticket, feel free.

          Show
          rowen Russell Owen added a comment - With Paul Price 's speed fix I'm finding that examples/runDecamTest.sh takes about 10 minutes on my Mac. A shorter test that results in some QA graphs would still be appreciated, but I think the pressure is off. If you want to just close this ticket, feel free.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Thanks, Russell Owen

          I think I essentially almost accidentally implemented the remaining part here of doing a Quick test as part of DM-4901, which was nominally about shifting to use YAML config files instead of the hard-coded.

          I'll ask you to review that ticket in a few days once DM-4956 passes review.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Thanks, Russell Owen I think I essentially almost accidentally implemented the remaining part here of doing a Quick test as part of DM-4901 , which was nominally about shifting to use YAML config files instead of the hard-coded. I'll ask you to review that ticket in a few days once DM-4956 passes review.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          After a bit of discussion, this is marked as Done with completion of DM-4901 and DM-5050.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - After a bit of discussion, this is marked as Done with completion of DM-4901 and DM-5050 .

            People

            • Assignee:
              wmwood-vasey Michael Wood-Vasey
              Reporter:
              rowen Russell Owen
              Watchers:
              Michael Wood-Vasey, Russell Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: