Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1119

EUPS responsiveness is sluggish due to rapid growth of eups-versioned packages

    XMLWordPrintable

    Details

    • Team:
      DM Science

      Description

      Eups is exhibiting a slowdown due to the rapidly increasing number of eups-versioned packages being maintained by it.

      It has been suggested that, for the short term, the buildbot system create a periodic purge of eups tags associated with user branches (for those would considered too old for debugging use).

      A longer term solution will be having separate build structures for mainline builds and user builds.

        Attachments

          Activity

          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          @RHL suggests (via hipchat moments ago) "Deploying the SQL backend is the real answer, but I need to do some more work and testing.  Lots of tags seem to be a problem but I haven't profiled to know why.  I'd try removing tags, and bugging me to work on the SQL stuff."

          The tag removal technique as tried a few weeks ago and didn't last very long. We are ready for the longer term solution.

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - @RHL suggests (via hipchat moments ago) "Deploying the SQL backend is the real answer, but I need to do some more work and testing.  Lots of tags seem to be a problem but I haven't profiled to know why.  I'd try removing tags, and bugging me to work on the SQL stuff." The tag removal technique as tried a few weeks ago and didn't last very long. We are ready for the longer term solution.
          Hide
          ktl Kian-Tat Lim added a comment -

          Removing tags is not enough (although we've added a lot). Removing versions is also necessary since the worst code (at https://github.com/RobertLuptonTheGood/eups/blob/master/python/eups/db/Database.py#L678) looks at all tags and all versions.

          Is there a shorter-term solution that avoids the above code?

          Show
          ktl Kian-Tat Lim added a comment - Removing tags is not enough (although we've added a lot). Removing versions is also necessary since the worst code (at https://github.com/RobertLuptonTheGood/eups/blob/master/python/eups/db/Database.py#L678 ) looks at all tags and all versions. Is there a shorter-term solution that avoids the above code?
          Hide
          rhl Robert Lupton added a comment -

          I didn't write that code and I'm not surprised that it's a disaster. Thanks for profiling, that's where I would have started.

          A SQL backend is almost certainly needed anyway, both for performance and to solve the locking issues properly, but maybe not urgently.

          Show
          rhl Robert Lupton added a comment - I didn't write that code and I'm not surprised that it's a disaster. Thanks for profiling, that's where I would have started. A SQL backend is almost certainly needed anyway, both for performance and to solve the locking issues properly, but maybe not urgently.
          Hide
          lguy Leanne Guy added a comment -

          Kian-Tat Lim, Robert Lupton is this still an issue?  If not (or if we are not planning to fix it) I will set to Won't Fix 

          Show
          lguy Leanne Guy added a comment - Kian-Tat Lim , Robert Lupton is this still an issue?  If not (or if we are not planning to fix it) I will set to Won't Fix 
          Hide
          price Paul Price added a comment -

          This is still an issue. I don't believe it will be going away without a relatively large investment of RHL time, which I think must be quite unlikely.

          Show
          price Paul Price added a comment - This is still an issue. I don't believe it will be going away without a relatively large investment of RHL time, which I think must be quite unlikely.
          Hide
          lguy Leanne Guy added a comment -

          Robert Lupton informs me that there are workarounds in place and that this is not a significant blocker for anyone. This will not be fixed without some significant time investment and is not a priority at this time. 

          Show
          lguy Leanne Guy added a comment - Robert Lupton informs me that there are workarounds in place and that this is not a significant blocker for anyone. This will not be fixed without some significant time investment and is not a priority at this time. 

            People

            Assignee:
            rhl Robert Lupton
            Reporter:
            robyn Robyn Allsman [X] (Inactive)
            Watchers:
            Kian-Tat Lim, Leanne Guy, Paul Price, Robert Lupton, Robyn Allsman [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: