Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29521

Pin sqlite in rubin-env to 3.34.0

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: conda
    • Labels:
      None
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      sqlite 3.35.0 through 3.35.3 change the way UNION ALL queries are planned. This causes a major performance regression in the Gen3 Butler's query graph generator (DM-29453). We will fall back to 3.34.0 which was previously used in the stack base container.

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          ktl Kian-Tat Lim added a comment -

          I'm doing this as new builds of rubin-env 0.4.1 through 0.4.3.

          Show
          ktl Kian-Tat Lim added a comment - I'm doing this as new builds of rubin-env 0.4.1 through 0.4.3.
          Hide
          Permalink
          tjenness Tim Jenness added a comment -

          Do sqlite know they've slowed things down so badly? ie was it a deliberate change because of some other critical problem?

          Show
          tjenness Tim Jenness added a comment - Do sqlite know they've slowed things down so badly? ie was it a deliberate change because of some other critical problem?
          Hide
          Permalink
          ktl Kian-Tat Lim added a comment -

          Yes, they appear to know: https://sqlite.org/forum/forumpost/737cf80ecefbee48f66ab5f9ee0b144f62c4b2694f2b56235f6f3c1903e8e9ef.

          It's not clear that the fix that has been added https://sqlite.org/src/info/9520bed2bd87dc56 will solve our problem (it looks like it definitely won't go back to the way it was in 3.34), but it might.

          Show
          ktl Kian-Tat Lim added a comment - Yes, they appear to know: https://sqlite.org/forum/forumpost/737cf80ecefbee48f66ab5f9ee0b144f62c4b2694f2b56235f6f3c1903e8e9ef . It's not clear that the fix that has been added https://sqlite.org/src/info/9520bed2bd87dc56 will solve our problem (it looks like it definitely won't go back to the way it was in 3.34), but it might.
          Hide
          Permalink
          ktl Kian-Tat Lim added a comment -

          New base container was built manually this morning; d_2021_03_31 was built on top of it and seems to have reduced the runtime of validate_drp_gen3 to about what it was.

          Show
          ktl Kian-Tat Lim added a comment - New base container was built manually this morning; d_2021_03_31 was built on top of it and seems to have reduced the runtime of validate_drp_gen3 to about what it was.

            People

            Assignee:
            ktl Kian-Tat Lim
            Reporter:
            ktl Kian-Tat Lim
            Reviewers:
            Tim Jenness
            Watchers:
            Kian-Tat Lim, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.