Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-715

Add new packages to conda env

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM, TCT
    • Labels:
      None

      Description

      I would like to add these new packages to the conda environment:

      • conda
      • backoff
      • pytest-openfiles
      • psycopg2

      conda is needed to implement DM-25431 so that we can report software versions properly again (for example boost is no longer reported because it's not an EUPS package). The package has to be in our own so that base can import it to query the local packages.

      We would like to use backoff in our S3 gen3 datastore to allow for automatic retries if the network connection drops. When we run thousands of jobs we do sometimes get blips that would be caught by including backoff.

      pytest-openfiles used to be part of our conda environment but it was removed when astropy dropped it as a dependency in release 4.0. I would like to add it back to see if it can allow us to remove the "memory test" from our tests (memory leaks are no longer checked, solely file leaks). Switching to pytest-openfiles would be more in line with other testing approaches and simplify the test files (and let us remove the psutil dependency).

      psycopg2 is used to connect to PostgreSQL databases. Now that we have standardized on postgres we need to be able to connect to them from butler.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            (sorry, messages crossed in the ether, or I'd have waited for the new RFC)

            Show
            jbosch Jim Bosch added a comment - (sorry, messages crossed in the ether, or I'd have waited for the new RFC)
            Hide
            tjenness Tim Jenness added a comment -

            The compromise then is to have the GitHub action install these packages so that the test run (and break) early. Then we also have a postgres database visible in Jenkins that can be used by ci_hsc_gen3. We only need psycopg to be added to the conda base env in this case (which Brian Van Klaveren has already done as described above).

            Show
            tjenness Tim Jenness added a comment - The compromise then is to have the GitHub action install these packages so that the test run (and break) early. Then we also have a postgres database visible in Jenkins that can be used by ci_hsc_gen3. We only need psycopg to be added to the conda base env in this case (which Brian Van Klaveren has already done as described above).
            Hide
            tjenness Tim Jenness added a comment -

            I've updated the RFC to include pyscopg2 but not other postgress packages.

            Show
            tjenness Tim Jenness added a comment - I've updated the RFC to include pyscopg2 but not other postgress packages.
            Hide
            swinbank John Swinbank added a comment - - edited

            We discussed this at the DM-CCB of 2020-07-29, at which I learnt that I am a blocker (�). To be clear — if it is possible to add pytest-openfiles and remove psutil in one change, without requiring a further RFC, then I am in favour of that approach. However, if (as I understand from Tim Jenness) such a change is not possible, then I am ok with simply adding the former and not removing the latter on this RFC.

            Show
            swinbank John Swinbank added a comment - - edited We discussed this at the DM-CCB of 2020-07-29, at which I learnt that I am a blocker (�). To be clear — if it is possible to add pytest-openfiles and remove psutil in one change, without requiring a further RFC, then I am in favour of that approach. However, if (as I understand from Tim Jenness ) such a change is not possible, then I am ok with simply adding the former and not removing the latter on this RFC.
            Hide
            tjenness Tim Jenness added a comment -

            Brian Van Klaveren do you have an implementation ticket for this that I can link to? I know you've been implementing it already.

            Show
            tjenness Tim Jenness added a comment - Brian Van Klaveren do you have an implementation ticket for this that I can link to? I know you've been implementing it already.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Brian Van Klaveren, Colin Slater, Jim Bosch, John Swinbank, Kian-Tat Lim, Leanne Guy, Michelle Butler [X] (Inactive), Tim Jenness, Wil O'Mullane
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.