Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2928

move old ingest scripts into and retire old packages

    XMLWordPrintable

Details

    Description

      This ticket implements RFC-57, by:

      • renaming datarel to daf_ingest (there is already a daf_ingest package, but it's completely empty, so I'll just force-push it all away)
      • removing everything from the renamed package that doesn't relate to ingest (including pruning dependencies)
      • removing ap and testing_endToEnd from the CI system

      Attachments

        Issue Links

          Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue blocks DM-1766 [ DM-1766 ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch added a comment -

            This is essentially ready, but I'm seeing a buildbot failure on the ticket branch I don't understand; the log is here.

            I've removed datarel and testing_endToEnd from lsst_distrib, and now, after running lsstDoxygen, something tries to do a eups list that includes datarel, and that fails. I don't see the string "datarel" in lsstsw anywhere but repos.yaml, and my understanding is that that shouldn't be a problem. frossie, jhoblitt, any ideas?

            jbosch Jim Bosch added a comment - This is essentially ready, but I'm seeing a buildbot failure on the ticket branch I don't understand; the log is here . I've removed datarel and testing_endToEnd from lsst_distrib, and now, after running lsstDoxygen , something tries to do a eups list that includes datarel , and that fails. I don't see the string "datarel" in lsstsw anywhere but repos.yaml, and my understanding is that that shouldn't be a problem. frossie , jhoblitt , any ideas?

            What I suspect is happening is that lsst_build is [re]generating the same bxxxx number because the repo head refs haven't changed. create_xlinkdocs.sh is generating the b number independently of lsst_build and assumes the build id is monotonically incrementing. The obvious problem is that that the two programs don't agree on the build id but I don't yet understand the conditions that put it into this state.

            jhoblitt Joshua Hoblitt added a comment - What I suspect is happening is that lsst_build is [re] generating the same bxxxx number because the repo head refs haven't changed. create_xlinkdocs.sh is generating the b number independently of lsst_build and assumes the build id is monotonically incrementing. The obvious problem is that that the two programs don't agree on the build id but I don't yet understand the conditions that put it into this state.

            A work around has been committed.

            jhoblitt Joshua Hoblitt added a comment - A work around has been committed.
            jbosch Jim Bosch added a comment -

            Thanks for the workaround fix. What package is create_xlinkdocs.sh in? It sounds like that's where the real fix needs to go, and I gather I need to address this before I test this ticket branch with buildbot again.

            jbosch Jim Bosch added a comment - Thanks for the workaround fix. What package is create_xlinkdocs.sh in? It sounds like that's where the real fix needs to go, and I gather I need to address this before I test this ticket branch with buildbot again.
            jbosch Jim Bosch added a comment -

            Nevermind; just saw that K-T had posted a link on HipChat to buildbot_scripts

            jbosch Jim Bosch added a comment - Nevermind; just saw that K-T had posted a link on HipChat to buildbot_scripts
            swinbank John Swinbank made changes -
            Parent Issue DM-1674 [ DM-1674 ] DM-1912 [ DM-1912 ]
            swinbank John Swinbank made changes -
            Parent DM-1912 [ 15945 ]
            Issue Type Technical task [ 10002 ] Story [ 10001 ]
            swinbank John Swinbank made changes -
            Epic Link DM-1912 [ 15945 ]
            swinbank John Swinbank made changes -
            Link This issue relates to DM-1674 [ DM-1674 ]
            swinbank John Swinbank made changes -
            Sprint Science Pipelines DM-S15-4 [ 159 ]
            jbosch Jim Bosch added a comment -

            So, the original problem here is that the buildbot scripts depend explicitly on datarel, assuming it's a top-level package whose Doxygen build depends on every other Doxygen build. So if we remove datarel, we're going to need another package that can fill that role, which should probably just be lsst_distrib (depending on how much of a Doxygen build we have to add to lsst_distrib to get this working). Or we'll need to make deeper changes to lsstDoxygen.

            jhoblitt, is there any way I can get a sandbox to test out changes to buildbot_scripts, or would it be easiest if I just reassigned the rest of this issue to you?

            jbosch Jim Bosch added a comment - So, the original problem here is that the buildbot scripts depend explicitly on datarel , assuming it's a top-level package whose Doxygen build depends on every other Doxygen build. So if we remove datarel, we're going to need another package that can fill that role, which should probably just be lsst_distrib (depending on how much of a Doxygen build we have to add to lsst_distrib to get this working). Or we'll need to make deeper changes to lsstDoxygen. jhoblitt , is there any way I can get a sandbox to test out changes to buildbot_scripts , or would it be easiest if I just reassigned the rest of this issue to you?

            jbosch yes - I put something together for you tomorrow. Perhaps hipchat will let me authenticate by then so I can pass you credentials...

            jhoblitt Joshua Hoblitt added a comment - jbosch yes - I put something together for you tomorrow. Perhaps hipchat will let me authenticate by then so I can pass you credentials...
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-2948 [ DM-2948 ]
            jbosch Jim Bosch added a comment -

            I'm taking a different approach to removing the block on DM-1766: just removing the dependency on ap from datarel (DM-2949), rather than removing datarel from CI (which is blocked by DM-2948). As such, I'm closing this as Won't Fix, and leaving it for the Data Access team to figure if and how they want to move ingest code from datarel later in DM-1674.

            jbosch Jim Bosch added a comment - I'm taking a different approach to removing the block on DM-1766 : just removing the dependency on ap from datarel ( DM-2949 ), rather than removing datarel from CI (which is blocked by DM-2948 ). As such, I'm closing this as Won't Fix, and leaving it for the Data Access team to figure if and how they want to move ingest code from datarel later in DM-1674 .
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Won't Fix [ 10405 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to RFC-22 [ RFC-22 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggered by RFC-57 [ RFC-57 ]

            People

              jbosch Jim Bosch
              jbosch Jim Bosch
              Jim Bosch, Joshua Hoblitt
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.