Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2928

move old ingest scripts into and retire old packages

    Details

      Description

      This ticket implements RFC-57, by:

      • renaming datarel to daf_ingest (there is already a daf_ingest package, but it's completely empty, so I'll just force-push it all away)
      • removing everything from the renamed package that doesn't relate to ingest (including pruning dependencies)
      • removing ap and testing_endToEnd from the CI system

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            This is essentially ready, but I'm seeing a buildbot failure on the ticket branch I don't understand; the log is here.

            I've removed datarel and testing_endToEnd from lsst_distrib, and now, after running lsstDoxygen, something tries to do a eups list that includes datarel, and that fails. I don't see the string "datarel" in lsstsw anywhere but repos.yaml, and my understanding is that that shouldn't be a problem. Frossie Economou, Joshua Hoblitt, any ideas?

            Show
            jbosch Jim Bosch added a comment - This is essentially ready, but I'm seeing a buildbot failure on the ticket branch I don't understand; the log is here . I've removed datarel and testing_endToEnd from lsst_distrib, and now, after running lsstDoxygen , something tries to do a eups list that includes datarel , and that fails. I don't see the string "datarel" in lsstsw anywhere but repos.yaml, and my understanding is that that shouldn't be a problem. Frossie Economou , Joshua Hoblitt , any ideas?
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            What I suspect is happening is that lsst_build is [re]generating the same bxxxx number because the repo head refs haven't changed. create_xlinkdocs.sh is generating the b number independently of lsst_build and assumes the build id is monotonically incrementing. The obvious problem is that that the two programs don't agree on the build id but I don't yet understand the conditions that put it into this state.

            Show
            jhoblitt Joshua Hoblitt added a comment - What I suspect is happening is that lsst_build is [re] generating the same bxxxx number because the repo head refs haven't changed. create_xlinkdocs.sh is generating the b number independently of lsst_build and assumes the build id is monotonically incrementing. The obvious problem is that that the two programs don't agree on the build id but I don't yet understand the conditions that put it into this state.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            A work around has been committed.

            Show
            jhoblitt Joshua Hoblitt added a comment - A work around has been committed.
            Hide
            jbosch Jim Bosch added a comment -

            Thanks for the workaround fix. What package is create_xlinkdocs.sh in? It sounds like that's where the real fix needs to go, and I gather I need to address this before I test this ticket branch with buildbot again.

            Show
            jbosch Jim Bosch added a comment - Thanks for the workaround fix. What package is create_xlinkdocs.sh in? It sounds like that's where the real fix needs to go, and I gather I need to address this before I test this ticket branch with buildbot again.
            Hide
            jbosch Jim Bosch added a comment -

            Nevermind; just saw that K-T had posted a link on HipChat to buildbot_scripts

            Show
            jbosch Jim Bosch added a comment - Nevermind; just saw that K-T had posted a link on HipChat to buildbot_scripts
            Hide
            jbosch Jim Bosch added a comment -

            So, the original problem here is that the buildbot scripts depend explicitly on datarel, assuming it's a top-level package whose Doxygen build depends on every other Doxygen build. So if we remove datarel, we're going to need another package that can fill that role, which should probably just be lsst_distrib (depending on how much of a Doxygen build we have to add to lsst_distrib to get this working). Or we'll need to make deeper changes to lsstDoxygen.

            Joshua Hoblitt, is there any way I can get a sandbox to test out changes to buildbot_scripts, or would it be easiest if I just reassigned the rest of this issue to you?

            Show
            jbosch Jim Bosch added a comment - So, the original problem here is that the buildbot scripts depend explicitly on datarel , assuming it's a top-level package whose Doxygen build depends on every other Doxygen build. So if we remove datarel, we're going to need another package that can fill that role, which should probably just be lsst_distrib (depending on how much of a Doxygen build we have to add to lsst_distrib to get this working). Or we'll need to make deeper changes to lsstDoxygen. Joshua Hoblitt , is there any way I can get a sandbox to test out changes to buildbot_scripts , or would it be easiest if I just reassigned the rest of this issue to you?
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Jim Bosch yes - I put something together for you tomorrow. Perhaps hipchat will let me authenticate by then so I can pass you credentials...

            Show
            jhoblitt Joshua Hoblitt added a comment - Jim Bosch yes - I put something together for you tomorrow. Perhaps hipchat will let me authenticate by then so I can pass you credentials...
            Hide
            jbosch Jim Bosch added a comment -

            I'm taking a different approach to removing the block on DM-1766: just removing the dependency on ap from datarel (DM-2949), rather than removing datarel from CI (which is blocked by DM-2948). As such, I'm closing this as Won't Fix, and leaving it for the Data Access team to figure if and how they want to move ingest code from datarel later in DM-1674.

            Show
            jbosch Jim Bosch added a comment - I'm taking a different approach to removing the block on DM-1766 : just removing the dependency on ap from datarel ( DM-2949 ), rather than removing datarel from CI (which is blocked by DM-2948 ). As such, I'm closing this as Won't Fix, and leaving it for the Data Access team to figure if and how they want to move ingest code from datarel later in DM-1674 .

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Jim Bosch, Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel