Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2949

remove dead code and dependencies from datarel

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: datarel
    • Labels:
      None

      Description

      Removing the datarel package entirely has proved to be difficult (DM-2928, DM-2948), so instead I'm simply going to remove non-ingest code (and dead ingest code) from the package, along with its dependencies on ap and testing_endToEnd. Other dependencies will be retained even if they aren't necessary for the code that will remain in datarel, to support lsstDoxygen's use of datarel as a top-level package for documentation generation.

        Attachments

          Issue Links

            Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue blocks DM-1766 [ DM-1766 ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            jbosch Jim Bosch added a comment -

            Changes on tickets/DM-2949 of datarel:

            $ git --no-pager log --reverse LSST/master..tickets/DM-2949
            commit 6e5f88858b38e7001cd27a46dfb8a702ffc46c8a
            Author: Jim Bosch <jbosch@astro.princeton.edu>
            Date:   Fri Jun 19 15:40:16 2015 -0400
             
                Remove relic pipeline and execution code.
                
                This removes all code related to Stages, the old weekly productions,
                and the JobOffice, leaving only ingest and a few utility functions.
             
            commit c82d5d93a4870580b3bb5ef1f601a83d60404622
            Author: Jim Bosch <jbosch@astro.princeton.edu>
            Date:   Fri Jun 19 15:43:03 2015 -0400
             
                Remove all ingest code that relies on lsst.ap, and the dependency on ap.
                
                Premise is that the code we'll want to start from when we get ingest working
                again is the code that doesn't use ap.
             
            commit fb5cabc0f8c8d879b95b8bd4359c9963a47c73cb
            Author: Jim Bosch <jbosch@astro.princeton.edu>
            Date:   Fri Jun 19 15:45:21 2015 -0400
             
                Re-enable unit tests for utility functions and ingest.
                
                Both of these tests were already working, and probably should have been
                enabled earlier; a comment in the SConscript suggests someone wanted
                to just disable all pex_harness-related tests, but got these as well.
                
                The ingest test does require access to a remote server, but I've verified
                that it skips the test and issues a warning when the server cannot be
                contacted, so I think it's safe.
            

            and lsst_distrib:

            $ git show LSST/master..tickets/DM-2949
            commit 29f62650a8dd25d7528f4eca44013bab354c539d
            Author: Jim Bosch <jbosch@astro.princeton.edu>
            Date:   Fri Jun 19 15:55:44 2015 -0400
             
                Remove dependency on testing_endToEnd, which is being retired.
             
            diff --git a/ups/lsst_distrib.table b/ups/lsst_distrib.table
            index 05389da..60420cb 100644
            --- a/ups/lsst_distrib.table
            +++ b/ups/lsst_distrib.table
            @@ -4,4 +4,3 @@ setupRequired(ctrl_execute)
             setupRequired(ctrl_platform_gordon)
             setupRequired(ctrl_platform_lsst)
             setupRequired(datarel)
            -setupRequired(testing_endtoend)
            

            Show
            jbosch Jim Bosch added a comment - Changes on tickets/ DM-2949 of datarel: $ git --no-pager log --reverse LSST/master..tickets/DM-2949 commit 6e5f88858b38e7001cd27a46dfb8a702ffc46c8a Author: Jim Bosch <jbosch@astro.princeton.edu> Date: Fri Jun 19 15:40:16 2015 -0400   Remove relic pipeline and execution code. This removes all code related to Stages, the old weekly productions, and the JobOffice, leaving only ingest and a few utility functions.   commit c82d5d93a4870580b3bb5ef1f601a83d60404622 Author: Jim Bosch <jbosch@astro.princeton.edu> Date: Fri Jun 19 15:43:03 2015 -0400   Remove all ingest code that relies on lsst.ap, and the dependency on ap. Premise is that the code we'll want to start from when we get ingest working again is the code that doesn't use ap.   commit fb5cabc0f8c8d879b95b8bd4359c9963a47c73cb Author: Jim Bosch <jbosch@astro.princeton.edu> Date: Fri Jun 19 15:45:21 2015 -0400   Re-enable unit tests for utility functions and ingest. Both of these tests were already working, and probably should have been enabled earlier; a comment in the SConscript suggests someone wanted to just disable all pex_harness-related tests, but got these as well. The ingest test does require access to a remote server, but I've verified that it skips the test and issues a warning when the server cannot be contacted, so I think it's safe. and lsst_distrib: $ git show LSST/master..tickets/DM-2949 commit 29f62650a8dd25d7528f4eca44013bab354c539d Author: Jim Bosch <jbosch@astro.princeton.edu> Date: Fri Jun 19 15:55:44 2015 -0400   Remove dependency on testing_endToEnd, which is being retired.   diff --git a/ups/lsst_distrib.table b/ups/lsst_distrib.table index 05389da..60420cb 100644 --- a/ups/lsst_distrib.table +++ b/ups/lsst_distrib.table @@ -4,4 +4,3 @@ setupRequired(ctrl_execute) setupRequired(ctrl_platform_gordon) setupRequired(ctrl_platform_lsst) setupRequired(datarel) -setupRequired(testing_endtoend)
            jbosch Jim Bosch made changes -
            Reviewers Kian-Tat Lim [ ktl ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            jbosch Jim Bosch added a comment -
            Show
            jbosch Jim Bosch added a comment - Buildbot run successful: http://lsst-buildx.ncsa.illinois.edu:8010/builders/DM_stack/builds/7224
            Hide
            ktl Kian-Tat Lim added a comment -

            Does it make sense to change the name of what's now left after this ticket to daf_ingest and replace datarel with a new package that just has doc and ups subdirectories (and, I guess, an SConstruct)?

            Show
            ktl Kian-Tat Lim added a comment - Does it make sense to change the name of what's now left after this ticket to daf_ingest and replace datarel with a new package that just has doc and ups subdirectories (and, I guess, an SConstruct )?
            Hide
            ktl Kian-Tat Lim added a comment -

            The removals all look appropriate to me, as does what is left behind. Sad to see some of that stuff go, particularly when we don't necessarily have good replacements for all of it yet...

            Show
            ktl Kian-Tat Lim added a comment - The removals all look appropriate to me, as does what is left behind. Sad to see some of that stuff go, particularly when we don't necessarily have good replacements for all of it yet...
            ktl Kian-Tat Lim made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            jbosch Jim Bosch added a comment -

            I don't think it makes sense to change the name yet, as we're also still using this package as a metapackage for the cross-linked Doxygen build (see DM-2948). That means we're still carrying more dependencies than the ingest code here needs, and we can't change the name without changing the buildbot scripts (which I can't do without help from SQuaRE, and they're busy).

            Show
            jbosch Jim Bosch added a comment - I don't think it makes sense to change the name yet, as we're also still using this package as a metapackage for the cross-linked Doxygen build (see DM-2948 ). That means we're still carrying more dependencies than the ingest code here needs, and we can't change the name without changing the buildbot scripts (which I can't do without help from SQuaRE, and they're busy).
            Hide
            jbosch Jim Bosch added a comment -

            Merged to master.

            Show
            jbosch Jim Bosch added a comment - Merged to master.
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            Hide
            ktl Kian-Tat Lim added a comment -

            Jim Bosch You missed the part about adding a new (nearly-empty) datarel that would serve as the metapackage for the cross-linked Doxygen build and the buildbot scripts.

            Show
            ktl Kian-Tat Lim added a comment - Jim Bosch You missed the part about adding a new (nearly-empty) datarel that would serve as the metapackage for the cross-linked Doxygen build and the buildbot scripts.
            Hide
            jbosch Jim Bosch added a comment -

            Oh, I did miss that part somehow, reading too fast - sorry about that. I'm still not sure it's worthwhile to do right now, as I think a better solution for the Doxygen/buildbot problem is to remove the need for a single top-level package. But we may want to do that first if Jacek Becla's team starts working on updating the ingest code before DM-2948 is addressed.

            Show
            jbosch Jim Bosch added a comment - Oh, I did miss that part somehow, reading too fast - sorry about that. I'm still not sure it's worthwhile to do right now, as I think a better solution for the Doxygen/buildbot problem is to remove the need for a single top-level package. But we may want to do that first if Jacek Becla 's team starts working on updating the ingest code before DM-2948 is addressed.
            swinbank John Swinbank made changes -
            Epic Link DM-1769 [ 15588 ]
            Hide
            swinbank John Swinbank added a comment -

            Adding this to DM-1769 since it was necessary to unblock DM-1766.

            Show
            swinbank John Swinbank added a comment - Adding this to DM-1769 since it was necessary to unblock DM-1766 .
            frossie Frossie Economou made changes -
            Link This issue is duplicated by DM-305 [ DM-305 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Jim Bosch, John Swinbank, Kian-Tat Lim
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.