Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11173

Check log-based provenance of weekly RC processing inputs

    XMLWordPrintable

Details

    • Story
    • Status: Won't Fix
    • Resolution: Done
    • None
    • None
    • None
    • 1
    • DRP F19-5, DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb), DRP S20-5 (Apr), DRP S20-6 (May), DRP F20-1 (June), DRP F20-2 (July), DRP F20-3 (Aug)
    • Data Release Production

    Description

      We'd like to be able to reproduce a RC processing run on a weekly release (or any rerun, but that's beyond the scope of this issue) from the files in the output data repository. hchiang2 believes the logs saved in the weekly RC data repositories (see https://confluence.lsstcorp.org/display/DM/Reprocessing+of+the+HSC+RC+dataset) contain enough information to reconstruct the command-lines run (including the data IDs passed).

      DRP team should check these logs to see if this is true. This activity should be limited to perhaps a half-day of work; if the information may be present but is too difficult to extract from the logs we should pursue other options.

      Attachments

        Activity

          The log record I was thinking is stored through pipe_base CmdLineTask. So it's at the level of CmdLineTask but not at ctrl_pool. It's not full provenance but might be enough to reproduce errors.

          Some examples from /project/hsc_rc/DM-11020/logs/ are:
          (singleFrame/CosmosI.o74938)

          root INFO: Running: -c /datasets/hsc/repo --rerun private/hchiang2/RC/w25 --id visit=1228..1232:2^1236..1248:2^19658^19660^19662^19680^19682^19684^19694^19696^19698^19708^19710^19712^30482..30504:2 ccd=0..8^10..103
          

          (mosaic/mosaic-74949.log)

          root INFO: Running: /home/hchiang2/stack/meas_mosaic/bin/mosaic.py /datasets/hsc/repo --rerun private/hchiang2/RC/w25 --numCoresForRead=12 --id tract=8767 ccd=0..8^10..103 visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598 --diagnostics --diagDir=/project/hsc_rc/w25_mosaic_diag/G
          

          (coadd/coaddWG.o74961)

          root INFO: Running: -c /datasets/hsc/repo --rerun private/hchiang2/RC/w25 --id tract=8766^8767 filter=HSC-G --selectId ccd=0..8^10..103 visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598
          

          (multiband/mbWide8766.o75611)

          root INFO: Running: -c /datasets/hsc/repo --rerun private/hchiang2/RC/w25:private/hchiang2/RC/w25_mb --id tract=8766 filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y patch=0,0^0,1^0,2^0,3^0,4^0,5^0,6^0,7^1,0^1,1^1,2^1,3^1,4^1,5^1,6^1,7^1,8^2,0^2,1^2,2^2,3^2,4^2,5^2,6^2,7^2,8^3,0^3,1^3,2^3,3^3,4^3,5^3,6^3,7^3,8^4,0^4,1^4,2^4,3^4,4^4,5^4,6^4,7^4,8^5,0^5,1^5,2^5,3^5,4^5,5^5,6^5,7^5,8^6,0^6,1^6,2^6,3^6,4^6,5^6,6^6,7^6,8^7,0^7,1^7,2^7,3^7,4^7,5^7,6^7,7^8,0^8,1^8,2^8,3^8,4^8,5^8,6^8,7^8,8
          

          hchiang2 Hsin-Fang Chiang added a comment - The log record I was thinking is stored through pipe_base CmdLineTask . So it's at the level of CmdLineTask but not at ctrl_pool . It's not full provenance but might be enough to reproduce errors. Some examples from /project/hsc_rc/ DM-11020 /logs/ are: (singleFrame/CosmosI.o74938) root INFO: Running: -c /datasets/hsc/repo --rerun private /hchiang2/RC/w25 --id visit= 1228 .. 1232 : 2 ^ 1236 .. 1248 : 2 ^ 19658 ^ 19660 ^ 19662 ^ 19680 ^ 19682 ^ 19684 ^ 19694 ^ 19696 ^ 19698 ^ 19708 ^ 19710 ^ 19712 ^ 30482 .. 30504 : 2 ccd= 0 .. 8 ^ 10 .. 103 (mosaic/mosaic-74949.log) root INFO: Running: /home/hchiang2/stack/meas_mosaic/bin/mosaic.py /datasets/hsc/repo --rerun private /hchiang2/RC/w25 --numCoresForRead= 12 --id tract= 8767 ccd= 0 .. 8 ^ 10 .. 103 visit= 9852 ^ 9856 ^ 9860 ^ 9864 ^ 9868 ^ 9870 ^ 9888 ^ 9890 ^ 9898 ^ 9900 ^ 9904 ^ 9906 ^ 9912 ^ 11568 ^ 11572 ^ 11576 ^ 11582 ^ 11588 ^ 11590 ^ 11596 ^ 11598 --diagnostics --diagDir=/project/hsc_rc/w25_mosaic_diag/G (coadd/coaddWG.o74961) root INFO: Running: -c /datasets/hsc/repo --rerun private /hchiang2/RC/w25 --id tract= 8766 ^ 8767 filter=HSC-G --selectId ccd= 0 .. 8 ^ 10 .. 103 visit= 9852 ^ 9856 ^ 9860 ^ 9864 ^ 9868 ^ 9870 ^ 9888 ^ 9890 ^ 9898 ^ 9900 ^ 9904 ^ 9906 ^ 9912 ^ 11568 ^ 11572 ^ 11576 ^ 11582 ^ 11588 ^ 11590 ^ 11596 ^ 11598 (multiband/mbWide8766.o75611) root INFO: Running: -c /datasets/hsc/repo --rerun private /hchiang2/RC/w25: private /hchiang2/RC/w25_mb --id tract= 8766 filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y patch= 0 , 0 ^ 0 , 1 ^ 0 , 2 ^ 0 , 3 ^ 0 , 4 ^ 0 , 5 ^ 0 , 6 ^ 0 , 7 ^ 1 , 0 ^ 1 , 1 ^ 1 , 2 ^ 1 , 3 ^ 1 , 4 ^ 1 , 5 ^ 1 , 6 ^ 1 , 7 ^ 1 , 8 ^ 2 , 0 ^ 2 , 1 ^ 2 , 2 ^ 2 , 3 ^ 2 , 4 ^ 2 , 5 ^ 2 , 6 ^ 2 , 7 ^ 2 , 8 ^ 3 , 0 ^ 3 , 1 ^ 3 , 2 ^ 3 , 3 ^ 3 , 4 ^ 3 , 5 ^ 3 , 6 ^ 3 , 7 ^ 3 , 8 ^ 4 , 0 ^ 4 , 1 ^ 4 , 2 ^ 4 , 3 ^ 4 , 4 ^ 4 , 5 ^ 4 , 6 ^ 4 , 7 ^ 4 , 8 ^ 5 , 0 ^ 5 , 1 ^ 5 , 2 ^ 5 , 3 ^ 5 , 4 ^ 5 , 5 ^ 5 , 6 ^ 5 , 7 ^ 5 , 8 ^ 6 , 0 ^ 6 , 1 ^ 6 , 2 ^ 6 , 3 ^ 6 , 4 ^ 6 , 5 ^ 6 , 6 ^ 6 , 7 ^ 6 , 8 ^ 7 , 0 ^ 7 , 1 ^ 7 , 2 ^ 7 , 3 ^ 7 , 4 ^ 7 , 5 ^ 7 , 6 ^ 7 , 7 ^ 8 , 0 ^ 8 , 1 ^ 8 , 2 ^ 8 , 3 ^ 8 , 4 ^ 8 , 5 ^ 8 , 6 ^ 8 , 7 ^ 8 , 8

          The new nominal path is now /datasets/hsc/repo/rerun/RC/w_2017_25/DM-11020/logs/

          hchiang2 Hsin-Fang Chiang added a comment - The new nominal path is now /datasets/hsc/repo/rerun/RC/w_2017_25/ DM-11020 /logs/
          jbosch Jim Bosch added a comment -

          Any objection to just closing this as Won't Fix? We've been carrying it from sprint to sprint for a long time, and I don't see us getting it before Gen3 gives us a better direction in which to focus any provenance work.

          jbosch Jim Bosch added a comment - Any objection to just closing this as Won't Fix? We've been carrying it from sprint to sprint for a long time, and I don't see us getting it before Gen3 gives us a better direction in which to focus any provenance work.

          Big, huge  from me on Won’t Fix-ing this one!

          lauren Lauren MacArthur added a comment - Big, huge   from me on Won’t Fix-ing this one!

          People

            lauren Lauren MacArthur
            jbosch Jim Bosch
            Hsin-Fang Chiang, Jim Bosch, Lauren MacArthur
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.