Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33457

RC2 reprocessing with w_2022_04 (gen2)

    XMLWordPrintable

    Details

    • Story Points:
      5
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Process the RC2 dataset with the w_2022_04 weekly.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Note that I had to comment out the following line in the fgcm fgcmOutputProductsRC2new.py config file:

            # Default reference catalog filter to use if filter not specified in exposure; if blank then filter must be specified in exposure
            # config.refObjLoader.defaultFilter='g, r, i, z, N921, y'
            

            as the defaultFilter has been deprecated (DM-26231). I don't think this will have any consequences (can you confirm Eli Rykoff?)

            Show
            lauren Lauren MacArthur added a comment - Note that I had to comment out the following line in the fgcm fgcmOutputProductsRC2new.py config file: # Default reference catalog filter to use if filter not specified in exposure; if blank then filter must be specified in exposure # config.refObjLoader.defaultFilter='g, r, i, z, N921, y' as the defaultFilter has been deprecated ( DM-26231 ). I don't think this will have any consequences (can you confirm Eli Rykoff ?)
            Hide
            lauren Lauren MacArthur added a comment -

            For jointcal, in order to best match current Gen3 defaults, I am running with the config override file:

            /home/lauren/tickets/DM-29008/configJointcalGen2_lam.py
            

            (also attached to DM-29008).

            Show
            lauren Lauren MacArthur added a comment - For jointcal , in order to best match current Gen3 defaults, I am running with the config override file: / home / lauren / tickets / DM - 29008 / configJointcalGen2_lam.py (also attached to DM-29008 ).
            Hide
            dtaranu Dan Taranu added a comment - - edited

            Did something go wrong with this run? I don't see an objectTable_tract output for any of the tract trying:

            ls -al /datasets/hsc/repo/rerun/RC/w_202[12]_*/DM-*/deepCoadd-results/merged/*/objectTable*.parq

            (I don't need it and am not planning on following this up, I was just trying to confirm that gen2to3 conversion caught everything that succeeded for DM-33855).

             

            Show
            dtaranu Dan Taranu added a comment - - edited Did something go wrong with this run? I don't see an objectTable_tract output for any of the tract trying: ls -al /datasets/hsc/repo/rerun/RC/w_202[12]_*/DM-*/deepCoadd-results/merged/*/objectTable*.parq (I don't need it and am not planning on following this up, I was just trying to confirm that gen2to3 conversion caught everything that succeeded for DM-33855 ).  
            Hide
            lauren Lauren MacArthur added a comment -

            Huh...nothing deeper than I don't appear to have run them (I guess I got distracted with the parity checks that did not include these tables and forgot to to back and finish off the processing. Well-spotted!)  I guess I jumped the gun on checking off the boxes in the weekly reprocessing spreadsheet (https://docs.google.com/spreadsheets/d/1KTwTetVJoRBXAb3tQKL3-XzYadRndyoIxY3dTyT85qk/edit#gid=0, sorry Brock Brendal [X]). I just ran them, so all should be in place now, but I'm not sure it's worth updating/re-running the gen2to3 conversion.

            Show
            lauren Lauren MacArthur added a comment - Huh...nothing deeper than I don't appear to have run them (I guess I got distracted with the parity checks that did not include these tables and forgot to to back and finish off the processing. Well-spotted!)  I guess I jumped the gun on checking off the boxes in the weekly reprocessing spreadsheet ( https://docs.google.com/spreadsheets/d/1KTwTetVJoRBXAb3tQKL3-XzYadRndyoIxY3dTyT85qk/edit#gid=0 , sorry Brock Brendal [X] ). I just ran them, so all should be in place now, but I'm not sure it's worth updating/re-running the gen2to3 conversion.
            Hide
            lauren Lauren MacArthur added a comment -

            Run is now complete. Saw the same set of errors and warnings as reported in DM-31477, DM-31815, DM-32419, & DM-32766 (no new ones noticed).

            Logs can be found at (or linked here):

            /datasets/hsc/repo/rerun/RC/w_2022_04/DM-33457/logs
            

            FGCM plots and logs are linked here

            pipe_analysis plots: gen2, gen3 + gen2 vs gen3.

            matchedVisitMetrics output linked here

            Modulo the [s]tellar locus issue of DM-34286 (which also seems to apply to the [s]kyObject* metrics), the metrics were successfully dispatched, ingested to SQuaSH, and now appear on the DRP metrics monthly Gen 2 (HSC RC2) dashboard in chronograph.

            Show
            lauren Lauren MacArthur added a comment - Run is now complete. Saw the same set of errors and warnings as reported in DM-31477 , DM-31815 , DM-32419 , & DM-32766 (no new ones noticed). Logs can be found at (or linked here ): / datasets / hsc / repo / rerun / RC / w_2022_04 / DM - 33457 / logs FGCM plots and logs are linked here pipe_analysis plots: gen2 , gen3 + gen2 vs gen3 . matchedVisitMetrics output linked here Modulo the [s] tellar locus issue of DM-34286 (which also seems to apply to the [s] kyObject* metrics), the metrics were successfully dispatched, ingested to SQuaSH, and now appear on the DRP metrics monthly Gen 2 (HSC RC2) dashboard in chronograph .
            Hide
            lauren Lauren MacArthur added a comment -

            Since you've already looked into this one, would you mind doing the final cursory check & approval?

            Show
            lauren Lauren MacArthur added a comment - Since you've already looked into this one, would you mind doing the final cursory check & approval?
            Hide
            dtaranu Dan Taranu added a comment -

            The plots certainly appear to be there, as do the objectTable_tract datasets.

            I think the missing datasets could be ingested fairly easily by re-running the gen2to3 conversion script, but I'm not completely sure that it would work without issue and concur that there isn't any reason to try. If someone does find a compelling need, please correct me.

            Show
            dtaranu Dan Taranu added a comment - The plots certainly appear to be there, as do the objectTable_tract datasets. I think the missing datasets could be ingested fairly easily by re-running the gen2to3 conversion script, but I'm not completely sure that it would work without issue and concur that there isn't any reason to try. If someone does find a compelling need, please correct me.
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks, Dan!

            Show
            lauren Lauren MacArthur added a comment - Thanks, Dan!

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Dan Taranu
              Watchers:
              Dan Taranu, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.