# RC2 reprocessing with w_2022_04 (gen2)

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
• Story Points:
5
• Team:
Data Release Production
• Urgent?:
No

#### Description

Process the RC2 dataset with the w_2022_04 weekly.

#### Activity

Hide
Lauren MacArthur added a comment -

Note that I had to comment out the following line in the fgcm fgcmOutputProductsRC2new.py config file:

 # Default reference catalog filter to use if filter not specified in exposure; if blank then filter must be specified in exposure # config.refObjLoader.defaultFilter='g, r, i, z, N921, y' 

as the defaultFilter has been deprecated (DM-26231). I don't think this will have any consequences (can you confirm Eli Rykoff?)

Show
Lauren MacArthur added a comment - Note that I had to comment out the following line in the fgcm fgcmOutputProductsRC2new.py config file: # Default reference catalog filter to use if filter not specified in exposure; if blank then filter must be specified in exposure # config.refObjLoader.defaultFilter='g, r, i, z, N921, y' as the defaultFilter has been deprecated ( DM-26231 ). I don't think this will have any consequences (can you confirm Eli Rykoff ?)
Hide
Lauren MacArthur added a comment -

For jointcal, in order to best match current Gen3 defaults, I am running with the config override file:

 /home/lauren/tickets/DM-29008/configJointcalGen2_lam.py 

(also attached to DM-29008).

Show
Lauren MacArthur added a comment - For jointcal , in order to best match current Gen3 defaults, I am running with the config override file: / home / lauren / tickets / DM - 29008 / configJointcalGen2_lam.py (also attached to DM-29008 ).
Hide
Dan Taranu added a comment - - edited

Did something go wrong with this run? I don't see an objectTable_tract output for any of the tract trying:

 ls -al /datasets/hsc/repo/rerun/RC/w_202[12]_*/DM-*/deepCoadd-results/merged/*/objectTable*.parq

(I don't need it and am not planning on following this up, I was just trying to confirm that gen2to3 conversion caught everything that succeeded for DM-33855).

Show
Dan Taranu added a comment - - edited Did something go wrong with this run? I don't see an objectTable_tract output for any of the tract trying: ls -al /datasets/hsc/repo/rerun/RC/w_202[12]_*/DM-*/deepCoadd-results/merged/*/objectTable*.parq (I don't need it and am not planning on following this up, I was just trying to confirm that gen2to3 conversion caught everything that succeeded for DM-33855 ).
Hide
Lauren MacArthur added a comment -

Huh...nothing deeper than I don't appear to have run them (I guess I got distracted with the parity checks that did not include these tables and forgot to to back and finish off the processing. Well-spotted!)  I guess I jumped the gun on checking off the boxes in the weekly reprocessing spreadsheet (https://docs.google.com/spreadsheets/d/1KTwTetVJoRBXAb3tQKL3-XzYadRndyoIxY3dTyT85qk/edit#gid=0, sorry Brock Brendal [X]). I just ran them, so all should be in place now, but I'm not sure it's worth updating/re-running the gen2to3 conversion.

Show
Lauren MacArthur added a comment - Huh...nothing deeper than I don't appear to have run them (I guess I got distracted with the parity checks that did not include these tables and forgot to to back and finish off the processing. Well-spotted!)  I guess I jumped the gun on checking off the boxes in the weekly reprocessing spreadsheet ( https://docs.google.com/spreadsheets/d/1KTwTetVJoRBXAb3tQKL3-XzYadRndyoIxY3dTyT85qk/edit#gid=0 , sorry Brock Brendal [X] ). I just ran them, so all should be in place now, but I'm not sure it's worth updating/re-running the gen2to3 conversion.
Hide
Lauren MacArthur added a comment -

Run is now complete. Saw the same set of errors and warnings as reported in DM-31477, DM-31815, DM-32419, & DM-32766 (no new ones noticed).

Logs can be found at (or linked here):

 /datasets/hsc/repo/rerun/RC/w_2022_04/DM-33457/logs 

FGCM plots and logs are linked here

pipe_analysis plots: gen2, gen3 + gen2 vs gen3.

Modulo the [s]tellar locus issue of DM-34286 (which also seems to apply to the [s]kyObject* metrics), the metrics were successfully dispatched, ingested to SQuaSH, and now appear on the DRP metrics monthly Gen 2 (HSC RC2) dashboard in chronograph.

Show
Lauren MacArthur added a comment - Run is now complete. Saw the same set of errors and warnings as reported in DM-31477 , DM-31815 , DM-32419 , & DM-32766 (no new ones noticed). Logs can be found at (or linked here ): / datasets / hsc / repo / rerun / RC / w_2022_04 / DM - 33457 / logs FGCM plots and logs are linked here pipe_analysis plots: gen2 , gen3 + gen2 vs gen3 . matchedVisitMetrics output linked here Modulo the [s] tellar locus issue of DM-34286 (which also seems to apply to the [s] kyObject* metrics), the metrics were successfully dispatched, ingested to SQuaSH, and now appear on the DRP metrics monthly Gen 2 (HSC RC2) dashboard in chronograph .
Hide
Lauren MacArthur added a comment -

Since you've already looked into this one, would you mind doing the final cursory check & approval?

Show
Lauren MacArthur added a comment - Since you've already looked into this one, would you mind doing the final cursory check & approval?
Hide
Dan Taranu added a comment -

The plots certainly appear to be there, as do the objectTable_tract datasets.

I think the missing datasets could be ingested fairly easily by re-running the gen2to3 conversion script, but I'm not completely sure that it would work without issue and concur that there isn't any reason to try. If someone does find a compelling need, please correct me.

Show
Dan Taranu added a comment - The plots certainly appear to be there, as do the objectTable_tract datasets. I think the missing datasets could be ingested fairly easily by re-running the gen2to3 conversion script, but I'm not completely sure that it would work without issue and concur that there isn't any reason to try. If someone does find a compelling need, please correct me.
Hide
Lauren MacArthur added a comment -

Thanks, Dan!

Show
Lauren MacArthur added a comment - Thanks, Dan!

#### People

Assignee:
Lauren MacArthur
Reporter:
Lauren MacArthur
Reviewers:
Dan Taranu
Watchers:
Dan Taranu, Lauren MacArthur