# Fixup Jointcal after DM-31825

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
• Story Points:
2
• Team:
Data Release Production
• Urgent?:
No

#### Description

On DM-31825, the sourceTable_visits that the ticket branches for jointcal and fgcmcal were tested with were different then the final RFC-807/DM-31825 sourceTable_visits.

100% my fault. Fixing up here.

#### Activity

Hide

OK, this is mostly a testing ticket. I built w40 source tables at u/yusra/RC2/w40 in gen3 /repo/main and private/yusra/RC2/DM-31825 in gen2 /datasets/hsc/repo.

So far I've tested:

### gen3:

• jointcal: pipetask run -b /repo/main/butler.yaml -o u/yusra/RC2/w40 -d "" --register-dataset-types -p $OBS_SUBARU_DIR/pipelines/DRP.yaml#jointcal &> /project/yusra/dp0tables/DM-31825/jointcal2.log • fgcm: pipetask run -b /repo/main/butler.yaml -o u/yusra/RC2/w40 -d "" --register-dataset-types -p$OBS_SUBARU_DIR/pipelines/DRP.yaml#fgcm &> /project/yusra/dp0tables/DM-31825/fgcmcal.log

### gen2:

• jointcal:

  jointcal.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/yusra/RC2/DM-31825 --id ccd=0..8^10..103 visit=1202^1204^1206^1208^1210^1212^1214^1216^1218^1220^23692^23694^23704^23706^23716^23718 filter=HSC-R tract=9813 --no-versions &> /project/yusra/dp0tables/jointcal_gen2.log &  jointcal.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/yusra/RC2/DM-31825 --id ccd=0..8^10..103 visit=1202^1208^1220^23716^23718 filter=HSC-R tract=9813 -j 4 --no-versions &> /project/yusra/dp0tables/jointcal_gen2_few_visits.log & 

• fgcm: Lauren MacArthur or Eli Rykoff I need help with this one. I don't think I've ever ran this on gen2, and can't find instructions. I don't think this was right: {{fgcmBuildStarsTable.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/yusra/RC2/DM-31825 --id filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y --no-versions &> build_stars_gen2.log }}}
Show
Yusra AlSayyad added a comment - OK, this is mostly a testing ticket. I built w40 source tables at u/yusra/RC2/w40 in gen3 /repo/main and private/yusra/RC2/ DM-31825 in gen2 /datasets/hsc/repo. So far I've tested: gen3: jointcal: pipetask run -b /repo/main/butler.yaml -o u/yusra/RC2/w40 -d "" --register-dataset-types -p $OBS_SUBARU_DIR/pipelines/DRP.yaml#jointcal &> /project/yusra/dp0tables/ DM-31825 /jointcal2.log fgcm : pipetask run -b /repo/main/butler.yaml -o u/yusra/RC2/w40 -d "" --register-dataset-types -p$OBS_SUBARU_DIR/pipelines/DRP.yaml#fgcm &> /project/yusra/dp0tables/ DM-31825 /fgcmcal.log gen2: jointcal: jointcal.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/yusra/RC2/DM-31825 --id ccd=0..8^10..103 visit=1202^1204^1206^1208^1210^1212^1214^1216^1218^1220^23692^23694^23704^23706^23716^23718 filter=HSC-R tract=9813 --no-versions &> /project/yusra/dp0tables/jointcal_gen2.log & jointcal.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/yusra/RC2/DM-31825 --id ccd=0..8^10..103 visit=1202^1208^1220^23716^23718 filter=HSC-R tract=9813 -j 4 --no-versions &> /project/yusra/dp0tables/jointcal_gen2_few_visits.log & fgcm: Lauren MacArthur or Eli Rykoff I need help with this one. I don't think I've ever ran this on gen2, and can't find instructions. I don't think this was right: {{fgcmBuildStarsTable.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/yusra/RC2/ DM-31825 --id filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y --no-versions &> build_stars_gen2.log }}}
Hide

Opened an initial jointcal PR. The "sourceId" in the columns list to be read from the parquet does no harm.
It's smart enough to recognize it, but bring it back as the index (same as if you have read the whole table). I don't really care if it's there or not. If you prefer gone, it's gone.

I don't THINK I need to make any changes to fgcmcal, but I want to test with gen2 before concluding that.

Show
Yusra AlSayyad added a comment - Opened an initial jointcal PR. The "sourceId" in the columns list to be read from the parquet does no harm. It's smart enough to recognize it, but bring it back as the index (same as if you have read the whole table). I don't really care if it's there or not. If you prefer gone, it's gone. I don't THINK I need to make any changes to fgcmcal, but I want to test with gen2 before concluding that.
Hide
Lauren MacArthur added a comment -

The effective instructions I follow for gen2 fgcm runs are in https://github.com/lsst-dm/gen2gen/blob/master/FGCM/all_steps.sh (you’ll need the configs files in https://github.com/lsst-dm/gen2gen/blob/master/FGCM/).  I’m not sure if it’s overkill to run the entire script…

Show
Lauren MacArthur added a comment - The effective instructions I follow for gen2 fgcm runs are in https://github.com/lsst-dm/gen2gen/blob/master/FGCM/all_steps.sh  (you’ll need the configs files in https://github.com/lsst-dm/gen2gen/blob/master/FGCM/ ).  I’m not sure if it’s overkill to run the entire script…
Hide
Eli Rykoff added a comment -

So Yusra AlSayyad I did run your suggested command and the star ingestion worked fine (which is relevant for this ticket). And I have confirmed that I don't use sourceId in fgcmcal so this isn't a problem. But it then crashed later on because the default fgcmcal LUT that's in the gen2 repo is not compatible with the current code; we use a separate one for the RC2 runs (because of the different filters), and fixing this is not a high priority since gen2 is disappearing anyway and we're not running it on non-RC2 data.

I concur that the fgcmcal is good to go for gen2/gen3.

For reading in sourceId in the columns list for jointcal, I'd like to leave it there because it is a column we want to make sure we're reading. I am unhappy that we're depending on idiosyncrasies of pandas in our code, but that's a discussion for another day.

Show
Eli Rykoff added a comment - So Yusra AlSayyad I did run your suggested command and the star ingestion worked fine (which is relevant for this ticket). And I have confirmed that I don't use sourceId in fgcmcal so this isn't a problem. But it then crashed later on because the default fgcmcal LUT that's in the gen2 repo is not compatible with the current code; we use a separate one for the RC2 runs (because of the different filters), and fixing this is not a high priority since gen2 is disappearing anyway and we're not running it on non-RC2 data. I concur that the fgcmcal is good to go for gen2/gen3. For reading in sourceId in the columns list for jointcal , I'd like to leave it there because it is a column we want to make sure we're reading. I am unhappy that we're depending on idiosyncrasies of pandas in our code, but that's a discussion for another day.
Hide

In that case, this is ready for review.

Show
Hide
Eli Rykoff added a comment -

Show
Hide

Thanks!

Show

#### People

Assignee:
Reporter:
Reviewers:
Eli Rykoff
Watchers:
Eli Rykoff, Lauren MacArthur, Yusra AlSayyad