# Create gaia dr2 refcat for ap_verify_hits2015

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
2
• Sprint:
AP S20-1 (December)
• Team:

#### Description

I'm using ap_verify_ci_hits2015 to test my gen2->gen3 repo converter code, and I ran into a problem due to the use of the gaia dr1 refcat, which uses htm depth=8. Gen3 repos all have to use depth=7 for the moment. The easiest solution is to just extract the necessary pixels from the new gaia dr2 refcat and replace the old one in ap_verify_hits. That also gains us an upgraded reference catalog.

This should be an easy change to scripts/gaia_HiTS_2015.py and then a bit of processing on lsst-dev.

#### Attachments

1. ap_verify_master_vs_22676.png
212 kB

#### Activity

Hide
John Parejko added a comment -

Meredith Rawls: I think you're probably in the best place to review this. I suspect it may change some of the measured metrics. Probably the best way to test it is to run through each dataset's pipeline and see if things are still happy?

Show
John Parejko added a comment - Meredith Rawls : I think you're probably in the best place to review this. I suspect it may change some of the measured metrics. Probably the best way to test it is to run through each dataset's pipeline and see if things are still happy?
Hide
John Parejko added a comment -

Question: the gen3 butler doesn't allow - in a dataset name, so the pan-starrs refcat in these test data have to be renamed. I would probably recommend calling it either ps1 (which is the data release it came from) or just panstarrs (even though the real name has a hyphen). Should I make that change on this ticket, or should we do it on another one? Would we need to change any configs other than the two in both ap_verify_hits config/ directories?

Show
John Parejko added a comment - Question: the gen3 butler doesn't allow - in a dataset name, so the pan-starrs refcat in these test data have to be renamed. I would probably recommend calling it either ps1 (which is the data release it came from) or just panstarrs (even though the real name has a hyphen). Should I make that change on this ticket, or should we do it on another one? Would we need to change any configs other than the two in both ap_verify_hits config/ directories?
Hide
Eric Bellm added a comment -

Note that there are two PS1 data releases (DR1 and DR2).

Show
Eric Bellm added a comment - Note that there are two PS1 data releases (DR1 and DR2).
Hide
Meredith Rawls added a comment -

A few things.

The changes in the datasets themselves look fine, and thanks (John) for explaining HTM trixels and "depth" to me via DM.

I ran ap_verify with the CI dataset on this ticket branch, and it failed with

 apPipe.ccdProcessor.calibrate.astromRefObjLoader INFO: Loading reference objects using center (154.967854, -5.784114) and radius 0.19703074992397315 deg apPipe FATAL: Failed on dataId={'date': '2015-02-18', 'filter': 'g', 'visit': 411420, 'hdu': 2, 'ccdnum': 5, 'object': 'Blind15A_40'}: RuntimeError: Unknown reference filter phot_g_mean_mag_flux

I think this failure is related to the new refcat and needs to be addressed before this ticket is done.

It then subsequently failed with

 AttributeError: lsst.pipe.base.config.Connections has no attribute taskName

Which, incidentally, I am beginning to think is the new "no locations for get" (in its pervasiveness and opaqueness, not what it actually means). I believe this is the ap_pipe bug Krzysztof Findeisen caught and fixed earlier this week, so for now, don't worry about it.

Regarding Pan-STARRS as the photometric refcat. Please specify in both datasets' READMEs that it includes Gaia DR2 and Pan-STARRS PS1 DRX (I don't know if it's DR1 or DR2?), per Eric's comment above. I am quite annoyed that gen3 does not permit hyphens in refcat names, as this is the kind of thing that will create significant overhead for folks who are just trying to do a processCcd rerun and suddenly it doesn't work anymore. That said, assuming it truly is necessary to banish hyphens from our refcats forever, I think it is fine to update the pan-starrs name to be hyphen free on this ticket. I think I would vote for pan_starrs or panstarrs (and not, e.g., PS1).

Show
Meredith Rawls added a comment - A few things. The changes in the datasets themselves look fine, and thanks (John) for explaining HTM trixels and "depth" to me via DM. I ran ap_verify with the CI dataset on this ticket branch, and it failed with apPipe.ccdProcessor.calibrate.astromRefObjLoader INFO: Loading reference objects using center ( 154.967854 , - 5.784114 ) and radius 0.19703074992397315 deg apPipe FATAL: Failed on dataId={ 'date' : '2015-02-18' , 'filter' : 'g' , 'visit' : 411420 , 'hdu' : 2 , 'ccdnum' : 5 , 'object' : 'Blind15A_40' }: RuntimeError: Unknown reference filter phot_g_mean_mag_flux I think this failure is related to the new refcat and needs to be addressed before this ticket is done. It then subsequently failed with AttributeError: lsst.pipe.base.config.Connections has no attribute taskName Which, incidentally, I am beginning to think is the new "no locations for get" (in its pervasiveness and opaqueness, not what it actually means). I believe this is the ap_pipe bug Krzysztof Findeisen caught and fixed earlier this week, so for now, don't worry about it. Regarding Pan-STARRS as the photometric refcat. Please specify in both datasets' READMEs that it includes Gaia DR2 and Pan-STARRS PS1 DRX (I don't know if it's DR1 or DR2?), per Eric's comment above. I am quite annoyed that gen3 does not permit hyphens in refcat names, as this is the kind of thing that will create significant overhead for folks who are just trying to do a processCcd rerun and suddenly it doesn't work anymore. That said, assuming it truly is necessary to banish hyphens from our refcats forever, I think it is fine to update the pan-starrs name to be hyphen free on this ticket. I think I would vote for pan_starrs or panstarrs (and not, e.g., PS1).
Hide
Krzysztof Findeisen added a comment -

Yes, I fixed it on DM-22641. (For the record, that error is not from the Butler at all: I was assigning to a config field that didn't exist. I think the AttributeError might have been raised by Python itself.)

Show
Krzysztof Findeisen added a comment - Yes, I fixed it on DM-22641 . (For the record, that error is not from the Butler at all: I was assigning to a config field that didn't exist. I think the  AttributeError might have been raised by Python itself.)
Hide
John Parejko added a comment -

Thank you, Meredith Rawls. I've made some changes and got a successful run of ci_hits2015 on my desktop (I'm running the full hits2015 test overnight now). Could you please try it again and see if it works for you?

Show
John Parejko added a comment - Thank you, Meredith Rawls . I've made some changes and got a successful run of ci_hits2015 on my desktop (I'm running the full hits2015 test overnight now). Could you please try it again and see if it works for you?
Hide
Meredith Rawls added a comment -

Thanks! I also ran the CI dataset and made a quick plot of the good DIA Sources on the sky on this ticket branch vs. the current master (plot is attached to this ticket). It looks essentially the same; the new version has 337 good sources vs. 362 good sources in master. Assuming nothing falls over when you run the full hits2015, I think this is all set.

Show
Meredith Rawls added a comment - Thanks! I also ran the CI dataset and made a quick plot of the good DIA Sources on the sky on this ticket branch vs. the current master (plot is attached to this ticket). It looks essentially the same; the new version has 337 good sources vs. 362 good sources in master. Assuming nothing falls over when you run the full hits2015, I think this is all set.
Hide
John Parejko added a comment -

Thanks Meredith Rawls; the full run didn't produce errors, so we should be good.

Merged and done.

Show
John Parejko added a comment - Thanks Meredith Rawls ; the full run didn't produce errors, so we should be good. Merged and done.

#### People

Assignee:
John Parejko
Reporter:
John Parejko
Reviewers:
Meredith Rawls
Watchers:
Eric Bellm, Ian Sullivan, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls