Hi Meredith Rawls, can you take a look at this reworking of the ApPipeTask configs? Thanks!

If you have any workflows on data other than ap_verify_hits2015 or the HSC datasets I mentioned in my last comment, I'd appreciate if you could try out the new system and tell me if the burden on the command-line user is too much.

Thank you for the clear use cases! I'm impressed you've gotten ap_pipe to work with HSC data.

It looks like you ran ap_pipe on lsst-dev, but I'm not sure what working directory you were in, and I'd like to take a look at the actual processed images if possible. Can you please point me to somewhere I can see them?

I may be missing something, but it looks like you moved some critical config info from ap_pipe into the dataset repo (i.e., ap_verify_hits2015) because it was DECam-specific. Did you generate a similar but HSC-style config file for refcats, filterMaps, etc. for your three use case test runs? I am concerned about requiring users to have a bunch of custom configs set in order to run ap_pipe at all, and I wonder if it would make sense for the obs_packages to have a set of default refcat/filterMap configs that advanced users may choose to override.

Everything was run from lsst-dev01:/scratch/krzys001/. I'll have to look at the PR comments about "critical config info", but the default refcats for ProcessCcdTask (DECam, HSC) work fine on lsst-dev.

I had to create a custom config for the third run (the one that used validation_data_hsc), but that's because Astrometry.net refcats are very incompatible with the LSST-style refcats we've been assuming.

OK, that makes sense. I am unhappy with the answer being "run ap_pipe on lsst-dev" but you're correct that the processCcd configs cover my concern on lsst-dev. I suppose it's impossible to know a priori what refcats are available in other places so I don't have a better suggestion for now.

I'd like to see the commands you used to run ap_pipe here documented in some way, so that when people ask "how can I run ap_pipe on my favorite HSC data?" they don't have to wade over to this ticket and/or your lsst-dev scratch directory. Now that we support both DECam and HSC, I think it would be helpful (e.g., to folks who might want to use ap_pipe with yet another camera) to have examples of each in the docs.

It's fine to close and merge this ticket, and please create a new one to add more examples to the docs.

I think it's less "you must run on lsst-dev" and more "if you use your own refcats, you are responsible for supporting them". Hopefully not a big issue in practice, since anybody who's run ProcessCcdTask has had to have solved the problem already.

I've created DM-15351 to deal with the examples question.

Thanks for the review!