[DM-40113] Update PS1 catalogs on tiger from v0 to v1 - Jira

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: None
Labels:
None

Story Points:
4
Epic Link:
f23a-a&p-preops
Team:
Data Release Production
Urgent?:
No

Description

Original title: Investigate why calibrate task cannot be run on tiger using w_2023_28

When trying to run calibrate on the Princeton tiger cluster using lsst_distrib on w_2023_28, e.g.:

DATAQUERY="instrument='HSC' AND visit=1228 AND detector=42"; \

$(which time) -f "Total runtime: %E" \

pipetask --long-log --log-file $LOGFILE \

run --register-dataset-types \

--instrument lsst.obs.subaru.HyperSuprimeCam \

-b $REPO \

-i HSC/runs/RC2/w_2023_25/DM-39770 \

-o u/lskelvin/scratch/test_calibrate \

-p $DRP_PIPE_DIR/pipelines/HSC/DRP-RC2.yaml#calibrate \

-d "$DATAQUERY"

the following error is returned:

ValueError: No version number found in refcat header metadata. Version 0 refcats are no longer supported: refcat fluxes must have nJy units.

It looks like this has been broken since w_2023_27, i.e., the above works on w_2023_26. It is worth noting however that the w_2023_26 test run does report:

...

INFO 2023-07-19T13:54:16.477-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 42, visit: 1228, ...})(loadReferenceObjects.py:753) - Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [149.88838285, 150.10916640], [1.99195064, 2.21258387] RA Dec

INFO 2023-07-19T13:54:16.772-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 42, visit: 1228, ...})(loadReferenceObjects.py:793) - Loaded 2546 reference objects

WARNING 2023-07-19T13:54:16.778-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 42, visit: 1228, ...})(loadReferenceObjects.py:805) - Found version 0 reference catalog with old style units in schema.

WARNING 2023-07-19T13:54:16.780-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 42, visit: 1228, ...})(loadReferenceObjects.py:806) - run `meas_algorithms/bin/convert_refcat_to_nJy.py` to convert fluxes to nJy.

WARNING 2023-07-19T13:54:16.780-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 42, visit: 1228, ...})(loadReferenceObjects.py:807) - See RFC-575 for more details.

INFO 2023-07-19T13:54:16.783-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 42, visit: 1228, ...})(loadReferenceObjects.py:153) - Converted refcat flux fields to nJy (name, units): (g_flux, ''); (r_flux, ''); (i_flux, ''); (z_flux, ''); (y_flux, ''); (i_fluxSigma, ''); (y_fluxSigma, ''); (r_fluxSigma, ''); (z_fluxSigma, ''); (g_fluxSigma, '')

INFO 2023-07-19T13:54:16.792-04:00 lsst.calibrate.photoCal.match.referenceSelection (calibrate:{instrument: 'HSC', detector: 42, visit: 1228, ...})(sourceSelector.py:705) - Selected 333/2546 references

INFO 2023-07-19T13:54:16.793-04:00 lsst.calibrate.photoCal.match (calibrate:{instrument: 'HSC', detector: 42, visit: 1228, ...})(directMatch.py:111) - Matched 39 from 388/3094 input and 333/2546 reference sources

...

This ticket captures effort to resolve this issue.

See also this Community post, ~~RFC-851~~ and ~~DM-34793~~ for further details.

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

239043_v0.fits
2.56 MB
20/Jul/23 4:52 PM
239043_v1.fits
2.49 MB
20/Jul/23 4:52 PM

Issue Links

relates to

RFC-851 Remove support for version 0 reference catalogs

Implemented

DM-34793 Remove version 0 refcat code

Done

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Lee Kelvin added a comment - 20/Jul/23 4:43 PM

On chatting with John Parejko over on Slack, our GAIA refcats are fine (they were never v0 catalogs), but our PS1 catalogs needed a version upgrade.

The convert_refcat_to_nJy.py script was removed on ~~DM-34793~~, so I've had to use an older Rubin Env (4.0.1) and weekly (w_2022_29) to run this conversion script.

At the time of writing, the script is running seemingly well, and so far ~3000 FITS files have been converted (~2%) in ~10 minutes. I therefore expect this process to take ~7 hours.

Show

Lee Kelvin added a comment - 20/Jul/23 4:43 PM On chatting with John Parejko over on Slack , our GAIA refcats are fine (they were never v0 catalogs), but our PS1 catalogs needed a version upgrade. The convert_refcat_to_nJy.py script was removed on DM-34793 , so I've had to use an older Rubin Env (4.0.1) and weekly (w_2022_29) to run this conversion script. At the time of writing, the script is running seemingly well, and so far ~3000 FITS files have been converted (~2%) in ~10 minutes. I therefore expect this process to take ~7 hours.

Hide

Permalink

Lee Kelvin added a comment - 20/Jul/23 4:53 PM

Updates do seem to be being applied correctly, e.g.:

239043_v0.fits 239043_v1.fits

The footprint column has been removed from the v1 catalogs - I assume that is to be expected?

Show

Lee Kelvin added a comment - 20/Jul/23 4:53 PM Updates do seem to be being applied correctly, e.g.: 239043_v0.fits 239043_v1.fits The footprint column has been removed from the v1 catalogs - I assume that is to be expected?

Hide

Permalink

Lee Kelvin added a comment - 21/Jul/23 4:47 PM - edited

The update script completed successfully, and all FITS files should now be in v1 nJy format.

I attempted a further test run using the command above, however, it now returns this error:

Exception RuntimeError: Integrity failure in Datastore.

Size of file file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/231858.fits (2799360)

does not match size recorded in registry of 2882880

A thread on Slack discussed the possible options, which seem to be A) remove the collection and re-ingest, or B) set the file_size registry record for these files to -1, which will disable file size checks. On balance, it was felt that option B) was the best course of action here.

On tiger, these commands were run in the PostgreSQL interactive terminal (psql -h localhost -U lsst lsst):

lsst=> SELECT COUNT(*) FROM file_datastore_records WHERE path ~ '^file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/.*fits$';

 count

--------

(1 row)

This number matches the number of FITS files on-disk bar 1 (the master_schema.fits file):

ps1_pv3_3pi_20170110$ ls -1 *.fits | wc -l

As an example, these files look like this in the database:

lsst=> SELECT * FROM file_datastore_records WHERE path ~ '^file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/.*fits$' LIMIT 3;

              dataset_id              |                               path                                |                         formatter                         | storage_class |    component    | checksum | file_size

--------------------------------------+-------------------------------------------------------------------+-----------------------------------------------------------+---------------+-----------------+----------+-----------

 4b65e9e3-d5cc-4bfd-87a4-b25742c98559 | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/133534.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |   1874880

 c581dc69-685c-42a5-a131-60767f42b23c | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/148960.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |   6672960

 581dcf18-28bf-45e0-8775-0d7cd8f45bc9 | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/163940.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |   1863360

(3 rows)

Finally, to roll out the modification of their file_size attributes, I ran this command:

BEGIN;

UPDATE file_datastore_records SET file_size=-1 WHERE path ~ '^file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/.*fits$';

# Once everything looks ok, then run:

COMMIT;

Once completed and committed, the example three records above now look like this:

lsst=> SELECT * FROM file_datastore_records WHERE path ~ '^file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/.*fits$' LIMIT 3;

              dataset_id              |                               path                                |                         formatter                         | storage_class |    component    | checksum | file_size

--------------------------------------+-------------------------------------------------------------------+-----------------------------------------------------------+---------------+-----------------+----------+-----------

 2acfd356-6c17-43f3-93da-ec5b30d01f3b | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/216189.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |        -1

 d4fe7ecc-c879-4e20-b57c-6c8ebb2f32d8 | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/190893.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |        -1

 2838c408-9995-495b-8798-a4a2affb49c7 | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/202970.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |        -1

(3 rows)

Show

Lee Kelvin added a comment - 21/Jul/23 4:47 PM - edited The update script completed successfully, and all FITS files should now be in v1 nJy format. I attempted a further test run using the command above, however, it now returns this error: Exception RuntimeError: Integrity failure in Datastore. Size of file file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/231858.fits (2799360) does not match size recorded in registry of 2882880 A thread on Slack discussed the possible options, which seem to be A) remove the collection and re-ingest, or B) set the file_size registry record for these files to -1 , which will disable file size checks. On balance, it was felt that option B) was the best course of action here. On tiger , these commands were run in the PostgreSQL interactive terminal ( psql -h localhost -U lsst lsst ): lsst=> SELECT COUNT(*) FROM file_datastore_records WHERE path ~ '^file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/.*fits$'; count -------- 130924 (1 row) This number matches the number of FITS files on-disk bar 1 (the master_schema.fits file): ps1_pv3_3pi_20170110$ ls -1 *.fits | wc -l 130925 As an example, these files look like this in the database: lsst=> SELECT * FROM file_datastore_records WHERE path ~ '^file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/.*fits$' LIMIT 3; dataset_id | path | formatter | storage_class | component | checksum | file_size --------------------------------------+-------------------------------------------------------------------+-----------------------------------------------------------+---------------+-----------------+----------+----------- 4b65e9e3-d5cc-4bfd-87a4-b25742c98559 | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/133534.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | 1874880 c581dc69-685c-42a5-a131-60767f42b23c | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/148960.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | 6672960 581dcf18-28bf-45e0-8775-0d7cd8f45bc9 | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/163940.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | 1863360 (3 rows) Finally, to roll out the modification of their file_size attributes, I ran this command: BEGIN; UPDATE file_datastore_records SET file_size=-1 WHERE path ~ '^file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/.*fits$'; # Once everything looks ok, then run: COMMIT; Once completed and committed, the example three records above now look like this: lsst=> SELECT * FROM file_datastore_records WHERE path ~ '^file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/.*fits$' LIMIT 3; dataset_id | path | formatter | storage_class | component | checksum | file_size --------------------------------------+-------------------------------------------------------------------+-----------------------------------------------------------+---------------+-----------------+----------+----------- 2acfd356-6c17-43f3-93da-ec5b30d01f3b | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/216189.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | -1 d4fe7ecc-c879-4e20-b57c-6c8ebb2f32d8 | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/190893.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | -1 2838c408-9995-495b-8798-a4a2affb49c7 | file:///projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/202970.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | -1 (3 rows)

Hide

Permalink

Lee Kelvin added a comment - 21/Jul/23 9:10 PM

A similar process for the Merian PostgreSQL database is also required. In summary:

merian=> SELECT COUNT(*) FROM file_datastore_records WHERE path ~ '^refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110.*fits$';

 count

--------

(1 row)

These currently look like this:

merian=> SELECT * FROM file_datastore_records WHERE path ~ '^refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110.*fits$' LIMIT 3;

              dataset_id              |                                      path                                       |                         formatter                         | storage_class |    component    | checksum | file_size

--------------------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------+---------------+-----------------+----------+-----------

 86773127-8409-49ba-8c36-942b5724e292 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_133534_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |   1874880

 7f7c35f7-08b0-4e9a-a703-41226ba5f8a1 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_148960_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |   6672960

 602fdd6a-7295-48e8-a4ff-612b262bae32 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_163940_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |   1863360

(3 rows)

Running the same script to resolve these new file_size values on-disk:

BEGIN;

UPDATE file_datastore_records SET file_size=-1 WHERE path ~ '^refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110.*fits$';

# Once everything looks ok, then run:

COMMIT;

And a final check to confirm the new values have been set:

merian=> SELECT * FROM file_datastore_records WHERE path ~ '^refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110.*fits$' LIMIT 3;

              dataset_id              |                                      path                                       |                         formatter                         | storage_class |    component    | checksum | file_size

--------------------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------+---------------+-----------------+----------+-----------

 f57a99d4-bf09-49bc-8e71-1b6c953da1f4 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_165522_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |        -1

 e6b87bd0-41a1-4037-9464-d034acd0f8f9 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_169370_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |        -1

 43eadbda-02fc-403a-aaa2-7a58a322f967 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_217109_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ |          |        -1

(3 rows)

Show

Lee Kelvin added a comment - 21/Jul/23 9:10 PM A similar process for the Merian PostgreSQL database is also required. In summary: merian=> SELECT COUNT(*) FROM file_datastore_records WHERE path ~ '^refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110.*fits$'; count -------- 130924 (1 row) These currently look like this: merian=> SELECT * FROM file_datastore_records WHERE path ~ '^refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110.*fits$' LIMIT 3; dataset_id | path | formatter | storage_class | component | checksum | file_size --------------------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------+---------------+-----------------+----------+----------- 86773127-8409-49ba-8c36-942b5724e292 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_133534_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | 1874880 7f7c35f7-08b0-4e9a-a703-41226ba5f8a1 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_148960_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | 6672960 602fdd6a-7295-48e8-a4ff-612b262bae32 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_163940_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | 1863360 (3 rows) Running the same script to resolve these new file_size values on-disk: BEGIN; UPDATE file_datastore_records SET file_size=-1 WHERE path ~ '^refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110.*fits$'; # Once everything looks ok, then run: COMMIT; And a final check to confirm the new values have been set: merian=> SELECT * FROM file_datastore_records WHERE path ~ '^refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110.*fits$' LIMIT 3; dataset_id | path | formatter | storage_class | component | checksum | file_size --------------------------------------+---------------------------------------------------------------------------------+-----------------------------------------------------------+---------------+-----------------+----------+----------- f57a99d4-bf09-49bc-8e71-1b6c953da1f4 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_165522_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | -1 e6b87bd0-41a1-4037-9464-d034acd0f8f9 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_169370_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | -1 43eadbda-02fc-403a-aaa2-7a58a322f967 | refcats/gen2/ps1_pv3_3pi_20170110/ps1_pv3_3pi_20170110_217109_refcats_gen2.fits | lsst.obs.base.formatters.fitsGeneric.FitsGenericFormatter | SimpleCatalog | __NULL_STRING__ | | -1 (3 rows)

Hide

Permalink

Erfan Nourbakhsh added a comment - 31/Jul/23 9:59 PM

Thank you Lee Kelvin for resolving this issue. This important refcat change on Tiger now allows us to continue running step 1 using the latest weekly version. I have tested it on my account and can confirm that everything functions as expected. See below:

Pipetask run

I ran HSC visit 1228, detector 24, through the calibrate task as instructed. Previously, we encountered issues when utilizing the older v0 PanSTARRS reference catalogs. However, thanks to your fix, the task is now functioning correctly and producing the desired results.

en7908@tiger2-sumire:~$ source /scratch/gpfs/HSC/LSST/stack/loadLSST.sh

en7908@tiger2-sumire:~$ setup lsst_distrib -t w_2023_30

en7908@tiger2-sumire:~$ eups list | grep lsst_distrib | grep setup

lsst_distrib          g4213664e8e+4c321e77d1    w_2023_30 current w_latest setup

en7908@tiger2-sumire:~$ pipetask --long-log \

> run \

> --instrument lsst.obs.subaru.HyperSuprimeCam \

> -b /projects/HSC/repo/main \

> -i HSC/runs/RC2/w_2023_25/DM-39770 \

> -o u/$USER/scratch/test_calibrate \

> -p $DRP_PIPE_DIR/pipelines/HSC/DRP-RC2.yaml#calibrate \

> -d "instrument='HSC' AND visit=1228 AND detector=24"

INFO 2023-07-31T15:28:28.616-04:00 lsst.ctrl.mpexec.cmdLineFwk ()(cmdLineFwk.py:646) - QuantumGraph contains 1 quanta for 1 tasks, graph ID: '1690831708.613665-177753'

Quanta   Tasks

------ ---------

     1 calibrate

INFO 2023-07-31T15:29:03.449-04:00 lsst.calibrate.detection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(detection.py:556) - Setting factor for negative detections equal to that for positive detections: 1.000000

INFO 2023-07-31T15:29:04.223-04:00 lsst.calibrate.detection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(detection.py:646) - Detected 2998 positive peaks in 2034 footprints and 17 negative peaks in 17 footprints to 5 +ve and 5 -ve sigma

INFO 2023-07-31T15:29:04.552-04:00 lsst.calibrate.detection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(detection.py:671) - Resubtracting the background after object detection

INFO 2023-07-31T15:29:05.423-04:00 lsst.calibrate.skySources (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(skyObjects.py:133) - Added 100 of 100 requested sky sources (100%)

INFO 2023-07-31T15:29:05.424-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:324) - Deblending 2134 sources

WARNING 2023-07-31T15:29:06.347-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523556676: skipping large footprint (area: 27710)

WARNING 2023-07-31T15:29:06.634-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523556885: skipping large footprint (area: 13705)

WARNING 2023-07-31T15:29:07.232-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523557232: skipping large footprint (area: 71357)

WARNING 2023-07-31T15:29:07.484-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523557428: skipping large footprint (area: 13659)

WARNING 2023-07-31T15:29:08.289-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523558260: skipping large footprint (area: 10225)

INFO 2023-07-31T15:29:08.383-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:493) - Deblended: of 2134 sources, 380 were deblended, creating 1018 children, total 3152 sources

INFO 2023-07-31T15:29:09.049-04:00 lsst.calibrate.measurement (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sfm.py:292) - Measuring 3152 sources (2134 parents, 1018 children)

INFO 2023-07-31T15:29:30.527-04:00 lsst.calibrate.applyApCorr (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(applyApCorr.py:238) - Applying aperture corrections to 3 instFlux fields

INFO 2023-07-31T15:29:30.791-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(calibrate.py:780) - Copying flags from icSourceCat to sourceCat for 295 sources

INFO 2023-07-31T15:29:30.803-04:00 lsst.calibrate.astrometry (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(astrometry.py:220) - Purged 2735 sources, leaving 417 good sources

INFO 2023-07-31T15:29:30.821-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:657) - Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [150.29392924, 150.51855942], [1.84332951, 1.97104299] RA Dec

INFO 2023-07-31T15:29:31.591-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:701) - Loaded 1766 reference objects

WARNING 2023-07-31T15:29:31.592-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:241) - Reference catalog pm_ra field is not an Angle; cannot apply proper motion.

INFO 2023-07-31T15:29:31.598-04:00 lsst.calibrate.astrometry.referenceSelector (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceSelector.py:703) - Selected 1766/1766 references

INFO 2023-07-31T15:29:32.230-04:00 lsst.calibrate.astrometry.matcher (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(matchPessimisticB.py:295) - Matched 272 sources

INFO 2023-07-31T15:29:32.404-04:00 lsst.calibrate.astrometry.matcher (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(matchPessimisticB.py:295) - Matched 271 sources

INFO 2023-07-31T15:29:32.570-04:00 lsst.calibrate.astrometry.matcher (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(matchPessimisticB.py:295) - Matched 271 sources

INFO 2023-07-31T15:29:32.709-04:00 lsst.calibrate.astrometry (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(astrometry.py:301) - Matched and fit WCS in 3 iterations; found 271 matches with mean and scatter = 0.063 +- 0.042 arcsec

INFO 2023-07-31T15:29:32.721-04:00 lsst.calibrate.photoCal.match.sourceSelection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceSelector.py:637) - Selected 369/3152 sources

INFO 2023-07-31T15:29:32.722-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:657) - Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [150.30616075, 150.51943701], [1.79922551, 2.01238208] RA Dec

INFO 2023-07-31T15:29:32.911-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:701) - Loaded 2563 reference objects

INFO 2023-07-31T15:29:32.919-04:00 lsst.calibrate.photoCal.match.referenceSelection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceSelector.py:703) - Selected 234/2563 references

INFO 2023-07-31T15:29:32.919-04:00 lsst.calibrate.photoCal.match (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(directMatch.py:112) - Matched 29 from 369/3152 input and 234/2563 reference sources

INFO 2023-07-31T15:29:32.920-04:00 lsst.calibrate.photoCal.reserve (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(reserveSourcesTask.py:110) - Reserved 0/29 sources

INFO 2023-07-31T15:29:32.921-04:00 lsst.calibrate.photoCal (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(photoCal.py:256) - Applying color terms for filter='HSC-I', config.photoCatName=ps1_pv3_3pi_20170110

INFO 2023-07-31T15:29:32.923-04:00 lsst.calibrate.photoCal (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(photoCal.py:414) - Magnitude zero point: 33.040059 +/- 0.000645 from 24 stars

INFO 2023-07-31T15:29:32.931-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(calibrate.py:640) - Photometric zero-point: 33.040059

INFO 2023-07-31T15:29:32.946-04:00 lsst.calibrate.postCalibrationMeasurement (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sfm.py:292) - Measuring 3152 sources (2134 parents, 1018 children)

INFO 2023-07-31T15:29:37.335-04:00 lsst.calibrate.computeSummaryStats (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(computeExposureSummaryStats.py:147) - Measuring exposure statistics

INFO 2023-07-31T15:29:45.739-04:00 lsst.ctrl.mpexec.singleQuantumExecutor (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(singleQuantumExecutor.py:265) - Execution of task 'calibrate' on quantum {instrument: 'HSC', detector: 24, visit: 1228, ...} took 51.710 seconds

INFO 2023-07-31T15:29:46.055-04:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:519) - Executed 1 quanta successfully, 0 failed and 0 remain out of total 1 quanta.

FITS headers

I examined one of the modified reference catalog FITS files and inspected its FITS header. As intended, the columns in this catalog are now expressed in units of nanojanskys, replacing the previous units of Janskys.

>>> import astropy.io.fits as fits

>>> hdulist = fits.open('/projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/231858.fits')

>>> hdu = hdulist[1]

>>> hdu.header

This gives:

XTENSION= 'BINTABLE'           / binary table extension

BITPIX  =                    8 / 8-bit bytes

NAXIS   =                    2 / 2-dimensional binary table

NAXIS1  =                  144 / width of table in bytes

NAXIS2  =                19346 / number of rows in table

PCOUNT  =                    0 / size of special data area

GCOUNT  =                    1 / one data group (required keyword)

TFIELDS =                   21 / number of fields in each row

TTYPE1  = 'id      '           / unique ID

TFORM1  = '1K      '           / format of field

TDOC1   = 'unique ID'

TCCLS1  = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE2  = 'coord_ra'           / position in ra/dec

TFORM2  = '1D      '           / format of field

TDOC2   = 'position in ra/dec'

TUNIT2  = 'rad     '

TCCLS2  = 'Angle   '           / Field template used by lsst.afw.table

TTYPE3  = 'coord_dec'          / position in ra/dec

TFORM3  = '1D      '           / format of field

TDOC3   = 'position in ra/dec'

TUNIT3  = 'rad     '

TCCLS3  = 'Angle   '           / Field template used by lsst.afw.table

TTYPE4  = 'parent  '           / unique ID of parent source

TFORM4  = '1K      '           / format of field

TDOC4   = 'unique ID of parent source'

TCCLS4  = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE5  = 'g_flux  '

TFORM5  = '1D      '           / format of field

TUNIT5  = 'nJy     '

TCCLS5  = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE6  = 'r_flux  '

TFORM6  = '1D      '           / format of field

TUNIT6  = 'nJy     '

TCCLS6  = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE7  = 'i_flux  '

TFORM7  = '1D      '           / format of field

TUNIT7  = 'nJy     '

TCCLS7  = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE8  = 'z_flux  '

TFORM8  = '1D      '           / format of field

TUNIT8  = 'nJy     '

TCCLS8  = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE9  = 'y_flux  '

TFORM9  = '1D      '           / format of field

TUNIT9  = 'nJy     '

TCCLS9  = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE10 = 'i_fluxErr'

TFORM10 = '1D      '           / format of field

TUNIT10 = 'nJy     '

TCCLS10 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE11 = 'y_fluxErr'

TFORM11 = '1D      '           / format of field

TUNIT11 = 'nJy     '

TCCLS11 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE12 = 'r_fluxErr'

TFORM12 = '1D      '           / format of field

TUNIT12 = 'nJy     '

TCCLS12 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE13 = 'z_fluxErr'

TFORM13 = '1D      '           / format of field

TUNIT13 = 'nJy     '

TCCLS13 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE14 = 'g_fluxErr'

TFORM14 = '1D      '           / format of field

TUNIT14 = 'nJy     '

TCCLS14 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE15 = 'coord_ra_err'

TFORM15 = '1E      '           / format of field

TCCLS15 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE16 = 'coord_dec_err'

TFORM16 = '1E      '           / format of field

TCCLS16 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE17 = 'epoch   '

TFORM17 = '1K      '           / format of field

TCCLS17 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE18 = 'pm_ra   '

TFORM18 = '1E      '           / format of field

TCCLS18 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE19 = 'pm_dec  '

TFORM19 = '1E      '           / format of field

TCCLS19 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE20 = 'pm_ra_err'

TFORM20 = '1E      '           / format of field

TCCLS20 = 'Scalar  '           / Field template used by lsst.afw.table

TTYPE21 = 'pm_dec_err'

TFORM21 = '1E      '           / format of field

TCCLS21 = 'Scalar  '           / Field template used by lsst.afw.table

HIERARCH REFCAT_FORMAT_VERSION = 1

HIERARCH AFW_TABLE_VERSION = 3

AFW_TYPE= 'SIMPLE  '           / Tells lsst::afw to load this as a Simple table.

Show

Erfan Nourbakhsh added a comment - 31/Jul/23 9:59 PM Thank you Lee Kelvin for resolving this issue. This important refcat change on Tiger now allows us to continue running step 1 using the latest weekly version. I have tested it on my account and can confirm that everything functions as expected. See below: Pipetask run I ran HSC visit 1228, detector 24, through the calibrate task as instructed. Previously, we encountered issues when utilizing the older v0 PanSTARRS reference catalogs. However, thanks to your fix, the task is now functioning correctly and producing the desired results. en7908@tiger2-sumire:~$ source /scratch/gpfs/HSC/LSST/stack/loadLSST.sh en7908@tiger2-sumire:~$ setup lsst_distrib -t w_2023_30 en7908@tiger2-sumire:~$ eups list | grep lsst_distrib | grep setup lsst_distrib g4213664e8e+4c321e77d1 w_2023_30 current w_latest setup en7908@tiger2-sumire:~$ pipetask --long-log \ > run \ > --instrument lsst.obs.subaru.HyperSuprimeCam \ > -b /projects/HSC/repo/main \ > -i HSC/runs/RC2/w_2023_25/DM-39770 \ > -o u/$USER/scratch/test_calibrate \ > -p $DRP_PIPE_DIR/pipelines/HSC/DRP-RC2.yaml#calibrate \ > -d "instrument='HSC' AND visit=1228 AND detector=24" INFO 2023-07-31T15:28:28.616-04:00 lsst.ctrl.mpexec.cmdLineFwk ()(cmdLineFwk.py:646) - QuantumGraph contains 1 quanta for 1 tasks, graph ID: '1690831708.613665-177753' Quanta Tasks ------ --------- 1 calibrate INFO 2023-07-31T15:29:03.449-04:00 lsst.calibrate.detection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(detection.py:556) - Setting factor for negative detections equal to that for positive detections: 1.000000 INFO 2023-07-31T15:29:04.223-04:00 lsst.calibrate.detection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(detection.py:646) - Detected 2998 positive peaks in 2034 footprints and 17 negative peaks in 17 footprints to 5 +ve and 5 -ve sigma INFO 2023-07-31T15:29:04.552-04:00 lsst.calibrate.detection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(detection.py:671) - Resubtracting the background after object detection INFO 2023-07-31T15:29:05.423-04:00 lsst.calibrate.skySources (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(skyObjects.py:133) - Added 100 of 100 requested sky sources (100%) INFO 2023-07-31T15:29:05.424-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:324) - Deblending 2134 sources WARNING 2023-07-31T15:29:06.347-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523556676: skipping large footprint (area: 27710) WARNING 2023-07-31T15:29:06.634-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523556885: skipping large footprint (area: 13705) WARNING 2023-07-31T15:29:07.232-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523557232: skipping large footprint (area: 71357) WARNING 2023-07-31T15:29:07.484-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523557428: skipping large footprint (area: 13659) WARNING 2023-07-31T15:29:08.289-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:354) - Parent 527473523558260: skipping large footprint (area: 10225) INFO 2023-07-31T15:29:08.383-04:00 lsst.calibrate.deblend (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceDeblendTask.py:493) - Deblended: of 2134 sources, 380 were deblended, creating 1018 children, total 3152 sources INFO 2023-07-31T15:29:09.049-04:00 lsst.calibrate.measurement (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sfm.py:292) - Measuring 3152 sources (2134 parents, 1018 children) INFO 2023-07-31T15:29:30.527-04:00 lsst.calibrate.applyApCorr (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(applyApCorr.py:238) - Applying aperture corrections to 3 instFlux fields INFO 2023-07-31T15:29:30.791-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(calibrate.py:780) - Copying flags from icSourceCat to sourceCat for 295 sources INFO 2023-07-31T15:29:30.803-04:00 lsst.calibrate.astrometry (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(astrometry.py:220) - Purged 2735 sources, leaving 417 good sources INFO 2023-07-31T15:29:30.821-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:657) - Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [150.29392924, 150.51855942], [1.84332951, 1.97104299] RA Dec INFO 2023-07-31T15:29:31.591-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:701) - Loaded 1766 reference objects WARNING 2023-07-31T15:29:31.592-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:241) - Reference catalog pm_ra field is not an Angle; cannot apply proper motion. INFO 2023-07-31T15:29:31.598-04:00 lsst.calibrate.astrometry.referenceSelector (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceSelector.py:703) - Selected 1766/1766 references INFO 2023-07-31T15:29:32.230-04:00 lsst.calibrate.astrometry.matcher (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(matchPessimisticB.py:295) - Matched 272 sources INFO 2023-07-31T15:29:32.404-04:00 lsst.calibrate.astrometry.matcher (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(matchPessimisticB.py:295) - Matched 271 sources INFO 2023-07-31T15:29:32.570-04:00 lsst.calibrate.astrometry.matcher (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(matchPessimisticB.py:295) - Matched 271 sources INFO 2023-07-31T15:29:32.709-04:00 lsst.calibrate.astrometry (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(astrometry.py:301) - Matched and fit WCS in 3 iterations; found 271 matches with mean and scatter = 0.063 +- 0.042 arcsec INFO 2023-07-31T15:29:32.721-04:00 lsst.calibrate.photoCal.match.sourceSelection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceSelector.py:637) - Selected 369/3152 sources INFO 2023-07-31T15:29:32.722-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:657) - Loading reference objects from ps1_pv3_3pi_20170110 in region bounded by [150.30616075, 150.51943701], [1.79922551, 2.01238208] RA Dec INFO 2023-07-31T15:29:32.911-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(loadReferenceObjects.py:701) - Loaded 2563 reference objects INFO 2023-07-31T15:29:32.919-04:00 lsst.calibrate.photoCal.match.referenceSelection (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sourceSelector.py:703) - Selected 234/2563 references INFO 2023-07-31T15:29:32.919-04:00 lsst.calibrate.photoCal.match (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(directMatch.py:112) - Matched 29 from 369/3152 input and 234/2563 reference sources INFO 2023-07-31T15:29:32.920-04:00 lsst.calibrate.photoCal.reserve (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(reserveSourcesTask.py:110) - Reserved 0/29 sources INFO 2023-07-31T15:29:32.921-04:00 lsst.calibrate.photoCal (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(photoCal.py:256) - Applying color terms for filter='HSC-I', config.photoCatName=ps1_pv3_3pi_20170110 INFO 2023-07-31T15:29:32.923-04:00 lsst.calibrate.photoCal (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(photoCal.py:414) - Magnitude zero point: 33.040059 +/- 0.000645 from 24 stars INFO 2023-07-31T15:29:32.931-04:00 lsst.calibrate (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(calibrate.py:640) - Photometric zero-point: 33.040059 INFO 2023-07-31T15:29:32.946-04:00 lsst.calibrate.postCalibrationMeasurement (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(sfm.py:292) - Measuring 3152 sources (2134 parents, 1018 children) INFO 2023-07-31T15:29:37.335-04:00 lsst.calibrate.computeSummaryStats (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(computeExposureSummaryStats.py:147) - Measuring exposure statistics INFO 2023-07-31T15:29:45.739-04:00 lsst.ctrl.mpexec.singleQuantumExecutor (calibrate:{instrument: 'HSC', detector: 24, visit: 1228, ...})(singleQuantumExecutor.py:265) - Execution of task 'calibrate' on quantum {instrument: 'HSC', detector: 24, visit: 1228, ...} took 51.710 seconds INFO 2023-07-31T15:29:46.055-04:00 lsst.ctrl.mpexec.mpGraphExecutor ()(mpGraphExecutor.py:519) - Executed 1 quanta successfully, 0 failed and 0 remain out of total 1 quanta. FITS headers I examined one of the modified reference catalog FITS files and inspected its FITS header. As intended, the columns in this catalog are now expressed in units of nanojanskys, replacing the previous units of Janskys. >>> import astropy.io.fits as fits >>> hdulist = fits. open ( '/projects/HSC/refcats/htm/ps1_pv3_3pi_20170110/231858.fits' ) >>> hdu = hdulist[ 1 ] >>> hdu.header This gives: XTENSION = 'BINTABLE' / binary table extension BITPIX = 8 / 8 - bit bytes NAXIS = 2 / 2 - dimensional binary table NAXIS1 = 144 / width of table in bytes NAXIS2 = 19346 / number of rows in table PCOUNT = 0 / size of special data area GCOUNT = 1 / one data group (required keyword) TFIELDS = 21 / number of fields in each row TTYPE1 = 'id ' / unique ID TFORM1 = '1K ' / format of field TDOC1 = 'unique ID' TCCLS1 = 'Scalar ' / Field template used by lsst.afw.table TTYPE2 = 'coord_ra' / position in ra / dec TFORM2 = '1D ' / format of field TDOC2 = 'position in ra/dec' TUNIT2 = 'rad ' TCCLS2 = 'Angle ' / Field template used by lsst.afw.table TTYPE3 = 'coord_dec' / position in ra / dec TFORM3 = '1D ' / format of field TDOC3 = 'position in ra/dec' TUNIT3 = 'rad ' TCCLS3 = 'Angle ' / Field template used by lsst.afw.table TTYPE4 = 'parent ' / unique ID of parent source TFORM4 = '1K ' / format of field TDOC4 = 'unique ID of parent source' TCCLS4 = 'Scalar ' / Field template used by lsst.afw.table TTYPE5 = 'g_flux ' TFORM5 = '1D ' / format of field TUNIT5 = 'nJy ' TCCLS5 = 'Scalar ' / Field template used by lsst.afw.table TTYPE6 = 'r_flux ' TFORM6 = '1D ' / format of field TUNIT6 = 'nJy ' TCCLS6 = 'Scalar ' / Field template used by lsst.afw.table TTYPE7 = 'i_flux ' TFORM7 = '1D ' / format of field TUNIT7 = 'nJy ' TCCLS7 = 'Scalar ' / Field template used by lsst.afw.table TTYPE8 = 'z_flux ' TFORM8 = '1D ' / format of field TUNIT8 = 'nJy ' TCCLS8 = 'Scalar ' / Field template used by lsst.afw.table TTYPE9 = 'y_flux ' TFORM9 = '1D ' / format of field TUNIT9 = 'nJy ' TCCLS9 = 'Scalar ' / Field template used by lsst.afw.table TTYPE10 = 'i_fluxErr' TFORM10 = '1D ' / format of field TUNIT10 = 'nJy ' TCCLS10 = 'Scalar ' / Field template used by lsst.afw.table TTYPE11 = 'y_fluxErr' TFORM11 = '1D ' / format of field TUNIT11 = 'nJy ' TCCLS11 = 'Scalar ' / Field template used by lsst.afw.table TTYPE12 = 'r_fluxErr' TFORM12 = '1D ' / format of field TUNIT12 = 'nJy ' TCCLS12 = 'Scalar ' / Field template used by lsst.afw.table TTYPE13 = 'z_fluxErr' TFORM13 = '1D ' / format of field TUNIT13 = 'nJy ' TCCLS13 = 'Scalar ' / Field template used by lsst.afw.table TTYPE14 = 'g_fluxErr' TFORM14 = '1D ' / format of field TUNIT14 = 'nJy ' TCCLS14 = 'Scalar ' / Field template used by lsst.afw.table TTYPE15 = 'coord_ra_err' TFORM15 = '1E ' / format of field TCCLS15 = 'Scalar ' / Field template used by lsst.afw.table TTYPE16 = 'coord_dec_err' TFORM16 = '1E ' / format of field TCCLS16 = 'Scalar ' / Field template used by lsst.afw.table TTYPE17 = 'epoch ' TFORM17 = '1K ' / format of field TCCLS17 = 'Scalar ' / Field template used by lsst.afw.table TTYPE18 = 'pm_ra ' TFORM18 = '1E ' / format of field TCCLS18 = 'Scalar ' / Field template used by lsst.afw.table TTYPE19 = 'pm_dec ' TFORM19 = '1E ' / format of field TCCLS19 = 'Scalar ' / Field template used by lsst.afw.table TTYPE20 = 'pm_ra_err' TFORM20 = '1E ' / format of field TCCLS20 = 'Scalar ' / Field template used by lsst.afw.table TTYPE21 = 'pm_dec_err' TFORM21 = '1E ' / format of field TCCLS21 = 'Scalar ' / Field template used by lsst.afw.table HIERARCH REFCAT_FORMAT_VERSION = 1 HIERARCH AFW_TABLE_VERSION = 3 AFW_TYPE = 'SIMPLE ' / Tells lsst::afw to load this as a Simple table.

Hide

Permalink

Lee Kelvin added a comment - 31/Jul/23 10:04 PM

Wonderful, thanks for such a detailed check Erfan Nourbakhsh, much appreciated! Marking this ticket as done!

Show

Lee Kelvin added a comment - 31/Jul/23 10:04 PM Wonderful, thanks for such a detailed check Erfan Nourbakhsh , much appreciated! Marking this ticket as done!

Data Management

Update PS1 catalogs on tiger from v0 to v1

Details

Description

Attachments

Attachments

Issue Links

Activity

Pipetask run

FITS headers

People

Dates

Jenkins