[DM-3141] Reduce verbosity of astrometry - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: meas_astrom
Labels:
None

Story Points:
1
Epic Link:
DRP DM-S15-4,5,6 Bugs and Papercuts
Sprint:
Science Pipelines DM-S15-5
Team:
Data Release Production

Description

The astrometry.net solver that runs by default in meas_astrom 10.1 is very verbose. Here's an example running HSC data with an SDSS reference catalog:

$ processCcd.py /tigress/HSC/HSC --output /tigress/pprice/lsst --id visit=904020 ccd=49 --clobber-config

: Loading config overrride file '/home/pprice/LSST/obs/subaru/config/processCcd.py'

WARNING: Unable to use psfex: No module named extensions.psfex.psfexPsfDeterminer

hscAstrom is not setup; using LSST's meas_astrom instead

Cannot import lsst.meas.multifit: disabling CModel measurements

Cannot import lsst.meas.extensions.photometryKron: disabling Kron measurements

Cannot enable shapeHSM ('MEAS_EXTENSIONS_SHAPEHSM_DIR'): disabling HSM shape measurements

Cannot import lsst.meas.extensions.photometryKron: disabling Kron measurements

Cannot enable shapeHSM ('MEAS_EXTENSIONS_SHAPEHSM_DIR'): disabling HSM shape measurements

: Loading config overrride file '/home/pprice/LSST/obs/subaru/config/hsc/processCcd.py'

: input=/tigress/HSC/HSC

: calib=None

: output=/tigress/pprice/lsst

CameraMapper: Loading registry registry from /tigress/pprice/lsst/_parent/registry.sqlite3

CameraMapper: Loading calibRegistry registry from /tigress/HSC/HSC/CALIB/calibRegistry.sqlite3

processCcd: Processing {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}

processCcd.isr: Performing ISR on sensor {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}

processCcd.isr WARNING: Cannot write thumbnail image; hsc.fitsthumb could not be imported.

afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE

processCcd.isr: Applying linearity corrections to Ccd 49

processCcd.isr.crosstalk: Applying crosstalk correction

afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE

: Empty WCS extension, using FITS header

processCcd.isr: Set 0 BAD pixels to 647.04

processCcd.isr WARNING: There were 6192 unmasked NaNs

processCcd.isr WARNING: Cannot write thumbnail image; hsc.fitsthumb could not be imported.

processCcd.isr: Flattened sky level: 647.130493 +/- 12.733898

processCcd.isr: Measuring sky levels in 8x16 grids: 648.106765

processCcd.isr: Sky flatness in 8x16 grids - pp: 0.024087 rms: 0.006057

processCcd.calibrate: installInitialPsf fwhm=5.88235294312 pixels; size=15 pixels

processCcd.calibrate.repair: Identified 80 cosmic rays.

processCcd.calibrate.detection: Detected 303 positive sources to 5 sigma.

processCcd.calibrate.detection: Resubtracting the background after object detection

processCcd.calibrate.initialMeasurement: Measuring 303 sources (303 parents, 0 children)

processCcd.calibrate.astrometry: Applying distortion correction

processCcd.calibrate.astrometry: Solving astrometry

LoadReferenceObjects: read index files

processCcd.calibrate.astrometry.solver: Number of selected sources for astrometry : 258

processCcd.calibrate.astrometry.solver: Got astrometric solution from Astrometry.net

LoadReferenceObjects: getting reference objects using center (1023.5, 2084.5) pix = Fk5Coord(320.3431396, 0.5002365, 2000.00) sky and radius 0.00194896 rad

LoadReferenceObjects: search for objects at Fk5Coord(320.3431396, 0.5002365, 2000.00) with radius 0.111667372351 deg

LoadReferenceObjects: found 495 objects

LoadReferenceObjects: trimmed 257 out-of-bbox objects, leaving 238

processCcd.calibrate.astrometry.solver: Fit WCS: use iter 2 because it had less linear scatter than the next iter: 0.307471 vs. 0.320229 pixels

processCcd.calibrate.astrometry: 186 astrometric matches

processCcd.calibrate.astrometry: Refitting WCS

processCcd.calibrate.astrometry: Astrometric scatter: 0.047945 arcsec (with non-linear terms, 174 matches, 12 rejected)

processCcd.calibrate.measurePsf: Measuring PSF

/tigress/HSC/LSST/stack10_1/Linux64/anaconda/2.1.0-4-g35ca374/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.

  warnings.warn("Mean of empty slice.", RuntimeWarning)

/tigress/HSC/LSST/stack10_1/Linux64/anaconda/2.1.0-4-g35ca374/lib/python2.7/site-packages/numpy/core/_methods.py:71: RuntimeWarning: invalid value encountered in double_scalars

  ret = ret.dtype.type(ret / rcount)

/home/pprice/LSST/meas/algorithms/python/lsst/meas/algorithms/objectSizeStarSelector.py:143: RuntimeWarning: invalid value encountered in less

  update = dist < minDist

processCcd.calibrate.measurePsf: PSF star selector found 163 candidates

processCcd.calibrate.measurePsf: PSF determination using 114/163 stars.

processCcd.calibrate.repair: Identified 92 cosmic rays.

processCcd.calibrate: Fit and subtracted background

processCcd.calibrate.measurement: Measuring 303 sources (303 parents, 0 children)

processCcd.calibrate.astrometry: Applying distortion correction

processCcd.calibrate.astrometry: Solving astrometry

processCcd.calibrate.astrometry.solver: Number of selected sources for astrometry : 258

Solver:

  Arcsec per pix range: 0.153025, 0.18516

  Image size: 2054 x 4186

  Quad size range: 205.4, 4662.78

  Objs: 0, 50

  Parity: 0, normal

  Use_radec? yes, (320.343, 0.500178), radius 1 deg

  Verify_pix: 1

  Code tol: 0.01

  Dist from quad bonus: yes

  Distractor ratio: 0.25

  Log tune-up threshold: inf

  Log bail threshold: -230.259

  Log stoplooking threshold: inf

  Maxquads 0

  Maxmatches 0

  Set CRPIX? no

  Tweak? no

  Indexes: 3

    /tigress/HSC/astrometry_net_data/sdss-dr9-fink-v5b/sdss-dr9-fink-v5b_and_263_0.fits

    /tigress/HSC/astrometry_net_data/sdss-dr9-fink-v5b/sdss-dr9-fink-v5b_and_263_1.fits

    /tigress/HSC/astrometry_net_data/sdss-dr9-fink-v5b/sdss-dr9-fink-v5b_and_263_2.fits

  Field: 258 stars

Quad scale range: [641.674, 2208.56] pixels

object 1 of 50: 0 quads tried, 0 matched.

object 2 of 50: 0 quads tried, 0 matched.

object 3 of 50: 0 quads tried, 0 matched.

object 4 of 50: 0 quads tried, 0 matched.

object 5 of 50: 0 quads tried, 0 matched.

object 6 of 50: 0 quads tried, 0 matched.

Got a new best match: logodds 787.099.

  log-odds ratio 787.099 (inf), 178 match, 1 conflict, 75 distractors, 220 index.

  RA,Dec = (320.343,0.500213), pixel scale 0.167612 arcsec/pix.

  Hit/miss:   Hit/miss: ++-+++++-++++++++++++--++-+--+++++-+-+++++++++-+++++++-+++++-+++++++++++++-++++++-++++++-+++++++-+++

Pixel scale: 0.167612 arcsec/pix.

Parity: pos.

processCcd.calibrate.astrometry.solver: Got astrometric solution from Astrometry.net

LoadReferenceObjects: getting reference objects using center (1023.5, 2084.5) pix = Fk5Coord(320.3431396, 0.5002365, 2000.00) sky and radius 0.00194896 rad

LoadReferenceObjects: search for objects at Fk5Coord(320.3431396, 0.5002365, 2000.00) with radius 0.111667328272 deg

LoadReferenceObjects: found 495 objects

LoadReferenceObjects: trimmed 257 out-of-bbox objects, leaving 238

processCcd.calibrate.astrometry.solver: Fit WCS: use iter 2 because it had less linear scatter than the next iter: 0.306732 vs. 0.320115 pixels

processCcd.calibrate.astrometry: 186 astrometric matches

processCcd.calibrate.astrometry: Refitting WCS

processCcd.calibrate.astrometry: Astrometric scatter: 0.048271 arcsec (with non-linear terms, 174 matches, 12 rejected)

processCcd.calibrate.photocal: Not applying color terms because config.applyColorTerms is False

processCcd.calibrate.photocal: Magnitude zero point: 30.685281 +/- 0.058711 from 173 stars

processCcd.calibrate: Photometric zero-point: 30.685281

processCcd.detection: Detected 1194 positive sources to 5 sigma.

processCcd.detection: Resubtracting the background after object detection

processCcd.deblend: Deblending 1194 sources

processCcd.deblend: Deblended: of 1194 sources, 143 were deblended, creating 358 children, total 1552 sources

processCcd.measurement: Measuring 1552 sources (1194 parents, 358 children)

processCcd WARNING: Persisting background models

processCcd: Matching icSource and Source catalogs to propagate flags.

processCcd: Matching src to reference catalogue

LoadReferenceObjects: getting reference objects using center (1023.5, 2087.5) pix = Fk5Coord(320.3429016, 0.5001781, 2000.00) sky and radius 0.00195667 rad

LoadReferenceObjects: search for objects at Fk5Coord(320.3429016, 0.5001781, 2000.00) with radius 0.112109149864 deg

LoadReferenceObjects: found 499 objects

LoadReferenceObjects: trimmed 261 out-of-bbox objects, leaving 238

processCcd.calibrate.astrometry.solver: Fit WCS: use iter 1 because it had less linear scatter than the next iter: 0.300624 vs. 0.300652 pixels

The verbosity of the astrometry module is out of proportion with the rest of the modules, which makes it difficult to follow the processing.

This is a pull request for fixes I have made.

Attachments

Activity

Ascending order - Click to sort in descending order

5 older comments

Hide

Permalink

Russell Owen added a comment - 14/Jul/15 4:53 PM

Thank you for the clarification of finalize(). The LSST code I started from called finalize() all over the place (e.g. in unit tests and such), so I thought it was necessary and felt it should be called automatically if possible. Adding finalize() to the context manager seemed like the obvious solution, but not having to call it at all is even better.

I would prefer to retain the other messages, but am willing to be overruled.

Show

Russell Owen added a comment - 14/Jul/15 4:53 PM Thank you for the clarification of finalize(). The LSST code I started from called finalize() all over the place (e.g. in unit tests and such), so I thought it was necessary and felt it should be called automatically if possible. Adding finalize() to the context manager seemed like the obvious solution, but not having to call it at all is even better. I would prefer to retain the other messages, but am willing to be overruled.

Hide

Permalink

Paul Price added a comment - 14/Jul/15 6:19 PM

One more commit on meas_astrom to review, please:

pprice@tiger-sumire:~/LSST/meas/astrom (tickets/DM-3141=) $ git --no-pager log --patch -1

commit 356dcf06308add871058e8bad51cf9df412eac06

Author: Paul Price <price@astro.princeton.edu>

Date:   Tue Jul 14 21:13:42 2015 -0400

    ANetBasicAstrometryTask: set name and log for ref loader

    The LoadAstrometryNetObjectsTask is not a subtask, so wasn't a

    proper part of the log hierarchy --- log messages were being

    printed with only the Task's _DefaultName.  Adding the log and

    a name to the constructor puts it in the usual log hierarchy.

diff --git a/python/lsst/meas/astrom/anetBasicAstrometry.py b/python/lsst/meas/astrom/anetBasicAstrometry.py

index c4ff217..95df12c 100644

--- a/python/lsst/meas/astrom/anetBasicAstrometry.py

+++ b/python/lsst/meas/astrom/anetBasicAstrometry.py

@@ -217,7 +217,8 @@ class ANetBasicAstrometryTask(pipeBase.Task):

         pipeBase.Task.__init__(self, config=config, **kwargs)

         self.config = config

         # this is not a subtask because it cannot safely be retargeted

-        self.refObjLoader = LoadAstrometryNetObjectsTask(config=self.config, andConfig=andConfig)

+        self.refObjLoader = LoadAstrometryNetObjectsTask(config=self.config, andConfig=andConfig, log=self.log,

+                                                         name="loadAN")

         self.refObjLoader._readIndexFiles()

     def memusage(self, prefix=''):

Show

Paul Price added a comment - 14/Jul/15 6:19 PM One more commit on meas_astrom to review, please: pprice@tiger-sumire:~/LSST/meas/astrom (tickets/DM-3141=) $ git --no-pager log --patch -1 commit 356dcf06308add871058e8bad51cf9df412eac06 Author: Paul Price <price@astro.princeton.edu> Date: Tue Jul 14 21:13:42 2015 -0400 ANetBasicAstrometryTask: set name and log for ref loader The LoadAstrometryNetObjectsTask is not a subtask, so wasn't a proper part of the log hierarchy --- log messages were being printed with only the Task's _DefaultName. Adding the log and a name to the constructor puts it in the usual log hierarchy. diff --git a/python/lsst/meas/astrom/anetBasicAstrometry.py b/python/lsst/meas/astrom/anetBasicAstrometry.py index c4ff217..95df12c 100644 --- a/python/lsst/meas/astrom/anetBasicAstrometry.py +++ b/python/lsst/meas/astrom/anetBasicAstrometry.py @@ -217,7 +217,8 @@ class ANetBasicAstrometryTask(pipeBase.Task): pipeBase.Task.__init__(self, config=config, **kwargs) self.config = config # this is not a subtask because it cannot safely be retargeted - self.refObjLoader = LoadAstrometryNetObjectsTask(config=self.config, andConfig=andConfig) + self.refObjLoader = LoadAstrometryNetObjectsTask(config=self.config, andConfig=andConfig, log=self.log, + name="loadAN") self.refObjLoader._readIndexFiles() def memusage(self, prefix=''):

Hide

Permalink

Russell Owen added a comment - 15/Jul/15 10:05 AM

Your change to how you call the LoadAstrometryNetObjectsTask looks like a good improvement.

This is presumably outside the scope of this ticket, but is finalize() is even a useful method? The name is deceptive and it just calls stop_an_logging, which has a very clear name, so I wonder if it should be ditched.

Show

Russell Owen added a comment - 15/Jul/15 10:05 AM Your change to how you call the LoadAstrometryNetObjectsTask looks like a good improvement. This is presumably outside the scope of this ticket, but is finalize() is even a useful method? The name is deceptive and it just calls stop_an_logging, which has a very clear name, so I wonder if it should be ditched.

Hide

Permalink

Paul Price added a comment - 15/Jul/15 10:52 AM

I agree about finalize(), but there's an awful lot of work that needs to be done in cleaning up the astrometry.net stuff, so I would just group it in with that. I don't know what the plans are for improving that code, but given we'll be moving to a different matcher and reference catalog, it's probably not a high priority.

Going to merge soon.

Show

Paul Price added a comment - 15/Jul/15 10:52 AM I agree about finalize() , but there's an awful lot of work that needs to be done in cleaning up the astrometry.net stuff, so I would just group it in with that. I don't know what the plans are for improving that code, but given we'll be moving to a different matcher and reference catalog, it's probably not a high priority. Going to merge soon.

Hide

Permalink

Paul Price added a comment - 15/Jul/15 12:23 PM

Thanks!

Merged to master.

Show

Paul Price added a comment - 15/Jul/15 12:23 PM Thanks! Merged to master.

People

Assignee:: Paul Price

Reporter:: Paul Price

Reviewers:: Russell Owen

Watchers:: Dominique Boutigny, Lauren MacArthur, Paul Price, Russell Owen, Simon Krughoff

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 14/Jul/15 2:39 PM

Updated:: 15/Jul/15 12:23 PM

Resolved:: 15/Jul/15 12:23 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Activity

People

Dates

Jenkins