Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12420

Write up a short how to for using synpipe

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      DRP F17-6, DRP S18-1, DRP S18-2, DRP S18-3, DRP S18-4, DRP S18-5
    • Team:
      Data Release Production

      Description

      Create a short document or community post which outlines the steps necessary for using synpipe to add fake sources to to real data.

        Attachments

          Issue Links

            Activity

            Hide
            nlust Nate Lust added a comment -

            Here is a first pass at the documentation. I can put it in whatever format is best to get the information out once you have looked it over. Let me know if you have any problems.

            Show
            nlust Nate Lust added a comment - Here is a first pass at the documentation. I can put it in whatever format is best to get the information out once you have looked it over. Let me know if you have any problems.
            Hide
            sophiereed Sophie Reed added a comment -

            I've been running through the steps, the first problem I hit was that I needed sky maps but this is probably just my lack of familiarity with the process. Then when I finished the steps in the single frame processing section it says I can skip to the final step at the bottom which didn't immediately work. Again my confusion here is probably just caused by lack of familiarity with the processing in general but I thought it could be more clear that either additional steps need to be run or the command given needs to be changed for the outputs of singleFrameDriver.

            Show
            sophiereed Sophie Reed added a comment - I've been running through the steps, the first problem I hit was that I needed sky maps but this is probably just my lack of familiarity with the process. Then when I finished the steps in the single frame processing section it says I can skip to the final step at the bottom which didn't immediately work. Again my confusion here is probably just caused by lack of familiarity with the processing in general but I thought it could be more clear that either additional steps need to be run or the command given needs to be changed for the outputs of singleFrameDriver.
            Hide
            sophiereed Sophie Reed added a comment -

            I am getting an error running runMatchFakes on the output of singleFrameDriver:

            [sr525@lsst-dev01 ~]$ runMatchFakes.py /datasets/hsc/repo/rerun/private/sreed/synPipe/sfd/ 1166 -o /datasets/hsc/repo/rerun/private/sreed/synPipe/extractedCats -c /scratch/sr525/synPipe/src_0_radec_Z.fits --ccd 40
            CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            Doing ccd 40
            Traceback (most recent call last):
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+6/bin/runMatchFakes.py", line 69, in <module>
            raCol=args.raCol, decCol=args.decCol)
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+6/python/lsst/synpipe/matchFakes.py", line 586, in returnMatchTable
            raCol=raCol, decCol=decCol)
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+6/python/lsst/synpipe/matchFakes.py", line 484, in returnMatchSingle
            minRad=minRad, raCol=raCol, decCol=decCol)
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+6/python/lsst/synpipe/matchFakes.py", line 335, in getFakeSources
            doc=availExtras[extraName]['doc'])
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/14.0-25-g0e8fc950e+3/python/lsst/afw/table/schema/schemaContinued.py", line 154, in addField
            field = Field[type](field, doc=doc, units=units,
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/utils/14.0-6-g4f52afe+1/python/lsst/utils/wrappers.py", line 409, in _getitem_
            return self._registry[key]
            KeyError: <class 'int'>

            Show
            sophiereed Sophie Reed added a comment - I am getting an error running runMatchFakes on the output of singleFrameDriver: [sr525@lsst-dev01 ~] $ runMatchFakes.py /datasets/hsc/repo/rerun/private/sreed/synPipe/sfd/ 1166 -o /datasets/hsc/repo/rerun/private/sreed/synPipe/extractedCats -c /scratch/sr525/synPipe/src_0_radec_Z.fits --ccd 40 CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 Doing ccd 40 Traceback (most recent call last): File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+6/bin/runMatchFakes.py", line 69, in <module> raCol=args.raCol, decCol=args.decCol) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+6/python/lsst/synpipe/matchFakes.py", line 586, in returnMatchTable raCol=raCol, decCol=decCol) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+6/python/lsst/synpipe/matchFakes.py", line 484, in returnMatchSingle minRad=minRad, raCol=raCol, decCol=decCol) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+6/python/lsst/synpipe/matchFakes.py", line 335, in getFakeSources doc=availExtras [extraName] ['doc'] ) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/14.0-25-g0e8fc950e+3/python/lsst/afw/table/schema/schemaContinued.py", line 154, in addField field = Field [type] (field, doc=doc, units=units, File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/utils/14.0-6-g4f52afe+1/python/lsst/utils/wrappers.py", line 409, in _ getitem _ return self._registry [key] KeyError: <class 'int'>
            Hide
            sophiereed Sophie Reed added a comment -

            When running:

            runMatchFakes.py /datasets/hsc/repo/rerun/private/sreed/synPipeCoaddDetections/ 9813 -f HSC-Z -o /scratch/sr525/synPipe/matchedFakesAll -c /scratch/sr525/synPipe/src_0_radec_Z.fits

            on multiBandDriver outputs I get the error:

            Doing patch 10,10
            skipping {'tract': 9813, 'patch': '10,10', 'filter': 'HSC-Z'}
            No match returns!
            Traceback (most recent call last):
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/14.0-28-g7de3051c8/python/lsst/afw/table/base/baseContinued.py", line 339, in _getattr_
            return getattr(self.table, name)
            AttributeError: 'lsst.afw.table.source.source.SourceTable' object has no attribute 'get'

            During handling of the above exception, another exception occurred:

            Traceback (most recent call last):
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+16/python/lsst/synpipe/matchFakes.py", line 409, in getAstroTable
            data=src.get(nameKey)))
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/14.0-28-g7de3051c8/python/lsst/afw/table/base/baseContinued.py", line 341, in _getattr_
            return getattr(self.columns, name)
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/14.0-28-g7de3051c8/python/lsst/afw/table/base/baseContinued.py", line 111, in __getColumns
            self._columns = self._getColumnView()
            lsst.pex.exceptions.wrappers.RuntimeError:
            File "include/lsst/afw/table/BaseColumnView.h", line 200, in static lsst::afw::table::BaseColumnView lsst::afw::table::BaseColumnView::make(const std::shared_ptr<lsst::afw::table::BaseTable>&, InputIterator, InputIterator) [with InputIterator = lsst::afw::table::CatalogIterator<__gnu_cxx::__normal_iterator<const std::shared_ptr<lsst::afw::table::SourceRecord>*, std::vector<std::shared_ptr<lsst::afw::table::SourceRecord>, std::allocator<std::shared_ptr<lsst::afw::table::SourceRecord> > > > >]
            Record data is not contiguous in memory. {0}
            lsst::pex::exceptions::RuntimeError: 'Record data is not contiguous in memory.'

            During handling of the above exception, another exception occurred:

            Traceback (most recent call last):
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+16/bin/runMatchFakes.py", line 69, in <module>
            raCol=args.raCol, decCol=args.decCol)
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+16/python/lsst/synpipe/matchFakes.py", line 593, in returnMatchTable
            astroTable = getAstroTable(slist, mags=True)
            File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+16/python/lsst/synpipe/matchFakes.py", line 410, in getAstroTable
            except lsst.pex.exceptions.LsstException:
            AttributeError: module 'lsst.pex.exceptions' has no attribute 'LsstException'

            This does not occur if --ccd is specified.

            Show
            sophiereed Sophie Reed added a comment - When running: runMatchFakes.py /datasets/hsc/repo/rerun/private/sreed/synPipeCoaddDetections/ 9813 -f HSC-Z -o /scratch/sr525/synPipe/matchedFakesAll -c /scratch/sr525/synPipe/src_0_radec_Z.fits on multiBandDriver outputs I get the error: Doing patch 10,10 skipping {'tract': 9813, 'patch': '10,10', 'filter': 'HSC-Z'} No match returns! Traceback (most recent call last): File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/14.0-28-g7de3051c8/python/lsst/afw/table/base/baseContinued.py", line 339, in _ getattr _ return getattr(self.table, name) AttributeError: 'lsst.afw.table.source.source.SourceTable' object has no attribute 'get' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+16/python/lsst/synpipe/matchFakes.py", line 409, in getAstroTable data=src.get(nameKey))) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/14.0-28-g7de3051c8/python/lsst/afw/table/base/baseContinued.py", line 341, in _ getattr _ return getattr(self.columns, name) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/14.0-28-g7de3051c8/python/lsst/afw/table/base/baseContinued.py", line 111, in __getColumns self._columns = self._getColumnView() lsst.pex.exceptions.wrappers.RuntimeError: File "include/lsst/afw/table/BaseColumnView.h", line 200, in static lsst::afw::table::BaseColumnView lsst::afw::table::BaseColumnView::make(const std::shared_ptr<lsst::afw::table::BaseTable>&, InputIterator, InputIterator) [with InputIterator = lsst::afw::table::CatalogIterator<__gnu_cxx::__normal_iterator<const std::shared_ptr<lsst::afw::table::SourceRecord>*, std::vector<std::shared_ptr<lsst::afw::table::SourceRecord>, std::allocator<std::shared_ptr<lsst::afw::table::SourceRecord> > > > >] Record data is not contiguous in memory. {0} lsst::pex::exceptions::RuntimeError: 'Record data is not contiguous in memory.' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+16/bin/runMatchFakes.py", line 69, in <module> raCol=args.raCol, decCol=args.decCol) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+16/python/lsst/synpipe/matchFakes.py", line 593, in returnMatchTable astroTable = getAstroTable(slist, mags=True) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/synpipe/master-g5b975e8f94+16/python/lsst/synpipe/matchFakes.py", line 410, in getAstroTable except lsst.pex.exceptions.LsstException: AttributeError: module 'lsst.pex.exceptions' has no attribute 'LsstException' This does not occur if --ccd is specified.
            Hide
            sophiereed Sophie Reed added a comment -

            On line 152 of the primer the multiBandDriver is missing a capital B.

            Show
            sophiereed Sophie Reed added a comment - On line 152 of the primer the multiBandDriver is missing a capital B.
            Hide
            swinbank John Swinbank added a comment -

            This was marked as “reviewed” two months ago — what are our plans for closing it out?

            Show
            swinbank John Swinbank added a comment - This was marked as “reviewed” two months ago — what are our plans for closing it out?
            Hide
            nlust Nate Lust added a comment -

            I have a princetonite who is unofficially reviewing this (as he is trying to make use of synpipe) and so I was letting a little rope out on it. However this got blocked by a change in how we do WCS and now synpipe is broken (its not tested so no ones fault) I need to make a ticket to address it (either me or someone else) and set this as blocked by that new ticket.

            Show
            nlust Nate Lust added a comment - I have a princetonite who is unofficially reviewing this (as he is trying to make use of synpipe) and so I was letting a little rope out on it. However this got blocked by a change in how we do WCS and now synpipe is broken (its not tested so no ones fault) I need to make a ticket to address it (either me or someone else) and set this as blocked by that new ticket.
            Hide
            sophiereed Sophie Reed added a comment -

            The config option for making fake star catalogues should probably be changed to make it clear that it is inputCat not inputCatalog.

            Show
            sophiereed Sophie Reed added a comment - The config option for making fake star catalogues should probably be changed to make it clear that it is inputCat not inputCatalog.

              People

              Assignee:
              nlust Nate Lust
              Reporter:
              nlust Nate Lust
              Reviewers:
              Sophie Reed
              Watchers:
              John Swinbank, Nate Lust, Sophie Reed
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.