Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30624

Document gen2 to gen3 refcat ingestion

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

      Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:

      butler convert $REPO --gen2root $GEN2REPO -C convert_refcats.py
      

      the script fails with:

      RuntimeError: No exposures given.
      

      (full output log attached to this ticket).

      This ticket aims to copy the relevant parts of code within `obs/base/gen2to3/rootRepoConverter.py` and place them into a new script within `meas_algorithms` use existing butler tools to provide updated instructions in the existing pipelines.lsst.io reference catalog documentation.

      Attachments

        Issue Links

          Activity

            lskelvin Lee Kelvin created issue -
            lskelvin Lee Kelvin made changes -
            Field Original Value New Value
            Attachment convert_refcats.log [ 49979 ]
            lskelvin Lee Kelvin made changes -
            Description The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

            Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:
            {code:sh}
            butler convert $REPO --gen2root $GEN2REPO -C $GEN2REPO/convert_refcats.py
            {code}
            the script fails with:
            {code:sh}
            RuntimeError: No exposures given.
            {code}
            (full output log attached to this ticket).
            The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

            Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:
            {code:sh}
            butler convert $REPO --gen2root $GEN2REPO -C $GEN2REPO/convert_refcats.py
            {code}
            the script fails with:
            {code:sh}
            RuntimeError: No exposures given.
            {code}
            (full output log attached to this ticket).

            This ticket aims to cut out the relevant parts of code within `obs/base/gen2to3/rootRepoConverter.py` and place them into a new script within `meas_algorithms`.
            lskelvin Lee Kelvin made changes -
            Description The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

            Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:
            {code:sh}
            butler convert $REPO --gen2root $GEN2REPO -C $GEN2REPO/convert_refcats.py
            {code}
            the script fails with:
            {code:sh}
            RuntimeError: No exposures given.
            {code}
            (full output log attached to this ticket).

            This ticket aims to cut out the relevant parts of code within `obs/base/gen2to3/rootRepoConverter.py` and place them into a new script within `meas_algorithms`.
            The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

            Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:
            {code:sh}
            butler convert $REPO --gen2root $GEN2REPO -C $GEN2REPO/convert_refcats.py
            {code}
            the script fails with:
            {code:sh}
            RuntimeError: No exposures given.
            {code}
            (full output log attached to this ticket).

            This ticket aims to copy the relevant parts of code within `obs/base/gen2to3/rootRepoConverter.py` and place them into a new script within `meas_algorithms`.
            lskelvin Lee Kelvin made changes -
            Summary Set up Gen2 to Gen3 Standalone RefCat Script Setup gen2 to gen3 standalone refcat conversion script
            lskelvin Lee Kelvin made changes -
            Description The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

            Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:
            {code:sh}
            butler convert $REPO --gen2root $GEN2REPO -C $GEN2REPO/convert_refcats.py
            {code}
            the script fails with:
            {code:sh}
            RuntimeError: No exposures given.
            {code}
            (full output log attached to this ticket).

            This ticket aims to copy the relevant parts of code within `obs/base/gen2to3/rootRepoConverter.py` and place them into a new script within `meas_algorithms`.
            The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

            Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:
            {code:sh}
            butler convert $REPO --gen2root $GEN2REPO -C convert_refcats.py
            {code}
            the script fails with:
            {code:sh}
            RuntimeError: No exposures given.
            {code}
            (full output log attached to this ticket).

            This ticket aims to copy the relevant parts of code within `obs/base/gen2to3/rootRepoConverter.py` and place them into a new script within `meas_algorithms`.
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-29543 [ DM-29543 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-29857 [ DM-29857 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30471 [ 509196 ] DM-30536 [ 511194 ]
            lskelvin Lee Kelvin made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lskelvin Lee Kelvin made changes -
            Description The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

            Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:
            {code:sh}
            butler convert $REPO --gen2root $GEN2REPO -C convert_refcats.py
            {code}
            the script fails with:
            {code:sh}
            RuntimeError: No exposures given.
            {code}
            (full output log attached to this ticket).

            This ticket aims to copy the relevant parts of code within `obs/base/gen2to3/rootRepoConverter.py` and place them into a new script within `meas_algorithms`.
            The current command line script `butler convert` performs a large number of tasks in addition to reference catalog conversion, including sky map mapping, raw data ingestion and data ID processing.

            Recent use of this command whilst setting up a new butler on a new machine highlighted the need for a standalone gen2 to gen3 refcat conversion script. When running:
            {code:sh}
            butler convert $REPO --gen2root $GEN2REPO -C convert_refcats.py
            {code}
            the script fails with:
            {code:sh}
            RuntimeError: No exposures given.
            {code}
            (full output log attached to this ticket).

            This ticket aims to -copy the relevant parts of code within `obs/base/gen2to3/rootRepoConverter.py` and place them into a new script within `meas_algorithms`- use existing butler tools to provide updated instructions in the existing [pipelines.lsst.io reference catalog documentation|https://pipelines.lsst.io/modules/lsst.meas.algorithms/creating-a-reference-catalog.html].
            lskelvin Lee Kelvin made changes -
            Summary Setup gen2 to gen3 standalone refcat conversion script Document gen2 to gen3 refcat ingestion
            lskelvin Lee Kelvin made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            lskelvin Lee Kelvin made changes -
            Resolution Done [ 10000 ]
            Status Done [ 10002 ] To Do [ 10001 ]
            lskelvin Lee Kelvin made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lskelvin Lee Kelvin made changes -
            Reviewers John Parejko [ parejkoj ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            lskelvin Lee Kelvin made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            lskelvin Lee Kelvin made changes -
            Story Points 2

            People

              lskelvin Lee Kelvin
              lskelvin Lee Kelvin
              John Parejko
              Ian Sullivan, John Parejko, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.