[DM-21915] Support Gen 3 ingestion of ap_verify datasets - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: ap_verify
Labels:

Story Points:
6
Epic Link:
f20a-ap-infrastructure
Sprint:
AP S20-5 (April), AP S20-6 (May), AP F20-1 (June), AP F20-2 (July)
Team:
Alert Production
Urgent?:
No

Description

lsst.ap.verify.DatasetIngestTask is responsible for ingesting an ap_verify dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in ingest_dataset.py that lets users choose between the two.

Note that the current dataset format includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket.

Some overlap with ~~DM-21862~~, which wants to use HiTS to run single-frame processing.

Attachments

Issue Links

blocks

DM-21919 Run ap_verify end-to-end in Gen 3

Done

DM-25806 Support parallel ap_verify ingestion in Gen 3

Done

has to be done after

DM-24260 Create Gen3 versions of ap_verify datasets

Done

is blocked by

DM-25786 Cannot import sphgeom objects in Gen 3

Done

DM-24844 Migrate ap_verify_testdata to obs_lsst

Done

DM-21016 Handle DECam instrument signature data in gen3

Done

relates to

DM-25945 Registry.queryCollections raises MissingCollectionError

To Do

DM-25970 ap_verify CI command line broken

Done

DM-21862 Extend Gen3 butler support for obs_decam

Done

DM-23616 Run converted ap_verify testdata through gen3 pipeline

Done

DM-21898 Create command-line tools for Gen3 repo administration

Won't Fix

DM-24314 Support following links in Butler operations

Done

DM-24365 Add relative symbolic link transfer mode to Gen 3 ingest

Done

DM-21939 Create Gen 3 AP Pipeline

Done

DM-26070 Add visit definition to ap_verify

Done

(1 is blocked by, 9 relates to)

Activity

Ascending order - Click to sort in descending order

Krzysztof Findeisen created issue - 23/Oct/19 9:29 PM

Krzysztof Findeisen made changes - 23/Oct/19 9:29 PM

Field	Original Value	New Value
Link		This issue blocks ~~DM-21888~~ [ ~~DM-21888~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 9:29 PM

Link

This issue relates to ~~DM-21862~~ [ ~~DM-21862~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 9:33 PM

Link

This issue relates to ~~DM-21898~~ [ ~~DM-21898~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 9:45 PM

Description

{{lsst.ap.verify.DatasetIngestTask}} is responsible for ingesting an {{ap_verify}} dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in {{ap_verify.py}} and {{ingest_dataset.py}} that lets users choose between the two.

Note that the current [dataset format|https://pipelines.lsst.io/modules/lsst.ap.verify/datasets-creation.html] includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket.

Some overlap with ~~DM-21862~~, which wants to use HiTS to run single-frame processing.

{{lsst.ap.verify.DatasetIngestTask}} is responsible for ingesting an {{ap_verify}} dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in {{ap_verify.py}} and {{ingest_dataset.py}} that lets users choose between the two; naturally, the former must also run {{ApPipeTask}} in Gen 3.

Note that the current [dataset format|https://pipelines.lsst.io/modules/lsst.ap.verify/datasets-creation.html] includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket.

Some overlap with ~~DM-21862~~, which wants to use HiTS to run single-frame processing.

Krzysztof Findeisen made changes - 23/Oct/19 9:46 PM

Description

{{lsst.ap.verify.DatasetIngestTask}} is responsible for ingesting an {{ap_verify}} dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in {{ap_verify.py}} and {{ingest_dataset.py}} that lets users choose between the two; naturally, the former must also run {{ApPipeTask}} in Gen 3.

Note that the current [dataset format|https://pipelines.lsst.io/modules/lsst.ap.verify/datasets-creation.html] includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket.

Some overlap with ~~DM-21862~~, which wants to use HiTS to run single-frame processing.

{{lsst.ap.verify.DatasetIngestTask}} is responsible for ingesting an {{ap_verify}} dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in {{ap_verify.py}} and {{ingest_dataset.py}} that lets users choose between the two; naturally, the former must also run {{ap_pipe}} as a Gen 3 pipeline.

Note that the current [dataset format|https://pipelines.lsst.io/modules/lsst.ap.verify/datasets-creation.html] includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket.

Some overlap with ~~DM-21862~~, which wants to use HiTS to run single-frame processing.

Krzysztof Findeisen made changes - 23/Oct/19 9:47 PM

Link

This issue has to be done after ~~DM-21886~~ [ ~~DM-21886~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 9:47 PM

Link

This issue has to be done after ~~DM-21885~~ [ ~~DM-21885~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:03 PM

Summary

Support Gen 3 ingestion of ap_verify datasets

Support Gen 3 ingestion/processing of ap_verify datasets

Krzysztof Findeisen made changes - 23/Oct/19 10:07 PM

Link

This issue is blocked by ~~DM-21862~~ [ ~~DM-21862~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:08 PM

Link

This issue is blocked by ~~DM-21886~~ [ ~~DM-21886~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:08 PM

Link

This issue is blocked by ~~DM-21885~~ [ ~~DM-21885~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:08 PM

Link

This issue has to be done after ~~DM-21885~~ [ ~~DM-21885~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:08 PM

Link

This issue has to be done after ~~DM-21886~~ [ ~~DM-21886~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:17 PM

Link

This issue blocks ~~DM-21919~~ [ ~~DM-21919~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:17 PM

Comment

[ Adding ~~DM-21885~~ and ~~DM-21886~~ as blockers for being able to actually use an ingested dataset in {{ap_verify}}. ]

Krzysztof Findeisen made changes - 23/Oct/19 10:18 PM

Summary	Support Gen 3 ingestion/processing of ap_verify datasets	Support Gen 3 ingestion of ap_verify datasets
Description	{{lsst.ap.verify.DatasetIngestTask}} is responsible for ingesting an {{ap_verify}} dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in {{ap_verify.py}} and {{ingest_dataset.py}} that lets users choose between the two; naturally, the former must also run {{ap_pipe}} as a Gen 3 pipeline. Note that the current [dataset format\|https://pipelines.lsst.io/modules/lsst.ap.verify/datasets-creation.html] includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket. Some overlap with ~~DM-21862~~, which wants to use HiTS to run single-frame processing.	{{lsst.ap.verify.DatasetIngestTask}} is responsible for ingesting an {{ap_verify}} dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in {{ingest_dataset.py}} that lets users choose between the two. Note that the current [dataset format\|https://pipelines.lsst.io/modules/lsst.ap.verify/datasets-creation.html] includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket. Some overlap with ~~DM-21862~~, which wants to use HiTS to run single-frame processing.
Story Points	6	4

Krzysztof Findeisen made changes - 23/Oct/19 10:18 PM

Link

This issue is blocked by ~~DM-21885~~ [ ~~DM-21885~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:18 PM

Link

This issue is blocked by ~~DM-21886~~ [ ~~DM-21886~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:19 PM

Link

This issue is blocked by ~~DM-21862~~ [ ~~DM-21862~~ ]

Krzysztof Findeisen made changes - 23/Oct/19 10:22 PM

Story Points

Krzysztof Findeisen made changes - 23/Oct/19 11:34 PM

Sprint

AP F19-6 (November) [ 958 ]

Jim Bosch made changes - 24/Oct/19 4:32 PM

Labels

gen3-middleware

gen2-deprecation-blocker gen3-middleware

Krzysztof Findeisen made changes - 24/Oct/19 5:30 PM

Link

This issue relates to ~~DM-21939~~ [ ~~DM-21939~~ ]

Krzysztof Findeisen made changes - 24/Oct/19 5:41 PM

Link

This issue blocks ~~DM-21888~~ [ ~~DM-21888~~ ]

Krzysztof Findeisen made changes - 31/Oct/19 6:33 PM

Epic Link

~~DM-21442~~ [ 423049 ]

Krzysztof Findeisen made changes - 31/Oct/19 7:30 PM

Labels

gen2-deprecation-blocker gen3-middleware

PairCoding gen2-deprecation-blocker gen3-middleware

Krzysztof Findeisen made changes - 31/Oct/19 9:20 PM

Link

This issue is blocked by ~~DM-21016~~ [ ~~DM-21016~~ ]

Krzysztof Findeisen made changes - 15/Nov/19 7:50 PM

Status

To Do [ 10001 ]

In Progress [ 3 ]

Krzysztof Findeisen made changes - 25/Nov/19 11:30 PM

Link

This issue is blocked by DM-22370 [ DM-22370 ]

Krzysztof Findeisen made changes - 06/Dec/19 10:40 PM

Sprint

AP F19-6 (November) [ 958 ]

John Swinbank made changes - 14/Dec/19 11:35 PM

Epic Link

~~DM-21442~~ [ 423049 ]

~~DM-22633~~ [ 427742 ]

Krzysztof Findeisen made changes - 16/Dec/19 9:36 PM

Rank

Ranked higher

Krzysztof Findeisen made changes - 16/Dec/19 9:36 PM

Rank

Ranked higher

Eric Bellm made changes - 15/Jan/20 12:09 AM

Sprint

AP S20-6 (May) [ 987 ]

Eric Bellm made changes - 15/Jan/20 12:09 AM

Sprint

AP S20-6 (May) [ 987 ]

Krzysztof Findeisen made changes - 26/Feb/20 11:05 PM

Comment

[ The config files included in the current datasets are not compatible with Gen 3, because in Gen 3 each task in a pipeline must be configured separately. ]

John Swinbank made changes - 31/Mar/20 4:48 PM

Link

This issue blocks ~~DM-24260~~ [ ~~DM-24260~~ ]

John Swinbank made changes - 31/Mar/20 4:48 PM

Urgent?

off

John Swinbank made changes - 31/Mar/20 4:48 PM

Sprint

AP S20-5 (April) [ 986 ]

Krzysztof Findeisen made changes - 02/Apr/20 10:12 PM

Link

This issue relates to ~~DM-24314~~ [ ~~DM-24314~~ ]

Krzysztof Findeisen made changes - 02/Apr/20 11:38 PM

Link

This issue is blocked by DM-22370 [ DM-22370 ]

John Swinbank made changes - 03/Apr/20 11:05 PM

Epic Link

~~DM-22633~~ [ 427742 ]

~~DM-24341~~ [ 433028 ]

Krzysztof Findeisen made changes - 06/Apr/20 8:58 PM

Link

This issue relates to ~~DM-24365~~ [ ~~DM-24365~~ ]

Krzysztof Findeisen made changes - 07/Apr/20 5:49 PM

Link

This issue has to be finished together with ~~DM-24260~~ [ ~~DM-24260~~ ]

Krzysztof Findeisen made changes - 07/Apr/20 5:49 PM

Link

This issue blocks ~~DM-24260~~ [ ~~DM-24260~~ ]

Krzysztof Findeisen made changes - 30/Apr/20 5:53 PM

Link

This issue relates to ~~DM-23616~~ [ ~~DM-23616~~ ]

John Swinbank made changes - 11/May/20 9:10 PM

Sprint

AP S20-5 (April) [ 986 ]

AP S20-5 (April), AP S20-6 (May) [ 986, 987 ]

Krzysztof Findeisen made changes - 11/May/20 9:21 PM

Link

This issue has to be done after ~~DM-24260~~ [ ~~DM-24260~~ ]

Krzysztof Findeisen made changes - 11/May/20 9:21 PM

Link

This issue has to be finished together with ~~DM-24260~~ [ ~~DM-24260~~ ]

Krzysztof Findeisen made changes - 11/May/20 9:32 PM

Link

This issue is blocked by ~~DM-24844~~ [ ~~DM-24844~~ ]

John Swinbank made changes - 04/Jun/20 12:38 AM

Epic Link

~~DM-24341~~ [ 433028 ]

~~DM-25145~~ [ 435263 ]

John Swinbank made changes - 04/Jun/20 7:00 PM

Sprint

AP S20-5 (April), AP S20-6 (May) [ 986, 987 ]

AP S20-5 (April), AP S20-6 (May), AP F20-1 (June) [ 986, 987, 1019 ]

Krzysztof Findeisen made changes - 02/Jul/20 2:21 PM

Link

This issue is blocked by ~~DM-25786~~ [ ~~DM-25786~~ ]

John Swinbank made changes - 02/Jul/20 6:27 PM

Sprint

AP S20-5 (April), AP S20-6 (May), AP F20-1 (June) [ 986, 987, 1019 ]

AP S20-5 (April), AP S20-6 (May), AP F20-1 (June), AP F20-2 (July) [ 986, 987, 1019, 1025 ]

Krzysztof Findeisen made changes - 06/Jul/20 1:54 PM

Link

This issue blocks ~~DM-25806~~ [ ~~DM-25806~~ ]

Krzysztof Findeisen made changes - 09/Jul/20 11:40 PM

Status	In Progress [ 3 ]	In Review [ 10004 ]
Reviewers		Jim Bosch [ jbosch ]

Jim Bosch made changes - 10/Jul/20 5:06 PM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Krzysztof Findeisen made changes - 13/Jul/20 11:16 PM

Link

This issue relates to DM-25945 [ DM-25945 ]

Krzysztof Findeisen made changes - 14/Jul/20 11:26 PM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

Krzysztof Findeisen made changes - 15/Jul/20 4:37 PM

Link

This issue relates to ~~DM-25970~~ [ ~~DM-25970~~ ]

Krzysztof Findeisen made changes - 22/Jul/20 10:42 PM

Link

This issue relates to ~~DM-26070~~ [ ~~DM-26070~~ ]

People

Assignee:: Krzysztof Findeisen

Reporter:: Krzysztof Findeisen

Reviewers:: Jim Bosch

Watchers:: Jim Bosch, John Parejko, Krzysztof Findeisen, Meredith Rawls, Tim Jenness

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 23/Oct/19 9:29 PM

Updated:: 22/Jul/20 10:42 PM

Resolved:: 14/Jul/20 11:26 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Issue Links

Activity

People

Dates

Jenkins