# Quantum graph ordering for component datasets

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
4
• Team:
Data Release Production
• Urgent?:
No

#### Description

GetTemplateTask takes two calexp components, calexp.bbox and calexp.wcs, as Input connections. When I build the graph for a complete AP pipeline (from ISR to diaPipe), getTemplate should appear after calibrate, taking those inputs from the calexp output by calibrate. Instead it appears off by itself, with only its output connected to the difference imaging task.

Probably relatedly, the index order of GetTemplateTask in the generated graph depends on the task name assigned to it in the pipeline yaml definition. See the attached image for an example: naming it "gretrieveTemplate" results in index=1 (thus before calibrate and resulting in an empty qgraph), whereas "retrieveTemplate" results in index=14 (thus after calibrate, and an executable pipeline).

#### Attachments

1. gretrieve vs retrieve qgraph.png
92 kB

#### Activity

Hide
Jim Bosch added a comment -

Hate to pile one more, "oh, can you also do X" on this ticket, but I've just noticed that the pipeline dot outputs don't seem to be deterministic: at least task nodes an dimensions seem to have arbitrary order (I'm guessing set iteration, but I'm not sure). If you see any easy ways to address that, please do.

Show
Jim Bosch added a comment - Hate to pile one more, "oh, can you also do X" on this ticket, but I've just noticed that the pipeline dot outputs don't seem to be deterministic: at least task nodes an dimensions seem to have arbitrary order (I'm guessing set iteration, but I'm not sure). If you see any easy ways to address that, please do.
Hide
Nate Lust added a comment -

Jim Bosch A note on your request. I can sort everything lexicographical, and because we add in the edges and let dot draw the graph everything comes out fine. However, the task index makes no sense. If I dont sort tasks the index makes more sense, but it is not deterministic because the topographical sorting that comes out of the pipeline is not deterministic. Which would you prefer?

Show
Nate Lust added a comment - Jim Bosch A note on your request. I can sort everything lexicographical, and because we add in the edges and let dot draw the graph everything comes out fine. However, the task index makes no sense. If I dont sort tasks the index makes more sense, but it is not deterministic because the topographical sorting that comes out of the pipeline is not deterministic. Which would you prefer?
Hide
Jim Bosch added a comment -

Oh, I had no idea the topological sorting that comes out of the pipeline isn't deterministic - I thought we broke ties lexicographically there, too. How is the task index actually used? Just something internal to the dot representation? It doesn't sound like something we want to display to the user.

Show
Jim Bosch added a comment - Oh, I had no idea the topological sorting that comes out of the pipeline isn't deterministic - I thought we broke ties lexicographically there, too. How is the task index actually used? Just something internal to the dot representation? It doesn't sound like something we want to display to the user.
Hide
Nate Lust added a comment -

Jim BoschI just removed displaying that index at all, since it is not really that informative anyway.

Tim JennessI added the metadata connection as a workaround as I hope to have metadata handled normally after a future ticket.

Either of you two or both want to review these extra commits since you requested them? Otherwise Andy Salnikov can you take another look?

Show
Nate Lust added a comment - Jim Bosch I just removed displaying that index at all, since it is not really that informative anyway. Tim Jenness I added the metadata connection as a workaround as I hope to have metadata handled normally after a future ticket. Either of you two or both want to review these extra commits since you requested them? Otherwise Andy Salnikov can you take another look?
Hide
Andy Salnikov added a comment -

Sorry I missed the comment three days ago. Re-reviewed, looks good, couple of comments on PR.

Show
Andy Salnikov added a comment - Sorry I missed the comment three days ago. Re-reviewed, looks good, couple of comments on PR.

#### People

Assignee:
Nate Lust
Reporter:
John Parejko
Reviewers:
Andy Salnikov
Watchers:
Andy Salnikov, Ian Sullivan, Jim Bosch, John Parejko, Kian-Tat Lim, Nate Lust, Tim Jenness