# obs_lsst has a race condition between tests and curated calibration ingestion

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
Architecture
• Urgent?:
No

#### Description

pytest as run by the tests SConscript attempts to scan all files, but that set can be in flux as the per-camera ingestCuratedCalibs.py executions can be running at the same time. There are no dependencies between the two.

I'm not sure if the correct solution is to sequence one before the other or to exclude those directories from pytest.

#### Activity

Hide
Robert Lupton added a comment -

Christopher Waters Are these your tests?  Thoughts?

Show
Robert Lupton added a comment - Christopher Waters Are these your tests?  Thoughts?
Hide
Christopher Waters added a comment -

I don't believe so.  My only thought is that if new calibrations were added, that might have increased the execution time long enough to start this being a problem, but I don't recall anything new being added in the past month.

Show
Christopher Waters added a comment - I don't believe so.  My only thought is that if new calibrations were added, that might have increased the execution time long enough to start this being a problem, but I don't recall anything new being added in the past month.
Hide
Kian-Tat Lim added a comment - - edited

ingestCuratedCalibs.py is part of the build (SConscripts in the camera directories), not the tests.

Show
Kian-Tat Lim added a comment - - edited ingestCuratedCalibs.py is part of the build (SConscripts in the camera directories), not the tests.
Hide
Tim Jenness added a comment -

This comment:

 # Note the ordering here is critical. LATISS is put at the end here to ensure # that the tests are run first and version.py is created, because creation of # of the defect registry required the camera to be instantiated. # If other cameras add defect generation they should add their build to # the end of this list, along with LATISS 

in the SConstruct file implies that these targets explicitly run after tests (they do also hard code a dependency on the python target) suggesting that the answer is to change the scanning code. Can someone point to an actual error report from this problem? Was there a Jenkins failure I can look at?

Show
Tim Jenness added a comment - This comment: # Note the ordering here is critical. LATISS is put at the end here to ensure # that the tests are run first and version.py is created, because creation of # of the defect registry required the camera to be instantiated. # If other cameras add defect generation they should add their build to # the end of this list, along with LATISS in the SConstruct file implies that these targets explicitly run after tests (they do also hard code a dependency on the python target) suggesting that the answer is to change the scanning code. Can someone point to an actual error report from this problem? Was there a Jenkins failure I can look at?
Hide
Kian-Tat Lim added a comment -

https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix-test/detail/stack-os-matrix-test/15/tests is an example of a failure.

My understanding from a quick glance at sconsUtils (https://github.com/lsst/sconsUtils/blob/master/python/lsst/sconsUtils/scripts.py#L208-L209) is that there is in fact no implied dependency ordering from the target list alone; any dependencies must be added separately.

Show
Kian-Tat Lim added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix-test/detail/stack-os-matrix-test/15/tests is an example of a failure. My understanding from a quick glance at sconsUtils ( https://github.com/lsst/sconsUtils/blob/master/python/lsst/sconsUtils/scripts.py#L208-L209 ) is that there is in fact no implied dependency ordering from the target list alone; any dependencies must be added separately.
Hide
Tim Jenness added a comment -

In the end I added the directories to the ignore list when running pytest.

There is also an extra unrelated fix to header stuff that I saw whilst testing this but I can remove that if needed. It's a gen2-specific problem where the curated calibs ingest calls normal raw data ingest which now calls fix_header and that causes and extra log message because curated calibrations say they are LATISS but they aren't raw latiss and header fixup gets confused.

Show
Tim Jenness added a comment - In the end I added the directories to the ignore list when running pytest. There is also an extra unrelated fix to header stuff that I saw whilst testing this but I can remove that if needed. It's a gen2-specific problem where the curated calibs ingest calls normal raw data ingest which now calls fix_header and that causes and extra log message because curated calibrations say they are LATISS but they aren't raw latiss and header fixup gets confused.
Hide
Kian-Tat Lim added a comment -

Looks fine. Thanks for dealing with this.

Show
Kian-Tat Lim added a comment - Looks fine. Thanks for dealing with this.

#### People

Assignee:
Tim Jenness
Reporter:
Kian-Tat Lim
Reviewers:
Kian-Tat Lim
Watchers:
Christopher Waters, Kian-Tat Lim, Robert Lupton, Tim Jenness