Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ctrl_pool
-
Labels:None
-
Story Points:0.5
-
Team:Architecture
-
Urgent?:No
Description
As has been noted on Slack, the log outputs from pipeline tasks currently seem to be getting swallowed when running the pipe_drivers scripts (yes, this is gen2...) E.g. a log from a w_2021_26 run was 65M and the equivalent log on a w_2021_34 run was only 5.2M (and missing a lot of important info…)
Eli Rykoff points out:
The problem I think is that ctrl_pool wasn’t updated, and there might be some funny logging stuff here: https://github.com/lsst/ctrl_pool/blob/master/python/lsst/ctrl/pool/log.py I certainly don’t know what to do with updating the MDC.
How to reproduce:
I have attached the logs from running the following with w_2021_34:
$ singleFrameDriver.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/lauren/logTest/w34 --batch-type slurm --mpiexec='-bind-to socket' --job logTest_w34 --cores 60 --time 900 --id visit=1228 ccd=0..8^10..103 |
versus running the following with w_2021_30:
$ singleFrameDriver.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun private/lauren/logTest/w30 --batch-type slurm --mpiexec='-bind-to socket' --job logTest_w30 --cores 60 --time 900 --id visit=1228 ccd=0..8^10..103 |
The differences should be pretty obvious with a quick glance (and I'm not sure what to make of the few lines that do make it into the w34 logs that have no indication of where they originated).
Of potential note: if I run with --cores 1, the relevant log output does seem to appear.
Attachments
Issue Links
Activity
This seems to work for me. I get messages like:
20275 INFO 2021-08-26T12:46:38.330-0500 singleFrameDriver.processCcd.charImage.repair: Identified 904 cosmic rays.
|
20268 INFO 2021-08-26T12:46:38.419-0500 singleFrameDriver.processCcd.charImage.measurement: Measuring 352 sources (352 parents, 0 children)
|
20278 INFO 2021-08-26T12:46:38.460-0500 singleFrameDriver.processCcd.isr: Measuring sky levels in 8x16 grids: 3293.878163.
|
20278 INFO 2021-08-26T12:46:38.460-0500 singleFrameDriver.processCcd.isr: Sky flatness in 8x16 grids - pp: 0.020951 rms: 0.005463.
|
20276 INFO 2021-08-26T12:46:38.605-0500 singleFrameDriver.processCcd.charImage.repair: Identified 779 cosmic rays.
|
20272 INFO 2021-08-26T12:46:38.724-0500 singleFrameDriver.processCcd.charImage.detection: Detected 1198 positive peaks in 282 footprints and 0 negative peaks in 0 footprints to 50 sigma
|
20278 INFO 2021-08-26T12:46:38.730-0500 singleFrameDriver.processCcd.charImage: Processing {'visit': 1228, 'ccd': 58, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0}
|
in my log output file.
I’m not in a position to test myself at the moment, but your log output looks good. Jenkins won’t probe this directly, but it wouldn’t hurt.
Thanks for the quick fix!
It might be that adding these lines to that log file somewhere is all that is needed:
lgr.addHandler(lsstLog.LogHandler())(this is what happens in command line task)