Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22227

measurePhotonTransferCurve.py fails to find raftName

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe
    • Labels:
      None
    • Team:
      DM Science

      Description

      Andrés reports that he can't run measurePhotonTransferCurve.py, with the failure reporting:

      RuntimeError: Unable to retrieve bias for DataId(initialdata=

      Unknown macro: {'detector'}

      , tag=set()): no such column: raftName.

      The problem's deep down in Butler Gen2, and the simplest "fix" is to lookup the raftName (and detectorName) before trying to run the ISR. This hack should not be needed in Gen3; it's equivalent to the calls to CalibTask.addMissingKeys().

        Attachments

          Activity

          Hide
          rhl Robert Lupton added a comment - - edited

          To support raftName=R22 detectorName=S20 (instead of detector=97) I also needed to modify the definition in obs_base to

          measurePhotonTransferCurveGainAndNoise:
          description: "Persisted output data (gain and noise) from the MeasurePhotonTransferCurveTask"
          persistable: ignored
          python: lsst.cp.pipe.ptc.PtcDataset
          storage: PickleStorage
          tables: 'raw'
          template: ''

          Note that I've added tables.

          Show
          rhl Robert Lupton added a comment - - edited To support raftName=R22 detectorName=S20 (instead of detector=97 ) I also needed to modify the definition in obs_base to measurePhotonTransferCurveGainAndNoise: description: "Persisted output data (gain and noise) from the MeasurePhotonTransferCurveTask" persistable: ignored python: lsst.cp.pipe.ptc.PtcDataset storage: PickleStorage tables: 'raw' template: '' Note that I've added tables.
          Hide
          plazas Andrés Alejandro Plazas Malagón added a comment - - edited

          Thanks for looking into this. 

          1) The following command (that uses --id detector=95) now runs to completion (with the branch tickets/DM-22227 of cp_pipe and w_2019_45 set up): 

           

          measurePhotonTransferCurve.py /project/shared/BOT/ --rerun /project/plazas/test --id detector=95 --visit-pairs 2019101200233,2019101200234 2019101200236,2019101200237 2019101200239,2019101200240 2019101200242,2019101200243 2019101200279,2019101200280 2019101200290,2019101200291 2019101200293,2019101200294 2019101200299,2019101200300 2019101200305,2019101200306 2019101200311,2019101200312 2019101200314,2019101200315 2019101200317,2019101200318 2019101200323,2019101200324 2019101200326,2019101200327 2019101200329,2019101200330 2019101200332,2019101200333 2019101200341,2019101200355 2019101200358,2019101200359 2019101200361,2019101200362 2019101200364,2019101200365 2019101200370,2019101200371 2019101200373,2019101200374 2019101200376,2019101200377 2019101200379,2019101200380 2019101200382,2019101200383 2019101200385,2019101200386 2019101200394,2019101200395 2019101200408,2019101200409 2019101200411,2019101200412 2019101200417,2019101200418 2019101200423,2019101200424 2019101200426,2019101200427 2019101200429,2019101200430 2019101200432,2019101200433 2019101200435,2019101200436 2019101200438,2019101200439 2019101200441,2019101200442 2019101200444,2019101200445 2019101200450,2019101200451 2019101200453,2019101200454 2019101200456,2019101200457 2019101200459,2019101200460 2019101200462,2019101200463 2019101200465,2019101200466 2019101200468,2019101200469 2019101200471,2019101200472 2019101200474,2019101200475 2019101200480,2019101200481 2019101200483,2019101200484 2019101300003,2019101300004 2019101300006,2019101300007 2019101300009,2019101300010 2019101300012,2019101300013 2019101300015,2019101300016 2019101300018,2019101300019 2019101300021,2019101300022 2019101300024,2019101300025 2019101300027,2019101300028 2019101300030,2019101300031 2019101300036,2019101300037 2019101300039,2019101300040 2019101300042,2019101300043 2019101300045,2019101300046 2019101300048,2019101300049 2019101300051,2019101300052 2019101300057,2019101300058 2019101300060,2019101300061 2019101300063,2019101300064 2019101300069,2019101300070 2019101300078,2019101300079 2019101300081,2019101300082 2019101300084,2019101300085 2019101300087,2019101300088 2019101300090,2019101300091 2019101300093,2019101300094 2019101300096,2019101300097 2019101300099,2019101300100 2019101300102,2019101300103 2019101300111,2019101300112 2019101300114,2019101300115 2019101300117,2019101300118 2019101300120,2019101300121 2019101300123,2019101300124 2019101300126,2019101300127 2019101300132,2019101300133 2019101300135,2019101300136 2019101300138,2019101300139 2019101300141,2019101300142 2019101300144,2019101300145 2019101300147,2019101300148 2019101300150,2019101300151 2019101300153,2019101300154 2019101300156,2019101300157 2019101300159,2019101300160 2019101300162,2019101300163 2019101300165,2019101300166 2019101300171,2019101300172 2019101300174,2019101300175 2019101300177,2019101300178 2019101300180,2019101300181 2019101300183,2019101300184 2019101300186,2019101300187 2019101300192,2019101300193 2019101300195,2019101300196 2019101300198,2019101300199 2019101300201,2019101300202 2019101300204,2019101300205 2019101300207,2019101300208 2019101300219,2019101300220 2019101300225,2019101300226 2019101300228,2019101300229 2019101300231,2019101300232 2019101300234,2019101300235 2019101300237,2019101300238 2019101300243,2019101300244 2019101300246,2019101300247 2019101300252,2019101300253 2019101300255,2019101300256 2019101300261,2019101300262 2019101300264,2019101300265 2019101300267,2019101300268 2019101300270,2019101300271 2019101300273,2019101300274 2019101300276,2019101300277 2019101300282,2019101300283 2019101300285,2019101300286 2019101300288,2019101300289 2019101300291,2019101300292 2019101300294,2019101300295 2019101300297,2019101300298 2019101300300,2019101300301 2019101300303,2019101300304 2019101300306,2019101300307 2019101300318,2019101300319 2019101300321,2019101300322 2019101300324,2019101300325 2019101300330,2019101300331 2019101300333,2019101300334 2019101300336,2019101300337 2019101300339,2019101300340 2019101300342,2019101300343 2019101300351,2019101300352 2019101300354,2019101300355 2019101300363,2019101300364 2019101300366,2019101300367 2019101300369,2019101300370 2019101300375,2019101300376 2019101300378,2019101300379 2019101300381,2019101300382 2019101300384,2019101300385 2019101300387,2019101300388 2019101300390,2019101300391 2019101300393,2019101300394 2019101300396,2019101300397 2019101300399,2019101300400 2019101300402,2019101300403 2019101300405,2019101300406 2019101300408,2019101300409 2019101300411,2019101300412 2019101300420,2019101300421 2019101300423,2019101300424 --clobber-config --clobber-versions -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 maxMeanSignal=100000 -c isr.doBias=False isr.doDark=False isr.doSuspect=False

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - - edited Thanks for looking into this.  1) The following command (that uses --id detector=95) now runs to completion (with the branch tickets/ DM-22227 of cp_pipe and w_2019_45 set up):    measurePhotonTransferCurve.py /project/shared/BOT/ --rerun /project/plazas/test --id detector=95 --visit-pairs 2019101200233,2019101200234 2019101200236,2019101200237 2019101200239,2019101200240 2019101200242,2019101200243 2019101200279,2019101200280 2019101200290,2019101200291 2019101200293,2019101200294 2019101200299,2019101200300 2019101200305,2019101200306 2019101200311,2019101200312 2019101200314,2019101200315 2019101200317,2019101200318 2019101200323,2019101200324 2019101200326,2019101200327 2019101200329,2019101200330 2019101200332,2019101200333 2019101200341,2019101200355 2019101200358,2019101200359 2019101200361,2019101200362 2019101200364,2019101200365 2019101200370,2019101200371 2019101200373,2019101200374 2019101200376,2019101200377 2019101200379,2019101200380 2019101200382,2019101200383 2019101200385,2019101200386 2019101200394,2019101200395 2019101200408,2019101200409 2019101200411,2019101200412 2019101200417,2019101200418 2019101200423,2019101200424 2019101200426,2019101200427 2019101200429,2019101200430 2019101200432,2019101200433 2019101200435,2019101200436 2019101200438,2019101200439 2019101200441,2019101200442 2019101200444,2019101200445 2019101200450,2019101200451 2019101200453,2019101200454 2019101200456,2019101200457 2019101200459,2019101200460 2019101200462,2019101200463 2019101200465,2019101200466 2019101200468,2019101200469 2019101200471,2019101200472 2019101200474,2019101200475 2019101200480,2019101200481 2019101200483,2019101200484 2019101300003,2019101300004 2019101300006,2019101300007 2019101300009,2019101300010 2019101300012,2019101300013 2019101300015,2019101300016 2019101300018,2019101300019 2019101300021,2019101300022 2019101300024,2019101300025 2019101300027,2019101300028 2019101300030,2019101300031 2019101300036,2019101300037 2019101300039,2019101300040 2019101300042,2019101300043 2019101300045,2019101300046 2019101300048,2019101300049 2019101300051,2019101300052 2019101300057,2019101300058 2019101300060,2019101300061 2019101300063,2019101300064 2019101300069,2019101300070 2019101300078,2019101300079 2019101300081,2019101300082 2019101300084,2019101300085 2019101300087,2019101300088 2019101300090,2019101300091 2019101300093,2019101300094 2019101300096,2019101300097 2019101300099,2019101300100 2019101300102,2019101300103 2019101300111,2019101300112 2019101300114,2019101300115 2019101300117,2019101300118 2019101300120,2019101300121 2019101300123,2019101300124 2019101300126,2019101300127 2019101300132,2019101300133 2019101300135,2019101300136 2019101300138,2019101300139 2019101300141,2019101300142 2019101300144,2019101300145 2019101300147,2019101300148 2019101300150,2019101300151 2019101300153,2019101300154 2019101300156,2019101300157 2019101300159,2019101300160 2019101300162,2019101300163 2019101300165,2019101300166 2019101300171,2019101300172 2019101300174,2019101300175 2019101300177,2019101300178 2019101300180,2019101300181 2019101300183,2019101300184 2019101300186,2019101300187 2019101300192,2019101300193 2019101300195,2019101300196 2019101300198,2019101300199 2019101300201,2019101300202 2019101300204,2019101300205 2019101300207,2019101300208 2019101300219,2019101300220 2019101300225,2019101300226 2019101300228,2019101300229 2019101300231,2019101300232 2019101300234,2019101300235 2019101300237,2019101300238 2019101300243,2019101300244 2019101300246,2019101300247 2019101300252,2019101300253 2019101300255,2019101300256 2019101300261,2019101300262 2019101300264,2019101300265 2019101300267,2019101300268 2019101300270,2019101300271 2019101300273,2019101300274 2019101300276,2019101300277 2019101300282,2019101300283 2019101300285,2019101300286 2019101300288,2019101300289 2019101300291,2019101300292 2019101300294,2019101300295 2019101300297,2019101300298 2019101300300,2019101300301 2019101300303,2019101300304 2019101300306,2019101300307 2019101300318,2019101300319 2019101300321,2019101300322 2019101300324,2019101300325 2019101300330,2019101300331 2019101300333,2019101300334 2019101300336,2019101300337 2019101300339,2019101300340 2019101300342,2019101300343 2019101300351,2019101300352 2019101300354,2019101300355 2019101300363,2019101300364 2019101300366,2019101300367 2019101300369,2019101300370 2019101300375,2019101300376 2019101300378,2019101300379 2019101300381,2019101300382 2019101300384,2019101300385 2019101300387,2019101300388 2019101300390,2019101300391 2019101300393,2019101300394 2019101300396,2019101300397 2019101300399,2019101300400 2019101300402,2019101300403 2019101300405,2019101300406 2019101300408,2019101300409 2019101300411,2019101300412 2019101300420,2019101300421 2019101300423,2019101300424 --clobber-config --clobber-versions -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 maxMeanSignal=100000 -c isr.doBias=False isr.doDark=False isr.doSuspect=False
          Hide
          plazas Andrés Alejandro Plazas Malagón added a comment -

          2). With respect to the other issue (raftName=R22 detectorName=S12 instead of detector=95), the following command ran to completion too (with the branch tickets/DM-22227 of cp_pipe and obs_base, and w_2019_45 set up): 

           

          /measurePhotonTransferCurve.py /project/shared/BOT/ --rerun /project/plazas/test --id raftName=R22 detectorName=S12 --visit-pairs 2019101200233,2019101200234 2019101200236,2019101200237 2019101200239,2019101200240 2019101200242,2019101200243 2019101200279,2019101200280 2019101200290,2019101200291 2019101200293,2019101200294 2019101200299,2019101200300 2019101200305,2019101200306 2019101200311,2019101200312 2019101200314,2019101200315 2019101200317,2019101200318 2019101200323,2019101200324 2019101200326,2019101200327 2019101200329,2019101200330 2019101200332,2019101200333 2019101200341,2019101200355 2019101200358,2019101200359 2019101200361,2019101200362 2019101200364,2019101200365 2019101200370,2019101200371 2019101200373,2019101200374 2019101200376,2019101200377 2019101200379,2019101200380 2019101200382,2019101200383 2019101200385,2019101200386 2019101200394,2019101200395 2019101200408,2019101200409 2019101200411,2019101200412 2019101200417,2019101200418 2019101200423,2019101200424 2019101200426,2019101200427 2019101200429,2019101200430 2019101200432,2019101200433 2019101200435,2019101200436 2019101200438,2019101200439 2019101200441,2019101200442 2019101200444,2019101200445 2019101200450,2019101200451 2019101200453,2019101200454 2019101200456,2019101200457 2019101200459,2019101200460 2019101200462,2019101200463 2019101200465,2019101200466 2019101200468,2019101200469 2019101200471,2019101200472 2019101200474,2019101200475 2019101200480,2019101200481 2019101200483,2019101200484 2019101300003,2019101300004 2019101300006,2019101300007 2019101300009,2019101300010 2019101300012,2019101300013 2019101300015,2019101300016 2019101300018,2019101300019 2019101300021,2019101300022 2019101300024,2019101300025 2019101300027,2019101300028 2019101300030,2019101300031 2019101300036,2019101300037 2019101300039,2019101300040 2019101300042,2019101300043 2019101300045,2019101300046 2019101300048,2019101300049 2019101300051,2019101300052 2019101300057,2019101300058 2019101300060,2019101300061 2019101300063,2019101300064 2019101300069,2019101300070 2019101300078,2019101300079 2019101300081,2019101300082 2019101300084,2019101300085 2019101300087,2019101300088 2019101300090,2019101300091 2019101300093,2019101300094 2019101300096,2019101300097 2019101300099,2019101300100 2019101300102,2019101300103 2019101300111,2019101300112 2019101300114,2019101300115 2019101300117,2019101300118 2019101300120,2019101300121 2019101300123,2019101300124 2019101300126,2019101300127 2019101300132,2019101300133 2019101300135,2019101300136 2019101300138,2019101300139 2019101300141,2019101300142 2019101300144,2019101300145 2019101300147,2019101300148 2019101300150,2019101300151 2019101300153,2019101300154 2019101300156,2019101300157 2019101300159,2019101300160 2019101300162,2019101300163 2019101300165,2019101300166 2019101300171,2019101300172 2019101300174,2019101300175 2019101300177,2019101300178 2019101300180,2019101300181 2019101300183,2019101300184 2019101300186,2019101300187 2019101300192,2019101300193 2019101300195,2019101300196 2019101300198,2019101300199 2019101300201,2019101300202 2019101300204,2019101300205 2019101300207,2019101300208 2019101300219,2019101300220 2019101300225,2019101300226 2019101300228,2019101300229 2019101300231,2019101300232 2019101300234,2019101300235 2019101300237,2019101300238 2019101300243,2019101300244 2019101300246,2019101300247 2019101300252,2019101300253 2019101300255,2019101300256 2019101300261,2019101300262 2019101300264,2019101300265 2019101300267,2019101300268 2019101300270,2019101300271 2019101300273,2019101300274 2019101300276,2019101300277 2019101300282,2019101300283 2019101300285,2019101300286 2019101300288,2019101300289 2019101300291,2019101300292 2019101300294,2019101300295 2019101300297,2019101300298 2019101300300,2019101300301 2019101300303,2019101300304 2019101300306,2019101300307 2019101300318,2019101300319 2019101300321,2019101300322 2019101300324,2019101300325 2019101300330,2019101300331 2019101300333,2019101300334 2019101300336,2019101300337 2019101300339,2019101300340 2019101300342,2019101300343 2019101300351,2019101300352 2019101300354,2019101300355 2019101300363,2019101300364 2019101300366,2019101300367 2019101300369,2019101300370 2019101300375,2019101300376 2019101300378,2019101300379 2019101300381,2019101300382 2019101300384,2019101300385 2019101300387,2019101300388 2019101300390,2019101300391 2019101300393,2019101300394 2019101300396,2019101300397 2019101300399,2019101300400 2019101300402,2019101300403 2019101300405,2019101300406 2019101300408,2019101300409 2019101300411,2019101300412 2019101300420,2019101300421 2019101300423,2019101300424 --clobber-config --clobber-versions -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 maxMeanSignal=100000 -c isr.doBias=False isr.doDark=False isr.doSuspect=False

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - 2). With respect to the other issue (raftName=R22 detectorName=S12 instead of detector=95), the following command ran to completion too (with the branch tickets/ DM-22227  of cp_pipe and obs_base, and w_2019_45 set up):    /measurePhotonTransferCurve.py /project/shared/BOT/ --rerun /project/plazas/test --id raftName=R22 detectorName=S12 --visit-pairs 2019101200233,2019101200234 2019101200236,2019101200237 2019101200239,2019101200240 2019101200242,2019101200243 2019101200279,2019101200280 2019101200290,2019101200291 2019101200293,2019101200294 2019101200299,2019101200300 2019101200305,2019101200306 2019101200311,2019101200312 2019101200314,2019101200315 2019101200317,2019101200318 2019101200323,2019101200324 2019101200326,2019101200327 2019101200329,2019101200330 2019101200332,2019101200333 2019101200341,2019101200355 2019101200358,2019101200359 2019101200361,2019101200362 2019101200364,2019101200365 2019101200370,2019101200371 2019101200373,2019101200374 2019101200376,2019101200377 2019101200379,2019101200380 2019101200382,2019101200383 2019101200385,2019101200386 2019101200394,2019101200395 2019101200408,2019101200409 2019101200411,2019101200412 2019101200417,2019101200418 2019101200423,2019101200424 2019101200426,2019101200427 2019101200429,2019101200430 2019101200432,2019101200433 2019101200435,2019101200436 2019101200438,2019101200439 2019101200441,2019101200442 2019101200444,2019101200445 2019101200450,2019101200451 2019101200453,2019101200454 2019101200456,2019101200457 2019101200459,2019101200460 2019101200462,2019101200463 2019101200465,2019101200466 2019101200468,2019101200469 2019101200471,2019101200472 2019101200474,2019101200475 2019101200480,2019101200481 2019101200483,2019101200484 2019101300003,2019101300004 2019101300006,2019101300007 2019101300009,2019101300010 2019101300012,2019101300013 2019101300015,2019101300016 2019101300018,2019101300019 2019101300021,2019101300022 2019101300024,2019101300025 2019101300027,2019101300028 2019101300030,2019101300031 2019101300036,2019101300037 2019101300039,2019101300040 2019101300042,2019101300043 2019101300045,2019101300046 2019101300048,2019101300049 2019101300051,2019101300052 2019101300057,2019101300058 2019101300060,2019101300061 2019101300063,2019101300064 2019101300069,2019101300070 2019101300078,2019101300079 2019101300081,2019101300082 2019101300084,2019101300085 2019101300087,2019101300088 2019101300090,2019101300091 2019101300093,2019101300094 2019101300096,2019101300097 2019101300099,2019101300100 2019101300102,2019101300103 2019101300111,2019101300112 2019101300114,2019101300115 2019101300117,2019101300118 2019101300120,2019101300121 2019101300123,2019101300124 2019101300126,2019101300127 2019101300132,2019101300133 2019101300135,2019101300136 2019101300138,2019101300139 2019101300141,2019101300142 2019101300144,2019101300145 2019101300147,2019101300148 2019101300150,2019101300151 2019101300153,2019101300154 2019101300156,2019101300157 2019101300159,2019101300160 2019101300162,2019101300163 2019101300165,2019101300166 2019101300171,2019101300172 2019101300174,2019101300175 2019101300177,2019101300178 2019101300180,2019101300181 2019101300183,2019101300184 2019101300186,2019101300187 2019101300192,2019101300193 2019101300195,2019101300196 2019101300198,2019101300199 2019101300201,2019101300202 2019101300204,2019101300205 2019101300207,2019101300208 2019101300219,2019101300220 2019101300225,2019101300226 2019101300228,2019101300229 2019101300231,2019101300232 2019101300234,2019101300235 2019101300237,2019101300238 2019101300243,2019101300244 2019101300246,2019101300247 2019101300252,2019101300253 2019101300255,2019101300256 2019101300261,2019101300262 2019101300264,2019101300265 2019101300267,2019101300268 2019101300270,2019101300271 2019101300273,2019101300274 2019101300276,2019101300277 2019101300282,2019101300283 2019101300285,2019101300286 2019101300288,2019101300289 2019101300291,2019101300292 2019101300294,2019101300295 2019101300297,2019101300298 2019101300300,2019101300301 2019101300303,2019101300304 2019101300306,2019101300307 2019101300318,2019101300319 2019101300321,2019101300322 2019101300324,2019101300325 2019101300330,2019101300331 2019101300333,2019101300334 2019101300336,2019101300337 2019101300339,2019101300340 2019101300342,2019101300343 2019101300351,2019101300352 2019101300354,2019101300355 2019101300363,2019101300364 2019101300366,2019101300367 2019101300369,2019101300370 2019101300375,2019101300376 2019101300378,2019101300379 2019101300381,2019101300382 2019101300384,2019101300385 2019101300387,2019101300388 2019101300390,2019101300391 2019101300393,2019101300394 2019101300396,2019101300397 2019101300399,2019101300400 2019101300402,2019101300403 2019101300405,2019101300406 2019101300408,2019101300409 2019101300411,2019101300412 2019101300420,2019101300421 2019101300423,2019101300424 --clobber-config --clobber-versions -c makePlots=True ptcFitType=POLYNOMIAL polynomialFitDegree=3 maxMeanSignal=100000 -c isr.doBias=False isr.doDark=False isr.doSuspect=False

            People

            • Assignee:
              rhl Robert Lupton
              Reporter:
              rhl Robert Lupton
              Reviewers:
              Andrés Alejandro Plazas Malagón
              Watchers:
              Andrés Alejandro Plazas Malagón, Robert Lupton
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel