Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23585

Test fake injection using the ap_verify_ci_hits2015 dataset.

    Details

      Description

      Create test fake sources for injection into the CI-HiTS2015 dataset. Use the Sophie Reed created tasks to insert said fakes into both the calexp and coadd. Reconfigure ImageDifference/ap_pipe to run over these fake inserted data. Make first check for results to see that the fakes are indeed processed. More detailed results tested on a later ticket.

        Attachments

          Activity

          Hide
          cmorrison Chris Morrison added a comment - - edited

          I was able to successfully inject fake sources using processCcdWithFakes.py and insertFakes.py into both the calexp and coadds of the CI-HiTS2015 dataset. The test ccdVisit that I ran was ccdnum=56, visit=411371.

          After running the ap_verify normally for this this ccdVisit, I created a set of uniformly spaced randoms at mag_g =21. I made them uniform so that they are easy to find by eye. There are two different difference image runs. In one we inject into the calexp only and difference against the regular coadd. In the other we inject the same sources into both the calexp and the coadd. The former is shown on the left while the later is shown on the right:

          You can see the regular pattern of positive sources in the left hand image while they are missing on the right, having been mostly subtracted in the diff. Zooming on on the right hand image shows that there are residuals of the injected sources.

          Finally, below is a plot of the DiaSources stored in the Apdb for three cases: a run with no fake injection (green), fakes injected into both (orange), fakes injected into the calexp only (blue). The points from the different samples are given different sizes to show when they overlap. The blue isolated blue points mostly show the injected point sources while the orange points mostly do not overlap with the injected points, having been mostly removed from in the difference.

           

          Currently this work and inputs to ap_pipe are done by manually editing the DecamMapper.yaml file to point to either the real or fake injected calexps and additionally moving the files containing the coadds to point to the real and injected data. This could be scripted up in Gen2 to be more automated or possibly through another method I'm not aware of. Gen3 should make this process more reproducible and easier given the ability to configure the input and output datasets.

          Show
          cmorrison Chris Morrison added a comment - - edited I was able to successfully inject fake sources using processCcdWithFakes.py and insertFakes.py into both the calexp and coadds of the CI-HiTS2015 dataset. The test ccdVisit that I ran was ccdnum=56, visit=411371. After running the ap_verify normally for this this ccdVisit, I created a set of uniformly spaced randoms at mag_g =21. I made them uniform so that they are easy to find by eye. There are two different difference image runs. In one we inject into the calexp only and difference against the regular coadd. In the other we inject the same sources into both the calexp and the coadd. The former is shown on the left while the later is shown on the right: You can see the regular pattern of positive sources in the left hand image while they are missing on the right, having been mostly subtracted in the diff. Zooming on on the right hand image shows that there are residuals of the injected sources. Finally, below is a plot of the DiaSources stored in the Apdb for three cases: a run with no fake injection (green), fakes injected into both (orange), fakes injected into the calexp only (blue). The points from the different samples are given different sizes to show when they overlap. The blue isolated blue points mostly show the injected point sources while the orange points mostly do not overlap with the injected points, having been mostly removed from in the difference.   Currently this work and inputs to ap_pipe are done by manually editing the DecamMapper.yaml file to point to either the real or fake injected calexps and additionally moving the files containing the coadds to point to the real and injected data. This could be scripted up in Gen2 to be more automated or possibly through another method I'm not aware of. Gen3 should make this process more reproducible and easier given the ability to configure the input and output datasets.
          Hide
          ebellm Eric Bellm added a comment -

          I'd just suggest attaching a log of the commands you used so someone could reproduce this.

          Show
          ebellm Eric Bellm added a comment - I'd just suggest attaching a log of the commands you used so someone could reproduce this.
          Hide
          cmorrison Chris Morrison added a comment -

          First I run ap_verify and output the results:

          ap_verify.py --dataset CI-HiTS2015 --output fakesRepo
          

          Second step is to run the processCcdWithFakes program:

          processCcdWithFakes.py fakesRepo/output/ --rerun testFakes --id visit=411420^411371^419802 tract=0 -C processCcdWithFakesConfig.py --clobber-config --clobber-versions

          The config file here points the program towards the correct fakes file.

          config.insertFakes.fakeType="/project/morriscb/src/ap_verify/testInsertFakes/starFakes.csv"
          

          From there we can re-run the ap_pipe.py processing if we edit the DecamMapper to reverse the calexp and fakes_calexp variables so that processing happens on the fakes.

          We can also at this point we can also insert the fakes into Coadds via the command:

          insertFakes.py fakesRepo/output --rerun testFakes --id tract=0 filter=g -c fakeType=/project/morriscb/src/ap_verify/testInsertFakes/starFakes.csv --clobber-config --clobber-versions
          

          In order to use these fakes within the test data, we have to copy them into the deepCoadd directly with the ap_verify_ci_hits2015 repository. There may a be a in code way to point at these, but the simplest way I found was doing this copy.

          From there we can run the ap_pipe.py processing, reusing the processCcd outputs since they are the ones with the fakes. Make sure to use make_apdb.py in ap_pipe to initialize an Apdb if you want to store the results in the apdb.

          ap_pipe.py fakesRepo/output --rerun testFakes --id visit=411420^411371^419802 -C apPipe.py --reuse-outputs-from "ccdProcessor" --clobber-config --clobber-versions
          

          You should have then produced fakes in both the calexp and Coadd and run through the full ap_processing.

          Show
          cmorrison Chris Morrison added a comment - First I run ap_verify and output the results: ap_verify.py --dataset CI-HiTS2015 --output fakesRepo Second step is to run the processCcdWithFakes program: processCcdWithFakes.py fakesRepo/output/ --rerun testFakes --id visit= 411420 ^ 411371 ^ 419802 tract= 0 -C processCcdWithFakesConfig.py --clobber-config --clobber-versions The config file here points the program towards the correct fakes file. config.insertFakes.fakeType= "/project/morriscb/src/ap_verify/testInsertFakes/starFakes.csv" From there we can re-run the ap_pipe.py processing if we edit the DecamMapper to reverse the calexp and fakes_calexp variables so that processing happens on the fakes. We can also at this point we can also insert the fakes into Coadds via the command: insertFakes.py fakesRepo/output --rerun testFakes --id tract= 0 filter=g -c fakeType=/project/morriscb/src/ap_verify/testInsertFakes/starFakes.csv --clobber-config --clobber-versions In order to use these fakes within the test data, we have to copy them into the deepCoadd directly with the ap_verify_ci_hits2015 repository. There may a be a in code way to point at these, but the simplest way I found was doing this copy. From there we can run the ap_pipe.py processing, reusing the processCcd outputs since they are the ones with the fakes. Make sure to use make_apdb.py in ap_pipe to initialize an Apdb if you want to store the results in the apdb. ap_pipe.py fakesRepo/output --rerun testFakes --id visit= 411420 ^ 411371 ^ 419802 -C apPipe.py --reuse-outputs-from "ccdProcessor" --clobber-config --clobber-versions You should have then produced fakes in both the calexp and Coadd and run through the full ap_processing.
          Hide
          cmorrison Chris Morrison added a comment -

          Ticket to script this up better so Eric Bellm can insert fakes himself and analyze them. DM-23631

          Show
          cmorrison Chris Morrison added a comment - Ticket to script this up better so Eric Bellm can insert fakes himself and analyze them.  DM-23631

            People

            • Assignee:
              cmorrison Chris Morrison
              Reporter:
              cmorrison Chris Morrison
              Reviewers:
              Eric Bellm
              Watchers:
              Chris Morrison, Eric Bellm, John Swinbank
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: