Ran both the current stack default and the new plugin system over two ccds (24 and 24) in the full HiTS2015 dataset. The goal is to test the relative processing speed of the new plugin system versus the previous implementation.
Attached are two dot plots showing the percent of time spent in each part of the code during ap_pipe. In the default setup, the function `update_dia_objects` takes up 9.49% of the total processing time while the new plugin system takes up 6.46%, a 30% improvement.
The function `update_dia_objects` includes both loading of DiaSources and storage of DiaObjects both of which have not changed and represent the same amount of overhead. The methods `_set_flux_stats` in the previous implementation and `diaCalculation.run` represent almost the same process with `diaCalculation.run` having slightly more calculations (updating mean positions, HTM indices, etc.). Even with these differences, `diaCalculation` is 5.79% of the processing vs `_set_flux_stats` taking 7.83%, again about a 30% reduction.
The reason for this is possibly two fold: First the new plugins use the filter name as an index in the Pandas dataframe instead of creating a filter mask for each set of DiaSources. Second, a bug was discovered in the Stetson Mean calculation that could cause extra iterations in this step.
Last test will be to check that the Ppdb has proper values within it.
w/ DiaCalculation plugins: