Is there an RFC that triggered this ticket?

No, this is planning for next cycle, not something that will be done soon.

For the reference of whoever eventually does end up doing this work: note the procedure for adding new packages, including the RFC process, in the developer docs.

RFC just created.

John Swinbank, could you please create a project for ngmix in lsst-dm so that I have a place to create a branch while I am waiting for the RFC process?

I missed this comment by Jim. I was intending to write a separate RFC for fitsio, but now I see there are several additional packages. We may not need them all. Won't know until we analyze what functionality requires which imports.

Show I create separate RFCs, or just one for all the required packages?

FYI, here are imports I find in ngmix which are in esheldon:

esutil, admom, images, covmatrix, mcmc, fitsio

Here are imports which are not in esheldon and are not (at least obviously) a part of Python or commonly setup python packages.

biggles,emshear,emcee,triangle,lensing,progressbar

Not sure yet what is really needed.

Let's put everything we need at first into one RFC if we can.

It may be a good idea to just email Erin to find out what features are enabled by the imports you've listed. esutil is already part of the stack, and only emcee is currently listed as a dependency in the ngmix readme (I and I happen to know that's for functionality we won't need at first), so I would not be surprised if we don't actually need anything else.

But the only way to know for sure is to ask Erin or actually try to build make the plugins and see what the code we want to use needs to import.

Perry Gee: I've created https://github.com/lsst-dm/ngmix for you. Please give me a shout if you need anything else (& remember to use JIRA's "@" syntax so I get notified that you're talking to me).

Here is Erin's response. At present I am unsure if we need to do either of these things.
--------------------------------------------
many of these packages are not necessary to use ngmix.

for normal usage you would not need any of them. If you wanted to fix
the covariance matrix for LMSimple you need covmatrix.

For mcmc runs using the affine invariant sampler you need emcee.
--------------------------------------------

I was able to run the simple examples while only importing fitsio, and I think that we can work around that requirement. So my suggestion is that we list minimal dependencies in the ups configuration for ngmix, and wait to see if we need more.

My conclusion for this round is that fitsio is not needed for basic usage. The observation types he uses can be constructed from numpy ndarrays.

There is also a little code in the persistance of mixtures which needs fitsio, but I think we can persist these ourselves when we use the code. We would need to read the fits file back ourselves and call the constructor of the mixture to do this.

Does that mean there are no third party packages required at all for now? What about statsmodel, which appears in the ticket description but hasn't been mentioned subsequently?

I don't exactly understand how this is supposed to work. When I imported python, it also brought in scipy and statsmodel, so I did not try to create a dependency on either package. But of course, if you were running your own python and didn't have these packages, you would be in trouble.

@jbosch, can you tell me how this dependency is supposed to be resolved?

Perry Gee: We need to create a package for every third party library that will be required to run ngmix in the stack.

That doesn't include scipy, since it's already packaged, but statsmodels isn't. Statsmodels also isn't included by default with Python: it may be pre-installed on your system, but that's just a coincidence: you can't assume the same applies to anybody else.

However, I do note that the statsmodels dependency is optional for ngmix: perhaps we don't actually need it. Can you check?

I use the Python included when you install anaconda with the Stack:

anaconda/2.2.0/pkgs/statsmodels-0.6.1-np19py27_0/lib/python2.7/site-packages

Is that not adequate? If not, do I need to create a statsmodels distributions as well?

You certainly can't assume that everybody else has also installed Anaconda in the same way, no. Otherwise, it wouldn't really be optional!

Some questions about third party package installation.

1. By default, my 3rd party eupspkg install untars and runs the setup.py in ngmix. That leaves the standard ngmix distribution directly under the Linux64/ngmix directory. It is a bit of a mess, with README.md and tests in the root. But I don't feel the need to clean it up after the install step is done. How do other 3rd party distributions handle this?

2. My eupspkg install step creates a python directory and transfers the ngmix build products to python/ngmix. So I then have 3 lsst directories in the root: python, ups, and upstream. I assume this is the easier way for LSST people to work with it. But I see that galsim installs under lib/python/galsim. Is there a best practice?

3. I don't understand from the eupspkg comments if I am supposed to use "create" or not. It is not discussed on the confluence page. I also don't get the tie to pkginfo, and whether I am supposed to assign the version number in pkginfo, or does that get assigned by eups?

Does that mean that I need to build a statsmodels distribution?

Erin only lists numpy as a non-optional requirement.

statsmodels: optional for importance sampling (multivariate student T distribution)
emcee: optional for doing MCMC fitting

I don't know enough about how we are going to use ngmix to know for sure what we need and what we don't. But the basic fits seem to run without it.

It sounds like we don't need statsmodels, then. Or are at least we won't need it until after we have emcee. My understanding is that Erin never really got the importance sampling working, so it's certainly not one of the first things we'll wrap.

I think that this install is consistent with the intended use of eupspkg, at least as far as how it gets prep'd and built.

I wanted to get install to do "python setup.py --home=.", which copies the build products to ./lib/python/ngmix. But I could not figure out how to get eupspkg to do that. If you know, please comment.

I am still unsure if I am supposed to assign VERSION in pkginfo or not.

The package is git@github.com:lsst-dm/ngmix.

Branch is u/pgee/DM-2253

Hey Perry,

The eupspkg stuff is frustratingly under-documented, but here are some pointers you might find useful:

• You don't need to create a pkginfo file, and hence no need to worry about what version number to put in it.
• eupspkg already knows how to install Python packages using the standard setup.py mechanism. From a very quick look, that seems to be all that ngmix uses. I don't think you need to create a eupspkg.cfg.sh file at all, or to add any special instructions to copy files around. Maybe I've misunderstood what you're trying to achieve, though, so do explain further if I'm incorrect.
• The .cfg file is used to tell SCons which libraries are needed to compile and link C++ code. However, you're not using SCons for this purpose – the compilation is all handled by distutils driven by setup.py – so I don't think you need the ngmix.cfg file either.

I just tested, and I'm able to build and install your package using only the contents of the upstream directory and the ups/ngmix.table file: I think everything else is unnecessary.

Perhaps I chose the wrong path for testing the package. I did the sequence of eupspkg create, fetch, prep, config, build, install because I thought that was what the eups distrib command would do. And I found that I needed pkginfo, so I supplied it. If this is not the correct way to test installation, please advise. BTW, I also noticed that GalSim had an SHA1 included, so I supplied one even though I wasn't sure if it was needed.

About the eupspkg.cfg.sh, the one step I couldn't figure out is how to get setup.py to install where I wanted it to. I really wanted to install either to lib/python or python, but I ended up using eupspkg.cfg.sh to copy to the desired location. I know that this isn't right, which is why I asked about this? How do I tell setup.py, driven by distutils, to copy the build results to the right directory? When I ran it manually, it wanted to install to the dist-packages of whatever was the default Python installation.

John Swinbank is correct in that if you let eupspkg do everything itself then it should all work fine. You can see that the other Python packages work like that (see pyfits and stsci_distutils. They end up with Python files installed into lib/python (See the table files).

Perry I remember getting a pkginfo problem as well, but that was related to not having it committed to a git repo, as the sha is what eupspkg uses to build version identities and such when doing test builds. Others have given you feedback here, but I will give you more when I have time to look at it in more detail. I suspect they are right though, that you will only need to put the tar in and add the ups info and all the rest of it will just work.

I'm sure that Tim and John know what they are talking about. I think that the problem is that using eupspkg prep standalone is not the right test. Because pyfits also asks to have the PRODUCT, VERSION, and FLAVOR defined, which is why I defined pkginfo in the first place. Let's try it in the 'eups dist install' environment, and see if that is the issue.

I can also get the setup.py in ngmix to install to lib/python, just not when I run 'eupspkg install'. So clearly there is a general lack of knowledge on my part about how eupspkg works.

You need eupspkg -e prep if you are doing a quick test yourself (once you've committed everything to git).

Hey Perry,

I pushed some suggested changes to the u/swinbank/DM-2253 branch. It seems to work for me, and I hope it answers some of your questions:

$ setup numpy

$ git clone git@github.com:lsst-dm/ngmix.git

  [ .. elided .. ]

$ cd ngmix

$ git checkout u/swinbank/DM-2253

$ eupspkg -er prep

eupspkg.prep: unpacking upstream/ngmix-25d1ca4.tar.gz ...

$ eupspkg -er config

$ eupspkg -er build

running build

running build_py

  [ .. elided .. ]

cc -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db48 build/temp.macosx-10.11-x86_64-2.7/ngmix/_gmix.o -o build/lib.macosx-10.11-x86_64-2.7/ngmix/_gmix.so

$ eupspkg -er install

running install

running build

  [ .. elided .. ]

Writing /Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/ngmix/u.swinbank.DM-2253-gafb404fcf3/lib/python/ngmix-0.1-py2.7.egg-info

$ eupspkg -er decl

eupspkg.decl: declared ngmix u.swinbank.DM-2253-gafb404fcf3 in /Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/ngmix/u.swinbank.DM-2253-gafb404fcf3 (eups declare options: none)

I'm not having any trouble getting it to build and install. But where is the runnable Python directory ending up? And does it match the PYTHONPATH?

I don't really get what -r is supposed to do, although it does change things. The directory which gets installed to is u.swinbank.DM-2253-gafb404fcf3/lib/python, so perhaps if I were on master, it would be lib/python?

Is it OK if I push these changes to master on LSST-DM, so I can try it?

Would be nice if -r were better doc'd.

Looking in the source, the best description of -r I could find is:

• for all other verbs:
  + the PREFIX (the location to which the package
  will be installed to) will be set to ./_eupspkg/binary. This can
  be overridden with -r switch, in which case the PREFIX will be
  set to EUPS' root (i.e., where it would ordinarily be
  installed).

Note that you don't want things to be installed into lib/python: you want them to be installed into your EUPS product stack, which is exactly what's happening in the example I send above: note that the first part of the path has the flavour (DarwinX86) etc specified as usual.

And yes, I can confirm that when you do this your PYTHONPATH is properly set up.

Thanks, John. That makes more sense. I was unclear about the install destination, and assumed that a further copy was still required.

John, since you now seem to have this ticket in hand, do you want to take over moving it to the DM stack?

Perry Gee – Good idea. Thanks!

Nate Lust – I'll move my user branch to tickets/DM-2253 shortly, perform some minor cleanups, and put it back in review to you.

Note that my understanding of the above discussion is that we are only packaging ngmix here. If anything else is needed, Perry Gee or Jim Bosch, who are much more familiar with the code than I am, should speak up now.

My understanding is that ngmix is indeed the only one we need right now.

Nate Lust – all yours on tickets/DM-2253. Thanks!

Looks good

Merged. Broken. Fixing on DM-5867.