Can you attach the outputs of the tests/.tests/*.failed files?

ptr.failed
tests/ptr

dyld: Library not loaded: libbase.dylib
Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/ptr
Reason: image not found
-------
testModuleImporter1.failed
tests/testModuleImporter1

dyld: Library not loaded: libbase.dylib
Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporter1
Reason: image not found
-------
testModuleImporter2.py.failed
tests/testModuleImporter2.py

Traceback (most recent call last):
File "tests/testModuleImporter2.py", line 25, in <module>
import testModuleImporterLib
File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 28, in <module>
_testModuleImporterLib = swig_import_helper()
File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 24, in swig_import_helper
_mod = imp.load_module('_testModuleImporterLib', fp, pathname, description)
ImportError: dlopen(/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so, 2): Library not loaded: libbase.dylib
Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so
Reason: image not found
-------

Looks like something has changed in how compiled Python modules need to be built; I think these tests are the first ones in the dependency tree that try to import a compiled module we build ourselves.

Not sure if this helps at all or not, but I'll add this anyway. I went into /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14, did a "setup -r ." and then executed the failed tests:

srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ tests/ptr

srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ tests/testModuleImporter1

ModuleImporter test succeeded.

srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ tests/testModuleImporter2.py

Traceback (most recent call last):

  File "tests/testModuleImporter2.py", line 25, in <module>

    import testModuleImporterLib

  File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 28, in <module>

    _testModuleImporterLib = swig_import_helper()

  File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 24, in swig_import_helper

    _mod = imp.load_module('_testModuleImporterLib', fp, pathname, description)

ImportError: dlopen(/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so, 2): Library not loaded: libbase.dylib

  Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so

  Reason: image not found

SQuaRE takes the ticket but we don't have El Capitain yet so it won't be imminent.

In investigating this further this afternoon, El Capitan disables the inclusion of LD_LIBRARY_PATH and DYLD_LIBRARY_PATH when a new shell is invoked. (This was reported by other developers on other projects). I ran a quick test:

$ cat boo.sh

#!/bin/bash

tests/ptr

$ ./boo.sh

dyld: Library not loaded: libbase.dylib

  Referenced from: /Users/srp/stack/EupsBuildDir/DarwinX86/base-11.0/base-11.0/tests/ptr

  Reason: image not found

./boo.sh: line 2: 47242 Trace/BPT trap: 5       tests/ptr

$ cat foo.sh

#!/bin/bash

export DYLD_LIBRARY_PATH=$PWD/lib

tests/ptr

$ ./foo.sh

$ 

Steve Pietrowicz - thanks, very helpful. From looking around, did you get an idea of whether this is a bug or deliberate behaviour (i.e. some kind of new security sandboxing feature?)

Seems to be a deliberate security feature. If you reboot, twiddle with your boot prom and disable the new security features you can use the stack. That doesn't seem to be a good solution though. Some thought is required. If you run a binary that binary does inherit the library path, if you run something with a #! the library path is sanitized.

Yes, definitely deliberate. For the python programs, using "python whatever.py" works. The "#!/usr/bin/env python" at the top is broken, so no more just invoking "whatever.py".

OS X 10.11.1 beta 2 is already out for developers. I haven't looked it yet to see if they're reverted the behavior, but I suspect they haven't and won't.

According to the System Integrity Protection documentation this is a feature, not a bug: I don't think there's any serious possibility that it will be reverted.

However. To quote that documentation:

When a process is started, the kernel checks to see whether the main executable is protected on disk or is signed with an special system entitlement. If either is true, then a flag is set to denote that it is protected against modification...
Any dynamic linker (dyld) environment variables, such as DYLD_LIBRARY_PATH, are purged when launching protected processes.

(my emphasis). It looks as though Steve Pietrowicz has demonstrated above that /bin/bash is a protected process. However, I'm speculating (I don't have a 10.11 system to check) that end-user installed shells, Python interpreters, etc are not. Worth checking.

John Swinbank is correct, but it's a little complicated because it depends on how that interpreter is invoked:

$ cat test.py

#!/usr/bin/env python

import os, sys

for i in os.environ:

    if i == "DYLD_LIBRARY_PATH":

        print "DYLD_LIBRARY_PATH found"

        sys.exit(0)

print "DYLD_LIBRARY_PATH not found"

$ python test.py

DYLD_LIBRARY_PATH found

$ ./test.py

DYLD_LIBRARY_PATH not found

But if you hardcode the path to python, you get different results.

$ cat test2.py

#!/Users/srp/stack/DarwinX86/anaconda/2.2.0/bin/python

import os, sys

for i in os.environ:

    if i == "DYLD_LIBRARY_PATH":

        print "DYLD_LIBRARY_PATH found"

        sys.exit(0)

print "DYLD_LIBRARY_PATH not found"

$ python test2.py

DYLD_LIBRARY_PATH found

$ ./test2.py

DYLD_LIBRARY_PATH found

$

It's not python that's blocking it — it's env.

It's actually the location of env.

If you copy env to /usr/local/bin/env in change to #!/usr/local/bin/env python

it works.

I believe we're not going to be able to make Python 2.7 an optional install from now on. On 10.11.x Python 2.7 is installed in /usr/bin, and since it's in that location, it doesn't get passed DYLD_LIBRARY_PATH. (Version is 2.7.10)

I updated the summary title of this ticket to reflect the reality of El Capitan not working regardless of beta status.

I've finally had a chance to look at this and I propose the following (which I have proof-of-concepted on El Capitan):

• Modify the scons build so that the path to the selected python is burned in to the shebang.
• Change the test driver in sconsUtils to set DYLD_LIBRARY_PATH in the same line of shell script that runs the test. This is made a bit more complicated by the inability to simply set the environment variable in scons and have it appear in the subprocess. (similar to how DM-3856 was implemented).
• If we do not want to modify scons we have to get the library path in through another mechanism: I suggest we just add a line to all the EUPS table files to define a LSST_LIBRARY_PATH and then copy that for test execution.
• Executable python scripts will need to have the python path baked in rather than using env. People didn't like the idea of using a autoconf-style .in suffix for the pre-modified file so I suggest we put them in a scripts directory as is, and process each file in their to the bin directory (which conceptually is little different to how the lib directory is populated at the moment during a build.

It is important to realize that none of this works if people try to build their stack with the system OS X python!

I have some sconsUtils patches that are sufficient to handle the test target (and only trigger for El Capitan), the shebang processing will need to be added and the rest is just moving things around and (possibly) tweaking all the table files.

Kian-Tat Lim Russell Owen I've made initial changes for El Capitan support. The remaining changes are all rote moves of executables to bin.src and modifications of the table files. I think it would make sense to get these changes reviewed before I embark on the bulk changes. After that I can incrementally do the changes and merges without further review. Changing 40 packages and then being told that I've done it wrong is not where I want to go.

This looks like a good approach to me. In the long run I hope we can move away from using environment variables to setup packages. But we don't have time for that right now, and your solution looks like a good fix that we can have now.

I like the way you are handling python files in bin/ and tests/. For bin/ rewriting the #! line makes sense, and I am grateful to you for not insisting on .in files. For tests/ the savings of not having to modify the files outweighs the minor inconvenience of making the tests not runnable.

Just an FYI, Xcode 7.1 was released yesterday, and 10.11.1 a few days before that. Won't change what's needed to be done to fix the stack, but I thought I'd mention it.

John, would you be able to have a quick look at this? There are 3 parts to this work:

1. Modify sconsUtils
2. Add the new LSST_LIBRARY_PATH variable to EUPS table files everywhere
3. Move all the .py executables in bin directories to bin.src

I have examples of all these in branches linked from this ticket but I'd like to get the all clear before I do the big push for steps 2 and 3 above. The main item for review is the sconsUtils changes that I will merge before starting on incrementally merging each package in turn (it will be much quicker if I can merge as I go without having to get each bit reviewed and then having to rebase everything).

Kian-Tat Lim and Russell Owen have agreed to the approach.

I am happy – indeed, keen – to look, but it'll probably take me a couple of days (I want to get another WBS-and-a-half of planning for the DMLT in first). If you're happy to wait for that, then this is on my todo list.

A couple of days is fine.

My review of the changes in sconsUtils (I'm working may way through the packages one at a time).

scripts.py: defaultTargets=("lib", "python", "tests", "examples", "doc", "shebang") is used in two places and had to be modified in two places. I suggest the default value be made a module-level constant.

scripts.py: if src is None then it appears that rewrite_shebang tries to re-match the first line of every file, even binary executables. This worries me for several reasons:

• No warning is logged if it runs on a python file and there is no match for the shebang line
• It seems odd and a bit unwise to read a binary executable in a line-oriented fashion
  I suggest if the file is a .py file then run the match-and-change code and warn if no match and use shutil.copyfile on all non-python files.

Several files use from X import *, where X is "eups" or "Scons.Script". Please either fix or file a ticket. These is forbidden and prevents pyflakes from running properly. Please fix or file a ticket. I am offering to do the work.

I appreciate many of the small cleanups you made, such as removing silly line breaks such as {{if }} and expanding one-line conditionals into two lines. Nonetheless, I am a bit grumpy about the many changes to spacing, such as:

• Changing {{ = }} to = in argument lists. Yes the lack of spaces is recommended in argument lists, but {{ = }} is allowed and is usually more readable unless there are many arguments, so this change made the code harder to read, in my opinion.
• Adding an extra space before the # in in-line comments.
• Wrapping a line that was already less than 110 chars
  These changes made the differences harder to understand without materially enhancing readability (and in the first case, degrading it, in my opinion). I suggest undoing those changes before committing.

installation.py: please consider removing locks = on line 338. The variable is not used, which needlessly upsets pyflakes.

The following minor annoyances are probably out of scope for this ticket, but if so, I request a new low-priority ticket:

Many files use % formatting with a bare object as the right hand side, instead of a tuple. This works but is mildly dangerous. In some cases the object is enclosed in paranthesis, but of course that has no effect without a comma after the object.priority ticket.

The doc strings are non-standard and so will not be seen by python's "help" function.

The other packages look great. So...overall it's fine. I had one or two substantive requested changes and some picky requests. If you want help with any of it let me know.

Thanks for the review.

• I will have a look at removing that defaultTargets duplication. I didn't put that code there to start with so I was trying not to change unrelated logic in this ticket.
• bin.src != bin. I am only expecting items to be placed in bin.src that need to have their shebang rewritten. It is a bug to put a binary executable in the bin.src directory.
• I'll see about adding a warning if no shebang is found. If a file is in bin.src it is there to be modified and if it is not being modified that is a bug. I should probably trigger that warning even when not on El Capitan.
• I do not want to worry about changing the import semantics in this ticket. I will file a ticket indicating the "import *" issue.
• Regarding the PEP-8 clean ups:
  • I am inclined to stick with the removal of spaces in keyword arguments (I don't think the LSST coding standard expresses an opinion on keyword arguments other than when defining defaults).
  • Two spaces for inline comment is the PEP-8 standard and the LSST coding standard.
  • I did not intend to wrap a line that didn't need wrapping. Scanning through the patch it's not entirely obvious to me which line is being referred to.
• Regarding readability of the changes, the reformatting was isolated to a single commit and the rest of the commits are nicely self-contained. I think this is one thing where GitHub PRs get in the way of clarity as they prefer you to view the entire change set as one item. I guess they expect a separate cleanup ticket. I can create an entirely new ticket for the sconsUtils PEP-8 work if people prefer that approach. @ktl ?

Fixing formatting with % is out of scope of this ticket.

The doc strings are documented to be non-standard. The readme file explicitly states that they are set up for doxygen because it is impossible to import this file outside of the context of scons and therefore help() will never work (and pydoc doesn't work either because pydoc relies on the ability to import the code). sconsUtils might challenge Jonathan Sick.

Thank you for the explanations.

The extra line break is in scripts.py at line 394:

    state.env.SwigLoadableModule("_" + name, src,

                                 LIBS=state.env.getLibs("main python"))

I'm glad to know that the doc strings cannot be found, but I still think using standard doc strings would make the code easier to read. And Doxygen will parse standard doc strings provided they begin with """!. But that is clearly out of scope for this ticket.

Regarding processing files in bin.src. I had not realize that only python files were even expected to appear in this directory, but I'm glad you will add a warning. A code comment might clarify things a bit. As I recall you prefer that python executables not a ".py" suffix, so the obvious filter won't work. I do wonder what other sort of executables we can expect. C++ and shell scripts both seem plausible. Other languages seem less likely in the DM stack, though I wonder if java can be expected in orchestration. Do we need to munge the shebang for shell scripts?

We might need to be able to spot other shebangs but I'll cross that bridge when we come across the problem. Shell scripts may be a bit of a problem as /bin/sh will not allow DYLD_LIBRARY_PATH to be passed in.

I do think the documentation in this package will need to be revisited. I'll save that for Jonathan Sick.

No plans that I'm aware of for Java in orchestration. The only thing in the github DM structure that I'm aware of that uses Java is the complex event processing software in ctrl_evmon, and that's not currently in use or even being distributed.

Closing this now that lsst_distrib builds on El Capitan. I will write up a tech note covering the details.

Qserv does not build because of issues with partition and qserv.

Review of technical note http://dmtn-001.lsst.codes/en/master/

This is a very clear and helpful document. Nonetheless, I do have a few suggested changes.

"The Summer 2015 version of the LSST software stack does not work with OS X El Capitan". That is true, but I encourage you to highlight the fact that no version of the stack up to and including Summer 2015. Also, the incompatibility extends beyond that point, and I am wondering if we can explicitly tag a new release soon that is compatible with El Capitan, to give a clearer marker, e.g. Winter 2016. In which case you could say something like "OS X 10.11 (El Capitan) is not compatible with any version of the LSST software stack prior to Winter 2016".

"scons runs the tests and so the tests do not have the correct environment; applications such as processCcd.py will also not function properly." I found this slightly confusing. I suggest you break it into two sentences and explain what you mean by "applications" or even use a different term (I don't think of our bin scripts as applications, but I am not sure what our official documentation calls them), e.g. "Executables in <package>/bin, such as processCcd.py, will also not function properly."

"2. /usr/bin/env can no longer be used to run python scripts from the command line." I suggest omitting the word "python" because this applies to any executable found with /usr/bin/env. I admit that will make the rest of the paragraph a bit messier, as you then need to point out that we only fix python scripts in bin.src because that is all we have, so far. I think it is worth the work to increase clarity and highlight a possible need to modify other kinds of scripts in the future.

"One additional complication on El Capitan is that Apple no longer distribute" is the start of an interesting paragraph. Clearly you view "Apple" as a collective noun, whereas I do not. This may be a British vs. American viewpoint or I may just be exposing my ignorance. Perhaps consider wording that makes the distinction less obvious, such as: "One additional complication on El Capitan is that Apple no longer distributes OpenSSL include files. Apple deprecated the use of OpenSSL in OS X 10.7 (Lion) and removed the include files in El Capitan (though the libraries remain for binary compatibility)." (where I used "distributes" but you would probably prefer "distribute").

In addition, it would help to briefly explain why we felt we could get away with simply removing SSL support. The tickets you refer to later explain this, but a brief note here would be comforting, e.g. "We did not need SSL support in those packages (see DM-4327 and DM-4334) so we disabled it".

I do not know what this means: "Executable shell scripts should ensure they run setup rather than relying on the setup of the parent shell." Could you please say more, or provide an example?

"If a package requires OpenSSL consider optionally using Apple CommonCrypto" I suggest a comma after OpenSSL. Also, I had to read it twice to understand it. I think by "optionally using Apple CommonCrypto" you mean as an alternative to OpenSSL, rather than making Apple CommonCrypto optional in the sense that it can be omitted. Perhaps it would be clearer to say "If a package requires OpenSSL, consider supporting Apple CommonCrypto as an alternative" or "If a package requires SSL, consider supporting both Apple CommonCrypto and OpenSSL".

The hyphen in "pre-requisite" is unnecessary.