Joshua Hoblitt added a comment - 19/Feb/19 11:55 AM How does git fetch -p result in the local checkout of a branch being removed? Consider this simple test: #!/bin/bash   set -e   mkdir gittest cd gittest/ # simulate a canonical remote repo ( mkdir canonical_repo cd canonical_repo git init --bare ) # add a couple of branches to said repo ( git clone canonical_repo user_clone cd user_clone touch a_file git add a_file git commit -m "a commit" git push origin master git checkout -b foo touch anotherfile git add anotherfile git commit -m "another commit" git push origin foo ) # lsst-build like checkout of foo branch ( git clone canonical_repo build_clone -b foo cd build_clone git branch -a ) # rm foo branch from canoncial_repo ( cd user_clone git push origin :foo ) # lsst-build like fetch -p ( cd build_clone git fetch -p git branch -a ) This simple example results in a repo that still has a local branch named foo : * foo remotes/origin/HEAD -> origin/master remotes/origin/master

Gabriele Comoretto [X] (Inactive) added a comment - 19/Feb/19 12:19 PM Joshua Hoblitt , this does not correspond with what in lsst_build tickets/ DM-3236 branch, where I keep both git fetch:           # update from origin         if not self.no_fetch:             # the line below should be equivalent to:             #     git.fetch("origin", "- force", " -prune")             #     git.fetch("origin", "- force", " -tags")             # but avoids the overhead of two (possibly remote) git calls.             git.fetch("-fup", "origin", "+refs/heads/ :refs/heads/ ", "refs/tags/ :refs/tags/ ")             git.fetch("-p")   I know, is not optimal, but this works, as demonstrated above  

Gabriele Comoretto [X] (Inactive) added a comment - 19/Feb/19 2:02 PM PR created: https://github.com/lsst/lsst_build/pull/27 It is probably more easy to discuss on the actual changes done in Git.

Joshua Hoblitt added a comment - 19/Feb/19 3:51 PM - edited I think I've convinced myself that this works but it is a bit scary. Consider this Dockerfile : FROM lsstsqre/newinstall   ARG LSST_BUILD_URL=https: //github.com/lsst/lsst_build ARG LSST_BUILD_REF=tickets/DM- 3236   USER root   # install modern git needed by pkgautoversion RUN yum install -y rh-git29 RUN echo ". /opt/rh/rh-git29/enable" > "/etc/profile.d/rh-git29.sh" RUN git config --global user.email "you@example.com" && \ git config --global user.name "Your Name"   # source loadLSST.bash by default for python + eups RUN echo ". $(pwd)/loadLSST.bash" > "/etc/profile.d/loadLSST.sh"   # install lsst-build RUN git clone "$LSST_BUILD_URL" -b "$LSST_BUILD_REF" RUN cd lsst_build && python setup.py install   # create bare repo testpk1.git with master & feature1 RUN mkdir -p testpkg1 && \ cd testpkg1 && \ git init && \ mkdir ups && \ touch ups/testpkg1.table && \ git add ups/testpkg1.table && \ git commit -m "fwv" && \ git checkout -b feature1 && \ touch feature1 && \ git add feature1 && \ git commit -m "add feature1" && \ git checkout master RUN mv testpkg1/.git testpkg1.git && \ rm -rf testpkg1 && \ cd testpkg1.git && \ git config --bool core.bare true RUN echo "testpkg1: $(pwd)/testpkg1.git" > repos.yaml   # create testpkg2 which depends on testpkg1 RUN mkdir -p testpkg2 && \ cd testpkg2 && \ git init && \ mkdir ups && \ echo "setupRequired(testpkg1)" > ups/testpkg2.table && \ git add ups/testpkg2.table && \ git commit -m "fwv" && \ git checkout -b feature1 && \ touch feature1 && \ git add feature1 && \ git commit -m "add feature1" && \ git checkout master RUN mv testpkg2/.git testpkg2.git && \ rm -rf testpkg2 && \ cd testpkg2.git && \ git config --bool core.bare true RUN echo "testpkg2: $(pwd)/testpkg2.git" >> repos.yaml   # lsst-build "build" dir RUN mkdir build   # prepare feature1 branch RUN lsst-build prepare --repos= "$(pwd)/repos.yaml" "$(pwd)/build" testpkg2 \ --ref feature1   # show clone branch state RUN cd build/testpkg1 && \ git branch -a RUN cd build/testpkg2 && \ git branch -a   # rm feature1 branch from testpkg1 RUN git clone testpkg1.git testpkg1 && \ cd testpkg1 && \ git push origin :feature1 && \ rm -rf testpkg1   # prepare feature1 branch RUN lsst-build prepare --repos= "$(pwd)/repos.yaml" "$(pwd)/build" testpkg2 \ --ref feature1   # show clone branch state RUN cd build/testpkg1 && \ git branch -a RUN cd build/testpkg2 && \ git branch -a The tail end of the stdout is: $ docker build . --no-cache ... Step 19/25 : RUN lsst-build prepare --repos="$(pwd)/repos.yaml" "$(pwd)/build" testpkg2 --ref feature1 ---> Running in fa9099c2d558 testpkg2: ok (0.1 sec). testpkg1: ok (0.1 sec). Removing intermediate container fa9099c2d558 ---> c9565612a26f Step 20/25 : RUN cd build/testpkg1 && git branch -a ---> Running in a26ab1a987f2 * feature1 master remotes/origin/HEAD -> origin/master remotes/origin/feature1 remotes/origin/master Removing intermediate container a26ab1a987f2 ---> 4c39ee2ab99a Step 21/25 : RUN cd build/testpkg2 && git branch -a ---> Running in 434fb449b3dd * feature1 master remotes/origin/HEAD -> origin/master remotes/origin/feature1 remotes/origin/master Removing intermediate container 434fb449b3dd ---> 841f3ac618da Step 22/25 : RUN git clone testpkg1.git testpkg1 && cd testpkg1 && git push origin :feature1 && rm -rf testpkg1 ---> Running in 937156873bd7 Cloning into 'testpkg1'... done. To /opt/lsst/software/stack/testpkg1.git - [deleted] feature1 Removing intermediate container 937156873bd7 ---> 89e11053c5d3 Step 23/25 : RUN lsst-build prepare --repos="$(pwd)/repos.yaml" "$(pwd)/build" testpkg2 --ref feature1 ---> Running in 979b9ba1df7b testpkg2: ok (0.0 sec). testpkg1: ok (0.0 sec). Removing intermediate container 979b9ba1df7b ---> fa1340a7f880 Step 24/25 : RUN cd build/testpkg1 && git branch -a ---> Running in b1a759c57cca * master remotes/origin/HEAD -> origin/master remotes/origin/master Removing intermediate container b1a759c57cca ---> cd2653ea59c0 Step 25/25 : RUN cd build/testpkg2 && git branch -a ---> Running in 7500e6b0834e * feature1 master remotes/origin/HEAD -> origin/master remotes/origin/feature1 remotes/origin/master Removing intermediate container 7500e6b0834e ---> 557848e3ba06 Successfully built 557848e3ba06 What appears to be happening is that the second invocation of git fetch -p is leaving the repo with a broken HEAD – I'm wondering if this is legitimately a bug. $ cat .git/HEAD ref: refs/heads/feature1 $ cat .git/refs/heads/feature1 cat: .git/refs/heads/feature1: No such file or directory I think this is probably acceptable, at least in the context of lsstsw where the git version in pinned, but really should have verbose explanation of the problem being solved and why it works as a code comment. An acceptance test that runs under travis is also needed specifically to catch if this breaks with future versions of git .

Joshua Hoblitt added a comment - 19/Feb/19 3:53 PM I've asked for the changes described in my previous comment on the gh pr.