Upgrading to nixpkgs master

[nh2]

The latest nixpkgs master has all fixes I had in my nixpkgs commit 50677e464359947f2a71cfd497c4022e3cdf8c7d.

So I'd like to just switch to a nixpkgs master commit.

[nh2]

Trying nh2/nixpkgs@efb033a, I get an issue compiling e2fsprogs because apparently the musl-based GCC doesn't understand __GNUC_PREREQ:

niklas@ares:/tmp/nix-build-e2fsprogs-1.44.4.drv-0/e2fsprogs-1.44.4/lib/e2p$ gcc -I. -I../../lib -I../../lib -I../../intl -I../../intl -g -O2 -DHAVE_CONFIG_H -c feature.c -o feature.o
In file included from ../../lib/ext2fs/ext2fs.h:97:0,
                 from feature.c:20:
../../lib/ext2fs/hashmap.h:20:19: error: missing binary operator before token "("
 #if __GNUC_PREREQ (4, 8)
                   ^
../../lib/ext2fs/hashmap.h:25:19: error: missing binary operator before token "("
 #if __GNUC_PREREQ (4, 8)
                   ^

My system gcc understands it:

niklas@ares:/tmp/nix-build-e2fsprogs-1.44.4.drv-0/e2fsprogs-1.44.4/lib/e2p$ /usr/bin/gcc -I. -I../../lib -I../../lib -I../../intl -I../../intl -g -O2 -DHAVE_CONFIG_H -c feature.c -o feature.o

CC @dtzWill

[nh2]

gcc -I. -I../../lib -I../../lib -I../../intl -I../../intl -g -O2 -DHAVE_CONFIG_H -c feature.c -o feature.o in the build environment reproduces the issue immediately.

[nh2]

Pretty sure the reason is the upgrade to musl 1.1.20 (NixOS/nixpkgs#46064); the commit NixOS/nixpkgs@e37c21c itself fails with that message but the commit before it, NixOS/nixpkgs@88ae8f7, doesn't.

[dtzWill]

Oh, I don't see this locally because I'm using the fix I committed to staging that addresses this :P.

Sorry about that. Can you try using that patch (however works best for you) and LMK if that fixes the issue for you (especially if it doesn't!).

This PR: NixOS/nixpkgs#45703

[nh2]

@dtzWill I see. Why did it work before though? Why does the musl update make this difference?

[nh2]

On NixOS/nixpkgs@88ae8f7 I also get a Cython build failure due to failing tests (when building sphinx, a dependency of servant):

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1202, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 11

----------------------------------------------------------------------
Ran 2589 tests in 938.163s

FAILED (errors=1)
Following tests excluded because of missing dependencies on your system:
   Cython.Coverage
   run.asyncio_generators
   run.coverage_api
   run.coverage_cmd
   run.coverage_nogil
   run.numpy_pythran
   run.py35_asyncio_async_def

Note this takes quite long to run: 3066.69s user 191.27s system 220% cpu 24:36.07 total

[nh2]

Perhaps related: https://github.com/NixOS/nixpkgs/blob/288f9c648f2e145f6af913b88eba0696aa288df7/pkgs/development/python-modules/Cython/default.nix#L15-L24

[dtzWill]

Don't suppose bumping to latest cython fixes things? O:)

Given the comments in the nix expression you linked disabling tests doesn't seem so bad BUT it'd be good to check the reasons for the failures first... :/.

[nh2]

I can reproduce it in the build environment; it looks a bit cleaner there:

niklas@ares:/tmp/nix-build-python2.7-Cython-0.28.3.drv-0/Cython-0.28.3$ HOME="$NIX_BUILD_TOP" /nix/store/z722f9n43493n99xwjr545s0h39x2b33-python-2.7.15/bin/python2.7 runtests.py -j1

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (c) and running longlongindex
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1195, in run_forked_test
    cid, result_code = os.waitpid(child_id, 0)
OSError: [Errno 4] Interrupted system call

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1202, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 139

----------------------------------------------------------------------
Ran 10798 tests in 2871.950s

FAILED (errors=2, skipped=50)
Following tests excluded because of missing dependencies on your system:
   Cython.Coverage
   run.asyncio_generators
   run.coverage_api
   run.coverage_cmd
   run.coverage_nogil
   run.numpy_pythran
   run.py35_asyncio_async_def

ALL DONE

I get 2 failures here though, and now the signal number is different.

[nh2]

The longlongindex fails nondeterministically.

The other one can be reproduced quickly like this (the last CLI argument is a match on a test name):

niklas@ares:/tmp/nix-build-python2.7-Cython-0.28.3.drv-0/Cython-0.28.3$ HOME="$NIX_BUILD_TOP" /usr/bin/time /nix/store/z722f9n43493n99xwjr545s0h39x2b33-python-2.7.15/bin/python2.7 runtests.py -j1 --verbose parallel 
...
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1202, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 139

----------------------------------------------------------------------
Ran 110 tests in 45.450s

That's much quicker than waiting 40 minutes for all the tests.

[nh2]

Workaround for the Cython issue: Disabling tests with dontCheck.

And this other workaround for another issue also works: pyca/pyopenssl#768 (comment)

[nh2]

Getting some opendjk issues when trying to evaluate allStackageExecutables, because of:

assert stdenv.hostPlatform.libc == "glibc";

(CC @Ericson2314 @dtzWill)

niklas@ares ~/src/haskell/static-haskell-nix (git)-[master] % NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build --no-link survey/default.nix -A allStackageExecutables --show-trace
error: while evaluating the attribute 'nativeBuildInputs' of the derivation 'sparkle-0.7.4' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating the attribute 'buildInputs' of the derivation 'inline-java-0.8.4' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'getOutput' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/attrsets.nix:452:23, called from undefined position:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:127:17, called from undefined position:
while evaluating 'overrideCabal' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:37:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/configuration-nix.nix:83:9:
while evaluating 'override' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:20, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:37:28:
while evaluating 'makeOverridable' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:67:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:29:
while evaluating 'drvScope' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/make-package-set.nix:88:18, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:69:12:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/hackage-packages.nix:122015:6, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/make-package-set.nix:88:27:
while evaluating 'mkDerivation' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:38:20, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/hackage-packages.nix:122018:6:
while evaluating 'override' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:20, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:38:25:
while evaluating 'makeOverridable' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:67:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:29:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/generic-builder.nix:13:1, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:69:12:
while evaluating 'optionalAttrs' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/attrsets.nix:322:25, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/generic-builder.nix:474:4:
while evaluating the attribute 'buildInputs' of the derivation 'openjdk-8u181b13' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'getOutput' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/attrsets.nix:452:23, called from undefined position:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:127:17, called from undefined position:
while evaluating 'callPackageWith' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:108:35, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/top-level/all-packages.nix:6823:13:
while evaluating 'makeOverridable' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:67:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:112:8:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/compilers/openjdk/bootstrap.nix:1:1, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:69:12:
assertion failed at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/compilers/openjdk/bootstrap.nix:7:1