Problem to compile a package with cffi dependencies

[Niks4925]

I've some problem to cross compile a package that uses the cffi dependency.
Here is a part of my build script:

# install cffi for build env as recommended
build-pip install cffi   
# just in case, I don't really know if it's usefull in this case
cross-expose cffi
# package that uses cryptography and so cffi
pip install azure-storage-blob==12.5.0

And the output in the console:

WARNING: CC is a compound command (['arm-gad-linux-gnueabi-gcc', '-march=armv7-a', '-mthumb-interwork', '-mfloat-abi=hard', '-mfpu=neon', '-mtune=cortex-a9', '--sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi'])
WARNING: This can cause issues for modules that don't expect it.
WARNING: Consider setting CC='arm-gad-linux-gnueabi-gcc' and CFLAGS='-march=armv7-a -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi'
WARNING: CXX is a compound command (['arm-gad-linux-gnueabi-g++', '-march=armv7-a', '-mthumb-interwork', '-mfloat-abi=hard', '-mfpu=neon', '-mtune=cortex-a9', '--sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi'])
WARNING: This can cause issues for modules that don't expect it.
WARNING: Consider setting CXX='arm-gad-linux-gnueabi-g++' and CXXFLAGS='-march=armv7-a -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi'
Collecting cffi
  Downloading https://files.pythonhosted.org/packages/50/ca/bbca0fd95b24a1d4f0d2e016f09f35ae68d4fe72bf34cc538d0a0d2d3e10/cffi-1.14.3-cp36-cp36m-manylinux1_x86_64.whl (400kB)
Collecting pycparser (from cffi)
  Downloading https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112kB)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.14.3 pycparser-2.20
You are using pip version 18.1, however version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting azure-storage-blob==12.5.0
  Downloading https://files.pythonhosted.org/packages/b8/84/7e51b3e1156bcb89a20b9ec641d4fced4800aa79daac3a403898c32046be/azure_storage_blob-12.5.0-py2.py3-none-any.whl (326kB)
Collecting azure-core<2.0.0,>=1.6.0 (from azure-storage-blob==12.5.0)
  Downloading https://files.pythonhosted.org/packages/4c/fa/46974f4a7ad78b27e3eda8a573cc0c2508849f0d7d360b61c07cc5b46014/azure_core-1.8.2-py2.py3-none-any.whl (122kB)
Collecting msrest>=0.6.10 (from azure-storage-blob==12.5.0)
  Downloading https://files.pythonhosted.org/packages/fa/f5/9e315fe8cb985b0ce052b34bcb767883dc739f46fadb62f05a7e6d6eedbe/msrest-0.6.19-py2.py3-none-any.whl (84kB)
Collecting cryptography>=2.1.4 (from azure-storage-blob==12.5.0)
  Downloading https://files.pythonhosted.org/packages/5d/4b/7bb135c5787c003cdbc44990c5f41908f0f37135e0bb554e880d90fd5f6f/cryptography-3.1.1.tar.gz (534kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /tmp/python-cross-env/cross/bin/python3.6 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k4ijf2ws --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=40.6.0 wheel "cffi>=1.8,!=1.11.3; platform_python_implementation != 'PyPy'":
  *******************************************************
  * Crossenv has leaked into another Python interpreter!
  * You should probably file a bug report.
  * Version 3.6.8 (default, Dec 25 2018, 00:00:00)
  [GCC 4.8.4]
  * Executable /tmp/python-cross-env/cross/bin/python3.6
  *******************************************************
  Collecting setuptools>=40.6.0
    Using cached https://files.pythonhosted.org/packages/6d/38/c21ef5034684ffc0412deefbb07d66678332290c14bb5269c85145fbd55e/setuptools-50.3.2-py3-none-any.whl
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/a7/00/3df031b3ecd5444d572141321537080b40c1c25e1caa3d86cdd12e5e919c/wheel-0.35.1-py2.py3-none-any.whl
  Collecting cffi!=1.11.3,>=1.8
    Downloading https://files.pythonhosted.org/packages/cb/ae/380e33d621ae301770358eb11a896a34c34f30db188847a561e8e39ee866/cffi-1.14.3.tar.gz (470kB)
  Collecting pycparser (from cffi!=1.11.3,>=1.8)
    Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
  Installing collected packages: setuptools, wheel, pycparser, cffi
    Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      Complete output from command /tmp/python-cross-env/cross/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-b363qank/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dcmdpimn/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-k4ijf2ws --compile --install-headers /tmp/python-cross-env/cross/include/site/python3.6/cffi:
      *******************************************************
      * Crossenv has leaked into another Python interpreter!
      * You should probably file a bug report.
      * Version 3.6.8 (default, Dec 25 2018, 00:00:00)
      [GCC 4.8.4]
      * Executable /tmp/python-cross-env/cross/bin/python3.6
      *******************************************************
      *******************************************************
      * Crossenv has leaked into another Python interpreter!
      * You should probably file a bug report.
      * Version 3.6.8 (default, Dec 25 2018, 00:00:00)
      [GCC 4.8.4]
      * Executable /tmp/python-cross-env/cross/bin/python3.6
      *******************************************************
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-arm-3.6
      creating build/lib.linux-arm-3.6/cffi
      copying cffi/api.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/verifier.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/error.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/model.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/recompiler.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/__init__.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/cparser.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/lock.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/commontypes.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/pkgconfig.py -> build/lib.linux-arm-3.6/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-arm-3.6/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-arm-3.6/cffi
      copying cffi/_embedding.h -> build/lib.linux-arm-3.6/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-arm-3.6/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-arm-3.6
      creating build/temp.linux-arm-3.6/c
      arm-gad-linux-gnueabi-gcc -march=armv7-a -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi -Wno-unused-result -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/tmp/python-cross-env/cross/include -I/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi/usr/include/python3.6m -c c/_cffi_backend.c -o build/temp.linux-arm-3.6/c/_cffi_backend.o
      c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
       #include <ffi.h>
                       ^
      compilation terminated.
      error: command 'arm-gad-linux-gnueabi-gcc' failed with exit status 1

You can see that the pip command uses --ignore-installed probably because I use a Debian environment to build, but I don't know if it's relevant. I've tried to deactivate it without success.

[benfogle]

I'm not sure exactly what's happening, see if the following works for you:

$ build-pip install cffi pycparser
$ cross-expose cffi pycparser
$ pip install azure-storage-blob==12.5.0

I think the fact that cross-pip could see cffi but not its dependency, pycparser was causing issues. I'm not sure why, and it might be a regression somewhere.

[Niks4925]

I've tried what you suggest without success.

I should have specified that I cross-compile libffi and openssl upstream for other needs of my application.
I don't know if it's relevant, but with these I managed to install the package by running crossenv and adding the libraries headers:

python3.6 -m crossenv --env CFLAGS="-I$(pwd)/libffi/arm-gad-linux-gnueabi/include -I${SYSROOT}/usr/local/ssl/include" "${SYSROOT}/usr/bin/python3.6" "${CROSSENV_PATH}"
  . "${CROSSENV_PATH}/bin/activate"
# this is still mandatory as it doesn't compile without it
build-pip install cffi
pip install azure-storage-blob==12.5.0

I don't know if there's something I misunderstood about crossenv (very likely) or if it's a bug.

[odidev]

You need to prepare the toolchain properly. you need to cross compile libffi-dev and install it in toolchain to resolve the header/libraries related issue.