Deployment (git push resin master) hangs in Running state

Product support
1

Hi,

I’ve been having issues deploying my app. The last two releases I’ve tried to deploy have been stuck during git push resin master:

...
[main]    Downloading https://files.pythonhosted.org/packages/ff/f4/385715ccc461885f3cedf57a41ae3c12b5fec3f35cce4c8706b1a112a133/setuptools-40.0.0-py2.py3-none-any.whl (567kB)
    [main]  Installing collected packages: setuptools
    [main]    Found existing installation: setuptools 39.1.0
    [main]      Uninstalling setuptools-39.1.0:
    [main]        Successfully uninstalled setuptools-39.1.0
    [main]  Successfully installed setuptools-40.0.0
    [main]  Collecting atomicwrites==1.1.5 (from -r requirements.txt (line 1))
    [main]    Downloading https://files.pythonhosted.org/packages/0a/e8/cd6375e7a59664eeea9e1c77a766eeac0fc3083bb958c2b41ec46b95f29c/atomicwrites-1.1.5-py2.py3-none-any.whl
    [main]  Collecting attrs==18.1.0 (from -r requirements.txt (line 2))
    [main]    Downloading https://files.pythonhosted.org/packages/41/59/cedf87e91ed541be7957c501a92102f9cc6363c623a7666d69d51c78ac5b/attrs-18.1.0-py2.py3-none-any.whl
    [main]  Collecting colorama==0.3.9 (from -r requirements.txt (line 3))
    [main]    Downloading https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
    [main]  Collecting more-itertools==4.2.0 (from -r requirements.txt (line 4))
    [main]    Downloading https://files.pythonhosted.org/packages/85/40/90c3b0393e12b9827381004224de8814686e3d7182f9d4182477f600826d/more_itertools-4.2.0-py3-none-any.whl (45kB)
    [main]  Collecting pluggy==0.7.1 (from -r requirements.txt (line 5))
    [main]    Downloading https://files.pythonhosted.org/packages/f5/f1/5a93c118663896d83f7bcbfb7f657ce1d0c0d617e6b4a443a53abcc658ca/pluggy-0.7.1-py2.py3-none-any.whl
    [main]  Collecting psycopg2==2.7.5 (from -r requirements.txt (line 6))
    [main]    Downloading https://files.pythonhosted.org/packages/b2/c1/7bf6c464e903ffc4f3f5907c389e5a4199666bf57f6cd6bf46c17912a1f9/psycopg2-2.7.5.tar.gz (426kB)
    [main]  Collecting py==1.5.4 (from -r requirements.txt (line 7))
    [main]    Downloading https://files.pythonhosted.org/packages/f3/bd/83369ff2dee18f22f27d16b78dd651e8939825af5f8b0b83c38729069962/py-1.5.4-py2.py3-none-any.whl (83kB)
    [main]  Collecting pytest==3.7.1 (from -r requirements.txt (line 8))
    [main]    Downloading https://files.pythonhosted.org/packages/d2/86/7b9513da923b94e48c2cf013ae4eae8184a36ebeb7fe27d386bc3db4f56f/pytest-3.7.1-py2.py3-none-any.whl (202kB)
    [main]  Collecting ruamel.yaml==0.15.44 (from -r requirements.txt (line 9))
    [main]    Downloading https://files.pythonhosted.org/packages/cd/8f/cce2403b82c6939ada5fd39488f0ce91729581e2d17436cba1c2020d2997/ruamel.yaml-0.15.44.tar.gz (280kB)
    [main]  Collecting six==1.11.0 (from -r requirements.txt (line 10))
    [main]    Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
    [main]  Collecting SQLAlchemy==1.2.10 (from -r requirements.txt (line 11))
    [main]    Downloading https://files.pythonhosted.org/packages/8a/c2/29491103fd971f3988e90ee3a77bb58bad2ae2acd6e8ea30a6d1432c33a3/SQLAlchemy-1.2.10.tar.gz (5.6MB)
    [main]  Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from pytest==3.7.1->-r requirements.txt (line 8)) (40.0.0)
    [main]  Installing collected packages: atomicwrites, attrs, colorama, six, more-itertools, pluggy, psycopg2, py, pytest, ruamel.yaml, SQLAlchemy
    [main]    Running setup.py install for psycopg2: started
    [main]      Running setup.py install for psycopg2: finished with status 'done'
    [main]    Running setup.py install for ruamel.yaml: started
    [main]      Running setup.py install for ruamel.yaml: finished with status 'done'
    [main]    Running setup.py install for SQLAlchemy: started
    [main]      Running setup.py install for SQLAlchemy: finished with status 'done'
    [main]  Successfully installed SQLAlchemy-1.2.10 atomicwrites-1.1.5 attrs-18.1.0 colorama-0.3.9 more-itertools-4.2.0 pluggy-0.7.1 psycopg2-2.7.5 py-1.5.4 pytest-3.7.1 ruamel.yaml-0.15.44 six-1.11.0
    [main]  Processing /
    [Info]  Still working...
    [Info]  Still working...
    [Info]  Still working...
    [Info]  Still working...
    [Info]  Still working...
    [Info]  Still working...
    [Info]  Still working...
    [Info]  Still working...
    [Info]  Still working...
    [Info]  Still working...

Resin dashboard shows that the deployment is Running.

App ID: 1203358
Release ID: 38709ce77cb0b364ae3ffc21a5c43a5e88343ffc
Dockerfile:

# Dockerfile based on best practices listed in https://resin.io/blog/our-dockerfile-tips-tricks/

# Use date-tagged base image to ensure reproducibility
FROM resin/raspberrypi3-python:3.6.5-slim-20180707
ENV INITSYSTEM on

# Copy the working directory
COPY . ./

# Start target platform emulation
#RUN [ "cross-build-start" ]

# Install psycopg2 dependencies
RUN apt-get update && \
	apt-get install -y libpq-dev && \
	apt-get install -y gcc

# Install requirements
RUN pip install --upgrade pip && \
    pip install --upgrade setuptools && \
	pip install -r requirements.txt && \
	pip install -e .[prod]

# Stop target platform emulation
#RUN [ "cross-build-end" ]

# Start script
CMD ["bash", "start.sh"]

Any ideas on how to resolve this issue?

4

Hi @smomni, when you do the push, what does it says, Building on <machine>, what’s the machine that you get, when it hangs?

6

Hi @imrehg, it says Building on arm01 .

7

Thanks, we are checking with the team and keep you posted.

8

@smomni can you also paste the contents of requirements.txt here too?

9

@imrehg requirements.txt:

atomicwrites==1.1.5
attrs==18.1.0
colorama==0.3.9
more-itertools==4.2.0
pluggy==0.7.1
psycopg2==2.7.5
py==1.5.4
pytest==3.7.1
ruamel.yaml==0.15.44
six==1.11.0
SQLAlchemy==1.2.10
10

Hey, also what’s in the .[prod] folder? It looks like something from that folder is being installed and it goes into an infinite loop. Was that same pip install -e step working if you have tried on your local development machine, whether the module/code in there can be installed successfully?

The best would be if you could share the whole project (if it’s up on Github for example), but if not, this above is our current lead in troubleshooting the hang.

11

pip install -e .[prod] installs the production variant of the python package by executing setup.py. Installation works locally on x86 architecture but fails on emulated ARM architecture (see cross-build directives in Dockerfile). Unfortunately, I’m unable to share the project as it is not open source (at least yet).

setup.py:

from setuptools import setup, find_packages
from connectivity import __version__

setup(
    name='connectivity',
    version=__version__,
    packages=find_packages(),
    extras_require={
        'dev': [],
        'test': ['pytest', 'pytest-cov'],
        'prod': ['RPi.GPIO==0.6.3']
    },
    entry_points={
        'console_scripts': [],
    },
)
12

So that basically means, that the extra package of RPi.GPIO==0.6.3 is attempted to be installed?

13

Yes, exactly.

14

Does that hang happen all the time?

I’ve tried it a couple of times with a basic test project based on the code you pasted here, and that seems to work fine. Cutting to the relevant part:

...
[main]  Successfully installed SQLAlchemy-1.2.10 atomicwrites-1.1.5 attrs-18.1.0 colorama-0.3.9 more-itertools-4.2.0 pluggy-0.7.1 psycopg2-2.7.5 py-1.5.4 pytest-3.7.1 ruamel.yaml-0.15.44 six-1.11.0
[main]  Obtaining file:///
[main]  Collecting RPi.GPIO==0.6.3 (from test==3.14)
[main]    Downloading https://files.pythonhosted.org/packages/e2/58/6e1b775606da6439fa3fd1550e7f714ac62aa75e162eed29dbec684ecb3e/RPi.GPIO-0.6.3.tar.gz
[main]  Installing collected packages: RPi.GPIO, test
[main]    Running setup.py install for RPi.GPIO: started
[main]      Running setup.py install for RPi.GPIO: finished with status 'done'
[main]    Found existing installation: test 3.14
[main]      Can't uninstall 'test'. No files were found to uninstall.
[main]    Running setup.py develop for test
[main]  Successfully installed RPi.GPIO-0.6.3 test
...

I wonder why in your log it says [main] Processing /…

One thing I would try. Add a working directory, and have the code there, instead of in / directly. Might be that some file system traversion happens.

So add:

WORKDIR /usr/src/app

for example, and have the COPY step after this setting. Then the local directory is assumed to be whatever you set in WORKDIR. This is just a hunch, not sure what other difference you might have compared to my test setup…

15

No, there were several successful deployments before this.

Now I managed to get the deployment working! I moved a conditional import statement from connectivity/__init__.py to module connectivity/listener.py. However, the same import statement was fine in the previous successful deployment (75f237582f5d35c8eeba8e68c51dfe2001f89f1f) so I’m not exactly sure why it failed now ¯_(ツ)_/¯

Thanks for the help!

16

Thanks for the feedback. Glad it works now, and just let us know if anything else comes up!