ok this is strange. If there is nothing special in your application, we might have a race condition somewhere that needs stamping out.
Is this state easy to replicate again and again on other devices?
Do you add the io.balena.features.balena-socket label in any container in your application?
Hi yep it’s replicable again and again but uses content from private repos which makes it hard to give to you guys as it’s company IP.
More than happy to do things to help repro for you.
I don’t add any Balena labels at all
Tim
Here are the key parts of the containers…
Docker-compose
version: '2.0'
volumes:
mongo-data:
services:
mongo:
image: mongo
restart: always
volumes:
- 'mongo-data:/data/db'
expose:
- "27017"
snips-server:
build:
context: ./
dockerfile: ./docker/snips/Dockerfile
command: yarn start
ports:
- 5000:5000
depends_on:
- mongo
runtime:
build:
context: .
dockerfile: ./docker/runtime/Dockerfile
command: yarn start
ports:
- 3001:3001
depends_on:
- mongo
- snips-server
api:
build:
context: .
dockerfile: ./docker/api/Dockerfile
command: yarn start
ports:
- 3000:3000
- 9229:9229
depends_on:
- mongo
- snips-server
- runtime
designer:
build:
context: .
dockerfile: ./services/ui/designer/Dockerfile
depends_on:
- mongo
- snips-server
- runtime
- api
ports:
- 80:80
Designer
FROM ***/** as workspaces
FROM balenalib/generic-aarch64-debian-node:latest-run as develop-stage
RUN apt-get update && \
apt-get install yarn && \
apt-get clean && rm -rf /var/lib/apt/lists/*
COPY --from=workspaces /app /app
ADD framework/common /app/framework/common
WORKDIR /app/services/ui/designer
ADD services/ui/designer .
ADD README.md public/docs/
ADD docs public/docs/docs/
ADD docs-v2 public/docs/docs-v2/
ADD integrations/connectors/alexa/README.md public/docs/integrations/connectors/alexa/
ADD integrations/connectors/http/README.md public/docs/integrations/connectors/http/
ADD integrations/connectors/msteams/README.md public/docs/integrations/connectors/msteams/
ADD integrations/statistics/README.md public/docs/integrations/statistics/
ADD integrations/statistics/statsd/README.md public/docs/integrations/statistics/statsd/
ADD kubernetes/README.md public/docs/kubernetes/
ADD kubernetes/eks/README.md public/docs/kubernetes/eks/
ADD kubernetes/azure/README.md public/docs/kubernetes/azure/
ADD kubernetes/fluentd/README.md public/docs/kubernetes/fluentd/
ADD kubernetes/helm/README.md public/docs/kubernetes/helm/
ADD kubernetes/minikube/README.md public/docs/kubernetes/minikube/
ADD services/ui/designer/README.md public/docs/services/ui/designer/
ADD tutorials public/docs/tutorials/
ADD services/cli/README.md public/docs/cli/
ADD services/cli/add-delete-bot.md public/docs/cli/
ADD services/cli/login-logout.md public/docs/cli/
ADD services/cli/tokens.md public/docs/cli/
ADD services/cli/view-bots.md public/docs/cli/
RUN yarn build
RUN pwd && ls && ls dist
# production stage
FROM nginx:1.15.7-alpine as production-stage
COPY --from=develop-stage /app /app
RUN rm -v /etc/nginx/conf.d/default.conf && \
cp /app/services/ui/designer/nginx.conf /etc/nginx/conf.d/default.conf
RUN ls /app/services/ui/designer
RUN cp -R /app/services/ui/designer/dist/* /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]
API
FROM **/** as workspaces
FROM balenalib/generic-aarch64-debian-node:latest-run
COPY --from=workspaces /app /app
COPY framework /app/framework
COPY integrations/ /app/integrations/
WORKDIR /app/services/api
COPY services/api .
RUN apt-get update && \
apt-get install yarn && \
apt-get clean && rm -rf /var/lib/apt/lists/*
CMD ["yarn","start"]
Runtime
FROM **/** as workspaces
FROM balenalib/generic-aarch64-debian-node:latest-run
RUN apt-get update && \
apt-get install tzdata yarn && \
apt-get clean && rm -rf /var/lib/apt/lists/*
COPY --from=workspaces /app /app
COPY framework/runtime /app/framework/runtime
COPY framework/common /app/framework/common
COPY framework/adapters /app/framework/adapters
COPY framework/realtime /app/framework/realtime
COPY integrations /app/integrations
WORKDIR /app/services/runtime
COPY services/runtime .
CMD ["yarn","start"]
NLP Server
FROM ***/*** as workspaces
FROM balenalib/generic-aarch64-ubuntu:latest-run
RUN apt-get -q update && apt-get install -yq --no-install-recommends python3 python3-dev gcc gfortran python3-pip build-essential curl file libblas3 liblapack3 liblapack-dev libblas-dev && \
rm -rf /var/lib/apt/lists/* && \
apt-get clean
RUN c_rehash
ENV PATH=/root/.cargo/bin:$PATH
# Install Rust
# RUN curl --proto '=https' --tlsv1.2 -sSf https://static.rust-lang.org/rustup/dist/aarch64-unknown-linux-gnu/rustup-init > rustup
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain 1.36.0 -y
# RUN ls && chmod a+x ./rustup
# RUN ./rustup --help
# RUN ./rustup update
# RUN ./rustup override set nightly-2019-09-10
# RUN ./rustup toolchain install
RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash \
&& [ -s "$HOME/.nvm/nvm.sh" ] && \. "$HOME/.nvm/nvm.sh" \
&& [ -s "$HOME/.nvm/bash_completion" ] && \. "$HOME/.nvm/bash_completion" \
&& nvm install 11.3.0 \
&& curl -o- -L https://yarnpkg.com/install.sh | bash
ENV PATH /bin/versions/node/v11.3.0/bin:/root/.yarn/bin:$PATH
RUN pip3 install --no-cache-dir --upgrade pip -v
RUN pip3 install --no-cache-dir setuptools
RUN /bin/bash -c "source $HOME/.cargo/env"
WORKDIR /root
WORKDIR /app/services/snips
RUN rm -fr /usr/local/bin/python && \
ln -s /usr/bin/python3 /usr/local/bin/python
ADD ./services/snips/requirements.txt /app/services/snips/requirements.txt
RUN pip3 install --no-cache-dir setuptools_rust
RUN pip3 install --no-cache-dir Cython
RUN pip3 install --no-cache-dir -r /app/services/snips/requirements.txt
RUN python3 -m snips_nlu download en
COPY --from=workspaces /app /app
ADD ./framework/common /app/framework/common
ADD ./framework/adapters/storage/mongodb-adapter /app/framework/adapters/storage/mongodb-adapter
ADD ./services/snips /app/services/snips
CMD [ "yarn", "start" ]
Hi I have a full log available for when the pi restarts downloading…
Can you let me know where to email it as its too large to upload.
Overnight, with one of the containers constantly downloading, I am now out of space on the device…
Oct 30 07:57:03 a01bdc6 033655e2e3d7[2208]: 2019-10-30T07:57:03.246+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
Oct 30 07:57:03 a01bdc6 033655e2e3d7[2208]: 2019-10-30T07:57:03.246+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1416M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
Oct 30 07:57:03 a01bdc6 033655e2e3d7[2208]: 2019-10-30T07:57:03.327+0000 E STORAGE [initandlisten] WiredTiger error (28) [1572422223:327814][1:0x7fade8d450], connection: __posix_file_write, 543: /data/db/journal/WiredTigerTmplog.0000000001: handle-write: pwrite: failed to write 128 bytes at offset 0: No space left on device Raw: [1572422223:327814][1:0x7fade8d450], connection: __posix_file_write, 543: /data/db/journal/WiredTigerTmplog.0000000001: handle-write: pwrite: failed to write 128 bytes at offset 0: No space left on device
Oct 30 07:57:03 a01bdc6 033655e2e3d7[2208]: 2019-10-30T07:57:03.327+0000 E STORAGE [initandlisten] WiredTiger error (28) [1572422223:327974][1:0x7fade8d450], connection: __log_fs_write, 220: journal/WiredTigerTmplog.0000000001: fatal log failure: No space left on device Raw: [1572422223:327974][1:0x7fade8d450], connection: __log_fs_write, 220: journal/WiredTigerTmplog.0000000001: fatal log failure: No space left on device
Oct 30 07:57:03 a01bdc6 033655e2e3d7[2208]: 2019-10-30T07:57:03.328+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1572422223:328020][1:0x7fade8d450], connection: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1572422223:328020][1:0x7fade8d450], connection: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic
Oct 30 07:57:03 a01bdc6 033655e2e3d7[2208]: 2019-10-30T07:57:03.328+0000 F - [initandlisten] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 414
Any ideas…
Would be great to get to the bottom of this for you guys,
Hi guys - any news on this as really want to get this platform deployed and online and work out what’s causing your platform to die on the download…
Hey Tim, it looks like support access on the device is expired, can you grant it for a longer amount of time? This is a strange issue and I suspect it might take a bit to get to the bottom of it. If there is any way you can help us come up with a minimal example that reliably hits the issue it would be really helpful. As for the logs: if you upload them through the forum, they will be attached to the ticket and any agent will be able to see them, unfortunately we still don’t have a good process for the case when the upload is too big. Can you maybe only upload an initial snippet of those logs that respects the size limit?
Here is the device… - I had to reload as something completely filled up the pi with all the downloads so mongo would no longer start…
fb82a2bb9bd9bc7aa77fa985e5ca1196
I am going to try and build a smaller container set which shows the error - give me a week or so…
I have uploaded the logs already so you should be able to see that from the other day - I also uploaded the info about the docker containers …
its currently in a state where is stuck downloading just one image!!!
If you go to the diagnostics tab, https://dashboard.balena-cloud.com/devices/fb82a2bb9bd9bc7aa77fa985e5ca1196/diagnostics
One of the checks is slow sd card. I know we have visited the slow sd card problem before…
I still think that is causing issues. The pi4 is fast. Balena Engine healthchecks should not time out. And they do… due to slow io
There is another thing we can try. Its a bit hacky but worth trying to check things out.
We can increase the timeout for balenaEngine.
As the filesystem is read-only, we remount it.
root@fb82a2b:~# mount -o remount,rw /
This remounts the filesystem as read-write.
Then this is the line to edit. Increase the timeout by a factor of 10 I guess.
root@fb82a2b:~# cat /lib/systemd/system/balena.service | grep Watch
WatchdogSec=360
root@fb82a2b:~#
then
systemctl daemon-reload
systemctl restart balena
I still think something strange is going on with file i/o on the pi4. it could be firmware/driver/sd card as the device definitely feels sluggish…
pi4 is still beta, so there might be some quirks…
I’ve done speed tests and it looks correct for the card.
Also - a single image being downloaded works fine. Two or more images causes a problem.
That can’t be a card problem?
Sorry - previous email was incorrect
A new image being uploaded is fine
Downloading a new version of an existing image or 2 images at the same time causes problems
I have run on multiple cards and the same issue - I have also run on a pi 3 and same issue
Tim
I’m trying the ‘hack’ to see the outcome
Tim
Done and this allowed the downloads to complete and get the containers started 
Tim
Thanks for trying out the timeout extension, that gives us a bit more information as to what is going on. If possible can you please try and strip down your application to a minimal reproduction and share it with us? I cannot stress how valuable it is to have an easily reproducible setup when investigating bugs like this
Will try and do it today - I’m on holiday so will see what I can reproduce for you