# All devices got offline simultaneously

**URL:** <https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114>\
**Category:** Product support\
**Tags:** network\
**Created:** [October 11, 2023, 1:45am UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114 "2023-10-11T01:45:04Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cota](https://avatars.discourse-cdn.com/v4/letter/c/5daacb/32.png) [@Cota](https://forums.balena.io/u/Cota)\
**Post date:** [October 11, 2023, 1:45am UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114/1 "2023-10-11T01:45:04Z")

</div>

Yesterday, all our devices (of all our fleets) got offline for 10 min simultaneously.

This situation was detected by monitoring MQTT connection from the container on our devices.

Their upstream networks are LTEs using multiple carriers.

After recovery, I checked the journal log of one of the devices. I attached the log file by replacing the hostname and the SSID.

It looks there was a problem on OpenVPN.

Could I ask the following questions?

- Were there any trouble on Balena’s OpenVPN system at the time although the status indicator [https://status.balena.io/](https://status.balena.io/) is all green?
- Will the container or the network stop their operation when such a OpenVPN problem happens?

[balena\_error\_20231011\_pub.txt](https://forums.balena.io/uploads/short-url/jw9QRRDd28JYytzflXnqv6Erg5j.txt) (86.7 KB)

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [October 11, 2023, 9:50am UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114/2 "2023-10-11T09:50:30Z")

</div>

Thanks for sharing @Cota

Are you using balenaCloud? I will research more on this!

---

<div class="post-metadata">

**Author:** ![Cota](https://avatars.discourse-cdn.com/v4/letter/c/5daacb/32.png) [@Cota](https://forums.balena.io/u/Cota)\
**Post date:** [October 11, 2023, 10:55am UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114/4 "2023-10-11T10:55:50Z")

</div>

Thank you for your response.

Yes. I’m using balenaCloud with Prototype plan.

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [October 11, 2023, 11:11am UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114/5 "2023-10-11T11:11:57Z")

</div>

@Cota could you please confirm what different carriers are you using? are the devices in the same geographical area?

We didn’t have any issue yesterday and it’s odd to see issues such as

```auto
Cannot resolve host address: cloudlink.balena-cloud.com:443

```

Do you know if the carriers use the same network provider?

---

<div class="post-metadata">

**Author:** ![Cota](https://avatars.discourse-cdn.com/v4/letter/c/5daacb/32.png) [@Cota](https://forums.balena.io/u/Cota)\
**Post date:** [October 11, 2023, 12:14pm UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114/8 "2023-10-11T12:14:35Z")

</div>

@mpous

Thank you for your clarification on the status of your system yesterday.

Our fleet is using Docomo, au and Softbank (Top 3 carriers in Japan).  
Their geographical areas are Tokyo, Kanagawa, Ibaraki, Ishikawa and Hyogo (Kanto, Hokuriku and Kansai region in Japan).

> Cannot resolve host address: [cloudlink.balena-cloud.com:443](http://cloudlink.balena-cloud.com:443)

Humm… It looks DNS problem. I don’t know whether the carriers use the same DNS.

I will search their DNS information and their status of yeesterday.

---

<div class="post-metadata">

**Author:** ![Cota](https://avatars.discourse-cdn.com/v4/letter/c/5daacb/32.png) [@Cota](https://forums.balena.io/u/Cota)\
**Post date:** [October 11, 2023, 1:01pm UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114/9 "2023-10-11T13:01:14Z")

</div>

Although the carriers are different, the SIM cards for the devices were provided by the same MVNO.

Now I suspect the DNS servers specified by the MVNO were down or under maintenance at that time.  
I’m asking the MVNO.

I’ll share the result soon after any answer comes.

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [October 11, 2023, 1:04pm UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114/10 "2023-10-11T13:04:23Z")

</div>

Thanks @Cota let us know if this is the case!

---

<div class="post-metadata">

**Author:** ![Cota](https://avatars.discourse-cdn.com/v4/letter/c/5daacb/32.png) [@Cota](https://forums.balena.io/u/Cota)\
**Post date:** [October 12, 2023, 4:20am UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114/11 "2023-10-12T04:20:30Z")

</div>

@mpous

Finally, the MVNO we are using told me that the period was for the scheduled maintenance of the network (including DNS servers) provider that the MVNO uses.

Unfortunately, the MVNO does not provide advance notification service. Then, I directly registered the maintenance notification service of the network provider today.

Thank you for your kind support.

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [October 19, 2023, 11:26am UTC](https://forums.balena.io/t/all-devices-got-offline-simultaneously/369114/12 "2023-10-19T11:26:58Z")

</div>

Thanks for the confirmation @Cota
