To date the best compression algorithm seems to be ZSTD.
Compresses a lot and in multicore.
In decompressing it is very fast.
What do you think if ZSTD was also added in the compression algorithms?
Hi! Can you elaborate a bit more about this. How do you see ZSTD used here?
Hi !, with this command :
zstd -T15 -d MYIMAGE.zstd -c | dd of=/dev/MYDISK bs=$((512 * 1024 ))
with the same hardware and with the same original image, it takes less than half the time compared to :
gunzip -c MYIMAGE.gz |dd of=/dev/MYDISK bs=512
The reason is simple, ZSTD uses multi core.
GZIP uses a single core.
Balena etcher is very performing but is slowed down by the compression protocol.
I hope I was understandable. 
Oh ok, so this is an idea about making etcher faster right? I was initially confused because the category of this thread is balenaEngine.
Forgive me if I got the category wrong.
I thought that being an engineering implementation, in fact you have to implement a new compression algorithm, this was the correct category!
In general in order to have performance gains, if we used ZSTD in etcher we would also need to distribute the images compressed in ZSTB. Most of the images out there are not distributed compressed in ZSTD. The Etcher team informed me that in the future we will add support for ZSTB but it’s not in the immediate plans. How do you see this being used in balenaEngine? Downloading images from our image registry in ZSTB?
TNX.
I try to implement zstd.
How Do you achieve writing the stream into the block device?
Can you tell me the part of the code that concerns the dd ?
Hi, Kirk. The streaming to the block device is done by our etcher-sdk, as well as all the other low-level logic of the etcher.
We don’t actually use dd, in fact, avoiding dd is one of the reasons we created Etcher, because there are a bunch of different versions, and they don’t verify the writes.
The etcher SDK is a very large project and has a lot of dependencies. Can you explain what you’re goal is, and maybe we can provide you with more specific advice?
Hey Kirk,
Just checking in here. Whenever you get a chance to provide some additional context, our team can get you some specific advice. Please let us know how we can help.
Also, just to let you know, there is an open issue for this already on GitHub, and you can track the progress of this feature there: Add zstd support · Issue #2856 · balena-io/etcher · GitHub