I did run Cameron’s script afterwards and that’s where I got the 0. Sorry, I meant to mention that. And yes, the delta was created, but only after I manually forced it by pushing it to a test device.
I’ll push another update for you to test.
curl --silent "https://delta.balena-cloud.com/api/v2/delta?src=1092753&dest=1095272' -H 'Authorization: Bearer $authToken"
The result of calling Cameron’s script: {my-app-name: 0}
Image 1092753 does not exist. How do you get the image IDs?
In any case, trying to manually generate a delta isn’t the most straightforward thing for several reasons. My suggestion is to use a single device which you pin/unpin and let it update which will do all the heavy lifting. When it completes updating, you can use the script Cameron shared and request details for the delta.
Right, I think you passed a release ID instead of image ID in that delta curl request. Like I said, I think it’s best if you let a device do the hard work.
Oh, yep I certainly did use the release ID rather than the image ID.
How do I find out the image ids? I don’t see it in the API documentation, nor do I see a way to get the image id from the dashboard.
Why does the delta api use the image id rather than the release id? It seems most other API interfaces use the release id, and since that’s what I use to push an update to the devices anyway I think it would make sense to use. What am I missing?
The only issue I see with having a dedicated update device for this is that often I have different devices running different releases, which means that for every release I want to update I need to run a delta, and in order to run the delta I must first get the update device to have the target source image, and each of those operations takes a lot of time. something like total_wait_time = time_to_update_device * 2 * number_of_deltas_to_compute which I think can easily become quite lengthy. It would be really wonderful to be able to simply run ./gen-delta.sh src dst a couple times and then to poll the delta difference at the end.
Question, when you say to let a device do the hard work, what hard work is happening? I had assumed that the majority of the delta computation would have to happen on your servers, prior to pushing the delta image to my device. Are you indicating that my update device takes on some of that computational load, or does specifying an update device just enable you to more easily infer the settings (like architecture, etc.) required for finding the delta?
How do I find out the image ids?
Perhaps the supervisor API (GET /v2/state/status, or other endpoints) could be useful for this purpose:
It can be queried locally in a device, or to balenaCloud from anywhere else:
I had assumed that the majority of the delta computation would have to happen on your servers, prior to pushing the delta image to my device.
Yes I believe that’s correct.
Are you indicating that my update device takes on some of that computational load,
No, I don’t think so.
or does specifying an update device just enable you to more easily infer the settings (like architecture, etc.) required for finding the delta?
I think it’s simply meant to simplify your code. I have the impression that some of logic and API queries are currently coded in the balena supervisor, and you would end up having to recode some of it yourself.
Very cool, thank you. I did not know about the get state endpoint in the supervisor API, that will be very useful.
I think it’s simply meant to simplify your code. I have the impression that some of logic and API queries are currently coded in the balena supervisor, and you would end up having to recode some of it yourself.
I’m wondering if anyone could let me know what additional work is required to use the delta API query? I would be happy to write the code and share a gen-delta.sh script with the community for the benefit of whomever else might like to use this feature.
Hi @cnr, before we go that route, could you please clarify what’s not sufficient with the steps we outlined earlier in this thread?
Thanks to your questions and comments here, we created a feature request on your behalf here: Should have delta generation & size estimate methods · Issue #747 · balena-io/balena-sdk · GitHub
So our SDKs would support this functionality out of the box. I’ll take a note to ping this thread when we release this SDK feature.
we created a feature request on your behalf here:
Wonderful, thank you!
could you please clarify what’s not sufficient with the steps we outlined earlier in this thread?
Oh, I’m not sure that there is an issue. On one hand, dfunckt’s statement:
In any case, trying to manually generate a delta isn’t the most straightforward thing for several reasons.
makes me think that perhaps I’m missing something.
On the other, I have a lot of information here, so I’ll take a crack at scripting this and I’ll share my results. Thank you all for your help 
I’m trying to find out how large the update to a device will be so that I don’t use more data than my device allows.
Here is the code I’m using to generate the delta between two images:
./check-configuration.sh || exit 1
if [ -z $1 -o -z $2 ]
then
echo "./gen-delta.sh src-id dest-id"
exit 244
fi
source ./balena.env
set -x
response=$(curl --silent "https://delta.$BASE_URL/api/v2/delta?src=$1&dest=$2" -H "Authorization: Bearer $authToken")
echo $response%
When run it returns no response
$ ./gen-delta.sh 1596694 1598161
But then when I try to run the code that calculates the delta it returns Promise {<pending>} and no difference between the two {my-app-name: null}. I’ve waited two hours for the delta to be computed. The command I’m running:
release1 = 1596694
release2 = 1598161
sdk.pine.get({
resource: 'release',
options: {
$filter: {
id:{ $in: [ release1, release2 ]},
},
$orderby: 'id asc',
$expand: {
image__is_part_of__release: {
$select: 'id',
$expand: {
image: {
$select: [
'id'
],
$expand: {
is_a_build_of__service: {
$select: 'service_name'
}
}
}
}
}
}
}
}).then(([r1, r2]) => {
r1services = { }
_.each(r1.image__is_part_of__release, (ipr) => {
r1services[ ipr.image[0].is_a_build_of__service[0].service_name] = ipr.image[0].id;
})
r2services = { }
_.each(r2.image__is_part_of__release, (ipr) => {
r2services[ ipr.image[0].is_a_build_of__service[0].service_name] = ipr.image[0].id;
})
deltaSizes = { }
return Promise.all(_.map(r1services, (id, name) => {
return sdk.pine.get({ resource: 'delta', options: { $filter: { originates_from__image: id, produces__image: r2services[name] }, $select: 'size' } }).then(([ img ]) => { if (img != null) { deltaSizes[name] = img.size } else { deltaSizes[name] = null }});
})).then(() => console.log(deltaSizes))
})
The result of running await sdk.pine.get({ resource: 'delta' }) is here: dashboard.balena-cloud.com-1605546813083.log (13.0 KB)
Ah, perhaps I’m once again trying with the release id not the image id. It’s still unclear to me how to get access to the image id without downloading the image onto a device as that is not an option for me.
Neither of these solutions work because they both require a supervisor to already have downloaded the image delta, which is exactly the thing I’m trying to avoid.
Following up on this. Need to know how to get the image id without having to download the image to a device
Hi @cnr,
I would like to better understand since the function you are running seems right, release1 and release2 must be the old and new release ids. To check that ids are correct, you can get a list of releases of your application by running the following request from the browser console:
await sdk.pine.get({
resource: 'release',
options: {
$filter: {
belongs_to__application: <APPLICATION_ID>
}
}
})
(<APPLICATION_ID> can be also found in the url after /apps/. )
this request will return you an array of releases, which contain an id property (this is the property you are interested in).
Also to prevent the request you made above from returning a Promise {<pending>} you have to add an await in front, so:
release1 = <OLD_RELEASE_ID>
release2 = <NEW_RELEASE_ID>
await sdk.pine.get({
resource: 'release',
options: {
$filter: {
id:{ $in: [ release1, release2 ]},
},
$orderby: 'id asc',
...
Before continuing the analysis, can you confirm that these are the steps you have taken? if you confirm me that this is what you have done so far, we need to understand why the value it returns to you is null. To help us in the analysis it would be useful to have the response of the first reqeust I sent you on this message (release request).
For those following along at home, the command I’m using to get all the releases in sorted order so that I can pick out my release id is:
releases = await sdk.pine.get({
resource: 'release',
options: {
$filter: {
belongs_to__application: <MY APPLICATION ID>
}
}
})
releases.sort((a, b) => (Date.parse(a.created_at) < Date.parse(b.created_at)) ? 1 : -1)
I then try to calculate the delta size with:
release1 = 1602208
release2 = 1605036
await sdk.pine.get({
resource: 'release',
options: {
$filter: {
id:{ $in: [ release1, release2 ]},
},
$orderby: 'id asc',
$expand: {
image__is_part_of__release: {
$select: 'id',
$expand: {
image: {
$select: [
'id'
],
$expand: {
is_a_build_of__service: {
$select: 'service_name'
}
}
}
}
}
}
}
}).then(([r1, r2]) => {
r1services = { }
_.each(r1.image__is_part_of__release, (ipr) => {
r1services[ ipr.image[0].is_a_build_of__service[0].service_name] = ipr.image[0].id;
})
r2services = { }
_.each(r2.image__is_part_of__release, (ipr) => {
r2services[ ipr.image[0].is_a_build_of__service[0].service_name] = ipr.image[0].id;
})
deltaSizes = { }
return Promise.all(_.map(r1services, (id, name) => {
return sdk.pine.get({ resource: 'delta', options: { $filter: { originates_from__image: id, produces__image: r2services[name] }, $select: 'size' } }).then(([ img ]) => { if (img != null) { deltaSizes[name] = img.size } else { deltaSizes[name] = null }});
})).then(() => console.log(deltaSizes))
})
This gives me the error:
VM68271:32 Uncaught ReferenceError: _ is not defined
at <anonymous>:31:5
at async <anonymous>:4
I don’t know what’s causing this error. It’s not because I have releases without deltas (tested that). It’s not because of the application I’m on (tested that). It doesn’t have to do with the newly added await token before invoking sdk.pine... I’m guessing this has to do with not having the underscore package installed. But this used to work as of two days ago… so I don’t know what’s changed.
Following up on this. Sorry, I’m not well versed in node, so I’m not quite sure how to debug this on my own.
Following up on this
Hi @cnr,
VM68271:32 Uncaught ReferenceError: _ is not defined
at <anonymous>:31:5
at async <anonymous>:4
This error comes because you are trying to use lodash but it is not imported. May I ask where are you trying this code? Are you trying it in the browser console ? If so there are two solutions:
var el = document.createElement('script');
el.src = "https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.15.0/lodash.min.js";
el.type = "text/javascript";
document.head.appendChild(el);
const release1 = 1602208;
const release2 = 1605036;
try {
const [r1, r2] = await sdk.pine.get({
resource: 'release',
options: {
$filter: {
id: { $in: [release1, release2] },
},
$orderby: 'id asc',
$expand: {
image__is_part_of__release: {
$select: 'id',
$expand: {
image: {
$select: [
'id'
],
$expand: {
is_a_build_of__service: {
$select: 'service_name'
}
}
}
}
}
}
}
});
const reduceServiceInfo = (accumulator, ipr) => {
return { ...accumulator, [ipr.image[0].is_a_build_of__service[0].service_name]: ipr.image[0].id };
}
let r1services = r1.image__is_part_of__release.reduce(reduceServiceInfo, {})
let r2services = r2.image__is_part_of__release.reduce(reduceServiceInfo, {})
try {
const deltas = await Promise.all(Object.entries(r1services).map(async ([name, id]) => {
const [delta] = await sdk.pine.get({
resource: 'delta',
options: {
$filter: { originates_from__image: id, produces__image: r2services[name] ?? null },
$select: 'size'
}
})
return { [name]: delta }
}));
console.log('Deltas', deltas)
} catch (err) {
console.log('Deltas error: ', err);
}
} catch (err) {
console.log('Releases error: ', err);
}
The second solution should work fine in any case, let me know how it goes.
I’m now running this script for anyone that would like a more automated way to generate deltas. This script relies on commit ids (things you generally know and can easily copy from the balena dashboard) rather than release ids.
You can get this script working by cloning this balena-playground repo and then adding it. I called it get-delta.sh
#!/bin/zsh
./check-configuration.sh || exit 1
if [ -z $1 -o -z $2 ]
then
echo "./get-delta.sh src-commit-id dest-commit-id"
echo "src-commit-id is the commit id of the release you want to start with, dest-commit-id is"\
"the commit id you want to push"
exit 244
fi
src="$1"
dst="$2"
source ./balena.env
# set -x
getIdFromCommit () {
commit="$1"
curl --silent -X GET \
"https://api.balena-cloud.com/v4/release?\$filter=belongs_to__application%20eq%20$APP_ID" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $authToken" | jq -r '.[][] | select(.commit=="'"$commit"'") | "\(.id)"'
}
srcId=$(getIdFromCommit $src)
dstId=$(getIdFromCommit $dst)
if [ -z "$srcId" ]
then
echo "No source id found for ${src}"
exit 245
elif [ -z "$dstId" ]
then
echo "No source id found for ${dst}"
exit 245
fi
echo "Generating delta for ${srcId} (${src:0:7}) to ${dstId} (${dst:0:7}) update." >&2
curl "https://delta.$BASE_URL/api/v2/delta?src=$srcId&dest=$dstId" -H "Authorization: Bearer $authToken" >&2
echo "We have no way to know if it succeeded because the balena API offers no feedback. "\
"A long wait is a good sign." >&2
echo "Now try running the following code in your developer console when logged into your balena application."\
'You can access it with CMD + SHIFT + I\nYou can copy the following code this command by running this command as `'"./get-delta.sh $src $dst | pbcopy"'` \n\n' >&2
read -d '' CODE <<-'THEEND'
var release1 = REPLACE_WITH_SRCID;
var release2 = REPLACE_WITH_DSTID;
try {
const [r1, r2] = await sdk.pine.get({
resource: 'release',
options: {
$filter: {
id: { $in: [release1, release2] },
},
$orderby: 'id asc',
$expand: {
image__is_part_of__release: {
$select: 'id',
$expand: {
image: {
$select: [
'id'
],
$expand: {
is_a_build_of__service: {
$select: 'service_name'
}
}
}
}
}
}
}
});
const reduceServiceInfo = (accumulator, ipr) => {
return { ...accumulator, [ipr.image[0].is_a_build_of__service[0].service_name]: ipr.image[0].id };
}
let r1services = r1.image__is_part_of__release.reduce(reduceServiceInfo, {})
let r2services = r2.image__is_part_of__release.reduce(reduceServiceInfo, {})
try {
const deltas = await Promise.all(Object.entries(r1services).map(async ([name, id]) => {
const [delta] = await sdk.pine.get({
resource: 'delta',
options: {
$filter: { originates_from__image: id, produces__image: r2services[name] ?? null },
$select: 'size'
}
})
return { [name]: delta }
}));
console.log('Deltas', deltas)
} catch (err) {
console.log('Deltas error: ', err);
}
} catch (err) {
console.log('Releases error: ', err);
}
THEEND
printf "$CODE" | sed 's/REPLACE_WITH_SRCID/'"$srcId"'/g' | sed 's/REPLACE_WITH_DSTID/'"$dstId"'/g'
But currently it’s not working.
For releases I know have a difference it’s returning no difference:
Deltas
[{…}]
0:
dame_gothel: undefined
__proto__: Object
The code is below, the difference between these should be on the order of megabytes:
var release1 = 1632183;
var release2 = 1632146;
try {
const [r1, r2] = await sdk.pine.get({
resource: 'release',
options: {
$filter: {
id: { $in: [release1, release2] },
},
$orderby: 'id asc',
$expand: {
image__is_part_of__release: {
$select: 'id',
$expand: {
image: {
$select: [
'id'
],
$expand: {
is_a_build_of__service: {
$select: 'service_name'
}
}
}
}
}
}
}
});
const reduceServiceInfo = (accumulator, ipr) => {
return { ...accumulator, [ipr.image[0].is_a_build_of__service[0].service_name]: ipr.image[0].id };
}
let r1services = r1.image__is_part_of__release.reduce(reduceServiceInfo, {})
let r2services = r2.image__is_part_of__release.reduce(reduceServiceInfo, {})
try {
const deltas = await Promise.all(Object.entries(r1services).map(async ([name, id]) => {
const [delta] = await sdk.pine.get({
resource: 'delta',
options: {
$filter: { originates_from__image: id, produces__image: r2services[name] ?? null },
$select: 'size'
}
})
return { [name]: delta }
}));
console.log('Deltas', deltas)
} catch (err) {
console.log('Deltas error: ', err);
}
} catch (err) {
console.log('Releases error: ', err);
}
Please please please I really need this to work. I have been trying to get the difference between the size of two images for over a year.
Hi, I have restructured the code into separate functions, so you can get each value step by step.
async function getRelease (commitId) {
const [release] = await sdk.pine.get({
resource: 'release',
options: {
$filter: {
commit: commitId,
},
}
});
console.table(release)
return release
}
async function getServiceImageIds (releaseId) {
const [release] = await sdk.pine.get({
resource: 'release',
options: {
$filter: {
id: releaseId,
},
$orderby: 'id asc',
$expand: {
image__is_part_of__release: {
$select: 'id',
$expand: {
image: {
$select: [
'id'
],
$expand: {
is_a_build_of__service: {
$select: 'service_name',
}
}
}
}
}
}
}
});
const reduceServiceInfo = (accumulator, ipr) => {
return {...accumulator, [ipr.image[0].is_a_build_of__service[0].service_name]: ipr.image[0].id};
}
let serviceImages = release.image__is_part_of__release.reduce(reduceServiceInfo, {})
console.table(serviceImages)
return serviceImages
}
async function getDelta(imageId) {
const [delta] = await sdk.pine.get({
resource: 'delta',
options: {
$filter: {produces__image: imageId},
$select: 'size',
}
})
console.table(delta)
return delta
}
If you have a commit, use getRelease('your commit id')to get the release id, it will display a table of values copy the id. Then you can get the services and their image id for a release getServiceImageIds(your_release_id) it will display a table of services and the image id, copy the image id. You can then get a delta for an image getDelta(your_image_id) which will display a table with the size of a delta. It will show the size of the delta for the service image between the previous commit and the commit you entered.
2 questions: