Logs that show size of upload

I did run Cameron’s script afterwards and that’s where I got the 0. Sorry, I meant to mention that. And yes, the delta was created, but only after I manually forced it by pushing it to a test device.

I’ll push another update for you to test.

curl --silent "https://delta.balena-cloud.com/api/v2/delta?src=1092753&dest=1095272' -H 'Authorization: Bearer $authToken"

The result of calling Cameron’s script: {my-app-name: 0}

Image 1092753 does not exist. How do you get the image IDs?

In any case, trying to manually generate a delta isn’t the most straightforward thing for several reasons. My suggestion is to use a single device which you pin/unpin and let it update which will do all the heavy lifting. When it completes updating, you can use the script Cameron shared and request details for the delta.

1 Like

Right, I think you passed a release ID instead of image ID in that delta curl request. Like I said, I think it’s best if you let a device do the hard work.

1 Like

Oh, yep I certainly did use the release ID rather than the image ID.

How do I find out the image ids? I don’t see it in the API documentation, nor do I see a way to get the image id from the dashboard.

Why does the delta api use the image id rather than the release id? It seems most other API interfaces use the release id, and since that’s what I use to push an update to the devices anyway I think it would make sense to use. What am I missing?

The only issue I see with having a dedicated update device for this is that often I have different devices running different releases, which means that for every release I want to update I need to run a delta, and in order to run the delta I must first get the update device to have the target source image, and each of those operations takes a lot of time. something like total_wait_time = time_to_update_device * 2 * number_of_deltas_to_compute which I think can easily become quite lengthy. It would be really wonderful to be able to simply run ./gen-delta.sh src dst a couple times and then to poll the delta difference at the end.

Question, when you say to let a device do the hard work, what hard work is happening? I had assumed that the majority of the delta computation would have to happen on your servers, prior to pushing the delta image to my device. Are you indicating that my update device takes on some of that computational load, or does specifying an update device just enable you to more easily infer the settings (like architecture, etc.) required for finding the delta?

How do I find out the image ids?

Perhaps the supervisor API (GET /v2/state/status, or other endpoints) could be useful for this purpose:
https://www.balena.io/docs/reference/supervisor/supervisor-api/#get-v2statestatus

It can be queried locally in a device, or to balenaCloud from anywhere else:
https://www.balena.io/docs/reference/supervisor/supervisor-api/#http-api-reference

I had assumed that the majority of the delta computation would have to happen on your servers, prior to pushing the delta image to my device.

Yes I believe that’s correct.

Are you indicating that my update device takes on some of that computational load,

No, I don’t think so.

or does specifying an update device just enable you to more easily infer the settings (like architecture, etc.) required for finding the delta?

I think it’s simply meant to simplify your code. I have the impression that some of logic and API queries are currently coded in the balena supervisor, and you would end up having to recode some of it yourself.

1 Like

Very cool, thank you. I did not know about the get state endpoint in the supervisor API, that will be very useful.

I think it’s simply meant to simplify your code. I have the impression that some of logic and API queries are currently coded in the balena supervisor, and you would end up having to recode some of it yourself.

I’m wondering if anyone could let me know what additional work is required to use the delta API query? I would be happy to write the code and share a gen-delta.sh script with the community for the benefit of whomever else might like to use this feature.

Hi @cnr, before we go that route, could you please clarify what’s not sufficient with the steps we outlined earlier in this thread?

Thanks to your questions and comments here, we created a feature request on your behalf here: https://github.com/balena-io/balena-sdk/issues/747

So our SDKs would support this functionality out of the box. I’ll take a note to ping this thread when we release this SDK feature.

we created a feature request on your behalf here:

Wonderful, thank you!

could you please clarify what’s not sufficient with the steps we outlined earlier in this thread?

Oh, I’m not sure that there is an issue. On one hand, dfunckt’s statement:

In any case, trying to manually generate a delta isn’t the most straightforward thing for several reasons.

makes me think that perhaps I’m missing something.

On the other, I have a lot of information here, so I’ll take a crack at scripting this and I’ll share my results. Thank you all for your help :slight_smile:

I’m trying to find out how large the update to a device will be so that I don’t use more data than my device allows.
Here is the code I’m using to generate the delta between two images:

./check-configuration.sh || exit 1

if [ -z $1 -o -z $2 ]
then
  echo "./gen-delta.sh src-id dest-id"
  exit 244
fi

source ./balena.env
set -x
response=$(curl --silent "https://delta.$BASE_URL/api/v2/delta?src=$1&dest=$2" -H "Authorization: Bearer $authToken")

echo $response%  

When run it returns no response

$ ./gen-delta.sh 1596694 1598161

But then when I try to run the code that calculates the delta it returns Promise {<pending>} and no difference between the two {my-app-name: null}. I’ve waited two hours for the delta to be computed. The command I’m running:

release1 = 1596694
release2 = 1598161

sdk.pine.get({
    resource: 'release',
    options: {
        $filter: {
            id:{ $in: [ release1, release2 ]},
        },
        $orderby: 'id asc',
        $expand: {
            image__is_part_of__release: {
                $select: 'id',
                $expand: {
                    image: {
                        $select: [
                            'id'
                        ],
                        $expand: {
                            is_a_build_of__service: {
                                $select: 'service_name'
                            }
                        }
                    }
                }
            }
        }
    }
}).then(([r1, r2]) => {
    r1services = { }
    _.each(r1.image__is_part_of__release, (ipr) => {
        r1services[ ipr.image[0].is_a_build_of__service[0].service_name] = ipr.image[0].id;
    })
    r2services = { }
    _.each(r2.image__is_part_of__release, (ipr) => {
        r2services[ ipr.image[0].is_a_build_of__service[0].service_name] = ipr.image[0].id;
    })

    deltaSizes = { }
    return Promise.all(_.map(r1services, (id, name) => { 
        return sdk.pine.get({ resource: 'delta', options: { $filter: { originates_from__image: id, produces__image: r2services[name] }, $select: 'size' } }).then(([ img ]) => { if (img != null) { deltaSizes[name] = img.size } else { deltaSizes[name] = null }});
    })).then(() => console.log(deltaSizes))
})

The result of running await sdk.pine.get({ resource: 'delta' }) is here: dashboard.balena-cloud.com-1605546813083.log (13.0 KB)

Ah, perhaps I’m once again trying with the release id not the image id. It’s still unclear to me how to get access to the image id without downloading the image onto a device as that is not an option for me.

Neither of these solutions work because they both require a supervisor to already have downloaded the image delta, which is exactly the thing I’m trying to avoid.

Following up on this. Need to know how to get the image id without having to download the image to a device

Hi @cnr,
I would like to better understand since the function you are running seems right, release1 and release2 must be the old and new release ids. To check that ids are correct, you can get a list of releases of your application by running the following request from the browser console:

await sdk.pine.get({
  resource: 'release',
  options: {
    $filter: {
      belongs_to__application: <APPLICATION_ID> 
    }
  }
})

(<APPLICATION_ID> can be also found in the url after /apps/. )
this request will return you an array of releases, which contain an id property (this is the property you are interested in).

Also to prevent the request you made above from returning a Promise {<pending>} you have to add an await in front, so:

release1 = <OLD_RELEASE_ID>
release2 = <NEW_RELEASE_ID>

await sdk.pine.get({
    resource: 'release',
    options: {
        $filter: {
            id:{ $in: [ release1, release2 ]},
        },
        $orderby: 'id asc',
...

Before continuing the analysis, can you confirm that these are the steps you have taken? if you confirm me that this is what you have done so far, we need to understand why the value it returns to you is null. To help us in the analysis it would be useful to have the response of the first reqeust I sent you on this message (release request).

For those following along at home, the command I’m using to get all the releases in sorted order so that I can pick out my image id is:

releases = await sdk.pine.get({
  resource: 'release',
  options: {
    $filter: {
      belongs_to__application: <MY APPLICATION ID>
    }
  }
})
releases.sort((a, b) => (Date.parse(a.created_at) < Date.parse(b.created_at)) ? 1 : -1)

I then try to calculate the delta size with:

release1 = 1602208
release2 = 1605036

await sdk.pine.get({
    resource: 'release',
    options: {
        $filter: {
            id:{ $in: [ release1, release2 ]},
        },
        $orderby: 'id asc',
        $expand: {
            image__is_part_of__release: {
                $select: 'id',
                $expand: {
                    image: {
                        $select: [
                            'id'
                        ],
                        $expand: {
                            is_a_build_of__service: {
                                $select: 'service_name'
                            }
                        }
                    }
                }
            }
        }
    }
}).then(([r1, r2]) => {
    r1services = { }
    _.each(r1.image__is_part_of__release, (ipr) => {
        r1services[ ipr.image[0].is_a_build_of__service[0].service_name] = ipr.image[0].id;
    })
    r2services = { }
    _.each(r2.image__is_part_of__release, (ipr) => {
        r2services[ ipr.image[0].is_a_build_of__service[0].service_name] = ipr.image[0].id;
    })

    deltaSizes = { }
    return Promise.all(_.map(r1services, (id, name) => { 
        return sdk.pine.get({ resource: 'delta', options: { $filter: { originates_from__image: id, produces__image: r2services[name] }, $select: 'size' } }).then(([ img ]) => { if (img != null) { deltaSizes[name] = img.size } else { deltaSizes[name] = null }});
    })).then(() => console.log(deltaSizes))
})

This gives me the error:

VM68271:32 Uncaught ReferenceError: _ is not defined
    at <anonymous>:31:5
    at async <anonymous>:4

I don’t know what’s causing this error. It’s not because I have releases without deltas (tested that). It’s not because of the application I’m on (tested that). It doesn’t have to do with the newly added await token before invoking sdk.pine... I’m guessing this has to do with not having the underscore package installed. But this used to work as of two days ago… so I don’t know what’s changed.

Following up on this. Sorry, I’m not well versed in node, so I’m not quite sure how to debug this on my own.