"error: unable to write sha1 filename" when trying to push my code to balena

I am currently getting the following errors when trying to push code to balena:

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)
$ git push balena dev_pulseaudio:master
Enter passphrase for key '/c/Users/IBM_ADMIN/.ssh/id_rsa':
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (6/6), 525 bytes | 43.00 KiB/s, done.
Total 6 (delta 4), reused 5 (delta 3)
error: unable to write sha1 filename ./objects/28/c24232ab573f0eb00f9704f835c6b3494a2c25: Permission denied
error: unable to write sha1 filename ./objects/0e/bcd597a229f2255ebd4aeb5fc7bae3eafbff7c: Permission denied
To git.balena-cloud.com:gh_janvda/pi3three.git
 ! [remote rejected] dev_pulseaudio -> master (unable to migrate objects to permanent storage)
error: failed to push some refs to 'gh_janvda@git.balena-cloud.com:gh_janvda/pi3three.git'

Apologies, but I have trouble replicating this issue. Could you try again and let me know if the issue is still present?

I want to deploy code in a local branch (which is different from master) to balena.
Note that deploying code in the master branch works for me (just tested it).

I have tried again deploying code from my local branch dev_pulseaudio and I am getting same error.

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)
$ git branch
* dev_pulseaudio
  master

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)
$ git status
On branch dev_pulseaudio
Your branch is ahead of 'origin/dev_pulseaudio' by 7 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)
$ git remote -v
balena  gh_janvda@git.balena-cloud.com:gh_janvda/pi3three.git (fetch)
balena  gh_janvda@git.balena-cloud.com:gh_janvda/pi3three.git (push)
origin  https://github.com/janvda/balena-google-assistant.git (fetch)
origin  https://github.com/janvda/balena-google-assistant.git (push)

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)
$ git push balena dev_pulseaudio:master
Enter passphrase for key '/c/Users/IBM_ADMIN/.ssh/id_rsa':
Counting objects: 21, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (21/21), 1.94 KiB | 153.00 KiB/s, done.
Total 21 (delta 13), reused 19 (delta 11)
error: unable to write sha1 filename ./objects/28/c24232ab573f0eb00f9704f835c6b3494a2c25: Permission denied
error: unable to write sha1 filename ./objects/0e/bcd597a229f2255ebd4aeb5fc7bae3eafbff7c: Permission denied
To git.balena-cloud.com:gh_janvda/pi3three.git
 ! [remote rejected] dev_pulseaudio -> master (unable to migrate objects to permanent storage)
error: failed to push some refs to 'gh_janvda@git.balena-cloud.com:gh_janvda/pi3three.git'

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)
$ git push balena dev_pulseaudio:master --force
Enter passphrase for key '/c/Users/IBM_ADMIN/.ssh/id_rsa':
Counting objects: 21, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (21/21), 1.94 KiB | 79.00 KiB/s, done.
Total 21 (delta 13), reused 19 (delta 11)
error: unable to write sha1 filename ./objects/28/c24232ab573f0eb00f9704f835c6b3494a2c25: Permission denied
error: unable to write sha1 filename ./objects/0e/bcd597a229f2255ebd4aeb5fc7bae3eafbff7c: Permission denied
To git.balena-cloud.com:gh_janvda/pi3three.git
 ! [remote rejected] dev_pulseaudio -> master (unable to migrate objects to permanent storage)
error: failed to push some refs to 'gh_janvda@git.balena-cloud.com:gh_janvda/pi3three.git'

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)

Hey there!

What version of git are you running in your machine? Can you try reproducing the issue with another git versions, and potentially an environment other than MinGW? The fact that the permissions error messages are not prefixed by remote: makes me think these errors are coming from the repository in your machine (maybe something weird with MinGW?) rather than on the remote one on our servers.

Can you double check that your user has the necessary permissions to write files in .git/objects? i.e. maybe try:

touch .git/objects/28/c24232ab573f0eb00f9704f835c6b3494a2c25
touch .git/objects/0e/bcd597a229f2255ebd4aeb5fc7bae3eafbff7c

Thanks for the response.
As requested:

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)
$ git --version
git version 2.17.1.windows.2

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)
$ touch .git/objects/28/c24232ab573f0eb00f9704f835c6b3494a2c25

BE02506@20FM-PC0CP7KD MINGW64 ~/balena/active/balena-google-assistant (dev_pulseaudio)

I will retry later from different machine.

Hey there,

Cool, lets see how it goes. In the mean-time, can you link me to the repo you are using so I can test it from my side? I initially thought it was one of our projects, but I can’t seem to find any called balena-google-assistant

Sure, I have just pushed my local changes also to github = https://github.com/janvda/balena-google-assistant

Thanks a lot!

I tried to reproduce the exact same scenario as you, and it all seems fine on git 2.22.0 on macOS Mojave. I’ll continue doing some research but I still think it might be an issue on the client side of things (we’ll see what happens in another computer)

1 Like

I had the same problem this afternoon, I couldn’t push my code anymore to my application because of the same type of error.

When I created a new application, I could push all commits without any problem.
Eventually I ended up pushing my code - commit by commit - to the existing application.
One commit seemed to cause the error, I removed the commit from my history and recommitted the changes.

1 Like

Very interesting. Do you remember what was the problematic commit? It might be good to have it as a part of our test suite

just did the same on MacOS Mojave and I am having the same issue.

Jans-MBP:balena-google-assistant jan$ git push balena dev_pulseaudio:master --force
Enumerating objects: 23, done.
Counting objects: 100% (23/23), done.
Delta compression using up to 12 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (21/21), 1.94 KiB | 1.94 MiB/s, done.
Total 21 (delta 13), reused 21 (delta 13)
error: unable to write sha1 filename ./objects/28/c24232ab573f0eb00f9704f835c6b3494a2c25: Permission denied
error: unable to write sha1 filename ./objects/0e/bcd597a229f2255ebd4aeb5fc7bae3eafbff7c: Permission denied
To git.balena-cloud.com:gh_janvda/pi3three.git
 ! [remote rejected] dev_pulseaudio -> master (unable to migrate objects to permanent storage)
error: failed to push some refs to 'gh_janvda@git.balena-cloud.com:gh_janvda/pi3three.git'
Jans-MBP:balena-google-assistant jan$ git --version
git version 2.20.1 (Apple Git-117)
Jans-MBP:balena-google-assistant jan$ 

Hi,
just tried on Ubuntu 18.04 and it worked with the following sequence of events:

git clone git@github.com:janvda/balena-google-assistant.git
cd balena-google-assistant
git remote add balena thomasr@git.balena-cloud.com:thomasr/rpitest-empty.git
git push balena master 

worked and

git checkout dev_pulseaudio
git push balena dev_pulseaudio:master --force

worked too.
I wonder if it is something in your balena git repository. To check that (and get working again) you could create a new application and check in as above.
If that fails too the problem is likely in your system, if it succeeds something might be going wrong in your balena repo,

Regards
Thomas

It strongly looks like there is a problem with the 2 commits on my remote Balena git repository. So the problem is not at my end or specific to the changes I have made to my git repository.

See also the comment from @thiver who reported a similar issue.

I also created a new branch not including the 2 commits and that I could push without a problem to balena.

Hi,

As my colleague mentioned above, can you please share with us more details about the problematic commits? So we can identify the issue and add it as a part of our test suite.

Hello, I’m experiencing the same issue:

$ git remote add balena jef@git.balena-cloud.com:jef/playerrpi3dev.git
$ git push balena master
Counting objects: 26, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (23/23), done.
Writing objects: 100% (26/26), 40.57 KiB | 2.03 MiB/s, done.
Total 26 (delta 13), reused 12 (delta 2)
error: unable to write sha1 filename ./objects/d8/df1e5045bfa9c6d0010ec85af44ecd56e020c9: 
Permission denied
To git.balena-cloud.com:jef/playerrpi3dev.git
 ! [remote rejected] master -> master (unable to migrate objects to permanent storage)
error: failed to push some refs to 'jef@git.balena-cloud.com:jef/playerrpi3dev.git'

I’ve already tried cloning my “origin” repo again, and adding the “balena” remote, but that did not change anything. Also just experiencing this since this morning (just got in the office).

I did a bunch of changes to my docker-compose.yml file and suspected one of these was the culprit causing some git hooks to fail.
Pushing commit by commit, the failing commit was just a simple addition to an application method.

@nurama can you please do git log -l 20 and paste the output here?

@thiver, can you also please do git log and paste here the problematic commit? Including a couple of commits above and below the problematic one would also be useful.

commit 4a70ed7922a0d4ca44a8e2732c0a1e4e4b6b8e1d (HEAD -> master, origin/master, origin/HEAD)

Author: Dirk Moors dirkmoors@gmail.com
Date: Thu Jun 20 09:24:02 2019 +0200

Debugging chromium flags

commit f55b6cb38e2c97bd192401c287447c28346dd0d4
Author: Dirk Moors dirkmoors@gmail.com
Date: Thu Jun 20 09:20:36 2019 +0200

Debugging stability

commit 5b0869f6a0aaf2b08cb219151cfabdd3c6612716
Author: Dirk Moors dirkmoors@gmail.com
Date: Wed Jun 19 11:59:13 2019 +0200

Bugfix

commit a411904fdc742c99f2411d926e0e465f77d0729d
Merge: 201f94b 46556f2
Author: Dirk Moors dirk@nurama.com
Date: Wed Jun 19 09:43:24 2019 +0000

Merged in TEC-500/lg-display-controller (pull request #3)

Added LG display controller logic

Approved-by: Dirk Moors <dirk@nurama.com>

commit 46556f225b3c35410d6de504e6e38844daa12564
Author: Dirk Moors dirkmoors@gmail.com
Date: Wed Jun 19 11:37:43 2019 +0200

Added LG display controller logic

commit 201f94babcb7a0d26d4d635429489eb125d7ec36
Author: Dirk Moors dirkmoors@gmail.com
Date: Tue Jun 18 17:34:16 2019 +0200

Updated some styling for network state view

commit 8e5aaa70a365e3358380795c91ab0fcfcbf09b33
Author: Dirk Moors dirkmoors@gmail.com
Date: Tue Jun 18 10:45:28 2019 +0200

Debugging watchdog

commit d4ffc01e078c7f0e92c73fae21b56f5cdaf83477
Author: Dirk Moors dirk@nurama.com
Date: Tue Jun 18 08:23:00 2019 +0000

Merged in TEC-493/offline-behavior-templates (pull request #2)

TEC-493/offline behavior templates

* Debugging offline behavior

* Debugging offline behavior

* Debugging

* Added extra error handling

:

@nurama thanks. Can you tell us more about the time it first happened? Did you make new commits at that time? Have you pushed before?

Hi, I experienced the issue for the first time this morning, as I got in the office. My last successful push was somewhere around 16:00 CET yesterday… I’ve created a simple new commit on top of my previous commit, so no strange things there…
Is there some way to “reset” the “remote” git repo so I can push a new, clean version or is there something else I can do?