Project structure and versioning

General
1

Hi all,

I believe that having a simple project structure, combined with a versioning system and good changelog is very important. However, it’s very hard to set it up right. Looking around on GitHub, most (big) repositories have fixed this one way or the other. And most Balena repositories also have a nice versioning system with automated changelog. But how do you set it up, what to use etc.
I’m probably not the only one struggling with these questions, so I thought, why not create a topic about it.

First, the project structure. I made some projects in the past for Balena and setting up the structure at the beginning of a project is always a real struggle for me. Do I use the same as last time? Did I like that structure? Can I improve it this time?

Most of the time I’m creating multi-container projects for Balena and Node.js or Golang projects to run on servers. For the Node.js and Golang projects, I’m trying to use the basics they’ve created for those projects. But for a multi-container Balena project, you can have Node.js, Golang and other containers inside one project. What’s the “best-practice” to set this up and attaching a version to it?

My structure is as follows:

- docker-compose.yml
- README.md
- .dockerignore
- .gitignore
- container-1-directory (e.g. webservice)
  - Dockerfile.template
  - entry.sh
  - directory with contents (e.g. Node.js)
- container-2-directory (e.g. redis)
- container-3-directory (e.g. frontend)

The structure is ok (imo), but should I split the containers to repositories and version them there, keep the commits there and use git submodules in the “main” repository? The main downside is that the changelog of the whole project is divided over multiple repositories instead of in 1 place.

I’m curious what the structure is of other Balena users!

Next up, versioning.

I saw Balena made their versioning system open source, which is awesome! But is this still the versioning system you guys use in all of your projects?

I have some questions about how other people version their projects.

  1. Do you use automated versioning? And if so, what do you use?
  2. Where do you begin when setting up versioning in a project / repository?
  3. How do you (automatically) validate if commits are using the right formatting? (like, add Change-type or Changelog-entry to the footer)
  4. Do you use Pull Requests to create versions and automated changelogs or commits?
  5. Any tips / tricks to begin with versioning your project / repository?

Like I said, I’m mostly using Node.js and Golang for projects, but versioning is useful for all of your repositories. So I’m looking for a way to implement a versioning system that can be used in all repositories, with automated changelogs.

I’d like to thank everyone in advance who’s willing to read all of this and help me (and others) out!

Regards,

Bart

1 Like
6

Hi Bart,

This subject could be a full semester course in college. :laughing:

I’ve supported development teams for quite a few years, and with every new project, the structure and versioning are always a challenge to set up, because they are subject to the needs of the project, and depend on literally everything about it:

  • How the team is set up and where they are.
  • What “stuff” will go into, and come out of the project.
  • Whether the team has specific needs around who can access what parts of the project (ask me what it’s like to develop for the US military).
  • Capabilities of the tools that will be used in the project’s development.
  • The specific needs around build, code review, integration, testing and deployment.
  • Concerns around breaking up the project into parts that can be handled individually if necessary.
  • Details around how developers will commit and create PRs (or not create PRs).
  • How the team wishes to present project progress.

Ideally, the resulting structure and versioning will be easy to manage and make sense to the dev team, end users and third parties. If it’s cumbersome or doesn’t make sense, you’re doing it wrong. There is good motivation to not over-complicate simple projects. And there must be allowance on large projects for some complexities that are necessary for quality management.

A balena application could be as simple or complex as any other I’ve encountered in my life. So, it’s hard to say that you should keep it super-simple. But here are some guidelines off the top of my head that I think will get you close to a great project setup.

  • Breaking things up: If you need parallel dev efforts that affect different large componenets of the project, it might make sense to break things up into different repositories. But keep in mind that their histories will not line up, and you will need to somehow join them through versioning or clever build techniques. On small projects, it doesn’t make sense at all - just use one repo, and organize the folders. The teams who develop for Amazon’s internal systems and AWS all treat every other Amazon team as an external system. Talk about breaking things up!
  • Pull requests: If you have one to three developers, you may not even need to use pull requests. But any more than about three developers and things will get crazy fast, and you will find bugs caused by botched merges, and other issues. Also, if you have to provide any reporting or accountability on your progress, using PRs will help provide solid info on that.
  • Automated versioning: Honestly, I’ve never used any kind of automated versioning, even on very large projects. Clearly balena uses it, and it seems to work quite well. So, it could be a useful tool. But it’s something you would need to set up and somehow “attach” to your repo and/or build system.
  • Git sub-modules: On every project I’ve ever supported, that used git sub-modules, everyone hated it, and we eventually got rid of them. The required manual overhead of keeping the sub-module commits correct within each branch gets to be a nightmare when there are more than a few branches in the repo. Developers can easily screw up sub-module commits, and many don’t understand how to properly use them in their local development environment. Also, it may limit what build tools you can use, because some can’t handle sub-modules (e.g. The balena build infrastructure does not support git sub-modules, IIRC). There are plenty of use cases where sub-modules are a great tool. But I have not seen it in web nor IoT development ever.
  • Branching strategy: This has proven to be a very impactful issue in most projects. And I have made a conscious decision to only allow the “gitflow” [1] branching strategy on projects at my work. There are other ways to manage branches, which are useful for some cases and team structures. But gitflow has proved itself to be the overall best throughout my career so far.

A lot of what you asked above are things that can be changed if you don’t like how it’s going. So don’t feel like you have to get it perfect right out of the starting gate. And chances are you will need to shift at some point in the future anyway, due to the project becoming more or less complex, or just changing directions.

[1] https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow
[2] https://semver.org/

Cheers!
Mark-

2 Likes
7

Hi Mark,

Thanks for your detailed response, it changed my perspective about some things!
At the moment, we’re a small development team, so using 1 repository per Balena project seems to make more sense. I’ll look into a course about CI/CD and versioning.

I’m still a fan of versioning and making a changelog. But the ‘automation’ is not a must. My main concern is forgetting to do some basic tasks, like adding a tag when a new release is published, so we can go back in the commits which things changed from one release to another. Or having a format for commits / branches.

I’m also fan of the ‘gitflow’ workflow, but after some time, not everyone (including myself) sticks to it. Or you push some changes in a feature/* branch what’s supposed to be a bugfix. So our team has to be more strict about these rules probably.

Your post definitely helps to give some perspective about when to use what! I know it’s not set in stone, but having someone else’s opinion is very helpful!

1 Like