git

tags : General Programming

Workflow §

• Watch How to be a git expert - YouTube

Features §

git subtrees §

• An alternative to git submodules, I used it to have anotehr private fork of benthos repo but issue is it becomes little complicated to maintain with actual upstream.

# NOTE: Manual steps
#       git remote add geekodour-benthos git@github.com:geekodour/benthos.git
#       git subtree add --prefix services/benthos geekodour-benthos main --squash
# NOTE: Probably should have used --squash when adding
service-benthos-upstream-fetch:
	git fetch geekodour-benthos main
	git subtree pull --prefix services/benthos geekodour-benthos main --squash
 
# NOTE: Force pushing subtree
# 		subtree command does not have a force push thing on it. We need to do it manually:
#		git subtree split --prefix=services/benthos -b temp_branch
#		git push geekodour-benthos temp_branch:main --force
#		git branch -D temp_branch
service-benthos-upstream-push:
	git subtree push --prefix services/benthos geekodour-benthos main

Git branching strategy against Gitflow §

How do we unusually use dev branch? §

Usually don’t think clearly what’s the usecase of the dev branch, its a relic from gitflow branching strategy but we end up using it for deployment pipeline as a way to deploy to development env(they don’t have to be coupled!), pre-prod testing(eg. always goes to dev before main kinds), sandbox testing(just want to try out this feature and see how it works etc.)

Do we really need develop branch? §

• When we use dev, what we essentially end up having develop and main as two forks of the codebase. And these two forks go out of sync due to whatever reason.
  • Eg. You tested the change on develop, you made a PR to main, you made some micro adjustments to the PR before it made its way to main. Boom! out of sync, We pushed something to develop and never cherry-picked it to main. etc. etc.
• The develop branch not only becomes a long lived feature-branch, it becomes a long-lived feature-branch that everyone keeps changing without any guard-rails.
• It also has a more fundamental flaw in a way that “if we need develop and main in sync”, why are we merging something to develop which is also not ready for main.

What would make this simpler? §

One line answer: better git branching strategy(trunkbased) + IssueOps + invariants/development practices + branch protection rules

Better git branching strategy §

• Instead of fighting develop we get rid of develop.
• There’s just one primary branch, and it’s main.
  • main gets deployed to production-environment, main gets deployed to development-environment
  • Everything else are short lived branches which come as PRs and get deleted as PRs get reviewed and merged/closed.
  • If the PR change needs its changed to be on development-environment, we can simply deploy the changes of the PR to the development-environment directly from the PR. This allows review a PR while its live on the development-environment.
• There are certain invariants and practices which allow such a flow to happen at teams of different sizes. (Eg. you might wonder, well I have 2 PRs that I need to test in development-environment how do I do that? these question are addressed by the certain invariants and practices)

• What’s under the hood? (More on the branching strategy)

  • See this reference on branching strategy
  • What we’re currently following is a stripped down version of the gitflow model. (The author themselves suggests using better alternatives, see linked article)
  • What I am suggesting here is the trunkbaseddevelopment(TBD). There’s also GithubFlow model which is a subset of TBD.

IssueOps §

IssueOps is just a buzzword for being able to trigger operations from github issue/PR comments. In this case, we wish to deploy changes of a PR into development-environment without merging our changes into main because we want to test it out. We just comment something like /deploy to develop and it’ll provision the development-environment with the changes of our PR.

• post-merge vs pre-merge deployments

  Usually, we do deployments after merge. But if it’s for testing, deployments before merge make more sense if the environment allows it. IssueOps lets us do pre-merge deployments

  • post-merge: merge to main > deploy production-environment > something broke > open a PR > roll back > pass checks/approvals > merge again
  • pre-merge: deploy on PR development-environment > something broke > fix in PR > deploy on PR … > merge to main

• What about multiple PRs

  • In certain cases, teams try to provision an entire ephemeral preview environment. Which solves if Emiway and KRSNA both try to see their changes on top of main. The deployment infra will spin two different environments and give both of them separate ephemeral links.
  • In most cases, ephemeral environments are not possible due to the nature of our environments. We resort to bathroom etiquette and let people pee in peace. We use locks.
  • Once we /deploy if we’re not done testing our changes, we can /lock any other deploys to development-environment. People can come /unlock it and then deploy their changes once they feel they need to test their changes. If you just /deploy and don’t /lock then their /deploy will simply override your changes if they were not merged to main yet. (This tooling is very nicely provided with branch-deploy)

• Decoupling branch to environment

  • We don’t want to map git branches to deploy environments. We’ve seen the problems it has caused us.
  • Additionally it just allows more ways for production-environment to drift from development-environment
  • We just have one branch main, that gets deployed to all of our environments.
  • With IssueOps, we can test the changes of the PR on top of main and see those changes in the environment that we want.(in our case, development-environment)

Invariants/development practices (for TBD) §

There are additional practices/tooling that makes TBD more realistic. These are written in details in the TBD website.

• PR: TDB doesn’t mandate using PRs, you could push direct to trunk(main). But that’s only okay for very small teams. We need to be using ONLY PR based workflow with shortlived-branches.
• Ready to release: main should always be in a ready to go to production state. Your PR changes must take that into account.
• Feature flags: A way to work against main for features that are bigger than one PR.
• Commit sizes: small incremental commits, tested and rolled out. Ideally should be as small as possible while still being “whole”.
• Others
  • Features writing practices such as branching by abstraction, few more abstract practices we need to keep in mind when writing our code.
  • Since we’ve only main, now we won’t be making PRs against main or develop, we won’t be making PRs against development-environment or production-environment. This is a little mindshift in how we do development.

• More on Feature flags

  • For features that take longer to implement we can use feature toggles.
  • This way, we don’t need to keep track of what features is in main, if it’s still getting tested, we would have a feature flag active on development-environment and it’ll be live there while production-environment will not have it even if the feature is there in main (guard-rail by the feature flag)
  • If we’re writing unit tests, we need to make sure we write tests for feature regardless of they’re getting enabled in any environment.
  • Links: Feature toggle, Feature Toggles (aka Feature Flags), Testing code that uses feature flags

Rollout strategy for IssueOps §

Maybe a mix of issueOps and a bit post merge deploy magic

• Use issueOps to deploy to development-environment
• Use merge to main as a trigger for deploy to production-environment
• Or we can go full overboard and use issueOps completely for both development-environment and production-environment as-well as keeping deploy-proudction-on-merge-to-main

Using branch-deploy §

• We could something that allows deploying main to development-environment. We could try different tooling or even write our own but Github already has a very nice tool as an action: branch-deploy
• It uses combination of github environments, github actions, branch protection rules, and github issue commands to allow us with a nice flow to deploy main into environments we want to deploy our change to.
• Gotchas about branch-deploy
  • Required reviewers is Enterprise feature for private repos, so if you have production env specified for branch-deploy anyone will be able to deploy to production-environment if their PR is approved.
    • Required-reviewers feature needs the PR to be approved by certain
  • We keep deploy to production on merge and use branch-deploy only to deploy branch changes to development env.

Other notes §

dev/prod deploy vs branch §

• Ideally we should not have production deploy/ci scripts in dev branch at all if branch per env is what we’re doing
• In env<>branch mapping ideally, we’d have a poc and production branch which is not the case. So we’re operating in a mixed mode which is not ideal.

What about schema change? §

• For databases, we expect the data to drift massively in development-environment and production-environment but not the schema.
• Database schema changes are tricky because unlike code, they may not be same in different environments as these changes are persistent unlike code. Changes can go from outside, changes can be deployed from a PR which did not land on main but because we deployed to development-environment it altered the schema etc.
• Make sure the schema change can be rolled back. (Use UP/DOWN convention that the respective migration library provides)
• You want to be extra careful with these changes. If you’re making a schema change that is not going in main, maybe rollback the schema change using deploy commands.
• If ever schema migration conflicts occur between environment, resolving them is a manual process dealing with the DB. with newer migrations fixing the broken migration etc.
• Hopefully this will only ever be a rarity.

Do we want to cut releases? §

• I think we only do it for martini at the moment
• But in most of our setups we don’t really need to do the releasing ceremony. Whatever is in main is expected to be live in production and life is good.

Github repo stuff §

Actions to be taken for each repo §

• Enable “Automatically delete head branches” after PR merge
  • This will delete short lived feature branches after PR is merged
  • If existing repo, scrub any unused branches.
• Add .gitmessage

Branch protection to be added §

Need it?	Description
YES	Require a pull request before merging, w min 1 required approval
YES	Dismiss stale pull request approvals when new commits are pushed
YES	Require status checks to pass before merging
Optional	Require branches to be up to date before merging*
Preferred	Require linear history (No merge commits!, only ff commits)

On “Require branches to be up to date before merging”, we don’t really need it at the moment so optional

• Eg. When some other merged PR added a new test case which this change might fail with
• Can be automated using a merge queue but we don’t have a strong usecase for this

git message §

Add .gitmessage so that the following automatically is populated when we try to write a commit message See Better Commit Messages with a .gitmessage Template

# <type>(optional scope): <description>
# <type>(<opt scope>): <message in present tense>
# [opt body]
# [opt footer(s)]
#
# types: fix,feat,chore,docs, style, refactor, perf, test etc.
# scopes: api,lang etc.
#
# Breaking change:
# 1. The text "BREAKING CHANGE" anywhere in the commit message
# 2. "!" after type is breaking change. Eg. fix!: some description

# Types:
# - `feat`: new feature for the user, not a new feature for build script
# - `fix`: bug fix for the user, not a fix to a build script
# - `docs`: changes to the documentation
# - `style`: formatting, missing semi colons, etc; no production code change
# - `refactor`: refactoring production code, eg. renaming a variable
# - `test`: adding missing tests, refactoring tests; no production code change
# - `chore`: updating grunt tasks etc; no production code change
#
# See https://www.conventionalcommits.org/