🐏 mogoz

Search

SearchSearch

Golang

Apr 20, 2025, 35 min read

tags : Programming Languages, Concurrency in Golang

Compiling, Linking and OS dependencies

Syscalls

How does go make syscalls?

  • Differs by Operating system
  • In Linux, because it has a stable ABI, it makes the syscalls directly skipping libc. w Linux, the kernel to userspace interface is stable(which syscalls use), but the in kernel interfaces are not stable.
  • Enabling or disabling CGO has nothing to do with whether or not go syscall goes via libc, the C code used via CGO ofcourse will use libc.
  • Go does create wrappers around some syscalls (TODO: Need to dig into this)
  • Support for different syscalls in different OS is incremental. Eg. If X syscall is available in OS A and B. Go might have support for X only in A as of the moment.

Places where go handles syscalls

Portability

  • syscalls are not portable by nature, they are specific to the system. We need to add a build tag for the ARCH and OS that syscall invocation is valid for.

Dynamic and Static Linking

Static linking

  • musl

    • Statically compiled Go programs, always, even with cgo, using musl
    • You can also statically link with musl, but note that musl lacks features that people might want to use non-pure Go in the first place. For example, musl does’t support arbitrary name resolvers, e.g. no LDAP support; it only supports DNS, just like the pure Go net package.
    • On the other hand musl does support os/user.

Dynamic linking

CGO

CGO is essentially utilizing C api calls to shared libraries exporting C interface. It is a tradeoff.

Using CGO

  • CGO_ENABLED=1
  • Some things are only available as C libraries, re-implementing that in Go would be costly.
  • CGO is also used in some parts of standard library. Eg. (net and os/user). It’s not a strict requirement though, you can use these packages w/o CGO and they’ll use stripped down version written in Go. But if you want the full thing, you have no other option than to enable CGO

Without using CGO

Cross Compilation

See Cross Compilation

Cross compilation in Golang

  • Unless you’re using a native cross compiler(eg. Clang, Golang Compiler), to cross-compile a program, you need to separately build and install a complete gcc+binutils toolchain for every individual arch that you want to target.
  • Which Go this is easy(cross compiler out of the box) + dependencies ensured to support.

Cross compilation and CGO

CGO allows us to access C libraries in the system we’re building/compiling on. It has no idea about C libraries of other systems. So mostly CGO is disabled by default if cross-compiling. However, if you need to cross-compile go code with CGO, you need a cross-compiling C compiler for the target machine. It can be done but it is a bit of PITA.

  • Using Zig

    CGO_ENABLED=1 GOOS=linux GOARCH=amd64 CC="zig cc -target x86_64-linux" CXX="zig c++ -target x86_64-linux" go build --tags extended

Others

Packages and Modules

  • Read Go Modules Reference
  • module-aware mode is the way 2 go me fren. (GO111MODULE=""/auto)
  • ditch gopath

Meta notes

  • package path and module path may look similar, the difference lies in the existence of go.mod file inside the directory. i.e Repository root need not be the place where the module is defined.

Packages

  • package path / import path
    • Identity of a package
    • module path + subdirectory
    • Eg. golang.org/x/net/html
  • Each directory that has Go source code inside, possibly including the root directory, is a package.
  • Example: x/... matches x as well as x’s subdirectories.

Module

  • The tree(module) with branches(packages) and leafs(*.go files) growing on branches.
  • packages sharing the same lifecycle(version number) are bundled into a module.
  • module path
    • Defined in the go.mod file by the module directive
    • Identity of a module
    • Acts as a prefix for package import paths within the module.
    • Eg. golang.org/x/net, golang.org/x/tools/gopls

Semantic versioning & versions from VCS

  • A version identifies an immutable snapshot of a module. Each version starts with the letter v + semantic versioning.
  • v0.0.0, v1.12.134, v8.0.5-pre, v9.2.2-beta+meta and v2.0.9+meta are valid versions.

VCS and pseudo versioning

  • We can also get modules from VCS using tags/branches/revisions/commits that don’t follow semantic versioning.
  • In these cases, the go command will replace golang.org/x/net@daa7c041 with v0.0.0-20191109021931-daa7c04131f5.
  • This is called pseudo-version. You usually won’t be typing a pseudo version by hand.

Why separate directory for Major versions

Golden rule: If an old package and a new package have the same import path ⇒ The new package must be backwards compatible with the old package.

  • v0 / pre-release suffix: Unstable, doesn’t need to be backwards compatible. No major version suffix directory allowed. So when starting new projects be under v0 as long as possible.
  • v1 : Defines the compatibility/stability. No major version suffix directory allowed.
  • v2 / v2+ : Since major version bump by definition means breaking changes, by the golden rule, we need it to have separate module path.

Building

What happens when go command tries to load a package?

  • When we try to load a package, indirectly we need to find the module path
  • It first looks into the build list, if not found it’ll try to fetch the module (latest version) from a module proxy mentioned in the GOPROXY env var.
  • go tidy / go get does this automatically.

Generating build list

  • When we run the go command, a list of final module versions is prepared from the go.mod file of main module + transitively required modules using minimal version selection. This final list of module+version is used for go{build,list,test,etc}. This is the build list
  • // indirect : This is added to go.mod of main module, when module is not directly required in the main module. So you should have all the dependencies in the go.mod file.

Workspaces

  • New feature 1.18+
  • You’re not meant to commit go.work files. They are meant for local changes only.
  • Has use and replace directives that can be useful for scratch work

Module Proxy

  • module proxy is an HTTP server that can respond to GET requests for certain paths
  • We don’t have a central package authority in the vein of npm or crates.io. Go modules have no names, only paths. The package management system uses the package path/module path to learn how to get the package. If it can’t find the package locally, it’ll try getting it from a module proxy.
  • module proxy related vars: GOPRIVATE, GONOPROXY, GOPROXY="https://proxy.golang.org,direct"
  • Different module proxies can have their own conventions (Eg. gopkg.in has some diff conventions)

Access private packages is a PITA

Project organization and dependencies

Standard Library

Project structure

  • multi-module monorepos is unusual
  • multi-package monorepo is common

Language topics

Pointers

  • There’s no pointer arithmetic in go
  • Go guarantees that, thing being pointed to will continue to be valid for the lifetime of the pointer. 
    func f() *int {
        i := 1
        return &i
} // Go will arrange memory to store i after f returns.

Methods

  • In general, all methods on a given type should have either value or pointer receivers, but not a mixture of both.

Context

Signaling and Request cancellation

  • Example: a client timeout - > your request context is canceled - > every I/O operations and long running processes will be canceled too
  • It’s not possible for a function that takes a context.Context to cancel it
    • It could do is newCtx, cancel := context.WithCancel(origCtx).
    • It can listen for Done on that ctx and do something(usually cancellation of ongoing task) based on it.
    • Done is triggered
      • when cancel() on the ctx is called. (requires WithCancel)
      • also can be triggered based on use on WithTimeout and WithDeadline
    • When a Context is canceled, all Contexts derived from it are canceled.
      • Eg. when cancel() is called on newCtx, newCtx and all Contexts derived from it are canceled. (origCtx is NOT canceled)
      • Eg. when cancel() is called on origCtx, origCtx and all Contexts derived from it are canceled. (origCtx and newCtx are canceled)
  • context.Background() is never canceled.

Storing values

  • The storage of values in a context is a bit controversial. main use case for “context” is cancellation signals.
  • In the above example, newCtx will have access to the same values as origCtx
  • context.Value() is like Thread Local Storage (see Threads, Concurrency) for goroutines but in a cheap suit.

Other notes on context

  • Context is that it should flow through your program.

    • Imagine a river or running water.
    • Do pass from function to function down your call stack, augmented as needed. (Usually as the first argument)
    • Don’t want to store it somewhere like in a struct.
    • Don’t want to keep it around any more than strictly needed.

  • When to create context?

    • Good practice to add a context to anything that might block on I/O, regardless of how long you assume it might take.
    • Context object is created with each request and expires when the request is over. (request is general sense)
    • context.Background()
      • Use pure context.Background() ONLY to handle your app lifecycle, never in a io/request function.
      • Just passing context.Background() there offers no functionality.
    • context.WithCancel
      • In io/request functions, use something like context.WithCancel(context.Background()) because that’ll allow you to cancel the context.
      • Fresh context
        • Eg. context.WithCancel(context.Background()), context.WithCancel(context.TODO())
      • Derived context
        • Eg. context.WithCancel(someExistingCtx)
    • context.TODO
      • Adding context to program later can be problematic, so consider using context.TODO if unsure what context to use. It’s similar to using context.Background() but it’s a clue to your future self that you are not sure about the context yet rather than you explicitly want a background context.

  • Separation of context

    • General rule: If the work(i/o) you’re about to perform can outlive the lifetime of outer function, you’d want to create a fresh context instead of deriving from the context of the outer function(if there is one)
      • Eg. HTTP requests context are not derived from the server context as you still want to process on-going request while the app shuts down. (IMPORTANT). You’d use http.Request.Context()
    • Think clearly about the boundaries and lifetimes, don’t mess app context to handle async function, internal consumer or request etc.

  • Context package and HTTP package

    • You can get the context from http.Request with .Context(). It’s like this is because the http package was written before context was a thing.
    • Outgoing client requests, the context is canceled when
      • We explicitly cancel the context
    • Incoming server requests, the context is canceled when
      • The client’s connection closes
      • The request is canceled (with HTTP/2)
      • The ServeHTTP method returns

  • Context an Instrumentation

    • Instrumentation libraries generally use the context to hold the current span, to which new child spans can be attached.

Resources on context

Maps

  • Go Maps are hashmap. O(1) ACR, O(n) WCR
  • A Map value is a pointer to a runtime.hmap structure.
  • Since it’s a pointer, it should be written as *map[int]int instead of map[int]int. Go team changed this historically cuz it was confusing anyway.
  • Maps change it’s structure
    • When you insert or delete entries
    • The map may need to rebalance itself to retain its O(1) guarantee
  • What the compiler does when you use map 
    v := m["key"]     // → runtime.mapaccess1(m, ”key", &v)
v, ok := m["key"] // → runtime.mapaccess2(m, ”key”, &v, &ok)
m["key"] = 9001   // → runtime.mapinsert(m, ”key", 9001)
delete(m, "key")  // → runtime.mapdelete(m, “key”)
  • If you want a map where you only care about the key and not the value, we can do: set := make(map[string]struct{}) so it’s assigned to empty struct.

Embedding interfaces & structs

Embedding Interface

// combines Reader and Writer interfaces
type ReadWriter interface {
    Reader
    Writer
}

Embedding Struct

  • Embedding directly, no additional bookkeeping

    • When invoked, the receiver of the method is the inner type not the outer one.
    • i.e when the Read method of a bufio.ReadWriter is invoked, receiver is the inner Reader and not ReadWriter.
     
// bufio.ReadWriter
type ReadWriter struct {
    *Reader  // *bufio.Reader
    *Writer  // *bufio.Writer
	*log.Logger
}
 
// - the type name of the field, ignoring the package
//   qualifier, serves as a field name
// - Name conflicts are ez resolvable
var poop ReadWriter
poop.Reader // refers to inner Reader
poop.Logger // refers to inner Logger

  • Embedding in-directly, additional bookkeeping

    type ReadWriter struct {
    reader *Reader
    writer *Writer
}
func (rw *ReadWriter) Read(p []byte) (n int, err error) {
    return rw.reader.Read(p)
}

Error and panics

See How we centralized and structured error handling in Golang | Hacker News

  • recover only makes sense inside defer
  • defer can modify named return values
    • We cannot return from defer but can use of named return values to have similar effect. eg. have a named return called err and set err in the defer function after recover.

Creation

  • errors.New
  • fmt.Errorf
  • which one?

Error wrapping

https://lukas.zapletalovi.com/posts/2022/wrapping-multiple-errors/

you wrap to give context about what went wrong in the function you called. You do not wrap to say that the function you called failed.

  • using fmt like this: fmt.Errorf("failed to create user: %w", err) (Go1.13): Useful to add info context
    • fmt.Errorf can also be used independently ofc when not wrapping error
    • %w is meant for error arguments. (so this is the differentator)
  • Go1.20 introduced errors.Join and then make fmt.Errorf accept multiple %w. We can use either to join multiple errors. Eg. from goroutines.

  • Error wrapping in practice

    package main
 
import (
	"errors"
	"fmt"
)
 
// common HTTP status codes
var NotFoundHTTPCode = errors.New("404")
var UnauthorizedHTTPCode = errors.New("401")
 
// database errors
var RecordNotFoundErr = errors.New("DB: record not found")
var AffectedRecordsMismatchErr = errors.New("DB: affected records mismatch")
 
// HTTP client errors
var ResourceNotFoundErr = errors.New("HTTP client: resource not found")
var ResourceUnauthorizedErr = errors.New("HTTP client: unauthorized")
 
// application errors (the new feature)
var UserNotFoundErr = fmt.Errorf("user not found: %w (%w)",
    RecordNotFoundErr, NotFoundHTTPCode)
var OtherResourceUnauthorizedErr = fmt.Errorf("unauthorized call: %w (%w)",
    ResourceUnauthorizedErr, UnauthorizedHTTPCode)
 
func handleError(err error) {
	if errors.Is(err, NotFoundHTTPCode) {
		fmt.Println("Will return 404")
	} else if errors.Is(err, UnauthorizedHTTPCode) {
		fmt.Println("Will return 401")
	} else {
		fmt.Println("Will return 500")
	}
	fmt.Println(err.Error())
}
 
func main() {
	handleError(UserNotFoundErr)
	handleError(OtherResourceUnauthorizedErr)
}

    “What may not look obvious from such artificial code snippet is that errors declarations are typically spread across many packages and it is not easy to keep track of all possible errors ensuring the required HTTP status codes. In this approach, all application-level wrapping errors declared in a single place also have HTTP codes wrapped inside them.”

Error comparison

  • Value comparison
    • Eg. if err != nil {
  • Type comparison
    • Basically type assert but useful when we want to check error type
    • Eg. if nerr, ok := err.(net.Error) {
    • Eg. if errors.As(err, &nerr) { but need to intanciate first, var nerr *net.Error
  • Existence check
    • This can be used when you wrap errors
    • if errors.Is(err, RecordNotFoundErr) {

Error stacktrace

Aliases

  • type byte = uint8
  • type rune = int32
  • type any = interface{}

Interfaces

  • Interfaces are just description of what something should resemble, by the methods.
  • The implementation of the interface can be done by a struct, int, func anything. Doesn’t matter. You can define a method on a func, on a int just the same way you can define a method on a struct.

Interface and Pointers

From chatgpt

Certainly. The statement “interfaces in Go are already pointer-like” refers to how interfaces are implemented and behaved in Go. This is an important concept that affects how we pass and use interfaces in Go programs. Let’s dive deeper into this:

Interface Structure: In Go, an interface is represented internally as a two-word structure:

The first word is a pointer to information about the type stored in the interface (called the “type descriptor”). The second word is a pointer to the actual data.

This structure is often referred to as an “interface value”. Passing Interfaces: When you pass an interface to a function, you’re actually passing this two-word structure. Even though it looks like you’re passing by value, you’re effectively passing a pointer to the data. No Need for Double Pointers: Because of this implementation, you don’t need to use pointers to interfaces (like *pgx.Tx) when you want to modify the underlying data. The interface itself already contains a pointer to the data.

io stuff

Ben Johnson has great blogpost series covering these in good depth

  • io
    • Abstractions on byte-stream
    • General io utility functions that don’t fit elsewhere.
  • bufio
    • Like io but with a buffer
    • Wraps io.Reader and io.Writer and helps w automatic buffering
  • bytes
    • Represent byte slice([]byte) as byte-stream (strings also provide this)
    • general operations on []byte.
    • bytes.Buffer implements io.Writer (useful for tests)
  • io/ioiutil (deprecated)
    • Deprecated
    • functionality moved to io or os packages

io

  • Reading

    • Read
      • returns io.EOF as normal part of usage
      • If you pass an 8-byte slice you could receive anywhere between 0 and 8 bytes back.
    • ReadFull
      • for strict reading of bytes into buffer.
    • MultiReader
      • Concat multiple readers into one
      • Things are read in sequence
      • Eg. Concat in memory header with some file reader
    • TeeReader
      • Like the tee command. Specify an duplicate writer when reader gets read. Might be useful for debugging etc.

  • Writing

    • MultiWriter
      • Duplicate writes to multiple writers. Similar to TeeReader tho but happens when writing shit
    • WriteString
      • An performance improvement on Write on packages that support it. Falls back to Write

  • Transferring btwn Reading & Writing

    • Copy : Allocates a 32KB temp buff to copy from src:Reader to dst:Writer
    • CopyBuffer : Provide your own buffer instead on letting Copy create one
    • CopyN : Similar to copy but you can set a limit on total bytes. Useful when reader is continuously growing for example or want to do limited read etc.
    • WriteTo and ReadFrom are optimized methods that are supposed to transfer data without additional allocation. If available, Copy will use these.

  • Files

    Usually, you have a continuous stream of bytes. But files are exceptions. You can do stuff like Seek w them.

  • Reading and Writing Bytes(uint8) & Runes(int32)

    • ByteReader
    • ByteWriter
    • ByteScanner
    • RuneReader
    • RuneScanner
    • There’s no RuneWriter btw

bytes and strings package

Provides a way to interface in-memory []byte and string as io.Reader and io.Writers

  • bytes package has 2 types
    • bytes.Reader which implements io.Reader (NewReader)
    • bytes.Buffer which implements io.Writer
  • bytes.Buffer is OK for tests etc
    • Consider bufio for proper usecases w buffer related io.
    • bytes.Buffer is a buffer with two ends
      • can only read from the start of it
      • can only write to the end of it
      • No seeking

strings, bytes, runes, characters

  • Formal for loop will loop through byte in string but for range loop will loop through rune
  • string : Readonly slice of bytes. NOT slice of characters.
  • “poop” is a string. `poop` is a raw string.
    • string can contain escape sequences, so they’re not always UTF-8.
    • raw string cannot contain escape sequences, only UTF-8 because Go source code is UTF-8. (almost always)
  • Unicode
    • See Unicode
    • code point U+2318, hex val 2318, (bytes e28c98) represents the symbol ⌘.
  • character
    • May be represented by a number of different sequences of code points
      • i.e different sequences of UTF-8 bytes
    • In Go, we call Unicode code points as rune (int32).

Encoding

Encoding vs Marshaling

  • Usually these mean the same thing, but Go has specific meanings.
  • x.Encoder & x.Decoder are for working w io.Writer & io.Reader (files eg.)
  • x.Marshaler & x.Unmarshaler are for working w []byte (in memory)

Encoding for Primitives vs Complex objects

  • Primitive stuff

    • bytes
      • Text encoding(base64)/ binary encoding
      • encoding package
        • BinaryMarshaler, BinaryUnmarshaler, TextMarshaler, TextUnmarshaler
        • These are not used so much because there’s not a single defined way to marshal an object to binary format. Instead we have Custom Protocols which is covered w other packages such as encoding/json etc.
      • encoding/hex, encoding/base64 etc.
    • integers
      • encoding/binary, wen we needs endian stuff and variable length encoding
      • For in-memory we have ByteOrder interface
      • For streams we have Read and Write. This also supports composite types but better to just use Custom Protocols.
    • string
      • ASCII, UTF8
      • unicode/utf16, encoding/ascii85, golang.org/x/text, fmt, strconv etc.

  • Complex obj stuff

    • Complex objects is where Custom Protocols comes in
    • This is mostly about encoding more complex stuff like language specific data structure etc.
    • Here we can go JSON, CSV, Protocol Buffers, MsgPack etc etc.
    • In a sense, Database‘es also encode data for us.
    • Example packages: encoding/json, encoding/xml, encoding/csv, encoding/gob. Other external stuff is always there like Protocol Buffers.

More on encoding/json

Generics in Go

Current limitations

  • Co-variance

    You can’t use

    RecipeServiceClient[
  templatefill.StartRequest,
  templatefill.StartResponse,
  templatefill.StopRequest,
  templatefill.StopResponse,
]

    where RecipeServiceClient[any,any,any,any] is expected. They’re different types

    • ATM, generic methods aren’t supported (where the generic type is scoped to the method, instead of the parent type)
    • i.e you can’t have type parameters scoped to a particular method
    • i.e Go’s generics do not support covariance, which means that even though if T satisfies the any interface, Client[T] is not a Client[any], as these are distinct and mutually exclusive types and not interchangeable in either direction.

Type Conversion and Type assertion

Type conversion

Eg. interface{}(42), here we’re taking 42(which is an int) and converting it into type interface{}

Type assertion

package main
 
import "fmt"
 
type t int
 
//this will panic
// func main() {
//   i := interface{}(42)
//   _ = i.(t)
// }
 
// because we have two _, ok, this will not panic
func main() {
	i := interface{}(42)
	if _, ok := i.(t); !ok {
		fmt.Println("assertion failed")
		return
	}
	fmt.Println("not hmm")
}
  • Can be only done on interfaces. (Not limited to interface{} though, but has to be interface)

Type switch

Main hatch: switch c := v.(type)

// p := map[string]interface{}{}
p := map[string]any{}
err := json.Unmarshal(data, &p)
for k, v := range p {
    switch c := v.(type) {
    case string:
        fmt.Printf("Item %q is a string, containing %q\n", k, c)
    case float64:
        fmt.Printf("Looks like item %q is a number, specifically %f\n", k, c)
    default:
        fmt.Printf("Not sure what type item %q is, but I think it might be %T\n", k, c)
    }
}

cgo

Go Pointer, Pass to GoGo Pointer, Pass to C
Go codeYESYES, must point to C memory
C codeNO
  • Go’s pointer type can contain C pointers aswell as Go pointers
  • Go pointers, passed to C may only point to data stored in C

Other topics

Deep copy

  • There’s no default deep copy, if primitive types assigning is copy by value but if struct contains pointers etc, you’d need to implement the deep copy yourself

time.Ticker

If an operation in a select case takes longer than the ticker duration (5 seconds in your case), the ticker will continue to tick every 5 seconds, but you’ll miss those ticks while the operation is running. When the operation finally completes, the next case that reads from terminationCheckTicker.C will only get the most recent tick - any ticks that occurred during the operation are dropped.

Application supervison

NOTE: We’re running multiple services here, the idea is similar to erlang supervisor trees. See https://www.jerf.org/iri/post/2930/

But instead of using https://github.com/thejerf/suture, we’re using oklog/run as suture doesn’t seem to handle net/http (doesn’t fit in the model). The idea is similar to errgroup package aswell.

Application architecture

  • Accept interfaces(broader types)
  • Return structs(specific types)

Handler vs HandlerFunc

  • See HandleFunc vs Handle : golang
  • Anything(struct/function etc.) that implements the http.Handler interface
  • The interface has the ServeHTTP method for handling HTTP requests and generating response
  • Avoid putting business logic in handlers the same way you won’t put business logic into controllers
  • http.HandlerFunc is an example of a handler (of type function) which implements http.Handler
  • When we write functions that contain the signature of http.HandlerFunc we’ve written a handler function.

Logging and Error Handling

  • In short: log the error once, at the point you handle it.
    • Only log the error where it is handled, otherwise wrap it and return it without logging.
    • At some point, you will log it as either an error if there is nothing you can do about it, or a warning if somehow you can recover from it (not panic recover).
    • However your log record will contain the trace from the point where the error occurred, so you have all the information you need.

Testing

Note on testing.Log

  • When using t.Log
    • this debug info printed only on test fail, else with -v. If program fails to compile nothing runs

Go and Databases

See Organising Database Access in Go – Alex Edwards 🌟

Notes on using sqlc with golang

Postgres Gotchas

  • PostgreSQL DEFAULT is for when you don’t provide a column value in INSERT statement. If you provide NULL as a value it’ll be considered as a value and DEFAULT won’t apply.

Go Performance

Garbage Collection

Base heap size is like the “starting point” memory usage when your program has loaded and initialized, but before it starts doing its main work. Think of it as the program’s “idle” memory state.

In Go specifically:

  • It’s dynamically determined based on what your program needs at startup
  • Can vary between runs of the same program
  • Affected by factors like OS memory layout, system state, and program inputs
  • Used by Go’s garbage collector as a reference point (GC typically allows heap to grow to 2x this size)

See Garbage collection

  • By default, Go GC allows the heap to grow up to 2x its current size before triggering garbage collection
  • Example case
    • Allocated Mem for executable: 250M
    • Base heap: 170MB
    • Potential growth allowed: up to 340MB (2 × 170MB)
    • Available memory: only 80MB (250-170)
    • Now the GC won’t kick in unless we reach 340M, but we have only allocated 250.
    • In this case, we can set GOMEMLIMIT=200MiB, which will trigger the GC at 200 instead of waiting for the 2x number.

Template (base-go)

About the template

Why this template?

  • Remove friction of setup of new project
  • Remove friction of writing the initial tests
  • Suitable for: Basic starter, CLI stuff

Project layout

  • Inspired by this layout.
  • Essentials
    • cmd/ : executable commands
    • docs/ : human readable documentation
    • internal/ : code not intended to be used by others
      • Contains code that others(other things inside the repo aswell outside) shouldn’t consume.
      • It isolates things in a way that files under the same import path can use it, exposing it is out of question. (??)
    • scripts/ : any scripts needed for meta-operations
  • Additional
    • api/ : OpenAPI/Swagger specs, JSON schema files, protocol definition files.
    • configs/ : Configuration file templates or default configs.
    • tests/ : Integration tests w data etc. Otherwise, test code lives in the same dir that its code is written in.
    • tools/ : Supporting tools for this project.

Style suggestions and other notes on Go

Workflow

  • Running tests in watch mode w gosumtest
  • Using go run instead of go build (use with entr): eg. git-entr go run main.go --some-flag

  • Debugging (dlv)

    • Usually use dlv test when you have a test file if you don’t have a main func.
      • You cd into the directory where the test is and then
        • dlv test -- -test.v or simply dlv test
    • funcs : lists all functions. funcs [some_func] filters
    • setting breakpoints
      • break [func_name]
    • Location specifiers: https://github.com/go-delve/delve/blob/master/Documentation/cli/locspec.md
    • Use p to print symbols etc.

  • Logging

    • Question: Is it okay to mix slog and log in same package? Eg. in some cases i’d like to avoid slog+osExit and simply use log.Fatal
    • slog doesn’t pass values by map unlike say logurus, which makes this a little bit weird but passing maps has performance implications but you could use attrs if you want.
    • Opinions on logging levels
      • My opinion
        • I think when using structured logging, using these levels has some merit unlike what the author suggests. Color coding, easy filtering etc.
      • Warning: It’s either an informational message, or an error condition. (prefer not using)
      • Fatal: In golang, it’s similar to panic. Better to avoid it only. (prefer not using)
      • Error: There’s no point in logging the error, either handle it or pass it up the stack. (prefer not using)

Basics

  • Practical Go: Real world advice for writing maintainable Go programs
  • Naming & Comments
    • Overall Docs should answer why not how, code is how, variables are code
    • Variable & Constants
      • Name: Should define its purpose(How). Do no mention the type in the name.
      • Comment: Should describe the contents(Why) not the purpose(How)
    • Getter&setters: Owner instead of GetOwner, SetOwner is fine.
    • Interface names have a -er
      • If methods of interface has name such as Read, Write, Close, Flush, String, try using the canonical signatures.
      • If the functionality of a method is same as provided by std lib, keep name same. Eg. If method prints sring representation, call it String instead of ToString
    • Prefer single words for methods, interfaces, and packages.
  • Global variables become an invisible parameter to every function in your program!
  • Using Must as a prefix for function or method names to indicate that the operation is expected to succeed or panic

  • Organizing files

    • All Go code is organized into packages. And we organize code and types by their functional responsibilities.
    • Keep things close
      • Keep types close to source, pref. in top of the file.
      • Code should be as close to where it’s used as possible. It might be tempting to put things into repo-root/internal but if it makes better sense, put it close to the source, maybe in its own /internal.

  • About main

    • 3 things: package main, main() and main.go
    • Executable program you need an entry point
      • package main + main() func is the entry point
    • The file with the main() inside the package main, is conventionally named main.go. But can be called anything else. But good to follow the convention.
    • i.e. Have a file(that we want to create an executable out of) inside a directory, name it main.go + package main + main() func. This will give us an executable when go build is run on it.
    • main packages are not importable, so don’t export stuff from it.
    • If your project is just a package and not an executable, it doesn’t need a main(), hence doesn’t need a main package at all.

  • About init()

    • Special function that is executed before main(), usually to perform any initialization tasks
    • Can be defined in any package, multiple such functions can be defined in the same package
    • Nothing related to being an executable unlike main()

Packages in Go

  • Creating packages

    • Go only allows one package per directory
    • Package Name/Path
      • What it provides, not what it contains.
      • No plurals for package name. don’t name a package httputils, name it httputil
      • Avoid overly broad package names like “common” and “util”.
    • Enforce vanity URLs. It ensures this package can only be imported via the mentioned path, even if the other url is serving it. 
      package datastore // import "cloud.google.com/go/datastore"

  • Exposing packages

    // from xeiaso
repo-root
├── cmd
│   ├── paperwork
│   │   ├── create // exposed to other parts of the module and outside
│   │   │   └── create.go
│   │   └── main.go
│   ├── hospital // not exposed cuz internal
│   │   ├── internal
│   │   │   └── operate.go
│   │   └── main.go
│   └── integrator // not exposed
│       ├── integrate.go
│       └── main.go
├── internal // not exposed
│   └── log_manipulate.go
└── web // exposed to other parts of the module and outside
    ├── error.go
    └── instrument.go
    • Things you don’t want to expose
      • Only internal use : /internal (subdirectories can have their own /internal)
      • Other things with main.go don’t get exposed as it’s typically used as an entry point for executable programs.
    • Things you want to expose
      • If web is used all over your package. (/repo-root/web)
      • From subdirectories/subpackage, close to the code. (repo-root/cmd/paperwork/create)
      • If exposing a package to users, avoid exposing your custom repository structure to your users. i.e Avoid having src/, pkg/ sections in your import paths. So stick to the two points above^

  • Vendoring

    • Why vendor?
      • I sometimes work offline, so vendoring is important for me
      • Helps to ensure that all files used for build are in a single file tree.
      • No network access we can build stuff always (we supposed to push the vendor to vcs)
    • Default behavior is, if vendor/ directory is present, the go command acts as if -mod=vendor otherwise -mod=readonly. I think sane defaults.

    • module cache

      • When -mod=mod (This is what go get and go mod tidy does, doesn’t need the flag)
      • Downloaded modules are stored in GOMODCACHE and made read-only. This cache may be shared by multiple Go projects developed on the same machine.

    • vendor

      • When -mod=vendor, go command will use the vendor/ directory
      • Will not use the network or the module cache
      • Useful things to know about vendoring
        • Local changes should not be made to vendored packages. Workspaces can probably help here.
        • go mod vendor omits go.mod and go.sum files for vendored dependencies
        • go mod vendor records the go version for each deps in vendor/modules.txt
        • go get, go mod download, go mod tidy will bypass vendor directory and download stuff as expected

Testing

  • Test code usually lives in the same dir as the code with <file>_test.go
  • fmt.X work inside tests but it’s not supposed to be used there, also it’ll format itself weird in the output. For logging in tests, use t.Log etc.
  • Assertion are not popular in go and I do not plan to use them, but if I need, there’s testify.
  • You don’t want to test private functions, those are implementation details. Better focus on testing the behavior.
  • If the test has > 3 mocks, might be time to reconsider code

  • Additional testing helper packages

    • unit tests & mocking : Go’s testing framework and dependency injection via interfaces
    • Acceptance tests: Black box test/Functional tests
      • These are usually separate packages or somthing that run against the running shit
      • Usually great for working w legacy codebase or unknown ones
    • github.com/approvals/go-approval-tests : For goldens

  • Service lifecycle

    • Waitgroups, errgroup or oklog/run.
    • The later 2 are alternatives trying to improve waitgroup’s interface. Choose based on preference.

  • Responses

    • We need to set w.Header() before calling w.WriteHeader
      • first w.Header().Set("Content-Type", "application/octet-stream")
      • then w.WriteHeader(http.StatusOK)
      • otherwise you are making change after headers were written to the response and it will have no effect.

Databases

  • Interface

    • These interfaces do need a driver to work with
    • database/sql
      • See Go database/sql tutorial
      • Basic usage
        • Write a query, pass in the necessary arguments, and scan the results back into fields.
        • Programmers are responsible for explicitly specifying the mapping between a SQL field and its value
      • sqlx
        • Extensions on go’s standard database/sql library. (superset of database/sql) Eg. allows you to avoid manual column <-> field mapping etc.
        • sqlx work only with the standard interface and not the native interface of pgx. So sqlx can be used with pgx with stdlib compatibility.
    • ORMs

  • Drivers

    • pgx (for postgres)
      • If needed, it still can be used with database/sql, sqlc, sqlx etc. But usually, if you’re only dealing with postgres, you don’t really need these additional layers. Simply pgx’s native interface, it can handle most things.
      • pgx’s native interface works with scany
      • It has postgree specific optimizations that are impossible in the standard driver. (for example support for Native Postgres types: arrays, json etc)
      • Alternatives are lib/pq etc. but pgx is good.

  • Pooling for postgres

    • Why pgxpool?

      • Usually it’s preferred to go with server side pooling than to even worry about client side pooling.
      • But if you’re using pgx in a concurrent application, eg. webserver where each web session will create a new database connection then you must you pgxpool whether or not you’re using pgbouncer on the server side because pgx.conn by itself is not threadsafe, i.e multiple goroutines cannot safely access it.
      • With pgxpool is you can have long standing connections with db and pgxpool also tries to restablish connections if they are closed due to environment factors. You don’t get it with plain pgx.Conn
      • By default pgx automatically uses prepared statements. Prepared statements are incompaptible with PgBouncer. This can be disabled by setting a different QueryExecMode in ConnConfig.DefaultQueryExecMode.

Patterns

See Design Patterns

Go and sqlite

  • have a dedicated object for writing to the database, and run db.SetMaxOpenConns(1) on it. I learned the hard way that if I don’t do this then I’ll get SQLITE_BUSY errors from two threads trying to write to the db at the same time.
  • if I want to make reads faster, I could have 2 separate db objects, one for writing and one for reading

Performance

Concurrency stuff 🌟

Graph View

GolangProgramming LanguagesConcurrency in GolangnsswitchCross CompilationThreadsConcurrencyUnicodeCustom ProtocolsProtocol BuffersDatabaseContext Free Grammar (CFG)ParsersDesign PatternsPostgreSQLGarbage collectionsqliteLoggingNotes of pprofGeneral ProgrammingCppBit ShiftingWebAssemblyOCI Ecosystemtailscale & cloudflaregrpc, protocol buffers and friends

Backlinks