Tools on Flavio Silva

🔥 Test DB integrations with Testcontainers

flavio1110@gmail.com (Flavio Silva) — Mon, 17 Apr 2023 14:00:00 +0100

Picture this, you have a critical and reasonably complicated piece of logic in your application that is handled in the database. Despite any change on it (or around it), you must have a 100% guarantee that piece continues to work just fine. So, what do you do?

Back in the ancient days…

Ancient cave art of many hand prints. (Credit: Petr Kratochvila/Shutterstock)

Before the “age of containers”, the solution would be a variant of the following:

Have a database with the schema aligned with the application’s version.
Ensure your CI tool has access to that database, to run the tests in your CI pipeline.
Create a script to arrange the necessary data for the test.
Write your testing using the DB.
Create a script to revert any DB changes performed by the test, such as inserts, updates, etc.

This is a very nice post about the Docker: Life Before and after.

What’s the problem with testing with a shared database?

Simply put, it’s complex, expensive to maintain, and has many other reasons to fail. For example:

The DB’s schema version is not the same as the application’s version you want to test.
There is a change in the network policy that blocks access to the database.
Someone’s test setup corrupted the data that you expected to have, so you arrange script fails, or the test fails because of missing data.

Because of the cost and flakiness of such tests, in many cases, the solution was to move logic from DB to the application’s code when performance was not a problem, or just manually test it from time to time and hope it doesn’t break because of an unforeseen reason (it-never-happened-in-the-last-30-minutes).

Containers everywhere

With the popularization of containers, not only developing, runnings, and deploying became easier, but also integrating and testing your external dependencies became easier, faster, and more dependable.

We can now effortlessly run not only the regular dependencies, like RDBMSs and Message Brokers, but we can emulate many cloud provider services using tools like localstack.

What about testing? Testcontainers to rescue!

Testcontainers makes it simple to create and clean up container-based dependencies for automated integration/smoke tests. The clean, easy-to-use API enables developers to programmatically define containers that should be run as part of a test and clean up those resources when the test is done.

It supports many popular languages/frameworks like Go, Java, .NET, Rust, and others. You can check the full list on its official site](https://www.testcontainers.org/) and because it’s open source, you check out and contribute to it in its repositories.

How to use Testcontainers on integration tests?

For this example, assume that one of the features of your software is to search for users based on a few criteria. Your users are stored in a PostgreSQL database and you query them using SQL. Something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


func searchPeople(ctx context.Context, db *sql.DB, params searchParams) ([]person, error) {
	query := `select first_name, last_name, city
		 from people
		 where ($1::text is null or first_name = $1)
		 and ($2::text is null or last_name = $2)
		 and ($3::text is null or city = $3)
		 order by first_name asc`

 var people []person

 //Execute query, check errors, populate people variable

 return people, nil
}

The search criteria will evolve, by adding new filters or adding new features like pagination. Therefore, we need to make sure that whenever we touch this query its existing behavior is not broken. Let’s write a test for it using Testcontainers.

Writing the test

To be able to run these tests against a real database, we need to:

Start the DB
Create the schema
Arrange data, Act, and Assert

Let’s see in detail how to perform each one of these steps.

1. Start the DB

Let’s create a function that will start a DB, return its connection string and a function to terminate it when we are done.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54


func startTestDB(ctx context.Context) (string, func(t *testing.T), error) {
	var envVars = map[string]string{
		"POSTGRES_USER": "user",
		"POSTGRES_PASSWORD": "super-secret",
		"POSTGRES_DB": "people",
		"PORT": "5432/tcp",
	}

	getConnString := func(host string, port nat.Port) string {
		return fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable",
			envVars["POSTGRES_USER"],
			envVars["POSTGRES_PASSWORD"],
			host,
			port.Port(),
			envVars["POSTGRES_DB"])
	}

	req := testcontainers.ContainerRequest{
		Image: "postgres:14",
		ExposedPorts: []string{envVars["PORT"]},
		Env: envVars,
		WaitingFor: wait.ForSQL(nat.Port(envVars["PORT"]), "pgx", getConnString).
 WithStartupTimeout(time.Second * 15),
	}
	pgC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: req,
		Started: true,
	})
	if err != nil {
		return "", nil, fmt.Errorf("failed to start db container :%w", err)
	}
	port, err := pgC.MappedPort(ctx, "5432/tcp")
	if err != nil {
		return "", nil, fmt.Errorf("failed to get mapped port :%w", err)
	}
	host, err := pgC.Host(ctx)
	if err != nil {
		return "", nil, fmt.Errorf("failed to get host :%w", err)
	}

	connString := fmt.Sprintf("postgres://%s:%s@%s:%d/%s?sslmode=disable",
		envVars["POSTGRES_USER"],
		envVars["POSTGRES_PASSWORD"],
		host,
		port.Int(),
		envVars["POSTGRES_DB"])

	terminate := func(t *testing.T) {
		if err := pgC.Terminate(ctx); err != nil {
			t.Fatalf("failed to terminate container: %s", err.Error())
		}
	}
	return connString, terminate, nil
}

The code above is self-explanatory if you are familiar with Go, but there are a few aspects that I’d like to highlight:

Testcontainers will map and assign a random port. Therefore, in this case, instead of connecting to 5432, it’s necessary to connect to the assigned port. This is similar to the hostname. Therefore, to get your correct connection string it’s needed to resolve port and host.
The container will automatically shut down when the application is finished. However, it is recommended to defer its termination as soon as it is started. For that, a terminate func is returned by the method above.

So far our test looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


func TestSearchPeople(t *testing.T) {
	ctx := context.Background()
	connString, terminate, err := startTestDB(ctx)
	if err != nil {
		t.Error(err)
	}
	defer terminate(t)

	db, err := openDB(connString)
	if err != nil {
		t.Fatal("fail to open DB", err)
	}
	// Continue
}

2. Create the schema

With a DB up and running we can initiate it with the expected schema, a.k.a migrate the DB schema.

In the real world, we would use something more sophisticated like go-migrate, but for the sake of simplicity let’s write something simpler.

1
2
3
4
5
6
7


func migrateDB(ctx context.Context, db *sql.DB) error {
	_, err := db.ExecContext(ctx, "create table if not exists people (first_name text,last_name text,city text)")
	if err != nil {
		return fmt.Errorf("failed to migrate DB: %w", err)
	}
	return nil
}

3. Arrange data, Act, and Assert

The last step before the actual test is to arrange the data. We could accomplish it together with the schema migration, however keeping these two parts separate helps to keep each test isolated, avoiding side effects when the data is tweaked.

For our tests, we need to insert 4 different people with distinct names and cities. Then we can use this data to act and assert the results. This is what the final test func looks like, testing every variation of our query.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


func TestSearchPeople(t *testing.T) {
	ctx := context.Background()
	// Start the DB
	connString, terminate, err := startTestDB(ctx)
	if err != nil {
		t.Error(err)
	}
	// Terminate the container when the func TestSearchPeople finishes
	defer terminate(t)

	// Open the the database
	db, err := openDB(connString)
	if err != nil {
		t.Fatal("fail to open DB", err)
	}

	// Migrate schema
	if err := migrateDB(ctx, db); err != nil {
		t.Fatal("fail to migrate DB", err)
	}

	insert := `insert into people values
			('Flavio', 'Silva', 'Olbia'),
			('Joost', 'Van Huis', 'Amsterdam'),
			('Aldben', 'Arimeritin', 'Istambul'),
			('Nando', 'Pelect', 'Perth');`

 // Arrange data
	if _, err := db.ExecContext(ctx, insert); err != nil {
		t.Fatal("fail to insert initial data", err)
	}

	// Several test cases
	t.Run("without filters", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{})
		assert.NoError(t, err)
		assert.Equal(t, 4, len(people))
	})

	t.Run("filtering by first name", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{firstName: strPtr("Joost")})
		assert.NoError(t, err)
		assert.Equal(t, 1, len(people))
		assert.Equal(t, "Joost", people[0].firstName)
	})
	t.Run("filtering by last name", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{lastName: strPtr("Arimeritin")})
		assert.NoError(t, err)
		assert.Equal(t, 1, len(people))
		assert.Equal(t, "Arimeritin", people[0].lastName)
	})
	t.Run("filtering by city", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{city: strPtr("Olbia")})
		assert.NoError(t, err)
		assert.Equal(t, 1, len(people))
		assert.Equal(t, "Olbia", people[0].city)
	})
}

With this test, we have the guarantee the covered scenarios are working. Profit!

You can check the results by running the tests.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


go test ./... -v
=== RUN TestSearchPeople
2023/05/09 15:48:55 github.com/testcontainers/testcontainers-go - Connected to docker:
 Server Version: 20.10.22
 API Version: 1.41
 Operating System: Docker Desktop
 Total Memory: 3932 MB
2023/05/09 15:48:55 Starting container id: 95dc2928dae3 image: docker.io/testcontainers/ryuk:0.3.4
2023/05/09 15:48:56 Waiting for container id 95dc2928dae3 image: docker.io/testcontainers/ryuk:0.3.4
2023/05/09 15:48:56 Container is ready id: 95dc2928dae3 image: docker.io/testcontainers/ryuk:0.3.4
2023/05/09 15:48:56 Starting container id: adedca079bc6 image: postgres:14
2023/05/09 15:48:57 Waiting for container id adedca079bc6 image: postgres:14
2023/05/09 15:48:58 Container is ready id: adedca079bc6 image: postgres:14
=== RUN TestSearchPeople/without_filters
=== RUN TestSearchPeople/filtering_by_first_name
=== RUN TestSearchPeople/filtering_by_last_name
=== RUN TestSearchPeople/filtering_by_city
--- PASS: TestSearchPeople (3.48s)
 --- PASS: TestSearchPeople/without_filters (0.00s)
 --- PASS: TestSearchPeople/filtering_by_first_name (0.00s)
 --- PASS: TestSearchPeople/filtering_by_last_name (0.00s)
 --- PASS: TestSearchPeople/filtering_by_city (0.00s)
PASS
ok 	github.com/flavio1110/go-for-csharp-devs/experiments/test-db-interactions	.724s

You can check the source code of this test on my experiments repository.

Final thoughts

Despite Testcontainers reducing the cost of integration tests and making them more reliable, it doesn’t remove the cost entirely. The tests will take more time to run (especially if where you are running the tests doesn’t have the image downloaded yet).

Nevertheless, in my opinion, the price is low compared to the benefit that it yields.

What about you? How do you write integration tests, and what are your main challenges?

🔥 Improve your Frontend skills with Frontend Mentor

flavio1110@gmail.com (Flavio Silva) — Tue, 11 Apr 2023 14:00:00 +0100

When studying HTML, CSS, and Javascript it’s hard to practice with real-world examples, so oftentimes we get stuck trying to come up with a good design and then building it.

Frontend Mentor to the recue!

Frontend Mentor offers front-end coding challenges and interesting projects to practice your HTML, CSS, and JavaScript. It is as close as it gets to work on a professional real-world project.

After creating your account, you can start one of its challeges. There are many free challenges that can get you started and give you plenty of things to play with. However, if you want take one step further, have access to premium challenges, figma and sketch design files, and private solutions, you can subscribe to the pro developer.

How does it work?

Once you start a challenge, you will download a zip for that challenge. For example the challenge Four card feature section, will give you a zip file containing the instructions, necessary images, basic design details, and the most important the design images for desktop and mobile!

These files are all you need to start hacking your front end!

After the challenge is complete you can submit it and get feedback from the community! That’s excellent!

I’ve been playing with some challenges myself, (the image of the banner is from my own implementation of the Four Cards feature section).

I’m not a Frontend specialist by no means, but you can find my solutions so far on my GitHub repository.

What about you? Do you know any other tool that offers such an immersive and near-real-world experience?

🐉 got: Embracing my lazyness with my custom git CLI

flavio1110@gmail.com (Flavio Silva) — Thu, 30 Mar 2023 14:00:00 +0100

got is a CLI written in Go, created on top of the git CLI, to make my life easier by shortening some commands I use daily.

Because it’s built on top of git, all git commands will also work just fine and I don’t need to keep switching between got and git!

If I want to clean up my local dead branches I can got rmb instead of git branch | grep -v "main" | xargs git branch -D (you can call me lazy. I accept that 😅).

xkcd Automation.

But… Why?

My first experience with a Distributed version control system (DVCS), was many years ago with mercurial hg, and despite taking some time to get used to the new way of working as compared to a centralized version control system, I got very used to its short commands, aliases, and simplicity. We could do something like hg pull or hg pul (that’s not a typo. This is an actual alias for hg pull), we were able to close a branch and commit a message in a single command like hg commit --close-branch -m 'closing this branch'. It was super nice and handy! Then, eventually, I started working with git and I was amazed by its differences, possibilities, and features. However, it felt more verbose for day-to-day tasks. So I ended up creating a bunch of aliases for the commands that I use more often, or commands that are longer and I always have to google it to remember. Sneak peek of my ~/.zshrc

1
2
3
4
5
6
7


#...
alias push='git push origin head'
alias stat='git status -s'
alias gbr="git branch | grep -v "main" | xargs git branch -D"
alias pick="git cherry-pick"
# and a few more...
#...

Speaking on zsh, if you use it with the go plugin, you will have a conflict with the alias got for go test.

The aliases work so well and are easy to maintain… But I wanted something fancier and all have the opportunity to go through the process of writing a CLI in Go.

How

Got is built using cobra, which makes the work so much easier. Cobra has its own CLI called Cobra Generator that helps bootstrap your CLI project and add commands. You can check the full documentation here, but here goes the basic usage:

Install

With go installed, open the terminal and execute the following command:

1

go install github.com/spf13/cobra-cli@latest

Booststraping your CLI

Navigate to a folder that you want to have your project, init a go module, then execute the cobra init. e.g

1
2


go mod init my-cli
cobra-cli init

Voilá! You have your custom CLI, ad you can run it with go run .! You will see a result like:

1
2
3
4
5
6


A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

You can now add a command with executing cobra-cli add ping, then execute the command again. You will see a result like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

Usage:
 my-cli [command]

Available Commands:
 completion Generate the autocompletion script for the specified shell
 help Help about any command
 ping A brief description of your command

Flags:
 -h, --help help for my-cli
 -t, --toggle Help message for toggle

Use "my-cli [command] --help" for more information about a command.

The foundation for your CLI is in place, you only need to worry about the actual logic of the commands because cobra will take care of all the pumbling for you.

You can check several examples in the cobra documentation, and use it as a base to create your shine CLI.

Ok, what about the logic in got?

In got I have two ways of executing some logic. 1) using go-git to perform some actions like iterate in all local branches and delete all except main, and 2) executing a git cli command from my Go application directly.

go-git

go-git is a highly extensible git implementation library written in pure Go. It can be used to manipulate git repositories at low level (plumbing) or high level (porcelain), through an idiomatic Go API. - (Description from its repo).

It is such a well-documented, flexible, and powerful lib. I highly recommend looking into it if you even thought to do something with git, or if you intend to create your own lib. 10/10!

The example below stages all files (including untracked) and commit them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


path, err := os.Getwd() // get the current path
exitIfError(err)

r, err := git.PlainOpen(path) // open the repository related to the path
exitIfError(err)

w, err := r.Worktree() // get the current worktree
exitIfError(err)

_, err = w.Add(".") // Stage all files
exitIfError(err)

_, err = w.Commit("commit yay!", &git.CommitOptions{}) // commit
exitIfError(err)

I have to say that for basic operations like this one, we could be better served by executing the git cli directly like in the example below. However, go-git opens several possibilities like in the support of several type of storage, such as in-memory, file system, or anything you can think of as long as you implement the Storer interface. At this moment I’m not using such features yet, but I’m planning to use them for some commands like got squash that will squash all commits of the current branch.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


cmd := exec.Command("git", "add", "-A")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stdout
_ = cmd.Run()


cmd = exec.Command("git", "commit", "-m", "commit yay!")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stdout
_ = cmd.Run()

Fallback to git

It is important because I wanted to “proxy” all unknown got commands to git. In this way, I can use got for my custom commands and the standard git commands without worrying about what is available where. The other benefit is I don’t need to implement things that are good enough or I don’t use so often.

Implementing it was fairly simple. The snippet below is part of the root.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


func Execute() {
	err := rootCmd.Execute()
	if err != nil {
		fallbackToGit()
	}
}

func fallbackToGit() {
	cmd := exec.Command("git", os.Args[1:]...)
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stdout
	_ = cmd.Run() // Skipping error here, because git already sends it to stdout, and I don't have anything else to do with it.

Conclusion and what’s next?

This small project started as an experiment for playing with writing a custom CLI, but I ended up creating something that is very useful for me. Win-win!

I still have a few commands I want to introduce in got, but my next step is to create a reasonable test suit, so I can have confidence that things work as they suppose to. This is especially important given it is responsible for my interaction with git. That’s a big deal!

Anyways, I created it to attend to my laziness, but if got looks interesting to you, feel free to use, fork, and contribute. 💪

I hope this post can help you to see how easy is to write a CLI using Go, and maybe can inspire you to identify things that you do daily and can be somehow optimized.

Till next time o/