Docker on Flavio Silva

🐔 What happened with my project on the Rinha de Backend challenge

flavio1110@gmail.com (Flavio Silva) — Mon, 28 Aug 2023 14:00:00 +0100

I participated in the Rinha de backend (pt-BR) or “Backend Rooster Fight” in English challenge and I was super excited to see the results of my super hacky and performant solution.

The results were scheduled to be published at 21:00 BRT (2 AM CEST my local time). So I stayed up til later that night grabbed popcorn, and waited for the results.

Well, I wish I hadn’t waited…

TL;DR; My solution didn’t run on the official test server because I built the docker image to linux/arm64 instead of linux/amd64 🤦🤦🤦 . I fixed the issue and ran the same tests afterward and I reached ~50% of points of the winner, mainly because in my solution I didn’t tolerate any risk of losing writes. Checkout my repository flavio1110/rinha-de-backend with my solution.

What was the Rinha de Backend?

Before we get into the details, a brief explanation of what was the challenge.

From July 28th to August 25th, the Backend Fight was held, a tournament in which the API that supported the most load during a stress test would be the winner. Participants had to implement an API with endpoints to create, query and search for ‘people’ (a kind of CRUD without UPDATE and DELETE). In the tournament, participants still had to deal with CPU and memory restrictions – each participant had to deliver the API in docker-compose format and could only use 1.5 CPU units and 3GB of memory. More details on technical aspects can be found in the instructions) – (translated from its repository in English).

On top of the description above, some things important to highlight are:

You can write your API with any language or framework, as long as you can build a docker image out of it
The solution has to run two instances of the API in parallel.
The requests will be balanced as you wish between these instances via Nginx.
You have to use one of the following data stores: MySQL, PostgreSQL, or MongoDB.

It was a super fun, informal, practical, and a very good opportunity to exercise and learn new things.

How the winners were determined?

To make the comparison easier and fun, the single parameter used to determine the winner of the challenge was the number of people inserted into the database.

How did I do?

POV you are looking at me while I check the results.

I didn’t. I was disqualified because the container with my APIs didn’t start. Therefore I got nothing. 🤦🤦🤦

Wrong platform?

Yes, that’s true. My solution didn’t even run because it failed to start the APIs. I built the images for linux/arm64 instead of linux/amd64.

It was a mistake on my end, and it made me super disappointed. I ran the tests on the CI pipelines, and it gave me confidence the image was properly built and I didn’t double-check the requirements. Shame on me.

Well, I was disqualified but I was curious to see how it would perform on a server with the same configuration as the one used for the official tests. So, I fixed the platform of the image, spun up an EC2 with the same configuration, and finally ran the tests. The results were a bit disappointing compared to the TOP 10.

How did I approach the problem?

I had two constraints in mind while designing and building:

It has to work – each endpoint has to do what it is supposed to do.
Keep things as simple as possible.

Each of these points brought a few tradeoffs that ended up limiting the overall performance of the solution.

You can check my solution on flavio1110/rinha-de-backend. It’s built with Go and PostgreSQL.

Given many APIs were performing super well, a few days before the last day of the challenge, the load for the stress test was doubled, and it had a massive impact on the performance of my API. You can check the comparison below:

Before…

Print of part of the report generated by Gatling of my solution run on Github BEFORE the load was doubled.

After…

Print of part of the report generated by Gatling of my solution run on Github AFTER the load was doubled.

What were the bottlenecks?

The inserts were the bottleneck. Because I didn’t tolerate the risk of losing writes and wanted to keep it simple, I missed the opportunity to introduce a distributed mechanism to validate the data before trying to insert it into the database and perform the inserts in batch.

Validating the uniqueness of a field Apelido (Nickname)

On my designed solution, without a distributed cache, it was not possible to perform without the DB in 100% of the cases.

I had an in-memory cache, but if the nickname wasn’t there because it wasn’t there because the entry was inserted via the other instance, that request would reach the DB, occupy a connection, take resources, etc.

Introducing a distributed cache like Redis would enable me to add and check the entry in a single place. As a result, we could decrease the memory necessary to run the APIs and move to the cache.

Batching inserts

I inserted each entry per a valid request because I didn’t want to risk losing valid writes in case there was any error between the construction of the batch and its execution.

This was a huge problem because it used too many DB connections and resources, and slowed down the creation request.

Did I like it?

Despite my results, I loved the challenge! I learned quite some tricks with nginx config and some insteresting stuff like using Redis or even PostgreSQL as a “PubSub”. The interaction with the community via Twitter and Github was super nice! I also liked the fact of working with something challenging and closer to reality. Loved it!

What could I have done differently?

Build the image using the correct platform.
A challenge is a challenge. I should have been more flexible with the writing.
Tweak the Nginx configuration to load balance based on fewer connections and disable logs.
Use a distributed cache and drop the read table.
Batch inserts.

What’s next?

Check the official results and look into the repositories of the participants. I guarantee I’ll learn something new.

Stay tuned on @rinhadebackend for the next challenges, I can’t wait for the next ones.

Other than that, I’ll apply the lessons learned to my solution and hopefully get better results. Watch flavio1110/rinha-de-backend and see how it will evolve.

This challenge is officially over, but you can still do it. Challenge accepted?

🔥 Test DB integrations with Testcontainers

flavio1110@gmail.com (Flavio Silva) — Mon, 17 Apr 2023 14:00:00 +0100

Picture this, you have a critical and reasonably complicated piece of logic in your application that is handled in the database. Despite any change on it (or around it), you must have a 100% guarantee that piece continues to work just fine. So, what do you do?

Back in the ancient days…

Ancient cave art of many hand prints. (Credit: Petr Kratochvila/Shutterstock)

Before the “age of containers”, the solution would be a variant of the following:

Have a database with the schema aligned with the application’s version.
Ensure your CI tool has access to that database, to run the tests in your CI pipeline.
Create a script to arrange the necessary data for the test.
Write your testing using the DB.
Create a script to revert any DB changes performed by the test, such as inserts, updates, etc.

This is a very nice post about the Docker: Life Before and after.

What’s the problem with testing with a shared database?

Simply put, it’s complex, expensive to maintain, and has many other reasons to fail. For example:

The DB’s schema version is not the same as the application’s version you want to test.
There is a change in the network policy that blocks access to the database.
Someone’s test setup corrupted the data that you expected to have, so you arrange script fails, or the test fails because of missing data.

Because of the cost and flakiness of such tests, in many cases, the solution was to move logic from DB to the application’s code when performance was not a problem, or just manually test it from time to time and hope it doesn’t break because of an unforeseen reason (it-never-happened-in-the-last-30-minutes).

Containers everywhere

With the popularization of containers, not only developing, runnings, and deploying became easier, but also integrating and testing your external dependencies became easier, faster, and more dependable.

We can now effortlessly run not only the regular dependencies, like RDBMSs and Message Brokers, but we can emulate many cloud provider services using tools like localstack.

What about testing? Testcontainers to rescue!

Testcontainers makes it simple to create and clean up container-based dependencies for automated integration/smoke tests. The clean, easy-to-use API enables developers to programmatically define containers that should be run as part of a test and clean up those resources when the test is done.

It supports many popular languages/frameworks like Go, Java, .NET, Rust, and others. You can check the full list on its official site](https://www.testcontainers.org/) and because it’s open source, you check out and contribute to it in its repositories.

How to use Testcontainers on integration tests?

For this example, assume that one of the features of your software is to search for users based on a few criteria. Your users are stored in a PostgreSQL database and you query them using SQL. Something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


func searchPeople(ctx context.Context, db *sql.DB, params searchParams) ([]person, error) {
	query := `select first_name, last_name, city
		 from people
		 where ($1::text is null or first_name = $1)
		 and ($2::text is null or last_name = $2)
		 and ($3::text is null or city = $3)
		 order by first_name asc`

 var people []person

 //Execute query, check errors, populate people variable

 return people, nil
}

The search criteria will evolve, by adding new filters or adding new features like pagination. Therefore, we need to make sure that whenever we touch this query its existing behavior is not broken. Let’s write a test for it using Testcontainers.

Writing the test

To be able to run these tests against a real database, we need to:

Start the DB
Create the schema
Arrange data, Act, and Assert

Let’s see in detail how to perform each one of these steps.

1. Start the DB

Let’s create a function that will start a DB, return its connection string and a function to terminate it when we are done.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54


func startTestDB(ctx context.Context) (string, func(t *testing.T), error) {
	var envVars = map[string]string{
		"POSTGRES_USER": "user",
		"POSTGRES_PASSWORD": "super-secret",
		"POSTGRES_DB": "people",
		"PORT": "5432/tcp",
	}

	getConnString := func(host string, port nat.Port) string {
		return fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable",
			envVars["POSTGRES_USER"],
			envVars["POSTGRES_PASSWORD"],
			host,
			port.Port(),
			envVars["POSTGRES_DB"])
	}

	req := testcontainers.ContainerRequest{
		Image: "postgres:14",
		ExposedPorts: []string{envVars["PORT"]},
		Env: envVars,
		WaitingFor: wait.ForSQL(nat.Port(envVars["PORT"]), "pgx", getConnString).
 WithStartupTimeout(time.Second * 15),
	}
	pgC, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
		ContainerRequest: req,
		Started: true,
	})
	if err != nil {
		return "", nil, fmt.Errorf("failed to start db container :%w", err)
	}
	port, err := pgC.MappedPort(ctx, "5432/tcp")
	if err != nil {
		return "", nil, fmt.Errorf("failed to get mapped port :%w", err)
	}
	host, err := pgC.Host(ctx)
	if err != nil {
		return "", nil, fmt.Errorf("failed to get host :%w", err)
	}

	connString := fmt.Sprintf("postgres://%s:%s@%s:%d/%s?sslmode=disable",
		envVars["POSTGRES_USER"],
		envVars["POSTGRES_PASSWORD"],
		host,
		port.Int(),
		envVars["POSTGRES_DB"])

	terminate := func(t *testing.T) {
		if err := pgC.Terminate(ctx); err != nil {
			t.Fatalf("failed to terminate container: %s", err.Error())
		}
	}
	return connString, terminate, nil
}

The code above is self-explanatory if you are familiar with Go, but there are a few aspects that I’d like to highlight:

Testcontainers will map and assign a random port. Therefore, in this case, instead of connecting to 5432, it’s necessary to connect to the assigned port. This is similar to the hostname. Therefore, to get your correct connection string it’s needed to resolve port and host.
The container will automatically shut down when the application is finished. However, it is recommended to defer its termination as soon as it is started. For that, a terminate func is returned by the method above.

So far our test looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


func TestSearchPeople(t *testing.T) {
	ctx := context.Background()
	connString, terminate, err := startTestDB(ctx)
	if err != nil {
		t.Error(err)
	}
	defer terminate(t)

	db, err := openDB(connString)
	if err != nil {
		t.Fatal("fail to open DB", err)
	}
	// Continue
}

2. Create the schema

With a DB up and running we can initiate it with the expected schema, a.k.a migrate the DB schema.

In the real world, we would use something more sophisticated like go-migrate, but for the sake of simplicity let’s write something simpler.

1
2
3
4
5
6
7


func migrateDB(ctx context.Context, db *sql.DB) error {
	_, err := db.ExecContext(ctx, "create table if not exists people (first_name text,last_name text,city text)")
	if err != nil {
		return fmt.Errorf("failed to migrate DB: %w", err)
	}
	return nil
}

3. Arrange data, Act, and Assert

The last step before the actual test is to arrange the data. We could accomplish it together with the schema migration, however keeping these two parts separate helps to keep each test isolated, avoiding side effects when the data is tweaked.

For our tests, we need to insert 4 different people with distinct names and cities. Then we can use this data to act and assert the results. This is what the final test func looks like, testing every variation of our query.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58


func TestSearchPeople(t *testing.T) {
	ctx := context.Background()
	// Start the DB
	connString, terminate, err := startTestDB(ctx)
	if err != nil {
		t.Error(err)
	}
	// Terminate the container when the func TestSearchPeople finishes
	defer terminate(t)

	// Open the the database
	db, err := openDB(connString)
	if err != nil {
		t.Fatal("fail to open DB", err)
	}

	// Migrate schema
	if err := migrateDB(ctx, db); err != nil {
		t.Fatal("fail to migrate DB", err)
	}

	insert := `insert into people values
			('Flavio', 'Silva', 'Olbia'),
			('Joost', 'Van Huis', 'Amsterdam'),
			('Aldben', 'Arimeritin', 'Istambul'),
			('Nando', 'Pelect', 'Perth');`

 // Arrange data
	if _, err := db.ExecContext(ctx, insert); err != nil {
		t.Fatal("fail to insert initial data", err)
	}

	// Several test cases
	t.Run("without filters", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{})
		assert.NoError(t, err)
		assert.Equal(t, 4, len(people))
	})

	t.Run("filtering by first name", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{firstName: strPtr("Joost")})
		assert.NoError(t, err)
		assert.Equal(t, 1, len(people))
		assert.Equal(t, "Joost", people[0].firstName)
	})
	t.Run("filtering by last name", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{lastName: strPtr("Arimeritin")})
		assert.NoError(t, err)
		assert.Equal(t, 1, len(people))
		assert.Equal(t, "Arimeritin", people[0].lastName)
	})
	t.Run("filtering by city", func(t *testing.T) {
		people, err := searchPeople(ctx, db, searchParams{city: strPtr("Olbia")})
		assert.NoError(t, err)
		assert.Equal(t, 1, len(people))
		assert.Equal(t, "Olbia", people[0].city)
	})
}

With this test, we have the guarantee the covered scenarios are working. Profit!

You can check the results by running the tests.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


go test ./... -v
=== RUN TestSearchPeople
2023/05/09 15:48:55 github.com/testcontainers/testcontainers-go - Connected to docker:
 Server Version: 20.10.22
 API Version: 1.41
 Operating System: Docker Desktop
 Total Memory: 3932 MB
2023/05/09 15:48:55 Starting container id: 95dc2928dae3 image: docker.io/testcontainers/ryuk:0.3.4
2023/05/09 15:48:56 Waiting for container id 95dc2928dae3 image: docker.io/testcontainers/ryuk:0.3.4
2023/05/09 15:48:56 Container is ready id: 95dc2928dae3 image: docker.io/testcontainers/ryuk:0.3.4
2023/05/09 15:48:56 Starting container id: adedca079bc6 image: postgres:14
2023/05/09 15:48:57 Waiting for container id adedca079bc6 image: postgres:14
2023/05/09 15:48:58 Container is ready id: adedca079bc6 image: postgres:14
=== RUN TestSearchPeople/without_filters
=== RUN TestSearchPeople/filtering_by_first_name
=== RUN TestSearchPeople/filtering_by_last_name
=== RUN TestSearchPeople/filtering_by_city
--- PASS: TestSearchPeople (3.48s)
 --- PASS: TestSearchPeople/without_filters (0.00s)
 --- PASS: TestSearchPeople/filtering_by_first_name (0.00s)
 --- PASS: TestSearchPeople/filtering_by_last_name (0.00s)
 --- PASS: TestSearchPeople/filtering_by_city (0.00s)
PASS
ok 	github.com/flavio1110/go-for-csharp-devs/experiments/test-db-interactions	.724s

You can check the source code of this test on my experiments repository.

Final thoughts

Despite Testcontainers reducing the cost of integration tests and making them more reliable, it doesn’t remove the cost entirely. The tests will take more time to run (especially if where you are running the tests doesn’t have the image downloaded yet).

Nevertheless, in my opinion, the price is low compared to the benefit that it yields.

What about you? How do you write integration tests, and what are your main challenges?