Programming on Flavio Silva

🧹 Tidy First? By Kent Beck - My review and takeaways

flavio1110@gmail.com (Flavio Silva) — Tue, 26 Dec 2023 14:00:00 +0100

“Messy code is a nuisance. “Tidying” code, to make it more readable, requires breaking it up into manageable sections. In this practical guide, author Kent Beck, creator of Extreme Programming and pioneer of software patterns, suggests when and where you might apply tidyings to improve your code while keeping the overall structure of the system in mind. (…)” O’Reilly - Tidy First

The book is clear, objective, and easy to read. Its simple explanations and examples on empirical software design carry sophisticated concepts on evaluating when, how, what, and if to tidy at all.

TL;DR: The book is an 8/10 and I recommend it to any programmer at any level of experience. It’s not a 10/10 because the first section about actual tydings with practical examples felt a bit unnecessary. I understand why it’s there, but the discussions presented in sections 2 and 3 were way more valuable for me. It might be the other way around depending on the experience though.

Sections

The author split the book into three sections: Tidyngs, Managing, and Theory.

Tydings

According to Kent Beck, tidyings are a subset of refactoring. Tydings are the cute, fuzzy little refactorings that nobody could possibly hate on. In this section, the author presents several tidyings that can be done to leave things better than before, e.g. adjusting the order based on reading or cohesion, improving comments, etc.

For people with little or no experience with refactorings, this is going to probably be the most valuable part. Personally, the “nobody could possibly hate on” piece, is the highlight of this section. It helped me to create a mental model to identify the line between a tidy and a “full-blown” refactor that would require much more work and discussion.

Managing

“Tyding is geek self-care” - Kent Beck, Tidy First?

Here is where things started to become more interesting for me. This section addresses when to start and stop tidying, and more importantly how to combine tyding, changing the structure of the code, with changing the behavior of the system.

In this section is the reason why I’d love to change the title of this book is “First, After, Later, Never”. It provides a simple and objective guideline to decide when and if to tidy something. I won’t spoil it here so you can read this part from the book and reach your conclusions.

Theory

This was my favorite section and every chapter was very interesting to read, digest, and process. Particularly the correlation between software and cash flow versus options. This analogy helped me to think more structurally about the real cost of software, its development, and its evolution.

Another highlight of this section is having a clear distinction between behavior and structural changes. Where the behavior changes are the ones related to what the software does and structural changes are the ones that support what the software needs to do. E.g. software for HR offers features like payroll, holiday planning, etc. These features are the behaviors, anything else to support these features is structural, such as the code organization, how data is stored, retrieved, etc. Understanding this distinction, their costs, and values helps to better judge when to assume debt and plan to pay/tidy.

Conclusion

This book is the first of a trilogy. The upcoming books are not yet released (as of the date of writing - 22 December 2023), and they will extend the scope of changes and impact. The first one is focused on the individual programmer (you), the second will be focused on the team, and the third will focus on all stakeholders.

📂 Go: Importig a CSV to PostgreSQL

flavio1110@gmail.com (Flavio Silva) — Sun, 26 Mar 2023 14:00:00 +0100

If you are using Go ad PostgreSQL, and need to performa a bulk import a CSV, it’s most likely you will find the COPY protocol is the feature that suits you better. In that direction, you will find examples using pgx CopyFrom that relies on the native protocol, and it’s fairly easy to use. However, depending how it’s used you can have an exponencial increase of memory consumption of your application making it unreliable and more expensive to run.

TL;DR;: Don’t load the file in memory and use pgx.CopyFromRows. Instead, use the io.Reader of the file and implement a custom pgx.CopyFromSource.

Checkout the repository with examples and details presented here.

Context

Most of the examples out there, are either using CopyFromRows or CopyFromSlice. However, the big problem is these two options require you to have the entire content in memory to use. This is not a big deal when dealing with small files, there won’t be concurrent usage, or you have infinite memory 😅.

How big is the problem?

Comparing the memory consumption for the two distinct approaches importing a file with ~16MB (1M rows).

Approach/metric	TotalAlloc	Sys
Stream file	61 MiB	12 Mib
Read entire file	84 MiB	58 Mib
	+37.70%	+346.15%

Yes, you read it right! using CopyFromRows obtained +346.15% of memory from the OS! 58 MiB instead of 12 MiB.

You can read more about the meaning of each metric on https://golang.org/pkg/runtime/#MemStats and find the source code and details of the comparisson on this repository..

What’s the most efficient way for using the COPY protocol with pgx?

Instead of reading the entire in memory, the idea is to stream each line of the file directly to PostgreSQL. In this way, we only need to keep the current line in-memory as opposed to the entire file.

How can we do it?

The CopyFrom method receives an implementation of the interface CopyFromSource.

Let’s implement this interface using a CSV file with the followng three columns: first_name, last_name, and city.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62


func newPeopleCopyFromSource(csvStream io.Reader) *peopleCopyFromSource {
 csvReader := csv.NewReader(csvStream)
 csvReader.ReuseRecord = true // reuse slice to return the record line by line
 csvReader.FieldsPerRecord = 3

 return &peopleCopyFromSource{
 reader: csvReader,
 isBOF: true, // first line is header
 record: make([]interface{}, len(peopleColumns)),
 }
}

type peopleCopyFromSource struct {
 reader *csv.Reader
 err error
 currentCsvRow []string
 record []interface{}
 isEOF bool
 isBOF bool
}

func (pfs *peopleCopyFromSource) Values() ([]any, error) {
 if pfs.isEOF {
 return nil, nil
 }

 if pfs.err != nil {
 return nil, pfs.err
 }

 // the order of the elements of the record array, must match with
 // the order of the columns in passed into the copy method
 pfs.record[0] = pfs.currentCsvRow[0]
 pfs.record[1] = pfs.currentCsvRow[1]
 pfs.record[2] = pfs.currentCsvRow[2]
 return pfs.record, nil
}

func (pfs *peopleCopyFromSource) Next() bool {
 pfs.currentCsvRow, pfs.err = pfs.reader.Read()
 if pfs.err != nil {

 // when get to the end of the file return false and clean the error.
 // If it's io.EOF we can't return an error
 if errors.Is(pfs.err, io.EOF) {
 pfs.isEOF = true
 pfs.err = nil
 }
 return false
 }

 if pfs.isBOF {
 pfs.isBOF = false
 return pfs.Next()
 }

 return true
}

func (pfs *peopleCopyFromSource) Err() error {
 return pfs.err
}

You can now use this implementation in the CopyFrom method. e.g.

1

_, err := pgxConn.CopyFrom(ctx, pgx.Identifier{"people"}, peopleColumns, newPeopleCopyFromSource(csvStream))

Conclusion

Using the CopyFrom with CopyFromRows or CopyFrom will significantly increase the memory comsultion of your application. The high memory usage can bring several problems like OOM errors, increase of costs, unavailability, etc.

By using a custom implementation of CopyFromSource will make your application much more efficient, reliable, and cheaper to ru.

You can find the entire source code of the examples above on this repository. There you will also find more deatils about the comparisson and the not-so-great implementation.

🔢 Go: Where is the decimal type?

flavio1110@gmail.com (Flavio Silva) — Tue, 21 Mar 2023 14:00:00 +0100

Go doesn’t have a primitive decimal type for arbitrary-precision fixed-point decimal numbers. Yes, you read it right. Therefore, if you need to deal with fixed-point precision there are two main options:

Use an external package like decimal, which introduces the decimal type. However, the current version (1.3.1), can “only” represent numbers with a maximum of 2^31 digits after the decimal point.
Use int64 to store and deal with these numbers. For e.g. given you need 6 precision digits, therefore 79.23, 23.00, and 54.123456, become respectively 79230000, 23000000, and 54123456.

There is an open proposal to add decimal float types (IEEE 754-2008) in the std lib. However, for now, it’s just a proposal being discussed, without guarantee it will be ever added.

🧠 Go resources for beginners

flavio1110@gmail.com (Flavio Silva) — Sun, 19 Mar 2023 14:00:00 +0100

Below a living list of compiled links for whoever is learning Go.

Official docs

Effective Go - https://go.dev/doc/effective_go
- A document that gives tips for writing clear, idiomatic Go code. A must-read for any new Go programmer. It augments the tour and the language specification, both of which should be read first.
Go Code Review Comments https://github.com/golang/go/wiki/CodeReviewComment
- This page collects common comments made during reviews of Go code, so that a single detailed explanation can be referred to by shorthands. This is a laundry list of common mistakes, not a comprehensive style guide. You can view this as a supplement to Effective Go.

Go Docs - https://go.dev/doc/
- Root for many useful documentations
FAQ - https://go.dev/doc/faq
- Answers to common questions about Go.

Blogs

Dave Cheney - https://dave.cheney.net/
- Specially the practical go section, it has TONS of good advice and real-world examples of how to deal with daily challenges. I highly recommend it.

Courses

The way to go - https://www.educative.io/courses/the-way-to-go
- It’s a course from educative.io, it goes from the basics concepts of the language to more advanced ones. It also brings very interesting insights into the differences between the approaches of Java/C# to Go..
How to code in go - https://www.digitalocean.com/community/tutorial_series/how-to-code-in-go
- A great collection of tutorials that cover basic Go concepts, ideal for beginers

Videos

Go in 100 seconds - https://www.youtube.com/watch?v=446E-r0rXHI&t=38s
- Short introduction about Go
Simplicity is complicated - https://www.youtube.com/watch?v=rFejpH_tAHM
- Rob Pike talks about how Go is often described as a simple language. It is not, it just seems that way. Rob explains how Go’s simplicity hides a great deal of complexity, and that both the simplicity and complexity are part of the design.
Concurrency is not parallelism - https://www.youtube.com/watch?v=qmg1CF3gZQ0&t=1582s
- Rob Pike talks about concurrency and how Go implements it
Understanding channels - https://www.youtube.com/watch?v=KBZlN0izeiY&t=1011s
- Channels provide a simple mechanism for goroutines to communicate, and a powerful construct to build sophisticated concurrency patterns. We will delve into the inner workings of channels and channel operations, including how they’re supported by the runtime scheduler and memory
Concurrency in Go - https://www.youtube.com/watch?v=\_uQgGS_VIXM&list=PLsc-VaxfZl4do3Etp_xQ0aQBoC-x5BIgJ
- Playlist with a few short videos about different components of concurrency in Go
Just for func - https://www.youtube.com/watch?v=H_4eRD8aegk&list=PL64wiCrrxh4Jisi7OcCJIUpguV_f5jGnZ
- A very complete playlist of tutorials given by Francesc Campoy, a past Developer Advocate for the Go team at Google, that cover simple to advanced topics in Go
Golang crash course - https://www.youtube.com/watch?v=SqrbIlUwR0U
- A 90 minutes video that covers most of Go features with cristal clear live coding examples, excelent for beginers to get a fast gist of Go

Podcasts

Go Time - Spotify
- Diverse discussions from around the Go and its community