Go on Flavio Silva

🤖 Building a propose-then-approve bot engine

flavio1110@gmail.com (Flavio Silva) — Tue, 14 Jul 2026 14:00:00 +0100

After every race weekend on apexclub.live I used to do the same three things by hand. Enter the starting grid, enter the final classifications, trigger the scoring calculation. Every step was a few clicks and a few minutes. The annoying part wasn’t the time. It was the kind of work. It’s detectable, it’s repetitive, and getting it wrong silently corrupts a season’s standings. So I wanted it gone, but only on my terms.

I wanted something autonomous enough that I didn’t have to remember. Deterministic enough that I could trust it. Small enough that I could still explain every line.

TL;DR: A recurring worker tick runs a cheap Detect query for each registered use case. If there’s nothing to do, no LLM runs. If there’s a candidate, the engine builds a proposal, persists it, sends me a Telegram message with Approve / Reject buttons, and only on my tap calls a typed executor to apply the change. The LLM proposes. The database and the human approve. The executor writes.

The shape of the problem

The original setup wasn’t bad. Race results came from analytics on Monday. The application had a Telegram bot I already trusted for one-off commands. I had a database I could read and write to. The only thing missing was glue.

A naive first attempt was to let the existing chat bot propose mutations during a conversation. Press a button, the model picks a tool, the tool runs. I tried it and closed it. The problem wasn’t Telegram. It was mixing recurring operations with a conversational tool loop. Most ticks have nothing to do: the race hasn’t finished, analytics hasn’t synced, results already exist, scores are already published. Paying for an LLM call to discover that “nothing is ready” is wasteful, and worse, “the model decided to call this tool” is not an audit record I want to publish against a league’s standings.

I wanted a concrete payload, a visible proposal, a named human decision, and a deterministic function that could be retried safely. So I split the work in five pieces:

a detect step that is just SQL, cheap and easy to reason about;
a propose step that builds a structured payload and a human-readable summary;
a persist step that stores the proposal before anything is sent anywhere;
a human decision delivered through a channel I already trust (Telegram, in my case);
a typed executor that applies an approved proposal and is required to be idempotent.

Each piece is small. The interesting bit is the contract between them.

The contract

A use case is anything that can detect a candidate, propose a payload, and apply it. The shape, in Go, is roughly:

1
2
3
4
5
6
7


type UseCase interface {
 Kind() string
 Schedule() string
 Detect(ctx context.Context) ([]Candidate, error)
 Propose(ctx context.Context, c Candidate) (*Proposal, error)
 Execute(ctx context.Context, a *Action) (summary string, err error)
}

The four methods divide responsibility cleanly:

Detect is a cheap precondition, usually a single SELECT.
Propose turns one candidate into a structured payload and an admin-facing summary.
Execute applies an approved action. Idempotency is required.
Kind is the dispatch and dedup key.
Schedule lives on the use case, not in central wiring.

The engine has three pieces that move a use case through that contract:

A runner that ticks on schedule, calls Detect, and for each candidate checks whether an action already exists for the same kind + subject reference. If not, it asks Propose to build a proposal and hands it to the coordinator.
A coordinator that owns persistence and decisions. It stores the proposal, sends the notification, atomically claims an approval, dispatches the executor, and records the outcome.
A registry that maps a kind such as insert_race_results to its executor. Registering the same kind twice panics at startup, which is the right moment to discover a wiring mistake.

The lifecycle of a persisted action is small enough to draw in text:

1
2
3


pending -> executing -> executed
 -> failed
pending -> rejected

A partial unique index allows only one open action per (kind, subject_ref). That single index is what makes the runner safe to call every 30 seconds across multiple application instances, and what makes a second admin press of “Approve” a no-op instead of a duplicate write.

The approval transport can be anything with two buttons. Telegram was the obvious choice because I was already living there for one-off commands. A web page could work just as well. It just has to call back into the same coordinator.

The flow, end to end

Here is the whole path of one piece of work, written the way the code actually runs it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


worker tick (every 30s)
 -> for each registered use case:
 Detect() with SQL
 -> zero candidates: stop
 -> candidate: is there already an action for (kind, subject)?
 yes: stop
 no: Propose(candidate) -> Proposal
 Coordinator.Submit(proposal)
 -> persist action with status=pending
 -> send approval message with action UID
 -> Telegram button press
 -> Coordinator.Approve(uid) (atomic: pending -> executing)
 -> Registry.Dispatch(kind).Execute(action)
 -> mark action executed or failed

The decision to put deterministic work first is the entire insight. The LLM only runs after SQL has already said “there is something to do here”, and only to convert that “something” into a payload a human can review.

Why this set of qualities

A few properties fall out of the shape, not from extra effort.

When nothing needs doing, the worker runs only SELECT queries. No model call, no proposal, no notification. I called that “cheap idle ticks” above, and I underestimated how much it mattered before I built it. Most ticks are idle. Saving a model call on every idle tick is the difference between a system I forget exists and a system I worry about.

Auditability is mostly free. Every action is a row: kind, subject reference, JSONB payload, the human-readable summary, the Telegram message identifiers, who decided it, when, and the final outcome. That’s a complete answer to “why did the standings change on Sunday at 23:14?”.

The executor contract is Execute(ctx, *Action). Rerunning it with the same action must converge to the same outcome. That requirement is what makes a button-mash, a network retry, or a panic mid-flight all safe. The row in pending / executing is the source of truth, and the executor decides whether to insert, update, or do nothing.

There is no global “AI: on” switch. The system is on whenever the worker is running with a configured approval channel. The absence of an admin to approve is the absence of writes.

Failure modes are independent. Each use case has its own schedule and its own advisory lock. A broken results matcher doesn’t prevent scores from being checked.

Two use cases, end to end

The same engine runs two unrelated capabilities on apexclub.live. They’re not hardcoded into the engine. They’re ordinary use cases that happen to be the first ones I needed.

Insert race results

The use case owns kind insert_race_results and schedule */30 * * * *.

Its Detect query is “active-season races in the past with no row in results yet”. When that returns nothing, no LLM runs.

When it returns a candidate, the use case:

matches the main race to a row in an analytics calendar (the LLM only picks from offered candidates, it can’t author the mapping);
pulls qualifying pole and classified finishers from the analytics tables;
resolves driver names to the application’s own grid entries (again, the LLM only picks from offered IDs);
assembles a structured result plus the P1–P22 finishing order;
refuses incomplete mappings and surfaces them as a manual-entry notice instead of a proposal;
persists the payload for approval.

On approval, Execute checks for an existing result before inserting and only writes final positions when none are already recorded. A retry after a partial failure converges without overwriting manual data.

Publish scores

The second use case owns kind publish_scores, also scheduled every 30 minutes. Its Detect query is “an active-season race with a results row, not yet finalized, no scores row, and at least one real starting position recorded”. Same shape, a SELECT that asks whether the next step’s preconditions are met.

The proposal payload is just a race identifier. On approval, the executor runs the existing scoring code and flips a finalized flag. Re-running the executor upserts and sets the same boolean, which converges safely.

There is no direct call between these two use cases. The database state is the hand-off: the results executor creates the data that makes scoring eligible on a later tick. Independent schedules, independent advisory locks, independent failure handling.

The traps, in order

Building the engine was less work than fixing the things I got wrong the first time. A few worth naming.

Approval state can stick. The first version moved an action from pending to executing before calling the executor, which prevented two button presses from racing. But if the executor panicked, the action stayed in executing, and because executing counted as open, the runner could never propose it again without a manual database update. The fix was to move the lifecycle into the coordinator: a panic is converted into an error, and Approve is the only transition that marks an action failed. Atomic claiming was only half the story. Every transition needs a recovery path.

Source identities are fuzzy. The first attempt matched races by season and round number. Real data proved that wrong. For one British Grand Prix, the main calendar said round 11 while analytics said round 9, and analytics round 11 was a different race with no results yet. The LLM’s job in this system is to resolve exactly the kind of ambiguity that round numbers and slightly different race names cause. The authority stays narrow: the model picks from offered source rows, never authors them. Deterministic code then checks whether the chosen mapping is safe enough to propose.

The approval message is part of the safety boundary. A Telegram message that shows only the top three drivers is not a complete proposal if the scoring system needs the full finishing order. Anything required for approval belongs in deterministic presentation, not in prose the model is free to shorten. The pattern that worked: build the visible fields from the assembled payload in code, and append the model-generated summary as commentary. The fields are guaranteed. The commentary is colour.

Preconditions encode data completeness. Scoring requires both a result and a starting grid. A result without a grid produces a plausible-looking but silently-wrong score. The Detect query for scoring was the right place to encode that requirement. The engine should refuse a valid-looking operation when its inputs would produce an incorrect result.

These weren’t architectural mistakes. They were the price of admitting the second iteration. The contract didn’t have to change. The implementations got better at honouring it.

Tradeoffs and limits

A few honest costs.

Latency is human-shaped. A use case that detects at 30 seconds and a human who approves five minutes later means a five-minute gap between readiness and execution. For race weekend admin that’s fine. For anything time-sensitive, propose-then-approve is the wrong shape.

Approval delivery is one channel. Today the only way to approve is Telegram. If the notification fails to send, the action is persisted in pending and stays there until a human notices or a redelivery job is added. Persistence beats loss, but it’s not the same as a guaranteed delivery.

One primary approver. Notifications go to the first ID in the allowlist. Every allowlisted user can press the buttons, but only one gets the message. A real multi-admin workflow needs a delivery and decision policy, not just an allowlist.

Rejected actions don’t auto-retry. The repository layer suppresses future proposals for a rejected (kind, subject_ref) to avoid nagging. There is no “retry this one” operation that doesn’t also require a policy or data change. For one-shot work that’s the right default. For evolving inputs it can feel stubborn.

Fuzzy matching is bounded. The model can only pick from offered candidates. That’s a strong guardrail and also a real ceiling. When the input source gets a new spelling or a new driver, the matcher will refuse rather than guess. Human review remains mandatory.

Where I’d take it next

The next changes I’d make are unglamorous:

A redelivery job for pending actions whose Telegram message failed.
A web approval page that calls the same coordinator. It doesn’t need to replace Telegram. It just needs to be a second transport.
An explicit retry / reopen transition for rejected subjects, gated by policy rather than by data manipulation.
More use cases. The shape proved out on results and scoring. The next two I’m eyeing are joker-round announcements and end-of-season standings exports. Both are detectable, both are one-shot, both have a payload I want to review before they go out.

The thing I would not change is the order. Cheap detect, deterministic propose, human approve, typed execute. Each piece is replaceable. The order is the engine.

The engine is small because it refuses to solve every agent problem. It works for recurring, detectable, one-shot changes with a payload that can be reviewed before it’s applied. “Autonomous” here means the system notices and prepares work by itself. Authority to change league results still belongs to a person.

What kind of recurring work would you trust it to propose, but not to approve?

🐔 What happened with my project on the Rinha de Backend challenge

flavio1110@gmail.com (Flavio Silva) — Mon, 28 Aug 2023 14:00:00 +0100

I participated in the Rinha de backend (pt-BR) or “Backend Rooster Fight” in English challenge and I was super excited to see the results of my super hacky and performant solution.

The results were scheduled to be published at 21:00 BRT (2 AM CEST my local time). So I stayed up til later that night grabbed popcorn, and waited for the results.

Well, I wish I hadn’t waited…

TL;DR; My solution didn’t run on the official test server because I built the docker image to linux/arm64 instead of linux/amd64 🤦🤦🤦 . I fixed the issue and ran the same tests afterward and I reached ~50% of points of the winner, mainly because in my solution I didn’t tolerate any risk of losing writes. Checkout my repository flavio1110/rinha-de-backend with my solution.

What was the Rinha de Backend?

Before we get into the details, a brief explanation of what was the challenge.

From July 28th to August 25th, the Backend Fight was held, a tournament in which the API that supported the most load during a stress test would be the winner. Participants had to implement an API with endpoints to create, query and search for ‘people’ (a kind of CRUD without UPDATE and DELETE). In the tournament, participants still had to deal with CPU and memory restrictions – each participant had to deliver the API in docker-compose format and could only use 1.5 CPU units and 3GB of memory. More details on technical aspects can be found in the instructions) – (translated from its repository in English).

On top of the description above, some things important to highlight are:

You can write your API with any language or framework, as long as you can build a docker image out of it
The solution has to run two instances of the API in parallel.
The requests will be balanced as you wish between these instances via Nginx.
You have to use one of the following data stores: MySQL, PostgreSQL, or MongoDB.

It was a super fun, informal, practical, and a very good opportunity to exercise and learn new things.

How the winners were determined?

To make the comparison easier and fun, the single parameter used to determine the winner of the challenge was the number of people inserted into the database.

How did I do?

POV you are looking at me while I check the results.

I didn’t. I was disqualified because the container with my APIs didn’t start. Therefore I got nothing. 🤦🤦🤦

Wrong platform?

Yes, that’s true. My solution didn’t even run because it failed to start the APIs. I built the images for linux/arm64 instead of linux/amd64.

It was a mistake on my end, and it made me super disappointed. I ran the tests on the CI pipelines, and it gave me confidence the image was properly built and I didn’t double-check the requirements. Shame on me.

Well, I was disqualified but I was curious to see how it would perform on a server with the same configuration as the one used for the official tests. So, I fixed the platform of the image, spun up an EC2 with the same configuration, and finally ran the tests. The results were a bit disappointing compared to the TOP 10.

How did I approach the problem?

I had two constraints in mind while designing and building:

It has to work – each endpoint has to do what it is supposed to do.
Keep things as simple as possible.

Each of these points brought a few tradeoffs that ended up limiting the overall performance of the solution.

You can check my solution on flavio1110/rinha-de-backend. It’s built with Go and PostgreSQL.

Given many APIs were performing super well, a few days before the last day of the challenge, the load for the stress test was doubled, and it had a massive impact on the performance of my API. You can check the comparison below:

Before…

Print of part of the report generated by Gatling of my solution run on Github BEFORE the load was doubled.

After…

Print of part of the report generated by Gatling of my solution run on Github AFTER the load was doubled.

What were the bottlenecks?

The inserts were the bottleneck. Because I didn’t tolerate the risk of losing writes and wanted to keep it simple, I missed the opportunity to introduce a distributed mechanism to validate the data before trying to insert it into the database and perform the inserts in batch.

Validating the uniqueness of a field Apelido (Nickname)

On my designed solution, without a distributed cache, it was not possible to perform without the DB in 100% of the cases.

I had an in-memory cache, but if the nickname wasn’t there because it wasn’t there because the entry was inserted via the other instance, that request would reach the DB, occupy a connection, take resources, etc.

Introducing a distributed cache like Redis would enable me to add and check the entry in a single place. As a result, we could decrease the memory necessary to run the APIs and move to the cache.

Batching inserts

I inserted each entry per a valid request because I didn’t want to risk losing valid writes in case there was any error between the construction of the batch and its execution.

This was a huge problem because it used too many DB connections and resources, and slowed down the creation request.

Did I like it?

Despite my results, I loved the challenge! I learned quite some tricks with nginx config and some insteresting stuff like using Redis or even PostgreSQL as a “PubSub”. The interaction with the community via Twitter and Github was super nice! I also liked the fact of working with something challenging and closer to reality. Loved it!

What could I have done differently?

Build the image using the correct platform.
A challenge is a challenge. I should have been more flexible with the writing.
Tweak the Nginx configuration to load balance based on fewer connections and disable logs.
Use a distributed cache and drop the read table.
Batch inserts.

What’s next?

Check the official results and look into the repositories of the participants. I guarantee I’ll learn something new.

Stay tuned on @rinhadebackend for the next challenges, I can’t wait for the next ones.

Other than that, I’ll apply the lessons learned to my solution and hopefully get better results. Watch flavio1110/rinha-de-backend and see how it will evolve.

This challenge is officially over, but you can still do it. Challenge accepted?

🐉 got: Embracing my lazyness with my custom git CLI

flavio1110@gmail.com (Flavio Silva) — Thu, 30 Mar 2023 14:00:00 +0100

got is a CLI written in Go, created on top of the git CLI, to make my life easier by shortening some commands I use daily.

Because it’s built on top of git, all git commands will also work just fine and I don’t need to keep switching between got and git!

If I want to clean up my local dead branches I can got rmb instead of git branch | grep -v "main" | xargs git branch -D (you can call me lazy. I accept that 😅).

xkcd Automation.

But… Why?

My first experience with a Distributed version control system (DVCS), was many years ago with mercurial hg, and despite taking some time to get used to the new way of working as compared to a centralized version control system, I got very used to its short commands, aliases, and simplicity. We could do something like hg pull or hg pul (that’s not a typo. This is an actual alias for hg pull), we were able to close a branch and commit a message in a single command like hg commit --close-branch -m 'closing this branch'. It was super nice and handy! Then, eventually, I started working with git and I was amazed by its differences, possibilities, and features. However, it felt more verbose for day-to-day tasks. So I ended up creating a bunch of aliases for the commands that I use more often, or commands that are longer and I always have to google it to remember. Sneak peek of my ~/.zshrc

1
2
3
4
5
6
7


#...
alias push='git push origin head'
alias stat='git status -s'
alias gbr="git branch | grep -v "main" | xargs git branch -D"
alias pick="git cherry-pick"
# and a few more...
#...

Speaking on zsh, if you use it with the go plugin, you will have a conflict with the alias got for go test.

The aliases work so well and are easy to maintain… But I wanted something fancier and all have the opportunity to go through the process of writing a CLI in Go.

How

Got is built using cobra, which makes the work so much easier. Cobra has its own CLI called Cobra Generator that helps bootstrap your CLI project and add commands. You can check the full documentation here, but here goes the basic usage:

Install

With go installed, open the terminal and execute the following command:

1

go install github.com/spf13/cobra-cli@latest

Booststraping your CLI

Navigate to a folder that you want to have your project, init a go module, then execute the cobra init. e.g

1
2


go mod init my-cli
cobra-cli init

Voilá! You have your custom CLI, ad you can run it with go run .! You will see a result like:

1
2
3
4
5
6


A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

You can now add a command with executing cobra-cli add ping, then execute the command again. You will see a result like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

Usage:
 my-cli [command]

Available Commands:
 completion Generate the autocompletion script for the specified shell
 help Help about any command
 ping A brief description of your command

Flags:
 -h, --help help for my-cli
 -t, --toggle Help message for toggle

Use "my-cli [command] --help" for more information about a command.

The foundation for your CLI is in place, you only need to worry about the actual logic of the commands because cobra will take care of all the pumbling for you.

You can check several examples in the cobra documentation, and use it as a base to create your shine CLI.

Ok, what about the logic in got?

In got I have two ways of executing some logic. 1) using go-git to perform some actions like iterate in all local branches and delete all except main, and 2) executing a git cli command from my Go application directly.

go-git

go-git is a highly extensible git implementation library written in pure Go. It can be used to manipulate git repositories at low level (plumbing) or high level (porcelain), through an idiomatic Go API. - (Description from its repo).

It is such a well-documented, flexible, and powerful lib. I highly recommend looking into it if you even thought to do something with git, or if you intend to create your own lib. 10/10!

The example below stages all files (including untracked) and commit them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


path, err := os.Getwd() // get the current path
exitIfError(err)

r, err := git.PlainOpen(path) // open the repository related to the path
exitIfError(err)

w, err := r.Worktree() // get the current worktree
exitIfError(err)

_, err = w.Add(".") // Stage all files
exitIfError(err)

_, err = w.Commit("commit yay!", &git.CommitOptions{}) // commit
exitIfError(err)

I have to say that for basic operations like this one, we could be better served by executing the git cli directly like in the example below. However, go-git opens several possibilities like in the support of several type of storage, such as in-memory, file system, or anything you can think of as long as you implement the Storer interface. At this moment I’m not using such features yet, but I’m planning to use them for some commands like got squash that will squash all commits of the current branch.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


cmd := exec.Command("git", "add", "-A")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stdout
_ = cmd.Run()


cmd = exec.Command("git", "commit", "-m", "commit yay!")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stdout
_ = cmd.Run()

Fallback to git

It is important because I wanted to “proxy” all unknown got commands to git. In this way, I can use got for my custom commands and the standard git commands without worrying about what is available where. The other benefit is I don’t need to implement things that are good enough or I don’t use so often.

Implementing it was fairly simple. The snippet below is part of the root.go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


func Execute() {
	err := rootCmd.Execute()
	if err != nil {
		fallbackToGit()
	}
}

func fallbackToGit() {
	cmd := exec.Command("git", os.Args[1:]...)
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stdout
	_ = cmd.Run() // Skipping error here, because git already sends it to stdout, and I don't have anything else to do with it.

Conclusion and what’s next?

This small project started as an experiment for playing with writing a custom CLI, but I ended up creating something that is very useful for me. Win-win!

I still have a few commands I want to introduce in got, but my next step is to create a reasonable test suit, so I can have confidence that things work as they suppose to. This is especially important given it is responsible for my interaction with git. That’s a big deal!

Anyways, I created it to attend to my laziness, but if got looks interesting to you, feel free to use, fork, and contribute. 💪

I hope this post can help you to see how easy is to write a CLI using Go, and maybe can inspire you to identify things that you do daily and can be somehow optimized.

Till next time o/

📂 Go: Importig a CSV to PostgreSQL

flavio1110@gmail.com (Flavio Silva) — Sun, 26 Mar 2023 14:00:00 +0100

If you are using Go ad PostgreSQL, and need to performa a bulk import a CSV, it’s most likely you will find the COPY protocol is the feature that suits you better. In that direction, you will find examples using pgx CopyFrom that relies on the native protocol, and it’s fairly easy to use. However, depending how it’s used you can have an exponencial increase of memory consumption of your application making it unreliable and more expensive to run.

TL;DR;: Don’t load the file in memory and use pgx.CopyFromRows. Instead, use the io.Reader of the file and implement a custom pgx.CopyFromSource.

Checkout the repository with examples and details presented here.

Context

Most of the examples out there, are either using CopyFromRows or CopyFromSlice. However, the big problem is these two options require you to have the entire content in memory to use. This is not a big deal when dealing with small files, there won’t be concurrent usage, or you have infinite memory 😅.

How big is the problem?

Comparing the memory consumption for the two distinct approaches importing a file with ~16MB (1M rows).

Approach/metric	TotalAlloc	Sys
Stream file	61 MiB	12 Mib
Read entire file	84 MiB	58 Mib
	+37.70%	+346.15%

Yes, you read it right! using CopyFromRows obtained +346.15% of memory from the OS! 58 MiB instead of 12 MiB.

You can read more about the meaning of each metric on https://golang.org/pkg/runtime/#MemStats and find the source code and details of the comparisson on this repository..

What’s the most efficient way for using the COPY protocol with pgx?

Instead of reading the entire in memory, the idea is to stream each line of the file directly to PostgreSQL. In this way, we only need to keep the current line in-memory as opposed to the entire file.

How can we do it?

The CopyFrom method receives an implementation of the interface CopyFromSource.

Let’s implement this interface using a CSV file with the followng three columns: first_name, last_name, and city.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62


func newPeopleCopyFromSource(csvStream io.Reader) *peopleCopyFromSource {
 csvReader := csv.NewReader(csvStream)
 csvReader.ReuseRecord = true // reuse slice to return the record line by line
 csvReader.FieldsPerRecord = 3

 return &peopleCopyFromSource{
 reader: csvReader,
 isBOF: true, // first line is header
 record: make([]interface{}, len(peopleColumns)),
 }
}

type peopleCopyFromSource struct {
 reader *csv.Reader
 err error
 currentCsvRow []string
 record []interface{}
 isEOF bool
 isBOF bool
}

func (pfs *peopleCopyFromSource) Values() ([]any, error) {
 if pfs.isEOF {
 return nil, nil
 }

 if pfs.err != nil {
 return nil, pfs.err
 }

 // the order of the elements of the record array, must match with
 // the order of the columns in passed into the copy method
 pfs.record[0] = pfs.currentCsvRow[0]
 pfs.record[1] = pfs.currentCsvRow[1]
 pfs.record[2] = pfs.currentCsvRow[2]
 return pfs.record, nil
}

func (pfs *peopleCopyFromSource) Next() bool {
 pfs.currentCsvRow, pfs.err = pfs.reader.Read()
 if pfs.err != nil {

 // when get to the end of the file return false and clean the error.
 // If it's io.EOF we can't return an error
 if errors.Is(pfs.err, io.EOF) {
 pfs.isEOF = true
 pfs.err = nil
 }
 return false
 }

 if pfs.isBOF {
 pfs.isBOF = false
 return pfs.Next()
 }

 return true
}

func (pfs *peopleCopyFromSource) Err() error {
 return pfs.err
}

You can now use this implementation in the CopyFrom method. e.g.

1

_, err := pgxConn.CopyFrom(ctx, pgx.Identifier{"people"}, peopleColumns, newPeopleCopyFromSource(csvStream))

Conclusion

Using the CopyFrom with CopyFromRows or CopyFrom will significantly increase the memory comsultion of your application. The high memory usage can bring several problems like OOM errors, increase of costs, unavailability, etc.

By using a custom implementation of CopyFromSource will make your application much more efficient, reliable, and cheaper to ru.

You can find the entire source code of the examples above on this repository. There you will also find more deatils about the comparisson and the not-so-great implementation.

🔢 Go: Where is the decimal type?

flavio1110@gmail.com (Flavio Silva) — Tue, 21 Mar 2023 14:00:00 +0100

Go doesn’t have a primitive decimal type for arbitrary-precision fixed-point decimal numbers. Yes, you read it right. Therefore, if you need to deal with fixed-point precision there are two main options:

Use an external package like decimal, which introduces the decimal type. However, the current version (1.3.1), can “only” represent numbers with a maximum of 2^31 digits after the decimal point.
Use int64 to store and deal with these numbers. For e.g. given you need 6 precision digits, therefore 79.23, 23.00, and 54.123456, become respectively 79230000, 23000000, and 54123456.

There is an open proposal to add decimal float types (IEEE 754-2008) in the std lib. However, for now, it’s just a proposal being discussed, without guarantee it will be ever added.

🧠 Go resources for beginners

flavio1110@gmail.com (Flavio Silva) — Sun, 19 Mar 2023 14:00:00 +0100

Below a living list of compiled links for whoever is learning Go.

Official docs

Effective Go - https://go.dev/doc/effective_go
- A document that gives tips for writing clear, idiomatic Go code. A must-read for any new Go programmer. It augments the tour and the language specification, both of which should be read first.
Go Code Review Comments https://github.com/golang/go/wiki/CodeReviewComment
- This page collects common comments made during reviews of Go code, so that a single detailed explanation can be referred to by shorthands. This is a laundry list of common mistakes, not a comprehensive style guide. You can view this as a supplement to Effective Go.

Go Docs - https://go.dev/doc/
- Root for many useful documentations
FAQ - https://go.dev/doc/faq
- Answers to common questions about Go.

Blogs

Dave Cheney - https://dave.cheney.net/
- Specially the practical go section, it has TONS of good advice and real-world examples of how to deal with daily challenges. I highly recommend it.

Courses

The way to go - https://www.educative.io/courses/the-way-to-go
- It’s a course from educative.io, it goes from the basics concepts of the language to more advanced ones. It also brings very interesting insights into the differences between the approaches of Java/C# to Go..
How to code in go - https://www.digitalocean.com/community/tutorial_series/how-to-code-in-go
- A great collection of tutorials that cover basic Go concepts, ideal for beginers

Videos

Go in 100 seconds - https://www.youtube.com/watch?v=446E-r0rXHI&t=38s
- Short introduction about Go
Simplicity is complicated - https://www.youtube.com/watch?v=rFejpH_tAHM
- Rob Pike talks about how Go is often described as a simple language. It is not, it just seems that way. Rob explains how Go’s simplicity hides a great deal of complexity, and that both the simplicity and complexity are part of the design.
Concurrency is not parallelism - https://www.youtube.com/watch?v=qmg1CF3gZQ0&t=1582s
- Rob Pike talks about concurrency and how Go implements it
Understanding channels - https://www.youtube.com/watch?v=KBZlN0izeiY&t=1011s
- Channels provide a simple mechanism for goroutines to communicate, and a powerful construct to build sophisticated concurrency patterns. We will delve into the inner workings of channels and channel operations, including how they’re supported by the runtime scheduler and memory
Concurrency in Go - https://www.youtube.com/watch?v=\_uQgGS_VIXM&list=PLsc-VaxfZl4do3Etp_xQ0aQBoC-x5BIgJ
- Playlist with a few short videos about different components of concurrency in Go
Just for func - https://www.youtube.com/watch?v=H_4eRD8aegk&list=PL64wiCrrxh4Jisi7OcCJIUpguV_f5jGnZ
- A very complete playlist of tutorials given by Francesc Campoy, a past Developer Advocate for the Go team at Google, that cover simple to advanced topics in Go
Golang crash course - https://www.youtube.com/watch?v=SqrbIlUwR0U
- A 90 minutes video that covers most of Go features with cristal clear live coding examples, excelent for beginers to get a fast gist of Go

Podcasts

Go Time - Spotify
- Diverse discussions from around the Go and its community