Postgresql on Flavio Silva

🤖 Building a propose-then-approve bot engine

flavio1110@gmail.com (Flavio Silva) — Tue, 14 Jul 2026 14:00:00 +0100

After every race weekend on apexclub.live I used to do the same three things by hand. Enter the starting grid, enter the final classifications, trigger the scoring calculation. Every step was a few clicks and a few minutes. The annoying part wasn’t the time. It was the kind of work. It’s detectable, it’s repetitive, and getting it wrong silently corrupts a season’s standings. So I wanted it gone, but only on my terms.

I wanted something autonomous enough that I didn’t have to remember. Deterministic enough that I could trust it. Small enough that I could still explain every line.

TL;DR: A recurring worker tick runs a cheap Detect query for each registered use case. If there’s nothing to do, no LLM runs. If there’s a candidate, the engine builds a proposal, persists it, sends me a Telegram message with Approve / Reject buttons, and only on my tap calls a typed executor to apply the change. The LLM proposes. The database and the human approve. The executor writes.

The shape of the problem

The original setup wasn’t bad. Race results came from analytics on Monday. The application had a Telegram bot I already trusted for one-off commands. I had a database I could read and write to. The only thing missing was glue.

A naive first attempt was to let the existing chat bot propose mutations during a conversation. Press a button, the model picks a tool, the tool runs. I tried it and closed it. The problem wasn’t Telegram. It was mixing recurring operations with a conversational tool loop. Most ticks have nothing to do: the race hasn’t finished, analytics hasn’t synced, results already exist, scores are already published. Paying for an LLM call to discover that “nothing is ready” is wasteful, and worse, “the model decided to call this tool” is not an audit record I want to publish against a league’s standings.

I wanted a concrete payload, a visible proposal, a named human decision, and a deterministic function that could be retried safely. So I split the work in five pieces:

a detect step that is just SQL, cheap and easy to reason about;
a propose step that builds a structured payload and a human-readable summary;
a persist step that stores the proposal before anything is sent anywhere;
a human decision delivered through a channel I already trust (Telegram, in my case);
a typed executor that applies an approved proposal and is required to be idempotent.

Each piece is small. The interesting bit is the contract between them.

The contract

A use case is anything that can detect a candidate, propose a payload, and apply it. The shape, in Go, is roughly:

1
2
3
4
5
6
7


type UseCase interface {
 Kind() string
 Schedule() string
 Detect(ctx context.Context) ([]Candidate, error)
 Propose(ctx context.Context, c Candidate) (*Proposal, error)
 Execute(ctx context.Context, a *Action) (summary string, err error)
}

The four methods divide responsibility cleanly:

Detect is a cheap precondition, usually a single SELECT.
Propose turns one candidate into a structured payload and an admin-facing summary.
Execute applies an approved action. Idempotency is required.
Kind is the dispatch and dedup key.
Schedule lives on the use case, not in central wiring.

The engine has three pieces that move a use case through that contract:

A runner that ticks on schedule, calls Detect, and for each candidate checks whether an action already exists for the same kind + subject reference. If not, it asks Propose to build a proposal and hands it to the coordinator.
A coordinator that owns persistence and decisions. It stores the proposal, sends the notification, atomically claims an approval, dispatches the executor, and records the outcome.
A registry that maps a kind such as insert_race_results to its executor. Registering the same kind twice panics at startup, which is the right moment to discover a wiring mistake.

The lifecycle of a persisted action is small enough to draw in text:

1
2
3


pending -> executing -> executed
 -> failed
pending -> rejected

A partial unique index allows only one open action per (kind, subject_ref). That single index is what makes the runner safe to call every 30 seconds across multiple application instances, and what makes a second admin press of “Approve” a no-op instead of a duplicate write.

The approval transport can be anything with two buttons. Telegram was the obvious choice because I was already living there for one-off commands. A web page could work just as well. It just has to call back into the same coordinator.

The flow, end to end

Here is the whole path of one piece of work, written the way the code actually runs it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


worker tick (every 30s)
 -> for each registered use case:
 Detect() with SQL
 -> zero candidates: stop
 -> candidate: is there already an action for (kind, subject)?
 yes: stop
 no: Propose(candidate) -> Proposal
 Coordinator.Submit(proposal)
 -> persist action with status=pending
 -> send approval message with action UID
 -> Telegram button press
 -> Coordinator.Approve(uid) (atomic: pending -> executing)
 -> Registry.Dispatch(kind).Execute(action)
 -> mark action executed or failed

The decision to put deterministic work first is the entire insight. The LLM only runs after SQL has already said “there is something to do here”, and only to convert that “something” into a payload a human can review.

Why this set of qualities

A few properties fall out of the shape, not from extra effort.

When nothing needs doing, the worker runs only SELECT queries. No model call, no proposal, no notification. I called that “cheap idle ticks” above, and I underestimated how much it mattered before I built it. Most ticks are idle. Saving a model call on every idle tick is the difference between a system I forget exists and a system I worry about.

Auditability is mostly free. Every action is a row: kind, subject reference, JSONB payload, the human-readable summary, the Telegram message identifiers, who decided it, when, and the final outcome. That’s a complete answer to “why did the standings change on Sunday at 23:14?”.

The executor contract is Execute(ctx, *Action). Rerunning it with the same action must converge to the same outcome. That requirement is what makes a button-mash, a network retry, or a panic mid-flight all safe. The row in pending / executing is the source of truth, and the executor decides whether to insert, update, or do nothing.

There is no global “AI: on” switch. The system is on whenever the worker is running with a configured approval channel. The absence of an admin to approve is the absence of writes.

Failure modes are independent. Each use case has its own schedule and its own advisory lock. A broken results matcher doesn’t prevent scores from being checked.

Two use cases, end to end

The same engine runs two unrelated capabilities on apexclub.live. They’re not hardcoded into the engine. They’re ordinary use cases that happen to be the first ones I needed.

Insert race results

The use case owns kind insert_race_results and schedule */30 * * * *.

Its Detect query is “active-season races in the past with no row in results yet”. When that returns nothing, no LLM runs.

When it returns a candidate, the use case:

matches the main race to a row in an analytics calendar (the LLM only picks from offered candidates, it can’t author the mapping);
pulls qualifying pole and classified finishers from the analytics tables;
resolves driver names to the application’s own grid entries (again, the LLM only picks from offered IDs);
assembles a structured result plus the P1–P22 finishing order;
refuses incomplete mappings and surfaces them as a manual-entry notice instead of a proposal;
persists the payload for approval.

On approval, Execute checks for an existing result before inserting and only writes final positions when none are already recorded. A retry after a partial failure converges without overwriting manual data.

Publish scores

The second use case owns kind publish_scores, also scheduled every 30 minutes. Its Detect query is “an active-season race with a results row, not yet finalized, no scores row, and at least one real starting position recorded”. Same shape, a SELECT that asks whether the next step’s preconditions are met.

The proposal payload is just a race identifier. On approval, the executor runs the existing scoring code and flips a finalized flag. Re-running the executor upserts and sets the same boolean, which converges safely.

There is no direct call between these two use cases. The database state is the hand-off: the results executor creates the data that makes scoring eligible on a later tick. Independent schedules, independent advisory locks, independent failure handling.

The traps, in order

Building the engine was less work than fixing the things I got wrong the first time. A few worth naming.

Approval state can stick. The first version moved an action from pending to executing before calling the executor, which prevented two button presses from racing. But if the executor panicked, the action stayed in executing, and because executing counted as open, the runner could never propose it again without a manual database update. The fix was to move the lifecycle into the coordinator: a panic is converted into an error, and Approve is the only transition that marks an action failed. Atomic claiming was only half the story. Every transition needs a recovery path.

Source identities are fuzzy. The first attempt matched races by season and round number. Real data proved that wrong. For one British Grand Prix, the main calendar said round 11 while analytics said round 9, and analytics round 11 was a different race with no results yet. The LLM’s job in this system is to resolve exactly the kind of ambiguity that round numbers and slightly different race names cause. The authority stays narrow: the model picks from offered source rows, never authors them. Deterministic code then checks whether the chosen mapping is safe enough to propose.

The approval message is part of the safety boundary. A Telegram message that shows only the top three drivers is not a complete proposal if the scoring system needs the full finishing order. Anything required for approval belongs in deterministic presentation, not in prose the model is free to shorten. The pattern that worked: build the visible fields from the assembled payload in code, and append the model-generated summary as commentary. The fields are guaranteed. The commentary is colour.

Preconditions encode data completeness. Scoring requires both a result and a starting grid. A result without a grid produces a plausible-looking but silently-wrong score. The Detect query for scoring was the right place to encode that requirement. The engine should refuse a valid-looking operation when its inputs would produce an incorrect result.

These weren’t architectural mistakes. They were the price of admitting the second iteration. The contract didn’t have to change. The implementations got better at honouring it.

Tradeoffs and limits

A few honest costs.

Latency is human-shaped. A use case that detects at 30 seconds and a human who approves five minutes later means a five-minute gap between readiness and execution. For race weekend admin that’s fine. For anything time-sensitive, propose-then-approve is the wrong shape.

Approval delivery is one channel. Today the only way to approve is Telegram. If the notification fails to send, the action is persisted in pending and stays there until a human notices or a redelivery job is added. Persistence beats loss, but it’s not the same as a guaranteed delivery.

One primary approver. Notifications go to the first ID in the allowlist. Every allowlisted user can press the buttons, but only one gets the message. A real multi-admin workflow needs a delivery and decision policy, not just an allowlist.

Rejected actions don’t auto-retry. The repository layer suppresses future proposals for a rejected (kind, subject_ref) to avoid nagging. There is no “retry this one” operation that doesn’t also require a policy or data change. For one-shot work that’s the right default. For evolving inputs it can feel stubborn.

Fuzzy matching is bounded. The model can only pick from offered candidates. That’s a strong guardrail and also a real ceiling. When the input source gets a new spelling or a new driver, the matcher will refuse rather than guess. Human review remains mandatory.

Where I’d take it next

The next changes I’d make are unglamorous:

A redelivery job for pending actions whose Telegram message failed.
A web approval page that calls the same coordinator. It doesn’t need to replace Telegram. It just needs to be a second transport.
An explicit retry / reopen transition for rejected subjects, gated by policy rather than by data manipulation.
More use cases. The shape proved out on results and scoring. The next two I’m eyeing are joker-round announcements and end-of-season standings exports. Both are detectable, both are one-shot, both have a payload I want to review before they go out.

The thing I would not change is the order. Cheap detect, deterministic propose, human approve, typed execute. Each piece is replaceable. The order is the engine.

The engine is small because it refuses to solve every agent problem. It works for recurring, detectable, one-shot changes with a payload that can be reviewed before it’s applied. “Autonomous” here means the system notices and prepares work by itself. Authority to change league results still belongs to a person.

What kind of recurring work would you trust it to propose, but not to approve?

🐔 What happened with my project on the Rinha de Backend challenge

flavio1110@gmail.com (Flavio Silva) — Mon, 28 Aug 2023 14:00:00 +0100

I participated in the Rinha de backend (pt-BR) or “Backend Rooster Fight” in English challenge and I was super excited to see the results of my super hacky and performant solution.

The results were scheduled to be published at 21:00 BRT (2 AM CEST my local time). So I stayed up til later that night grabbed popcorn, and waited for the results.

Well, I wish I hadn’t waited…

TL;DR; My solution didn’t run on the official test server because I built the docker image to linux/arm64 instead of linux/amd64 🤦🤦🤦 . I fixed the issue and ran the same tests afterward and I reached ~50% of points of the winner, mainly because in my solution I didn’t tolerate any risk of losing writes. Checkout my repository flavio1110/rinha-de-backend with my solution.

What was the Rinha de Backend?

Before we get into the details, a brief explanation of what was the challenge.

From July 28th to August 25th, the Backend Fight was held, a tournament in which the API that supported the most load during a stress test would be the winner. Participants had to implement an API with endpoints to create, query and search for ‘people’ (a kind of CRUD without UPDATE and DELETE). In the tournament, participants still had to deal with CPU and memory restrictions – each participant had to deliver the API in docker-compose format and could only use 1.5 CPU units and 3GB of memory. More details on technical aspects can be found in the instructions) – (translated from its repository in English).

On top of the description above, some things important to highlight are:

You can write your API with any language or framework, as long as you can build a docker image out of it
The solution has to run two instances of the API in parallel.
The requests will be balanced as you wish between these instances via Nginx.
You have to use one of the following data stores: MySQL, PostgreSQL, or MongoDB.

It was a super fun, informal, practical, and a very good opportunity to exercise and learn new things.

How the winners were determined?

To make the comparison easier and fun, the single parameter used to determine the winner of the challenge was the number of people inserted into the database.

How did I do?

POV you are looking at me while I check the results.

I didn’t. I was disqualified because the container with my APIs didn’t start. Therefore I got nothing. 🤦🤦🤦

Wrong platform?

Yes, that’s true. My solution didn’t even run because it failed to start the APIs. I built the images for linux/arm64 instead of linux/amd64.

It was a mistake on my end, and it made me super disappointed. I ran the tests on the CI pipelines, and it gave me confidence the image was properly built and I didn’t double-check the requirements. Shame on me.

Well, I was disqualified but I was curious to see how it would perform on a server with the same configuration as the one used for the official tests. So, I fixed the platform of the image, spun up an EC2 with the same configuration, and finally ran the tests. The results were a bit disappointing compared to the TOP 10.

How did I approach the problem?

I had two constraints in mind while designing and building:

It has to work – each endpoint has to do what it is supposed to do.
Keep things as simple as possible.

Each of these points brought a few tradeoffs that ended up limiting the overall performance of the solution.

You can check my solution on flavio1110/rinha-de-backend. It’s built with Go and PostgreSQL.

Given many APIs were performing super well, a few days before the last day of the challenge, the load for the stress test was doubled, and it had a massive impact on the performance of my API. You can check the comparison below:

Before…

Print of part of the report generated by Gatling of my solution run on Github BEFORE the load was doubled.

After…

Print of part of the report generated by Gatling of my solution run on Github AFTER the load was doubled.

What were the bottlenecks?

The inserts were the bottleneck. Because I didn’t tolerate the risk of losing writes and wanted to keep it simple, I missed the opportunity to introduce a distributed mechanism to validate the data before trying to insert it into the database and perform the inserts in batch.

Validating the uniqueness of a field Apelido (Nickname)

On my designed solution, without a distributed cache, it was not possible to perform without the DB in 100% of the cases.

I had an in-memory cache, but if the nickname wasn’t there because it wasn’t there because the entry was inserted via the other instance, that request would reach the DB, occupy a connection, take resources, etc.

Introducing a distributed cache like Redis would enable me to add and check the entry in a single place. As a result, we could decrease the memory necessary to run the APIs and move to the cache.

Batching inserts

I inserted each entry per a valid request because I didn’t want to risk losing valid writes in case there was any error between the construction of the batch and its execution.

This was a huge problem because it used too many DB connections and resources, and slowed down the creation request.

Did I like it?

Despite my results, I loved the challenge! I learned quite some tricks with nginx config and some insteresting stuff like using Redis or even PostgreSQL as a “PubSub”. The interaction with the community via Twitter and Github was super nice! I also liked the fact of working with something challenging and closer to reality. Loved it!

What could I have done differently?

Build the image using the correct platform.
A challenge is a challenge. I should have been more flexible with the writing.
Tweak the Nginx configuration to load balance based on fewer connections and disable logs.
Use a distributed cache and drop the read table.
Batch inserts.

What’s next?

Check the official results and look into the repositories of the participants. I guarantee I’ll learn something new.

Stay tuned on @rinhadebackend for the next challenges, I can’t wait for the next ones.

Other than that, I’ll apply the lessons learned to my solution and hopefully get better results. Watch flavio1110/rinha-de-backend and see how it will evolve.

This challenge is officially over, but you can still do it. Challenge accepted?

📂 Go: Importig a CSV to PostgreSQL

flavio1110@gmail.com (Flavio Silva) — Sun, 26 Mar 2023 14:00:00 +0100

If you are using Go ad PostgreSQL, and need to performa a bulk import a CSV, it’s most likely you will find the COPY protocol is the feature that suits you better. In that direction, you will find examples using pgx CopyFrom that relies on the native protocol, and it’s fairly easy to use. However, depending how it’s used you can have an exponencial increase of memory consumption of your application making it unreliable and more expensive to run.

TL;DR;: Don’t load the file in memory and use pgx.CopyFromRows. Instead, use the io.Reader of the file and implement a custom pgx.CopyFromSource.

Checkout the repository with examples and details presented here.

Context

Most of the examples out there, are either using CopyFromRows or CopyFromSlice. However, the big problem is these two options require you to have the entire content in memory to use. This is not a big deal when dealing with small files, there won’t be concurrent usage, or you have infinite memory 😅.

How big is the problem?

Comparing the memory consumption for the two distinct approaches importing a file with ~16MB (1M rows).

Approach/metric	TotalAlloc	Sys
Stream file	61 MiB	12 Mib
Read entire file	84 MiB	58 Mib
	+37.70%	+346.15%

Yes, you read it right! using CopyFromRows obtained +346.15% of memory from the OS! 58 MiB instead of 12 MiB.

You can read more about the meaning of each metric on https://golang.org/pkg/runtime/#MemStats and find the source code and details of the comparisson on this repository..

What’s the most efficient way for using the COPY protocol with pgx?

Instead of reading the entire in memory, the idea is to stream each line of the file directly to PostgreSQL. In this way, we only need to keep the current line in-memory as opposed to the entire file.

How can we do it?

The CopyFrom method receives an implementation of the interface CopyFromSource.

Let’s implement this interface using a CSV file with the followng three columns: first_name, last_name, and city.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62


func newPeopleCopyFromSource(csvStream io.Reader) *peopleCopyFromSource {
 csvReader := csv.NewReader(csvStream)
 csvReader.ReuseRecord = true // reuse slice to return the record line by line
 csvReader.FieldsPerRecord = 3

 return &peopleCopyFromSource{
 reader: csvReader,
 isBOF: true, // first line is header
 record: make([]interface{}, len(peopleColumns)),
 }
}

type peopleCopyFromSource struct {
 reader *csv.Reader
 err error
 currentCsvRow []string
 record []interface{}
 isEOF bool
 isBOF bool
}

func (pfs *peopleCopyFromSource) Values() ([]any, error) {
 if pfs.isEOF {
 return nil, nil
 }

 if pfs.err != nil {
 return nil, pfs.err
 }

 // the order of the elements of the record array, must match with
 // the order of the columns in passed into the copy method
 pfs.record[0] = pfs.currentCsvRow[0]
 pfs.record[1] = pfs.currentCsvRow[1]
 pfs.record[2] = pfs.currentCsvRow[2]
 return pfs.record, nil
}

func (pfs *peopleCopyFromSource) Next() bool {
 pfs.currentCsvRow, pfs.err = pfs.reader.Read()
 if pfs.err != nil {

 // when get to the end of the file return false and clean the error.
 // If it's io.EOF we can't return an error
 if errors.Is(pfs.err, io.EOF) {
 pfs.isEOF = true
 pfs.err = nil
 }
 return false
 }

 if pfs.isBOF {
 pfs.isBOF = false
 return pfs.Next()
 }

 return true
}

func (pfs *peopleCopyFromSource) Err() error {
 return pfs.err
}

You can now use this implementation in the CopyFrom method. e.g.

1

_, err := pgxConn.CopyFrom(ctx, pgx.Identifier{"people"}, peopleColumns, newPeopleCopyFromSource(csvStream))

Conclusion

Using the CopyFrom with CopyFromRows or CopyFrom will significantly increase the memory comsultion of your application. The high memory usage can bring several problems like OOM errors, increase of costs, unavailability, etc.

By using a custom implementation of CopyFromSource will make your application much more efficient, reliable, and cheaper to ru.

You can find the entire source code of the examples above on this repository. There you will also find more deatils about the comparisson and the not-so-great implementation.