If you are already doing code reviews, you might not think a lot about it. But what if it's a missed opportunity? What if you could get even more out of your Pull Requests?

Note: In this article, I might use pull request (PR) and code review interchangeably. To be precise, a pull request is a thing you create in some tool, containing some description, diff of changes and can be approved/declined, and code review is the process of reading the code. For the sake of this article, I don't feel like these two need to be differentiated.

Code review goals

Let's start by answering "Why should we even do code reviews on our team?". I think it's not that straightforward to answer, there's no single answer to this question. As usual, it depends and varies from team to team, but here are some things I find beneficial from reviewing the code:

Knowledge sharing

This is the biggest thing in my opinion. The ability to easily transfer knowledge between team members. And it does not even matter whether they are seniors, juniors, or any mix of these. Knowledge sharing is always happening and is always beneficial. In my case, it happened multiple times that some innocent and small comment on PR started a bigger discussion, sometimes even one that I could call a philosophical one, like the int vs var/let. And I think it's a good thing, since such discussions, if are moderated properly (so we're exchanging our opinions and giving arguments, instead of trying to push as hard as possible onto our favourite method), can lead to the growth of all members within the team. Code reviews are giving opportunities to such discussions, they act as conversation starters. After all, it's not a regular practice (at least not in my case) to approach my coworkers and ask them random questions, like "Hey, how do you usually decide that the function is already too long?" or "Have you learned something new recently you could share with me?".

Quality improvement

This is another big benefit of code reviews. You always have at least a second pair of eyes, at minimum skimming through the code and checking if you made some of the most obvious mistakes. Depending on the type of change and reviewer, you might get a more detailed review or just a quick look at the code. While I prefer more detailed reviews, they also take more time and energy, so I can understand it's not always possible.

When I was at the university, I had classes on waterfall methodology. In waterfall, you have multiple separate stages (requirements / design / implementation / ..) and identifying a bug in each next stage increases the cost of fixing a problem by 10x. I'm not sure how true it is, but I believe that something similar happens on pull requests - when you catch a bug in PR instead of in a test environment, you save yourself some time. If we compare it to catching a bug in a production environment, the cost is usually even higher. It does not matter whether we're talking about business logic bugs, or technical ones (like memory leaks), it's always better not to introduce them.

Tests

Depending on your policy on testing, PR is a good point to ask "Is it possible to add some tests?". I also sometimes sit down and try to understand what the tests are testing and whether there are some problems in the tests themselves, or in test coverage.

Keeping up with changes

If the project is big enough, everyone can't know what's happening in every module. Code reviews are a cheap way of keeping either everyone or some subset of developers up to date with the codebase, without investing too much time into it. They don't have to do the implementation, all they need to do is to spend significantly less time from time to time checking out the PRs.

What should not be a part of a code review

There are also some things that PRs and code reviews are not about. These are things like:

• Code style, formatting, newlines, etc.

• Build checks (reviewer making sure that app builds)

• Test checks (reviewer running the tests on their machine)

These things can be easily automated. Once the code does not compile or tests are not passing, it should be impossible to merge the code into some sort of main branch and there's no need to waste the reviewer's time to check this.

The same thing goes with formatting - as a team, you should agree on a set of rules, set up automatic linter and either enforce linting or fail the build on failed linter check.

Code review is about code, not about the author's ego

As a code review author, remember: it's not about you. It's about the code. What I mean by that is that when someone points out that you've introduced a bug, there's a problem with your architecture, or the code is hard to understand, try not to take it personally. There's an API rule which I try to follow - Assume Positive Intent. It simply means that you should assume the person on the other side has positive intent. This is especially true in the remote world - it's hard to pass the emotions through the review comments. Whenever there's a space for uncertainty whether the person has good intent or is just malicious, always assume the good intent. It's a really small thing, but can massively improve the atmosphere at work. From my experience in almost all cases, people have good intents at work and even when they're wrong or are defending something obviously incorrect, it's possible to calmly talk it through. And most importantly - it always might be you who's wrong. This way you avoid being a jerk.

Different approaches

I've been working in multiple teams, and each team had way different approach to code reviews. In some teams we had no code review and I think this approach is wrong and no matter what. The code review should be a part of every development team.

I like the 'deep dive' approach to PRs. So spending a long time reviewing the code and trying to understand all the "what's and why's" of the reviewed code. But there's also a different approach which I also think is fine.

A "shallow review". It takes a small amount of time and it's just meant to catch some bigger or more obvious problems. The reviewer skims over the code and stops only after something catches his eye. I would recommend this technique for experienced teams where you expect that you can trust all the devs that they know what they're doing and they're taking responsibility for their code - without taking responsibility it's a recipe for tragedy.

Code review comments

Writing good code review comments is an art. What I've found working the best is 'ask don't tell'. So instead of writing 'Add some unit tests', ask: 'Is it possible to add some unit tests?'. This way you open the gates for discussion. Because it's not always that the other person does not want to do it - sometimes there's a good reason why it wasn't done in the first place. I cannot count how many times I've approached the code with the attitude 'Why the hell is it done in such a weird way? Let me fix it real quick' only to realize an hour or three later why and back off.

Second thing - be polite. It might be the most important point. If you write a comment 'Why are you using library X instead of Y, are you stupid?', I can almost guarantee you that the other person will either do everything not to do your way or will back off without any further discussion. This is of course a bit exaggerated example, but I've seen cases similarly extreme to this one.

Let's also try to approach the problem from the other side. What if you've received a comment ending with '...are you stupid?' How can you approach it? I believe there is only one good solution - to talk it through. First I would maybe go to another coworker and ask 'Do you also think that this comment is not appropriate?', to gather a second opinion. When I'm in emotions I don't always think straight so it's good to have someone to ensure you're not overreacting. And then when I get the confirmation, I'd go and talk with someone. Usually it would be a team leader or manager, but if you're feeling brave, you can go to the person directly. If the person is not just 'mean by definition' but is having a bad day, I can guarantee you that they will appreciate your openness a lot. And if they won't, you don't want to work with them and the problem is way deeper.

Remember to NEVER respond in anger. If you feel like one of the comments touched you deeply, go for a coffee or a quick walk. Go to the nearest window and look at the nature for 3 minutes. Go to another coworker to talk about their kids. Do everything but respond right away. These 3 minutes can save the atmosphere and give you bonus points for your ability to handle difficult situations. Another outcome might be that when you come back with a cookie and coffee, you'll realize that it's not that bad, the reviewer was kind of right and maybe you're the one having a bad day. You won't become a drama queen.

Another idea for review comments is adding comment size or relevancy to it. One of the techniques is the 't-shirt size' thingy. At the very beginning of each comment, you put a t-shirt size. Let it be:

• [s]: you can freely ignore this comment

• [m]: it's something you should either fix or we should discuss

• [l]: I'm convinced that there's a bug here

or a lighter version of it, just marking 'irrelevant' comments as nitpick with 'nit'. The difference between a regular comment and a nit comment is that nit comment is usually about the thing where you either want to slightly suggest some change but it's so small that it does not really matter if it stays or not. It might be also something that is a personal preference which you know you should not enforce. It can also be a 'fyi' comment - when a thing is implemented in a correct way, but there's a new feature of the language that allows to simplify it.

Have a call besides comments

When you have a non-trivial review, it sometimes means that you've received or created multiple comments. Some might be big and important, and others you might be unsure of.

How you could approach it as a reviewer is the following: first of all sit down and try to understand the code and changes by yourself. Go file by file and review it the best you can. If you don't understand something, write a comment that you don't understand or you're unsure of something. After you've done the whole review, grab the author on a call. From my experience, they tend to be long calls, since you're probably discussing potential problems and solutions. On the call you can walk comment by comment and clear all unknowns, ultimately saving time, and also bringing both of you onto the same page. Sometimes people are willing to just 'do what the reviewer wrote without too much thinking' instead of trying to understand. This way you can also explain the 'why's' behind the comments and your thinking.

The same goes when you're the author of PR. If you feel like you don't fully understand what reviewer meant by the lengthy and complex comment (or maybe a small and simple one?), just ask them to hop on a call. In the long term it tends to save a lot of time.

What should I look for at the code review

It also varies a lot from team to team. But usually, some core things remain the same among different teams. These are:

• Is the business logic implemented according to requirements?

• Are there any technical problems with implementation? Could it be improved / optimized / rewritten according to team standards?

• Typos

• Tests - are there any tests? Is the coverage enough? Are the tests on a correct level? Are there any cases not covered by tests?

• Documentation - are the changes documented properly?

• Overall compliance with team agreements

If you have agreed to have some steps as part of code review (for example, you should manually test each PR before merging) you could try to make a template which will either automatically or manually be pasted into the PR description so that you can publicly tick off points you already did and every reviewer can check it out without explicitly asking.

Review in browser vs local IDE

Should you review the code in the browser or should you use your local IDE? I've used multiple online platforms: GitHub, Bitbucket, Upsource, Azure DevOps. They all differ by a bit but have one thing in common - they are not IDE. They all don't have the capabilities of your local dev environment. You cannot run tests (okay, you have the CI pipeline, but you get my point). You can sometimes go to the definition of function or variable, but not always. You don't have the full capabilities of mouse-over something to check its type and usages. It's closer to Notepad than IDE.

But on the other hand, they are all always one click away. You don't have to checkout the branch locally. You don't have to stash your changes. You can click some URL in a browser and that's it, you're ready to review. Also, they sometimes have some nice features like marking which commit / file / batch of changes you've already reviewed and it's saved for later.

The decision to do a review online or locally might seem to be minor and unimportant, but I think that it's significant. Maybe not crucial, but it might affect the way you write code. For example, let/var vs type name - if you are in your local IDE, you can look it up by hovering the mouse over it. In the browser, it's usually not available. In such case, you should write full type names for readability.

Another thing is the context. When I review the code, in bigger PRs I like to checkout the code locally and open it in IDE. Then I start reviewig the code in some file and jump around between functions trying to understand the flow. So instead of going top-to-bottom by filename or folder name, I would be guided by the flow of logic. This way I'm reviewing not only the code as is, but I'm also able to build a bigger picture about the changes and business logic inside. I also have easy access to the rest of the codebase, so I can quickly look up code that is not part of reviewed changes.

So, which one is better? I don't know. I think that it depends on the team you're in. I am comfortable with doing both.

Code review size

How big my code review should be? It also depends. If you have to wait a week for the review, I'd say 'as big as possible'. If you can get your code reviewed right away, I think that 'as small as possible' might be the answer. But I don't have a definite answer here. Smaller PRs encourage reviewers to do the review faster. I am also aware that unfortunately there are a lot of people who will become over-nitpicky with small PRs and when they get a big PR, they will skim over it, approving the changes without actually reviewing.

Another factor is the completeness of the feature you're implementing. Should you split PR into (code) and (tests) PRs? Probably no. But depending on the exact details of a feature you're implementing, you could try splitting it into 'vertical slices' - implement a part of a feature that is isolated and a complete piece of code, make a PR, merge, and start working on another small piece. You need to try out different sizes and see what works best.

When should I review PRs

No later than half a day from any point of the day. Code review is often a blocking process, meaning that if you don't unblock your peers regularly, they get to the point where they have 7 PRs open and they are getting lost in it, not to mention reviewing this all. If you don't like to be interrupted during your day, you have at minimum two points when you're already 'not in the zone'. The first one is when you've just started working in the morning. The second one is after your lunch. Reviewing code is working towards the whole team's goal so you shouldn't wait with review longer than necessary. I try to do the reviews as often as possible. If I'm busy with some critical tasks and someone asks me for a review, then I ask them if the review can wait. It should not happen too often, but in reality it does sometimes. Then they at least know that they'll have to wait. They can find someone else to review their code, wait for me to finish off what I'm doing currently or we could discuss re-prioritization. I'm talking about bigger reviews (let's say 30+ files, 100 lines changed in each) requiring even up to a couple of hours to review. Tiny PRs have a smaller risk of having this problem.

When should I create PR

Again, this depends on multiple factors. The first one might be CI pipelines and tests. Are there any slow-running tests that can be run only in the pipeline? Or the ones being too slow to run locally? If the answer is yes, then it might be beneficial to create PR right away if you have a pipeline tied to PR creation.

Another thing to consider is catching big problems early. The earlier in implementation you find that the architecture decision you've made is bad, the less time you've wasted working on a code that will get deleted anyway.

If none of the above applies, I'd say that you should probably create PR once the feature is fully implemented, or at least 90% implemented (so that you have a chance to implement the missing test while someone reviews the code, or improve insignificant things alongside with PR changes from comments.

Pair programming - can we skip the review?

What if you pair-programmed a feature? Can it be treated as already reviewed?

The answer I give to this question is "probably yes". It of course depends on how cautious you are while coding. In some cases, a quick self-review would be beneficial.

Self-review

I like self-reviewing the code before I hand it over to my teammates. It has two main benefits:

1. You gain credibility in your team - once you create a PR, there's a high level of trust that your code works. It has advantages in situations like yearly reviews where you can bring it up.

2. You can catch a lot of things by yourself, without even bringing another person to the table. You save everyone's time, and once reviewers see that you're not wasting their time submitting half-baked reviews, they at least should appreciate it and make the review faster.

So, how to do it? In the same way as you would do any other review. Grab your favourite tool, create a draft PR (or some equivalent of that) and go file by file, looking for potential problems, leftover code and typos.

I don't think that self-review can fully replace second pair of eyes looking at your code, but it's a good habit.

Self-review has also a second benefit. You don't need another person to do it. What it means is that it's a perfect tool for some 'emergency' situations when you don't have a reviewer. This technique saved me some time and stress when I was doing quick hotfixes outside of regular business hours when no one else was working. To do a self-review, create a PR as you would normally do, and go for a 5-minute break. This way you are leaving 'the zone' and when you get back to your computer to make a review, you are as close to being an independent reviewer as possible.

Should author of PR fix code outside of theirs changes?

It often happens that there's a piece of code that needs to be refactored but is not a part of current changes. These are either some TODOs, code containing bugs, or the whole module written in the 'old way'. It happens that you receive a comment asking ‘Hey, could you fix this TODO while you’re here?’ Should you touch it or not?

As usual, it depends. If it's a small and low-risk change, do it. Maybe not all of them if there are multiple, but just a few. Try to make the surroundings just a little bit nicer. If you don't do it, the code will slowly rot and turn into unmaintainable spaghetti over the years.

Sometimes the changes are too big to address them right away. What you could do is create a ticket or write down on your list 'refactors to be done while I'm bored'. And then when you have some slack time between two tasks just start grabbing things from that list.

Let's fast-forward to today. Today I'm writing F# code in Rider, a .NET IDE from JetBrains. But since the project I'm working on is quite big and F# is still a niche, the whole experience if far from ideal. It's better than 5 years ago, but still it's quite often that the code highlight just stops working or function references do not load.

Going back from nice and cosy C# environment where the whole IDE guided me through almost everything into the dark pits of F# development where you're kind of left by yourself is sometimes hard. There are times when I feel like I would be writing some C at the university many years ago - instead of having a nice 'it seems like you forgot to close the parenthesis somewhere around this line' I get the 'AAAAAAAAAAAAAAA THIS CODE IS ALL WRONG, THERE IS SOME ERROR SOMEWHERE IN THIS FILE BUT I DONT KNOW WHERE SO I WILL UNDERLINE EVERYTHING IN RED' which is not too helpful. But this is also an opportunity. Opportunity to step back and think about my needs and wants when it comes to my code editor.

What I'm using today

The project consists of backend in F# and frontend in React. Currently I'm using at maximum 60% of Rider's capabilities (the remaining 40% is not working because it's F# not C#) for backend projects, and nvim for frontend project. I'm not fully happy about this setup since it's not convenient to switch back and forth between this two editors. I should call Rider good enough, but after I've experienced the unlimited possibilites of customizing my experience with nvim, I don't enjoy the plainness of Rider. I have vim motions plugin installed, but it's only part of the neovim experience™. On the other hand, the amount of hours I had to put into my nvim config might be discouraging, especially at work. In my free time it's fine for me to spend 4 hours tinkering with my configs. At work it's not feasible to do it on a regular basis. I just need something that works out of the box, or at least something as close to this as possible.

Why I don't use nvim for F#

There are basically two reasons why I still stick to Rider as my main IDE: debugging and tests. I already can and do sometimes run all projects from console anyway. I have configured nvim so that I have a file tree, fuzzy find and jumping around the code using LSP so that I can go to definition or list all usages.

It is technically possible to run debugger inside nvim, but I still have something I would call a mental bareer - in Rider it just works perfectly. I would have to spend probably tens of hours trying to set up debugging and learning it, with no guarantee that I won't decide that it makes no sense and return to Rider. Especially that I've already looked into it and it seems like there are a lot of people using nvim regularly who dropped this idea of working in .NET environment in nvim because of Rider being too good out of the box.

The second problem I imagine I would have is running tests. I mean the ability to run single test from under the cursor. It should be possible to set it up in a way that I could just run the test from under cursor or in the worst case I'd just copy the name of the test I'd like to run and pass it at a parameter to console command, but hitting (CMD + TR) which just works at all times out of the box is too convenient.

Visual Studio Code

There is a middle ground between a bulky, performance-heavy editor (Rider) and configuration-heavy editor (neovim) and it's VS Code. At some point of my career I was amazed by how well vscode actually works. You want to run a python script? Just open the file, click 'yes, I want to install everything' on a popup asking if you want to run this file, and then just hit F5. At least this is my experience with vscode. It just works. But the problem I have with vscode is that it's another editor with its unique properties and shortcuts that I'd have to learn.

I was already very comfortable using mostly Rider for my daily work, with some support of vscode for some more generic and non-C# tasks. Then I've discovered neovim and decided to spend a lot of time to not only configure it, but also to learn how to use it properly. To learn unique features, find plugins, set up shortcuts and them learn them all. And the most importantly, to learn vim motions. And at the moment I'm more than happy with being fluent in nvim, but the problem with having this knowledge and skills is that I don't want to go back to the comfy, working out of the box editors where I can change only this much things and the editor's plugins list feels short.

I'm unhappy about my current editors

When I use neovim, I miss the approachability of Rider or vscode. When I use Rider, I miss the speed and lightness on nvim or vscode. When I use vscode, it all feels meh. It's the thing made of compromises. It's okay in everything but does not excel in any of my needs. Or maybe it's only my image of vscode which is untrue? But I don't feel like trying to change my mind here. I don't like vscode. Maybe it's the overwhelming amount of things that are happenig in the background? I'm not sure, right now I am unable to name it, I just don't feel like using vscode more than I have to.

I think that the reason for this whole dilemma is the fact that at some point I've decided to pick up nvim. If I wouldn't do that, I would be 'stuck' with Rider, not knowing that there is more than that. But now I know and this leads to me being lost. Maybe one day I will sit down and start tinkering with the neovim and try to set it in a way that it mimicks Rider's nicest features, at least to some extent. So that it will be good enough to replace it. Way better at some things, and just okay at others.

Zsh, plugins, gitconfig and neovim. For neovim I already have a separate repo to store my config but it was still one more thing to remember.

I've decided to do a research (again) and this time I've managed to come up with a solution. There are multiple ways to approach this problem. Some of them require additional applications, and also you could use bare git repository.

I've went through a couple of solutions and found out that chezmoi works best for me.

Managing dotfiles with chezmoi

My needs are not big. I just want to commit selected dotfiles, and then be able to replace and edit them. And I want to do it across multiple machines. What chezmoi gives me is simplicity.

All you need to do is to install chezmoi, and then follow the quick start guide, like so:

chezmoi init
chezmoi add ~/.zshrc
chezmoi cd
git commit -m 'init'
git remote add # set remote to your repo
git branch -M main
git push

and that's it. Now you have your `.zshrc` backed up in your repo. In order to have this file on another computer, all you need to execute is

chezmoi init https://github.com/$GITHUB_USERNAME/dotfiles.git
chezmoi diff # to see what changes will be applied
chezmoi apply

and that's it. You have successfully synced `.zshrc` file between two computers. And now, you can type

chezmoi edit ~/.zshrc
chezmoi diff #to check what's going to change
chezmoi apply
chezmoi cd
git commit -m 'changes'
git push

and in case you'd edit the file directly (so you forgot to edit it via chezmoi), there is a re-add parameter, so you just type

chezmoi re-add
chezmoi cd
git commit -m 'changes'
git push

and all good!

Another good thing about chezmoi is that it works out of the box with the whole directories - I wanted to sync my `~/.oh-my-zsh` and all it required was

chezmoi add ~/.oh-my-zsh

In case you'd want to use your own editor (like nvim for example) in `chezmoi edit`, all you need to do is to set env variable:

export EDITOR="nvim"

Managing per machine config

I'm using the same .gitconfig on my personal and on my work laptop. Obviously I want to have a different email on both. In order to do this, I need to convert `~/.gitconfig` file in chezmoi from file to template.

chezmoi add --template ~/.gitconfig

and change entry in `.gitconfig` to take email from local variables:

[user]
    email = {{ .email | quote }}

Now chezmoi will tell me if the email is missing, on `chezmoi diff` or `chezmoi apply` command:

chezmoi: .gitconfig: template: dot_gitconfig.tmpl:7:12: executing "dot_gitconfig.tmpl" at <.email>: map has no entry for key "email"

All I need to do is to go to `~/.config/chezmoi/chezmoi.toml` and add email entry, like so:

[data]
    email = "work.email@email.com"

and ofc do exactly same thing on personal computer with personal email. There are ways of further automating things, but I'm not there yet - you can create variables and whole configs per machine name or os type, and even download packages via package manager.

Chezmoi has everything or even slightly more than I was expecting from it, while keeping it stupid simple. I cannot recommend it enough.

You can find my chezmoi repo here

• Full ownership: you build it, you own it

• My users (or metrics) will tell me when things are off

• Single owner of the project

• Canary releases

• Simplicity over everything

• No time for bullshit meetings

For the past almost three years I've been working for a company that I could call a seasoned startup. The company was 10-ish years old, but a lot of things were organized in a way similar to how startups are organized. Bus factor of one - if one person disappears, we have a problem. Not exactly a paralyzing problem since there was usually more than one person which more or less knew the domain, but usually a single person was committing to the particular service.

It also had some other consequences, like having not much support from teammates in daily work. I've been in so-called backend team, which during a couple of years of existing, slowly and steadily transformed into a core team. We did backend. We did frontend. We did data processing. We did everything, we owned every most crucial piece of software. Okay, maybe not everything (yet!), but we owned a big piece of the company's software, and it was just working. I've owned somewhere around 20 different services from various domains, which means that I was the main person responsible for fixing them when something went wrong, sometimes also during weekends or even vacations.

It took years of slow and consistent work, to prove that we're able to deliver. I want to describe my experience of working in this team, what I've learned, and how we worked.

Full ownership: you build it, you own it

This is the most important prerequisite to this all. I had to take full ownership of the services I maintain. I was the only active contributor to the services I owned. Why? Because I will not fix things on Saturday evening that someone else broke by their carelessness. Of course, we had a code review process so I was not left alone, but it was my responsibility to test it before deployment, and if something went wrong, it was also my responsibility to fix it. We had no strict SLAs, but usually I fixed any problems within minutes or hours. It was possible thanks to good monitoring.

My users (or metrics) will tell me when things are off

How do you know that your service is not working the way it should work? Usually the answer could be straightforward: you open the application and check if it works or not. In my case it was not that simple. One of the areas I was solely responsible for was crawlers. Crawling of the internet is hard and unpredictable. I feel like I've seen it all when it comes to the ways of handling errors and failures, and some of them were creative. Response with HTTP code 200 and some carefully, hand-crafted error page. Multiple chains of redirects. HTTP 403 instead of 404 on a page which has been removed. Geo blockers of all sorts. Different sorts of anti-bot protections. Even honeypots for crawlers!

In such a volatile environment it's really hard to know quickly if something is broken or not, especially in distributed services. Crawlers were processing hundreds of messages per second, so the failure rate was also high at all times. Exceptions were thrown all the time, in every second. Some of them were expected and recoverable, and some of them were unknown unknowns - the things we haven't thought of. Known exceptions are fine - we know what is happening, and can easily and safely determine without a huge rush what's up and fix it relatively slowly. With the unexpected ones, it's a different story. I had a lot of alerts on multiple metrics, such as:

• too many unexpected exceptions during the last hour

• no successful crawl globally for the past hour

• no successful crawl in one country for the past 4/6/12h (depending on how often we were running crawlers)

• avg. amount of successful crawls over the past 3 days vs the past 7 days

• queue count exceeded the threshold

• messages on the queue are older than 4h

• queue message count exceeding some threshold

• and many more

Some of the alerts were more important than others. After some time of working on the project, I had a feeling when it's something that needs immediate attention and when it's a problem that can wait for a day or three, or maybe even I expect that it will resolve by itself. Because sometimes it was totally out of my control - what if the website we're trying to crawl is down? Even worse - what if it's down because we're crawling it a bit too hard? Then the only solution to fix this problem is to wait. And since they might not like the fact that we're crawling them, we cannot write an email saying

Hey, we're crawling you and it seems like our crawlers are failing, could you check it and let us know what's up?

But also there were times when I made some changes, deployed them to prod and I was worried that I might have introduced a bug. And it did happen from time to time. But since I had a lot of alerts, usually I knew right away and was able to fix & deploy the change quickly.

Single owner of the project

I feel like this might be a very underrated thing. It forces people (at least the right people) to shift their thinking. If there is only one contributor to the project, after some time of working you know the whole codebase by heart. You know all the why's and where's (okay, maybe not all). I'll mention crawlers again. I've spent multiple months working almost exclusively on maintaining and improving crawlers. It taught me a lot about the domain and the project. I could understand the project deeply. I got to experience the consequences of my code and design decisions. I had no one else to blame for bugs other than me. When I implemented a feature and I did not fully understand the why's behind it, I usually had to re-do the whole thing a month later, since "You know what? In my head it was working differently, could you change it please?". And such reality taught me a lot. I shifted my thinking about features from "what is needed today" to "today they ask for X, they need Y, and it probably will evolve into Z in the future". Let's take a look at the simple example. Our crawlers had pretty advanced configuration options, allowing for example to set retries on some exceptions, and also to specify which error codes we don't want to retry. When the requirement came in that 'website X is returning 404 (not found) instead of 429 (too many requests), could you please make a config so that we can retry 404s?' I transformed it in my head into 'we need a mechanism of specifying HTTP code into action'. Whenever it was possible, I was trying to make features as generic as possible. Instead of implementing "a flag to interpret 404 as 429" I've implemented "When , then ". And pretty often I was right - usually weeks or months later another requirement came in: "Hey, do you remember that site that was doing 404 on too many requests? We have a new similar website, but they're returning 403s. Help pls". And if I anticipated the future requirements correctly, I was able to save usually a couple of days of development and testing.

Canary releases

Crawlers were running in multiple countries. Some countries were more important to work continuously than others. In other words, we could afford some countries to be broken, and others we could not. So for bigger releases, I've used canary releases. By bigger releases I mean changes in code that I knew that were touching multiple areas, so they can break in multiple places. What I did was I released one country, and then waited for a day or three, depending on importance, size, deadlines and my current capabilities. After the period, I contacted all stakeholders, I've checked the logs, and if everything was looking good, I deployed another batch of countries, this time more significant ones. And for the very last batch, I was leaving the most important and the biggest countries. This way I had multiple checkpoints when I was making sure that everything was fine. Also, it made me extend the deployment period over days or even weeks, so the first country that got deployed had already a significant amount of data proving that the code works fine, even before the last batch of canary has been deployed to prod.

Simplicity over everything

Do you know that feeling when you sit down and try to understand the code, but the number of virtual classes, abstractions, and inheritance makes you jump from file to file and you need to spend hours debugging? I know it very well. Usually the story behind such code in places I've worked in was that there was this 10x developer who was brilliant. He coded the core of the project over the weekend, then over the next 3 months he built more features on top of that, and then he decided it was time to leave the company. And guess who needs to maintain this little monster? Not the person who wrote it!

When I'm alone in the project, I have a lot of room to do things my way. And my way means simple. I like simple code. If there are too many things going on at once, I cannot be productive, since I'm blocked by the complexity. I need to dive deep between smart functions, doing smart things in non-explicit way so that the code is more compressed. But it does not mean that I don't enjoy it when things are one-liners. Let's take regex as an example. Whenever possible, I try to avoid regex. I'd rather write 10 lines of code, doing step-by-step filtering over 1 line of regex doing things all at once (of course there's more to it, like performance, but it's just a simple example). And the only reason is that I know that it will be hard for me to wrap my head around the regex in 6 months. And if it's step by step, then it's easier to not only debug but also read the code.

No time for bullshit meetings

This requires some kind of bravery. To be able to hop into the meeting, and ask 'Hey folks, do you need me at this meeting? I have some deadlines to meet and I feel like I will not contribute to this discussion'. Of course, it does not mean that I was allowed to not participate in any meeting, it's just that I had a sense that my time was worth something - after all, I was the only person responsible for delivering my project on time. And I had to explain myself why I missed the deadline. And if people were pushing for me to do things that I felt were wasting my time and it was unjustified by the business requirements, I was escalating it above so that the 'business owner' of the project (someone who wanted my feature to be completed) could escalate and negotiate on my time allocation. I was just being clear that doing thing X will delay thing Y, that's it.

All the code for this post is available on my GitHub.

What is property-based testing?

Let’s say that you have a system that allows you to order fruits and tools. Strange combination, I know, but bear with me. An order has some properties that need to be fulfilled to be valid. For example, items ordered need to have count of at least one. Or the total price has to be non-negative. Or the name needs to be one of the predefined values of items present in the system. To unit test order handling, you would have to come up with a bunch of test cases. But what if I told you that all you need is to define a set of rules that the order needs to follow, and you will have auto-generated data, each time being different? This is what property-based testing is about. You create generators that will create data for you, and then in your test, you just say “Give me a list of orders”, and generators will create them for you. And it will run (by default) 100 times, each time with different input data. This way you can test more cases with less effort (usually). I have to admit that not always it’s easier to write property-based tests, as they require a lot of upfront thinking, but they are worth it when you have a complex calculations system on a big set of data with many different combinations and it would be hard to cover all the possible cases with unit tests. Now let’s get to the actual code.

Our first property-based test

[Fact]
public void SimpleTest()
{
    Prop.ForAll<int>(x => x >= 0).QuickCheckThrowOnFailure();
}

Let’s try running the test:

System.Exception
Falsifiable, after 1 test (0 shrinks) (10979290545183390790,6994218916298689381)
Last step was invoked with size of 2 and seed of (11283820414104960607,2916118026088292885):
Original:
-1
with exception:
System.Exception: Expected true, got false.

This test is failing. Why? Because we use the default generator for integers, which is not limited to non-negative numbers. Let’s break down what we can see here:

• Falsifiable, after 1 test - this means that our test failed after only one try

• (0 shrinks) - shrinking is an interesting, however advanced topic. It's a process of trying to find the smallest input data that will still make the test fail. For simple types it works pretty well out of the box: for numbers, it tries to find the case closest to zero, for lists, it will try to find the shortest list that still makes the test fail.

• (10979290545183390790,6994218916298689381) - this is the seed that was used to generate the data. This one's super useful if a test fails after 60 runs. Using the seed you can run the test again with the failing data right away, skipping the successful runs.

• Original: -1 - this is the input data for which the test failed

Simple type generator

Okay, so we have a failing test, but how to make it pass? We need to write a generator that will generate only non-negative numbers.

public static class SimpleTypesGenerators
{
    // this is a generator that will generate only non-negative numbers
    public static Arbitrary<int> OverrideIntArb() =>
        ArbMap.Default.GeneratorFor<int>().Where(x => x >= 0).ToArbitrary();
}

public class SimpleTypesTests
{
    // Another way of running PBT, also specifying the generator
    [Property(Arbitrary = new[] { typeof(SimpleTypesGenerators) })]
    public Property Generator_OverridingInt(int p)
    {
        return (p >= 0).ToProperty();
    }
}

Let’s run the test and see the output:

Ok, passed 100 tests.

As you can see, not only did the test pass, but it also ran 100 times. This is the default number of runs for FsCheck.

Now let’s try changing the condition in the test to p <= 50 and see what happens:

FsCheck.Xunit.PropertyFailedException
Falsifiable, after 61 tests (0 shrinks) (7331799476358367170,1508067056380619747)
Last step was invoked with size of 61 and seed of (6419148108620199268,14754055774596855233):
Original:
57

As you can see, the test failed after 61 runs, and the input data failing the test was 57.

Generators of custom types

Let’s try to create a generator for a custom record type:

public record PositiveInt(int Value);

public static class SimpleTypesGenerators
{
    // Generator, used to generate the values
    public static Gen<PositiveInt> GetPositiveInt() =>
        ArbMap.Default.GeneratorFor<int>().Where(x => x >= 0).Select(x => new PositiveInt(x));
}

This is the generator for the PositiveInt record. To be able to use it in tests, we need to create an Arbitrary from it:

public static class SimpleTypesGenerators
{
    public static Arbitrary<PositiveInt> PositiveIntArb() =>
        Arb.From(GetPositiveInt());}

Now we can use it in our tests:

[Fact]
public void AnotherWayOfRunningTests()
{
    var prop = Prop.ForAll<PositiveInt>(p => p.Value >= 0);
    prop.Check(Config.Default);
}

And the test is passing.

More complex types

Now let’s try to create a generator for order. First of all, here’s the order object:

public record Order(Guid Id, String Name, OrderStatus Status, int Quantity, decimal Price);

// enum with possible statuses
public enum OrderStatus
{
    New,
    InProgress,
    Done
}

// list of possible names for orders - names from outside of the list are illegal and should not be generated
private static readonly List<string> FruitNames = new List<string> { "Apple", "Banana", "Cherry", "Elderberry" };
private static readonly List<string> ToolNames = new List<string> { "Axe", "Hammer", "Screwdriver", "Wrench" };

Let’s try to quickly create a generator to generate orders:

public static class ComplexTypesGenerators
{
    // Choose one of the statuses - you can specify only a subset of the statuses you want to generate
    // if needed, you can split it into multiple generators, having different subsets of statuses
    public static Gen<OrderStatus> OrderStatusGenerator()
    {
        return Gen.Elements<OrderStatus>(OrderStatus.New, OrderStatus.InProgress, OrderStatus.Done);
    }

    // Generator returning a name from two lists, with weight attached.
    // This means that 1/4 of the time it will generate a name from the list of tools,
    // and 3/4 it will be the name from the list of fruits
    public static Gen<String> NameGenerator()
    {
        return Gen.Frequency(
            (3, Gen.Elements<String>(FruitNames)),
            (1, Gen.Elements<String>(ToolNames))
        );
    }

    // Generator for price - generates a number between 1 and 100, and then converts it to decimal
    public static Gen<decimal> PriceGenerator()
    {
        return Gen.Choose(1, 100).Select(x => x * 1.0m);
    }

    // I have found this old Linq syntax to be the most readable.
    // As an alternative, you can also use the syntax from the function below
    public static Gen<Order> OrderGenerator()
    {
        return 
            from name in NameGenerator()
            from qty in Gen.Choose(1, 10)
            from price in PriceGenerator()
            from status in OrderStatusGenerator()
            select new Order(Guid.NewGuid(), name, status, qty, qty * price);
    }

    // This is an alternative way of writing the same generator as above,
    // but chaining selects gets unreadable quickly
    public static Gen<Order> AnotherWayOfOrderGeneration()
    {
        return NameGenerator().SelectMany(name =>
        {
            return Gen.Choose(1, 10).Select(qty =>
                new Order(Guid.NewGuid(),
                    name,
                    OrderStatusGenerator().Sample(1, 1).Single(),
                    qty,
                    qty * PriceGenerator().Sample(1, 1).Single())
            );
        });

    public static Arbitrary<Order> OrderArb() =>
        Arb.From(OrderGenerator());

And now let’s write some tests using the above generator:

public class ComplexTypesTests
{
    private readonly ITestOutputHelper _testOutputHelper;

    public ComplexTypesTests(ITestOutputHelper testOutputHelper)
    {
        _testOutputHelper = testOutputHelper;
    }

    [Fact]
    public void SingleOrderTest()
    {
        var prop = Prop.ForAll<Order>(order =>
        {
            _testOutputHelper.WriteLine(order.ToString());
            Assert.True(order.Price >= 1);
        });
        prop.Check(Config.QuickThrowOnFailure
            // Specify the generator class for the test
            .WithArbitrary(new[] { typeof(ComplexTypesGenerators) }));
    }

    [Fact]
    public void OrdersListTest()
    {
        // Having a generator for a complex type, we have a generation of a list of such types for free.
        Prop.ForAll<List<Order>>(orders =>
            {
                _testOutputHelper.WriteLine(orders.Count.ToString());
                // Some asserts
            })
            .Check(Config.QuickThrowOnFailure
                .WithArbitrary(new[] { typeof(ComplexTypesGenerators) }));
    }
}

A few first outputs of the tests:

SingleOrderTest, writing the single object to the console:

Order { Id = ec0b5817-d124-4211-ace9-7a52253a6766, Name = Banana, Status = InProgress, Quantity = 10, Price = 280,0 }
Order { Id = be828369-ff32-4b5a-b6b1-c9dfc17650bb, Name = Cherry, Status = Done, Quantity = 1, Price = 11,0 }
Order { Id = 93d6c498-851d-4118-b727-a0f1f812004c, Name = Cherry, Status = New, Quantity = 9, Price = 81,0 }
Order { Id = 86df039e-3975-4200-b7c4-e68443218584, Name = Banana, Status = Done, Quantity = 1, Price = 54,0 }
Order { Id = b28c0be2-d9fb-4e9c-9b4e-0a0e7ed48c76, Name = Banana, Status = InProgress, Quantity = 10, Price = 60,0 }
Order { Id = de99993a-bc4c-4c51-826e-99e88f30beec, Name = Banana, Status = New, Quantity = 7, Price = 273,0 }
Order { Id = 17d2b4c9-87be-40bb-92f2-e6be6bcca2d5, Name = Axe, Status = New, Quantity = 1, Price = 18,0 }
Order { Id = 1b49b225-fa60-4424-8e93-923b63846d3e, Name = Apple, Status = InProgress, Quantity = 4, Price = 228,0 }
Order { Id = 87a52d4d-9ca8-448c-b643-76751a23c671, Name = Elderberry, Status = InProgress, Quantity = 5, Price = 50,0 }
Order { Id = 0581488e-499c-4cc5-8928-d5ffab089b79, Name = Banana, Status = Done, Quantity = 5, Price = 195,0 }
Order { Id = 923d5fce-3ea8-46d4-8bc9-b634b6f9f296, Name = Axe, Status = New, Quantity = 5, Price = 100,0 }
(...)

OrdersListTest, writing the count of the input list to the console:

2
3
1
1
5
2
6
0
5
4
11
11
5
9
14
4
19
0
5
4
2
21
19
24
(...)

As you can see, we have a lot of data generated, and the more data we need to generate, the more useful it becomes.

Using generators outside of property tests

It is possible to use generators in regular unit tests. It might be useful if you already have a generator and you don’t want to run it as a PBT (for any reason). However, I feel like there is no reason to do that, because why not just use PBT?

public class GeneratorInNonPbtTests
{
   [Fact]
    public void NonPbtUnitTest()
    {
        var positiveInt = SimpleTypesGenerators.GetPositiveInt().Sample(numberOfSamples: 1, size: 1).First();
        Assert.True(positiveInt.Value >= 0);
    }
}

Properties of tests

There are a few properties of the tests that you can use to control the behavior of your tests. I’ll focus on the two most interesting ones: WithArbitrary and WithReplay:

prop.Check(Config.QuickThrowOnFailure
    // Specify the generator class for the test
    .WithArbitrary(new[] { typeof(ComplexTypesGenerators) })
    .WithReplay("(6419148108620199268,14754055774596855233)"));

WithArbitrary is used to specify the generator class for the test.

WithReplay is used to replay the test with the same seed that was used to generate the failing data. You can find the seed in the output of the failing test.

Using attribute on the test method it looks similar:

[Property(Arbitrary = new[] { typeof(SimpleTypesGenerators) }, Replay = "(6419148108620199268,14754055774596855233)")]

Summary

I am not using property-based testing daily. For simple functions and simple inputs, it’s usually easier to write unit tests. But when I have complex functions in which I know that I might have a log of edge cases, here’s where PBT comes in handy.

The best part about PBT is that it’s repeatable randomness. You can always re-run the failed test with the same seed, getting the same input data. However, there are also some downsides. The biggest one I’ve found is that there is no documentation for FsCheck 3. For me, it was trial and error, and I hope that this article will help you to get started with FsCheck. If you have any troubles, I can recommend looking at the tests of the project itself and playing around with the code — I found it helpful.

Enjoy testing!

First step: Git and no code reviews

This is the first step. You have a Git repository, but you don’t know how to use it. You have a single branch, maybe collaborating with someone, but you’re not using any of the git features besides pull, push, and commit. This is a good starting point. This is the best way to start, but it’s nonsustainable long-term.

Starting small: develop and main branch

The next step is to start using branches. Let’s say you have some application being deployed manually to the production. And by this, I mean that you have to either copy files to the server or click some button in your IDE. You have no CICD pipeline, no automation, etc., but you’d like still to be able to make a hotfix to production while working on the new features. The flow then would be the following:

• you have a main branch, which is always in the state of the production

• you have a develop branch, which is the branch where you're working on the new features

• at every moment you can switch from the develop branch to the main branch, make a hotfix, and then switch back to develop without much disruption. You don't have to make sure that the develop works fine, there is no heavy testing required before deploying a hotfix

Slow but steady: develop, main, and feature branches

This is the approach that works pretty well if you have multiple contributors. The flow is similar to the previous one:

• you again have a main branch which is in the state of production

• you have a develop branch that should be working at all times, but it might happen that sometimes it's broken or untested

• you have feature branches, which you create to develop new features

• once you finish the feature, you merge it into the develop branch

• once you have enough changes, you merge the develop branch into the main branch, do some extensive testing, and then deploy main to the production

This approach is good because you push conflicts between the developers into the merge phase. If multiple people are working on the same branch, there is a high chance that they will have to constantly resolve conflicts. Especially if they’re working on similar areas of the code. The second advantage is that you can review and test features separately, before merging them into the develop branch. This way you need to spend less time on testing the whole application, and if one developer decides to break the build for a week because of a big change (which I don't think is a good approach, but it happens), the other developers can still work on their features without disruptions.

Trunk-based development

Trunk-based development is the feature branches approach but with a little twist. You have main, develop and feature branches, but the goal is to merge feature branches as soon as possible into the develop branch, and possibly also into the main branch. This way you avoid the long-living branches, which are hard to merge. Of course, you need to make sure that your changes will not affect the production, so usually feature toggles are used to ensure that the code is unreachable, but it's still in the codebase. This approach is good not only because of fewer conflicts but also because the code reviews are smaller, so there is a higher chance of getting the code reviewed right away, without the need to wait days for the review. Have you ever had a situation where you've waited for a week to get your code reviewed? If yes then your process is broken. This is a deeper topic, but one of the ways to try and fix it is to lower the friction on the code reviews. So instead of working for 3 weeks on a single feature, and having 100 files changed, you split your work into small batches of let's say 20 files which you produce over 3 days. This way you might get your code reviewed faster, as the code reviewer will get less anxiety from seeing how many files have changed.

Release branches

Release branches are useful in two situations:

• When you don’t have an automated CICD pipeline and you want to track when and which version has been deployed to production, or

• When you have a versioned product, like a desktop application with multiple major versions (version 1, 2, 3, etc.) with long-term support for each of them

In the first case, I’m sorry for you. In the second one, I’m also sorry. These two situations are hard to work with. In the first case, you need to track releases in case of rollback or hotfixes — it might be that you have merged some changes into main already but you found a bug both on main and production, and these are two different bugs. Then it’s easier to just checkout the release branch and fix the bug, without the need to worry about the changes which are already in the main branch. This can also be done with tags. It’s a bit more complicated (just slightly), but the general principle is the same.

In the second case imagine a scenario in which you have 3 versions of your application that you support — versions 4, 5, and 6. And in version 5 you’ve found a security vulnerability. You need to fix it in versions 5 and 6. Usually how it’s done is that you fix the bug in version 5 (start with the lowest version you have) and then depending on the changes and if it’s possible, you either cherry-pick the fix to subsequent versions, or you manually re-apply the changes. You repeat the process for each version having this bug.

My current way

I’m currently working in a company where all developers are working mostly on separate services. We have a lot of services and each service is developed by a single developer. The workflow I am using is the following:

• I have the main branch, which is the production

• I have feature branches that are long-living and are merged into the main branch once they're done

• Feature branches take a lot of time to develop — sometimes it happens that it’s even a week or two

• If I have a feature taking more than 2 weeks, I usually split the work into smaller feature branches, trying to deliver some value as soon as possible, slowly expanding the feature

Of course, this approach has its downsides. The biggest one is that code reviews are sometimes massive. It’s pretty usual for us to have 100 or even 300 files changed in a single code review. But on the other hand, the process is lightweight. We need to review these 300 files anyway, so why not batch them together and do one single round of review instead of 10 rounds of 30 files each? What helps a lot is the fact that there is most of the time a single person working on the service, so we don’t have to worry about conflicts.

Summary

The process of code reviews is very team- and codebase-dependent. There will be a different process for a team working on a monolith, and possibly a different one for a team working on microservices. What I am looking for while working on the improvements of the process is simplicity. The process should be as lightweight as possible. But sometimes it’s just not possible to have a simple process. But always try to make it just a bit simpler.

Bonus: Stories from the past

Welcome to the 90s: SVN and no code reviews

As I’m typing this, I feel old. However, I don’t consider myself old. I started my professional career in an old-style company, writing Delphi code stored in SVN. The approach of SVN is different than the one used by Git — instead of being distributed, it’s centralized. This means that two people can’t edit the same file at the same time. It was enforced by locks — everyone could lock the file at the start of the work, and then release the lock after doing the changes. As I think of it right now, I find it pretty funny that I needed to walk to my coworker’s desk and ask him when he was done with the file so that he’d release the lock on the file and I could edit it. I remember that when I was opening any file with an active lock on it, I was getting a message that the file was locked by someone else. I was able to see who and when locked it. The worst part was when someone was on vacation. Then we had to do the ancient way of merging changes — I could steal the lock, and then the person had to copy their changes to the separate file, download my changed file, and then manually apply their changes from the separate file. No one thought about any form of code review. It was just chaos.

Version Control From Hell: Dropbox as a version control system

I once had the pleasure of working with a self-taught programmer who did not need to collaborate with anyone. He used Dropbox as his version control system. As he implemented the feature, or after some time, he just zipped the whole project and sent it to the Dropbox account. And to avoid paying for the storage, he had dozens of accounts with free tiers. He had the whole system of managing this whole mess and I’m impressed that it worked for him — he knew what is happening and when he had any problems, he was able to restore the old version. But enough of the funny stories from the past, let’s move on to present times.

{
    "1": {
        "name": "Erik",
        "age": 30
    },
    "2": {
        "name": "Jane",
        "age": 25
    },
    "3": {
        "name": "Alex",
        "age": 35
    }
}

Since it was not an internal file but an external one over which I had no control, at first I got a bit frustrated and started to look for an opportunity to say "This file is not following the specification, fix it!". I mean, just look at this! Have you seen a JSON file with keys being numbers? How am I supposed to access this data? My first thought was that it was invalid JSON because the keys need to be a valid variable name. And I don't know any programming language that allows for a variable name to start with a number, not even mentioning that it's a single-digit number.

So I went to the Internet browser and started reading the JSON specification. This is what I've found:

Member Names

Implementation and profile defined member names used in a JSON:API document MUST be treated as case sensitive by clients and servers, and they MUST meet all of the following conditions:

    Member names MUST contain at least one character.
    Member names MUST contain only the allowed characters listed below.
    Member names MUST start and end with a “globally allowed character”, as defined below.

To enable an easy mapping of member names to URLs, it is RECOMMENDED that member names use only non-reserved, URL safe characters specified in RFC 3986.
Allowed Characters

The following “globally allowed characters” MAY be used anywhere in a member name:

    U+0061 to U+007A, “a-z”
    U+0041 to U+005A, “A-Z”
    U+0030 to U+0039, “0-9”
    U+0080 and above (non-ASCII Unicode characters; not recommended, not URL safe)

Additionally, the following characters are allowed in member names, except as the first or last character:

    U+002D HYPHEN-MINUS, “-“
    U+005F LOW LINE, “_”
    U+0020 SPACE, “ “ (not recommended, not URL safe)

This was followed by a long list of illegal characters. But let's slowly break it down, shall we?

Member names MUST contain at least one character Okay, so one character key is allowed. Fine, I've already seen variables named x or y in the code, so I can live with that.

Member names MUST contain only the allowed characters listed below Member names MUST start and end with a “globally allowed character”, as defined below. Easy, still no problem. They are a-z, A-Z, right?

U+0061 to U+007A, “a-z” U+0041 to U+005A, “A-Z” U+0030 to U+0039, “0-9”

Okay, now I'm confused. But there are characters not allowed in the first and last character, it cannot be that the name can start with a number, right?

Hell no! It just cannot be a minus, underscore, and space. But it can be a number. I was sitting shocked. How am I supposed to deserialize it into any object?

The realisation

I was sitting shocked. How am I supposed to deserialize it into any object? I've never seen a JSON file like this before. I started googling it and the solution is simple. It's a dictionary. It's a dictionary with keys being numbers. And it's a valid JSON file. It's a valid JSON file that I can deserialize into a dictionary.

So basically such JSON can be deserialized into a Dictionary<string, MyObject> where MyObject is a class with name and age in this case. My main mistake in the whole thing is that usually if there is a list of objects, it's an array, not a dictionary, and within this array, I'm used to having objects with keys being strings, not numbers.

I feel like this might be an obvious thing for front-end developers, but I'm not one of them. I'm a backend developer. Even though I now know the answer, I'm still shocked that "1" and "2" are valid keys in JSON.

It’s hard for me to think in terms of infrastructure. I’m bad at bash. I’m bad at networking. I’m bad at everything that is needed to make it work. At least this is what I was thinking for multiple years. And right now slowly I’ve started convincing myself that the infrastructure work is not only not that hard, but also can be pretty enjoyable.

I’ve always had problems with the infrastructure, hosting, and making something that can be used by me or someone else. Since I can remember I always wanted to become a backend developer. I was never attracted by frontend work. I did not like database work at first. I hated it. In my 2nd year of university, I even decided that no one would ever force me to use this awful technology which is databases. Reality hit me hard pretty quickly, but I still did not enjoy database work. I think it might have bad tooling — most DB tools are pure monstrosities taken right from the 90s.

Also, any frontend work was a problem for me — I just did not enjoy it. Nothing too personal, it was just boring.

At multiple points in my career, I was a generalist, a one-man army. I was able to just fill the shoes of any position that was required at that moment. If there was a need for frontend developer, I was a poor frontend developer. If there was a need to record and process some promotional video, conduct training, or write long documentation for a project, I was able to do it all, and usually, I was not that unsatisfied about it. I even at some point was responsible for publishing the backend application I’ve maintained into the internet. I’m calling it this strange way because it had nothing to do with infrastructure or hosting — I just created a web application in Azure and then clicked the ‘publish’ button in Visual Studio. And that was it.

Later on, I had an opportunity to work with hosting on virtual machines. It was horrible as I think of it right now — we had a virtual machine with IIS installed, and deployment was manual — we just generated a .zip package using pipeline and we had to grab this package, upload it by hand into a virtual machine, and then replace the folder which contained production application. At some point I got fluent in it, however, I still did not feel like it’s infrastructure work. It was some Windows Server 2012, and using a remote desktop I could click out most of the things. At that time I started a bit of powershell scripting to automate some processes. But still, there was some ‘mysterious end boss’ over there, which was cloud. Scary sounding names, like ‘EC2’, ‘S3’, ‘VPS’, ‘Proxy’, etc. And the entry barrier was a bit too high for me. I was afraid. As I think of it right now, the biggest problem for me back then was that I had no idea what I was doing. Once I’ve created an account on AWS, and started following some tutorials on how to deploy the ‘hello world’ application to the cloud. And the thing that stopped me from progressing at some point was just stupid — I managed to create something by mistake which started to cost me money. As I created the account, they promised that I wouldn’t have to pay and there would be no costs!

It even wasn’t about the money, because it was somewhere around 1$, it was about the possibility of failure — it was so easy to make a wrong step and be charged for things that I did not need. I just wanted to play around with things, and I ended up in a minefield where every step could cost me money, and also there was no good way of monitoring what was happening. I could not set up triggers like ‘if I exceed 10$/mo, just stop everything, delete all resources, and send me an email’.

Then I joined a company where we use cloud technologies, mostly Azure stack. I’ve learned a bunch of things in practice. I’ve learned to use docker. I’ve touched Kubernetes. I used things like queues, table storage, and S3. And this all from a user perspective — I did not have to worry about hosting all things. I did not have to worry about managing Kubernetes, pipelines, etc. This all was managed by other people. This gave me a good opportunity to slowly learn how to use it, and how to approach things. The thing I’m the most happy about is that I’ve learned Kubernetes. Many terms might sound scary at first: deployment, job, cronjob, service, ingress, etc. But since other people were managing it, and I also had colleagues willing to help me and explain to me how to use this whole thing, it did not sound that scary.

And then after some time, as I got comfortable with just using the cloud, I started to think that it would be nice to be able to jump a bit deeper into the infrastructure. I’ve started to think about self-hosting. I had (and still have) a mental barrier from buying a VPS because I’m afraid I will just get demotivated and stop using it. I think this fear is irrational, but still exists. I don’t feel like fully committing with my money and betting that I will not get bored in a week. I don’t like wasting money.

I have started small, the way I’m comfortable: I took my old laptop I was not using anymore, and installed Ubuntu on it. I’ve been always afraid of Linux and bash, but a few months prior I started using WSL and it helped me a lot — I got a bit more comfortable with bash, it gave me a slow and safe intro to dotfiles, shell configuration and overall managing of Linux. So I managed to take a first step — I installed Minikube on my Ubuntu, and I was able to access some sample application from another computer inside of my local network. And that’s it. Over there I’ve stopped again. The reasons were multiple: I had no meaningful application to host, and I had problems with high ping locally (I have a ping of 50–300ms between two laptops in the same room, I suspect that it’s caused by the radio antenna which is just above my apartment), and I was again afraid. Afraid that if I would expose my laptop to the internet since I have no idea what I’m doing, I will get hacked in a matter of minutes. So I had a working cluster, and this was it when it came to the infrastructure for a couple of months.

A couple of days ago I came up with a brilliant idea. My fiance goes to the gym with her friend, and they attend some organized classes together. The problem is that the registration for those classes opens sometime 2 days before the classes (the exact time of opening is not disclosed) and more people want to attend than there are free spots. So my fiance and her friend had to go into the website and refresh it from time to time during the day if they wanted to sign up. And then I realized that I have all that I need — I have skills to write a crawler, I have a Kubernetes cluster which I don’t even need to expose to the internet, and then I’ll be able to solve an actual real-life problem, and also have a good excuse to make another take on the infrastructure.

So I took my old laptop, which still had Minikube installed, turned it on, and checked if everything was still working. And it was! So my first step was to create a PostgreSQL database, hosted on my local Kubernetes. It was not required for the task (just a little nice to have for the near future), but I’ve decided to use this small spark of motivation to make it happen. The crawler part and hosting it was the easy part, relative to what I thought was waiting for me with this database. And I was partially right — I had some problems. I had to go manually onto the machine with Kubernetes and clean up some corrupted files after failed attempts. But I’m happy with that because not only I’ve managed to do it, but also I’ve learned a few things. This way not only do I get more comfortable with working with Linux, but also I get more confidence that I am capable of fixing things by myself.

Right now the status of everything is:

• I have a Kubernetes cluster running on my old laptop — I have a PostgreSQL database running on this cluster — I have a .NET crawler deployed to the cluster, which sends emails once there is a free spot on the class

What I am working on right now, and the plan for the future:

• I want to expose the cluster to the internet, using some kind of proxy / tunnel / reverse proxy (preferably a free one, I’m exploring options using Cloudflare right now) — I want the crawler to use the database to store and read some data — I want to make some frontend for the crawler which will be exposed publicly to the internet

So far I’d say that I’m not only happy with my progress, but also proud of myself that I am capable of working with infrastructure and it’s not as hard as I thought it was. I’m looking forward to the next steps, and I’m excited to see where this journey will take me.

Here you can find the whole project on GitHub

Business logic & application

Imagine you have a pretty simple cronjob to create. It’s a small console application which grabs the data from one table in the database, transforms it, and then inserts the data back into the database. In our case, it will be logs. We have a table into which raw logs are being collected. You can think of it as of some automatically gathered logs of all the events happening in the production cluster, which needs to be filtered and cleaned up in order to store & browse them long term, to save the space.

create table raw_logs
(
    id uuid primary key,
    message text not null,
    severity int not null,
    source text not null,
    timestamp timestamp not null,
    additional_field1 text,
    additional_field2 text
);

Most of the fields from the raw_logs table are not interesting for us, we only care about message and severity. Message is just a message associated with log, and severity is enum stored as int values, being:

Info = 1,
Warning = 2,
Error = 3,

We have a task to filter and transform the data, so that another application can process it without knowing the exact details of the raw logs. The rules are as following:

• if log is info, we don’t care about it

• if log is warning, we only care about the amount of such logs

• if log is error, we care about its message

And we want to save the result into another PostgreSQL table. So let’s get into implementation of the C# application. Foremost, we need a way to read raw logs from the DB. We need a DB DTO:

public class RawLog
{
    public readonly Guid Id;
    public readonly string Message;
    public readonly Severity Severity;
}

private  RawLog(Guid id, string message, Severity severity)
{
    Id = id;
    Message = message;
    Severity = severity;
}

And a function reading the logs in DatabaseReader class:

public async Task<List<RawLogDto>> ReadRawData()
{
    string sql = $"select * from raw_logs";
    var query = await _connection.QueryAsync<RawLogDto>(sql);
    return query.ToList();
}

So now we have the raw logs, now we want to filter out only the ones we care about — warnings and errors. We don’t want to operate on DB Dtos so in this step we also map DB Dtos into another objects. We do this in the Worker class, which will be the main point of our logic.

private static IEnumerable<ITransformResult> Transform(List<RawLogDto> data)
    {
        var logs = data
            .Select(RawLog.FromDto)
            .Select(x =>
            {
                ITransformResult result = x.Severity switch
                {
                    Severity.Unknown => throw new InvalidOperationException("Unknown log severity"),
                    Severity.Info => new DropTransformResult(),
                    Severity.Warning => new WarningTransformResult(),
                    Severity.Error => new ErrorTransformResult(x.Id, x.Message),
                    _ => throw new ArgumentOutOfRangeException()
                };
                return result;
            });
        return logs;
    }

Not only that, but we also need to add mapping function from DB Dto to business logic object:

public static RawLog FromDto(RawLogDto dto)
{
    return new RawLog(dto.Id, dto.Message, (Severity) dto.Severity);
}

Now as we have transformed logs, we want to convert them into the final DB-writable DTOs in a final form, so: if the log is error we want each individual messages, if it’s warning we only want count.

private static IEnumerable<LogWriteDbDto> ToDbDto(IEnumerable<ITransformResult> data)
{
    foreach (var d in data)
    {
        switch (d)
        {
            case ErrorTransformResult e:
                yield return new LogWriteDbDto(Severity.Error, e.Message);
                break;
            case WarningTransformResult:
                yield return new LogWriteDbDto(Severity.Warning, null);
                break;
            case DropTransformResult:
                break;
            default:
                throw new InvalidOperationException("Unknown worker result");
        }
    }
}

And we also need to have the LogWriteDbDto:

public class LogWriteDbDto(Severity Severity, string? Message)
{
    public Guid Id { get; init; } = Guid.NewGuid();
    public Severity Severity { get; init; } = Severity;
    public string? Message { get; init; } = Message;
}

Finally, as we have the logs transformed and processed, we need to have a DatabaseWriter:

public async Task WriteTransformedLogs(List<LogWriteDbDto> dto)
{
    var sql = "INSERT INTO transformed_logs (id, severity, message) VALUES (@id, @severity, @message)";
    await _connection.ExecuteAsync(sql, dto);
}

With the writer, here comes the SQL table:

create table transformed_logs
(
    id uuid primary key,
    message text,
    severity int not null
);

And now as we have all the pieces, we can patch them up all together:

public async Task DoAsync()
{
    var data = await _databaseReader.ReadRawData();
    var transformed = Transform(data);
    var dbDtos = ToDbDto(transformed);
    await _databaseWriter.WriteTransformedLogs(dbDtos.ToList());
}

And then in Program.cs we can have all setup, configuration, HostBuilder, etc., and ultimately we have one entry point: Worker.DoAsync(), which is responsible for all the business logic.

In my case, the whole Program.cs looks like this:

// Connection string of local databse, hosted on inside Docker
const string connectionString = "Host=localhost:5432;Username=postgres;Password=admin;Database=postgres";
await using var databaseReader = new DatabaseReader(connectionString);
await using var databaseWriter = new DatabaseWriter(connectionString);
var worker = new Worker(databaseReader, databaseWriter);
await worker.DoAsync();

Console.WriteLine("Goodbye");

Testing

Now as we have the application working, we can get into the testing part. Although testing the whole application would be nice, I think it would be way more complicated than just testing the Worker.DoAsync() method, using the actual PostgreSQL database.

First, we need to be able to spin up the PostgreSQL container. For this I’m using the TestContainers and TestContainers.PostgreSql libraries:

public PostgreSqlContainer Get()
{
    return new PostgreSqlBuilder()
        .WithImage("postgres:16")
        .WithCleanUp(true)
        .Build();
}

As we have the container, we can already use it in our integration test:

[Fact]
public async Task LogsTransformationTest()
{
    // Arrange
    var container = new TestContainerFactory().Get();
    await container.StartAsync();
}

As a prerequisite to these tests, we need to have a working DB with the schema identical to the production one. We can achieve this by storing the whole schema in a schema.sql file, for the ease of use I've put it into test's project, however it's possible to add it as a solution folder so that it's accessible from multiple projects. In our case, schema.sql is pretty short as it just creates two tables:

create table raw_logs
( .. );

create table transformed_logs
( .. );

And we also need to have some test data in the raw_logs table. I have a second file, called data.sql:

insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('c154b819-3e7b-4fa8-bf8b-5de0b7e35d3f', 'Info log 1', 1, 'Source 1', '2024-03-08 12:00:00', 'Value 1', 'Value 2');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('f6a4dc05-2b70-44b6-90cb-238bb6f19a93', 'Warning log 1', 2, 'Source 2', '2024-03-08 12:10:00', 'Value 3', 'Value 4');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('ecb27850-1a6c-4b2d-aa30-63f94ff541f8', 'Error log 1', 3, 'Source 3', '2024-03-08 12:20:00', 'Value 5', 'Value 6');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('bae4b6b2-8e46-4224-967a-40528ed06a68', 'Info log 2', 1, 'Source 4', '2024-03-08 12:30:00', 'Value 7', 'Value 8');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('a8cfdb45-d6c1-4814-a65c-58d942f52a69', 'Warning log 2', 2, 'Source 5', '2024-03-08 12:40:00', 'Value 9', 'Value 10');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('c3bf6d44-515d-4945-b3b7-810620fb3e82', 'Error log 2', 3, 'Source 6', '2024-03-08 12:50:00', 'Value 11', 'Value 12');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('cfd5dab5-1342-47b8-8c96-04ec5e5795ec', 'Info log 3', 1, 'Source 7', '2024-03-08 13:00:00', 'Value 13', 'Value 14');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('c68b5e19-1097-4ec3-967e-67190d22c3f5', 'Warning log 3', 2, 'Source 8', '2024-03-08 13:10:00', 'Value 15', 'Value 16');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('eddb52b0-70d7-45b3-9fb4-c40ddbf5ec9b', 'Error log 3', 3, 'Source 9', '2024-03-08 13:20:00', 'Value 17', 'Value 18');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('1b9e8608-fa0b-4b4c-98cf-96b84c4bf8a6', 'Info log 4', 1, 'Source 10', '2024-03-08 13:30:00', 'Value 19', 'Value 20');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('b01d1e47-dcb4-472f-b6f4-7b20b14e2b70', 'Warning log 4', 2, 'Source 11', '2024-03-08 13:40:00', 'Value 21', 'Value 22');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('e4c5245b-8631-4f1e-93ee-42b2e87eab07', 'Error log 4', 3, 'Source 12', '2024-03-08 13:50:00', 'Value 23', 'Value 24');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('3dfe7d9d-dcaa-438f-9a2d-8e10962886b0', 'Info log 5', 1, 'Source 13', '2024-03-08 14:00:00', 'Value 25', 'Value 26');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('293fb847-9cfb-4a4c-b22d-6f2a1b319b16', 'Warning log 5', 2, 'Source 14', '2024-03-08 14:10:00', 'Value 27', 'Value 28');
insert into raw_logs (id, message, severity, source, timestamp, additional_field1, additional_field2) values ('f2430cf6-1c6e-4343-9b73-b3b594b62814', 'Error log 5', 3, 'Source 15', '2024-03-08 14:20:00', 'Value 29', 'Value 30');

Now we need to read .sql files and apply them onto container:

public async Task ExecuteScriptFromFile(string connectionString, string path)
{
    var fileText = await FileReader.ReadFile(path);
    await using var connection = new NpgsqlConnection(connectionString);
    await connection.OpenAsync();
    await connection.ExecuteAsync(fileText);
}

[Fact]
public async Task LogsTransformationTest()
{
    ...
    var testDbWriter = new TestDbWriter();
    string schemaPath = @"./DatabaseScripts/schema.sql";
    string dataPath = @"./DatabaseScripts/data.sql";
    await testDbWriter.ExecuteScriptFromFile(container.GetConnectionString(), schemaPath);
    await testDbWriter.ExecuteScriptFromFile(container.GetConnectionString(), dataPath);
}

Now we can add worker factory and call it from test:

public static class WorkerFactory
{
    public static Worker Get(string connectionString)
    {
        var databaseReader = new DatabaseReader(connectionString);
        var databaseWriter = new DatabaseWriter(connectionString);
        return new Worker(databaseReader, databaseWriter);
    }
}

[Fact]
public async Task LogsTransformationTest()
{
    ...
    var worker = WorkerFactory.Get(container.GetConnectionString());
    // Act
    await worker.DoAsync();
}

Now we have all the work done, and the results should be waiting for us in db. So we need test db reader for the assertions:

public class TestDbReader
{
    public async Task<List<TransformedLogs>> ReadAllLogs(string connectionString)
    {
        await using var connection = new NpgsqlConnection(connectionString);
        await connection.OpenAsync();
        var logs = await connection.QueryAsync<TransformedLogs>("SELECT * FROM transformed_logs");
        return logs.ToList();
    }
}

public class TransformedLogs
{
    public Guid Id { get; init; }
    public string? Message { get; init; }
    public Severity Severity { get; init; }
}

Going back to the test, we can use newly created DbReader to assert that worker did its work as expected:

[Fact]
public async Task LogsTransformationTest()
{
    ...
    // Assert
    var logs = await new TestDbReader().ReadAllLogs(container.GetConnectionString());
    logs.Should().HaveCount(10);
    logs.Where(x => x.Severity is Severity.Warning).Should().HaveCount(5);
    logs.Where(x => x.Severity is Severity.Error).Should().HaveCount(5);
    logs.Where(x => x.Severity is Severity.Error).Select(x => x.Message).All(x => x is null).Should().BeFalse();

I’m here using FluentAssertions library to improve the asserts - instead of

Assert.Equals(5, logs.Where(x => x.Severity is Severity.Warning));

I can write

logs.Where(x => x.Severity is Severity.Warning).Should().HaveCount(5);

And now, as a final touch, we should clean up the test container:

[Fact]
public async Task LogsTransformationTest()
{
    ...
    // Cleanup
    await container.DisposeAsync(); 
}

So the whole test looks like this:

[Fact]
public async Task LogsTransformationTest()
{
    // Arrange
    var container = new TestContainerFactory().Get();
    await container.StartAsync();
    var worker = WorkerFactory.Get(container.GetConnectionString());

    var testDbWriter = new TestDbWriter();
    string schemaPath = @"./DatabaseScripts/schema.sql";
    string dataPath = @"./DatabaseScripts/data.sql";
    await testDbWriter.ExecuteScriptFromFile(container.GetConnectionString(), schemaPath);
    await testDbWriter.ExecuteScriptFromFile(container.GetConnectionString(), dataPath);

    // Act
    await worker.DoAsync();

    // Assert
    var logs = await new TestDbReader().ReadAllLogs(container.GetConnectionString());
    logs.Should().HaveCount(10);
    logs.Where(x => x.Severity is Severity.Warning).Should().HaveCount(5);
    logs.Where(x => x.Severity is Severity.Error).Should().HaveCount(5);
    logs.Where(x => x.Severity is Severity.Error).Select(x => x.Message).All(x => x is null).Should().BeFalse();

    // Cleanup
    await container.DisposeAsync();
}

Conclusion

I think this way of testing provides a nice balance between the complexity of test and level of confidence provided. Running this test takes only a couple seconds and the best thing about it is that it tests the whole depth of the business part of the application — it tests not only business rules, but also db queries, SQL writes etc. It’s possible to test if your DB implementations do not struggle with multiple instances — you can easily create 10 workers and execute them in parallel to test if there will be no errors while having 10 or even 100 instances trying to run at the same time. It’s invaluable to be able to test how multiple instances behave locally, and to do it each time you run your test suite. Having integration tests done this way, you don’t need to have additional infrastructure to run db tests in your pipelines, which provides better scaling of your tests. Also, by creating a new DB each time, you are avoiding any problems with parallel test runs on the same db, resulting in a broken state — tests are more consistent and separated. This technique has served me well, and I hope you will also find it useful.