My infrastructure journey — part 1
A couple of months ago I decided to give it a try and get into self-hosted. The problem is that I’m not the best in terms of this — it’s…
Are you bad at infrastructure? I also used to be. A couple of months ago I decided to give it a try and get into self-hosted and oh boy, it’s been an amazing and rewarding journey! Let me share my story with you.
It’s hard for me to think in terms of infrastructure. I’m bad at bash. I’m bad at networking. I’m bad at everything that is needed to make it work. At least this is what I was thinking for multiple years. And right now slowly I’ve started convincing myself that the infrastructure work is not only not that hard, but also can be pretty enjoyable.
I’ve always had problems with the infrastructure, hosting, and making something that can be used by me or someone else. Since I can remember I always wanted to become a backend developer. I was never attracted by frontend work. I did not like database work at first. I hated it. In my 2nd year of university, I even decided that no one would ever force me to use this awful technology which is databases. Reality hit me hard pretty quickly, but I still did not enjoy database work. I think it might have bad tooling — most DB tools are pure monstrosities taken right from the 90s.
Also, any frontend work was a problem for me — I just did not enjoy it. Nothing too personal, it was just boring.
At multiple points in my career, I was a generalist, a one-man army. I was able to just fill the shoes of any position that was required at that moment. If there was a need for frontend developer, I was a poor frontend developer. If there was a need to record and process some promotional video, conduct training, or write long documentation for a project, I was able to do it all, and usually, I was not that unsatisfied about it. I even at some point was responsible for publishing the backend application I’ve maintained into the internet. I’m calling it this strange way because it had nothing to do with infrastructure or hosting — I just created a web application in Azure and then clicked the ‘publish’ button in Visual Studio. And that was it.
Later on, I had an opportunity to work with hosting on virtual machines. It was horrible as I think of it right now — we had a virtual machine with IIS installed, and deployment was manual — we just generated a .zip package using pipeline and we had to grab this package, upload it by hand into a virtual machine, and then replace the folder which contained production application. At some point I got fluent in it, however, I still did not feel like it’s infrastructure work. It was some Windows Server 2012, and using a remote desktop I could click out most of the things. At that time I started a bit of powershell scripting to automate some processes. But still, there was some ‘mysterious end boss’ over there, which was cloud. Scary sounding names, like ‘EC2’, ‘S3’, ‘VPS’, ‘Proxy’, etc. And the entry barrier was a bit too high for me. I was afraid. As I think of it right now, the biggest problem for me back then was that I had no idea what I was doing. Once I’ve created an account on AWS, and started following some tutorials on how to deploy the ‘hello world’ application to the cloud. And the thing that stopped me from progressing at some point was just stupid — I managed to create something by mistake which started to cost me money. As I created the account, they promised that I wouldn’t have to pay and there would be no costs!
It even wasn’t about the money, because it was somewhere around 1$, it was about the possibility of failure — it was so easy to make a wrong step and be charged for things that I did not need. I just wanted to play around with things, and I ended up in a minefield where every step could cost me money, and also there was no good way of monitoring what was happening. I could not set up triggers like ‘if I exceed 10$/mo, just stop everything, delete all resources, and send me an email’.
Then I joined a company where we use cloud technologies, mostly Azure stack. I’ve learned a bunch of things in practice. I’ve learned to use docker. I’ve touched Kubernetes. I used things like queues, table storage, and S3. And this all from a user perspective — I did not have to worry about hosting all things. I did not have to worry about managing Kubernetes, pipelines, etc. This all was managed by other people. This gave me a good opportunity to slowly learn how to use it, and how to approach things. The thing I’m the most happy about is that I’ve learned Kubernetes. Many terms might sound scary at first: deployment, job, cronjob, service, ingress, etc. But since other people were managing it, and I also had colleagues willing to help me and explain to me how to use this whole thing, it did not sound that scary.
And then after some time, as I got comfortable with just using the cloud, I started to think that it would be nice to be able to jump a bit deeper into the infrastructure. I’ve started to think about self-hosting. I had (and still have) a mental barrier from buying a VPS because I’m afraid I will just get demotivated and stop using it. I think this fear is irrational, but still exists. I don’t feel like fully committing with my money and betting that I will not get bored in a week. I don’t like wasting money.
I have started small, the way I’m comfortable: I took my old laptop I was not using anymore, and installed Ubuntu on it. I’ve been always afraid of Linux and bash, but a few months prior I started using WSL and it helped me a lot — I got a bit more comfortable with bash, it gave me a slow and safe intro to dotfiles, shell configuration and overall managing of Linux. So I managed to take a first step — I installed Minikube on my Ubuntu, and I was able to access some sample application from another computer inside of my local network. And that’s it. Over there I’ve stopped again. The reasons were multiple: I had no meaningful application to host, and I had problems with high ping locally (I have a ping of 50–300ms between two laptops in the same room, I suspect that it’s caused by the radio antenna which is just above my apartment), and I was again afraid. Afraid that if I would expose my laptop to the internet since I have no idea what I’m doing, I will get hacked in a matter of minutes. So I had a working cluster, and this was it when it came to the infrastructure for a couple of months.
A couple of days ago I came up with a brilliant idea. My fiance goes to the gym with her friend, and they attend some organized classes together. The problem is that the registration for those classes opens sometime 2 days before the classes (the exact time of opening is not disclosed) and more people want to attend than there are free spots. So my fiance and her friend had to go into the website and refresh it from time to time during the day if they wanted to sign up. And then I realized that I have all that I need — I have skills to write a crawler, I have a Kubernetes cluster which I don’t even need to expose to the internet, and then I’ll be able to solve an actual real-life problem, and also have a good excuse to make another take on the infrastructure.
So I took my old laptop, which still had Minikube installed, turned it on, and checked if everything was still working. And it was! So my first step was to create a PostgreSQL database, hosted on my local Kubernetes. It was not required for the task (just a little nice to have for the near future), but I’ve decided to use this small spark of motivation to make it happen. The crawler part and hosting it was the easy part, relative to what I thought was waiting for me with this database. And I was partially right — I had some problems. I had to go manually onto the machine with Kubernetes and clean up some corrupted files after failed attempts. But I’m happy with that because not only I’ve managed to do it, but also I’ve learned a few things. This way not only do I get more comfortable with working with Linux, but also I get more confidence that I am capable of fixing things by myself.
Right now the status of everything is:
I have a Kubernetes cluster running on my old laptop — I have a PostgreSQL database running on this cluster — I have a .NET crawler deployed to the cluster, which sends emails once there is a free spot on the class
What I am working on right now, and the plan for the future:
I want to expose the cluster to the internet, using some kind of proxy / tunnel / reverse proxy (preferably a free one, I’m exploring options using Cloudflare right now) — I want the crawler to use the database to store and read some data — I want to make some frontend for the crawler which will be exposed publicly to the internet
So far I’d say that I’m not only happy with my progress, but also proud of myself that I am capable of working with infrastructure and it’s not as hard as I thought it was. I’m looking forward to the next steps, and I’m excited to see where this journey will take me.