pabis.eu

I learned about Portsentry from a book called "Practical Linux System Administration" by Kenneth Hess. It is a very simple yet useful tool to detect port scans or malicious bots trying to gather information about our publicly exposed instances (or even private ones given they got infected). Portsentry simply listens...

If you follow my blog, you probably know that I am huge fan of YubiKeys. I tinker with them a lot. One of the most annoying things about multi-factor authentication is when you have to get up your butt and go get your phone to open Google Authenticator and get...

Today, we are going to explore a way to perform a blue/green deployment of an RDS database (MySQL and PostgreSQL) when all the infrastructure is created from CloudFormation and Terraform. You might ask yourself, why even bother to do a blue/green deployment of a database? The answer is very, very...

Someone decided to use CloudFormation as the IaC tool of choice, once in the past. Why? Nobody knows why 🤷. It's cool that it managed the state for me and I don't need to worry about someone stealing it or managing permissions or doing backups. But we want to be...

In the previous post, we have created a Lambda Authorizer for API Gateway that used default factory YubiKey OTP and YubiCloud for validation. Today, we will use the second slot of the YubiKey OTP (long-press) to create a custom AES key and IDs. We will use official YubiKey CLI tool...

Do you have a YubiKey? It has a button (or a touch sensor), right? So, plug it in, open a text editor, focus on the text field and press your YubiKey. Then press again. You should see some random lowercase characters. Weird, right? What is that string after all? Let's...

Building SOCI containers in a CI/CD pipeline when your control over the environment is restricted was pretty challenging. I went through many trials and errors and eventually came up with a solution that looked very difficult at first. I will first give you a brief story of all the struggles,...

Germany's AWS Summit usually took place in Berlin around May and June. But this year, it was held in Hamburg 🛳️ for the first time. The event was a great experience as always, and with each year, I feel like I benefit from it more and more efficiently. Apparently, in...

In Azure there's no Systems Manager, that's pretty obvious when you know that it's an AWS product. However, as both clouds are very large providers, Azure should offer something similar, right? As our focus is Session Manager, there's no direct equivalent that works in the same way. However, Microsoft offers...

As the third approach to creating a new NAT instance on AWS, I want to show you how to cleanly and simply create a NAT instance that will use the latest Amazon Linux 2023 AMI. This can help you save a lot of money compared to using NAT Gateways if...

Image Builder is a service that allows you to build AMIs and Docker images step-by-step using a recipe you define. This is an equivalent service to HashiCorp's Packer but completely AWS managed that you can use from your browser (AWS Console). However, as Infrastructure as Code is a best practice,...

LDAP is a pretty known and widely used protocol in the world of IT, although it is not as comfortable to use as OAuth or OpenID Connect. Nevertheless, you can still often encounter enterprises that need to use it, therefore it is worth to gain some experience with it. In...

S3 native feature of presigned URL is great - you can save on compute costs as everything will be handled by AWS for you. However, there's not much you can set when generating the URL, basically just `Content-Type` and `Content-Disposition`. But to set tags on the object, you need another...

We use HTTP headers all the time. In requests we do, we often send cookies, authorization headers with bearer tokens, content length, we get responses with new cookies, expiration dates for caching. We don't think too much about this, even when building things like a REST application. However, it is...

We all know that we should store our configuration in Git. But I have seen very often, that the config is kept together with the code. Why is it a problem? If you want to publish a new configuration, you have to rebuild and restart the whole app (unless you...

LDAP ist eine ganz normale Sache in der IT-Welt, obwohl es nicht so bequem ist wie OAuth oder OpenID Connect. Trotzdem wird es immer noch weitgehend genutzt. Darum lohnt es sich, etwas Erfahrung mit LDAP zu sammeln. In diesem Artikel zeige ich euch, wie man eine Webseite mit Active-Directory- oder...

Waiting for your deployment to finish. What a painful experience. You have to wait for build. You have to wait for tests. And then you have to wait for the startup. 10 minutes later, the new task definition is up and running. Then you realize that you made a mistake...

Today, I will be guiding you through implementation of a script that creates a CloudFormation templated based on given instructions to a large language model. With this, I will also demonstrate how to connect ell library to AWS Bedrock. Moreover, we will provide the agent with tools to fetch most...

Ell is a quite new Python library that is similar to LangChain. In my opinion it feels simpler to use but is not so featureful. It is mostly useful as a higher level API on top of LLM providers (such as OpenAI or Anthropic) and creation of agentic tools -...

Often times in order to connect to AWS services from our local machines or external systems, we resolve to using IAM user credentials. However, as we know, these are easy to be compromised when leaked. Unless you have very strict IAM policies that for example check the source IP, leaked...

In the last post, we constructed all the necessary resources to create a Bedrock Knowledge Base utilizing OpenSearch Serverless. These were the CloudFormation templates, Lambda function for index creation and other static things like IAM Roles and S3 Buckets. Today, we will go through the process of combining all these...

One thing that grinds my gears about AWS OpenSearch Serverless is that it is "serverless". That means that despite the promise of serverless services, such as Lambda or Fargate, where you pay only for what you use, OpenSearch uses OCUs (capacity units) constantly. It's not a large amount but it...

This is an update to my recent post about creating daily Git diffs between today's and previous day's `HEAD` commits. However, in the previous post, I relied on CodeCommit that is unfortunately being slowly shut down. What's more, a lot of users prefer external Git providers like GitHub, GitLab, or...

One thing I need for a future project is a way to collect all the changes in a Git repository (on a specific branch) and store them in some place such as S3 bucket. I thought about it extensively and came with such a solution you can find on the...

Creating tasks (to-dos) in Habitica is not the most effective and pleasant thing using the UI (although tools like Trello or Jira are far from better). Compared to editing a simple and single text file, whether formatted as Markdown or not, is much easier and gives a better overview (at...

Habitica is an excellent tool for keeping yourself sticking to what you planned for yourself. Although it doesn't work for everyone, it is nevertheless worth trying. (This is my third try in using Habitica and it seems to be working with the longest streak of 949 days.) As discussed in...

I use Habitica daily. It is a very strong motivator to keep me on track with my daily tasks. However, I already collected all the free pets and mounts and so I don't need most of the items I collect. Selling them is a pain as you have to sell...

As you see on my "resume" page, I uploaded my public key for encrypting emails back when I was starting this website. However, moving private key from one place to another is never a good option. My YubiKey cost me 55€ and I barely use it for FIDO/U2F. Why not...

World Backup Day is long behind us but it's never too late. I was thinking about methods to backup my data at home. I have some USB disks, old computers that still contain some data - maybe valuable, maybe not. Usual cloud solutions might be of decent price -...

One of the best practices for running and developing a service is to separate each environment into its own account. AWS Organizations enables this to be done easily. However, we often see examples where environment names are subdomains of one of our product's domain, such as `dev.example.com` or `staging.example.com`. Let's...

Last Friday I joined the live stream of AWS Power Hour: Architecting Professional Season 2. As in every episode, we went through some example questions. I decided to take one of them and implement a working solution. The question I picked states: there's a company that uses CloudFront to host...

In one previous posts I wrote how to create a NAT instance on AWS to save on NAT Gateway costs. However, there were two issues with that solution: it worked only for one subnet, it didn't work on Amazon Linux 2023 and it didn't save the state permanently (reboot makes...

I came to a conclusion that a thing that is not written down and explained is not learned. So as I am preparing for AWS MLS-C01, I need to refresh some of my small machine learning knowledge and expand it further. I decided to go with an exercise where I...

In this post I would like to explore the possibilities of Athena Workgroups and Lake Formation to control costs and access to data in S3. We will implement a limit on the amount of data scanned by Athena and hide some columns and rows from the user via Lake Formation...

In January 2023 I decided to start writing a technological blog, mostly about field of my profession which is DevOps, Cloud and Infrastructure. I had this idea in mind since some time but was hesitant to do so. My greatest fear was writing something stupid. I decided to ask colleagues...

Amazon Aurora has a great feature that can save you hours of recovery from a backup in case you need to quickly restore a database. Backtracking let's you do point-in-time recovery into the same cluster. The only downside is the need to reconnect. However, what if we rewind our database...

Multi-factor authentication is a must-have for any serious developer and administrator. It's Security 101: you should always use second factor if offered by the service. This is true also for AWS but not only in the Management Console. AWS STS offers time-limited tokens that can be configured (with the power...

We all have been there. Well, not exactly, some of us planned ahead to keep the options open. But we all feared Oracle and wanted to jump to MariaDB. And now we learned about all the benefits of Aurora and we are still on this RDS, the button for "Create...

Ok, Hashi, that's it. I'm giving up Terraform. And even OpenTofu doesn't taste now. Amazon convinced me. CloudFormation is the leader of Infrastructure as Code. But today, I'll break even more rules. CloudWatch Metrics? Nah, let's put our CPU and Memory usage into Timestream. Because why not? Having a graph...

MongoDB, a NoSQL database used since a long time as the primary database of many web applications. AWS offers a managed solution for MongoDB called DocumentDB. It is highly available, easy to scale read capacity horizontally and the storage can scale automatically. However, even with those features, we are still...

As the preparation for SCS-C02 exam, some questions came up for me regarding S3 access logging. Are anonymous requests logged? Are website requests logged? Are CloudFront requests logged? I wanted to test it out to see whether CloudTrail Data trail and Server Access Logging collect all those events or are...

In one of the [previous posts](https://pabis.eu/blog/2023-09-11-Slim-SSM-Parameter-Store-Go.html), we were implementing a small application in Go that was helpful to download configuration files from SSM Parameter Store. However, SSM is limited to 4KB or 8KB with Advanced tier. To solve this problem, today we will implement a similar application that will download...

I recently started learning for AWS Security Specialty with Stephane Maarek's course and was surprised that such basic problem as losing SSH access was not needed for SysOps Administrator. However, it was exciting to see all the possible methods that are available to resolve this issue. Today, I will try...

AWS Image Builder is a service that allows you to create AMIs using a recipe - a step-by-step pipeline that applies different "components" to the image and performs tests at the end. Pipelines that perform recipes can be scheduled to run on different dates and the resulting AMIs can be...

We often use Alpine Linux for our containers. It's small, it has small footprint on memory so it is perfect for containers. However, if we want to interface with AWS services, we either have to: use SDK, use HTTP API or use AWS CLI. AWS CLI weighs a bit (200MB),...

Additional block volumes in AWS EC2 are not deleted by default (although root ones can also be marked to be retained). It is a good practice to keep data that we want to keep on a separate volume, such as MySQL database files. However, when creating things with IaC solutions...

Previously we have created a Python script that pulls data from Google Play Developer Console. The repository contained a main function that used Prometheus client to expose the metrics under `/metrics` endpoint. Today, I will explain it and we will further develop the project by putting it inside a container...

Feel the excitement. Just connect the API, place a TV or a monitor in a visible place at your home or office and stare at the graphs. Is my update liked? Are people reaching my product? What are the sentiments about the latest app I released? Grafana is a...

When preparing for AWS SysOps Associate Certification, I wondered how can I learn more about some of the components I never used. I decided to use Elastic File System which is an infinitely expanding, managed NFS service. We recently ran out of space on our Nextcloud instance's EBS volume, so...

Not so long ago AWS announced that Lambda functions will be stopped if they run infinitely or excessively in a loop. A common pattern for that case is when Lambda is executed from SQS queue or SNS topic and submits a message to the same queue or topic. There are...

Sometimes we have projects that build long or that we don't need builds for very often. We want to get the automated builds but only on demand. In GitLab we can set the job to manual. However, if we create jobs for each branch, the list of new pipelines will...

When we generate presigned URL for S3 uploads there's not much we can set in advance. We can set the key, some metadata like `Content-Type` and `Content-Disposition`. What if we wanted to specify some metadata that cannot be set in advance? In this example the user will first create a...

If we are given an image with packed PHP application and it also contains resources like images, plain HTML files, we would like to serve them directly, without involvement of FPM. On a single instance, whether it is a container or a VM, it is easy - we just point...

It's obvious that among the best practices in terms of security is to deploy some of your instance in a private network, where the outside world can't access them directly. They should be hidden behind a load balancer or some other instance. However, that prevents the instance from accessing the...

Configuring the cloud environment is not an easy task. Despite a lot of effort being taken away from ops engineers, it still takes a lot to ensure that our systems are configured correctly, and more important, securely. Today, let me introduce you to AWS Config and AWS Security Hub. ...

Previously we created a passthrough in our monolith service into the fruits microservice. This is useful for users that still use the old endpoints, the old frontend. However, in order to save on bandwidth, we prefer for the users to call the microservice directly. For that we will change our...

In the previous post we implemented the Fruits microservice. However, it is still not delivered to our users. To make everything as smooth as possible, we will keep the current monolith endpoints and route them to the microservice - implementing a strangler pattern [^1]. In the future, we plan to...

Today we will create a simple routine for scanning our instances using [Trivy](https://aquasecurity.github.io/trivy/). It is a quite lightweight tool for discovering vulnerabilities, stored secrets and other security concerns in the given targets. It can scan files in the filesystem, Docker images, and even AWS infrastructure. In this post we will...

In today's post we will focus on splitting the code from the monolith into an actual separate service. Previously we have already prepared the new database (Redis) and migrated the table from the old SQLite database. Code for the monolith version in the previous post is tagged as [`1.1`...

In the previous post we discussed and designed how do we want to split our monolith service into an independent `accounts` and `fruits` services. There are multiple approaches to do it. Let's go with the data first approach. Using this technique, we migrate the data first and during code implementation,...

Deploying with Ansible is easy. Just copy files over, restart services, and observe a new version of the app. But that might break some connections if some user was just waiting for the server's response. What to do then? Base infrastructure...

This post will be more of my hypothetical deliberation on splitting a monolith. I stumbled multiple times upon a software that was logically split in code into separate modules and programs but still used the same database, often read and write from common tables. Today, using a simple example we...

In the previous post we created a Datadog check and monitor to keep track of how many packages can be upgraded on our systems, when was the last time we rebooted the system and what is the major release of the operating system. Today, we will replicate the same metrics...

Updating your systems, especially those that are constantly connected to the Internet, is very important from a security standpoint. Currently, state of the art, cloud-native solution is to use containers and orchestrators such as Docker and Kubernetes. There's also an option to use virtual machines and images, made with Packer...

In this post we will go through the process of converting the playbook from the previous posts into a reusable role. At the last step, we will go through the process of publishing the role in Ansible Galaxy. The finished process is [available here](https://galaxy.ansible.com/ppabis/smartfox). ...

In this post, we will schedule a scan of our website with `sslscan` using GitLab CI and Docker. We will also use a custom Docker image to run the scan and store the results in a file. We will also create a stage to analyze the results from the scan...

Previously we adapted our playbook to configure more aspects of SmartFox like ports, copying Zone files and extensions. In this post, let's make the playbook support more platforms, namely RedHat based systems, like AlmaLinux, and ARM CPU systems, like AWS Graviton. We will also fix an issue where SmartFox archive...

In the previous post, we created a playbook with which we could deploy SmartFox server on a Debian/Ubuntu machine. It contained basic configuration like admin username, password and SSL certificate parameters. In this part, we are going to extend the playbook and make it more configurable. If you didn't...

SmartFoxServer is a software that acts as a framework for building backend for online games. It lets game developers to focus on the gameplay, while SmartFox takes care of connection, authentication, room management, in-game chat, etc. It is very simple to install and configure. I shouldn't need to introduce Ansible...

So you got your new job as a sole DevOps Engineer. Congratulations! For the first months your roam around your company's AWS. You see all the manually created EC2 instances, RDS databases, Elastic Load Balancers, etc. Well, they say a better practice is to use Infrastructure as Code than to...