I am working on auto-scaling in Kubernetes and I was seeing something weird in my system and only now I realised that auto-scaling based on relative metrics (like CPU usage) really has to be done o...

In my opinion one of the main responsibilities of a data engineer is to build the infrastructure and clean data for efficient use by various stakeholders (analysts, data scientists…). Data platfor...

Introduction I am an avid fan of PostgreSQL database due to many reasons. What has been lacking in PostgreSQL is a simple, “built-in”, high-availability setup like Cassandra, MongoDB, Kafka… have...

I spent a bit of time looking into alternative load balancing solutions to Nginx/HA Proxy. It seems Envoy is by far the leader in this space and the product seems really good. Probably the best a...

This will be a quick guide on a very easy way to do machine learning competitions. This approach is very general and easily applicable to other competitions. I just want to make it clear that using...

I wanted to share a couple of tips for easier development on AWS. 1. Use a local version of AWS for development and testing Try something like localstack to stand up a local AWS environment. This...

I’ve published online the presentation on Apache Spark I’ve made for a introductory lecture to graduate students at Maastricht University. If interested, please take a look at the presentation here.

Apache Airflow Using Apache Airflow to build reusable ETL on AWS Redshift Apache Kafka + Spark Streaming + Redshift Streaming Tweets to Snowflake Data Warehouse with Spark Structured Streaming a...

I wrote a detailed article showing how to load 6GB of data into Snowflake using the PUT and COPY INTO commands. Then I evaluated the performance of joins and how caching and instance size affects ...

I wrote a blog post describing my 10 favorite features of Snowflake. You can find the full blog post here. A small preview: