How I halved the runtime of my PostgreSQL dbt model using DuckDB With dbt, you can build fairly cross compatible SQL models with only a bit of effort. DuckDB is a new database which is similar to...
Best practice have a version of Python for your project independent of your system and global Python we can achieve this using pyenv deterministically manage the dependencie...
Baseline of features is fron Nginx here. Features Nginx HA Proxy Envoy Load balancer + + + HTTP/TCP/UDP sup...
I’ve recently read the “Advanced Python Mastery” course by David Beazley (can be found here). While there is a lot of material, here are some things I’ll try to incorporate more of: 1) Use abstra...
The Design and Implementation of Modern Column-Oriented Database Systems Probably the best paper to read Massively Parallel Databases and MapReduce Systems very interesting comparison of ...
Data storage Data lake: Object store + storing data in a columnar format examples: S3 + Athena (Presto) PROs: cheaper than...
Data engineering specific: SQL (advanced SQL, database internals, query tuning, administration…) Good coding (Python + a JVM language) Data pipeline design Data modeling techniques Arch...
Have a config driven and easily extensible library for your ETL You can easily add new extractors and your existing loaders will take care of loading the data. Important to abst...
Bunch of scripts Single run-everything script Hacky “homemade” dependency control Hard-coded values eg. data sources, queries… ETL jobs that create d...
You will need to choose between full and incremental extracts Always prefer full extracts and partitioned snapshots if the data is small enough If this is not possible, then des...