Technology
Theme
Period
Aug 2026
Building Idempotent Infrastructure: A Guide to provisioning your infrastructure with Data Product…
As data architecture scales, keeping your real-world infrastructure, like S3 buckets, database schemas, or Snowflake roles, in sync with your data governance platform is a big challenge.
Aug 2026
Can small LLMs match Claude Sonnet for agentic coding?
The cost of AI coding assistants is climbing and not just because prices are increasing.
Aug 2026
Making your self-hosted LLM production-ready
I wanted to host Gemma 431B and had an H100 GPU with 80 GB of VRAM. The math looked straightforward: 31 billion parameters × 2 bytes per parameter ≈ 62 GB, making the H100 a great fit for my model.
Jul 2026
Migrating a Data Platform: 20% technology, 80% everything else
At some point, when the bills become too high, or the platform is not scalable anymore, almost every organisation with a data platform reaches the same crossroads: “We need to migrate”.
Jun 2026
Not every data question deserves a full data product (and that's okay)
When we started building the Data Product Portal, we had a clear conviction: data product thinking is the right approach for getting value out of data in organisations. A data product has an owner.
Jun 2026
Beyond the Cron: Reclaiming Your Data Intervals in Hybrid Scheduling
Many data pipelines are either on a time schedule or triggered by events. Events are ideal for maximum data freshness, but are unpredictable.
May 2026
uv scripts: micro-production situationship
Sometimes you are not trying to start a Python project at all.
May 2026
Portable S3 security for EU clouds
At the core of most modern data platforms sits a lakehouse on S3-compatible storage. It is the central store for all organisational data, from HR to sales and beyond.
May 2026
Dezoom! How to pitch a data platform to your leadership & organization.
This article has been co-written with my colleague, Jelle De Vleminck. Thanks for all the ideas coming from your gigantic reading list as…
May 2026
Data Platforms for humans
At Dataminded we build Data Platforms for a living. We’re good at it and we have all kinds of things to say about the technical stuff.
Apr 2026
Your cloud strategy after the hyperscaler era
For years, choosing a cloud provider was mostly a question of features, price, and convenience.
Apr 2026
The Boring Stuff That Keeps You in the 5%
95% of GenAI pilots fail. What do you actually need to take GenAI from POC to production?
Apr 2026
Authorizing AWS Principals on Azure
How to delegate trust from Entra to AWS IAM through Cognito, authorizing Azure actions without needing long-lived credentials.
Apr 2026
Scaling your Data Platform with Reference Data Products
A well-designed data platform makes it safe and easy to launch the infrastructure required to build successful data products. The platform…
Apr 2026
You Built a Data Mesh, But Your Metrics Are Still a Mess. Here’s Why.
At Dataminded, we work with organizations at various stages of their data journey. One pattern we keep running into: teams that have…
Mar 2026
DuckLake Wants to Fix the Lakehouse. Can It?
While I was on the bench between projects at Dataminded, I had a chance to explore DuckLake and lakehouse architecture in more depth. A…
Mar 2026
From Idea to Implementation: Building an MCP Server for our Data Product Portal
Over the past weeks I’ve been experimenting with building an MCP (Model Context Protocol) server for our Data Product Portal. The portal is…
Mar 2026
From knowledge graphs, to star schemas, to data products
How three powerful concepts come together. A Data Engineer’s reflection.
Mar 2026
How to set up Databricks for data products?
Companies increasingly rely on data products for decision-making, insights, and AI/ML. data products separate data in meaningful entities…
Mar 2026
Musings from the Second Summit on Data Product Oriented Architectures
What is the state of the art in data products? Last week we joined the Second Summit on Data Product Oriented Architectures to find out…
Mar 2026
Why Your Next Data Catalog Should Be a Marketplace
Explore why traditional data catalogs are failing to meet business needs and our vision for the solution: a data product marketplace.
Mar 2026
You Don’t Have a Data Platform Without Excel
The Most Used Feature is “Export”: Why Your Data Stack Needs a Spreadsheet Strategy
Dec 2025
Using AWS IAM with STS as an identity Provider
How EKS tokens are created, and how we can use the same technique to use AWS IAM as an identity provider.
Nov 2025
Slaying the Terraform Monostate Beast
You start out building your data platform. You choose Terraform because you want to do it the right way and put your infra in code. There’s…
Nov 2025
Introducing Conveyor: Build, Deploy & Scale Your Data Projects
Today, the team at Data Minded is proud to announce the launch of Conveyor, our managed, cloud-based data platform.
Oct 2025
How to Prevent Crippling Your Infrastructure When AWS US-EAST-1 Fails
What the October 2025 outage reminded us about dependencies, preparedness, and the people behind the cloud.
Oct 2025
When writing SQL isn't enough: debugging PostgreSQL in production
While writing efficient SQL queries is essential, it is not enough to operate a database at scale. I will illustrate this using 3 issues
Sep 2025
The State of Data Work in 2025: Insights From 32 In-Depth Conversations
Understanding the real challenges faced by data professionals is essential for strategic planning. Our team recently completed an extensive…
Sep 2025
The ROI Challenge: Why Measuring Data’s Value is Hard, but Crucial
Here’s a story playing out in many data-driven organizations, and it often follows a familiar three-act structure.
Sep 2025
Locking down your data: fine-grained data access on EU Clouds
This post investigates how to restrict data access for both SQL and Spark/Python applications when using EU clouds
Aug 2025
Rethinking the data product workbench in the age of AI
In this blogpost we will explore the challenges related to building data products share our vision for a data product workbench using AI.
Jul 2025
The Mystery of Folders on AWS S3
The difference between objects and files, and what the AWS Console really does when you press the “create folder” button.
Jul 2025
Portable by design: Rethinking data platforms in the age of digital sovereignty
Recent geo-political and legal rulings have triggered us to investigate how data platforms can be designed for portability across providers
Jun 2025
Cloud Independence: Testing a European Cloud Provider Against the Giants
Somewhere in Europe. You might be running a small online business, managing a mid-sized automotive supplier, or leading a global…
Apr 2025
Why the ‘Private’ API Gateway of AWS Might Not Be as Secure as You Think
Designing secure applications is a challenge for everyone. A big part of this is based on who can access what. In this blog I want to dig…
Apr 2025
The Data Engineer’s guide to optimizing Kubernetes
By default Kubernetes is not optimal for running batch workloads. I tackle spot instances, tweaks in autoscaling and node efficiency…
Apr 2025
Are your AKS logging costs too high? Here’s how to reduce them
Explore how to reduce the cost of logging on Azure by analyzing your applications and investigating Basics log analytics tables in Azure.
Apr 2025
Integrating MegaLinter to Automate Linting Across Multiple Codebases. A Technical Description.
If you’re not familiar with linters, or specifically with MegaLinter, please take a look at my previous article on the topic. In contrast…
Apr 2025
Running thousands of Spark applications without losing your cool
I explain how to troubleshoot and detect problematic Spark applications at scale as well as show how this can be used to reduce your costs.
Apr 2025
The building blocks of successful Data Teams
Based on my experience I will elaborate on key criteria for building successful data teams
Apr 2025
Stop loading bad quality data
Rule number one of having good quality data: Stop loading bad quality data. Really, it’s that simple. I see so many companies make this…
Mar 2025
What Is Data Product Thinking?
Data product thinking is gaining momentum in the data world right now so we recently organised a live online learning session for Conveyor…
Mar 2025
Debugging Running Pods on Kubernetes
Exploring Kubernetes’s debugging feature, kubectl debug, and extending kubectl debug to support volume mounts
Mar 2025
Head-to-head comparison of dbt SQL engines
Compare usage and performance of dbt against 3 popular open-source SQL engines, namely: Spark, Trino and Duckdb
Mar 2025
Improve the security of pods on kubernetes
In this article we will show you four settings you should enable on your pods to improve the security of your kubernetes cluster.
Mar 2025
Improving Apache Spark performance on k8s
This post talks about adding external disks to your Kubernetes executors in order to speed up your spark jobs using the TPC-DS benchmark.
Mar 2025
Why data engineers should be more like software engineers
Data engineers are better when using a product mindset as well as software best practices: cicd pipelines, test code, develop iteratively.
Mar 2025
Source-Aligned Data Products: The Foundation of a Scalable Data Mesh
Introduction
Dec 2024
Data Product Thinking Rethought
On November 27, 2024, at Startplatz in Düsseldorf, we had a fantastic evening where data professionals gathered for the Data Product…
Dec 2024
Data Product Portal Integrations 1: OIDC
How to integrate Open ID Connect with the Data Product Portal
Dec 2024
Data Product Portal Integrations 2: Helm
How to install portal in your production environment using Helm
Dec 2024
Data Modelling In A Data Product World
Many organisations are hitting the limits of data warehousing, especially as they grow in size. They often see adopting data products as a…
Dec 2024
Why not to build your own data platform
A round-table discussion summary on imec’s approach to their data platform
Nov 2024
Beyond the Buzzwords: Let’s Talk About the Real Challenges in Data
We’re on a mission to cut through the noise and talk about the real challenges that data teams face in our fast-moving industry. And what…
Oct 2024
From Good AI to Good Data Engineering. Or how Responsible AI interplays with High Data Quality
The intersection of artificial intelligence (AI) and data engineering has become increasingly critical. As AI technologies proliferate, the…
Oct 2024
A glimpse into the life of a data leader
Last week, Dataminded organised a data leadership roundtable. We invited 20 decision takers of large data organisations, from Belgium, the…
Oct 2024
Why rising cloud costs are the silent killers of data platforms
Building data platforms in the cloud is changing. Gone are the days that you would manually set up a few EC2 instances and run some modest…
Oct 2024
Quack, Quack, Ka-Ching: Cut Costs by Querying Snowflake from DuckDB
How to leverage Snowflake’s support for interoperable open lakehouse technology — Iceberg — to save money.
Oct 2024
How to Effectively Structure Data for Self-Service Data Teams
For years, data platforms — particularly data lakes and lakehouses — have relied on the medallion architecture. This tiered system…
Sep 2024
How we democratized data access with Streamlit and Microsoft-powered automation
How we democratized data access with Streamlit and Microsoft-powered automation Within the community of data professionals, the term “data governance” often conjures up an image of a large …
Sep 2024
Unlocking the new Power of Advanced Analytics
In recent years advanced analytics has become a cornerstone for businesses aiming to gain deeper insights and make informed decisions. This…
Aug 2024
How To Conquer The Complexity Of The Modern Data Stack
The more people you add to a team, the more lines of communication you introduce. The same holds for the number of tools in your data…
Aug 2024
The Data Product Portal Integrates With Your Preferred Data Platform
A couple of weeks ago we announced the release of the Data Product Portal as an open source repository. The Data Product Portal is an…
Aug 2024
How To Reduce Pressure On Your Data Teams
In August 2016, BARC published the results of a global survey on Data-Driven Decision-Making in Business.Results are astonishing: only 22%…
Aug 2024
Microsoft Fabric’s Migration Hurdles: My Experience
It has been more than a year since Microsoft announced their new all-mighty Fabric data platform. It offers a wide range of capabilities…
Aug 2024
Data Stability with Python: How to Catch Even the Smallest Changes
As a data engineer, it is nearly always the safest option to run data pipelines every X minutes. This allows you to sleep well at night…
Jul 2024
Why You Should Build A User Interface To Your Data Platform
Modern data platforms are complex. If you look at reference architectures, like the one from A16Z below, it contains 30+ boxes. Each box…
Jul 2024
Clear signals: Enhancing communication within a data team
Clear, effective communication is as crucial to successful data engineering as technical expertise.
Jul 2024
The State of Data Products in 2024
Gartner has released their hype cycle for data management 2024 quite recently and has identified Data Products at the gate of the peak of…
Jul 2024
Unlock Insights & Learnings: Dataminded Newsletter — June/July 2024 Edition
Welcome to the June — July 2024 edition of the Dataminded Newsletter! These months have been full of events and exciting developments for…
Jul 2024
Demystifying Device Flow
Implementing OAuth 2.0 Device Authorization Grant with AWS Cognito and FastAPI
Jul 2024
The Missing Piece to Data Democratization is More Actionable Than a Catalog
As of the nineties, with the advent of Business Intelligence, organizations do attempt to install data driven decision making and do aim to…
Jun 2024
Introducing Data Product Portal: An open source tool for scaling your data products
In the fast-evolving world of data, companies are discovering that the key to success for scaling their data initiatives is not to rely on…
Jun 2024
Short feedback cycles on AWS Lambda
A Makefile that enables to iterate quickly
May 2024
Prompt Engineering for a Better SQL Code Generation With LLMs
Picture yourself as a marketing executive tasked with optimising advertising strategies to target different customer segments effectively…
May 2024
Age of DataFrames 2: Polars edition
In this publication, I showcase some Polars tricks and features.
May 2024
A 5-step approach to improve data platform experience
A guide to turning user feedback into continuous platform improvement.
Apr 2024
Querying Hierarchical Data with Postgres
Hierarchical data is prevalent and simple to store, but querying it can be challenging. This post will guide you through the process of…
Apr 2024
Unlock Insights & Learnings: Data Minded Newsletter — March/April 2024 Edition
Welcome to the March — April 2024 edition of the Data Minded Newsletter! We’re thrilled to share some exciting updates, valuable insights…
Mar 2024
How to organize a data team to get the most value out of data
To state the obvious: a data team is there to bring value to the company. But is it this obvious? Haven’t companies too often created a…
Mar 2024
Securely use Snowflake from VS Code in the browser
At Conveyor we help you to build, deploy, and scale your data products. One way we do that is by offering IDEs that run in the cloud. A…
Mar 2024
The benefits of a data platform team
Shift your focus from maintenance to value
Mar 2024
Becoming Clout* certified
Hot takes about my experience with cloud certifications
Mar 2024
You can use a supercomputer to send an email but should you?
Discover the next evolution in data processing with DuckDB and Polars
Mar 2024
Two Lifecycle Policies Every S3 Bucket Should Have
Abandoned multipart uploads and expired delete markers: what are they, and why you must care about them thanks to bad AWS defaults.
Feb 2024
How we used GenAI to make sense of the government
We built a RAG chatbot with AWS Bedrock and GPT to answer questions about the Flemish government
Feb 2024
My key takeaways for building a data engineering platform
Having been a member of a product team for two years, I aim to share three valuable insights that I have gained.
Feb 2024
Leveraging Pydantic for validation.
Ensuring clean and reliable input is crucial for building robust services. One powerful tool that simplifies this process is Pydantic, a…
Jan 2024
7 Lessons Learned migrating dbt code from Snowflake to Trino
Just change the target in profiles.yml, right?
Jan 2024
Everyone to the data dance floor: a story of trust
Wandering through the vast realm of the internet, on stumbles upon a countless amount of articles about companies transitioning from…
Jan 2024
Growing your data program with a use-case-driven approach
How to Bridge the Gap Between Strategic Vision and Tactical Execution in Data Initiatives
Jan 2024
SAP CDC with Azure Data Factory
Using a self-hosted Integration Runtime
Jan 2024
Quacking Queries in the Azure Cloud with DuckDB
This post describes 2 Duckdb extensions that enable you to read data from Azure blob storage. It also shows code for both Python and dbt.
Jan 2024
3 software design patterns that every software data engineer should know
With real-life examples
Jan 2024
Harnessing AWS MSK and AWS EMR for Real-time Analytics
A Deep Dive into Building a Scalable Analytics Pipeline, automating the infrastructure with Terraform
Jan 2024
How to run PySpark jobs in an Amazon EMR Serverless Cluster with Terraform
Ignite Your Data Revolution and Harness the Power of Amazon EMR Serverless
Jan 2024
Pulumi vs. Terraform: Choosing your IaC Tool
Similarities and differences
Jan 2024
Step-by-Step Guide to deploy a Kafka Cluster with AWS MSK and Terraform
A Comprehensive Walkthrough to Deploying and Managing Kafka Clusters with Amazon MSK and Terraform
Dec 2023
Making the Airflow web UI faster
It’s a nice sunny Wednesday afternoon in Belgium, and my colleague Jan and I decide to go for a walk in the cozy city of Leuven. As we…
Dec 2023
Connecting to Databases using JDBC from the CLI
A quick guide to using the CLI tool sqlline to connect to any database with JDBC drivers
Nov 2023
Terra-Do’s and Terra-Don’ts — a few common issues with Terraform iterables and how to avoid them
One of the most common issues I observe when teaching Terraform is improper iteration over resources, data sources and modules, which in…
Nov 2023
Upserting Data using Spark and Iceberg
Use Spark and Iceberg’s MERGE INTO syntax to efficiently store daily, incremental snapshots of a mutable source table.
Nov 2023
Polars dataframe’s plugins and extensibility: getting started
Interesting feature of Polars explained.
Oct 2023
Twelve-Factor Python applications using Pydantic Settings
A look at Pydantic Settings and how it can help you reliably deploy applications across environments
Oct 2023
From notebook hell to container heaven
This article is the first chapter of a 3-parts tutorial
Oct 2023
How we reduced our docker build times by 40%
This post describes two ways to speed up building your Docker images: caching build info remotely, using the link option when copying files
Sep 2023
Soft Skills in a Tech World: A Psychologist’s Journey to IT Consulting
Five factors that facilitated my career switch to IT
Sep 2023
Testing frameworks in dbt
When I was struggling to find the right way to write tests in dbt, I got great inspiration from Mariah Rogers’ talk about testing at…
Aug 2023
Navigating the Iceberg: unit testing iceberg tables with Pyspark
The table format iceberg has gained a lot traction and has caused great excitement across the data-landscape. The architecture allows…
Jul 2023
Automating refactoring across teams and projects
Conquering breaking changes Chaos: Leveraging automatic refactoring tools across teams and projects.
Jul 2023
Cross-DAG Dependencies in Apache Airflow: A Comprehensive Guide
Exploring four methods to effectively manage and scale your data workflow dependencies with Apache Airflow.
Jun 2023
Enhance Your ETL Ingestion: Unlocking the Power of the Apache Iceberg Table Format
Regardless of all the changes that are happening in the data landscape today, data ingestion and ETL processes still play an important role…
May 2023
How Leading Data Organizations Achieve Success: Prioritize People, Process, and Product
Technology is not the biggest challenge anymore when building data platforms
Apr 2023
Write Cookiecutters faster
This one’s about writing cookiecutter templates. 🍪 If you’ve ever done that, you might recognize the following git log
Apr 2023
Use dbt and Duckdb instead of Spark in data pipelines
Dbt has become very popular for transformation on top of your data warehouse. We see potential to use dbt with Duckdb on top of a data…
Dec 2022
Make Gitpod Open Sites in the Browser
Gitpod configuration for opening links in the browser instead of in the terminal.
Nov 2022
Slash your cloud bill by moving data workloads to cost-effective compute
These are tough times for technology
Nov 2022
Content is no longer the king … It’s all about experiential marketing!
My honest takeaways on where HubSpot can do better & be better implemented!
Nov 2022
The rise of remote development environments
Gitpod and Codespaces are the first remote development environments that we would use ourselves and may also be useful for you.
Oct 2022
Debugging Google Application Default Credentials
Inspecting gcloud application default credentials, Google access tokens, and ID tokens through the refresh token grant & token…
Oct 2022
Not every m5.4xlarge is created equally
You might be surprised to learn that not every instance of the same type has the exact same amount of memory on AWS and this can have some…
Oct 2022
Mastering Cloud: can we do better than certificates?
A flipped classroom approach can help you get productive in the cloud more quickly.
Oct 2022
Will Dagger revolutionize CI/CD?
A hands-on exploration of Dagger CI/CD pipelines
Oct 2022
ML Pipelines in Azure Machine Learning Studio the right way
An opinionated way to get started quickly
Sep 2022
AWS Web Identity Token Authentication in Legacy Applications
How to run make applications without support for AWS web identity token authentication run in environments requiring it
Sep 2022
CI/CD for data projects: Why manual deployments are not good enough
CI/CD for data projects: Why manual deployments are not good enough Data engineering is slowly but surely adopting many of the best practices from Software engineering. Two of these best practices …
Sep 2022
Containerizing Git credential helpers
How to expose Git credential helpers to containerized processes, allowing the use of Bitbucket, Github and Gitlab inside of Docker on…
Aug 2022
Aws security for software engineers
Security breaches are more and more common. This preview shows you what we will talk about during our webinar on AWS security.
Aug 2022
Dagger vs. the current state of CI/CD
How Dagger tries to solve some of the pain points of current CI systems.
Aug 2022
What does it take to build a data platform
On August 9th, I will host a webinar on what it takes to build a data platform and which lessons I learned from helping my clients.
Jul 2022
Make Spark resilient against spot interruptions on kubernetes
Based on our experience of running spark in production at our customers, we discuss 3 ways to improve the resilience of spark on kubernetes
Jun 2022
Why I will not build my next data platform myself
The blogpost explains the difference between can a company build it’s own data platform and should it build it themself.
Apr 2022
Porting a data platform from AWS to Azure
We recently created an Azure version of our data platform and this blogpost elaborates on our learnings/issues to support a new cloud…
Apr 2022
Using kubebuilder in production
Some extra tips on using kubebuilder in production
Apr 2022
Batch orchestration on Azure flowchart
Managed solutions vs building it yourself
Apr 2022
The different ways to configure AKS connectivity
This post elaborates on the different ways to configure AKS within your network as well as explains the most important configuration…
Apr 2022
Why you should not use IAM users
AWS provides two main options for giving users access to your AWS resources: IAM users and roles. You should avoid the former.
Apr 2022
The 6 pillars of data maturity
In our previous blog, we already explained that you’ll have to put in some effort if you want to grow your data maturity. Now, let’s take a…
Apr 2022
Why grow in data maturity?
Every company pays lip service to data. Who takes action?
Apr 2022
Automatically following insiders transactions on the belgian stock market with Serverless on AWS
Recently I was reading the newspaper and I saw an article in which the owner family of D’Ieteren, a company listed on the Belgian stock…
Mar 2022
Keep Long-Lived AWS Credentials Out of Untrusted Environments
Generate and use short-lived AWS session credentials, keeping your AWS IAM keys secure
Mar 2022
Running project-specific CI/CD pipelines for a monorepo in AWS
Implementing a flexible (yet powerful) CI/CD setup for monorepos using AWS CodeBuild, CodePipeline, and Lambda Functions.
Mar 2022
Running Containers on Windows Subsystem for Linux (WSL 2)
How to install and automatically start Docker Engine on WSL distributions such as Ubuntu.
Mar 2022
Three methods for obtaining GCP access tokens
Using user credentials, service account credentials or the metadata service to obtain access tokens from Google’s identity service
Feb 2022
From notebook hell to container heaven, Part II.
This article is the second chapter of a 3-parts tutorial
Feb 2022
Why you should govern data access through Purpose-Based Access Control
PBAC is a powerful data access governance strategy that can make your data access policies more practical and secure.
Jan 2022
Datafy feature release Q4 2021
Today we want to share with you the features the Datafy team has been working in the last part of 2021.
Dec 2021
How to access private Git repositories during a Docker image build
A complete guide to building images that require access to SSH keys during the build process.
Nov 2021
Customizing SageMaker Notebook Instances
How SageMaker’s lifecycle configuration works, a collection of useful startup scripts and how to bundle them
Nov 2021
Sending mail from Google Cloud Build
A simple solution for sending messages from a GCP Cloud Build pipeline
Oct 2021
Backing up and restoring SageMaker Notebook Instances
How to store and restore SageMaker Notebook instances to and from S3, for example for migration to Amazon Linux 2
Sep 2021
What I wish I knew before going into Data Engineering
Disclaimer: this is my opinion, not necessarily the one of my employer or any organisation.
Sep 2021
Authentication on GCP with Docker: Application Default Credentials
How applications magically authenticate themselves with GCP through their environment, and how to make locally running containers magic too
Sep 2021
What is lakeFS: A Critical Survey
A critical introduction to lakeFS, a new metadata layering solution that brings Git-like operations and versioning to object storage.
Aug 2021
Mastering the Google Cloud Platform SDK tools
A look at some lesser-known GCP SDK settings and features that make your day-to-day interactions with GCP more enjoyable.
May 2021
How RDS proxy allowed us to run Airflow 200% more efficient
Running multiple Airflow instances on the same RDS means a lot of open database connections. RDS proxy can make this setup more efficient.
May 2021
Storing Snowplow bad row events in BigQuery
How to use Cloud Functions and a BigQuery schema generator to make Snowplow bad row schema violation events easily queryable
Apr 2021
CORS and the SOP explained
Introduction to Cross-Origin Resource Sharing (CORS) and the Same-Origin Policy (SOP). Structured as a dialogue, and focused on the why.
Apr 2021
How to Access Key Vaults from Azure Batch Jobs
The cheapest and simplest way of running computational jobs on Azure is by using Azure Batch. This service enables you to launch managed…
Mar 2021
Consulting 101 for data engineering
At Data Minded, we are data engineers first and foremost. But in reality, we do a lot of consulting. What is consulting really? I recently…
Mar 2021
What to consider before choosing Argo Workflow?
To go full Kubernetes-native or not?
Jan 2021
Tracking prevention in the modern browser
An overview of cross-domain tracking prevention in the modern browser, specifically of Intelligent Tracking Prevention (ITP) in WebKit.
Jan 2021
Joining Spark Datasets
Ever wanted to do better than joins on Apache Spark DataFrames? Now you can!
Dec 2020
How to share tabular data in a privacy-preserving way
Adding noise to existing rows, only adding noise to outcomes of tasks performed on that data, or synthetic data generation? An intuition.
Dec 2020
Why DBT will one day be bigger than Spark
The world of data is moving and shaking again. Ever since Hadoop came around, people were offloading workloads from their data warehouses…
Dec 2020
Learnings from the AWS Data Analytics Speciality
Last year I blogged about how I got my AWS Pro Architect certificate, which you can read about here…
Nov 2020
Ingesting custom event sources with Snowplow
How to use Snowplow to ingest data from an unsupported source, such as Auth0’s log streaming service.
Nov 2020
How I prepared for the Azure Data Engineer Associate (DP-200 and DP-201) exams
As the title says, this article is about how I personally prepared for the aforementioned exams and not about how one should best prepare…
Nov 2020
Running Spark 3 on AKS with Azure AD integration
Do you want to run Spark 3 on AKS in pro mode? Meaning no more “just copy-paste the storage account access key into the source code, and…
Oct 2020
Combine AWS cloud with OVH storage for handling sensitive EU data
Now that the Privacy Shield has been invalidated, there are some legal disputes whether European companies and government agencies can…
Sep 2020
Data Quality Libraries: The Right Fit
A high-level comparison of TensorFlow Data Validation, Great Expectations, and Deequ
Sep 2020
How to deploy analytics workloads
“It works on my machine”. That’s great. But now how do you make sure it runs in production, repeatedly and reliably? Here we share our…
Jul 2020
A summary of cookies
A short, technical summary of cookie concepts such as “third party cookies” and attributes such as a expiration date, host-only et cetera.
Jun 2020
Save money on MSK
Let’s be blunt here for a second: MSK is not a mature managed service. The author of that post may have changed his mind in the meantime…
Jun 2020
Enrichment and batch processing in Snowplow
A close look at Snowplow’s enrichment component as well as the deprecation of its batch pipeline
Jun 2020
Import SQL Server data in BigQuery
A list of four approaches for a one-off data dump from a RDBMS like SQL Server to BigQuery, and an in-depth look at how to use Apache…
Jun 2020
How to conditionally disable modules in terraform
Terraform doesn’t support the count parameter on modules. A proposal was made for a enabled parameter, but this is also not yet present…
Apr 2020
Running through the Google GCP Cloud Foundation Toolkit Setup
An experience report of applying the CFT toolkit foundation steps
Mar 2020
Cloud story: Spin down unneeded infrastructure quickly with terraform to save OPEX
The benefit of infrastructure as code and cloud: If you don’t need it, spin it down.
Feb 2020
Google Pub/Sub: Putting a number on the lack of order
Evaluating Pub/Sub’s lack of order. Making good decisions about watermarks
Jan 2020
The data product lifecycle
Your organisation wants to dive head-first into data and AI but you don’t really know where to start? Data&AI is on the radar of most…
Nov 2019
AWS MSK secure Python Kafka client
How to write a secured python client for AWS MSK using TLS for encryption and authentication
Oct 2019
Organize your data lake using Lighthouse
Lighthouse is an open source library (using Apache Spark and Scala) that we developed at Data Minded, with the aim of providing a way to…
Oct 2019
How to build a cleaning pipeline with BigQuery and DataFlow on GCP
I have a small script running on my phone which sends a set of information to the cloud every 5 minutes. I wanted to build a dataset with…
Oct 2019
Hooray, I’m an AWS Certified Pro Architect. Now what?
Is it worth it to spend time and money collecting advanced certifications? In this blog I share my opinion and lessons learned. TL/DR: Yes…
Sep 2019
Avoiding copy paste in Terraform: Two approaches for multi-environment Infra as code setups
Terraform workspaces vs symlinks and overrides
Aug 2019
Test new Kafka application with prod data with Kafka streams applications running on K8S
Or how to pipe messages from one kafka cluster to another cluster through kubectl and your local machine
Feb 2019
Capture clickstream data with Azure Application Insights
Raw clickstream data is a valuable data source in almost any analytics project. But it’s not always easy to capture. Free tools like…
Nov 2018
Little known Spark DataFrame join types
Probably most of you know the basic join types from SQL: left, right, inner and outer. Since these are supported by most of the…
Nov 2018
Run Spark Jobs on Azure Batch using Azure Container Registry and Blob storage
This is another one of those “how to” blogs that can hopefully help people get up-and-running quickly because it took me a while to figure…
Sep 2018
Hell’s Kitchen — IT edition
Imagine you start a new job in the kitchen of a fancy restaurant. You just graduated from your culinary studies and you’re eager to learn…
Jul 2018
Good leaders take ownership of everything in their world
Last week, yet another political fight started in Belgium. You can read all about it here http://www.standaard.be/cnt/dmf20180719_03624520…
May 2018
Connect to AWS Athena using Datagrip
Datagrip is a great database IDE, just like the other IDEs from Jetbrains: Pycharm, IntelliJ, … In this blog, I describe how to connect an…
Nov 2017
3 years Data Minded!
This month Data Minded turns 3 years already. Like every startup, we’ve had our ups and downs. But looking back I’m extremely proud of the…
Nov 2017
Cloud is a business strategy, not an IT implementation detail
British Airways is not having their best weekend: A big IT outage, resulting in a lockdown at Heathrow, all apparently caused by a single…
Nov 2017
Order some 21st century management with your big data lunch
Why is it that every company today wants to have an Hadoop-enabled data lake, large-scale data pipelines in Spark, and artificial…
Nov 2017
Our three favourite data analytics tools of 2015
In this blog, we look back at the 3 data analytics tools that shaped 2015 for us. This is by no means an objective blog, and I’m sure your…
No matches found. Try a different search.
