S1:E4

Data Ingestion using PyAirbyte

Data Platform
18 min

Host

Jonny DaenenJonny Daenen

Guest

Tarik JamoulleTarik Jamoulle

Move your Google Drive documents straight into Postgres using Python and PyAirbyte. In this Technical Explorations episode, Jonny and Tarik from Dataminded show how they ingest internal meeting transcripts (Facts at Breakfast, Learning Over Lunch) from Google Drive into a relational table, ready for querying and AI use cases.

Read transcript →

What you'll see

  • ▸Configure PyAirbyte to read from a Google Drive folder
  • ▸Authenticate with a Google service account (JSON key)
  • ▸Convert Airbyte output into a clean pandas DataFrame
  • ▸Load the processed data into a Postgres table
  • ▸Discuss performance limits, API rate limits, and batching
  • ▸Reflect on when PyAirbyte is great for PoCs vs. production setups
  • ▸How many connectors Airbyte offers and what PyAirbyte can reuse
  • ▸Trade-offs of code-first ingestion vs. point-and-click UI
  • ▸Ideas for the next step: using MindsDB and LLMs to query this knowledge base

Chapters

  1. 00:00Intro
  2. 01:18What is Airbyte? (and 600+ connectors)
  3. 04:11Demo: Google Drive → Postgres
  4. 09:22Q: How do you get the table structure?
  5. 10:43Scale & format limits (many files, PDFs, images)
  6. 12:45Setting up Google Drive: auth & permissions
  7. 14:44Running it in production: Airflow + Docker
  8. 15:15Next up: MindsDB + verdict

Tags

Data IngestionPyAirbyteAirbyteData IntegrationGoogle DrivePostgresPythonData ConnectorsOpen SourceData Pipelines

Listen on

Technologies

PyAirbyteAirbyteGoogle DrivePostgresPythonpandas

Additional material