End-to-End Cloud Data Platform

An ELT Pipeline for NYC Taxi & Weather Analysis

Project Visuals

Project Overview

Data Engineering

This project is a complete, end-to-end data platform built on AWS. It automates the ingestion, transformation, and analysis of NYC taxi trip and weather data to uncover the impact of precipitation on travel times.

The entire cloud environment was provisioned using Terraform, establishing a reproducible and version-controlled infrastructure. An ELT pipeline, orchestrated by Apache Airflow, extracts raw data from a public data lake and a live weather API, loading it into a Redshift data warehouse. Data is then transformed, cleaned, and validated using dbt, with a suite of over 50 data quality tests to ensure accuracy. The final, analysis-ready data is presented in the interactive Streamlit dashboard.

My Contributions

  • Cloud Architecture Design (AWS & Terraform)
  • Modern ELT Pipeline (Airflow & dbt)
  • Data Quality & Testing (dbt)
  • Data Warehouse Performance Tuning (Redshift)
  • Interactive BI Dashboard (Streamlit)
Image

His expertise in ML is strong, Kevin is always ready to illuminate even the most tangled concepts. Kevin's dedication and clear explanations propelled our team across the finish line, leaving us all immensely grateful for his contribution.

— Jared Benedict

Image

Kevin is not only a great, knowledgeable teammate, but also an exceptional technical leader, who always ensured our milestones were met. Given the opportunity, I would happily work with Kevin again.

— Jarrod Pelley

Image

His commitment to excellence is truly commendable; he approached every challenge with a determined mindset, setting a high standard for the entire team. Kevin's collaborative spirit made him an invaluable team player. He seamlessly integrated with our group, fostering a positive and productive atmosphere.

— Vipul Koti

Image

Excuse me for using a sports metaphor but Kevin was like Michael Jordan to our dream team. He has an incredible grasp for ML and a good understanding of what it takes to succeed in the field. His unique ability to communicate and his commitment and determination inspired us to do our best work.

— Clivens LaGuerre

Image

He consistently showcased leadership skills, building out a plan for our project, working with team members to utilize their skills, and making sure that everything on our project roadmap was completed and successful.

— Eric Nagel