AWS hosted enterprise Data Lake with both batch and realtime data pipelines.

Talks co-organized with PyAmsterdam about ETL/ELT in action in the cloud

A simple shell script to delete multiple tables based on table name prefix.

politician stock market activity web scraping project

Data Lakehouse solution for data produced by STEDI Step Trainer sensors and the mobile app so that it can train the machine learning module.

AWS Glue scripts and IAC templates to get started easy. code from AWS Glue Immersion days with Danske Bank 8th+9th of July 2022.

a toolkit that provides an object-oriented interface for working with parquet datasets on AWS

Implementation of ETL data pipeline to load data from S3 to snowflake and refresh tableau datasource in AWS

Data lake project for a US based Insurance Company

Intro to streaming data with Kafka, Spark and AWS Glue

End-to-end batch and streaming data pipeline on AWS to process user ratings and activity data. Leverages Amazon RDS, Glue, S3, Kinesis, and PostgreSQL with pgvector for real-time recommendation generation and model training.

This project is based for legacy applications that works with positional files to process data. The objetive is read these positional files when they arrives in AWS S3, and then send to a dataware-house like AWS Redshift, and finally read the results with a Business Intelligence tool as AWS QuickSight.

Get the dataset intro a S3 bucket, use AWS glue to transform the dataset, write a Lambda script to clean the dataset, query the dataset via AWS Athena then build a dashboard using AWS Quicksight.

Working with Glue Data Catalog and running the using S3 Event Notification and creating the entire stack using AWS CloudFormation

Movie Recommendation System using Big Data

This project repo 📺 offers a robust solution meticulously crafted to efficiently manage, process, and analyze YouTube video data leveraging the power of AWS services. Whether you're diving into structured statistics or exploring the nuances of trending key metrics, this pipeline is engineered to handle it all with finesse.

A small walkthrough how to create an AWS Glue Job Pipeline with AWS CDK

SkillShift provides insights on evolving skill requirements across industries offering detailed analysis on skill demand, workplace culture, and industry trends, helping professionals make informed decisions about career development in a dynamic job market.