Data Engineering Master Class using AWS Analytics Services

Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lake or Data Warehouse and then from Data Lake or Data Warehouse to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using AWS Analytics Stack. It includes services such as Glue, Elastic Map Reduce (EMR), Lambda Functions, Athena, EMR, Kinesis, and many more.

Here are the high–level steps which you will follow as part of the course.

Setup Development Environment

Getting Started with AWS

Storage – All about AWS s3 (Simple Storage Service)

User Level Security – Managing Users, Roles and Policies using IAM

Infrastructure – AWS EC2 (Elastic Cloud Compute)

Data Ingestion using AWS Lambda Functions

Development Life Cycle of Pyspark

Overview of AWS Glue Components

Setup Spark History Server for AWS Glue Jobs

Deep Dive into AWS Glue Catalog

Exploring AWS Glue Job APIs

AWS Glue Job Bookmarks

Getting Started with AWS EMR

Deploying Spark Applications using AWS EMR

Streaming Pipeline using AWS Kinesis

Consuming Data from AWS s3 using boto3 ingested using AWS Kinesis

Populating GitHub Data to AWS Dynamodb

Overview of Amazon AWS Athena

Amazon AWS Athena using AWS CLI

Amazon AWS Athena using Python boto3