Mastering Databricks & Apache spark -Build ETL data pipeline

Welcome to the course on Mastering Databricks & Apache spark –Build ETL data pipeline

Databricks combines the best of data warehouses and data lakes into a lakehouse architecture. In this course we will be learning how to perform various operations in Scala, Python and Spark SQL. This will help every student in building solutions which will create value and mindset to build batch process in any of the language. This course will help in writing same commands in different language and based on your client needs we can adopt and deliver world class solution. We will be building end to end solution in azure databricks.

Key Learning Points

We will be building our own cluster which will process our data and with one click operation we will load different sources data to Azure SQL and Delta tables

After that we will be leveraging databricks notebook to prepare dashboard to answer business questions

Based on the needs we will be deploying infrastructure on Azure cloud

These scenarios will give student 360 degree exposure on cloud platform and how to step up various resources

All activities are performed in Azure Databricks

Fundamentals

Databricks

Delta tables

Concept of versions and vacuum on delta tables