Percorso di Certificazione Google Cloud "Professional Data Engineer"

Data Engineering on Google Cloud Platform

Module 1: Introduction to Data Engineering
Explore the role of a data engineer
Analyze data engineering challenges
Intro to BigQuery
Data Lakes and Data Warehouses
Demo: Federated Queries with BigQuery
Transactional Databases vs Data Warehouses
Website Demo: Finding PII in your dataset with DLP API
Partner effectively with other data teams
Manage data access and governance
Build production-ready pipelines
Review GCP customer case study
Lab: Analyzing Data with BigQuery
Module 2: Building a Data Lake
Introduction to Data Lakes
Data Storage and ETL options on GCP
Building a Data Lake using Cloud Storage
Optional Demo: Optimizing cost with Google Cloud Storage classes and Cloud Functions
Securing Cloud Storage
Storing All Sorts of Data Types
Video Demo: Running federated queries on Parquet and ORC files in BigQuery
Cloud SQL as a relational Data Lake
Lab: Loading Taxi Data into Cloud SQL
Module 3: Building a Data Warehouse
The modern data warehouse
Intro to BigQuery
Demo: Query TB+ of data in seconds
Getting Started
Loading Data
Video Demo: Querying Cloud SQL from BigQuery
Lab: Loading Data into BigQuery
Exploring Schemas
Demo: Exploring BigQuery Public Datasets with SQL using INFORMATION_SCHEMA
Schema Design
Nested and Repeated Fields
Demo: Nested and repeated fields in BigQuery
Lab: Working with JSON and Array data in BigQuery
Optimizing with Partitioning and Clustering
Demo: Partitioned and Clustered Tables in BigQuery
Preview: Transforming Batch and Streaming Data
Module 4: Introduction to Building Batch Data Pipelines
EL, ELT, ETL
Quality considerations
How to carry out operations in BigQuery
Demo: ELT to improve data quality in BigQuery
Shortcomings
ETL to solve data quality issues
Module 5: Executing Spark on Cloud Dataproc
The Hadoop ecosystem
Running Hadoop on Cloud Dataproc
GCS instead of HDFS
Optimizing Dataproc
Lab: Running Apache Spark jobs on Cloud Dataproc
Module 6: Serverless Data Processing with Cloud Dataflow
Cloud Dataflow
Why customers value Dataflow
Dataflow Pipelines
Lab: A Simple Dataflow Pipeline (Python/Java)
Lab: MapReduce in Dataflow (Python/Java)
Lab: Side Inputs (Python/Java)
Dataflow Templates
Dataflow SQL
Module 7: Manage Data Pipelines with Cloud Data Fusion and Cloud Composer
Building Batch Data Pipelines visually with Cloud Data Fusion
Components
UI Overview
Building a Pipeline
Exploring Data using Wrangler
Lab: Building and executing a pipeline graph in Cloud Data Fusion
Orchestrating work between GCP services with Cloud Composer
Apache Airflow Environment
DAGs and Operators
Workflow Scheduling
Optional Long Demo: Event-triggered Loading of data with Cloud Composer, Cloud Functions, Cloud Storage, and BigQuery
Monitoring and Logging
Lab: An Introduction to Cloud Composer
Module 8: Introduction to Processing Streaming Data
Processing Streaming Data
Module 9: Serverless Messaging with Cloud Pub/Sub
Cloud Pub/Sub
Lab: Publish Streaming Data into Pub/Sub
Module 10: Cloud Dataflow Streaming Features
Cloud Dataflow Streaming Features
Lab: Streaming Data Pipelines
Module 11: High-Throughput BigQuery and Bigtable Streaming Features
BigQuery Streaming Features
Lab: Streaming Analytics and Dashboards
Cloud Bigtable
Lab: Streaming Data Pipelines into Bigtable
Module 12: Advanced BigQuery Functionality and Performance
Analytic Window Functions
Using With Clauses
GIS Functions
Demo: Mapping Fastest Growing Zip Codes with BigQuery GeoViz
Performance Considerations
Lab: Optimizing your BigQuery Queries for Performance
Optional Lab: Creating Date-Partitioned Tables in BigQuery
Module 13: Introduction to Analytics and AI
What is AI?
From Ad-hoc Data Analysis to Data Driven Decisions
Options for ML models on GCP
Module 14: Prebuilt ML model APIs for Unstructured Data
Unstructured Data is Hard
ML APIs for Enriching Data
Lab: Using the Natural Language API to Classify Unstructured Text
Module 15: Big Data Analytics with Cloud AI Platform Notebooks
What’s a Notebook
BigQuery Magic and Ties to Pandas
Lab: BigQuery in Jupyter Labs on AI Platform
Module 16: Production ML Pipelines with Kubeflow
Ways to do ML on GCP
Kubeflow
AI Hub
Lab: Running AI models on Kubeflow
Module 17: Custom Model building with SQL in BigQuery ML
BigQuery ML for Quick Model Building
Demo: Train a model with BigQuery ML to predict NYC taxi fares
Supported Models
Lab Option 1: Predict Bike Trip Duration with a Regression Model in BQML
Lab Option 2: Movie Recommendations in BigQuery ML
Module 18: Custom Model building with Cloud AutoML
Why Auto ML?
Auto ML Vision
Auto ML NLP
Auto ML Tables

Workshop
Module 1: Data Engineer Certification Overview

Module 2: Google Big Data Fundamentals
Google Big Data History and Overview
Choosing the Right Storage Option
Securing Your Data on Google Cloud Platform
Architecting Data Processing Solutions on GCP

Module 3: Storing Binary Data
Storing Binary Data with Google Cloud Storage
Exercise: Google Cloud Storage
Understanding Persistent Disks Storage
Exercise: Disks and Snapshots

Module 4: Storing Relational Data
Modeling Relational Data
Moving Relational Databases to Cloud SQL
Exercise: Google Cloud SQL Quickstart
Exploiting Spanner for Massively Scalable Relational Systems
Exercise: Google Cloud Spanner Quickstart

Module 5: Managed NoSQL Solutions
Understanding NoSQL Storage
Simplifying Structured Storage with Cloud Firestore and Datastore
Exercise: Google Cloud Datastore/Firestore Quickstart
Storing Massive Data Sets with Bigtable
Choosing between Firestore and Bigtable
Caching Data using Memorystore

Module 6: Big Data Processing and Analytics
Migrating Hadoop and Spark Jobs to Google Cloud Dataproc
Exercise: Creating Dataproc Clusters
Big Data Warehousing and Analytics with BigQuery
Denormalizing Data for Query Optimization in BigQuery
Exercise: Querying Data with BigQuery
Choosing Big Data Processing Strategies

Module 7: Data Processing Pipelines
Programming ETL Pipelines with Google Cloud Dataflow
Simplify Dataflow coding using Templates
Exercise: Google Cloud Dataflow
Designing Real-time Data Processing Systems
Leveraging Pub/Sub for Scalable, Asynchronous Messaging
Preparing Data for Analysis with Cloud DataPrep

Module 8: Visualization and Analytics
Manipulating and Analyzing Data with Cloud Datalab
Building Dashboards with Data Studio

Module 9: Machine Learning Fundamentals
Machine Learning Use Cases and Algorithms
Training and Evaluating Models
Feature Engineering
Analyzing Machine Learning Case Studies
Programming Models with TensorFlow
Exercise: Getting Started with TensorFlow
Serverless, NoOps Training with Google Cloud MLE
Exercise: GCP Machine Learning
Automating machine Learning with AutoML and BigQuery ML