Dataproc Cookbook : Running Spark and Hadoop Workloads in Google Cloud

Get up to speed with Dataproc, the fully managed and highly scalable service for running open source big data tools and frameworks, including Hadoop, Spark, Flink, and Presto.

This cookbook shows data engineers, data scientists, data analysts, and cloud architects how to use Dataproc, integrated with Google Cloud, for data lake modernization, ETL, and secure data science at a fraction of the cost. Narasimha Sadineni from Google and former Googler Anu Venkataraman show you how to set up and run Hadoop and Spark jobs on Dataproc.

You'll learn how to create Dataproc clusters and run data engineering and data science workloads in long-running, ephemeral, and serverless ways.

In the process, you'll gain an understanding of Dataproc, orchestration, logging and monitoring, Spark History Server, and migration patterns. This cookbook includes hands-on examples for configuring, logging, securing clusters, and migrating from on-prem to Dataproc.

You'll learn how to:Create Dataproc clusters on Compute Engine and Kubernetes EngineRun data science workloads on DataprocExecute Spark jobs on Dataproc ServerlessOptimize Dataproc clusters to be cost effective and performantMonitor Spark jobs in various waysOrchestrate various workloads and activitiesUse different methods for migrating data and workloads from existing Hadoop clusters to Dataproc