Course Overview

• Big Data Processing & Distribution

Harnessing the complexity of large amounts of data is a challenge in itself.

But Big Data processing is more than that: originally characterized by the 3 Vs of Volume, Velocity and Variety, the concepts popularized by Hadoop and Google require dedicated computing and storage solutions (both software and infrastructure), which will be explored in this module.

Objectives

By the end of this module, participants will be able to:

• Understand the differences and usage between main distributed computing architectures (HPC, Big Data, Cloud, CPU vs GPGPU)
• Implement the distribution of simple operations via the Map/Reduce principle in PySpark
• Understand the principle of Kubernetes and object storage
• Deploy a Big Data Processing Platform (Pangeo) on the Cloud
• Either implement the distribution of data wrangling/cleaning and training machine learning algorithms using Dask
• Or implement a NDVI processing at scale using Pangeo platform.

About myself

• Guillaume Eynard-Bontemps
• CNES (Centre National d’Etudes Spatiales - French Space Agency)
• Since 2016:
  • 6 years in CNES Computing Center team
  • Technical lead of the team
  • 3 years of using Dask/Python on HPC and in the Cloud
  • A bit of Kubernetes and Google Cloud
• Before that: 5 years on Hadoop and Spark
• Originally: Software developer (a lot of Java)

First quizz

I’ll try to propose some quizz to be sure you’re following!

Let’s try it.

What is this course module main subject?

• Answer A: Cloud computing
• Answer B: Data Distribution & Processing
• Answer C: Machine Learning

Answer link Key: nf

Big Data & Distributed Computing (3.5h)

• Introduction to Big Data and its ecosystem (45min)
  • What is Big Data?
  • Legacy “Big Data” ecosystem
  • Big Data use cases
  • Big Data to Machine Learning
• Big Data platforms, Hadoop, HPC & Beyond (1h45min)
  • Hadoop, HDFS and MapReduce
  • Datalakes, Data Pipelines
  • HPC Platforms Architecture
  • From HPC to Big Data to Cloud and High Performance Data Analytics
  • BI vs Big Data
  • Hadoop legacy: Spark, Dask, Object Storage …
• Spark
  • Spark Introduction (30min)
  • Exercise: Play with MapReduce through Spark (30min)

Cloud, Kubernetes & Object stores (3.5h)

• The Cloud in 30 minutes (30min)
  • What’s the Cloud, Virtualization
  • Cloud history, layers, Engines
  • Usage revolution and new Data processing standard
• Containers Orchestration (1h)
  • Container orchestration and Kubernetes
  • Processing Platform over Kubernetes (Databricks, Coiled)
• Object storage (30min)
  • Object storage concepts: POSIX vs Object
  • Object storage architecture
  • Processing tools interfaces, Cloud Optimized files
• Deploy a Data processing platform on the Cloud based on Kubernetes and Dask (1.5h)
  • Exercise: DaskHub or Dask Kubernetes or Pangeo

Python ecosystem for data processing and Dask at scale (3.5h)

• The rise of the Python ecosystem for Data Processing (1h)
  • Data Science programming languages
  • Pydata stack (Pandas, Numpy, Matlplotlib, Jupyter)
  • Distributed and sctientific processing (Dask, PySpark)
  • Data vizualisation (Seaborn, Plotly, Pyviz)
  • Machine and Deep Learning (Sickit Learn, TensorFlow, Pytorch)
  • Jupyter notebooks, Binder, Google Colab
• Dask Presentation (30min)
• Dask/Pangeo practice:
  • Either play with Pangeo notebooks (in the cloud or on HAL cluster
  • Or try to answer Supaero evaluation notebook!

Quizz

What will we do today (multiple choices)?

• Answer A: Explain what’s Big Data
• Answer B: See what’s Hadoop
• Answer C: Dig into Cloud computing
• Answer D: Take a nap
• Answer E: Some Spark

Answer link Key: dy