Object Storage concepts

Object storage is a computer data storage that manages data as objects, as opposed to other storage architectures like file systems which manages data as a file hierarchy, and block storage which manages data as blocks within sectors and tracks. Each object typically includes the data itself, a variable amount of metadata, and a globally unique identifier. (Wikipedia)

But Why?

• Scalability. Scale-out, infinitly.
• Security/Reliability/Availability. Erasure Coding.
• Cost. Commodity hardware.
• Performance (bandwith). Just need good network, and scale.

POSIX vs Object Storage

Object store Architecture

Ceph @CNES

Quizz

What makes object storage efficient? (multiple choices)

• Answer A: The use of high-end and complex harware for storage solutions
• Answer B: The multiplication of standard servers with JBOD storage
• Answer C: High performance network
• Answer D: POSIX API
• Answer E: Dedicated API developed for simple transactions

Answer link Key: mz

Object storage interface and libraries

• Just HTTP Rest Calls: GET, PUT
• With authentification and authorizations: Access key and secret access key.
• Easier with AWS CLI or alike:

# Needs a keys file in your $HOME dir, or to dynamically obtain a key
aws s3 cp /tmp/foo/ s3://bucket/ --recursive --exclude "*" --include "*.jpg"

Interfaces and libraries for major programming languages:

• Boto3 for Python: not just S3, all AWS services
• Higher level for Python: s3fs: Pythonic file interface for S3
• Other languages:
  • Java SDK
  • C++ SDK
• Other Object storage interfaces:
  • Google Cloud Storage
  • Azure Blob Storage

Processing framework and Object store

All Major Data processing framework are compatible with S3 like interfaces.

• Pandas dataframes
• Spark
• Dask

Just replace / or file:// by s3://.

import dask.dataframe as dd
df = dd.read_csv('s3://bucket/path/to/data-*.csv')
df = dd.read_parquet('gcs://bucket/path/to/data-*.parq')

More on the next part of the course.

ARCO Data

Analysis Ready Cloud Optimized Data.

Thanks to Ryan Abernathey.

What is Analysis Ready?

• Think in Datasets, not data files
• No need for tedious homogenizing, cleaning steps
• Curated and cataloged

ARCO Data (2)

Analysis Ready Cloud Optimized Data.

Thanks to Ryan Abernathey.

Cloud Optimized Geotiff

Zarr

Parquet

• Chunked binary file
• Compressed
• Metadata easily accessible

See more

Quizz

Object storage is always performant for scientific data processing.

Answer link Key: qp