Parallel Computing with Python

This course is being taught at as part of Master For Smart Data Science ENSAI Rennes.

Run Jupyter notebooks with docker

Get docker app

• Mac

• Windows

• Linux

You can run these notebooks with Docker. The following command starts a container with the Notebook server listening for HTTP connections on port 8888 and 4040 without authentication configured.

git clone https://github.com/pnavaro/big-data.git
docker run --rm -v $PWD/big-data:/home/jovyan/ -p 8888:8888 -p 4040:4040 pnavaro/big-data

References

Books

• Python for Data Analysis by Wes McKinney.

• Python Data Science Handbook by Jake VanderPlas

Software documentation

• Pandas.

• Dask

• PySpark

• Apache Arrow

• Parquet

• GCSFS

• Dask.distributed

• fastparquet

Tutorials

• Python

  • Analyzing and Manipulating Data with Pandas Beginner: SciPy 2016 Tutorial by Jonathan Rocher.

• Dask

  • Dask Examples

  • Parallel Data Analysis with Dask Dask tutorial at PyCon 2018 by Tom Augspurger.

  • Parallelizing Scientific Python with Dask SciPy 2018 Tutorial by James Crist and Martin Durant

  • Parallelizing Scientific Python with Dask, SciPy 2017 Tutorial by James Crist.

  • Parallel Python: Analyzing Large Datasets Intermediate, SciPy 2016 Tutorial by Matthew Rocklin.

  • Parallel Data Analysis in Python, SciPy 2017 Tutorial by Matthew Rocklin, Ben Zaitlen & Aron Ahmadia.

  • Matthew Rocklin - Streaming Processing with Dask

  • Jacob Tomlinson - Dask Video Tutorial 2020

• Hadoop

  • Writing an Hadoop MapReduce Program in Python by Michael G. Noll.

• Spark

  • GetOting Started with Apache Spark Tutorial - Databricks

  • Hortonworks Data Tutorials

Blog posts

• Why Polars uses less memory than Pandas

• Reducing Pandas memory usage #1: lossless compression

• Reducing Pandas memory usage #2: lossy compression

• Reducing Pandas memory usage #3: Reading in chunks

• Don’t use Hadoop - your data isn’t that big

• Format Wars: From VHS and Beta to Avro and Parquet overview of Hadoop File formats.

• Should you replace Hadoop with your laptop? by Vicki Boykis.

• Implementing MapReduce with multiprocessing by Doug Hellmann.

• Deploying Dask on YARN by Jim Crist.

• Native Hadoop file system (HDFS) connectivity in Python by Wes McKinney.

• Working Notes from Matthew Rocklin (must read)

• Large SVDs with Dask

• Machine Learning – 7 astuces pour scaler Python sur de grands datasets

• The Best Format to Save Pandas Data

Online courses

• DataCamp Cheat Sheets

• Outils pour le Big Data by Pierre Nerzic. 🇫🇷

• wikistat - Ateliers Big Data by Philippe Besse. 🇫🇷

• Data Science and Big Data with Python by Steve Phelps.

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.