Modern laptops and tools make local data processing surprisingly powerful
I wonder sometimes that our laptop hardware is getting so much better that sometimes it is easier to open the laptop, crunch the numbers, and save them if required. This workflow is much more hassle-free, cost effective and super fast compared to spinning up a Databricks notebook with a Spark cluster to explore a subset of my data.
I do understand that for daily jobs and large-scale processing, where a “heavy” amount of data needs to be processed, something like Databricks is super helpful.
Nothing is faster than local processing for small exploratory analysis, avoiding the whole network latency, cluster spin-up times, etc. Mind you small here can be upwards of 1 million records.
I wonder if the industry will start moving in this direction from everything cloud to a hybrid setup.
What’s your opinion?
