u/LtLfTp12

I plan on building a pipeline that ingests 2d grid data (lat,long,value) thats updated frequently but I have zero clue on how exactly to store it, as I would have 4 columns essentially (timestamp, lat, long, value). So unique key per row would be combination of time/lat/long.

Would columnar format still be best? It wont be large, coordinates are integer values, data is updated every \~5 min and its provided in a json format.

My current pipeline ingests normal timeseries data, saves raw in S3 and transformed in Postgres, but I’m not sure if it’s best to have this data treated the same. Was thinking maybe a different file format and keep solely in S3? The values doesn’t need any processing

Ultimate goal is to in the frontend build a live, last n-days animation of the globe to see how the values change

I plan on building a pipeline that ingests 2d grid data (lat,long,value) thats updated frequently but I have zero clue on how exactly to store it, as I would have 4 columns essentially (timestamp, lat, long, value). So unique key per row would be combination of time/lat/long.

Would columnar format still be best? It wont be large, coordinates are integer values, data is updated every ~5 min and its provided in a json format.

My current pipeline ingests normal timeseries data, saves raw in S3 and transformed in Postgres, but I’m not sure if it’s best to have this data treated the same. Was thinking maybe a different file format and keep solely in S3? The values doesn’t need any processing

Ultimate goal is to in the frontend build a live, last n-days animation of the globe to see how the values change

I’m looking for historical solar wind data for a personal project I’m working on. I’m looking for data in minute increments timestamped to L1 Lagrange point ideally from beginning of 2025. I’ve been collecting rtsw data from NOAA SWPC for a month now but would like data I can attach before it.

However finding something like this has not been easy…

OMNIWEB - minute increments available but time shifted to Earth bow shock

NCEI - minute increments but no data from around July/August 2025

CDAWEB - not in minute increments

For NCEI supposedly there was a problem with DSCOVR but these endpoints are still serving DSCOVR data. Then again it’s tagged as inactive.

Any help would be greatly appreciated.