Help with Extract -> Load process logic in a personal project
Hello! I'm working on a small data project using a NASA API with Python. The idea is to extract the orbital elements and close approach (asteroids, comets, etc) objects (and there orbital elements) of each planet in order to perform some physics simulations with the data.
I was planning to use a Cloudflare R2 bucket to store the raw API responses, then transform them into a PostgreSQL database, and use FastAPI to consume the data.
I'm not sure how to perform the extraction and loading processes. In theory, each planet will follow this path:
Fetch the orbit elements for the own planet (use API 1);
Fetch the close approach object (use API 2);
Fetch the orbit elements for each of the close approach objects (use API 1).
I use two APIs: one for the orbital elements and one for the close approaches. Should I fetch all the data, store it in memory, and then send it to the bucket? Or would it be better to load it right away with each API call? So, when fetching the close approach objects' orbit elements, get the JSON from the bucket and then use the first API and store that raw response in the bucket