Spark Structured Streaming Realtime Mode with Stream to Batch-Batch dataframe join?
Trying to implement a Realtime-valid streaming job, but got an error STREAMING_REAL_TIME_MODE.BATCH_BATCH_JOIN_NOT_SUPPORTED.
Pretty sure that this is because I'm using a view (that joins 2 tables) as a static part of the stream-static join, but I can't find any references to this error OR that there's a limitation (either for structured Streaming or the realtime mode in particular) that says "you can't join a stream to a pre-joined dataframe".
Does anyone know where can I read more about this?
And on the other side of the question, I get the restriction is probably due to sheer complexity of having to manage (potentially) multiple incoming records to the joined batch, but I still didn't see anything mentioned about this anywhere.
Edit: it's definitely a limitation of the Realtime mode specifically