u/PretendGrimes

I'm doing a round of interviews with a bunch of different companies, but I was really unaware of what the market range for the position would be (I'm a DE in LATAM looking for nearshoring jobs). I realized that I was setting a low bar in many of them (I realized that I could ask for 5,500 ~ 6,000 and wouldn't get pushback on it)

Now there's a very promising position that I hope to get where, in the HR interview, I asked for 5,200 and the recruiter basically nodded so I assume that that was okay.
If I do make it through and receive an offer from them, is it okay for me to try to get to soemthing higher, like 6,000? If so, how do I frame that?

I thought about saying that I received a counter proposal from my current job or a new offer, but I don't know if they would ask for proof of that

I work for a robotics company and in my job as a data engineer, we basicailly handle IoT data that comes from MySQL dbs in every robot unit. The data amount itself is pretty low, most table don't get past 30GB, we wouldn't even need to have that in the cloud (but try telling that to my boss)

The data is picked up by Kinesis Data Streams and processed by Kinesis Firehose, which lands them in a S3 raw bucket as JSON files, not queryable on that format.

This data in then queued by an SQS and processed by a lambda (mostly pandas code that does deduplication, schema validation and add some new columns) every 15 minutes and loads it to what I would call a silver layer (although I'm not really quite sure if that would be it).

Then we have another lambda that basically runs a bunch of merge statements from this silver layer into a gold layer (it does some joins, parses a few JSON fields, convert a few columns)

All of that is within S3 Tables (when I arrived we had TONS of metadata overbloat, which was racking up too much cost, S3 Tables kinda sorted that for me but I'm already pissed at how badly integrated is with most of the other AWS services)

I also don't think the current way we handle schema versioning is ideal (we have a folder for every table in the silver layer and every time we change something in the schema we add a new JSON file to it, then the lambda function that runs for Silver always picks up the latest schema version)

I few like this is not an ideal framework for modern data engineering, how can I improve all of that? I'm not a senior engineer yet so I feel like I still lack some of the grasp of what the industry standards are (and my teams is literally only me so there's no one else to help on that)