Should I just work in prod?
EDIT: I don't actually mean I'll work in prod, you don't need to ward me off of it. I'm just looking for different ways to develop so I don't duplicate efforts and raise costs. Probably not the best title to give it, but I was in a rush
Hello,
I'm developing a Data Warehouse project using Azure (Azure Data Factory, Azure SQL Server, VNets, VMs, Bastion, etc. etc.). I'm handling PHI in the Data warehouse, which is why there's so much infrastructure.
I basically plan to install a SHIR for the source (it's an on-premises server), link it to ADF, set up a VNet and use private endpoints for the connection from ADF to the actual data warehouse (Azure SQL Database). Since the database is closed from public connections, and I can't just 'whitelist my IP', I wanted to set up a Virtual Machine on the same VNet, get into it via Bastion, and do the necessary developing/querying there.
I also want to do this all in Bicep, using one version-controlled project so that I can use CI/CD and also not lose my head looking for files since I'm working solo.
The plan is good and all, but because we're cost conscious, I'm using my Azure free account to develop a smaller version to test and break before working on the official pay-as-you-go company account. I'm using different data and my own laptop.
However, the Azure free restrictions are so intense that I had to wire up extra modules and infrastructure just to get things working. I needed to set up another VNet in a different region because that's the only place SQL server and SQL database were available, I had to set up peering, and ultimately scrapped working with the VM and Bastion at all in dev because there were no available VM SKUs. On top of that, since I want to develop with DACPAC's and a sqlproj in my version controlled project, I have to use a mock database. ATP, my project doesn't even look like the structure of what I want prod to be anymore.
I feel like I'm literally chipping away at a different problem and not working on my actual project assigned to me. I don't even know how my dev and prod branches will look in practice with so intensely discordant environments. But I need to prioritize security, costs, and keeping things neat and organized for myself. I need a second opinion on this because it doesn't feel right.