u/WiseWeird6306

Hi,

Why are executives thinking, if enough context is given, AI tools like Claude will solve/build/guide everything?

In my organization, my data architecture was not build on DE fundamentals and we have accumulated a shit load of technical debt. We are told to build context (which we should any ways) and feed everything to Claude to solve it. While I agree to use for faster fixes at logic level it cannot help design or think in right way how to architect workspaces and warehouse and model the tables. It cannot be a startegist and decide the data strategy for the company. Am I thinking it the wrong way? I feel very disappointed that data professionals advice is not being heard, instead that we are being advised to put everything on Claude. Is this ok? I advice, view points.

Hi everyone,

Here's the context: we have lot of bad data and data mismatch between em what we get and report and what we see from external reporting.

The communication goes on between stakeholders, us and Dev team in another country. We are a mid size company.

We have been hammered with lot of data inconsistency questions from C suit and they expect resolution or answers from 2 days to a week. For some questions I feel like that is reasonable but for questions that will involve business definition and data process capture change it seems we need more time? Is that a reasonable ask for time? how long does it take you guys to resolve data mismatch issues and deliver it?

What is a reasonable resolution time?

given that we have very poor documentation. Fact stays in someone's head or lost tickets.

Data issues are mostly like- why aren't the number matching- mostly cause the process changed or we are not considering a data point that should be flowing in.

I wanted to understand/get views on how do data analyst/analytics engineers/data engineers take out time to experiment/build and test things while you are always on fire fighting mode solving existing data issues and flawed/shabby medallion structures, tables and reports?

When rebuilding old reports again in sematic model, how you guys think about following start schema or snowflake schema?

I feel like as business builds up we lose track of schemas and get whatever we want in table. But it becomes messy and not manageable in long run.

Is it possible to strictly follow star or snowflake schema? How?

Also if we are building tables from self service perspective such that 1 table has most info and end user just needs to do , select * from table A, that means we are building 1 large denormalized table. Is that a good practice? Or should we build a report with 4-5 well built tables?

While denormalized tables will be good for self service and quick edit, I think it might lead to two problems:

a) too many tables for each report

b) if the report demands aggregated and non aggregated data, one denormalized aggregated table won't be helpful. For example if you have a huge transaction table then you aggregate it at Business level but then you realize you need some data at transactions level then one denormalized table is not good enough. What are your thoughts

When rebuilding old reports again in sematic model, how you guys think about following start schema or snowflake schema?

I feel like as business builds up we lose track of schemas and get whatever we want in table. But it becomes messy and not manageable in long run.

Is it possible to strictly follow star or snowflake schema? How?

Also if we are building tables from self service perspective such that 1 table has most info and end user just needs to do , select * from table A, that means we are building 1 large denormalized table. Is that a good practice? Or should we build a report with 4-5 well built tables?

While denormalized tables will be good for self service and quick edit, I think it might lead to two problems:

a) too many tables for each report

b) if the report demands aggregated and non aggregated data, one denormalized aggregated table won't be helpful. For example if you have a huge transaction table then you aggregate it at Business level but then you realize you need some data at transactions level then one denormalized table is not good enough. What are your thoughts

Can someone explain me how to understand the difference between them?

What I know-

1. Primary key is a column or set of columns that uniquely identifies each row. It may or may not have a business meaning

2. Grain of the table - one row or line item describing what it is, like one row per daily customer session

3. Group by- we use this to get one line item per item of that group. For example something grouped by business type and country, will get me data for unique combination of business type and country

Now I need clarification here-

A primary key should ALWAYS be in a group by statement in SQL or not, if it is needed in the output - True?

A column in group by is not necessary a primary key -True?

Columns defining the grain of the column consists of primary key and other cols (what is the nature of these other cols?)

I am asking these cause while aggregating data I am not sure if I should group all the cols, like sometimes you bring a col whose info you need but aggregating by it will repeat data. Some people say to me to aggregate data by primary key only but what if I have more cols other than primary key. Please correct me if you find flaws in my statements/concept/scenarios.

Can someone explain me how to understand the difference between them?

What I know-

1. Primary key is a column or set of columns that uniquely identifies each row. It may or may not have a business meaning

2. Grain of the table - one row or line item describing what it is, like one row per daily customer session

3. Group by- we use this to get one line item per item of that group. For example something grouped by business type and country, will get me data for unique combination of business type and country

Now I need clarification here-

A primary key should ALWAYS be in a group by statement in SQL or not, if it is needed in the output - True?

A column in group by is not necessary a primary key -True?

Columns defining the grain of the column consists of primary key and other cols (what is the nature of these other cols?)

I am asking these cause while aggregating data I am not sure if I should group all the cols, like sometimes you bring a col whose info you need but aggregating by it will repeat data. Some people say to me to aggregate data by primary key only but what if I have more cols other than primary key. Please correct me if you find flaws in my statements/concept/scenarios.