Links:

• Preprint: https://doi.org/10.5281/zenodo.22028087
• GitHub (R Function and Simulation): https://github.com/tjleestjohn/Entropic-Scree

TL;DR: I am sharing an open-source, non-parametric, and model-agnostic information-theoretic diagnostic framework for estimating the intrinsic rank of a dataset that is more broadly applicable than methods bound by strict generative assumptions (like standard PCA) or spatial distance metrics (like Kernel PCA and TWO-NN) when evaluating high-dimensional, mixed-type tabular datasets. It uses Normalized Mutual Information to compress non-linear expansions back towards their true generative roots. This avoids the dimensional inflation that can occur within strictly linear matrices, while also mitigating the structural instability that affects RKHS and Euclidean metric estimators when generative roots are entangled or sparse.

Where Standard Baselines Encounter Limits in Complex Data

When dealing with mixed data types, heavy non-linearities, or sparse environments ($m > N$), established baselines encounter structural limitations:

• Linear Estimators (PCA): Standard PCA and classical factor models evaluate linear variance. They tend to separate non-linear dependencies (like polynomials or interactions) into "Spurious Orthogonal Dimensions," which can lead to an overestimation of the true rank. Furthermore, in sparse regimes, PCA is mathematically capped, restricting the maximum extractable rank to $N-1$.
• Spatial Metrics (Kernel PCA & Euclidean): Kernel PCA (RBF) can lose its structural elbow when generative roots are entangled, as its infinite-dimensional space lacks a finite-sample boundary to cleanly separate signal from combinatorial noise. Similarly, topological estimators (like TWO-NN) can become unreliable because high-dimensional distance concentration affects local neighborhood calculations across mixed-data margins.

The Information-Theoretic Approach: The Entropic Scree

This framework evaluates pure probability mass rather than linear variance or physical distance.

• The Metric Space: It constructs a pairwise matrix using Information-Theoretic Jaccard Similarity (Variation of Information). Because it relies on Shannon entropy, it’s invariant to marginal shape mismatches (e.g., correlating a continuous wave with a discrete binary flag).
• Bypassing the Rank Ceiling: By executing a double-centering transformation (cMDS), it moves the data into a topological information space, explicitly bypassing the $N-1$ sample-size ceiling.
• Compressing the Manifold: The algorithm acts as a bivariate filter, inherently compressing the probability mass of non-linear combinations back towards their Intrinsic Generative Rank.

The Factor Analysis Connection

If you are used to interpreting standard EFA or PCA factor loadings, the Entropic Scree requires a fundamental shift in how you read the extracted axes:

• Bipolar Modules = Estrangement, not Inverse Correlation: In classical linear frameworks, variables loading on opposite ends of a factor are inversely correlated. In the Entropic Scree, because the matrix evaluates shared information, a negative inner product indicates structural estrangement. Variables loading with opposite signs share zero mutual information. The eigensolver anchors its poles into completely decoupled sub-networks, allowing you to cleanly untangle independent variable clusters directly from the primary axes.
• Informational Gravity (AIG/FSIG): Instead of calculating traditional "percentage of variance explained," the framework introduces Average and Factor-Specific Informational Gravity. These metrics translate abstract spectral variance into actionable, "variable-equivalent" footprints, mapping exactly how much topological weight each underlying factor controls.

I'd love to hear from this community. How are you handling intrinsic rank extraction and factor analysis in heavily non-linear, sample-starved environments? Feel free to pull the repo, test it out on your own messy tabular data, and let me know what you think.

Full arXiv Preprint: https://arxiv.org/abs/2603.12288

Paper Simulation Github: https://github.com/tjleestjohn/from-garbage-to-gold

Hi r/artificial,

It's a dirty little secret to many of us... sometimes, downstream AI/ML models perform surprisingly well when you just hand them raw, error-prone tabular data instead of heavily curated feature sets. Despite this, the vast majority of our field tends to be fiercely loyal to "Garbage In, Garbage Out" (GIGO). While automated ETL pipelines are absolutely essential for structuring data, our workflows are still bottlenecked with endless manual cleaning and aggressive imputation just to curate pristine, error-free tables.

My co-authors and I recently released a preprint on arXiv (From Garbage to Gold) arguing that treating GIGO as a universal law can sometimes be a trap... especially in the context of big data (many columns). That the bottleneck due to manual data cleaning can actively lower the predictive ceiling of our models when latent causes drive the system's behavior.

To be clear upfront: we are not arguing against ETL. Parsing JSON, handling schema evolution, and standardizing types is non-negotiable.

What we are arguing against is the universal assumption that "clean" data (via manual data scrubbing and aggressive imputation) is non-negotiable for big data predictive AI/ML modeling.

Here is why the traditional mindset can be limiting:

1. We conflate two different types of "noise" (Predictor Error and Structural Uncertainty).

Usually, we just lump all noise into one big bucket. But if you split that noise into two specific categories, the math changes completely:

• Predictor Error: Random typos, dropped logs, or transient glitches.
• Structural Uncertainty: The inherent, unresolvable gap between recorded metrics and the complex, hidden reality they represent.

We spend months manually scrubbing data because the threat of data errors is obvious, while Structural Uncertainty is often an afterthought at best. However, when latent causes drive a system, manual scrubbing fixes noise due to errors, but it fundamentally cannot fix the noise due to Structural Uncertainty.

On the other hand, the paper shows that in this context, if you use a comprehensive, high-dimensional data architecture, a flexible model can actually triangulate the hidden drivers reliably despite the presence of data errors. When keeping a massive amount of messy, highly correlated variables (even if error-prone), the sheer volume of redundant signals allows the model to drown out individual errors (bypassing the cleaning bottleneck) and simultaneously overcome Structural Uncertainty.

This redefines "data quality." It's not only about how accurately the variables are measured. It's also about how the portfolio of variables comprehensively and redundantly covers the latent drivers of the system.

2. Manual cleaning is a bottleneck on dimensionality (The Practical Problem).

To overcome Structural Uncertainty, modern AI/ML models want to find the underlying latent drivers of a system (think Representation Learning but with tabular data). To do this, however, they need a high-dimensional set of variables that contains Informative Collinearity in order to mathematically triangulate the hidden drivers.

The moment you introduce manual cleaning, you create a human bottleneck. Because we cannot manually clean 10,000 variables, we are forced to drop 9,900 of them. By artificially restricting the predictor space to make it "clean enough to model," we can harm the data architecture's inherent potential to triangulate those latent drivers. We sacrifice the model's actual predictive ceiling just to satisfy the GIGO heuristic.

Ultimately, this suggests we should focus mostly on extracting, loading, and increasing observational fidelity with automated tools, but that, in contexts characterized by latent drivers, we should stop letting manual cleaning bottlenecks restrict the scale of our AI/ML models.

Thoughts?: Have you run into situations where your data science teams actually got better predictive results by bypassing the manually cleaned tables and pulling massive dimensionality straight from the raw ELT layers?

I'd love to hear your experiences or thoughts. Happy to discuss all serious comments or questions.

Full disclosure: the preprint is a 120-page beast. It’s long because it doesn't just pitch the core theory with a qualitative argument. It gives the full mathematical treatment to everything which takes space. We also dig into edge cases, what happens when assumptions like Local Independence are violated (e.g., systematic errors exist), broader implications (like a link to Benign Overfitting and efficient feature selection strategies that make this high-d strategy practical with finite compute), a deep-dive simulation, failure modes, and a huge agenda for future research (because we do not claim the paper is the final word on the matter).

It's a major commitment upfront but may save you time and money in the long term, while also enhancing the predictive ceiling of your tabular AI/ML models.

Full Paper: https://arxiv.org/abs/2603.12288

Paper Simulation Github: https://github.com/tjleestjohn/from-garbage-to-gold

Hi r/dataengineering,

It's an open secret to many of us... sometimes, downstream ML models perform surprisingly well when you just hand them raw, error-prone data instead of heavily curated feature sets. Despite this, our field is fiercely loyal to "Garbage In, Garbage Out" (GIGO). While automated ETL pipelines are absolutely essential for structuring data and increasing observational fidelity, we still bottleneck our workflows with endless manual cleaning and aggressive imputation just to curate pristine, error-free tables.

My co-authors and I recently released a preprint (From Garbage to Gold) arguing that treating GIGO as a universal law can sometimes be a trap... especially in the context of big data (many columns). That manual cleaning can actively lower the predictive ceiling of our models when latent causes drive the system's behavior.

To be clear upfront: we are not arguing against ETL. Parsing JSON, handling schema evolution, and standardizing types is non-negotiable.

What we are arguing against is the universal assumption that "clean" data (via manual data scrubbing and aggressive imputation) is non-negotiable for big data predictive ML modeling.

Here is why the traditional mindset can be limiting:

1. We conflate two different types of "noise" (Predictor Error and Structural Uncertainty).

Usually, we just lump all noise into one big bucket. But if you split that noise into two specific categories, the math changes completely:

• Predictor Error: Random typos, dropped logs, or transient glitches.
• Structural Uncertainty: The inherent, unresolvable gap between recorded metrics and the complex, hidden reality they represent.

We spend months manually scrubbing data because we treat all "bad data" as a single enemy. However, when latent causes drive a system, manual scrubbing fixes Predictor Error, but it fundamentally cannot fix the Structural Uncertainty inherent to the fixed predictor set.

On the other hand, the paper shows that in this context, if you use a comprehensive, high-dimensional data architecture, a flexible model can actually triangulate the hidden drivers reliably. When keeping a massive amount of messy, highly correlated variables (even if error-prone), the sheer volume of redundant signals allows the model to drown out individual errors (bypassing cleaning) and simultaneously overcome Structural Uncertainty.

This redefines "data quality." It's not only about how accurately the variables are measured. It's also about how the portfolio of variables comprehensively and redundantly covers the latent drivers of the system.

2. Manual cleaning is a bottleneck on dimensionality (The Practical Problem).

To overcome Structural Uncertainty, modern ML models want to find the underlying latent drivers of a system (think Representation Learning but with tabular data). To do this, they need a high-dimensional set of variables that contains Informative Collinearity in order to mathematically triangulate the hidden drivers.

The moment you introduce manual cleaning, you create a human bottleneck. Because we cannot manually clean 10,000 variables, we are forced to drop 9,900 of them. By artificially restricting the predictor space to make it "clean enough to model," we harm the data architecture's inherent potential to triangulate those latent drivers. We sacrifice the model's actual predictive ceiling just to satisfy the GIGO heuristic.

Ultimately, this suggests DEs should focus mostly on extracting, loading, and increasing observational fidelity with automated tools, but that, in contexts characterized by latent drivers, we should stop letting manual cleaning bottlenecks restrict the scale of our ML models.

Thoughts?: Have you run into situations where your data science teams actually got better predictive results by bypassing the manually cleaned tables and pulling massive dimensionality straight from the raw ELT layers?

I'd love to hear your experiences or thoughts. Happy to discuss all serious comments or questions.

Full disclosure: the preprint is a 120-page beast. It’s long because it doesn't just pitch the core theory with a qualitative argument. It gives the full mathematical treatment to everything which takes space. We also dig into edge cases, what happens when assumptions like Local Independence are violated (e.g., systematic errors exist), broader implications (like a link to Benign Overfitting and efficient feature selection strategies that make this high-d strategy practical with finite compute), a deep-dive simulation, failure modes, and a huge agenda for future research (because we do not claim the paper is the final word on the matter). It's a major commitment upfront but may save you time long term in practice.

Full Paper: https://arxiv.org/abs/2603.12288

Hi r/analytics,

"Garbage In, Garbage Out" is a deeply entrenched mindset. We spend up to 80% of our time cleaning tabular data because GIGO is obviously true. But... what if this idea is sometimes holding our models back?

It's not unheard of. I'm sure many of you have noticed your models sometimes perform surprisingly well on raw, uncurated data.

To help explain this, my co-authors and I recently released a preprint called From Garbage to Gold (G2G) that basically says that sometimes GIGO is wrong. The paper discusses when and why error-prone data can actually be used to create SOTA prediction models.

In the context of big data driven by latent causes, it turns out that aggressively cleaning your data can actually blind your models to the exact signals they need to see.

The core of the paper is about how we define "noisy" data. Usually, we just lump all noise into one big bucket. But if you split that noise into two specific categories, the math changes completely:

• Category 1: Predictor Error. This is the classic garbage. Typos, sensor glitches, reporting delays, or just weird recording errors.
• Category 2: Structural Uncertainty. This is the inherent, probabilistic gap between a predictor and the actual hidden force driving the system. Basically, even a "perfectly" measured variable is still just a limited, imperfect proxy for reality.

Here’s the catch: traditional cleaning only fixes Category 1. You can spend six months making a dataset "flawless," but your model is still going to hit a performance ceiling because you did nothing to solve for Category 2.

Our paper shows that if you use a broad, high-dimensional architecture, a flexible model can actually triangulate the hidden truth. That when keeping a massive amount of messy, highly correlated variables (even if error-prone), the sheer volume of redundant signals allows the model to drown out individual errors (bypassing cleaning) and simultaneously overcome Structural Uncertainty.

Ultimately, this redefines "data quality." It's not only about how accurate the variables are measured. It's also about the how the portfolio of variables comprehensively and redundantly covers the latent drivers of the system.

Full disclosure: the preprint is a 120-page beast. It’s long because it doesn't just pitch the core theory. It gives the full mathematical treatment to everything which takes space. We also dig into edge cases, what happens when assumptions like Local Independence are violated, broader implications (like a link to Benign Overfitting and efficient feature selection strategies), a deep-dive simulation, failure modes, and a huge agenda for future research (because we do not claim the paper is the final word on the matter).

Would love to get your thoughts on this.

Happy to discuss or answer any serious questions.