Classical Machine Learning

Why This Matters

AI-001 introduced the "Russian dolls" picture: AI contains Machine Learning, which contains Deep Learning. It is easy to read that nesting and quietly assume deep learning is simply the better version of ML that replaced everything older. It has not. Most of the world's transactional, tabular business data, loan applications, insurance claims, sensor readings, sales records, is still best modeled by the classical techniques in this lesson. A bank's credit-scoring model, a payment processor's real-time fraud detector, and the majority of Kaggle's tabular-data competition winners all reach for gradient boosting or a random forest before they reach for a neural network. Knowing when to pick classical ML over deep learning (AI-003) is a practical skill, not a historical footnote.

Everyday Analogy

Imagine two ways to predict tomorrow's weather. One approach: a veteran forecaster who has tracked pressure, humidity, and wind patterns for thirty years and can tell you, with a clear rule of thumb, exactly why they expect rain, "pressure's dropping fast and the wind shifted southeast, that combination has meant rain 80% of the time in my records." Another approach: a vast simulation that ingests millions of data points and outputs a probability with no simple explanation of why. Both can be accurate. But if you only have thirty years of local data (not millions of data points), and you need to explain the forecast to a regulator, the veteran forecaster's transparent rule of thumb is often the better tool, not because it's more advanced, but because it fits the data and the requirement. Classical ML is that veteran forecaster: simpler, transparent, and often exactly right-sized for the data most organizations actually have.

Regression: The Baseline Predictors

Linear regression predicts a continuous number from input features by fitting a straight line (or hyperplane, in multiple dimensions): output = w1×feature1 + w2×feature2 + ... + bias. AI-002's house-price example is exactly this: price predicted from size, bedrooms, and distance to center, each with a learned weight. It is the simplest possible model that still learns from data rather than following hand-written rules, and it remains a strong first baseline for any numeric-prediction problem because it is fast to train, easy to inspect (each weight tells you the direction and size of that feature's effect), and hard to overfit on small datasets.

Logistic regression adapts the same idea to a yes/no (or multi-class) prediction: instead of outputting a raw number, it outputs a probability between 0 and 1 by passing the weighted sum through a sigmoid function. Despite the name, it is a classification method, not a regression method, used for problems like "will this transaction default" or "is this email spam." It is one of the most widely deployed models in finance and healthcare specifically because its output is a calibrated probability with clearly interpretable weights, a regulator or auditor can be shown exactly which factors pushed a prediction toward "high risk" and by how much.

Decision Trees and Ensembles

A decision tree predicts by asking a sequence of yes/no questions about the input features, "Is income above $50,000? If yes, is credit history longer than 5 years?", branching until it reaches a final prediction at a leaf node. Trees are trained by repeatedly picking the question that best splits the data into more homogeneous groups. A single tree is easy to visualize and explain, but tends to overfit, memorizing quirks of the training data rather than learning generalizable patterns.

Ensemble methods fix this by combining many trees:

• Random forests train many decision trees, each on a random subset of the data and a random subset of features, then average their predictions. Averaging cancels out each individual tree's overfitting quirks, producing a far more robust model while keeping most of a single tree's interpretability (feature importance scores show which inputs mattered most across the forest).
• Gradient boosting builds trees sequentially rather than independently: each new tree is trained specifically to correct the errors the previous trees made, gradually reducing the overall prediction error. XGBoost, along with LightGBM and CatBoost, are the dominant gradient boosting implementations in industry, prized for consistently winning tabular-data competitions and for training and predicting far faster than a comparable neural network on the same structured data.

The practical difference: random forests are simpler to tune and more resistant to overfitting out of the box; gradient boosting typically achieves higher accuracy with careful tuning, which is why it, rather than random forests, dominates competitive leaderboards.

Clustering: Learning Without Labels

Everything above is supervised learning (AI-002): every training example comes with a known correct answer. k-means clustering is the classical example of unsupervised learning: given unlabeled data, group similar points together with no "correct" group labels provided at all.

K-means works by picking k initial group centers, assigning every data point to its nearest center, recomputing each center as the average of its assigned points, and repeating until the assignments stop changing. The result is k clusters of similar data points, useful for problems like customer segmentation (grouping shoppers by purchasing behavior without predefined categories) or anomaly detection (points that don't fit well into any cluster are flagged as unusual). The number of clusters, k, is chosen by the practitioner, often by trying several values and looking for the point where adding more clusters stops meaningfully improving the grouping (the "elbow method").

When Classical ML Still Beats Deep Learning

Deep learning's advantage is learning its own feature representations from raw, unstructured data, images, audio, free text, where hand-engineering features is impractical (AI-003). That advantage mostly evaporates on structured, tabular data, and several forces then favor classical ML instead:

• Small data. Deep learning needs large datasets to learn reliable patterns without overfitting; gradient boosting and random forests routinely outperform neural networks on datasets with only thousands, or even hundreds, of rows, exactly the size of data most business problems actually provide.
• Interpretability requirements. Regulated industries (lending, insurance, healthcare) often require explaining why a model made a decision. Logistic regression's weights and a random forest's feature importances are directly inspectable; a neural network's millions of weighted connections are not, without extra explainability tooling layered on top.
• Low latency and low compute. A gradient boosting model can score a transaction in microseconds on a single CPU core; a comparable neural network typically needs more compute and, often, a GPU to hit the same latency, a real cost difference at the scale of a payment processor scoring every transaction in real time.
• Feature engineering already exists. When a business already has well-understood, hand-crafted features (income, credit history length, transaction frequency), the main advantage of deep learning, automatic feature learning from raw data, has less to contribute.

The standard practical guidance: start with gradient boosting or logistic regression on tabular data, reach for deep learning when the input is unstructured (images, audio, long text) or the dataset is large enough to need it.

Real-World Showcase

• Credit scoring models at banks and lenders overwhelmingly use logistic regression or gradient boosting, not deep learning, both for interpretability under lending regulations and because the underlying data (income, debt, payment history) is already structured and tabular.
• Kaggle tabular-data competitions are dominated by XGBoost, LightGBM, and CatBoost year after year; a review of top-placing solutions across Kaggle's structured-data competitions consistently shows gradient boosting variants outperforming deep learning approaches on the same tabular datasets.
• Real-time fraud detection systems at payment processors like Visa and Mastercard rely heavily on gradient boosting and ensemble models, chosen specifically because they can score a transaction in the milliseconds available before a card is approved or declined, a latency budget that heavier deep learning models struggle to meet at that scale.

Try It Yourself

1. Take a prediction problem you understand (predicting a house price, predicting whether a customer churns) and sketch what features a linear or logistic regression would use. Which features would have obviously positive weights, and which negative?
2. Compare a single decision tree to a random forest on paper: why would averaging many overfit trees produce a model that generalizes better than any one of them alone?
3. Pick a business problem with only a few hundred labeled examples available. Argue, using this lesson's tradeoffs, why gradient boosting would likely outperform a deep neural network trained from scratch on that same small dataset.

Common Mistakes

• Assuming deep learning is always the more advanced, and therefore better, choice, regardless of data size or structure. On small tabular datasets, it frequently underperforms simpler classical models.
• Using k-means clustering and expecting it to produce meaningful groups without first scaling or normalizing features, since k-means measures distance directly, features on very different numeric scales can dominate the clustering for no meaningful reason.
• Treating a single decision tree as production-ready. Its tendency to overfit means the ensemble version (random forest or gradient boosting) is almost always the right choice over one lone tree.
• Choosing a model based on hype rather than the interpretability requirements of the domain, deploying an uninterpretable model in lending or healthcare when regulation requires an explainable one.
• Ignoring classical ML baselines entirely and jumping straight to a neural network, then being surprised when a well-tuned gradient boosting model matches or beats it with a fraction of the training time and compute.

Key Takeaways

• Linear and logistic regression remain the simplest, most interpretable baselines for numeric and categorical prediction, and are still deployed widely in regulated industries.
• Decision trees are easy to understand but prone to overfitting; random forests and gradient boosting (XGBoost, LightGBM, CatBoost) fix this by combining many trees.
• K-means clustering is the classical unsupervised technique for grouping unlabeled data, distinct from the supervised methods above.
• Classical ML wins over deep learning (AI-003) on small tabular datasets, when interpretability is required, and when latency or compute budgets are tight.
• Credit scoring, fraud detection, and most Kaggle tabular-data competitions are real-world proof that classical ML is not a historical footnote, it is still the right default tool for structured data.