Tabular benchmark: from-scratch vs scikit-learn
Generated by projects/tabular_benchmark/src/tb_benchmark.py. Metrics are deterministic under seed 42; wall times vary slightly per run.
Setup
- Split: 80/20 train/test,
random_state=42, identical split for both implementations.
- Features standardized with train-fold mean/std (via
mlfp.standardize).
- Metrics computed with
ml_first_principles.metrics for both sides.
- Hyperparameters matched where the objectives are the same (alpha for ridge/lasso, depth/estimators for trees, k for KNN).
Datasets
- Diabetes (regression): 442 patients, 10 physiological features (age, sex, BMI, blood pressure, 6 serum measurements); target is a quantitative measure of disease progression one year after baseline.
- Breast cancer (Wisconsin) (classification): 569 tumors, 30 features computed from digitized images of fine-needle aspirates (radius, texture, concavity, ...); binary target (0 = malignant, 1 = benign).
Regression — Diabetes
| Model |
r2 (scratch) |
r2 (sklearn) |
rmse (scratch) |
rmse (sklearn) |
| LinearRegression |
0.4526 |
0.4526 |
53.8534 |
53.8534 |
| RidgeRegression |
0.4541 |
0.4541 |
53.7775 |
53.7775 |
| LassoRegression |
0.4669 |
0.4669 |
53.1472 |
53.1467 |
Classification — Breast cancer (Wisconsin)
| Model |
accuracy (scratch) |
accuracy (sklearn) |
f1 (scratch) |
f1 (sklearn) |
| LogisticRegression |
0.9825 |
0.9386 |
0.9859 |
0.9489 |
| DecisionTreeClassifier |
0.9386 |
0.9474 |
0.9504 |
0.9577 |
| RandomForestClassifier |
0.9649 |
0.9649 |
0.9722 |
0.9722 |
| KNeighborsClassifier |
0.9474 |
0.9474 |
0.9577 |
0.9577 |
| GaussianNB |
0.9649 |
0.9649 |
0.9722 |
0.9722 |
| LinearSVC |
0.9649 |
0.9561 |
0.9726 |
0.9645 |
Timing
| Task |
Model |
Fit scratch (ms) |
Fit sklearn (ms) |
Predict scratch (ms) |
Predict sklearn (ms) |
| regression |
LinearRegression |
0.37 |
1.01 |
0.11 |
0.16 |
| regression |
RidgeRegression |
0.11 |
0.76 |
0.01 |
0.11 |
| regression |
LassoRegression |
4.11 |
0.84 |
0.01 |
0.11 |
| classification |
LogisticRegression |
20.53 |
6.83 |
0.06 |
0.30 |
| classification |
DecisionTreeClassifier |
361.34 |
4.45 |
0.10 |
0.21 |
| classification |
RandomForestClassifier |
1257.35 |
33.72 |
3.40 |
1.40 |
| classification |
KNeighborsClassifier |
0.02 |
0.52 |
4.33 |
16.43 |
| classification |
GaussianNB |
0.40 |
1.51 |
0.10 |
0.31 |
| classification |
LinearSVC |
309.33 |
1.57 |
0.02 |
0.28 |
Findings
- Closed-form linear models match sklearn essentially exactly. LinearRegression (lstsq), RidgeRegression (same normal equations, same unpenalized-intercept centering) and LassoRegression (same \(\frac{1}{2n}\Vert r\Vert^2 + \alpha\Vert w\Vert_1\) objective via coordinate descent) agree with sklearn to 4 decimal places on both metrics.
- The logistic gap is early stopping in disguise. Both sides solve the same unregularized objective, yet scratch scores noticeably higher accuracy. Breast cancer is nearly linearly separable, so sklearn's lbfgs drives weights toward the diverging max-likelihood solution and overfits, while 1000 plain gradient-descent steps stop far short of convergence — an accidental implicit regularizer, not a better algorithm.
- Remaining classifiers land within a couple of test samples. KNN and GaussianNB are deterministic and agree with sklearn; the DecisionTree and the seeded RandomForest follow the same Gini/binary-split and bootstrap/feature-subset procedures but consume different RNG streams and break threshold ties differently, so predictions differ on 0-2 of the 114 test samples. LinearSVC differs by one sample: mini-batch subgradient descent on mean-hinge vs liblinear's coordinate solver on sum-hinge.
- Speed is where scratch honestly lags. The scratch tree grower evaluates candidate splits in Python loops, so tree/forest fitting is roughly 40-80x slower than sklearn's Cython implementation, and the mini-batch SVC loop is ~200x slower than liblinear. Closed-form linear models and vectorized KNN are as fast as (or faster than) sklearn at this dataset size, where sklearn's per-call overhead dominates.