Skip to content
All projects

Project 09 // Machine learning · Oct 2026

Can a Model Nowcast Census Vacancy from Mail Data?

An honest evaluation: in forward-chaining, the test that matches real use, no model beat last year’s number.

Reusing last year’s Census vacancy rate gave the lowest error (1.023 points) among the models compared, ahead of ridge (1.054). Its 80% intervals cover 81% overall but only 57% in Champaign.
MAE: last year vs ridge, gap not distinguishable
1.02 vs 1.05
Lowest error, in points (last year’s value)
1.02
Pooled interval coverage
81%
Coverage in Champaign
57%

In short

The question

Can a model that combines last year’s Census vacancy rate with this year’s USPS movement predict this year’s Census rate, and does the answer hold in every county?

What I built

  1. 01
    Census + USPS CSVs6 counties, 84 county-years
  2. 02
    SQLite bronze / silver / gold15 quality checks stop the run if any fail
    • SQL pipelines
    • Data-quality testing
  3. 03
    NumPy gradient descentWritten from scratch and tested against scikit-learn
    • Gradient descent from scratch
    • Testing with pytest
  4. 04
    8 models × 3 testsRandom CV, forward-chaining, leave-one-county-out
    • Evaluation design
  5. 05
    Intervals by county80% intervals, checked county by county
    • Conformal prediction

What it found

Rise-or-fall accuracy
  • Always guess “down”64.8%
  • My logistic model57.4%
  • With more features51.9%
  • In a realistic nowcast, carrying last year’s Census value forward had the lowest error: mean absolute error 1.023 points, versus 1.054 for my ridge model (3% worse; with only 78 rows, I cannot say whether that gap is real), 1.119 for random forest, and 1.215 for gradient boosting.
  • The evaluation choice decided the conclusion: ridge ties persistence under random CV (0.945 vs 0.947) and beats it leaving one county out (ridge 0.914). Only the scheme that matches real use shows the loss.
  • For rise-or-fall predictions, always guessing “down” is right 64.8% of the time; my from-scratch logistic model reaches 57.4%, and adding features drops it to 51.9%.
  • Python
  • NumPy
  • scikit-learn
  • SQLite
  • pytest

Full case study // tap a section to open it

01Problem & context

Can a model that combines last year’s Census vacancy rate with this year’s USPS movement predict this year’s Census rate, and does the answer hold in every county?

Census vacancy estimates arrive about nine months after the year ends, while USPS data comes out every quarter. A reliable early estimate would let a local economic development office see housing problems sooner, and if it is not reliable, it is better to know before anyone acts on its numbers.

02My role & approach

Follow-up to my housing vacancy project: SQL pipeline, from-scratch gradient-descent models, eight-model comparison, three evaluation designs, and prediction intervals. I chose the questions and designed the evaluations, and used AI tools to double-check code for edge cases.

  1. 01

    Built a bronze → silver → gold pipeline in SQLite over Census ACS table B25002 and HUD USPS vacancy data for 6 counties (84 county-years, 78 modeling rows), with 15 data-quality checks that stop the run if any fail.

  2. 02

    Wrote linear and logistic regression trained by gradient descent from scratch in NumPy, with tests against closed-form and scikit-learn results.

  3. 03

    Compared eight regression models and five direction classifiers under random 5-fold CV (×20), forward-chaining (test years 2015–2024), and leave-one-county-out.

  4. 04

    Added split-conformal 80% prediction intervals, calibrated only on earlier years, and checked their coverage county by county.

Problems I hit and how I handled them

  • A 2012 USPS definition change makes the broad vacancy rate jump 2.2–4.9 points in every county, so the pipeline zeroes that year’s change and adds a post-2012 flag; otherwise a model learns a fake trend.
  • There is no 1-year Census release for 2020, so 2021 is compared with 2019 and the model table records the two-year gap.
  • Making the test match real use: only forward-chaining (train on earlier years, predict the next) reflects how a nowcast would actually be run.
  • Proving the from-scratch code is right: tests check that gradient-descent regression matches the closed-form solution and that logistic regression matches scikit-learn with the same penalty.
03Tech stack
  • Python
  • NumPy
  • scikit-learn
  • SQLite
  • pytest

Skills shown

  • Evaluation design
  • Gradient descent from scratch
  • Conformal prediction
  • SQL pipelines
  • Data-quality testing
  • Testing with pytest
04More results
  • 80% intervals cover 81% of values pooled, but only 57% in Champaign and 100% in Cook and King: a pooled number can hide the county a model serves worst.
05What I learned & would improve

Lessons

  • The evaluation design decided the conclusion: two standard schemes showed a small win, and only forward-chaining, which matches real use, showed the loss.
  • A pooled number can hide the place a model serves worst: 81% interval coverage overall, 57% in Champaign (few test years per county, so a noisy estimate).
  • With 78 rows, simple linear models beat trees, and adding features made the classifier worse.

What I would do next

  • Add more counties; the same pipeline works for any county with a 1-year ACS estimate.
  • Add ACS margins of error to separate model error from Census sampling noise.
  • Use the quarterly USPS series directly so the latest quarter can carry more weight.
Next projectNetflix Titles ETL
Esc

↑ ↓ to moveEnter to open