Section 6.4: Monitoring and the Closing Loop

A service that answers is not yet a service anyone can watch. This section makes the running service report on itself: one JSON object per log line, a middleware that times and counts every request, a /health that names the model version it serves and counts what it refused and what failed, and every stored prediction tagged with that version and its latency. Then it sends a moved market through the service, measures how far the live inputs have drifted from the training rows with SciPy's two-sample KS test and the population stability index, the second written out by hand, retrains a challenger on fresh labeled rows, and moves the alias only when the challenger wins.

Section 6.3 shipped the service. Here it runs, and the failures that matter now are the ones a running service hides. A database restart leaves a single connection closed, so every prediction after it fails with a 500 while /health still says ok and Docker still calls the container healthy, until a connection pool checks each connection before handing it out. And the moved market raises nothing at all: 600 listings priced, 0 errors, latency unchanged, while the error on those listings climbs from 56,562 to 92,296, because nothing in the serving stack knows what a correct price is.

Lessons

  1. Logging Predictions and Service Health - structured logs, one JSON object per line, a middleware that times and counts every request, a health report beyond "ok", model version and latency in every stored prediction, and a connection pool that survives a database restart.
  2. Data Drift and the Retraining Loop - a moved market through the service, a healthy service that prices wrong, the two-sample KS test, the population stability index, drift.py and the retraining trigger, and a challenger against the champion, then the redeploy.

By the end you will be able to:

  • Log every request as structured JSON, count what a service served, refused and failed, and report it with the model version it serves.
  • Query the stored predictions by model version for their count, latency and prices, and keep the service answering through a database restart.
  • Measure drift between training and live inputs with a KS test and the population stability index, and tell a real shift from the noise of a small sample.
  • Retrain a challenger on fresh labeled rows, promote it only when it beats the champion, and redeploy the version /health then names.