Principles
Build for the difficult afternoon.
Know the normal state
Record the handful of signals that describe a healthy system. An alert without a baseline is only noise.
Separate experiments
New components should have clear boundaries, independent credentials, and an obvious way to leave.
Preserve the exit
Before changing a working service, decide exactly how the previous state will be restored.