Skip to content
Admin portal → Health/admin/health3 min

Health ​

One screen that says whether FMS is working. It refreshes itself every fifteen seconds.

Developers and owners only. Other roles do not see Health in the top menu.

The system health screen
Overall state, open incidents, components, recent alerts.
The system health screen

Click anywhere, or press Esc, to close.

The components ​

The components grid
Green across the grid means the plumbing is fine.
The components grid

Click anywhere, or press Esc, to close.

ComponentWhat red means
DatabaseFMS cannot reach its own storage. Nothing will work. Escalate immediately.
Background workersNever red. Off (grey) on a sandbox whose background work is stopped: its queue, emails and checks wait. Start it on Sandboxes. Production's always run.
Outbox workerMessages are queued but nothing is sending them. Payments and emails are affected.
Outbox crm workerNewsletter tag updates are queued but not being sent. Payments are not affected. Amber, not red.
Event queueRed: a payment-side message failed for good (a dead letter). Replay it on Dead letters. Amber: only newsletter syncs are dead, or work is piling up faster than it is being done. Every new dead letter sends a new alert, even while the row is already red. Dead staff emails are listed but never turn it red: on their own they turn it amber and send one alert, never one per email. If they pile up, the mail relay is refusing them; check Outgoing email.
Webhook freshnessStripe has not told us anything for a while. Cancellations and renewals will look stale. Counted for the environment shown only.
Member emailOnly shown when FMS sends email itself. Amber means member mail is stuck in the queue, or an email is marked Failed on Outgoing email and nobody has pressed Retry or Discard on it yet. The detail says which.
Log historyNever red. Amber means the log table or the Audit trail has grown past its size limit. Nothing is lost; tell Thomas.
Off-site backupOn the live console only. Amber with not configured: no copy of the database leaves this server. Set a target on Backup. Amber with NOT encrypted: copies leave the server without the backup key (Encrypted copies). Amber or red otherwise: last night's copy did not reach the storage. Green with sidecar: copies are made outside the console, so FMS cannot see them.
Stripe API versionNever red. Amber on the live console while STRIPE_API_VERSION is not set: FMS follows each Stripe account's default version, which changes the day someone upgrades the account in Stripe. The row names the version each account uses now; whoever runs the server sets that one. Amber when an account's default differs from the version FMS sends: recreate that account's webhook endpoint on Connections.
Stripe — account nameOne row per connected Stripe account. Red: Stripe refuses that account's key (revoked or rotated), or did not answer twice in a row. Amber: a member's renewal was due more than two days ago and Stripe has sent nothing since — the webhook may have been deleted. A quiet account with nothing due stays green. Open the account from Connections, fix its keys and press Re-check under Summary. The row turns green at once.
Opn — account nameOne row per connected Opn Payments account. Red: Omise refuses the account's secret key (rolled or revoked), or did not answer twice in a row. Open the account from Connections, fix its keys and press Re-check under Summary. The row turns green at once.
Mailchimp connectionRed: Mailchimp refuses the key, the stored key cannot be read, no audience is chosen, the audience is gone, or it did not answer twice in a row. Newsletter tags wait in the queue until it is fixed.
Newsletter archiveOnly shown once Connections names the archive. Red: FMS could not reach it twice in a row. Amber: it has not synced with Mailchimp for two days, its last sync stopped early, or its site is not built. Members' payments are not affected.

Issues the archive can't fetch ​

The newsletter archive card lists any issue Mailchimp keeps refusing, with the date it started failing and Mailchimp's reason. The archive tries each one again on every sync and still rebuilds without it, so the card stays green.

If an issue stays on the list for days, open it in Mailchimp by the code shown beside its reason. It is usually an issue that was deleted or never finished sending.

Recent alerts ​

The recent alerts list
What went wrong and whether it cleared.
The recent alerts list

Click anywhere, or press Esc, to close.

Most alerts clear themselves once the underlying problem passes. An alert that stays open needs a person.

What to do when something is red ​

  1. Wait two minutes and let it refresh. Brief blips are normal.
  2. If it is still red, open Dead letters and see whether messages are stuck.
  3. Tell Thomas what the component was and what the alert said.

Do not restart anything yourself.

Internal staff guide. Screenshots come from the rehearsal rig and never show real members. Project status (for developers and admins).