Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Change Verification

Verify whether a released change worked and whether it caused regressions

Change Verification verifies a released code or configuration change against production evidence. Cleric checks whether the intended outcome occurred, looks for regressions, and continues monitoring signals that need more time.

How It Works

Automatic verification begins when a configured Slack release notification runs the verify agent. For each release, Cleric:

  1. Identifies what shipped, where and when it shipped, and whether the expected version is live.

  2. Records each change. Cleric skips changes without a production outcome to verify. For every tracked change, it reads the source diff and records expectations about the running system.

  3. Checks each expectation against production logs, metrics, traces, infrastructure state, or live configuration. It resolves the expectation when the evidence is sufficient or monitors it with a scheduled check when more observation time is needed.

  4. Updates the change and expectation statuses as new evidence arrives.

When a release contains several changes, Cleric assesses each change independently. This keeps an anomaly or missing data for one change from obscuring the results for the others.

Baseline Assumptions

Automatic Change Verification assumes that your environment has:

  • Release notifications: A Slack channel where a bot posts the deployed service or component, environment, rollout time, and version. Pull request or commit links improve the analysis.

  • Source access: A connected source-control integration that lets Cleric read change descriptions and diffs.

  • Production access: Connected integrations that expose the logs, metrics, traces, infrastructure state, or live configuration needed to assess the change.

Cleric reports missing access or missing telemetry as a limitation. It does not treat an empty query result as proof until it has validated the query and its filters.

Set Up Automatic Change Verification

1

Prepare the Release Channel

Choose the Slack channel that receives release notifications. A channel dedicated to deployments works best because a catch-all trigger can process every bot message without matching unrelated alerts.

Invite Cleric to the channel with /invite @Cleric if it is not already a member.

2

Add a Bot Message Trigger

Open Settings > Automations > Triggers and click Add trigger. Keep the Bot message trigger type selected, then configure:

  • Channel: Select the release channel.

  • And matches keyword(s): Use * when every bot message in the channel is a release notification. Otherwise use a distinctive phrase that appears in release messages, such as deployment completed.

  • Then > Run agent: Select verify.

Place this trigger above broader triggers that could match the same release message. The first matching trigger runs.

3

Validate the Next Release

After the next release notification, confirm that Cleric reacts to the message and posts a link in its thread. The link opens the Changes page scoped to that release. Confirm that Cleric identified the correct changes, target environment, and rollout time.

For more detail about trigger matching and ordering, see Configuring Automations.

Run Change Verification From Slack

Run the built-in agent from Slack by prefixing the request with !verify. Change verification cannot be started from web chat.

For a pull request:

For another type of release, describe the change and target environment:

If Cleric cannot identify the change or environment from the request and connected systems, it asks for the missing detail.

Last updated