Common Use Cases

Ideas for how to use Cleric

New to Cleric? Start with the Quick Start Guide to run your first investigation, then return here for more examples.

This guide provides a comprehensive library of prompts showing various ways to use Cleric. In addition to automated alert triage, use Cleric to troubleshoot production issues, run proactive health scans, analyze observability data, or assist with incident retrospectives.

Use Case Guides

• Change Verification: Verify that a released change produced its intended outcome and did not cause regressions.

Quick Tips

• Be specific: Help Cleric scope its investigation by including details like time windows (and timezones!), identifiers, and resource names

• Describe the symptom: Include what users are seeing and any error messages, and attach images when appropriate

• Ask for evidence: Request links to dashboards, log queries, and traces and ask Cleric to cite its sources

• Follow up: Ask for additional hypotheses, further investigation, and remediation proposals

Example Investigation Flow

Here's what a complete investigation looks like:

@Cleric Users are reporting that checkout is failing intermittently since 2pm CET with 503s. Check the GCP logs and correlate with any recent deployments.

Cleric's response:

**Root Cause:** Database connection pool exhaustion in checkout-service

**Timeline:**
- 2:00 PM CET: Traffic increased 3.5x due to marketing campaign launch
- 2:15 PM CET: Connection pool reached 100% utilization (10/10 connections)
- 2:15 PM CET: First 503 errors appeared in logs

**Evidence:**
• GCP Logs show "waiting for available database connection" warnings
• Connection pool metrics show sustained saturation since 2:15 PM (10/10 connections)
• Error rate: 12% of checkout requests (847 failures / 7,058 requests)

Follow-up conversation:

What's the typical connection pool usage outside of incidents like this?

Last 30 days:
• Baseline: 4-6 connections (40-60% of pool)
• Peak hours (9am-5pm CET): 7-8 connections (70-80%)
• Off-hours: 2-3 connections (20-30%)

Today: 10/10 sustained (100% utilization, unusual)

Customizing Your Prompts

You can modify any prompt by specifying:

Dimension	Examples
Data source	"using Prometheus...", "check Datadog for...", "search GCP logs for..."
Scope	"for checkout-api", "in the UAT environment", "in the us-west region"
Time window	"in the last 4 hours", "since 2pm PST", "between Jan 15 2-4pm UTC"
Output format	"with evidence links", "as a timeline", "summarize in bullet points"

Investigate Issues

Understand what's happening when something goes wrong.

Incident/bug investigation

Prefix any prompt with !investigate to trigger a full deep investigation instead of conversational mode. See Deep Investigations.

Users are reporting that checkout is failing intermittently since 2pm CET with 503s. Check the GCP logs and correlate with any recent deployments.

Look for logs in the last 24h related to app installations in the canary instance and tell me if you see anything indicating a failure. The user reports seeing an error about a missing field.

Service health

Check payments health since 9am PST, include evidence links.

Example response

Payments service health check (since 9:00 AM PST):

**Status:** ✅ Healthy

**Metrics:**
• Uptime: 100% (no restarts or crashes)
• Error rate: 0.08% (3 errors / 3,847 requests) - within normal range
• P95 latency: 245ms (baseline: 220-250ms)
• Memory usage: 1.2GB / 2.0GB (60% - normal)

**Recent Activity:**
• No deployments since yesterday 3:15 PM PST
• Traffic volume: 15% above baseline (expected for Monday morning)

**Alerts:** None active

**Evidence:** [Datadog dashboard →] | [Error logs →] | [Metrics →]

Everything looks healthy. The 3 errors were timeout exceptions from the external payment gateway (Stripe) and were automatically retried successfully.

Are there any open heartbeat alerts for the marketing site?

Debugging errors

Debug why Prometheus in the platform cluster has unhealthy ingesters since 10am EST

Example response

**Root Cause:** Disk space exhaustion on ingester nodes

**Timeline:**
• 9:45 AM EST: Disk usage reached 95% on ingester-2 and ingester-3
• 10:00 AM EST: Ingesters marked as unhealthy by Prometheus
• 10:15 AM EST: Write requests began failing with "insufficient disk space" errors

**Impact:**
• Write path degraded: 40% of metrics samples dropped
• Read path operational: Query performance unaffected
• Data loss: ~2 hours of metrics for affected series
• Node count: 3 ingesters, 2 unhealthy

**Evidence:**
• Ingester logs show "no space left on device" errors, disk at 98% utilization
• WAL segment size increased 300% in last 7 days due to high cardinality metrics

Debug why the KEDA autoscaler has been throwing errors for the last 4 hours

Logs & metrics analysis

Using GCP logs, get the most common errors across all our customers in the last day.

Search Datadog logs for any errors related to user 3bc96ec9-9367-4e40-8673-6a2fc9ceb87b

Check the last 7 days of error logs for the acme instance. Group the logs into top 3 error patterns with counts and first/last seen and print it as a markdown table.

Example response

Top 3 error patterns (last 7 days):

| Error Pattern | Count | First Seen | Last Seen | Severity |
|--------------|-------|------------|-----------|----------|
| `ConnectionPoolTimeoutException: Could not acquire connection from pool` | 1,247 | Jan 23, 08:15 AM | Jan 29, 11:42 PM | ERROR |
| `ValidationError: Missing required field 'customer_id' in request` | 892 | Jan 24, 02:30 PM | Jan 30, 09:15 AM | ERROR |
| `HTTPException: 503 Service Unavailable from pricing-api` | 456 | Jan 26, 10:00 AM | Jan 29, 06:30 PM | ERROR |

**Analysis:**
• Connection pool errors: steady increase (avg 178/day)
• Validation errors: spiked Jan 27 (402 occurrences), then decreased
• 503 errors from pricing-api: intermittent, clustered around peak hours

Show the top 10 services with highest error rate increase vs baseline since midnight

CI/CD failures

Investigate why the last 3 web-app pipeline runs failed at the same post-deploy step waiting for the new version to come up.

Proactive Scans

Run fleet-wide scans to catch issues before they become incidents, or to improve observability data hygiene.

Warning spike scan

Run a proactive system health check. Find services with a spike in WARNING logs as compared to baseline. List the warning patterns and whether they correlate with any recent deploys.

Example response

3 services with elevated WARNING logs:

**1. inventory-api** (+245% vs baseline)
• Warning pattern: "Cache miss rate above threshold: 45%"
• Baseline: 12 warnings/hour → Current: 41 warnings/hour
• Correlation: Deployed v2.8.0 on Jan 28 at 2pm (Redis cache configuration changed)

**2. notification-service** (+180% vs baseline)
• Warning pattern: "Retry queue depth exceeds 1000 messages"
• Baseline: 5 warnings/hour → Current: 14 warnings/hour
• Correlation: No recent deployments (last deploy: Jan 20)

**3. analytics-service** (+120% vs baseline)
• Warning pattern: "Query execution time exceeded 5s threshold"
• Baseline: 8 warnings/hour → Current: 18 warnings/hour
• Correlation: No deployments, data volume increased 3x on Jan 27

**Summary:**
• 1 of 3 spikes correlates with deployment
• 2 spikes related to external factors (email service, data volume)
• No critical issues detected

Log clean up

Identify the top 5 services that produced the most logs over the last 7 days. Propose recommendations of changes we can make to reduce log volumes.

Incident Retrospectives

Get help answering retrospective questions or building post-incident timelines.

Structured retrospective

Use the attached conversation log to produce a structured retrospective report. Establish start time and mitigation time, build an evidence-based timeline, and identify contributing factors. Include evidence links.

Timeline construction

Help me build a timeline for the incident yesterday between 2pm and 4pm PST.

Example response

**Incident Timeline: Jan 29, 2pm - 4pm PST**

**14:00 - Detection**
• PagerDuty alert: "High API Error Rate - checkout-service" [Alert →]
• User reports begin in #support channel (5 reports of checkout failures)

**14:05 - Initial Response**
• On-call engineer acknowledges alert
• Error rate: 15% (baseline: <0.5%)
• P95 latency: 3.2s (baseline: 450ms)

**14:12 - Investigation Begins**
• Checkout-service logs show database connection timeouts
• Connection pool: 30/30 connections in use (100% utilization)
• No recent deployments to checkout-service

**14:18 - Root Cause Identified**
• Database primary experiencing high CPU (95% utilization)
• Slow query detected: Complex JOIN query from analytics-service
• Query running since 13:58 (14 minutes before incident)

**14:25 - Mitigation Started**
• Analytics team contacted
• Long-running query killed on database primary
• Connection pool begins recovering

**14:32 - Service Recovery**
• Error rate drops to 2%
• Latency returns to baseline (480ms)
• Connection pool utilization: 45%

**14:45 - Full Resolution**
• Error rate: <0.5% (normal)
• All services healthy
• User reports cease

**15:30 - Post-Incident Actions**
• Query timeout added to analytics-service (30s limit)
• Database query monitoring alerts configured
• Incident review scheduled for Jan 31

**Duration:** 45 minutes (detection to resolution)
**Impact:** ~15% of checkout transactions failed during incident window

What were the contributing factors to the web-app outage last Tuesday?

Retrieve Knowledge

Query your documentation, your codebase, or Cleric's knowledge.

Documentation lookups

What does our GitOps Runbook say about the debugging flow?

Search Confluence for the on-call escalation process.

Codebase exploration

Investigate under what circumstances a `user_created` event would be tagged with `source` of "system unknown"

Issue/tenant lookup

What tenant does issue_id 4c4d742c-2532-4941-8106-7ba112b7fa92 belong to?

Tool capabilities

Which clusters are visible to you when using the datasource 'logs-nonprod'?

What can you do with your AWS credentials?

Memory queries

Search your memories for how to identify a tenant from a user id.

What do you know about the payments service?

Example response

Here's what I remember about the payments service:

**Service Details:**
• Owner: Payments team (#payments-oncall for escalations)
• Primary dependencies: Stripe API, inventory-api, pricing-api
• Database: Uses payments_db with read replicas in us-east and us-west

**Known Behaviors:**
• Memory usage spikes to ~2.5GB during month-end batch processing (expected behavior)
• Read replica lag >500ms triggers automatic fallback to primary database
• Timeout threshold for Stripe API calls: 10 seconds

**Troubleshooting Guidance:**
• When investigating latency: Check read replicas first, then Stripe API status
• When investigating errors: 90% of payment failures are due to external gateway issues, not our service
• Recent deployments: Always check if inventory-api was also deployed (frequent cause of integration issues)

Remediate & Handoff

Take action on issues: summarize findings, create tickets, or open pull requests.

On the web issue page, use More actions on a suggested fix to open a prepared prompt in Claude Code, Codex, Conductor, or Cursor, or choose Copy to clipboard. The prompt includes the fix and investigation context. Select the intended repository in your coding agent before applying the fix.

With GitHub or Bitbucket connected, choose Create PR to have Cleric implement the suggested fix and open a pull request for review.

Summarization

Generate a detailed summary of this issue, including a mermaid dataflow diagram of the current implementation.

Investigation diagram

Create a mermaid diagram representing the path of your investigation. If existing, dead-ends should be represented and the main "golden" path (from alert to conclusion) should be prominent.

Create tickets

Create a GitHub issue that summarizes the problem and potential solutions.

Create a Jira ticket in project SRE with the root cause and remediation steps.

Open pull requests

Add debug logs to this flow and create a draft PR for me to review.

Monitoring Production

Keep an eye on production systems and verify changes are working as expected.

Verify a fix

The memory leak fix for auth-service was deployed at 3pm. Compare memory usage before and after the deploy.

We rolled back the payments-api to the previous version. Verify that the 500 errors have stopped.

Scheduled follow-ups

Ask Cleric to check back after a delay. This is useful when you've pushed a fix and want to verify it later without watching dashboards yourself.

I just deployed a fix for the connection pool issue. Check the error rate for checkout-service in 30 minutes and let me know if it's back to normal.

Check the error logs for payments-api again in an hour and tell me if the timeout errors have stopped since the deploy at 2pm.