Opens in a new tab
Guide

The event arrived twice. Did the ticket get created once?

In brief

Recognizing a repeated incident event does not establish what Slack or Jira did with the first one. This reference design tracks arrivals and destination results separately.

7 min read

Is this the right starting point?
Before you read
Practical event-driven integration, incident tooling and API recovery experience with durable state, retries and access controls.
What you will take away
Review a recoverable incident-integration design that preserves destination identity, distinguishes intake deduplication from reconciliation and exposes pending synchronization.
Sources
Two matching cards sit beside an open book whose red cords lead to three blue trays: one contains a ticket, one is empty, and one is partly covered by folded paper.
On this page5 sections

The same incident event arrives again. The integration recognizes its identifier and avoids starting a second workflow. Then someone asks for the Jira ticket. The saved record has no confirmed answer.

Recognizing the repeated event was useful. It did not finish the work the first delivery started. Connecting PagerDuty, Jira and Slack means keeping track of both: which events have arrived, and what each destination has actually done.

Consider a hypothetical interruption inside that first workflow. Slack creates the incident channel and responders start using it, but the worker crashes before saving the channel identifier. The replacement worker needs to find the existing channel and establish the ticket's state, rather than treat the whole event as either untouched or complete.

This reference design saves progress outside the process that may crash so recovery can finish the missing steps separately. It is not an installable product or a tested deployment. A channel that is ready and a ticket still waiting to synchronize are two different results the design must be able to report together.

Give the next worker something to come back to

Before connecting the systems, agree which one owns each part of the incident. PagerDuty might own its lifecycle, Jira the follow-up and Slack the discussion. Identity, acknowledgment, severity and resolution still need agreed sources. A responder joining halfway through should not have to decide which of three conflicting screens tells the right story.

The integration needs a record of its own progress too. Give the incident a stable internal identifier and save the linked destination identifiers, incoming event identifiers, completed steps and work still pending. Keep this record in storage that survives the running assistant or worker. That is what makes it durable.

A replacement worker can use a saved channel identifier to check that channel directly. Our imagined crash happened before that identifier was saved, so recovery needs a supported way to locate the channel using the recorded incident identity. If that lookup cannot establish the result, leave it unresolved for an operator. A blank field in the integration's record is not proof that Slack did nothing.

The sequence below saves the verified incoming event before working through every destination. Deduplication means recognizing a repeated event rather than treating it as a new incident. Reconciliation means checking what actually happened in each destination and bringing the saved record into agreement with it.

Verified event → durable intake → deduplication
              → incident-state update
              → queued destination actions
              → reconcile results and record destination IDs
              → reviewed summary with links to evidence

With those steps recorded separately, the display can say that the channel is ready while the ticket is still waiting to synchronize. Responders can use the channel now. A single completion checkmark would lose the distinction they need.

Persist progress between incident-system writes
Persist progress between incident-system writes. Reference design: a durable incident record links provider IDs and completed steps. Slack and Jira may succeed independently; reconcile each destination rather than replaying the whole workflow.
A reference design, not a tested deployment. The saved incident record connects each destination to its completed and pending steps, so a restart can preserve an existing conversation while finding out what happened to the ticket.
Follow the saved incident record

The drawing separates incoming events, saved incident state, Slack and Jira actions, and the final account of what completed. Each destination can succeed independently. Recovery checks those results separately instead of starting the entire sequence again. Diagram labels: Verified incoming event: Validate, persist and deduplicate; Authoritative incident state: Incident ID, lifecycle, event IDs and queued work; Slack destination: Create or locate channel; store ID; Jira destination: Create required follow-up; store ID; Reconcile + report: Expose pending sync; review summaries against evidence.

The same event can arrive more than once

Build the first version in test workspaces and services, with approved data handling and identities limited to the permissions they need. Keep credentials in an appropriate secret store, outside checked-in configuration. A Kubernetes ConfigMap is for ordinary configuration, not secret storage. Verify incoming signatures with the provider's supported method, rejecting invalid or stale requests before putting them on the work queue.

The Slack Events API documents retries and best-effort delivery. Acknowledge a valid event promptly under that delivery contract, then let the longer work continue asynchronously, outside the immediate response. Save the event identifiers already handled so the same delivery does not start another copy of the workflow.

That solves one problem, but not our missing channel identifier. Recognizing that the event arrived earlier tells the worker not to start a second workflow. It still needs to discover whether the first workflow created the channel or ticket. Receiving an event and finishing its destination writes are separate facts.

Sometimes a destination answers but asks the integration to slow down. Jira Cloud's rate-limit documentation describes the HTTP 429 response, meaning too many requests, and its retry guidance. Follow the response's waiting instructions and the operation's retry-safety requirements. Cap retries and leave work visible for review when those attempts run out.

Watch how long queued work waits as well. If incoming events keep arriving faster than destination actions finish, the queue ages even while the process answers its health checks. Being alive and keeping up are different things to tell the responder waiting for a ticket link.

Keep the channel people are already using

In a test of this design, walk one incident through the destinations in order:

  1. Save the test incident's identity before creating anything that belongs to it.
  2. Create or locate its discussion channel, then save that channel's destination identifier.
  3. If policy requires a follow-up record, create it with a link to the same incident.
  4. Put the destination links in the authoritative record and check that responders can open them.
  5. When a write times out, establish the destination's state before deciding whether to send it again.

Now consider a hypothetical partial failure: Slack creates the channel, but the Jira update times out. The people in the channel begin investigating while the integration waits. Recovery needs to preserve that conversation and find out what happened to the ticket.

Replaying the whole workflow risks creating a second coordination space. Deleting the first channel to tidy the machine's records would be worse: it would discard the investigation underway. Each consequential write needs its own reconciliation, and the responders' work is now part of what the recovery must preserve.

“Incident exists; Jira synchronization pending” is an honest, useful status. Slack's success does not finish the whole workflow, and Jira's delay need not stop people using a valid channel. In controlled testing, make each destination unavailable separately and check that the implementation can express both parts of that outcome.

A repeated low-consequence notification may be tolerable. Decide that for the particular operation. Do not carry that tolerance over to assigning work, closing an incident or making another change that redirects the response.

A new responder needs the story so far

Someone joining the channel needs to know the impact, the current hypothesis, what people actually did and when the next update is due. Generated text can assemble those details from the durable incident record, with links that let the reader follow them back.

A new claim of recovery or a proposed resolution still needs review by an authorized responder. People act differently after being told the service has recovered. A fluent summary should not make that decision for them.

Keep maintained runbooks available instead of asking a model to invent a recovery procedure for each event. Keep the discussion under the organization's retention policy too. Creating a channel for one incident does not make its contents disposable afterward; archiving the channel and deleting its evidence are different actions.

Test the restart while the conversation continues

The revealing tests interrupt the sequence where two systems can disagree. Deliver a test event twice. Let a resolution event arrive out of order. Restart the worker after a destination write but before it saves the result. Include expired credentials and simulated rate-limit responses, then measure how long the disagreement stays visible. Do not generate rate-limit traffic against Atlassian cloud tenants; its documentation prohibits that testing.

Process uptime tells you the integration is running. Synchronization age tells you how far its account of the incident has fallen behind. Track both so a running worker cannot conceal a queue of unfinished handoffs.

The Slack operations guide covers communication fallback when Slack is unavailable. The agent skill contract addresses later remediation capabilities. Responders need a way to continue when Slack or the assistant disappears, and an actual implementation must demonstrate these behaviors in its own environment before production use.

When that repeated event arrives, the recovered record should answer more than “seen before.” It should show which destination steps are complete, which still need work and which have an unknown result. The responder asking for the ticket can then get its confirmed link, or a visible pending item for someone to resolve. Recognizing the message prevents duplicate work; accounting for the destinations tells people what remains.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next