Some customers using WhatsApp inboxes with account-level webhooks were
reporting receiving duplicate `message_created` webhook deliveries for
every incoming message. Upon inspection, here's what we found
- Both payloads are identical.
- No errors appear in the application logs
- Webhook URL is only configured in one place.
This meant, the system was sending the webhooks twice. For some context,
there's a know related issue... Meta's WhatsApp Business API can deliver
the same webhook notification multiple times for a single message. The
codebase already acknowledges this — there's a comment in
`IncomingMessageBaseService#process_messages` noting that "multiple
webhook events can be received against the same message due to
misconfigurations in the Meta business manager account." A deduplication
guard exists, but it doesn't actually work under concurrency.
### Rationale
The existing dedup was a three-step sequence: check Redis (`GET`), check
the database, then set a Redis flag (`SETEX`). Two Sidekiq workers
processing duplicate Meta webhooks simultaneously would both complete
the `GET` before either executed the `SETEX`, so both would proceed to
create a message. The `source_id` column has a non-unique index, so the
database wouldn't catch the duplicate either. Each message then
independently fires `after_create_commit`, dispatching two
`message_created` webhook events to the customer.
```
Worker A Worker B
│ │
▼ ▼
Redis GET key ──► nil Redis GET key ──► nil
│ │
│ ◄── both pass guard ──► │
│ │
▼ ▼
Redis SETEX key Redis SETEX key
│ │
▼ ▼
BEGIN transaction BEGIN transaction
INSERT message INSERT message
DELETE Redis key ◄─┐ │
COMMIT │ DELETE Redis key
│ COMMIT
│ │
└── key gone before ───┘
B's commit lands
▼ ▼
after_create_commit after_create_commit
dispatch MESSAGE_CREATED dispatch MESSAGE_CREATED
│ │
▼ ▼
WebhookJob ──► n8n WebhookJob ──► n8n
(duplicate!)
```
There was a second, subtler problem visible in the diagram: the Redis
key was cleared *inside* the database transaction, before the
transaction committed. This opened a window where neither the Redis
check nor the database check would see the in-flight message.
The fix collapses the check-and-set into a single `SET NX EX` call,
which is atomic in Redis. The key is no longer eagerly cleared — it
expires naturally after 24 hours. The database lookup
(`find_message_by_source_id`) remains as a fallback for messages that
were created before the lock expired.
```
Worker A Worker B
│ │
▼ ▼
Redis SET NX ──► OK Redis SET NX ──► nil
│ │
▼ ▼
proceeds to create returns early
message normally (lock already held)
```
### Implementation Notes
The lock logic is extracted into `Whatsapp::MessageDedupLock`, a small
class that wraps a single `Redis SET NX EX` call. This makes the
concurrency guarantee testable in isolation — the spec uses a
`CyclicBarrier` to race two threads against the same key and asserts
exactly one wins, without needing database writes,
`use_transactional_tests = false`, or monkey-patching.
Because the Redis lock now persists (instead of being cleared
mid-transaction), existing WhatsApp specs needed an `after` hook to
clean up `MESSAGE_SOURCE_KEY::*` keys between examples. Transactional
fixtures only roll back the database, not Redis.
213 lines
8.0 KiB
Ruby
213 lines
8.0 KiB
Ruby
require 'rails_helper'
|
|
|
|
describe Whatsapp::IncomingMessageWhatsappCloudService do
|
|
describe '#perform' do
|
|
after do
|
|
Redis::Alfred.scan_each(match: 'MESSAGE_SOURCE_KEY::*') { |key| Redis::Alfred.delete(key) }
|
|
end
|
|
|
|
let!(:whatsapp_channel) { create(:channel_whatsapp, provider: 'whatsapp_cloud', sync_templates: false, validate_provider_config: false) }
|
|
let(:params) do
|
|
{
|
|
phone_number: whatsapp_channel.phone_number,
|
|
object: 'whatsapp_business_account',
|
|
entry: [{
|
|
changes: [{
|
|
value: {
|
|
contacts: [{ profile: { name: 'Sojan Jose' }, wa_id: '2423423243' }],
|
|
messages: [{
|
|
from: '2423423243',
|
|
image: {
|
|
id: 'b1c68f38-8734-4ad3-b4a1-ef0c10d683',
|
|
mime_type: 'image/jpeg',
|
|
sha256: '29ed500fa64eb55fc19dc4124acb300e5dcca0f822a301ae99944db',
|
|
caption: 'Check out my product!'
|
|
},
|
|
timestamp: '1664799904', type: 'image'
|
|
}]
|
|
}
|
|
}]
|
|
}]
|
|
}.with_indifferent_access
|
|
end
|
|
|
|
context 'when valid attachment message params' do
|
|
it 'creates appropriate conversations, message and contacts' do
|
|
stub_media_url_request
|
|
stub_sample_png_request
|
|
described_class.new(inbox: whatsapp_channel.inbox, params: params).perform
|
|
expect_conversation_created
|
|
expect_contact_name
|
|
expect_message_content
|
|
expect_message_has_attachment
|
|
end
|
|
|
|
it 'increments reauthorization count if fetching attachment fails' do
|
|
stub_request(
|
|
:get,
|
|
whatsapp_channel.media_url('b1c68f38-8734-4ad3-b4a1-ef0c10d683')
|
|
).to_return(
|
|
status: 401
|
|
)
|
|
|
|
described_class.new(inbox: whatsapp_channel.inbox, params: params).perform
|
|
expect(whatsapp_channel.inbox.conversations.count).not_to eq(0)
|
|
expect(Contact.all.first.name).to eq('Sojan Jose')
|
|
expect(whatsapp_channel.inbox.messages.first.content).to eq('Check out my product!')
|
|
expect(whatsapp_channel.inbox.messages.first.attachments.present?).to be false
|
|
expect(whatsapp_channel.authorization_error_count).to eq(1)
|
|
end
|
|
end
|
|
|
|
context 'when invalid attachment message params' do
|
|
let(:error_params) do
|
|
{
|
|
phone_number: whatsapp_channel.phone_number,
|
|
object: 'whatsapp_business_account',
|
|
entry: [{
|
|
changes: [{
|
|
value: {
|
|
contacts: [{ profile: { name: 'Sojan Jose' }, wa_id: '2423423243' }],
|
|
messages: [{
|
|
from: '2423423243',
|
|
image: {
|
|
id: 'b1c68f38-8734-4ad3-b4a1-ef0c10d683',
|
|
mime_type: 'image/jpeg',
|
|
sha256: '29ed500fa64eb55fc19dc4124acb300e5dcca0f822a301ae99944db',
|
|
caption: 'Check out my product!'
|
|
},
|
|
errors: [{
|
|
code: 400,
|
|
details: 'Last error was: ServerThrottle. Http request error: HTTP response code said error. See logs for details',
|
|
title: 'Media download failed: Not retrying as download is not retriable at this time'
|
|
}],
|
|
timestamp: '1664799904', type: 'image'
|
|
}]
|
|
}
|
|
}]
|
|
}]
|
|
}.with_indifferent_access
|
|
end
|
|
|
|
it 'with attachment errors' do
|
|
described_class.new(inbox: whatsapp_channel.inbox, params: error_params).perform
|
|
expect(whatsapp_channel.inbox.conversations.count).not_to eq(0)
|
|
expect(Contact.all.first.name).to eq('Sojan Jose')
|
|
expect(whatsapp_channel.inbox.messages.count).to eq(0)
|
|
end
|
|
end
|
|
|
|
context 'when invalid params' do
|
|
it 'will not throw error' do
|
|
described_class.new(inbox: whatsapp_channel.inbox, params: { phone_number: whatsapp_channel.phone_number,
|
|
object: 'whatsapp_business_account', entry: {} }).perform
|
|
expect(whatsapp_channel.inbox.conversations.count).to eq(0)
|
|
expect(Contact.all.first).to be_nil
|
|
expect(whatsapp_channel.inbox.messages.count).to eq(0)
|
|
end
|
|
end
|
|
|
|
context 'when message is a reply (has context)' do
|
|
let(:reply_params) do
|
|
{
|
|
phone_number: whatsapp_channel.phone_number,
|
|
object: 'whatsapp_business_account',
|
|
entry: [{
|
|
changes: [{
|
|
value: {
|
|
contacts: [{ profile: { name: 'Pranav' }, wa_id: '16503071063' }],
|
|
messages: [{
|
|
context: {
|
|
from: '16503071063',
|
|
id: 'wamid.ORIGINAL_MESSAGE_ID'
|
|
},
|
|
from: '16503071063',
|
|
id: 'wamid.REPLY_MESSAGE_ID',
|
|
timestamp: '1770407829',
|
|
text: { body: 'This is a reply' },
|
|
type: 'text'
|
|
}]
|
|
}
|
|
}]
|
|
}]
|
|
}.with_indifferent_access
|
|
end
|
|
|
|
context 'when the original message exists in Chatwoot' do
|
|
it 'sets in_reply_to to reference the existing message' do
|
|
# Create a conversation and the original message that will be replied to first
|
|
contact = create(:contact, phone_number: '+16503071063', account: whatsapp_channel.account)
|
|
contact_inbox = create(:contact_inbox, contact: contact, inbox: whatsapp_channel.inbox, source_id: '16503071063')
|
|
conversation = create(:conversation, contact: contact, inbox: whatsapp_channel.inbox, contact_inbox: contact_inbox)
|
|
|
|
original_message = create(:message,
|
|
conversation: conversation,
|
|
source_id: 'wamid.ORIGINAL_MESSAGE_ID',
|
|
content: 'Original message')
|
|
|
|
described_class.new(inbox: whatsapp_channel.inbox, params: reply_params).perform
|
|
|
|
reply_message = whatsapp_channel.inbox.messages.last
|
|
expect(reply_message.content).to eq('This is a reply')
|
|
expect(reply_message.content_attributes['in_reply_to']).to eq(original_message.id)
|
|
expect(reply_message.content_attributes['in_reply_to_external_id']).to eq('wamid.ORIGINAL_MESSAGE_ID')
|
|
end
|
|
end
|
|
|
|
context 'when the original message does not exist in Chatwoot' do
|
|
it 'does not set in_reply_to (discards the reply reference)' do
|
|
described_class.new(inbox: whatsapp_channel.inbox, params: reply_params).perform
|
|
|
|
reply_message = whatsapp_channel.inbox.messages.last
|
|
expect(reply_message.content).to eq('This is a reply')
|
|
expect(reply_message.content_attributes['in_reply_to']).to be_nil
|
|
expect(reply_message.content_attributes['in_reply_to_external_id']).to be_nil
|
|
end
|
|
end
|
|
end
|
|
end
|
|
|
|
# Métodos auxiliares para reduzir o tamanho do exemplo
|
|
|
|
def stub_media_url_request
|
|
stub_request(
|
|
:get,
|
|
whatsapp_channel.media_url('b1c68f38-8734-4ad3-b4a1-ef0c10d683')
|
|
).to_return(
|
|
status: 200,
|
|
body: {
|
|
messaging_product: 'whatsapp',
|
|
url: 'https://chatwoot-assets.local/sample.png',
|
|
mime_type: 'image/jpeg',
|
|
sha256: 'sha256',
|
|
file_size: 'SIZE',
|
|
id: 'b1c68f38-8734-4ad3-b4a1-ef0c10d683'
|
|
}.to_json,
|
|
headers: { 'content-type' => 'application/json' }
|
|
)
|
|
end
|
|
|
|
def stub_sample_png_request
|
|
stub_request(:get, 'https://chatwoot-assets.local/sample.png').to_return(
|
|
status: 200,
|
|
body: File.read('spec/assets/sample.png')
|
|
)
|
|
end
|
|
|
|
def expect_conversation_created
|
|
expect(whatsapp_channel.inbox.conversations.count).not_to eq(0)
|
|
end
|
|
|
|
def expect_contact_name
|
|
expect(Contact.all.first.name).to eq('Sojan Jose')
|
|
end
|
|
|
|
def expect_message_content
|
|
expect(whatsapp_channel.inbox.messages.first.content).to eq('Check out my product!')
|
|
end
|
|
|
|
def expect_message_has_attachment
|
|
expect(whatsapp_channel.inbox.messages.first.attachments.present?).to be true
|
|
end
|
|
end
|