feat: captain decides if conversation should be resolved or kept open (#13336)

# Pull Request Template

## Description

captain decides if conversation should be resolved or open

Fixes
https://linear.app/chatwoot/issue/AI-91/make-captain-resolution-time-configurable

Update: Added 2 entries in reporting events:
`conversation_captain_handoff` and `conversation_captain_resolved`

## Type of change

Please delete options that are not relevant.

- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update

## How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details
for your test configuration.

LLM call decides that conversation is resolved, drops a private note
<img width="1228" height="438" alt="image"
src="https://github.com/user-attachments/assets/fb2cf1e9-4b2b-458b-a1e2-45c53d6a0158"
/>

LLM call decides conversation is still open as query was not resolved
<img width="1215" height="573" alt="image"
src="https://github.com/user-attachments/assets/2d1d5322-f567-487e-954e-11ab0798d11c"
/>


## Checklist:

- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream
modules

---------

Co-authored-by: Sojan Jose <sojan@pepalo.com>

enterprise/app/jobs/captain/inbox_pending_conversations_resolution_job.rb

@@ -1,15 +1,16 @@
 class Captain::InboxPendingConversationsResolutionJob < ApplicationJob
+  CAPTAIN_INFERENCE_RESOLVE_ACTIVITY_REASON = 'no outstanding questions'.freeze
+  CAPTAIN_INFERENCE_HANDOFF_ACTIVITY_REASON = 'pending clarification from customer'.freeze
+
   queue_as :low
 
   def perform(inbox)
     return if inbox.account.captain_disable_auto_resolve
 
-    Current.executed_by = inbox.captain_assistant
-
-    resolvable_conversations = inbox.conversations.pending.where('last_activity_at < ? ', Time.now.utc - 1.hour).limit(Limits::BULK_ACTIONS_LIMIT)
-    resolvable_conversations.each do |conversation|
-      create_outgoing_message(conversation, inbox)
-      conversation.resolved!
+    if inbox.account.feature_enabled?('captain_tasks')
+      perform_with_evaluation(inbox)
+    else
+      perform_time_based(inbox)
     end
   ensure
     Current.reset
@@ -17,18 +18,118 @@ class Captain::InboxPendingConversationsResolutionJob < ApplicationJob
 
   private
 
-  def create_outgoing_message(conversation, inbox)
+  def perform_time_based(inbox)
+    Current.executed_by = inbox.captain_assistant
+
+    resolvable_pending_conversations(inbox).each do |conversation|
+      create_resolution_message(conversation, inbox)
+      conversation.resolved!
+    end
+  end
+
+  def perform_with_evaluation(inbox)
+    Current.executed_by = inbox.captain_assistant
+
+    resolvable_pending_conversations(inbox).each do |conversation|
+      evaluation = evaluate_conversation(conversation, inbox)
+      next unless still_resolvable_after_evaluation?(conversation)
+
+      if evaluation[:complete]
+        resolve_conversation(conversation, inbox, evaluation[:reason])
+      else
+        handoff_conversation(conversation, inbox, evaluation[:reason])
+      end
+    end
+  end
+
+  def evaluate_conversation(conversation, inbox)
+    Captain::ConversationCompletionService.new(
+      account: inbox.account,
+      conversation_display_id: conversation.display_id
+    ).perform
+  end
+
+  def resolvable_pending_conversations(inbox)
+    inbox.conversations.pending
+         .where('last_activity_at < ?', auto_resolve_cutoff_time)
+         .limit(Limits::BULK_ACTIONS_LIMIT)
+  end
+
+  def still_resolvable_after_evaluation?(conversation)
+    conversation.reload
+    conversation.pending? && conversation.last_activity_at < auto_resolve_cutoff_time
+  rescue ActiveRecord::RecordNotFound
+    false
+  end
+
+  def auto_resolve_cutoff_time
+    Time.now.utc - 1.hour
+  end
+
+  def resolve_conversation(conversation, inbox, reason)
+    create_private_note(conversation, inbox, "Auto-resolved: #{reason}")
+    create_resolution_message(conversation, inbox)
+    conversation.with_captain_activity_context(
+      reason: CAPTAIN_INFERENCE_RESOLVE_ACTIVITY_REASON,
+      reason_type: :inference
+    ) { conversation.resolved! }
+    conversation.dispatch_captain_inference_resolved_event
+  end
+
+  def handoff_conversation(conversation, inbox, reason)
+    create_private_note(conversation, inbox, "Auto-handoff: #{reason}")
+    create_handoff_message(conversation, inbox)
+    conversation.with_captain_activity_context(
+      reason: CAPTAIN_INFERENCE_HANDOFF_ACTIVITY_REASON,
+      reason_type: :inference
+    ) { conversation.bot_handoff! }
+    conversation.dispatch_captain_inference_handoff_event
+    send_out_of_office_message_if_applicable(conversation.reload)
+  end
+
+  def send_out_of_office_message_if_applicable(conversation)
+    # Campaign conversations should never receive OOO templates — the campaign itself
+    # serves as the initial outreach, and OOO would be confusing in that context.
+    return if conversation.campaign.present?
+
+    ::MessageTemplates::Template::OutOfOffice.perform_if_applicable(conversation)
+  end
+
+  def create_private_note(conversation, inbox, content)
+    conversation.messages.create!(
+      message_type: :outgoing,
+      private: true,
+      sender: inbox.captain_assistant,
+      account_id: conversation.account_id,
+      inbox_id: conversation.inbox_id,
+      content: content
+    )
+  end
+
+  def create_resolution_message(conversation, inbox)
     I18n.with_locale(inbox.account.locale) do
       resolution_message = inbox.captain_assistant.config['resolution_message']
       conversation.messages.create!(
-        {
-          message_type: :outgoing,
-          account_id: conversation.account_id,
-          inbox_id: conversation.inbox_id,
-          content: resolution_message.presence || I18n.t('conversations.activity.auto_resolution_message'),
-          sender: inbox.captain_assistant
-        }
+        message_type: :outgoing,
+        account_id: conversation.account_id,
+        inbox_id: conversation.inbox_id,
+        content: resolution_message.presence || I18n.t('conversations.activity.auto_resolution_message'),
+        sender: inbox.captain_assistant
       )
     end
   end
+
+  def create_handoff_message(conversation, inbox)
+    handoff_message = inbox.captain_assistant.config['handoff_message']
+    return if handoff_message.blank?
+
+    conversation.messages.create!(
+      message_type: :outgoing,
+      sender: inbox.captain_assistant,
+      account_id: conversation.account_id,
+      inbox_id: conversation.inbox_id,
+      content: handoff_message,
+      preserve_waiting_since: true
+    )
+  end
 end

enterprise/app/models/enterprise/activity_message_handler.rb

@@ -6,18 +6,30 @@ module Enterprise::ActivityMessageHandler
     key = captain_activity_key
     return unless key
 
-    I18n.t(key, user_name: Current.executed_by.name, reason: Current.captain_resolve_reason, locale: locale)
+    I18n.t(key, user_name: Current.executed_by.name, reason: captain_status_reason, locale: locale)
   end
 
   private
 
+  def captain_status_reason
+    captain_activity_reason.presence
+  end
+
   def captain_activity_key
-    if resolved? && Current.captain_resolve_reason.present?
-      'conversations.activity.captain.resolved_by_tool'
-    elsif resolved?
-      'conversations.activity.captain.resolved'
-    elsif open?
-      'conversations.activity.captain.open'
-    end
+    return captain_resolved_activity_key if resolved?
+    return captain_open_activity_key if open?
+  end
+
+  def captain_resolved_activity_key
+    return 'conversations.activity.captain.resolved_by_tool' if captain_activity_reason_type == :tool && captain_status_reason.present?
+    return 'conversations.activity.captain.resolved_with_reason' if captain_status_reason.present?
+
+    'conversations.activity.captain.resolved'
+  end
+
+  def captain_open_activity_key
+    return 'conversations.activity.captain.open_with_reason' if captain_status_reason.present?
+
+    'conversations.activity.captain.open'
   end
 end

enterprise/app/models/enterprise/conversation.rb

@@ -1,8 +1,30 @@
 module Enterprise::Conversation
+  attr_accessor :captain_activity_reason, :captain_activity_reason_type
+
+  def dispatch_captain_inference_resolved_event
+    dispatch_captain_inference_event(Events::Types::CONVERSATION_CAPTAIN_INFERENCE_RESOLVED)
+  end
+
+  def dispatch_captain_inference_handoff_event
+    dispatch_captain_inference_event(Events::Types::CONVERSATION_CAPTAIN_INFERENCE_HANDOFF)
+  end
+
   def list_of_keys
     super + %w[sla_policy_id]
   end
 
+  def with_captain_activity_context(reason:, reason_type:)
+    previous_reason = captain_activity_reason
+    previous_reason_type = captain_activity_reason_type
+
+    self.captain_activity_reason = reason
+    self.captain_activity_reason_type = reason_type
+    yield
+  ensure
+    self.captain_activity_reason = previous_reason
+    self.captain_activity_reason_type = previous_reason_type
+  end
+
   # Include select additional_attributes keys (call related) for update events
   def allowed_keys?
     return true if super
@@ -13,4 +35,10 @@ module Enterprise::Conversation
     changed_attr_keys = attrs_change[1].keys
     changed_attr_keys.intersect?(%w[call_status])
   end
+
+  private
+
+  def dispatch_captain_inference_event(event_name)
+    dispatcher_dispatch(event_name)
+  end
 end

enterprise/lib/captain/conversation_completion_schema.rb

@@ -0,0 +1,4 @@
+class Captain::ConversationCompletionSchema < RubyLLM::Schema
+  boolean :complete, description: 'Whether the conversation is complete and can be closed'
+  string :reason, description: 'Brief explanation of why the conversation is complete or incomplete'
+end

enterprise/lib/captain/conversation_completion_service.rb

@@ -0,0 +1,68 @@
+# Evaluates whether a conversation is complete and can be auto-resolved.
+# Used by InboxPendingConversationsResolutionJob to determine if inactive
+# conversations should be resolved or handed off to human agents.
+#
+# NOTE: This service intentionally does NOT count toward Captain usage limits.
+# The response excludes the :message key that Enterprise::Captain::BaseTaskService
+# checks for usage tracking. This is an internal operational evaluation,
+# not a customer-facing value-add, so we don't charge for it.
+class Captain::ConversationCompletionService < Captain::BaseTaskService
+  RESPONSE_SCHEMA = Captain::ConversationCompletionSchema
+
+  pattr_initialize [:account!, :conversation_display_id!]
+
+  def perform
+    content = format_messages_as_string
+    return default_incomplete_response('No messages found') if content.blank?
+
+    response = make_api_call(
+      model: InstallationConfig.find_by(name: 'CAPTAIN_OPEN_AI_MODEL')&.value.presence || GPT_MODEL,
+      messages: [
+        { role: 'system', content: prompt_from_file('conversation_completion') },
+        { role: 'user', content: content }
+      ],
+      schema: RESPONSE_SCHEMA
+    )
+
+    return default_incomplete_response(response[:error]) if response[:error].present?
+
+    parse_response(response[:message])
+  end
+
+  private
+
+  def prompt_from_file(file_name)
+    Rails.root.join('enterprise/lib/captain/prompts', "#{file_name}.liquid").read
+  end
+
+  def format_messages_as_string
+    messages = conversation_messages(start_from: 0)
+    messages.map do |msg|
+      sender_type = msg[:role] == 'user' ? 'Customer' : 'Assistant'
+      "#{sender_type}: #{msg[:content]}"
+    end.join("\n")
+  end
+
+  def parse_response(message)
+    return default_incomplete_response('Invalid response format') unless message.is_a?(Hash)
+
+    {
+      complete: message['complete'] == true,
+      reason: message['reason'] || 'No reason provided'
+    }
+  end
+
+  def default_incomplete_response(reason)
+    { complete: false, reason: reason }
+  end
+
+  def event_name
+    'captain.conversation_completion'
+  end
+
+  def build_follow_up_context?
+    false
+  end
+end
+
+Captain::ConversationCompletionService.prepend_mod_with('Captain::ConversationCompletionService')

enterprise/lib/captain/prompts/conversation_completion.liquid

@@ -0,0 +1,21 @@
+You are evaluating whether a customer support conversation is complete and can be safely closed. When in doubt, keep the conversation OPEN. It is far better to hand off to a human agent unnecessarily than to close a conversation where the customer still needs help.
+
+The conversation may be in any language. Apply these criteria based on the intent and meaning of messages, regardless of language.
+
+A conversation is INCOMPLETE (keep open) if ANY of these apply:
+- The assistant suggested the customer try something or take an action — they may still be attempting it
+- The assistant directed the customer to an external resource, link, or contact — they may still be following up
+- The assistant asked a question or requested information that the customer hasn't provided
+- The customer asked a question that wasn't fully answered
+- The customer asked for something the assistant couldn't do — even if the assistant explained why, the customer's need is unmet
+- The customer raised multiple questions or issues and not all were addressed
+
+A conversation is COMPLETE only if ALL of these are true:
+- The assistant's answer fully addressed the customer's question or issue and is self-contained — it requires no further action from the customer
+- There are no unanswered questions, unmet requests, or outstanding follow-ups from either side
+- Note: customers often do not explicitly say thanks or confirm resolution. If the assistant gave a complete, self-contained answer and the customer had no follow-up, that is sufficient. Do not require explicit gratitude or confirmation.
+
+Analyze the conversation and respond with ONLY a JSON object (no other text):
+{"complete": true, "reason": "brief explanation"}
+or
+{"complete": false, "reason": "brief explanation"}

enterprise/lib/captain/tools/resolve_conversation_tool.rb

@@ -10,12 +10,7 @@ class Captain::Tools::ResolveConversationTool < Captain::Tools::BasePublicTool
 
     log_tool_usage('resolve_conversation', { conversation_id: conversation.id, reason: reason })
 
-    Current.captain_resolve_reason = reason
-    begin
-      conversation.resolved!
-    ensure
-      Current.captain_resolve_reason = nil
-    end
+    conversation.with_captain_activity_context(reason: reason, reason_type: :tool) { conversation.resolved! }
 
     "Conversation ##{conversation.display_id} resolved#{" (Reason: #{reason})" if reason}"
   end

enterprise/lib/enterprise/captain/conversation_completion_service.rb

@@ -0,0 +1,12 @@
+# Overrides the quota check from Enterprise::Captain::BaseTaskService
+# so that conversation completion evaluation always runs regardless of
+# the customer's Captain usage quota. This is an internal operational
+# check, not a customer-facing feature — it should never be blocked
+# by quota exhaustion.
+module Enterprise::Captain::ConversationCompletionService
+  private
+
+  def responses_available?
+    true
+  end
+end