feat(ee): Add Captain features (#10665)

Migration Guide: https://chwt.app/v4/migration

This PR imports all the work related to Captain into the EE codebase. Captain represents the AI-based features in Chatwoot and includes the following key components:

- Assistant: An assistant has a persona, the product it would be trained on. At the moment, the data at which it is trained is from websites. Future integrations on Notion documents, PDF etc. This PR enables connecting an assistant to an inbox. The assistant would run the conversation every time before transferring it to an agent.
- Copilot for Agents: When an agent is supporting a customer, we will be able to offer additional help to lookup some data or fetch information from integrations etc via copilot.
- Conversation FAQ generator: When a conversation is resolved, the Captain integration would identify questions which were not in the knowledge base.
- CRM memory: Learns from the conversations and identifies important information about the contact.

---------

Co-authored-by: Vishnu Narayanan <vishnu@chatwoot.com>
Co-authored-by: Sojan <sojan@pepalo.com>
Co-authored-by: iamsivin <iamsivin@gmail.com>
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>

enterprise/app/services/captain/tools/firecrawl_service.rb

@@ -0,0 +1,40 @@
+class Captain::Tools::FirecrawlService
+  def initialize
+    @api_key = InstallationConfig.find_by!(name: 'CAPTAIN_FIRECRAWL_API_KEY').value
+    raise 'Missing API key' if @api_key.nil?
+  end
+
+  def perform(url, webhook_url = '')
+    HTTParty.post(
+      'https://api.firecrawl.dev/v1/crawl',
+      body: crawl_payload(url, webhook_url),
+      headers: headers
+    )
+  rescue StandardError => e
+    raise "Failed to crawl URL: #{e.message}"
+  end
+
+  private
+
+  def crawl_payload(url, webhook_url)
+    {
+      url: url,
+      maxDepth: 50,
+      ignoreSitemap: false,
+      limit: 10,
+      webhook: webhook_url,
+      scrapeOptions: {
+        onlyMainContent: false,
+        formats: ['markdown'],
+        excludeTags: ['iframe']
+      }
+    }.to_json
+  end
+
+  def headers
+    {
+      'Authorization' => "Bearer #{@api_key}",
+      'Content-Type' => 'application/json'
+    }
+  end
+end

enterprise/app/services/captain/tools/simple_page_crawl_service.rb

@@ -0,0 +1,38 @@
+class Captain::Tools::SimplePageCrawlService
+  attr_reader :external_link
+
+  def initialize(external_link)
+    @external_link = external_link
+    @doc = Nokogiri::HTML(HTTParty.get(external_link).body)
+  end
+
+  def page_links
+    sitemap? ? extract_links_from_sitemap : extract_links_from_html
+  end
+
+  def page_title
+    title_element = @doc.at_xpath('//title')
+    title_element&.text&.strip
+  end
+
+  def body_text_content
+    ReverseMarkdown.convert @doc.at_xpath('//body'), unknown_tags: :bypass, github_flavored: true
+  end
+
+  private
+
+  def sitemap?
+    @external_link.end_with?('.xml')
+  end
+
+  def extract_links_from_sitemap
+    @doc.xpath('//loc').to_set(&:text)
+  end
+
+  def extract_links_from_html
+    @doc.xpath('//a/@href').to_set do |link|
+      absolute_url = URI.join(@external_link, link.value).to_s
+      absolute_url
+    end
+  end
+end