feat: Advanced Search Backend (#12917)

## Description Implements comprehensive search functionality with advanced filtering capabilities for Chatwoot (Linear: CW-5956). This PR adds: 1. **Time-based filtering** for contacts and conversations (SQL-based search) 2. **Advanced message search** with multiple filters (OpenSearch/Elasticsearch-based) - **`from` filter**: Filter messages by sender (format: `contact:42` or `agent:5`) - **`inbox_id` filter**: Filter messages by specific inbox - **Time range filters**: Filter messages using `since` and `until` parameters (Unix timestamps in seconds) - **90-day limit enforcement**: Automatically limits searches to the last 90 days to prevent performance issues The implementation extends the existing `Enterprise::SearchService` module for advanced features and adds time filtering to the base `SearchService` for SQL-based searches. ## API Documentation ### Base URL All search endpoints follow this pattern: ``` GET /api/v1/accounts/{account_id}/search/{resource} ``` ### Authentication All requests require authentication headers: ``` api_access_token: YOUR_ACCESS_TOKEN ``` --- ## 1. Search All Resources **Endpoint:** `GET /api/v1/accounts/{account_id}/search` Returns results from all searchable resources (contacts, conversations, messages, articles). ### Parameters | Parameter | Type | Description | Required | |-----------|------|-------------|----------| | `q` | string | Search query | Yes | | `page` | integer | Page number (15 items per page) | No | | `since` | integer | Unix timestamp (contacts/conversations only) | No | | `until` | integer | Unix timestamp (contacts/conversations only) | No | ### Example Request ```bash curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search?q=customer" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` ### Example Response ```json { "payload": { "contacts": [...], "conversations": [...], "messages": [...], "articles": [...] } } ``` --- ## 2. Search Contacts **Endpoint:** `GET /api/v1/accounts/{account_id}/search/contacts` Search contacts by name, email, phone number, or identifier with optional time filtering. ### Parameters | Parameter | Type | Description | Required | |-----------|------|-------------|----------| | `q` | string | Search query | Yes | | `page` | integer | Page number (15 items per page) | No | | `since` | integer | Unix timestamp - filter by last_activity_at | No | | `until` | integer | Unix timestamp - filter by last_activity_at | No | ### Example Requests **Basic search:** ```bash curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/contacts?q=john" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Search contacts active in the last 7 days:** ```bash SINCE=$(date -v-7d +%s) curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/contacts?q=john&since=${SINCE}" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Search contacts active between 30 and 7 days ago:** ```bash SINCE=$(date -v-30d +%s) UNTIL=$(date -v-7d +%s) curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/contacts?q=john&since=${SINCE}&until=${UNTIL}" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` ### Example Response ```json { "payload": { "contacts": [ { "id": 42, "email": "john@example.com", "name": "John Doe", "phone_number": "+1234567890", "identifier": "user_123", "additional_attributes": {}, "created_at": 1701234567 } ] } } ``` --- ## 3. Search Conversations **Endpoint:** `GET /api/v1/accounts/{account_id}/search/conversations` Search conversations by display ID, contact name, email, phone number, or identifier with optional time filtering. ### Parameters | Parameter | Type | Description | Required | |-----------|------|-------------|----------| | `q` | string | Search query | Yes | | `page` | integer | Page number (15 items per page) | No | | `since` | integer | Unix timestamp - filter by last_activity_at | No | | `until` | integer | Unix timestamp - filter by last_activity_at | No | ### Example Requests **Basic search:** ```bash curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/conversations?q=billing" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Search conversations active in the last 24 hours:** ```bash SINCE=$(date -v-1d +%s) curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/conversations?q=billing&since=${SINCE}" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Search conversations from last month:** ```bash SINCE=$(date -v-30d +%s) UNTIL=$(date +%s) curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/conversations?q=billing&since=${SINCE}&until=${UNTIL}" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` ### Example Response ```json { "payload": { "conversations": [ { "id": 123, "display_id": 45, "inbox_id": 1, "status": "open", "messages": [...], "meta": {...} } ] } } ``` --- ## 4. Search Messages (Advanced) **Endpoint:** `GET /api/v1/accounts/{account_id}/search/messages` Advanced message search with multiple filters powered by OpenSearch/Elasticsearch. ### Prerequisites - OpenSearch/Elasticsearch must be running (`OPENSEARCH_URL` env var configured) - Account must have `advanced_search` feature flag enabled - Messages must be indexed in OpenSearch ### Parameters | Parameter | Type | Description | Required | |-----------|------|-------------|----------| | `q` | string | Search query | Yes | | `page` | integer | Page number (15 items per page) | No | | `from` | string | Filter by sender: `contact:{id}` or `agent:{id}` | No | | `inbox_id` | integer | Filter by specific inbox ID | No | | `since` | integer | Unix timestamp - searches from this time (max 90 days ago) | No | | `until` | integer | Unix timestamp - searches until this time | No | ### Important Notes - **90-Day Limit**: If `since` is not provided, searches default to the last 90 days - If `since` exceeds 90 days, returns `422` error: "Search is limited to the last 90 days" - All time filters use message `created_at` timestamp ### Example Requests **Basic message search:** ```bash curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Search messages from a specific contact:** ```bash curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&from=contact:42" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Search messages from a specific agent:** ```bash curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&from=agent:5" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Search messages in a specific inbox:** ```bash curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&inbox_id=3" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Search messages from the last 7 days:** ```bash SINCE=$(date -v-7d +%s) curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&since=${SINCE}" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Search messages between specific dates:** ```bash SINCE=$(date -v-30d +%s) UNTIL=$(date -v-7d +%s) curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&since=${SINCE}&until=${UNTIL}" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Combine all filters:** ```bash SINCE=$(date -v-14d +%s) curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&from=contact:42&inbox_id=3&since=${SINCE}" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` **Attempt to search beyond 90 days (returns error):** ```bash SINCE=$(date -v-120d +%s) curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&since=${SINCE}" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` ### Example Response (Success) ```json { "payload": { "messages": [ { "id": 789, "content": "I need a refund for my purchase", "message_type": "incoming", "created_at": 1701234567, "conversation_id": 123, "inbox_id": 3, "sender": { "id": 42, "type": "contact" } } ] } } ``` ### Example Response (90-day limit exceeded) ```json { "error": "Search is limited to the last 90 days" } ``` **Status Code:** `422 Unprocessable Entity` --- ## 5. Search Articles **Endpoint:** `GET /api/v1/accounts/{account_id}/search/articles` Search help center articles by title or content. ### Parameters | Parameter | Type | Description | Required | |-----------|------|-------------|----------| | `q` | string | Search query | Yes | | `page` | integer | Page number (15 items per page) | No | ### Example Request ```bash curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/articles?q=installation" \ -H "api_access_token: YOUR_ACCESS_TOKEN" ``` ### Example Response ```json { "payload": { "articles": [ { "id": 456, "title": "Installation Guide", "slug": "installation-guide", "portal_slug": "help", "account_id": 1, "category_name": "Getting Started", "status": "published", "updated_at": 1701234567 } ] } } ``` --- ## Technical Implementation ### SQL-Based Search (Contacts, Conversations, Articles) - Uses PostgreSQL `ILIKE` queries by default - Optional GIN index support via `search_with_gin` feature flag for better performance - Time filtering uses `last_activity_at` for contacts/conversations - Returns paginated results (15 per page) ### Advanced Search (Messages) - Powered by OpenSearch/Elasticsearch via Searchkick gem - Requires `OPENSEARCH_URL` environment variable - Requires `advanced_search` account feature flag - Enforces 90-day lookback limit via `Limits::MESSAGE_SEARCH_TIME_RANGE_LIMIT_DAYS` - Validates inbox access permissions before filtering - Returns paginated results (15 per page) --- ## Type of change - [x] New feature (non-breaking change which adds functionality) - [x] Enhancement (improves existing functionality) --- ## How Has This Been Tested? ### Unit Tests - **Contact Search Tests**: 3 new test cases for time filtering (`since`, `until`, combined) - **Conversation Search Tests**: 3 new test cases for time filtering - **Message Search Tests**: 10+ test cases covering: - Individual filters (`from`, `inbox_id`, time range) - Combined filters - Permission validation for inbox access - Feature flag checks - 90-day limit enforcement - Error handling for exceeded time limits ### Test Commands ```bash # Run all search controller tests bundle exec rspec spec/controllers/api/v1/accounts/search_controller_spec.rb # Run search service tests (includes enterprise specs) bundle exec rspec spec/services/search_service_spec.rb ``` ### Manual Testing Setup A rake task is provided to create 50,000 test messages across multiple inboxes: ```bash # 1. Create test data bundle exec rake search:setup_test_data # 2. Start OpenSearch mise elasticsearch-start # 3. Reindex messages rails runner "Message.search_index.import Message.all" # 4. Enable feature flag rails runner "Account.first.enable_features('advanced_search')" # 5. Test via API or Rails console ``` --- ## Checklist - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (this PR description) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules --- ## Additional Notes ### Requirements - **OpenSearch/Elasticsearch**: Required for advanced message search - Set `OPENSEARCH_URL` environment variable - Example: `export OPENSEARCH_URL=http://localhost:9200` - **Feature Flags**: - `advanced_search`: Account-level flag for message advanced search - `search_with_gin` (optional): Account-level flag for GIN-based SQL search ### Performance Considerations - 90-day limit prevents expensive long-range queries on large datasets - GIN indexes recommended for high-volume search on SQL-based resources - OpenSearch/Elasticsearch provides faster full-text search for messages ### Breaking Changes - None. All new parameters are optional and backward compatible. ### Frontend Integration - Frontend PR tracking advanced search UI will consume these endpoints - Time range pickers should convert JavaScript `Date` to Unix timestamps (seconds) - Date conversion: `Math.floor(date.getTime() / 1000)` ### Error Handling - Invalid `from` parameter format is silently ignored (filter not applied) - Time range exceeding 90 days returns `422` with error message - Missing `q` parameter returns `422` (existing behavior) - Unauthorized inbox access is filtered out (no error, just excluded from results) --------- Co-authored-by: iamsivin <iamsivin@gmail.com> Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com> Co-authored-by: Pranav <pranav@chatwoot.com> Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
2026-01-07 15:30:49 +05:30
parent 566de02385
commit 59cbf57e20
49 changed files with 3816 additions and 385 deletions
--- a/app/javascript/shared/composables/specs/useExpandableContent.spec.js
+++ b/app/javascript/shared/composables/specs/useExpandableContent.spec.js
@@ -0,0 +1,130 @@
+import { nextTick } from 'vue';
+import { useExpandableContent } from '../useExpandableContent';
+
+// Mock VueUse composables
+vi.mock('@vueuse/core', () => ({
+  useToggle: vi.fn(initialValue => {
+    let value = initialValue;
+    const toggle = newValue => {
+      value = newValue !== undefined ? newValue : !value;
+    };
+    return [{ value }, toggle];
+  }),
+  useResizeObserver: vi.fn((element, callback) => {
+    // Store callback for manual triggering in tests
+    if (element.value) {
+      callback();
+    }
+  }),
+}));
+
+describe('useExpandableContent', () => {
+  let originalGetComputedStyle;
+
+  beforeEach(() => {
+    // Mock window.getComputedStyle
+    originalGetComputedStyle = window.getComputedStyle;
+    window.getComputedStyle = vi.fn(() => ({
+      lineHeight: '20px',
+    }));
+  });
+
+  afterEach(() => {
+    window.getComputedStyle = originalGetComputedStyle;
+    vi.clearAllMocks();
+  });
+
+  it('returns expected properties', () => {
+    const result = useExpandableContent();
+
+    expect(result).toHaveProperty('contentElement');
+    expect(result).toHaveProperty('isExpanded');
+    expect(result).toHaveProperty('needsToggle');
+    expect(result).toHaveProperty('toggleExpanded');
+    expect(result).toHaveProperty('checkOverflow');
+  });
+
+  it('initializes with default values', () => {
+    const { isExpanded, needsToggle } = useExpandableContent();
+
+    expect(isExpanded.value).toBe(false);
+    expect(needsToggle.value).toBe(false);
+  });
+
+  it('checkOverflow sets needsToggle to true when content overflows', async () => {
+    const { contentElement, needsToggle, checkOverflow } =
+      useExpandableContent();
+
+    // Mock element with overflow
+    contentElement.value = {
+      scrollHeight: 100, // Much larger than 2 lines (40px)
+    };
+
+    checkOverflow();
+    await nextTick();
+
+    expect(needsToggle.value).toBe(true);
+  });
+
+  it('checkOverflow sets needsToggle to false when content fits', async () => {
+    const { contentElement, needsToggle, checkOverflow } =
+      useExpandableContent();
+
+    // Mock element without overflow
+    contentElement.value = {
+      scrollHeight: 30, // Less than 2 lines (40px)
+    };
+
+    checkOverflow();
+    await nextTick();
+
+    expect(needsToggle.value).toBe(false);
+  });
+
+  it('respects custom maxLines option', async () => {
+    const { contentElement, needsToggle, checkOverflow } = useExpandableContent(
+      {
+        maxLines: 3,
+      }
+    );
+
+    // Mock element that fits in 3 lines but not 2
+    contentElement.value = {
+      scrollHeight: 50, // Fits in 3 lines (60px) but not 2 lines (40px)
+    };
+
+    checkOverflow();
+    await nextTick();
+
+    expect(needsToggle.value).toBe(false);
+  });
+
+  it('uses defaultLineHeight when computed style is unavailable', async () => {
+    window.getComputedStyle = vi.fn(() => ({
+      lineHeight: 'normal', // Not a valid number
+    }));
+
+    const { contentElement, needsToggle, checkOverflow } = useExpandableContent(
+      {
+        defaultLineHeight: 16,
+      }
+    );
+
+    // Mock element that overflows with 16px line height (32px max)
+    contentElement.value = {
+      scrollHeight: 40,
+    };
+
+    checkOverflow();
+    await nextTick();
+
+    expect(needsToggle.value).toBe(true);
+  });
+
+  it('handles null contentElement gracefully', () => {
+    const { checkOverflow } = useExpandableContent();
+
+    // Should not throw when contentElement is null
+    expect(() => checkOverflow()).not.toThrow();
+  });
+});
--- a/app/javascript/shared/composables/useExpandableContent.js
+++ b/app/javascript/shared/composables/useExpandableContent.js
@@ -0,0 +1,62 @@
+import { ref, computed, nextTick, onMounted } from 'vue';
+import { useToggle, useResizeObserver } from '@vueuse/core';
+
+/**
+ * Composable for handling expandable content with "Read more / Read less" functionality.
+ * Detects content overflow and provides toggle state for expansion.
+ *
+ * @param {Object} options - Configuration options
+ * @param {number} [options.maxLines=2] - Maximum number of lines before showing toggle
+ * @param {number} [options.defaultLineHeight=20] - Fallback line height if computed style is unavailable
+ * @param {boolean} [options.useResizeObserverForCheck=false] - Use ResizeObserver for continuous overflow checking
+ * @returns {Object} - Composable state and methods
+ */
+export function useExpandableContent(options = {}) {
+  const {
+    maxLines = 2,
+    defaultLineHeight = 20,
+    useResizeObserverForCheck = false,
+  } = options;
+
+  const contentElement = ref(null);
+  const [isExpanded, toggleExpanded] = useToggle(false);
+  const needsToggle = ref(false);
+
+  const showReadMore = computed(() => needsToggle.value && !isExpanded.value);
+  const showReadLess = computed(() => needsToggle.value && isExpanded.value);
+
+  /**
+   * Checks if content overflows the maximum allowed height
+   * and updates needsToggle accordingly
+   */
+  const checkOverflow = () => {
+    if (!contentElement.value) return;
+
+    const element = contentElement.value;
+    const computedStyle = window.getComputedStyle(element);
+    const lineHeight =
+      parseFloat(computedStyle.lineHeight) || defaultLineHeight;
+    const maxHeight = lineHeight * maxLines;
+
+    needsToggle.value = element.scrollHeight > maxHeight;
+  };
+
+  // Setup overflow checking based on configuration
+  if (useResizeObserverForCheck) {
+    useResizeObserver(contentElement, checkOverflow);
+  } else {
+    onMounted(() => {
+      nextTick(checkOverflow);
+    });
+  }
+
+  return {
+    contentElement,
+    isExpanded,
+    needsToggle,
+    showReadMore,
+    showReadLess,
+    toggleExpanded,
+    checkOverflow,
+  };
+}