Commit Graph

58 Commits

Author SHA1 Message Date
Shivam Mishra
9a9398b386 feat: validate OpenAPI spec using Skooma (#13623)
Adds Skooma-based OpenAPI validation so SDK-facing request specs can
assert that documented request and response contracts match real Rails
behavior. This also upgrades the spec to OpenAPI 3.1 and fixes contract
drift uncovered while validating core application and platform
resources.

Closes
None

Why
We want CI to catch OpenAPI drift before it reaches SDK consumers. While
wiring validation in, this PR surfaced several mismatches between the
documented contract and what the Rails endpoints actually accept or
return.

What this change does
- Adds Skooma-backed OpenAPI validation to the request spec flow and a
dedicated OpenAPI validation spec.
- Migrates nullable schema definitions to OpenAPI 3.1-compatible unions.
- Updates core SDK-facing schemas and payloads across accounts,
contacts, conversations, inboxes, messages, teams, reporting events, and
platform account resources.
- Documents concrete runtime cases that were previously missing or
inaccurate, including nested `profile` update payloads, multipart avatar
uploads, required profile update bodies, nullable inbox feature flags,
and message sender types that include both `Captain::Assistant` and
senderless activity-style messages.
- Regenerates the committed Swagger JSON and tag-group artifacts used by
CI sync checks.

Validation
- `bundle exec rake swagger:build`
- `bundle exec rspec spec/swagger/openapi_spec.rb`

---------

Co-authored-by: Sojan Jose <sojan@pepalo.com>
2026-03-10 18:33:55 -07:00
Pranav
6a7cbcf5ba fix: Fixes reply-to in WhatsApp Cloud API (#13467)
This change https://github.com/chatwoot/chatwoot/pull/13371 broke the
functionality. When a user replies to a WhatsApp message, the reply
context wasn't being properly stored in Chatwoot due to #13371

WhatsApp sends reply messages with a `context` field containing the
original message ID:
```json
{
    "messages": [{
      "context": {
        "from": "phone_number",
        "id": "wamid.ORIGINAL_MESSAGE_ID"
      },
      "from": "phone_number",
      "id": "wamid.REPLY_MESSAGE_ID",
      "text": { "body": "This is a reply" }
    }]
  }
```
However, the in_reply_to_external_id was being overridden when building
the message because content_attributes was explicitly set to either {
external_echo: true } or {}, which discarded the reply-to information.
2026-02-06 14:01:01 -08:00
Sojan Jose
9eb3ee44a8 Revert "chore: Upgrade Rails to 7.2.2 and update Gemfile dependencies (#11037)"
This reverts commit ef6ba8aabd.
2026-02-03 21:09:42 -08:00
Sojan Jose
ef6ba8aabd chore: Upgrade Rails to 7.2.2 and update Gemfile dependencies (#11037)
Upgrade rails to 7.2.2 so that we can proceed with the rails 8 upgrade
afterwards
 
 # Changelog
- `.circleci/config.yml` — align CI DB setup with GitHub Actions
(`db:create` + `db:schema:load`) to avoid trigger-dependent prep steps.
- `.rubocop.yml` — add `rubocop-rspec_rails` and disable new cops that
don't match existing spec style.
- `AGENTS.md` — document that specs should run without `.env` (rename
temporarily when present).
- `Gemfile` — upgrade to Rails 7.2, switch Azure storage gem, pin
`commonmarker`, bump `sidekiq-cron`, add `rubocop-rspec_rails`, and
relax some gem pins.
- `Gemfile.lock` — dependency lockfile updates from the Rails 7.2 and
gem changes.
- `app/controllers/api/v1/accounts/integrations/linear_controller.rb` —
stringify params before passing to the Linear service to keep key types
stable.
- `app/controllers/super_admin/instance_statuses_controller.rb` — use
`MigrationContext` API for migration status in Rails 7.2.
- `app/models/installation_config.rb` — add commentary on YAML
serialization and future JSONB migration (no behavior change).
- `app/models/integrations/hook.rb` — ensure hook type is set on create
only and guard against missing app.
- `app/models/user.rb` — update enum syntax for Rails 7.2 deprecation,
serialize OTP backup codes with JSON, and use Ruby `alias`.
- `app/services/crm/leadsquared/setup_service.rb` — stringify hook
settings keys before merge to keep JSON shape consistent.
- `app/services/macros/execution_service.rb` — remove macro-specific
assignee activity workaround; rely on standard assignment handlers.
- `config/application.rb` — load Rails 7.2 defaults.
- `config/storage.yml` — update Azure Active Storage service name to
`AzureBlob`.
- `db/migrate/20230515051424_update_article_image_keys.rb` — use
credentials `secret_key_base` with fallback to legacy secrets.
- `docker/Dockerfile` — add `yaml-dev` and `pkgconf` packages for native
extensions (Ruby 3.4 / psych).
- `lib/seeders/reports/message_creator.rb` — add parentheses for clarity
in range calculation.
- `package.json` — pin Vite version and bump `vite-plugin-ruby`.
- `pnpm-lock.yaml` — lockfile changes from JS dependency updates.
- `spec/builders/v2/report_builder_spec.rb` — disable transactional
fixtures; truncate tables per example via Rails `truncate_tables` so
after_commit callbacks run with clean isolation; keep builder spec
metadata minimal.
- `spec/builders/v2/reports/label_summary_builder_spec.rb` — disable
transactional fixtures + truncate tables via Rails `truncate_tables`;
revert to real `resolved!`/`open!`/`resolved!` flow for multiple
resolution events; align date range to `Time.zone` to avoid offset gaps;
keep builder spec metadata minimal.
- `spec/controllers/api/v1/accounts/macros_controller_spec.rb` — assert
`assignee_id` instead of activity message to avoid transaction-timing
flakes.
- `spec/services/telegram/incoming_message_service_spec.rb` — reference
the contact tied to the created conversation instead of
`Contact.all.first` to avoid order-dependent failures when other specs
leave data behind.
-
`spec/mailers/administrator_notifications/shared/smtp_config_shared.rb`
— use `with_modified_env` instead of stubbing mailer internals.
- `spec/services/account/sign_up_email_validation_service_spec.rb` —
compare error `class.name` for parallel/reload-safe assertions.
2026-02-03 14:29:26 -08:00
Pranav
e9e6de5690 fix: Increase the parallelism config to fix flaky tests, revert bad commits (#13410)
The specs break only in Circle CI, we have to figure out the root cause
for the same. At the moment, I have increased the parallelism to fix
this.
2026-01-30 12:49:31 -08:00
Vishnu Narayanan
7d68c25c97 chore: upgrade node to v24.13 (#13291)
- upgrade node to v24.13

Co-authored-by: Sojan Jose <sojan@pepalo.com>
2026-01-15 20:28:13 -08:00
Vishnu Narayanan
ab83a663f0 chore: upgrade node to 24.x LTS (#13004)
Upgrade node to `24.12` LTS

FIxes https://github.com/chatwoot/chatwoot/issues/12546
Fixes https://linear.app/chatwoot/issue/CW-5707/nodejs-23-reached-eol
2026-01-12 18:10:23 +05:30
Vinay Keerthi
59cbf57e20 feat: Advanced Search Backend (#12917)
## Description

Implements comprehensive search functionality with advanced filtering
capabilities for Chatwoot (Linear: CW-5956).

This PR adds:
1. **Time-based filtering** for contacts and conversations (SQL-based
search)
2. **Advanced message search** with multiple filters
(OpenSearch/Elasticsearch-based)
- **`from` filter**: Filter messages by sender (format: `contact:42` or
`agent:5`)
   - **`inbox_id` filter**: Filter messages by specific inbox
- **Time range filters**: Filter messages using `since` and `until`
parameters (Unix timestamps in seconds)
- **90-day limit enforcement**: Automatically limits searches to the
last 90 days to prevent performance issues

The implementation extends the existing `Enterprise::SearchService`
module for advanced features and adds time filtering to the base
`SearchService` for SQL-based searches.

## API Documentation

### Base URL
All search endpoints follow this pattern:
```
GET /api/v1/accounts/{account_id}/search/{resource}
```

### Authentication
All requests require authentication headers:
```
api_access_token: YOUR_ACCESS_TOKEN
```

---

## 1. Search All Resources

**Endpoint:** `GET /api/v1/accounts/{account_id}/search`

Returns results from all searchable resources (contacts, conversations,
messages, articles).

### Parameters
| Parameter | Type | Description | Required |
|-----------|------|-------------|----------|
| `q` | string | Search query | Yes |
| `page` | integer | Page number (15 items per page) | No |
| `since` | integer | Unix timestamp (contacts/conversations only) | No
|
| `until` | integer | Unix timestamp (contacts/conversations only) | No
|

### Example Request
```bash
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search?q=customer" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

### Example Response
```json
{
  "payload": {
    "contacts": [...],
    "conversations": [...],
    "messages": [...],
    "articles": [...]
  }
}
```

---

## 2. Search Contacts

**Endpoint:** `GET /api/v1/accounts/{account_id}/search/contacts`

Search contacts by name, email, phone number, or identifier with
optional time filtering.

### Parameters
| Parameter | Type | Description | Required |
|-----------|------|-------------|----------|
| `q` | string | Search query | Yes |
| `page` | integer | Page number (15 items per page) | No |
| `since` | integer | Unix timestamp - filter by last_activity_at | No |
| `until` | integer | Unix timestamp - filter by last_activity_at | No |

### Example Requests

**Basic search:**
```bash
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/contacts?q=john" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Search contacts active in the last 7 days:**
```bash
SINCE=$(date -v-7d +%s)
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/contacts?q=john&since=${SINCE}" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Search contacts active between 30 and 7 days ago:**
```bash
SINCE=$(date -v-30d +%s)
UNTIL=$(date -v-7d +%s)
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/contacts?q=john&since=${SINCE}&until=${UNTIL}" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

### Example Response
```json
{
  "payload": {
    "contacts": [
      {
        "id": 42,
        "email": "john@example.com",
        "name": "John Doe",
        "phone_number": "+1234567890",
        "identifier": "user_123",
        "additional_attributes": {},
        "created_at": 1701234567
      }
    ]
  }
}
```

---

## 3. Search Conversations

**Endpoint:** `GET /api/v1/accounts/{account_id}/search/conversations`

Search conversations by display ID, contact name, email, phone number,
or identifier with optional time filtering.

### Parameters
| Parameter | Type | Description | Required |
|-----------|------|-------------|----------|
| `q` | string | Search query | Yes |
| `page` | integer | Page number (15 items per page) | No |
| `since` | integer | Unix timestamp - filter by last_activity_at | No |
| `until` | integer | Unix timestamp - filter by last_activity_at | No |

### Example Requests

**Basic search:**
```bash
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/conversations?q=billing" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Search conversations active in the last 24 hours:**
```bash
SINCE=$(date -v-1d +%s)
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/conversations?q=billing&since=${SINCE}" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Search conversations from last month:**
```bash
SINCE=$(date -v-30d +%s)
UNTIL=$(date +%s)
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/conversations?q=billing&since=${SINCE}&until=${UNTIL}" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

### Example Response
```json
{
  "payload": {
    "conversations": [
      {
        "id": 123,
        "display_id": 45,
        "inbox_id": 1,
        "status": "open",
        "messages": [...],
        "meta": {...}
      }
    ]
  }
}
```

---

## 4. Search Messages (Advanced)

**Endpoint:** `GET /api/v1/accounts/{account_id}/search/messages`

Advanced message search with multiple filters powered by
OpenSearch/Elasticsearch.

### Prerequisites
- OpenSearch/Elasticsearch must be running (`OPENSEARCH_URL` env var
configured)
- Account must have `advanced_search` feature flag enabled
- Messages must be indexed in OpenSearch

### Parameters
| Parameter | Type | Description | Required |
|-----------|------|-------------|----------|
| `q` | string | Search query | Yes |
| `page` | integer | Page number (15 items per page) | No |
| `from` | string | Filter by sender: `contact:{id}` or `agent:{id}` |
No |
| `inbox_id` | integer | Filter by specific inbox ID | No |
| `since` | integer | Unix timestamp - searches from this time (max 90
days ago) | No |
| `until` | integer | Unix timestamp - searches until this time | No |

### Important Notes
- **90-Day Limit**: If `since` is not provided, searches default to the
last 90 days
- If `since` exceeds 90 days, returns `422` error: "Search is limited to
the last 90 days"
- All time filters use message `created_at` timestamp

### Example Requests

**Basic message search:**
```bash
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Search messages from a specific contact:**
```bash
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&from=contact:42" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Search messages from a specific agent:**
```bash
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&from=agent:5" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Search messages in a specific inbox:**
```bash
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&inbox_id=3" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Search messages from the last 7 days:**
```bash
SINCE=$(date -v-7d +%s)
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&since=${SINCE}" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Search messages between specific dates:**
```bash
SINCE=$(date -v-30d +%s)
UNTIL=$(date -v-7d +%s)
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&since=${SINCE}&until=${UNTIL}" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Combine all filters:**
```bash
SINCE=$(date -v-14d +%s)
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&from=contact:42&inbox_id=3&since=${SINCE}" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

**Attempt to search beyond 90 days (returns error):**
```bash
SINCE=$(date -v-120d +%s)
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/messages?q=refund&since=${SINCE}" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

### Example Response (Success)
```json
{
  "payload": {
    "messages": [
      {
        "id": 789,
        "content": "I need a refund for my purchase",
        "message_type": "incoming",
        "created_at": 1701234567,
        "conversation_id": 123,
        "inbox_id": 3,
        "sender": {
          "id": 42,
          "type": "contact"
        }
      }
    ]
  }
}
```

### Example Response (90-day limit exceeded)
```json
{
  "error": "Search is limited to the last 90 days"
}
```
**Status Code:** `422 Unprocessable Entity`

---

## 5. Search Articles

**Endpoint:** `GET /api/v1/accounts/{account_id}/search/articles`

Search help center articles by title or content.

### Parameters
| Parameter | Type | Description | Required |
|-----------|------|-------------|----------|
| `q` | string | Search query | Yes |
| `page` | integer | Page number (15 items per page) | No |

### Example Request
```bash
curl -X GET "https://app.chatwoot.com/api/v1/accounts/1/search/articles?q=installation" \
  -H "api_access_token: YOUR_ACCESS_TOKEN"
```

### Example Response
```json
{
  "payload": {
    "articles": [
      {
        "id": 456,
        "title": "Installation Guide",
        "slug": "installation-guide",
        "portal_slug": "help",
        "account_id": 1,
        "category_name": "Getting Started",
        "status": "published",
        "updated_at": 1701234567
      }
    ]
  }
}
```

---

## Technical Implementation

### SQL-Based Search (Contacts, Conversations, Articles)
- Uses PostgreSQL `ILIKE` queries by default
- Optional GIN index support via `search_with_gin` feature flag for
better performance
- Time filtering uses `last_activity_at` for contacts/conversations
- Returns paginated results (15 per page)

### Advanced Search (Messages)
- Powered by OpenSearch/Elasticsearch via Searchkick gem
- Requires `OPENSEARCH_URL` environment variable
- Requires `advanced_search` account feature flag
- Enforces 90-day lookback limit via
`Limits::MESSAGE_SEARCH_TIME_RANGE_LIMIT_DAYS`
- Validates inbox access permissions before filtering
- Returns paginated results (15 per page)

---

## Type of change

- [x] New feature (non-breaking change which adds functionality)
- [x] Enhancement (improves existing functionality)

---

## How Has This Been Tested?

### Unit Tests
- **Contact Search Tests**: 3 new test cases for time filtering
(`since`, `until`, combined)
- **Conversation Search Tests**: 3 new test cases for time filtering
- **Message Search Tests**: 10+ test cases covering:
  - Individual filters (`from`, `inbox_id`, time range)
  - Combined filters
  - Permission validation for inbox access
  - Feature flag checks
  - 90-day limit enforcement
  - Error handling for exceeded time limits

### Test Commands
```bash
# Run all search controller tests
bundle exec rspec spec/controllers/api/v1/accounts/search_controller_spec.rb

# Run search service tests (includes enterprise specs)
bundle exec rspec spec/services/search_service_spec.rb
```

### Manual Testing Setup
A rake task is provided to create 50,000 test messages across multiple
inboxes:

```bash
# 1. Create test data
bundle exec rake search:setup_test_data

# 2. Start OpenSearch
mise elasticsearch-start

# 3. Reindex messages
rails runner "Message.search_index.import Message.all"

# 4. Enable feature flag
rails runner "Account.first.enable_features('advanced_search')"

# 5. Test via API or Rails console
```

---

## Checklist

- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [x] I have made corresponding changes to the documentation (this PR
description)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules

---

## Additional Notes

### Requirements
- **OpenSearch/Elasticsearch**: Required for advanced message search
  - Set `OPENSEARCH_URL` environment variable
  - Example: `export OPENSEARCH_URL=http://localhost:9200`
- **Feature Flags**:
  - `advanced_search`: Account-level flag for message advanced search
- `search_with_gin` (optional): Account-level flag for GIN-based SQL
search

### Performance Considerations
- 90-day limit prevents expensive long-range queries on large datasets
- GIN indexes recommended for high-volume search on SQL-based resources
- OpenSearch/Elasticsearch provides faster full-text search for messages

### Breaking Changes
- None. All new parameters are optional and backward compatible.

### Frontend Integration
- Frontend PR tracking advanced search UI will consume these endpoints
- Time range pickers should convert JavaScript `Date` to Unix timestamps
(seconds)
- Date conversion: `Math.floor(date.getTime() / 1000)`

### Error Handling
- Invalid `from` parameter format is silently ignored (filter not
applied)
- Time range exceeding 90 days returns `422` with error message
- Missing `q` parameter returns `422` (existing behavior)
- Unauthorized inbox access is filtered out (no error, just excluded
from results)

---------

Co-authored-by: iamsivin <iamsivin@gmail.com>
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
Co-authored-by: Pranav <pranav@chatwoot.com>
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
2026-01-07 15:30:49 +05:30
Vishnu Narayanan
08b9134486 feat: speed up circleci and github actions (#12849)
# 🚀 Speed up CI/CD test execution with parallelization

## TL;DR

- **Problem**: CI tests took 36-42 minutes per commit, blocking
developer workflow
- **Solution**: Implemented 16-way parallelization + optimized slow
tests + fixed Docker builds
- **Impact**: **1,358 hours/month saved** (7.7 FTE) across GitHub
Actions + CircleCI
  - GitHub Actions tests: 36m → 7m (82% faster)
  - Backend tests: 28m → 4m (87% faster with 16-way parallelization)
  - CircleCI tests: 42m → 7m (83% faster)
  - Docker builds: 34m → 5m (85% faster)
- **Result**: 5-6x faster feedback loops, 100% success rate on recent
runs

---

## Problem
CI test runs were taking **36-42 minutes per commit push** (GitHub
Actions: 36m avg, CircleCI: 42m P95), creating a significant bottleneck
in the development workflow.

## Solution
This PR comprehensively restructures both CI test pipelines to leverage
16-way parallelization and optimize test execution, reducing test
runtime from **36-42 minutes to ~7 minutes** - an **82% improvement**.

---

## 📊 Real Performance Data (Both CI Systems)

### GitHub Actions

#### Before (develop branch - 5 recent runs)
```
Individual runs: 35m 29s | 36m 1s | 40m 0s | 36m 4s | 34m 18s
Average: 36m 22s
```

#### After (feat/speed_up_ci branch - 9 successful runs)
```
Individual runs: 6m 39s | 7m 2s | 6m 53s | 6m 26s | 6m 52s | 6m 42s | 6m 45s | 6m 40s | 6m 37s
Average: 6m 44s
Range: 6m 26s - 7m 2s
```

**Improvement**:  **81.5% faster** (29m 38s saved per run)


#### Backend Tests Specific Impact
With 16-way parallelization, backend tests show dramatic improvement:
- **Before**: 27m 52s (sequential execution)
- **After**: 3m 44s (longest of 16 parallel runners)
  - Average across runners: 2m 30s
  - Range: 1m 52s - 3m 44s
- **Improvement**:  **86.6% faster** (24m 8s saved)
---

### CircleCI

#### Before (develop branch - CircleCI Insights)
```
Duration (P95): 41m 44s
Runs: 70 (last 30 days)
Success Rate: 84%
```

#### After (feat/speed_up_ci branch - Last 2 pipeline runs)
```
Run 1 (1h ago): 7m 7s
  ├─ lint: 4m 12s
  ├─ frontend-tests: 5m 36s
  ├─ backend-tests: 6m 23s
  ├─ coverage: 20s
  └─ build: 1s

Run 2 (2h ago): 7m 21s
  ├─ lint: 3m 47s
  ├─ frontend-tests: 5m 4s
  ├─ backend-tests: 6m 33s
  ├─ coverage: 19s
  └─ build: 1s

Average: 7m 14s
Success Rate: 100% 
```

**Improvement**:  **82.7% faster** (34m 30s saved per run)

---

## 🐳 Related Work: Docker Build Optimization

As part of the broader CI/CD optimization effort, Docker build
performance was improved separately in **PR #12859**.

### Docker Build Fix (Merged Separately)
**Problem**: Multi-architecture Docker builds (amd64/arm64) were taking
~34 minutes due to cache thrashing

**Solution**: Added separate cache scopes per platform in
`.github/workflows/test_docker_build.yml`:
```yaml
cache-from: type=gha,scope=${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ matrix.platform }}
```

**Results** (measured from November 2025 data):
- **Before**: 34.2 minutes/run average (15,547 minutes across 454 runs)
- **After**: 5 minutes/run
- **Improvement**: 85% faster, 29.2 minutes saved per run
- **Frequency**: 25.2 runs/day
- **Monthly savings**: **369 hours** (46 developer-days)

This prevents different architectures from invalidating each other's
caches and contributes 27% of total CI/CD time savings.

---

## 🎯 Key Findings

### Both CI Systems Now Perform Similarly
- **CircleCI**: 7m 14s average
- **GitHub Actions**: 6m 44s average
- **Difference**: Only 30 seconds apart (remarkably consistent!)

### Combined Performance
- **Average improvement across both systems**: **82.1% faster**
- **Time saved per commit**: ~32 minutes
- **Developer feedback loop**: 36-42 minutes → ~7 minutes

### Success Rate Improvement
- **CircleCI**: 84% → 100% (on feat/speed_up_ci branch)
- **GitHub Actions**: 100% (all 9 recent runs successful)
- Fixed all test isolation issues that caused intermittent failures

### Impact at Scale (Based on Real November 2025 Data)
- **CI runs per day**: **30.8 average** for tests, **25.2** for Docker
builds
  - Measured from GitHub Actions Usage Metrics (18 days)
  - Weekdays: 38-54 runs/day
  - Peak: up to 68 runs in a single day
- **This PR (test suite only)**:
  - **Daily time saved**: **15.3 hours** (GitHub Actions + CircleCI)
- **Monthly time saved**: **458 hours** (57 developer-days) on GitHub
Actions
  - Additional **531 hours** (66 developer-days) on CircleCI
- **Combined with Docker optimization** (PR #12859): **1,358
hours/month** (see Summary)
- **Developer experience**: 5-6x faster iteration cycles

---

## Code Changes

### 1. **Backend Test Parallelization (16x)**
Both CI systems now use 16-way parallelization with identical
round-robin test distribution:

```bash
# Distribute tests evenly across 16 runners
SPEC_FILES=($(find spec -name '*_spec.rb' | sort))
for i in "${!SPEC_FILES[@]}"; do
  if [ $(( i % 16 )) -eq $RUNNER_INDEX ]; then
    TESTS="$TESTS ${SPEC_FILES[$i]}"
  fi
done
```

**Why round-robin over timing-based?**
- CircleCI's timing-based splitting grouped similar tests together
- This caused race conditions with OAuth callback tests (Linear,
Shopify, Notion)
- Round-robin ensures even distribution and better test isolation
- Both CI systems now behave identically

### 2. **Frontend Test Optimization**
Enabled Vitest thread parallelization in `vite.config.ts`:

```typescript
pool: 'threads',
poolOptions: {
  threads: {
    singleThread: false,
  },
},
```

### 3. **CI Architecture Restructuring**
Split monolithic CI jobs into parallel stages:
- **Lint** (backend + frontend) - runs independently for fast feedback
- **Frontend tests** - runs in parallel with backend
- **Backend tests** - 16-way parallelized across runners
- **Coverage** - aggregates results from test jobs
- **Build** (CircleCI only) - final job for GitHub status check
compatibility

### 4. **Critical Test Optimization**
**report_builder_spec.rb**: Changed `before` to `before_all`
- Reduced execution time from **19 minutes to 1.2 minutes** (16x
speedup)
- Setup now runs once instead of 21 times
- Single biggest performance improvement after parallelization

---

## Test Stability Fixes (10 spec files)

Parallelization exposed latent test isolation issues that were fixed:

### Object Identity Comparisons (6 files)
Tests were comparing Ruby object instances instead of IDs:
- `spec/models/integrations/hook_spec.rb` - Use `.pluck(:id)` for
comparisons
- `spec/enterprise/models/captain/scenario_spec.rb` - Compare IDs
instead of objects
- `spec/models/notification_spec.rb` - Compare IDs for sort order
validation
- `spec/models/account_spec.rb` - Compare IDs in scope queries
- `spec/services/widget/token_service_spec.rb` - Compare class names
instead of class objects
- `spec/models/concerns/avatarable_shared.rb` - Use `respond_to` checks
for ActiveStorage

### Database Query Caching
- `spec/jobs/delete_object_job_spec.rb` - Added `.reload` to force fresh
database queries

### Test Expectations Timing
- `spec/jobs/mutex_application_job_spec.rb` - Removed flaky unlock
expectation after error block
  - Related to original PR #8770
- Expectation after error block never executes in parallel environments

### Timezone Handling
- `spec/mailers/account_notification_mailer_spec.rb` - Fixed date
parsing at timezone boundaries
  - Changed test time from 23:59:59Z to 12:00:00Z

### Test Setup
- `spec/builders/v2/report_builder_spec.rb` - Optimized with
`before_all`

---

##  CircleCI GitHub Integration Fix

### Problem
GitHub PR checks were stuck on "Waiting for status" even when all
CircleCI jobs passed. GitHub was expecting a job named `build` but the
workflow only had a workflow named "build".

### Solution
Added an explicit `build` job that runs after all other jobs:

```yaml
build:
  steps:
    - run:
        name: Legacy build aggregator
        command: echo "All main jobs passed"
  requires:
    - lint
    - coverage
```

This ensures GitHub's required status checks work correctly.


---

##  Testing & Validation

-  **GitHub Actions**: 9 successful runs, consistent 6m 26s - 7m 2s
runtime
-  **CircleCI**: 2 successful runs, consistent 7m 7s - 7m 21s runtime
-  Both CI systems produce identical, consistent results
-  GitHub PR status checks complete correctly
-  Success rate improved from 84% to 100% (recent runs)
-  No test regressions introduced
-  All flaky tests fixed (callback controllers, mutex jobs, etc.)

---

## 🎉 Summary

This PR delivers an **82% improvement** in test execution time across
both CI systems:

- **GitHub Actions tests**: 36m → 7m (81.5% faster)
- Backend tests specifically: 28m → 4m (86.6% faster)
- **CircleCI tests**: 42m → 7m (82.7% faster)
- **Developer feedback loop**: 5-6x faster
- **Test stability**: 84% → 100% success rate

### 📊 Total CI/CD Impact (All Optimizations)

Based on real November 2025 data, combining this PR with Docker build
optimization (PR #12859):

**Monthly Time Savings**: **1,358 hours/month** = **170
developer-days/month** = **7.7 FTE**

| System | Runs/Day | Before | After | Savings | Monthly Impact |
|--------|----------|---------|--------|---------|----------------|
| **GitHub Actions Tests** | 30.8 | 36.5m | 6.7m | 29.8m/run | 458 hrs
(34%) |
| **GitHub Actions Docker** | 25.2 | 34.2m | 5.0m | 29.2m/run | 369 hrs
(27%) |
| **CircleCI Tests** | 30.8 | 41.7m | 7.2m | 34.5m/run | 531 hrs (39%) |

*Data source: GitHub Actions Usage Metrics (November 2025, 18 days),
CircleCI Insights (30 days)*

The combined optimizations save the equivalent of **nearly 8 full-time
developers** worth of CI waiting time every month, significantly
improving developer velocity and reducing CI costs. All test isolation
issues exposed by parallelization have been fixed, ensuring reliable and
consistent results across both CI platforms.

woot woot !!!

---------
2025-11-19 15:32:48 +05:30
Sojan Jose
0a9edd4c3b ci(circleci): switch coverage reporting to Qlty orb (#12337) 2025-08-31 00:39:34 +05:30
Sojan Jose
bc42aec68e chore: upgrade ruby version to 3.4.4 (#11524)
- Chore upgrade ruby version to 3.4.4 before we migrate to rails 7.2
over #11037
2025-05-21 19:40:07 +05:30
Shivam Mishra
f2a7e1da6b fix: Corepack pnpm issue (#10840)
So, a while back Circle CI builds and Heroku builds started to fail. 

From all the threads I read, it seems like the [npm registry rotated
it's signing
keys](https://github.com/pnpm/pnpm/issues/9014#issuecomment-2616589753)
New pnpm versions were signed with the new key. Corepack, however,
bundles a static set of trusted keys (from Node’s release), so it
continued verifying signatures only against the old key. When it
encountered packages signed with the new key, Corepack’s integrity check
failed with “Cannot find matching keyid” errors.This mismatch caused
Corepack’s integrity check to fail with “Cannot find matching keyid”
errors.

Workarounds include the following

1. Updating Corepack (to 0.31.0), they [upgraded their
package](https://github.com/nodejs/corepack/releases/tag/v0.31.0) to
include the new integrity check keys. But we seldom control what's going
on with the CI, also, updating this across our scripts is going to be a
painful task. Besides Heroku has [made some
fixes](https://github.com/heroku/buildpacks-nodejs/pull/1010) around
this
2. Disabling integrity checks 🔥 #YOLO
3. Pinning `pnpm` to older versions, or pinning it to a newer version
with the checksum in place.

Doing the third one here, running `corepack use pnpm@9.15.5` fixes this,
[ref](https://github.com/pnpm/pnpm/issues/9014#issuecomment-2623761494)

We can get rid of this over time as CDN caches used by build systems are
refreshed. But the change in this PR is not disruptive in anyway, only
rigidly secure.

Fixes: https://github.com/chatwoot/chatwoot/issues/10832

---

Here are the threads to follow 

- https://github.com/pnpm/pnpm/issues/9014
- https://github.com/pnpm/pnpm/issues/9029
- https://github.com/nodejs/corepack/issues/612
- https://github.com/nodejs/corepack/issues/616
- https://github.com/heroku/buildpacks-nodejs/pull/1010

---------

Co-authored-by: Vishnu Narayanan <vishnu@chatwoot.com>
2025-02-05 10:12:29 -08:00
Pranav
d070743383 feat(ee): Add Captain features (#10665)
Migration Guide: https://chwt.app/v4/migration

This PR imports all the work related to Captain into the EE codebase. Captain represents the AI-based features in Chatwoot and includes the following key components:

- Assistant: An assistant has a persona, the product it would be trained on. At the moment, the data at which it is trained is from websites. Future integrations on Notion documents, PDF etc. This PR enables connecting an assistant to an inbox. The assistant would run the conversation every time before transferring it to an agent.
- Copilot for Agents: When an agent is supporting a customer, we will be able to offer additional help to lookup some data or fetch information from integrations etc via copilot.
- Conversation FAQ generator: When a conversation is resolved, the Captain integration would identify questions which were not in the knowledge base.
- CRM memory: Learns from the conversations and identifies important information about the contact.

---------

Co-authored-by: Vishnu Narayanan <vishnu@chatwoot.com>
Co-authored-by: Sojan <sojan@pepalo.com>
Co-authored-by: iamsivin <iamsivin@gmail.com>
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
2025-01-14 16:15:47 -08:00
Vishnu Narayanan
e3109dbb22 chore: pin bundler version to 2.5.x in circleci (#10596) 2024-12-17 16:31:55 +05:30
Vishnu Narayanan
ee02923ace chore: fix circleci on vite build (#10214)
- Switch to pnpm based build
- Switch circleci from docker to machine to have more memory
- Fix frontend and backend tests

Fixes
https://linear.app/chatwoot/issue/CW-3610/fix-circle-ci-for-vite-build
---------

Co-authored-by: Shivam Mishra <scm.mymail@gmail.com>
Co-authored-by: Pranav <pranavrajs@gmail.com>
Co-authored-by: Pranav <pranav@chatwoot.com>
2024-10-07 15:27:41 +05:30
Pranav
9de8c27368 feat: Use vitest instead of jest, run all the specs anywhere in app/ folder in the CI (#9722)
Due to the pattern `**/specs/*.spec.js` defined in CircleCI, none of the
frontend spec in the folders such as
`specs/<domain-name>/getters.spec.js` were not executed in Circle CI.

This PR fixes the issue, along with the following changes: 
- Use vitest instead of jest
- Remove jest dependancies
- Update tests to work with vitest

---------

Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
2024-07-10 08:32:16 -07:00
Sojan Jose
31bcdaa3dd chore: Upgrade ruby to 3.3.3 (#9664)
- Upgrade the ruby version to 3.3.3

---------

Co-authored-by: Vishnu Narayanan <iamwishnu@gmail.com>
2024-07-05 12:20:07 +05:30
Shivam Mishra
e2a6dc3e04 chore: Upgrade Node.js to v20 (#7759)
Co-authored-by: Vishnu Narayanan <vishnu@chatwoot.com>
Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
2023-08-28 19:20:35 +05:30
Sojan Jose
480f34803b feat: Response Bot using GPT and Webpage Sources (#7518)
This commit introduces the ability to associate response sources to an inbox, allowing external webpages to be parsed by Chatwoot. The parsed data is converted into embeddings for use with GPT models when managing customer queries.

The implementation relies on the `pgvector` extension for PostgreSQL. Database migrations related to this feature are handled separately by `Features::ResponseBotService`. A future update will integrate these migrations into the default rails migrations, once compatibility with Postgres extensions across all self-hosted installation options is confirmed.

Additionally, a new GitHub action has been added to the CI pipeline to ensure the execution of specs related to this feature.
2023-07-21 18:11:51 +03:00
Pranav Raj S
4e8d17f017 chore: Update specs and warnings in console (#7467) 2023-07-05 18:32:55 -07:00
Sojan Jose
373f5f5b64 chore: Disable coverage reports for .vue files (#7214)
- Disable coverage reporting for .vue files 

fixes: https://linear.app/chatwoot/issue/CW-1331
2023-05-30 19:18:19 +05:30
Sojan Jose
022383d942 chore: Upgrade to Rails 7 (#6719)
fixes: #6736
2023-05-06 10:44:52 +05:30
Giau. Tran Minh
59964a4f41 swagger: fixed invalid specification (#5485)
Currently, the swagger spec doesn't follow the Swagger 2.0 specification. So, I facing 4 errors when trying generate the Golang client for chatwoot.
Due to the spec, the binary field should use format: binary beside type: string

Signed-off-by: Giau. Tran Minh <hello@giautm.dev>
Co-authored-by: Sojan Jose <sojan@pepalo.com>
2023-02-16 17:47:26 +05:30
Vishnu Narayanan
b1ec67d110 chore: upgrade ruby to 3.1.3 (#5555)
* chore: update to ruby 3.1.3

* chore: ping docker version to alpine3.16 for nodev16.x

Starting with Node 17, nodejs switched to OpenSSL3. The docker builds
are installing node18.xx with alpine-3.1.3.

From Node.js 17's announcement post:

    If you hit an ERR_OSSL_EVP_UNSUPPORTED error in your application
with Node.js 17, it’s likely that your application or a module you’re
using is attempting to use an algorithm or key size which is no longer
allowed by default with OpenSSL 3.0. A new command-line option,
--openssl-legacy-provider, has been added to revert to the legacy
provider as a temporary workaround for these tightened restrictions.

Looks like a webpack issue. This is fixed in webpacl 5+ and we are on
webpack4 at the moment.
Solutions

    Upgrade webpack.
    Pin nodejs version to be 16.x.x
    Use  --openssl-legacy-provider as a workaround.

Pin docker version to alpine3.16 branch to have node16.x by default

ref:
https://github.com/chatwoot/chatwoot/pull/5555#issuecomment-1379778532

* chore: update webmock

* chore: fix ruby gem path in dockerfile

* chore: switch to node16 in circleci

* chore: update ruby version in linux installer script

* chore: update ruby version in linux installer script

* chore: fix circleci

* chore: fix circleci

* feat: upgrade node version to 16.x in linux installer

* chore: update systemd files

Co-authored-by: Sojan Jose <sojan@chatwoot.com>
2023-01-24 23:55:07 +05:30
Sojan Jose
8b0e95ece8 fix: Flakiness in CI pipeline (#5562)
- Fixing the recent flakiness in CI pipelines
2022-10-05 10:59:31 -07:00
Sojan Jose
afe31a3156 chore: parallel runs in circle (#2810)
- Enable parallel runs in circle

Co-authored-by: Tejaswini <tejaswini@chatwoot.com>
Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
2022-09-06 19:17:33 +05:30
Sojan Jose
4187428729 chore: Update dependencies to the latest versions (#5033) 2022-07-15 09:51:59 +07:00
Sojan Jose
ea1a27c7d4 fix: Twilio channel selection when MessagingServiceSid is empty (#5040)
- Fixes channel selection logic in incoming_message_service for Twilio messages

ref: #4242
2022-07-14 15:16:07 +02:00
Vishnu Narayanan
568c30e93e chore: Update ruby and docker base image to 3.0.4 (#4693)
* chore: Update ruby version to 3.0.4

* chore: update ruby version in docker workflow

* chore: update ruby version to 3.1.2

* Upgrade vue-jest to remove deasync

* Revert to 3.0.4 to see if deasync issue is fixed

* fix: script failure if pg/redis not opted in

Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
2022-05-27 17:33:24 +05:30
Sojan Jose
baa3efaf0e chore: Update cache name in Circle CI builds (#4738)
Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
2022-05-24 22:15:57 +05:30
Vishnu Narayanan
ec0ea0b1dc chore: Upgrade to next gen docker convenience image (#4310)
* chore: Upgrade to next gen docker convenience image

https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034/16

* switch to postgres:14.1 and redis:6.2 images
2022-03-28 18:15:38 +05:30
Sojan Jose
823c0ab6a7 chore: Use Round Robin service for team assignment (#4237)
Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
2022-03-28 14:38:07 +05:30
Jan-David
e500d1216b Fix: Make swagger doc compliant to OpenAPI (#3394)
This fixes issues in the swagger.json file. The motivation to do so is to be able to generate API clients using https://openapi-generator.tech Doing so will require further changes to the api spec, but this seems like a good first step since it is now "valid" according to editor.swagger.io and openapi-generator validate.

Fixes #2806
2021-11-17 18:55:15 +05:30
Sojan Jose
0e0632be22 chore: Minor Housekeeping tasks (#3169)
- Limit Rack attack to production environments
- Make the long-running data migration optional
2021-10-08 15:45:45 +05:30
Vishnu Narayanan
5d3cce12d5 fix: Disable rack attack gem in circleCI (#3167) 2021-10-08 12:14:23 +05:30
Sivin Varghese
5debe9e8ee fix: Add agent typing notification on all channels (#2855)
Fixes CVE-2021-22881

Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
2021-08-23 21:34:23 +05:30
Sojan Jose
ab54d9c629 chore: Upgrade rails and ruby versions (#2400)
ruby version: 3.0.2
rails version: 6.1.4
2021-08-03 20:11:52 +05:30
Sojan Jose
fa37f8e185 chore: Support Github Codespaces (#2510)
Note: Update the ruby version 2.7.3. For production Linux VM, install 2.7.3 and follow the upgrade steps as shown in https://www.chatwoot.com/docs/self-hosted/deployment/linux-vm#upgrading-to-a-newer-version-of-chatwoot
2021-06-29 19:18:10 +05:30
Adam Zysko
61d26f71c1 chore: bump ruby from 2.7.1 to 2.7.2 (#1426)
Closes:  #1399
2020-11-20 15:38:13 +05:30
Sojan Jose
4f83d5451e Chore: Routine Bugfixes and enhancements (#979)
- Fix slack scopes
- Docs for authentication
Fixes: #704 , #973
2020-06-25 23:35:16 +05:30
Sojan Jose
1ef8d03e18 Feature: Slack - receive messages, create threads, send replies (#974)
Co-authored-by: Pranav Raj S <pranav@thoughtwoot.com>
2020-06-22 13:19:26 +05:30
Sojan Jose
05ea6308f2 Chore: Update ruby and rails versions (#831) 2020-05-08 12:13:23 +05:30
Sojan Jose
8e59564793 Chore: Update ruby version (#665) 2020-04-05 12:11:50 +05:30
Pranav Raj S
c4e2a84f65 Feature: Agent Profile Update with avatar (#449)
* Feature: Agent Profile Update with avatar
* Add Update Profile with name, avatar, email and password
2020-02-16 17:20:38 +05:30
Sojan Jose
77473dc2aa Chore: Include avatar url in sign_in response (#501)
- include avatar url in sign_in response ( fixes #500 )
- fix circle ci builds
2020-02-15 22:57:48 +05:30
Sony Mathew
d4b3ba4baa [#139] Send conversation emails (#442)
* [#139] Delayed emails for conversations

* Added the setex and get methods to Redis wrapper
* Set the priorities for the sidekiq queues
* Was not able to use mailhog for testing email in local, switched back to letter opener and added comments on using the SMTP settings
* Added after create hood in messages to queue the sending of mail after 2 minutes using sidekiq worker and also set the redis key for the conversation to avoid the email sending for every message
* Added the sidekiq worker to send the email and delete the conversation redis key
* Added the mailer and mail template
* mailer sends the last 10 messages along with the new messages from the time it was queued

* Send email only in development or if smtp config is set

* Send email only in development or if smtp config is set
* Set the SMTP_PORT in production variable

* Adding redis to circle CI

* Specs for the conversation email changes

* Added specs for conversation email sidekiq worker
* Added specs for conversation mailer
* Added specs in message model for the after create hook for notify email

* Send emails only when there is a reply from agent

* set development to use mailhog

* Adding comments for using letter opener
2020-01-23 23:14:07 +05:45
Sojan Jose
a8e8416d48 Chore: Cache gemfiles in circle builds [#338] (#343) 2019-12-03 19:26:38 +05:30
Sojan Jose
6ca403542d Chore: Setup test coverage in codeclimate [#332] (#333) 2019-12-02 08:57:11 +05:30
Sojan Jose
9c6a101768 [Chore] Add bullet gem #249 (#281)
* closes #249 add bullet gem

- upgrade gems
- fix the warning in the console
- fix the deprecation message for where not
2019-11-25 23:55:18 +05:30
Sony Mathew
da9ac8b26d 🔥Docker and environment variables cleanup (#270)
* Added dotenv-rails gem to manage environment variables

* Added dotenv-rails gem to manage environment variables
* Removed figaro which was used earlier for this purpose

* Standardized variable names

* Changed all env variables to be upper case. This included changes in files referencing env variables.
* Added example env file with all variables set to empty value
* Removed the earlier setup of copying application.yml and database.yml and the scripts and documentation associated to this

* Docker setup

* Added docker file for building the docker images
* Added entrypoint.sh script which is referenced inside the Docker image
* Cloned the Procfile for development using docker which has slight change compared to regular procfile
* Added the docker-compose.yml which has 3 service's configuration, postgres, redis and chatwoot server and a mounted volume for postgres

* Added docker related info to documentation

* Added the docker setup info in the documentation
* Added info for using`rbenv` instead of rvm for managing ruby versions
* Updated the documentation for environment variables to have one about `dotenv-rails` gem and removed the documentation about the old copy paste method used by figaro

* Changing the postgres database, username and password as environment variables

* Removed database.yml from gitignore
* Made the postgres databse, username and password as environemnt variables
* Added this in documentation

* Added a quick setup page

* Added quick setup page
* Removed the docs from README and added link to the docs in website
* Removed the figaro related things from circle.ci config

* Adding external volume for redis in docker compose

* Added instructions for adding the redis volume in docs
2019-11-24 01:27:39 +05:30