Commit Graph

16 Commits

Author SHA1 Message Date
Sojan Jose
9eb3ee44a8 Revert "chore: Upgrade Rails to 7.2.2 and update Gemfile dependencies (#11037)"
This reverts commit ef6ba8aabd.
2026-02-03 21:09:42 -08:00
Sojan Jose
ef6ba8aabd chore: Upgrade Rails to 7.2.2 and update Gemfile dependencies (#11037)
Upgrade rails to 7.2.2 so that we can proceed with the rails 8 upgrade
afterwards
 
 # Changelog
- `.circleci/config.yml` — align CI DB setup with GitHub Actions
(`db:create` + `db:schema:load`) to avoid trigger-dependent prep steps.
- `.rubocop.yml` — add `rubocop-rspec_rails` and disable new cops that
don't match existing spec style.
- `AGENTS.md` — document that specs should run without `.env` (rename
temporarily when present).
- `Gemfile` — upgrade to Rails 7.2, switch Azure storage gem, pin
`commonmarker`, bump `sidekiq-cron`, add `rubocop-rspec_rails`, and
relax some gem pins.
- `Gemfile.lock` — dependency lockfile updates from the Rails 7.2 and
gem changes.
- `app/controllers/api/v1/accounts/integrations/linear_controller.rb` —
stringify params before passing to the Linear service to keep key types
stable.
- `app/controllers/super_admin/instance_statuses_controller.rb` — use
`MigrationContext` API for migration status in Rails 7.2.
- `app/models/installation_config.rb` — add commentary on YAML
serialization and future JSONB migration (no behavior change).
- `app/models/integrations/hook.rb` — ensure hook type is set on create
only and guard against missing app.
- `app/models/user.rb` — update enum syntax for Rails 7.2 deprecation,
serialize OTP backup codes with JSON, and use Ruby `alias`.
- `app/services/crm/leadsquared/setup_service.rb` — stringify hook
settings keys before merge to keep JSON shape consistent.
- `app/services/macros/execution_service.rb` — remove macro-specific
assignee activity workaround; rely on standard assignment handlers.
- `config/application.rb` — load Rails 7.2 defaults.
- `config/storage.yml` — update Azure Active Storage service name to
`AzureBlob`.
- `db/migrate/20230515051424_update_article_image_keys.rb` — use
credentials `secret_key_base` with fallback to legacy secrets.
- `docker/Dockerfile` — add `yaml-dev` and `pkgconf` packages for native
extensions (Ruby 3.4 / psych).
- `lib/seeders/reports/message_creator.rb` — add parentheses for clarity
in range calculation.
- `package.json` — pin Vite version and bump `vite-plugin-ruby`.
- `pnpm-lock.yaml` — lockfile changes from JS dependency updates.
- `spec/builders/v2/report_builder_spec.rb` — disable transactional
fixtures; truncate tables per example via Rails `truncate_tables` so
after_commit callbacks run with clean isolation; keep builder spec
metadata minimal.
- `spec/builders/v2/reports/label_summary_builder_spec.rb` — disable
transactional fixtures + truncate tables via Rails `truncate_tables`;
revert to real `resolved!`/`open!`/`resolved!` flow for multiple
resolution events; align date range to `Time.zone` to avoid offset gaps;
keep builder spec metadata minimal.
- `spec/controllers/api/v1/accounts/macros_controller_spec.rb` — assert
`assignee_id` instead of activity message to avoid transaction-timing
flakes.
- `spec/services/telegram/incoming_message_service_spec.rb` — reference
the contact tied to the created conversation instead of
`Contact.all.first` to avoid order-dependent failures when other specs
leave data behind.
-
`spec/mailers/administrator_notifications/shared/smtp_config_shared.rb`
— use `with_modified_env` instead of stubbing mailer internals.
- `spec/services/account/sign_up_email_validation_service_spec.rb` —
compare error `class.name` for parallel/reload-safe assertions.
2026-02-03 14:29:26 -08:00
Vishnu Narayanan
08b9134486 feat: speed up circleci and github actions (#12849)
# 🚀 Speed up CI/CD test execution with parallelization

## TL;DR

- **Problem**: CI tests took 36-42 minutes per commit, blocking
developer workflow
- **Solution**: Implemented 16-way parallelization + optimized slow
tests + fixed Docker builds
- **Impact**: **1,358 hours/month saved** (7.7 FTE) across GitHub
Actions + CircleCI
  - GitHub Actions tests: 36m → 7m (82% faster)
  - Backend tests: 28m → 4m (87% faster with 16-way parallelization)
  - CircleCI tests: 42m → 7m (83% faster)
  - Docker builds: 34m → 5m (85% faster)
- **Result**: 5-6x faster feedback loops, 100% success rate on recent
runs

---

## Problem
CI test runs were taking **36-42 minutes per commit push** (GitHub
Actions: 36m avg, CircleCI: 42m P95), creating a significant bottleneck
in the development workflow.

## Solution
This PR comprehensively restructures both CI test pipelines to leverage
16-way parallelization and optimize test execution, reducing test
runtime from **36-42 minutes to ~7 minutes** - an **82% improvement**.

---

## 📊 Real Performance Data (Both CI Systems)

### GitHub Actions

#### Before (develop branch - 5 recent runs)
```
Individual runs: 35m 29s | 36m 1s | 40m 0s | 36m 4s | 34m 18s
Average: 36m 22s
```

#### After (feat/speed_up_ci branch - 9 successful runs)
```
Individual runs: 6m 39s | 7m 2s | 6m 53s | 6m 26s | 6m 52s | 6m 42s | 6m 45s | 6m 40s | 6m 37s
Average: 6m 44s
Range: 6m 26s - 7m 2s
```

**Improvement**:  **81.5% faster** (29m 38s saved per run)


#### Backend Tests Specific Impact
With 16-way parallelization, backend tests show dramatic improvement:
- **Before**: 27m 52s (sequential execution)
- **After**: 3m 44s (longest of 16 parallel runners)
  - Average across runners: 2m 30s
  - Range: 1m 52s - 3m 44s
- **Improvement**:  **86.6% faster** (24m 8s saved)
---

### CircleCI

#### Before (develop branch - CircleCI Insights)
```
Duration (P95): 41m 44s
Runs: 70 (last 30 days)
Success Rate: 84%
```

#### After (feat/speed_up_ci branch - Last 2 pipeline runs)
```
Run 1 (1h ago): 7m 7s
  ├─ lint: 4m 12s
  ├─ frontend-tests: 5m 36s
  ├─ backend-tests: 6m 23s
  ├─ coverage: 20s
  └─ build: 1s

Run 2 (2h ago): 7m 21s
  ├─ lint: 3m 47s
  ├─ frontend-tests: 5m 4s
  ├─ backend-tests: 6m 33s
  ├─ coverage: 19s
  └─ build: 1s

Average: 7m 14s
Success Rate: 100% 
```

**Improvement**:  **82.7% faster** (34m 30s saved per run)

---

## 🐳 Related Work: Docker Build Optimization

As part of the broader CI/CD optimization effort, Docker build
performance was improved separately in **PR #12859**.

### Docker Build Fix (Merged Separately)
**Problem**: Multi-architecture Docker builds (amd64/arm64) were taking
~34 minutes due to cache thrashing

**Solution**: Added separate cache scopes per platform in
`.github/workflows/test_docker_build.yml`:
```yaml
cache-from: type=gha,scope=${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ matrix.platform }}
```

**Results** (measured from November 2025 data):
- **Before**: 34.2 minutes/run average (15,547 minutes across 454 runs)
- **After**: 5 minutes/run
- **Improvement**: 85% faster, 29.2 minutes saved per run
- **Frequency**: 25.2 runs/day
- **Monthly savings**: **369 hours** (46 developer-days)

This prevents different architectures from invalidating each other's
caches and contributes 27% of total CI/CD time savings.

---

## 🎯 Key Findings

### Both CI Systems Now Perform Similarly
- **CircleCI**: 7m 14s average
- **GitHub Actions**: 6m 44s average
- **Difference**: Only 30 seconds apart (remarkably consistent!)

### Combined Performance
- **Average improvement across both systems**: **82.1% faster**
- **Time saved per commit**: ~32 minutes
- **Developer feedback loop**: 36-42 minutes → ~7 minutes

### Success Rate Improvement
- **CircleCI**: 84% → 100% (on feat/speed_up_ci branch)
- **GitHub Actions**: 100% (all 9 recent runs successful)
- Fixed all test isolation issues that caused intermittent failures

### Impact at Scale (Based on Real November 2025 Data)
- **CI runs per day**: **30.8 average** for tests, **25.2** for Docker
builds
  - Measured from GitHub Actions Usage Metrics (18 days)
  - Weekdays: 38-54 runs/day
  - Peak: up to 68 runs in a single day
- **This PR (test suite only)**:
  - **Daily time saved**: **15.3 hours** (GitHub Actions + CircleCI)
- **Monthly time saved**: **458 hours** (57 developer-days) on GitHub
Actions
  - Additional **531 hours** (66 developer-days) on CircleCI
- **Combined with Docker optimization** (PR #12859): **1,358
hours/month** (see Summary)
- **Developer experience**: 5-6x faster iteration cycles

---

## Code Changes

### 1. **Backend Test Parallelization (16x)**
Both CI systems now use 16-way parallelization with identical
round-robin test distribution:

```bash
# Distribute tests evenly across 16 runners
SPEC_FILES=($(find spec -name '*_spec.rb' | sort))
for i in "${!SPEC_FILES[@]}"; do
  if [ $(( i % 16 )) -eq $RUNNER_INDEX ]; then
    TESTS="$TESTS ${SPEC_FILES[$i]}"
  fi
done
```

**Why round-robin over timing-based?**
- CircleCI's timing-based splitting grouped similar tests together
- This caused race conditions with OAuth callback tests (Linear,
Shopify, Notion)
- Round-robin ensures even distribution and better test isolation
- Both CI systems now behave identically

### 2. **Frontend Test Optimization**
Enabled Vitest thread parallelization in `vite.config.ts`:

```typescript
pool: 'threads',
poolOptions: {
  threads: {
    singleThread: false,
  },
},
```

### 3. **CI Architecture Restructuring**
Split monolithic CI jobs into parallel stages:
- **Lint** (backend + frontend) - runs independently for fast feedback
- **Frontend tests** - runs in parallel with backend
- **Backend tests** - 16-way parallelized across runners
- **Coverage** - aggregates results from test jobs
- **Build** (CircleCI only) - final job for GitHub status check
compatibility

### 4. **Critical Test Optimization**
**report_builder_spec.rb**: Changed `before` to `before_all`
- Reduced execution time from **19 minutes to 1.2 minutes** (16x
speedup)
- Setup now runs once instead of 21 times
- Single biggest performance improvement after parallelization

---

## Test Stability Fixes (10 spec files)

Parallelization exposed latent test isolation issues that were fixed:

### Object Identity Comparisons (6 files)
Tests were comparing Ruby object instances instead of IDs:
- `spec/models/integrations/hook_spec.rb` - Use `.pluck(:id)` for
comparisons
- `spec/enterprise/models/captain/scenario_spec.rb` - Compare IDs
instead of objects
- `spec/models/notification_spec.rb` - Compare IDs for sort order
validation
- `spec/models/account_spec.rb` - Compare IDs in scope queries
- `spec/services/widget/token_service_spec.rb` - Compare class names
instead of class objects
- `spec/models/concerns/avatarable_shared.rb` - Use `respond_to` checks
for ActiveStorage

### Database Query Caching
- `spec/jobs/delete_object_job_spec.rb` - Added `.reload` to force fresh
database queries

### Test Expectations Timing
- `spec/jobs/mutex_application_job_spec.rb` - Removed flaky unlock
expectation after error block
  - Related to original PR #8770
- Expectation after error block never executes in parallel environments

### Timezone Handling
- `spec/mailers/account_notification_mailer_spec.rb` - Fixed date
parsing at timezone boundaries
  - Changed test time from 23:59:59Z to 12:00:00Z

### Test Setup
- `spec/builders/v2/report_builder_spec.rb` - Optimized with
`before_all`

---

##  CircleCI GitHub Integration Fix

### Problem
GitHub PR checks were stuck on "Waiting for status" even when all
CircleCI jobs passed. GitHub was expecting a job named `build` but the
workflow only had a workflow named "build".

### Solution
Added an explicit `build` job that runs after all other jobs:

```yaml
build:
  steps:
    - run:
        name: Legacy build aggregator
        command: echo "All main jobs passed"
  requires:
    - lint
    - coverage
```

This ensures GitHub's required status checks work correctly.


---

##  Testing & Validation

-  **GitHub Actions**: 9 successful runs, consistent 6m 26s - 7m 2s
runtime
-  **CircleCI**: 2 successful runs, consistent 7m 7s - 7m 21s runtime
-  Both CI systems produce identical, consistent results
-  GitHub PR status checks complete correctly
-  Success rate improved from 84% to 100% (recent runs)
-  No test regressions introduced
-  All flaky tests fixed (callback controllers, mutex jobs, etc.)

---

## 🎉 Summary

This PR delivers an **82% improvement** in test execution time across
both CI systems:

- **GitHub Actions tests**: 36m → 7m (81.5% faster)
- Backend tests specifically: 28m → 4m (86.6% faster)
- **CircleCI tests**: 42m → 7m (82.7% faster)
- **Developer feedback loop**: 5-6x faster
- **Test stability**: 84% → 100% success rate

### 📊 Total CI/CD Impact (All Optimizations)

Based on real November 2025 data, combining this PR with Docker build
optimization (PR #12859):

**Monthly Time Savings**: **1,358 hours/month** = **170
developer-days/month** = **7.7 FTE**

| System | Runs/Day | Before | After | Savings | Monthly Impact |
|--------|----------|---------|--------|---------|----------------|
| **GitHub Actions Tests** | 30.8 | 36.5m | 6.7m | 29.8m/run | 458 hrs
(34%) |
| **GitHub Actions Docker** | 25.2 | 34.2m | 5.0m | 29.2m/run | 369 hrs
(27%) |
| **CircleCI Tests** | 30.8 | 41.7m | 7.2m | 34.5m/run | 531 hrs (39%) |

*Data source: GitHub Actions Usage Metrics (November 2025, 18 days),
CircleCI Insights (30 days)*

The combined optimizations save the equivalent of **nearly 8 full-time
developers** worth of CI waiting time every month, significantly
improving developer velocity and reducing CI costs. All test isolation
issues exposed by parallelization have been fixed, ensuring reliable and
consistent results across both CI platforms.

woot woot !!!

---------
2025-11-19 15:32:48 +05:30
Sojan Jose
98f4a6f797 chore: Ensure admin notification mailer specs are order agnostic (#12472)
## Summary
- update the admin notification base mailer spec to ignore ordering when
verifying administrator email addresses
- extend the channel and integrations admin notification mailer specs to
cover multiple administrators without relying on recipient order

------
https://chatgpt.com/codex/tasks/task_e_68cc7457cf788326a765f116ceab1732

Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
2025-09-19 16:37:57 +05:30
Pranav
e863a52262 chore: Update account deletion email copy (#12317)
Update the email copies for account deletion emails 

## Manual Deletion 

<img width="1378" height="678" alt="Screenshot 2025-08-29 at 2 41 40 PM"
src="https://github.com/user-attachments/assets/63d7ad97-ad51-4a8b-9ef3-d427fa467f8a"
/>

## Inactive Deletion

<img width="2946" height="1808" alt="image"
src="https://github.com/user-attachments/assets/bb50d08c-8701-4f93-af29-0b1d948a4009"
/>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Sojan Jose <sojan@pepalo.com>
2025-08-31 16:01:41 +02:00
Vishnu Narayanan
d40a59f7fa feat: automate account deletion (#11406)
- Automate the deletion of accounts that have requested deletion via
account settings.
- Add a Sidekiq job that runs daily to find accounts that have requested
deletion and have passed the 7-day window.
- This job deletes the account and then soft-deletes users if they do
not belong to any other account.
- This job also sends an email to the Chatwoot instance admin for
compliance purposes.
- The Chatwoot instance admin's email is configurable via the
`CHATWOOT_INSTANCE_ADMIN_EMAIL` global config.

---------

Co-authored-by: Sojan Jose <sojan@pepalo.com>
2025-05-23 12:58:13 +05:30
Muhsin Keloth
246deab684 feat: Instagram reauthorization (#11221)
This PR is part of https://github.com/chatwoot/chatwoot/pull/11054 to
make the review cycle easier.
2025-04-03 14:30:48 +05:30
Pranjal Kushwaha
0dc2af3c78 feat: Ability to delete account for administrators (#1874)
## Description

Add account delete option in the user account settings.

Fixes #1555 

## Type of change

- [ ] New feature (non-breaking change which adds functionality)


![image](https://user-images.githubusercontent.com/40784971/110349673-edcc5200-8058-11eb-8ded-a31d15aa0759.png)

![image](https://user-images.githubusercontent.com/40784971/110349778-0c324d80-8059-11eb-9291-abfbffedde5e.png)


## Checklist:

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my own code
- [ ] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules

---------

Co-authored-by: Sojan Jose <sojan@pepalo.com>
Co-authored-by: Sojan Jose <sojan.official@gmail.com>
Co-authored-by: Muhsin Keloth <muhsinkeramam@gmail.com>
2025-04-03 10:41:39 +05:30
Liam
5d9fb55370 feat: Export contact improvements (#8895)
This pull request enhances the export contacts feature by adding a confirmation step before exporting. Previously, clicking the export button would trigger the export action without confirmation.

Additionally, it ensures that only the intended recipient receives the export email, addressing the previous behaviour where all administrators received it.

Fixes: #8504

Co-authored-by: Sojan Jose <sojan@pepalo.com>
2024-02-20 17:11:03 +05:30
Pranav Raj S
653e0335c0 fix: Handle PermissionDeniedError for Dialogflow processor (#8252) 2023-10-30 13:24:03 -07:00
Sojan Jose
acb7debd3f chore: Contact import improvements (#7787)
- Ensure existing contact information is updated on data import
- Refactor the existing job to make it more readable
- Fixes issues with import files in the wrong encoding
fixes: #7307
2023-08-23 23:24:47 -07:00
Tejaswini Chile
23ca6d56f9 feat: Contact Exports (#7258) 2023-06-13 09:18:43 +05:30
Sojan Jose
7ab7bac6bf chore: Enable the new Rubocop rules (#7122)
fixes: https://linear.app/chatwoot/issue/CW-1574/renable-the-disabled-rubocop-rules
2023-05-19 14:37:10 +05:30
Sojan Jose
8ea0660862 chore: Add reauthorization prompt for Whatsapp Channel (#5929)
- Add reauthorization prompt for Whatsapp Channel

fixes: #5782
2022-11-24 14:50:32 +03:00
Sojan Jose
a4c87f2052 chore: Handle attachments in Whatsapp Channel (#3299)
send and receive attachments in 360Dialog WhatsApp channels
2021-11-11 13:03:48 +05:30
Sojan Jose
e46aa1aa64 chore: Mark Facebook accounts for reconnection (#2405)
fixes: #2037
2021-08-01 18:15:39 +05:30