LeadMagnet/leadchat
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
5,546 Commits 1 Branch 0 Tags
08b9134486976778360e87500d130bd19e0790c3
Commit Graph

14 Commits

Author SHA1 Message Date
Vishnu Narayanan
08b9134486 feat: speed up circleci and github actions (#12849) 
# 🚀 Speed up CI/CD test execution with parallelization

## TL;DR

- **Problem**: CI tests took 36-42 minutes per commit, blocking
developer workflow
- **Solution**: Implemented 16-way parallelization + optimized slow
tests + fixed Docker builds
- **Impact**: **1,358 hours/month saved** (7.7 FTE) across GitHub
Actions + CircleCI
  - GitHub Actions tests: 36m → 7m (82% faster)
  - Backend tests: 28m → 4m (87% faster with 16-way parallelization)
  - CircleCI tests: 42m → 7m (83% faster)
  - Docker builds: 34m → 5m (85% faster)
- **Result**: 5-6x faster feedback loops, 100% success rate on recent
runs

---

## Problem
CI test runs were taking **36-42 minutes per commit push** (GitHub
Actions: 36m avg, CircleCI: 42m P95), creating a significant bottleneck
in the development workflow.

## Solution
This PR comprehensively restructures both CI test pipelines to leverage
16-way parallelization and optimize test execution, reducing test
runtime from **36-42 minutes to ~7 minutes** - an **82% improvement**.

---

## 📊 Real Performance Data (Both CI Systems)

### GitHub Actions

#### Before (develop branch - 5 recent runs)
```
Individual runs: 35m 29s | 36m 1s | 40m 0s | 36m 4s | 34m 18s
Average: 36m 22s
```

#### After (feat/speed_up_ci branch - 9 successful runs)
```
Individual runs: 6m 39s | 7m 2s | 6m 53s | 6m 26s | 6m 52s | 6m 42s | 6m 45s | 6m 40s | 6m 37s
Average: 6m 44s
Range: 6m 26s - 7m 2s
```

**Improvement**: ⚡ **81.5% faster** (29m 38s saved per run)


#### Backend Tests Specific Impact
With 16-way parallelization, backend tests show dramatic improvement:
- **Before**: 27m 52s (sequential execution)
- **After**: 3m 44s (longest of 16 parallel runners)
  - Average across runners: 2m 30s
  - Range: 1m 52s - 3m 44s
- **Improvement**: ⚡ **86.6% faster** (24m 8s saved)
---

### CircleCI

#### Before (develop branch - CircleCI Insights)
```
Duration (P95): 41m 44s
Runs: 70 (last 30 days)
Success Rate: 84%
```

#### After (feat/speed_up_ci branch - Last 2 pipeline runs)
```
Run 1 (1h ago): 7m 7s
  ├─ lint: 4m 12s
  ├─ frontend-tests: 5m 36s
  ├─ backend-tests: 6m 23s
  ├─ coverage: 20s
  └─ build: 1s

Run 2 (2h ago): 7m 21s
  ├─ lint: 3m 47s
  ├─ frontend-tests: 5m 4s
  ├─ backend-tests: 6m 33s
  ├─ coverage: 19s
  └─ build: 1s

Average: 7m 14s
Success Rate: 100% ✅
```

**Improvement**: ⚡ **82.7% faster** (34m 30s saved per run)

---

## 🐳 Related Work: Docker Build Optimization

As part of the broader CI/CD optimization effort, Docker build
performance was improved separately in **PR #12859**.

### Docker Build Fix (Merged Separately)
**Problem**: Multi-architecture Docker builds (amd64/arm64) were taking
~34 minutes due to cache thrashing

**Solution**: Added separate cache scopes per platform in
`.github/workflows/test_docker_build.yml`:
```yaml
cache-from: type=gha,scope=${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ matrix.platform }}
```

**Results** (measured from November 2025 data):
- **Before**: 34.2 minutes/run average (15,547 minutes across 454 runs)
- **After**: 5 minutes/run
- **Improvement**: 85% faster, 29.2 minutes saved per run
- **Frequency**: 25.2 runs/day
- **Monthly savings**: **369 hours** (46 developer-days)

This prevents different architectures from invalidating each other's
caches and contributes 27% of total CI/CD time savings.

---

## 🎯 Key Findings

### Both CI Systems Now Perform Similarly
- **CircleCI**: 7m 14s average
- **GitHub Actions**: 6m 44s average
- **Difference**: Only 30 seconds apart (remarkably consistent!)

### Combined Performance
- **Average improvement across both systems**: **82.1% faster**
- **Time saved per commit**: ~32 minutes
- **Developer feedback loop**: 36-42 minutes → ~7 minutes

### Success Rate Improvement
- **CircleCI**: 84% → 100% (on feat/speed_up_ci branch)
- **GitHub Actions**: 100% (all 9 recent runs successful)
- Fixed all test isolation issues that caused intermittent failures

### Impact at Scale (Based on Real November 2025 Data)
- **CI runs per day**: **30.8 average** for tests, **25.2** for Docker
builds
  - Measured from GitHub Actions Usage Metrics (18 days)
  - Weekdays: 38-54 runs/day
  - Peak: up to 68 runs in a single day
- **This PR (test suite only)**:
  - **Daily time saved**: **15.3 hours** (GitHub Actions + CircleCI)
- **Monthly time saved**: **458 hours** (57 developer-days) on GitHub
Actions
  - Additional **531 hours** (66 developer-days) on CircleCI
- **Combined with Docker optimization** (PR #12859): **1,358
hours/month** (see Summary)
- **Developer experience**: 5-6x faster iteration cycles

---

## Code Changes

### 1. **Backend Test Parallelization (16x)**
Both CI systems now use 16-way parallelization with identical
round-robin test distribution:

```bash
# Distribute tests evenly across 16 runners
SPEC_FILES=($(find spec -name '*_spec.rb' | sort))
for i in "${!SPEC_FILES[@]}"; do
  if [ $(( i % 16 )) -eq $RUNNER_INDEX ]; then
    TESTS="$TESTS ${SPEC_FILES[$i]}"
  fi
done
```

**Why round-robin over timing-based?**
- CircleCI's timing-based splitting grouped similar tests together
- This caused race conditions with OAuth callback tests (Linear,
Shopify, Notion)
- Round-robin ensures even distribution and better test isolation
- Both CI systems now behave identically

### 2. **Frontend Test Optimization**
Enabled Vitest thread parallelization in `vite.config.ts`:

```typescript
pool: 'threads',
poolOptions: {
  threads: {
    singleThread: false,
  },
},
```

### 3. **CI Architecture Restructuring**
Split monolithic CI jobs into parallel stages:
- **Lint** (backend + frontend) - runs independently for fast feedback
- **Frontend tests** - runs in parallel with backend
- **Backend tests** - 16-way parallelized across runners
- **Coverage** - aggregates results from test jobs
- **Build** (CircleCI only) - final job for GitHub status check
compatibility

### 4. **Critical Test Optimization**
**report_builder_spec.rb**: Changed `before` to `before_all`
- Reduced execution time from **19 minutes to 1.2 minutes** (16x
speedup)
- Setup now runs once instead of 21 times
- Single biggest performance improvement after parallelization

---

## Test Stability Fixes (10 spec files)

Parallelization exposed latent test isolation issues that were fixed:

### Object Identity Comparisons (6 files)
Tests were comparing Ruby object instances instead of IDs:
- `spec/models/integrations/hook_spec.rb` - Use `.pluck(:id)` for
comparisons
- `spec/enterprise/models/captain/scenario_spec.rb` - Compare IDs
instead of objects
- `spec/models/notification_spec.rb` - Compare IDs for sort order
validation
- `spec/models/account_spec.rb` - Compare IDs in scope queries
- `spec/services/widget/token_service_spec.rb` - Compare class names
instead of class objects
- `spec/models/concerns/avatarable_shared.rb` - Use `respond_to` checks
for ActiveStorage

### Database Query Caching
- `spec/jobs/delete_object_job_spec.rb` - Added `.reload` to force fresh
database queries

### Test Expectations Timing
- `spec/jobs/mutex_application_job_spec.rb` - Removed flaky unlock
expectation after error block
  - Related to original PR #8770
- Expectation after error block never executes in parallel environments

### Timezone Handling
- `spec/mailers/account_notification_mailer_spec.rb` - Fixed date
parsing at timezone boundaries
  - Changed test time from 23:59:59Z to 12:00:00Z

### Test Setup
- `spec/builders/v2/report_builder_spec.rb` - Optimized with
`before_all`

---

##  CircleCI GitHub Integration Fix

### Problem
GitHub PR checks were stuck on "Waiting for status" even when all
CircleCI jobs passed. GitHub was expecting a job named `build` but the
workflow only had a workflow named "build".

### Solution
Added an explicit `build` job that runs after all other jobs:

```yaml
build:
  steps:
    - run:
        name: Legacy build aggregator
        command: echo "All main jobs passed"
  requires:
    - lint
    - coverage
```

This ensures GitHub's required status checks work correctly.


---

## ✅ Testing & Validation

- ✅ **GitHub Actions**: 9 successful runs, consistent 6m 26s - 7m 2s
runtime
- ✅ **CircleCI**: 2 successful runs, consistent 7m 7s - 7m 21s runtime
- ✅ Both CI systems produce identical, consistent results
- ✅ GitHub PR status checks complete correctly
- ✅ Success rate improved from 84% to 100% (recent runs)
- ✅ No test regressions introduced
- ✅ All flaky tests fixed (callback controllers, mutex jobs, etc.)

---

## 🎉 Summary

This PR delivers an **82% improvement** in test execution time across
both CI systems:

- **GitHub Actions tests**: 36m → 7m (81.5% faster)
- Backend tests specifically: 28m → 4m (86.6% faster)
- **CircleCI tests**: 42m → 7m (82.7% faster)
- **Developer feedback loop**: 5-6x faster
- **Test stability**: 84% → 100% success rate

### 📊 Total CI/CD Impact (All Optimizations)

Based on real November 2025 data, combining this PR with Docker build
optimization (PR #12859):

**Monthly Time Savings**: **1,358 hours/month** = **170
developer-days/month** = **7.7 FTE**

| System | Runs/Day | Before | After | Savings | Monthly Impact |
|--------|----------|---------|--------|---------|----------------|
| **GitHub Actions Tests** | 30.8 | 36.5m | 6.7m | 29.8m/run | 458 hrs
(34%) |
| **GitHub Actions Docker** | 25.2 | 34.2m | 5.0m | 29.2m/run | 369 hrs
(27%) |
| **CircleCI Tests** | 30.8 | 41.7m | 7.2m | 34.5m/run | 531 hrs (39%) |

*Data source: GitHub Actions Usage Metrics (November 2025, 18 days),
CircleCI Insights (30 days)*

The combined optimizations save the equivalent of **nearly 8 full-time
developers** worth of CI waiting time every month, significantly
improving developer velocity and reducing CI costs. All test isolation
issues exposed by parallelization have been fixed, ensuring reliable and
consistent results across both CI platforms.

woot woot !!!

---------
 2025-11-19 15:32:48 +05:30
Bruno Shiohei
4304e06748 fix: Updating the Ubuntu version to run the tests (#11260) 
In my previous PR I got some errors that when checking it is possible to
conclude that the version of ubuntu 20.04 in the Github Actions images
to run the tests is no longer supported.

![image](https://github.com/user-attachments/assets/f637e8da-3d15-43a1-8800-6c2785faf70c)

Possible to check at this url:
https://github.com/actions/runner-images/issues/11101

## What's changed
Changing the Ubuntu version in the image to run the tests.
Old version: 20.04
New version: 22.04

## BEFORE:

![image](https://github.com/user-attachments/assets/bd2dfdcd-3913-425e-a57c-504c7604d70a)

## NOW:

![image](https://github.com/user-attachments/assets/437ef89b-89d0-4e58-8e3c-ee3073dad417)
 2025-04-08 16:36:06 -07:00
Shivam Mishra
f2a7e1da6b fix: Corepack pnpm issue (#10840) 
So, a while back Circle CI builds and Heroku builds started to fail. 

From all the threads I read, it seems like the [npm registry rotated
it's signing
keys](https://github.com/pnpm/pnpm/issues/9014#issuecomment-2616589753)
New pnpm versions were signed with the new key. Corepack, however,
bundles a static set of trusted keys (from Node’s release), so it
continued verifying signatures only against the old key. When it
encountered packages signed with the new key, Corepack’s integrity check
failed with “Cannot find matching keyid” errors.This mismatch caused
Corepack’s integrity check to fail with “Cannot find matching keyid”
errors.

Workarounds include the following

1. Updating Corepack (to 0.31.0), they [upgraded their
package](https://github.com/nodejs/corepack/releases/tag/v0.31.0) to
include the new integrity check keys. But we seldom control what's going
on with the CI, also, updating this across our scripts is going to be a
painful task. Besides Heroku has [made some
fixes](https://github.com/heroku/buildpacks-nodejs/pull/1010) around
this
2. Disabling integrity checks 🔥 #YOLO
3. Pinning `pnpm` to older versions, or pinning it to a newer version
with the checksum in place.

Doing the third one here, running `corepack use pnpm@9.15.5` fixes this,
[ref](https://github.com/pnpm/pnpm/issues/9014#issuecomment-2623761494)

We can get rid of this over time as CDN caches used by build systems are
refreshed. But the change in this PR is not disruptive in anyway, only
rigidly secure.

Fixes: https://github.com/chatwoot/chatwoot/issues/10832

---

Here are the threads to follow 

- https://github.com/pnpm/pnpm/issues/9014
- https://github.com/pnpm/pnpm/issues/9029
- https://github.com/nodejs/corepack/issues/612
- https://github.com/nodejs/corepack/issues/616
- https://github.com/heroku/buildpacks-nodejs/pull/1010

---------

Co-authored-by: Vishnu Narayanan <vishnu@chatwoot.com>
 2025-02-05 10:12:29 -08:00
Pranav
d070743383 feat(ee): Add Captain features (#10665) 
Migration Guide: https://chwt.app/v4/migration

This PR imports all the work related to Captain into the EE codebase. Captain represents the AI-based features in Chatwoot and includes the following key components:

- Assistant: An assistant has a persona, the product it would be trained on. At the moment, the data at which it is trained is from websites. Future integrations on Notion documents, PDF etc. This PR enables connecting an assistant to an inbox. The assistant would run the conversation every time before transferring it to an agent.
- Copilot for Agents: When an agent is supporting a customer, we will be able to offer additional help to lookup some data or fetch information from integrations etc via copilot.
- Conversation FAQ generator: When a conversation is resolved, the Captain integration would identify questions which were not in the knowledge base.
- CRM memory: Learns from the conversations and identifies important information about the contact.

---------

Co-authored-by: Vishnu Narayanan <vishnu@chatwoot.com>
Co-authored-by: Sojan <sojan@pepalo.com>
Co-authored-by: iamsivin <iamsivin@gmail.com>
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
 2025-01-14 16:15:47 -08:00
Vishnu Narayanan
ee02923ace chore: fix circleci on vite build (#10214) 
- Switch to pnpm based build
- Switch circleci from docker to machine to have more memory
- Fix frontend and backend tests

Fixes
https://linear.app/chatwoot/issue/CW-3610/fix-circle-ci-for-vite-build
---------

Co-authored-by: Shivam Mishra <scm.mymail@gmail.com>
Co-authored-by: Pranav <pranavrajs@gmail.com>
Co-authored-by: Pranav <pranav@chatwoot.com>
 2024-10-07 15:27:41 +05:30
Shivam Mishra
42f6621afb feat: Vite + vue 3 💚 (#10047) 
Fixes https://github.com/chatwoot/chatwoot/issues/8436
Fixes https://github.com/chatwoot/chatwoot/issues/9767
Fixes https://github.com/chatwoot/chatwoot/issues/10156
Fixes https://github.com/chatwoot/chatwoot/issues/6031
Fixes https://github.com/chatwoot/chatwoot/issues/5696
Fixes https://github.com/chatwoot/chatwoot/issues/9250
Fixes https://github.com/chatwoot/chatwoot/issues/9762

---------

Co-authored-by: Pranav <pranavrajs@gmail.com>
Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com>
 2024-10-02 00:36:30 -07:00
Vishnu Narayanan
0d1b474e60 chore: upgrade to latest versions of gh helper actions (#8896) 
https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

Fixes: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/
 2024-02-09 10:56:36 +04:00
Shivam Mishra
e2a6dc3e04 chore: Upgrade Node.js to v20 (#7759) 
Co-authored-by: Vishnu Narayanan <vishnu@chatwoot.com>
Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
 2023-08-28 19:20:35 +05:30
Sojan Jose
480f34803b feat: Response Bot using GPT and Webpage Sources (#7518) 
This commit introduces the ability to associate response sources to an inbox, allowing external webpages to be parsed by Chatwoot. The parsed data is converted into embeddings for use with GPT models when managing customer queries.

The implementation relies on the `pgvector` extension for PostgreSQL. Database migrations related to this feature are handled separately by `Features::ResponseBotService`. A future update will integrate these migrations into the default rails migrations, once compatibility with Postgres extensions across all self-hosted installation options is confirmed.

Additionally, a new GitHub action has been added to the CI pipeline to ensure the execution of specs related to this feature.
 2023-07-21 18:11:51 +03:00
Vishnu Narayanan
26f164d6a0 fix: foss_spec github action (#6495) 
* fix: foss_spec gh action
* chore: upload log artifact in foss_spec gh action
* chore: ping foss_spec gh action runner os version to ubuntu-20.04
* fix: set nodeversion to 16
 2023-02-20 23:37:41 +05:30
Vishnu Narayanan
b1ec67d110 chore: upgrade ruby to 3.1.3 (#5555) 
* chore: update to ruby 3.1.3

* chore: ping docker version to alpine3.16 for nodev16.x

Starting with Node 17, nodejs switched to OpenSSL3. The docker builds
are installing node18.xx with alpine-3.1.3.

From Node.js 17's announcement post:

    If you hit an ERR_OSSL_EVP_UNSUPPORTED error in your application
with Node.js 17, it’s likely that your application or a module you’re
using is attempting to use an algorithm or key size which is no longer
allowed by default with OpenSSL 3.0. A new command-line option,
--openssl-legacy-provider, has been added to revert to the legacy
provider as a temporary workaround for these tightened restrictions.

Looks like a webpack issue. This is fixed in webpacl 5+ and we are on
webpack4 at the moment.
Solutions

    Upgrade webpack.
    Pin nodejs version to be 16.x.x
    Use  --openssl-legacy-provider as a workaround.

Pin docker version to alpine3.16 branch to have node16.x by default

ref:
https://github.com/chatwoot/chatwoot/pull/5555#issuecomment-1379778532

* chore: update webmock

* chore: fix ruby gem path in dockerfile

* chore: switch to node16 in circleci

* chore: update ruby version in linux installer script

* chore: update ruby version in linux installer script

* chore: fix circleci

* chore: fix circleci

* feat: upgrade node version to 16.x in linux installer

* chore: update systemd files

Co-authored-by: Sojan Jose <sojan@chatwoot.com>
 2023-01-24 23:55:07 +05:30
Vishnu Narayanan
568c30e93e chore: Update ruby and docker base image to 3.0.4 (#4693) 
* chore: Update ruby version to 3.0.4

* chore: update ruby version in docker workflow

* chore: update ruby version to 3.1.2

* Upgrade vue-jest to remove deasync

* Revert to 3.0.4 to see if deasync issue is fixed

* fix: script failure if pg/redis not opted in

Co-authored-by: Pranav Raj S <pranav@chatwoot.com>
 2022-05-27 17:33:24 +05:30
Vishnu Narayanan
45099f40f1 fix: ce spec action for PRs from forks (#4587) 
CE spec action was failing for PRs from external forks at the checkout stage. This PR modifies the checkout action to use the full repo name and branch.

#4586
 2022-04-29 14:38:10 +05:30
Vishnu Narayanan
75ce5345a9 feat: add gh action to build Chatwoot CE/foss docker image (#4406) 
Github action to build and push chatwoot-ce(foss) edition images. This action will run on merges to master, develop and when tags are created. Corresponding docker tags are as follows.

GitHub branch/tag  -->	docker tag
----
master 	--> latest-ce
develop 	---> develop-ce
v2.3.2 	---> v2.3.2-ce
v* 	---> v*-ce

Fixes #4388
 2022-04-06 21:14:04 +05:30