fix: stream attachment handling in workers (#12870)

We’ve been watching Sidekiq workers climb from ~600 MB at boot to
1.4–1.5 GB after an hour whenever attachment-heavy jobs run. This PR is
an experiment to curb that growth by streaming attachments instead of
loading the whole blob into Ruby: reply-mailer inline attachments,
Telegram uploads, and audio transcriptions now read/write in chunks. If
this keeps RSS stable in production we’ll keep it; otherwise we’ll roll
it back and keep digging
This commit is contained in:
Sojan Jose
2025-12-05 13:02:53 -08:00
committed by GitHub
parent a971ff00f8
commit cc86b8c7f1
12 changed files with 203 additions and 74 deletions

View File

@@ -28,13 +28,14 @@ RSpec.describe Captain::Llm::PdfProcessingService do
context 'when uploading PDF to OpenAI' do
let(:mock_client) { instance_double(OpenAI::Client) }
let(:pdf_content) { 'PDF content' }
let(:blob_double) { instance_double(ActiveStorage::Blob) }
let(:pdf_file) { instance_double(ActiveStorage::Attachment) }
before do
allow(document).to receive(:openai_file_id).and_return(nil)
# Use a simple double for ActiveStorage since it's a complex Rails object
pdf_file = double('pdf_file', download: pdf_content) # rubocop:disable RSpec/VerifiedDoubles
allow(document).to receive(:pdf_file).and_return(pdf_file)
allow(pdf_file).to receive(:blob).and_return(blob_double)
allow(blob_double).to receive(:open).and_yield(StringIO.new(pdf_content))
allow(OpenAI::Client).to receive(:new).and_return(mock_client)
# Use a simple double for OpenAI::Files as it may not be loaded