fix: Added authentication to FireCrawl API, remove unused RobinAI references (#10737)

- Fixed Firecrawl webhook payloads to ensure proper data handling and
delivery.
- Removed unused Robin AI code to improve codebase cleanliness and
maintainability.
- Implement authentication for the Firecrawl endpoint to improve
security. A key is generated to secure the webhook URLs from FireCrawl.

---------

Co-authored-by: Pranav <pranavrajs@gmail.com>

enterprise/app/models/captain/document.rb

@@ -29,6 +29,7 @@ class Captain::Document < ApplicationRecord
 
   validates :external_link, presence: true
   validates :external_link, uniqueness: { scope: :assistant_id }
+  validates :content, length: { maximum: 200_000 }
   before_validation :ensure_account_id
 
   enum status: {