Reaction Commerce Forums

Use Amazon S3 for Product Images

OOTB Reaction Commerce v3 provides two storage adapters:

  1. One for Mongo GridFS.
  2. Another for Meteor CollectionFS.

By default GridFS is used and media uploaded via the API using Reaction Admin are stored in MongoDB in several tables starting with cfs_gridfs. The DB-based file storage works well for small amounts of media and all the basics are there currently for managing product images.

For each product image uploaded via local disk the API kicks off a saveImage/local job via the createSaveImageJob utility function in the api-plugin-files package. The job is added to the Jobs collection in MongoDB and the status is updated as soon as the job is processed.

As image/local jobs are processed file workers are kicked off and 5 copies of various image quality are created using sharp. The same transformations occur for image/remote jobs as well. This creates 5 images for every 1 uploaded.

Using MongoDB presents scalaing issues:

There are quite a few drawbacks to using blobs to store media. First, database storage is typically expensive. When your catalog grows, you don’t want to have 50GB of media in a MongoDB instance. It doesn’t make sense. Second, with a lot of traffic, it’ll clog up your app <-> database link because it’ll use so much bandwidth. Not a good thing either, and can cause some pretty serious issues in production during traffic spikes. These are both large-scale examples, but they’re definitely real.

—Loan Laux, Oct 29, 2019

Another scaling issue occurs in development when restoring database backups which slows down development and reduces a businesses ability to fail fast—especially under limited bandwidth.

To overcome this shortcoming of Reaction an Object Storage adapter was created by Loan:

The adapter was made possible as a result of the thoughtful abstractions in reaction-file-collections. As the S3 adapter was created some forks of api-plugin-files have shown up using it. Based on the docs for the S3 adapter it should be fairly straight-forward to implement. However, it seems there are some issues which arise due to assumptions made related to use of GridFS including:

  • Files from S3 may need to be streamed back through Reaction API instead of downloaded by users directly using a URL to, for example, CloudFlare, an S3 endpoint or a server.
  • The README for reaction-file-collections suggests multiple Stores can be be created to keep some images in the DB and offload others to S3 but provides no working implementation.

These limitations of scale become apparent fairly quickly in new Reaction Commerce apps. And it’s not clear what the best solutions is yet to handle the problem. Let’s discuss.