Product Import from RabbitMq


#1

Hi,

I’m new to Reaction Commerce & Meteor, my background is with PHP, Magento, etc.
However, I started a new pet project aiming to connect Reaction Commerce with Magento, sending data from one to another using RabbitMq queues.

I started out with this product import plugin https://github.com/getoutfitted/reaction-product-importer (confirmed that is working in browser ).
Added RabbitMq module to meteor: “meteor add jakobloekke:rabbitmq”

Added a new file in /imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js ( I’ll paste the content in a new post bellow ).
I had to wrap the callbacks with Meteor.bindEnvironment because I was getting error “Meteor code must always run within a Fiber.”.

When I start Reaction with a message in RabbitMq queue I receive the following error:
Exception in callback of async function: Error: Price must be a number
at getErrorObject (packages/aldeed_collection2-core.js:480:15)
at ns.Collection.doValidate (packages/aldeed_collection2-core.js:462:13)
at ns.Collection.Mongo.Collection.(anonymous function) (packages/aldeed_collection2-core.js:214:25)
at ns.Collection.Mongo.Collection.(anonymous function) [as insert] (packages/dispatch_run-as-user.js:325:19)
at Object.ProductImporter.createTopLevelProduct (imports/plugins/custom/reaction-product-importer/server/api/productImporter.js:189:36)
at imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js:54:49
at Function.
.each._.forEach (packages/underscore.js:147:22)
at imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js:53:15
at runWithEnvironment (packages/meteor.js:1240:24)

I tried to comment out the code related to price in this file https://github.com/getoutfitted/reaction-product-importer/blob/development/server/api/productImporter.js , lines 154 - 161.
After that, when I restarted Reaction, the product was created, with variants and options, but I couldn’t add it to cart because of the error bellow:

12:29:40.291Z ERROR Reaction:
[ { name: ‘items.0.product.price’,
type: ‘expectedObject’,
value: 0 } ] 'Invalid keys. Error adding to cart.'
12:29:40.290Z ERROR Reaction: Price must be an object
Error: Price must be an object
at getErrorObject (packages/aldeed_collection2-core.js:480:15)
at ns.Collection.doValidate (packages/aldeed_collection2-core.js:462:13)
at ns.Collection.Mongo.Collection.(anonymous function) (packages/aldeed_collection2-core.js:214:25)
at ns.Collection.Mongo.Collection.(anonymous function) [as update] (packages/dispatch_run-as-user.js:325:19)
at DDPCommon.MethodInvocation.cartAddToCart (server/methods/core/cart.js:448:29)
at packages/check.js:128:16
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:15)
at Object._failIfArgumentsAreNotAllChecked (packages/check.js:127:41)
at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1765:18)
at DDP._CurrentMethodInvocation.withValue (packages/ddp-server/livedata_server.js:719:19)
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:15)
at DDPServer._CurrentWriteFence.withValue (packages/ddp-server/livedata_server.js:717:46)
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:15)
at Promise (packages/ddp-server/livedata_server.js:715:46)
at new Promise ()
at Session.method (packages/ddp-server/livedata_server.js:689:23)
at packages/ddp-server/livedata_server.js:559:43

I also got the following error that was temporarily ‘fixed’ by commenting the lines 191-193 from same file reaction-product-importer/blob/development/server/api/productImporter.js .

Exception in callback of async function: Error: Access Denied [access-denied]
at DDPCommon.MethodInvocation.productsUpdateProductTags (server/methods/catalog.js:991:13)
at packages/check.js:128:16
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:15)
at Object.failIfArgumentsAreNotAllChecked (packages/check.js:127:41)
at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1765:18)
at DDP.CurrentMethodInvocation.withValue (packages/ddp-server/livedata_server.js:1686:15)
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1186:15)
at resolve (packages/ddp-server/livedata_server.js:1684:36)
at new Promise ()
at Server.applyAsync (packages/ddp-server/livedata_server.js:1683:12)
at Server.apply (packages/ddp-server/livedata_server.js:1622:26)
at Server.call (packages/ddp-server/livedata_server.js:1604:17)
at imports/plugins/custom/reaction-product-importer/server/api/productImporter.js:195:12
at Array.forEach ()
at Function.
.each.
.forEach (packages/underscore.js:139:11)
at Object.ProductImporter.createTopLevelProduct (imports/plugins/custom/reaction-product-importer/server/api/productImporter.js:194:5)
at imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js:54:49
at Function.
.each..forEach (packages/underscore.js:147:22)
at imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js:53:15
at runWithEnvironment (packages/meteor.js:1240:24)
=> awaited here:
at Promise.await (/Users/mac/.meteor/packages/promise/.0.10.0.wb3q6c.q34++os+web.browser+web.cordova/npm/node_modules/meteor-promise/promise_server.js:60:12)
at Server.apply (packages/ddp-server/livedata_server.js:1635:14)
at Server.call (packages/ddp-server/livedata_server.js:1604:17)
at imports/plugins/custom/reaction-product-importer/server/api/productImporter.js:195:12
at Array.forEach ()
at Function.
.each..forEach (packages/underscore.js:139:11)
at Object.ProductImporter.createTopLevelProduct (imports/plugins/custom/reaction-product-importer/server/api/productImporter.js:194:5)
at imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js:54:49
at Function.
.each.
.forEach (packages/underscore.js:147:22)
at imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js:53:15
at runWithEnvironment (packages/meteor.js:1240:24)

Finally, the last error I receive when adding a product to queue after Reaction was started and product is picked up by RabbitMq consumer.
The error is:

Exception in callback of async function: TypeError: Cannot read property ‘namedContext’ of null
at ns.Collection.doValidate (packages/aldeed_collection2-core.js:303:32)
at ns.Collection.Mongo.Collection.(anonymous function) (packages/aldeed_collection2-core.js:214:25)
at ns.Collection.Mongo.Collection.(anonymous function) [as insert] (packages/dispatch_run-as-user.js:325:19)
at Object.ProductImporter.createMidLevelVariant (imports/plugins/custom/reaction-product-importer/server/api/productImporter.js:239:36)
at imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js:60:56
at Function.
.each..forEach (packages/underscore.js:147:22)
at imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js:59:19
at Function.
.each._.forEach (packages/underscore.js:147:22)
at imports/plugins/custom/reaction-product-importer/server/methods/rabbitmq.js:53:15
at runWithEnvironment (packages/meteor.js:1240:24)

Can you give me a hint about what am I doing wrong to get the mentioned errors?

I’ll post the content of rabbitmq.js file bellow and an example of product data from RabbitMq.

Thanks in advance!


#2

Content of rabbitmq.js file:

import { Meteor } from 'meteor/meteor';
import { check } from 'meteor/check';
import { Packages } from '/lib/collections';
import { Reaction, Logger } from '/server/api';
import { _ } from 'meteor/underscore';
import { ProductImporter } from '../api';

Logger.warn('RBMQ import file loaded');

Meteor.startup(function () {
    var options = {
        host: '127.0.0.1',
        port: 5672,
        login: 'guest',
        password: 'guest',
        vhost: '/'
    };

    let i = 0;

    const handle = Meteor.setInterval(() => {
      i++;
      const shopId = Reaction.getShopId();

      if (shopId) {
        RabbitMQ.ensureConnection(options);
        return Meteor.clearInterval(handle);
      }

      if (i > 30) {
        // stop checking and warn if the shopId isn't available within 30 secs
        Meteor.clearInterval(handle);
        Logger.warn("Error getting shopId for 'RabbitMQ.ensureConnection()'");
      }

      return null;
    }, 1000);

});

RabbitMQ.on('ready', Meteor.bindEnvironment( function () {

    RabbitMQ.connection.queue('product.updates', {durable: true, autoDelete: false}, Meteor.bindEnvironment( function (q) {

        q.subscribe({ack: true}, Meteor.bindEnvironment( function (productsList) {

            // your code for handling incoming messages goes here ...
            Logger.warn('!!!!!!!!!!!! Message consume start !!!!!!!!!');

            check(productsList, [Object]);
            let productsById = ProductImporter.groupBy(productsList, 'productId');

            _.each(productsById, function (product) {
                let productId = ProductImporter.createTopLevelProduct(product);
                let ancestors = [productId];
                // group each variant by variant title
                let variantGroups = ProductImporter.groupBy(product, 'variantTitle');

                _.each(variantGroups, function (variantGroup) {
                  let variantGroupId = ProductImporter.createMidLevelVariant(variantGroup, ancestors);
                  let variantAncestors = ancestors.concat([variantGroupId]);
                  // create each sub variant
                  _.each(variantGroup, function (variant) {
                    ProductImporter.createVariant(variant, variantAncestors);
                  });
                });

                Logger.warn('!!!!!!!!!!!! Inside _.each !!!!!!!!!');
            });

            Logger.warn('!!!!!!!!!!!! Message consumed !!!!!!!!!');

            q.shift(); // Only necessary if {ack: true}

        }));
    }));

}));

#3

Example of RabbitMq queue entry:

[  
   {  
      "productId":"1817",
      "topProductType":"simple",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"Breathe-Easy Tank PDP Title",
      "vendor":"Default Vendor",
      "handle":"breathe-easy-tank",
      "variantTitle":"XS",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - XS - Purple",
      "optionTitle":"Purple",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":"<p>The Breathe Easy Tank is so soft, lightweight, and comfortable, you won't even know it's there -- until its high-tech Cocona&reg; fabric starts wicking sweat away from your body to help you stay dry and focused. Layer it over your favorite sports bra and get moving.<\/p>\n<p>&bull; Machine wash\/dry.<br \/>&bull; Cocona&reg; fabric.<\/p>"
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"XS",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - XS - White",
      "optionTitle":"White",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"XS",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - XS - Yellow",
      "optionTitle":"Yellow",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"S",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - S - Purple",
      "optionTitle":"Purple",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"S",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - S - White",
      "optionTitle":"White",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"S",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - S - Yellow",
      "optionTitle":"Yellow",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"M",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - M - Purple",
      "optionTitle":"Purple",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"M",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - M - White",
      "optionTitle":"White",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"M",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - M - Yellow",
      "optionTitle":"Yellow",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"L",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - L - Purple",
      "optionTitle":"Purple",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"L",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - L - White",
      "optionTitle":"White",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"L",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - L - Yellow",
      "optionTitle":"Yellow",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"XL",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - XL - Purple",
      "optionTitle":"Purple",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"XL",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - XL - White",
      "optionTitle":"White",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   },
   {  
      "productId":"1817",
      "topProductType":"",
      "productTitle":"Breathe-Easy Tank",
      "pageTitle":"",
      "vendor":"Default Vendor",
      "handle":"",
      "variantTitle":"XL",
      "variantType":"variant",
      "title":"Breathe-Easy Tank - XL - Yellow",
      "optionTitle":"Yellow",
      "price":"34.0000",
      "qty":100,
      "weight":"1.0000",
      "taxable":"true",
      "hashtags":"Shop",
      "metatags":"",
      "description":""
   }
]

#4

So the basic problem is that your product is not mapping to the schema. If you bypass this checking it’s going to cause problems upstream. I know that product importer is a little outdated, so you may need to make adjustments to the mapping.


#5

Hi,

Mapping should be OK because I tested the product importer in browser and it’s working fine. The original product import behaviour was to parse a CSV file on client side and send product data to server to be processed.

Now, with my changes, the calls are just server side.
Probably that’s why I receive error ‘Access Denied’ when calling productsUpdateProductTags method?
What puzzles me though is why I receive error ‘Price must be a number’ when trying to add product from RabbitMq, but when adding to cart the error is ‘Price must be an object’.
I’m sure the correct type of Price field is object and product importer accepts price as object when ran from browser.

Thanks!
Florian


#6

Note that there are two separate schemas for products, one for top-level products called “simple” (which require an object for a price") and one called “variant” which require a number for a price. When inserting them you need to select with { type: "simple" } or { type: "variant" }. I suspect that’s what’s happening here is that it’s trying to validate a variant with the “simple” schema.