How to do a automatic JSON import / sync?


#1

Hey there.

At the moment I’m looking at all the different shop systems to find a great soluation for a business idea that was roaming around in my mind for quite some time. After some research I found reaction and so far I am really impressed by the performance and easy to use attitude of the backend.

Unfortunately I am completely new to node.js. I am one the good PHP dinosaurs :slight_smile: But I am very willing to get into node as this is definetly a great advance in tech.

My goal will be to gather products from an ERP that so far exports it’s products into one or more JSONs in regular cycles. So for two nights I was trying to figure out if reaction and it’s reaction.import method is somehow able to collect these JSON files and interpret them correctly.

But I am already stuck with reading those files. Can anybody please give me a hint how to proceed in a way that would make sense for my futur development? So far I created a simple plugin that on server side does basically this:

Content of the init.js:
import { Meteor} from “meteor/meteor”;
import { Reaction } from “/server/api”;

Meteor.startup(function () {
	Reaction.Import.process(Assets.getText("custom/Products.json"), ["Products from API"], Reaction.Import.product);
	Reaction.Import.flush();
});

To rule out errors I basically grabbed a copy of the Products.json (the fixture file in /private/data/) modified the product id and put it in /private/custom/

While firing up reaction it throws an error which I just can’t figure out

/Users/XYZ/.meteor/packages/meteor-tool/.1.4.4_2.1sgods7++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:280
throw(ex);
^
Error: Match error: Expected object, got undefined
at exports.check (packages/check.js:57:15)
at Object.Import.product (server/api/core/import.js:214:1)
at Object.Import.process (server/api/core/import.js:430:1)
at imports/plugins/custom/importer/server/init.js:6:18
at Function.time (/Users/XYZ/reaction/.meteor/local/build/programs/server/profile.js:309:28)
at /Users/XYZ/reaction/.meteor/local/build/programs/server/boot.js:312:13
at /Users/XYZ/reaction/.meteor/local/build/programs/server/boot.js:353:5
at Function.run (/Users/XYZ/reaction/.meteor/local/build/programs/server/profile.js:510:12)
at /Users/XYZ/reaction/.meteor/local/build/programs/server/boot.js:351:11
=> Exited with code: 1

So my first question is: Is this a meteor config problem or a problem of my plugin. Its init.js gets mentioned.

My second question is: If I manage to get the import done somehow, how do I sync reaction with those files regulary?

Any help would be highly appreciated!

Regards from Berlin
bob


#2

re: import

I’m not exactly sure if that is your error, but where you have ["Products from API"] you want to have the name of a field that should be used for the key field (in the case of Products, it should be “title”).

Re: syncing

You should look at the Job Control package, this allows you to queue jobs to run at certain intervals and you could leverage that to go out and periodically sync files.


#3

Hello Brent. Thanks for the super fast reply and please excuse my very late one :slight_smile:

I replaced the string with “title” but it didnt solve the error, that is thrown. Do you have any other idea what the reason could be?

I will also dig into Job Control tonight and hope to find a solution, thanks for the hint!

The more I get to know reaction the more I like it. Let’s see, maybe it will become my shop system of choice! :slight_smile:

Regards
Bob


#4

I’m trying to eliminate possible error sources, but without success so far.

Is set up a complete fresh installation on a different computer and I get the same error. So I guess it is not related to a wrong basic configuration as there basically is none I altered.

Next thing I tried is to follow the exact procedure described in the documentation regarding the import of shipping info from JSON.

init.js:

import { Meteor} from "meteor/meteor";
import { Reaction } from "/server/api";

Meteor.startup(function () {
	Reaction.Import.process(Assets.getText("custom/Shipping.json"), ["name"], Reaction.Import.shipping);
	Reaction.Import.flush();
});

The “custom/Shipping.json” contains the exact structure as the one in /data/Shipping.js". And as far as I understand the latter is the one reaction imports it’s shipping info on start. Right? The only things I changed is the name and label tags to identify the new information later on.

So when I start up reaction it starts with no error. But when I look into my reaction backend, I can’t find the new shipping info.

I’m stuck :frowning:

FYI, the custom Shipping.json looks like this:
> [{
> “name”: “test_shipping”,
> “methods”: [{
> “name”: “test1”,
> “label”: “Test 1”,
> “group”: “Ground”,
> “rate”: 0,
> “validLocales”: [{
> “deliveryBegin”: 2,
> “deliveryEnd”: 7
> }],
> “validRanges”: [{
> “begin”: 50
> }]
> }, {
> “name”: “test2”,
> “label”: “Test 2”,
> “group”: “Ground”,
> “rate”: 4.95,
> “validLocales”: [{
> “deliveryBegin”: 2,
> “deliveryEnd”: 7
> }]
> }, {
> “name”: “test3”,
> “label”: “Test 3”,
> “group”: “Priority”,
> “rate”: 7.95,
> “validLocales”: [{
> “deliveryBegin”: 1,
> “deliveryEnd”: 3
> }]
> }],
> “provider”: {
> “name”: “test_shipping”,
> “label”: “Test”
> }
> }]


#5

Well, there could be a lot of things going on. What you want to do is not use Meteor.startup, you want to use a hook so that your import runs after the system in initialized. You will want to run your code in an afterInit hook like this:

https://github.com/reactioncommerce/reaction-example-plugin/blob/master/server/init.js#L51

#6

Hey Brent. That actually worked a little better. On start reaction doesn’t throw the error anymore :slight_smile: Thanks!

My plugin init.js now looks like that

import { Reaction } from "/server/api";
import { Hooks } from "/server/api"
Hooks.Events.add("afterCoreInit", () => {
	Reaction.Import.process(Assets.getText("custom/Shipping.json"), ["name"], Reaction.Import.shipping);
	Reaction.Import.flush();
});

But the thing is, I now expected the new shipping information to show up in the reaction backend. The sidebar menu still only lists the default “Flat rate” and “Shippo”. My “Test” isn’t there. Is there anything I have to do else to get the plugin initialized?

Regards
bob


#7

If you want to have a whole other method of shipping (parallel to Flat Rate and Shippo) then you need to do more than just add a db record, you will need to create a plugin, as that Collection is just for maintaining the Flat Rate shipping method. Basically you will want to emulate the code that’s in the shipping-rates plugin.


#8

Ah ok. My intend was just to use the shipping to test the import, because the shipping method and fixture JSON looked less complex. My main goal still is to import products from a JSON.

So I used my last state of the init.js which didnt throw an error: Reaction.Import.process(Assets.getText(“custom/Products.json”), ["_id"], Reaction.Import.product);

But with plugin I get the error again :frowning:
> > /Users/XYZ/.meteor/packages/meteor-tool/.1.4.4_2.1sgods7++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/server-lib/node_modules/fibers/future.js:280
> > throw(ex);
> > ^
> > Error: Match error: Expected object, got undefined
> > at exports.check (packages/check.js:57:15)
> > at Object.Import.product (server/api/core/import.js:214:1)
> > at Object.Import.process (server/api/core/import.js:430:1)
> > at imports/plugins/custom/importer/server/init.js:7:18
> > at server/api/hooks.js:60:1
> > at Array.reduce (native)x
> > at Object.Hooks.Events.run (server/api/hooks.js:60:1)
> > at Object.module.export.exports.default.init (server/api/core/core.js:46:18)
> > at module.export.exports.default (server/startup/init.js:15:2)
> > at module.export.exports.default (server/startup/index.js:18:2)
> > at server/main.js:8:2
> > at Function.time (/Users/XYZ/reaction/.meteor/local/build/programs/server/profile.js:309:28)
> > at /Users/XYZ/reaction/.meteor/local/build/programs/server/boot.js:312:13
> > at /Users/XYZ/reaction/.meteor/local/build/programs/server/boot.js:353:5
> > at Function.run (/Users/XYZ/reaction/.meteor/local/build/programs/server/profile.js:510:12)
> > at /Users/XYZ/reaction/.meteor/local/build/programs/server/boot.js:351:11

My problem with the error is that it doesnt give me enough information on what the problem is. Debugging is quite hard like that :frowning:


#9

Can you try your code against this branch: joykare-testing-2066? I think I may have discovered a bug in the Import code that you may be running into as well and it’s fixed on that branch.


#10

Thanks Brent! I pulled the branch succesfuly. While starting up reaction I encountered quite some error which I first have to figure out, before I can test my code. I’ll get back to you once I got rid of the problems.

Warning: unable to resolve "es2015" in presets of /Users/XYZ/reaction/.babelrc
=> Errors prevented startup:
   
   While processing files with ecmascript (for target web.browser):

   /Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/plugin.js:131:13:
   Plugin 0 specified in "foreign" provided an invalid property of
   "__wrapped__"
   at Plugin.init
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/plugin.js:131:13)
   at Function.normalisePlugin
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:152:12)
   at
   /Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:184:30
   at Array.map (native)
   at Function.normalisePlugins
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:158:20)
   at OptionManager.mergeOptions
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:234:36)
   at
   /Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:265:14
   at
   /Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:323:22
   at Array.map (native)
   at OptionManager.resolvePresets
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:275:20)
   at OptionManager.mergePresets
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:264:10)
   at OptionManager.mergeOptions
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:249:14)
   at OptionManager.init
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)
   at File.initOptions
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/index.js:212:65)
   at new File
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/index.js:135:24)
   at Pipeline.transformFromAst
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/pipeline.js:69:16)
   at transform
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/meteor-babel/index.js:52:30)
   at compile
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/meteor-babel/index.js:70:14)
   at Cache.Cp.get
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/meteor-babel/cache.js:94:19)
   at Object.compile
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/meteor-babel/index.js:27:23)
   at Object.Babel.compile (packages/babel-compiler.js:48:24)
   at packages/babel-compiler.js:174:22
   at Function.time (/tools/tool-env/profile.js:305:10)
   at profile (packages/babel-compiler.js:217:20)
   at BabelCompiler.BCp.processOneFileForTarget
   (packages/babel-compiler.js:173:20)
   at BabelCompiler.<anonymous> (packages/babel-compiler.js:105:26)
   at Array.forEach (native)
   at BabelCompiler.BCp.processFilesForTarget
   (packages/babel-compiler.js:104:14)
   
   
   While processing files with less (for target web.browser):
   imports/plugins/included/default-theme/client/styles/main.less:150: Unknown
   import: {}/node_modules/react-select/less/select.less
   
   While processing files with ecmascript (for target os.osx.x86_64):

   /Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/logger.js:41:11:
   [BABEL] /imports/plugins/core/discounts/lib/collections/schemas/config.js:
   Unknown option: foreign.Children. Check out
   http://babeljs.io/docs/usage/options/ for more information about options.
   
   A common cause of this error is the presence of a configuration options
   object without the corresponding preset name. Example:
   
   Invalid:
   `{ presets: [{option: value}] }`
   Valid:
   `{ presets: [['presetName', {option: value}]] }`
   
   For more detailed information on preset configuration, please see
   http://babeljs.io/docs/plugins/#pluginpresets-options.
   at Logger.error
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/logger.js:41:11)
   at OptionManager.mergeOptions
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:226:20)
   at
   /Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:265:14
   at
   /Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:323:22
   at Array.map (native)
   at OptionManager.resolvePresets
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:275:20)
   at OptionManager.mergePresets
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:264:10)
   at OptionManager.mergeOptions
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:249:14)
   at OptionManager.init
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)
   at File.initOptions
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/index.js:212:65)
   at new File
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/file/index.js:135:24)
   at Pipeline.transformFromAst
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/babel-core/lib/transformation/pipeline.js:69:16)
   at transform
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/meteor-babel/index.js:52:30)
   at compile
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/meteor-babel/index.js:76:14)
   at Cache.Cp.get
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/meteor-babel/cache.js:94:19)
   at Object.compile
   (/Users/XYZ/.meteor/packages/ecmascript/.0.7.3.1soiyxg++os+web.browser+web.cordova/plugin.compile-ecmascript.os/npm/node_modules/meteor/babel-compiler/node_modules/meteor-babel/index.js:27:23)
   at Object.Babel.compile (packages/babel-compiler.js:48:24)
   at packages/babel-compiler.js:174:22
   at Function.time (/tools/tool-env/profile.js:305:10)
   at profile (packages/babel-compiler.js:217:20)
   at BabelCompiler.BCp.processOneFileForTarget
   (packages/babel-compiler.js:173:20)
   at BabelCompiler.<anonymous> (packages/babel-compiler.js:105:26)
   at Array.forEach (native)
   at BabelCompiler.BCp.processFilesForTarget
   (packages/babel-compiler.js:104:14)

#11

You need to do a meteor npm install after switching to that branch. There are different packages installed.


#12

Thanks Brent, that one worked.

Reaction is starting with no errors regarding my import plugin. So my guess is that it’s being loaded. But there is no new product showing up in the shop so far.

Is there anything else I have to do to get my test product (/private/custom/Products.json) into the database? As far as I understand the “Reaction.Import.flush();” should do the trick, but it doesnt so far.

But in general I really have to say thank you a lot for all your support to novices like me. I really appreciate that!


#13

I have to apologize, it worked :slight_smile: It was my fault of course, I put it in the wrong directory. My test product just showed up, yes!!

Thank you, Brent. I guess your fix will be part of a future master branch, right?


#14

Can I ask you another question?

Is there an update or sync function in the import module? I’m asking because when I try to import the same file again, but with altered product values, reaction throws an error that I’m trying to import data with a duplicate key.

That’s basically correct, but not very handy as I want to update existing products and not to import them again. Is there a way to update existing products instead of just importing them as new products with new IDs?


Import products from CSV file