1.4.2 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory


#21

@jeremy so I just did

reaction build --build-arg TOOL_NODE_FLAGS="--max-old-space-size=2048"  mycustom

with the same results, should I be doing that form or the

TOOL_NODE_FLAGS="--max-old-space-size=2048" reaction build  mycustom

form?

(building again now using the second one to see if there is any difference)

another thing that is a factor here is that I am constantly building from a very fresh state. Which specifically clones a fresh copy of reaction, performs meteor npm i, and then goes into a new reaction, which on my system /tmp is a ramdisk, and therefore refreshed daily or whenever I boot the machine


#22

@jeremy et al the second one fails with another faillog

  | +-- loose-envify@1.3.1  deduped
  | `-- warning@3.0.0  deduped
  `-- velocity-animate@1.5.0  deduped


[-] Building Meteor application...


Even with METEOR_ALLOW_SUPERUSER or --allow-superuser, permissions in your app
directory will be incorrect if you ever attempt to perform any Meteor tasks as
a normal user. If you need to fix your permissions, run the following command
from the root of your project:

  sudo chown -Rh <username> .meteor/local

cfs:tempstore: updating npm dependencies -- combined-stream...
cfs:gridfs: updating npm dependencies -- mongodb, gridfs-stream...

<--- Last few GCs --->

  380514 ms: Scavenge 1393.3 (1455.3) -> 1393.3 (1455.3) MB, 8.1 / 0 ms (+ 4.0 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
  382025 ms: Mark-sweep 1393.3 (1455.3) -> 1386.9 (1455.3) MB, 1510.8 / 0 ms (+ 6.0 ms in 2 steps since start of marking, biggest step 4.0 ms) [last resort gc].
  383515 ms: Mark-sweep 1386.9 (1455.3) -> 1393.3 (1455.3) MB, 1489.9 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3a090da37399 <JS Object>
    1: charCodeAt(aka charCodeAt) [native string.js:~53] [pc=0x29acc3a39186] (this=0x16381d71e9 <String[99]\: {                                                                                             // 1\n>,q=0)
    2: /* anonymous */ [/root/.meteor/packages/meteor-tool/.1.5.2_1.ogej5c++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/source-map/lib/so...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
/opt/build_scripts/build-meteor.sh: line 30:  5163 Aborted                 (core dumped) meteor build --directory $APP_BUNDLE_DIR
The command '/bin/sh -c cd $APP_SOURCE_DIR &&             $BUILD_SCRIPTS_DIR/install-deps.sh &&             $BUILD_SCRIPTS_DIR/install-node.sh &&             $BUILD_SCRIPTS_DIR/install-phantom.sh &&             $BUILD_SCRIPTS_DIR/install-mongo.sh &&             $BUILD_SCRIPTS_DIR/install-meteor.sh &&             $BUILD_SCRIPTS_DIR/build-meteor.sh &&             $BUILD_SCRIPTS_DIR/post-build-cleanup.sh' returned a non-zero code: 134

Error: Docker build failed. Exiting.

#23

Yeah, your second option wouldn’t work. Env vars in your local shell don’t pass through to a Docker build. That’s specifically why we had to use the --build-arg deal. It’s the only way to set values inside the container at build time.

In case you’re interested, this is how that happens in the Dockerfile…

So did the first command work for you?

For what it’s worth, I found that I needed to set it to 2048 on my Macbook and 4096 would fail, but we set it to 4096 on CircleCI because lower would fail. So I’m not exactly sure how to calculate the magic number there, but it’s worth experimenting a bit on different machines because it’s clearly not consistent.


#24

for plain old reaction to dev on, I find that installing Meteor 1.5.1 does the trick:

export METEOR_VERSION=1.5.1
curl https://raw.githubusercontent.com/reactioncommerce/base/master/scripts/install-meteor.sh | bash
reaction

However, the build comand, ends up installing Meteor 1.5.2.1 at some point in the chain. Which leads me to think at the at the moment the only way I can get a successful build is to pass that in as a build-arg:

reaction build --build-arg METEOR_VERSION=1.5.1  mycustom151

But that doesn’t seem to work either, 1.5.2.1 is in the final error message:

==== JS stack trace =========================================

Security context: 0x8e04ad37399 <JS Object>
    1: new constructor(aka SourceNode) [/root/.meteor/packages/meteor-tool/.**1.5.2_1**.ogej5c++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/source-map/lib/source-node.js:~35] [pc=0x3cc5326f48da] (this=0x1457f163339 <a SourceNode with map 0x27be2d2969b9>,aLine=1,aColumn=178,aSource=0x1457f163389 <String[62]: imports/plugins/core/ui/client/components/button/i...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
/opt/build_scripts/build-meteor.sh: line 30:  5175 Aborted                 (core dumped) meteor build --directory $APP_BUNDLE_DIR
The command '/bin/sh -c cd $APP_SOURCE_DIR &&             $BUILD_SCRIPTS_DIR/install-deps.sh &&             $BUILD_SCRIPTS_DIR/install-node.sh &&             $BUILD_SCRIPTS_DIR/install-phantom.sh &&             $BUILD_SCRIPTS_DIR/install-mongo.sh &&             $BUILD_SCRIPTS_DIR/install-meteor.sh &&             $BUILD_SCRIPTS_DIR/build-meteor.sh &&             $BUILD_SCRIPTS_DIR/post-build-cleanup.sh' returned a non-zero code: 134

How do I force meteor 1.5.1 in the build?

And the --build-arg TOOL_NODE_FLAGS="--max-old-space-size=2048" doesn’t seem to be helping. Though I might not have found the magic number for any of my build systems. So far I’ve tried 2048, 4096, 9096, and 14096.

EDIT: thanks @jeremy for letting me know the way to enforce meteor version for the build is thus:

echo 'METEOR@1.5.1'> .meteor/release

so success on one system and failure on the next system! Yay I love inconsistent errors!

successful build system 1 docker info

failure build system 2 docker info

the successful system does have 20 more gigs of ram than the other, but surely 12 gigs of ram should be sufficient to build in? And the second system with less RAM has over 20 gigs of swap enabled, and the first (successful) system has no swap enabled.

Quick table of success/fail of reaction build mycustom on those two systems, amongst the tag v1.4.3 and the marketplace branch

Meteor 1.5.1

System v143 marketplace
1 success success
2 success fail

Meteor 1.5.2.1

System v143 marketplace
1 fail fail
2 fail fail

Meteor 1.5.2.2

System v143 marketplace
1 fail fail
2 fail fail

Do other people here see similar results?


#25

I get unpredictable failures now in both of my user environments. The above table should not be looked as merely what I got this morning, and may or may not have relevance anymore.

The only reliable way I have to build at this point in time is to use the build script in my dockerized reaction dev environment. Which I made as kind of an experiment just to see if it would work, but at the moment it is the only reliable way for me to build locally.


#26

You could use the included CircleCI config. We use that every day and haven’t had the memory issue since setting the build flag months ago. You would just need to set a few environment variables in the CircleCI dashboard, but it would otherwise work fine without any modifications. Just add your Reaction fork to Circle and set these vars…

$DOCKER_USER - Docker Hub username
$DOCKER_PASS - Docker Hub password
$DOCKER_EMAIL - Docker Hub email
$DOCKER_NAMESPACE - the image org/name [Default]: reactioncommerce/reaction

Any pushes to master with a version tag formatted like v1.2.3 will build/tag your image and then push it to Docker Hub. If you want to customize how that works, everything is in this directory…

https://github.com/reactioncommerce/reaction/tree/master/.circleci


#27

@jeremy I’m getting out of memory errors on circleci too!, and again after a git fetch --all; git merge upstream master where upstream is the reactioncommerce/reaction repo. I am doing the same with marketplace as well

My fork of reaction can be seen here

The only thing I’ve added to this install is a theme

Though I get the out of memory error when I clone a fresh copy of reaction as well (notice all the circleci builds are master and marketplace, and I try and not do work in those branches on purpose), the table from yesterday was done by checking out clean copies and echoing into .meteor/release.


#28

OK, I may have nailed it down to one thing, setting those environment variables

and now I have a success

Is reaction build mycustom waiting around on those same env vars?

Interestingly enough, both master and marketplace are still failing


#29

Look at this one, I pushed the very same build that passes at reaction here

results in failure for me here

there is no code difference there, they are the same commit. The only difference must be env vars, and I have set the dockerhub ones that @jeremy provided

submitting a github issue

and just making a list of similar issues in meteor:

https://github.com/meteor/meteor/issues/8157
which leads to a more recent:
https://github.com/meteor/meteor/issues/9075

https://github.com/AdmitHub/meteor-buildpack-horse/issues/173

dying on minifier:


#30

I am still having the exact same issue.
Still stuck at v1.4.1, where it works fine.


#31

I have a table of success/fail at the github issue @yair do you have the same results?


#32

Yes, with an ubuntu machine. 8 gigs of ram.


#33

Same here, iMac with 16GB, fresh download of Reaction (master branch).

It was suggested to use

export TOOL_NODE_FLAGS="–max-old-space-size=2048"

and now I get

Installing NPM packages…

module.js:540
throw err;
^

Error: Cannot find module ‘…/lib/utils/unsupported.js’
at Function.Module._resolveFilename (module.js:538:15)
at Function.Module._load (module.js:468:25)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at /Users/scott/.meteor/packages/meteor-tool/.1.6.1.1t8atz8++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/bin/npm:19:21
at Object. (/Users/scott/.meteor/packages/meteor-tool/.1.6.1.1t8atz8++os.osx.x86_64+web.browser+web.cordova/mt-os.osx.x86_64/dev_bundle/bin/npm:92:3)
at Module._compile (module.js:643:30)
at Object.Module._extensions…js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)

Error: Node modules were not successfully installed. Exiting.


Fatal Error on starting the reaction
#34

Have you tried running reaction reset -y && reaction? This should empty node_modules and install dependencies from scratch.


#35

reaction reset -y gives the same error.
Error: Cannot find module ‘…/lib/utils/unsupported.js’


#36

I am having this same issue (OS X 10.12.6, 8GB RAM).

tsenkov$ reaction -v

Node: 9.11.1
NPM: 5.6.0
Meteor Node: 8.11.1
Meteor NPM: 5.6.0
Reaction CLI: 0.29.0
Docker: 18.03.0-ce