1.4.2 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory


#1

On a default reaction init; cd reaction; reaction I have seen this more than once:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory                                                    │················································································································
/opt/build_scripts/build-meteor.sh: line 30:  5163 Aborted                 (core dumped) meteor build --directory $APP_BUNDLE_│················································································································
DIR  

on machines with tons of available ram:

tekromancer@apollo:~/reaction$ free -m
              total        used        free      shared  buff/cache   available
Mem:          64548       34425        2262        1377       27859       28414
Swap:          5683         643        5040

Anyone else seeing this?


#2

I am getting the same error, but I thought I am doing something wrong. Looks like I am not along.


#3

Funny thing is it does seem intermittent, I do reaction again, and it works, but then reaction build fails, etc.


#4

It’s a first time I am trying to run RC at my local machine and it’s failing during “reaction run”. Sometimes after Proxy started, sometimes after Mongo started


#5

@gordondm try another branch like

git checkout v1.4.1

or

git checkout marketplace

both of those are running fine for me, only 1.4.2 seems to be affected


#6

@thoth it seems don’t work for me…


#7

@mikicho v1.4.1? or marketplace?


#8

@thoth the problem occur in both branches


#9

Can you copy and paste the error?


#10

v1.4.1 branch:

PS E:\projects\js\MAYtsWebsite\reaction> git checkout v1.4.1
Note: checking out 'v1.4.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at edf85dc9b... Release 1.4.1 (#2583)
M       packages/gridfs/.npm/package/.gitignore
M       packages/gridfs/.npm/package/README
M       packages/gridfs/.npm/package/npm-shrinkwrap.json
M       packages/tempstore/.npm/package/.gitignore
M       packages/tempstore/.npm/package/README
M       packages/tempstore/.npm/package/npm-shrinkwrap.json
PS E:\projects\js\MAYtsWebsite\reaction> reaction

A newer version of Reaction exists on Github.

Current version: 1.4.1
Available version: 1.4.2

To update, run: reaction pull


Using settings file at settings/dev.settings.json

Setting up plugin imports...

Setting up style imports...

[[[[[ ~\E\projects\js\MAYtsWebsite\reaction ]]]]]

=> Started proxy.
=> Meteor 1.5.2 is available. Update this project with 'meteor update'.
=> Started MongoDB.
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

Marketplace branch:

PS E:\projects\js\MAYtsWebsite\reaction> git checkout marketplace
Checking out files: 100% (844/844), done.
Switched to a new branch 'marketplace'
M       packages/gridfs/.npm/package/.gitignore
M       packages/gridfs/.npm/package/README
M       packages/gridfs/.npm/package/npm-shrinkwrap.json
M       packages/tempstore/.npm/package/.gitignore
M       packages/tempstore/.npm/package/README
M       packages/tempstore/.npm/package/npm-shrinkwrap.json
Branch marketplace set up to track remote branch marketplace from origin.
PS E:\projects\js\MAYtsWebsite\reaction> reaction

Using settings file at settings/dev.settings.json

Setting up plugin imports...

Installing dependencies for connectors...

npm WARN reaction-connectors@1.0.0 No repository field.
Installing dependencies for connectors-shopify...

npm WARN reaction-connectors-shopify@1.0.0 No repository field.
Setting up style imports...

[[[[[ ~\E\projects\js\MAYtsWebsite\reaction ]]]]]

=> Started proxy.
Warning: unable to resolve "env" in presets of /E/projects/js/MAYtsWebsite/reaction/package.json

Unable to resolve some modules:

  "recompose" in
/E/projects/js/MAYtsWebsite/reaction/imports/plugins/included/sms/client/containers/smsSettingsContainer.js
(web.browser)

If you notice problems related to these missing modules, consider running:

  meteor npm install --save recompose

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

#11

Any update, I am getting same issue with 1.4.1 branch.

George


#12

Hi, is there a way to increase the memory allocation? I have the same problem with both versions.


#13

I am having the same issue with version 1.4.2 and the new meteor version 1.5.2
It might be related to this: https://github.com/meteor/meteor/issues/8648


#14

On version 1.4.0 and 1.4.2 -

/opt/build_scripts/build-meteor.sh: line 30: 5136 Killed meteor build --directory $APP_BUNDLE_DIR
The command ‘/bin/sh -c cd $APP_SOURCE_DIR && $BUILD_SCRIPTS_DIR/install-deps.sh && $BUILD_SCRIPTS_DIR/install-node.sh && $BUILD_SCRIPTS_DIR/install-phantom.sh && $BUILD_SCRIPTS_DIR/install-mongo.sh && $BUILD_SCRIPTS_DIR/install-meteor.sh && $BUILD_SCRIPTS_DIR/build-meteor.sh && $BUILD_SCRIPTS_DIR/post-build-cleanup.sh’ returned a non-zero code: 137

Error: Docker build failed. Exiting.


#15

It seems this is most prevalent during a first run. In most cases, restart reaction and it goes through just fine. Let me know if this problem ever persists, and if so watch the memory usage and let me know at how much it is consuming when it finally dies (and how much memory is left over in your system).

I have been looking into this a bit more to see why its consuming over 3 gigs of ram when it does crash (*I’ve caught it a few times now at just over 3 gigs when it dies) and I have 12 gigs of RAM in this machine, 3 gigs is no sweat (with just under 9 gig free at the time the out of memory fail happens).


#16

Reaction commerce - could you please advice on this - I updated to the latest reaction commerce 1.4.3, meteor 1.5.2.1, try several versions of nodejs in docker, OS X, Amazon Linux, 16GB RAM, increased node memory allocation using --max-old-space-size=XXXX up to 10g, no luck, still same error on same place.

Thank you, George

cfs:tempstore: updating npm dependencies – combined-stream…
cfs:gridfs: updating npm dependencies – mongodb, gridfs-stream…
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

<— Last few GCs —>

241868 ms: Scavenge 1387.8 (1455.3) -> 1387.8 (1455.3) MB, 2.3 / 0 ms (+ 0.9 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
243032 ms: Mark-sweep 1387.8 (1455.3) -> 1381.8 (1455.3) MB, 1163.5 / 0 ms (+ 2.6 ms in 2 steps since start of marking, biggest step 1.6 ms) [last resort gc].
244140 ms: Mark-sweep 1381.8 (1455.3) -> 1387.1 (1455.3) MB, 1108.0 / 0 ms [last resort gc].

<— JS stacktrace —>

==== JS stack trace =========================================

Security context: 0x250501237399
1: charCodeAt(aka charCodeAt) [native string.js:~53] [pc=0x1a3314807846] (this=0x3e999d4f7cc1 <String[125]: return _this; // 19\n>,q=0)
2: /* anonymous /(aka / anonymous */) [/root/.meteor/packages/meteor-tool/.1.5.2_1.ogej5c++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x8…

/opt/build_scripts/build-meteor.sh: line 30: 5162 Aborted meteor build --directory $APP_BUNDLE_DIR
The command ‘/bin/sh -c cd $APP_SOURCE_DIR && $BUILD_SCRIPTS_DIR/install-deps.sh && $BUILD_SCRIPTS_DIR/install-node.sh && $BUILD_SCRIPTS_DIR/install-phantom.sh && $BUILD_SCRIPTS_DIR/install-mongo.sh && $BUILD_SCRIPTS_DIR/install-meteor.sh && $BUILD_SCRIPTS_DIR/build-meteor.sh && $BUILD_SCRIPTS_DIR/post-build-cleanup.sh’ returned a non-zero code: 134

Error: Docker build failed. Exiting.


#17

I’m getting the exact same error - bump and please help us solve this!


#18

I was hoping building with a huge amount of space set for build would help:

reaction build --build-arg TOOL_NODE_FLAGS="--max-old-space-size=9096"  mycustom

gleaned from this post which led me to this gitter.im I bumped up the memory considerably as I have plenty of RAM in this laptop, but yet I am still running into this issue. But these flags don’t appear to affect the results.

Using strace -c -f -o /tmp/strace.out reaction build mycustom shows that it is just waiting around on something, here’s a few examples of when it dies:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 92.03    8.682352         485     17894      1507 futex
  3.48    0.328133         165      1984           pselect6
  3.30    0.311139          73      4240         9 epoll_wait
------ ----------- ----------- --------- --------- ----------------
100.00    9.434512                102542      9638 total
1 marketplace % cat /tmp/xstrace.out 
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 86.36   15.520420         881     17620      1595 futex
  9.79    1.760298       11894       148        56 wait4
  2.18    0.391664         161      2433           pselect6

#19

Yeah, I’ve actually found that more isn’t necessarily better with this issue. For example, --max-old-space-size=4096 has worked better for me on some Linux machines, but CircleCI will time out and fail with anything higher than --max-old-space-size=2048. So I guess you need to experiment a little.


#20

@jeremy thanks for the reply! I’m finding this issue to be endemic to high RAM machines and you may have provided an answer. Perhaps I need to go the other way.

edit: adding link to this issue