Problem to run some cells

Module 6. Ensemble of models Ensemble based on boosting
#1

Hello
Thanks for this mooc wich is very great.
I’ve a problem running somme cells : I get the following error even after having rebooted my PC :
OSError: [Errno 12] Cannot allocate memory

I’ve this problem with the correction of the exercice M6.03 :

train_scores, test_scores = validation_curve(
    adaboost, data_train, target_train,
    param_name="n_estimators", param_range=param_range,
    scoring="neg_mean_absolute_error", n_jobs=-1)
And in the lesson about GBDT :
gradient_boosting = GradientBoostingRegressor(n_estimators=200)
cv_results_gbdt = cross_validate(
    gradient_boosting, data, target, scoring="neg_mean_absolute_error",
    n_jobs=-1,
)

What can I do ?
Best regards and anticipated thanks

Sophie

#2

This is typically when you don’t have enough RAM available. Are you running the code online or locally?

#3

Thank you for your quick answer.
I am not sure whether I run it locally or online.
What I do is opening the module on the site, and then typing the strokes Ctrl+Ent directly on the “cell” which contains the code to execute.
If there is another way to launch the run and ensure it happens on the remote platform rather than on my PC, please tell me.
I am all the more disturbed by this error that it takes place immediately after my command. If it were a matter of RAM disponibility, would it not happen a little later? And when I reduce the number of estimators, I have the same error although it should be less greedy in memory.
Regards,
Sophie

#4

Hi,
I found it : it’s the parameter : n_jobs=2 or n_jobs=-1 wich generates the error. If I don’t give this parameter, it’s working.
Can you explain me why ?
Thanks
Sophie

#5

Yes so this is on our server where we limit the RAM by user however it is too much limited. @brospars @lesteve @ogrisel we should probably look at this.

So why it is working with n_jobs=2 and n_jobs=-1 is technical but I will try to explain. Internally, scikit-learn will parallelize the computation for the validation curve. Indeed, you can train N problems with the N different parameters required to build the curve. Ideally, you could run the N problems on N processor cores, if N processor cores are available. In practice, the server is limited to 4 cores for a user. n_jobs=-1 will use the 4 cores in parallel, n_jobs=2 will use only 2 cores. So up to now, it would look like that setting n_jobs will just make the execution faster. However, there is a cost regarding memory. Indeed, the potentially N operations will be run in an isolated process, and each process will need the original data to fit/score the model. So you need to copy N times the original data. In short and to simplify, n_jobs=2 will run twice faster and take twice memory.

In practice, it is a bit more complex: you will not get a x2 acceleration but we will probably not allocate x2 memory, but it is a good approximation to understand what is going on.

#6

Thanks very much for answering so clearly and so fast.
I seize the opportunity to congratulate you on this whole MOOC whose quality is really great, including the involvment of the teaching team.

1 Like
#7

Note, when this happens, it should be possible to free some memory by clicking on the “File” menu of jupyter and then clicking “Running notebooks” and then clicking “shutdown” for the notebooks you no longer need.