How to gracefully handle OOM #3582
Comments
It depends ;-) Many methods have a hard requirement and there is no way to perform the calculation with less memory. The plane wave grids are a good example of that. However, there are a few cases where we can trade memory for compute time, e.g. caching of ERIs or COSMA. That being said, for the time being I'm afraid you'll have to use try-and-error to find the right settings. |
|
When I enable use ulimit within the container, it seems to be OK. As a side note, I've observed that when using the CP2K docker recipes and converting them to |
Is it possible to deal with OOM situations gracefully?
I'm dealing with input structures which are somewhat unphysical in terms of box size and or particle positions. It's hard to filter this out beforehand. In any case, when he tries to construct the PW basis, he runs OOM because probably the number of plane waves is exceptionally high because the box is unnecessarily large.
The problem is that it triggers an OOM of the kind that it kills everything else that was running on the node. Ideally, I'd use e.g
ulimitto handle this more gracefully. When I useulimit -vand set it to 2GB (roughly the available memory per core, those errors are in fact successfully caught by the OS and it no longer makes everything crash. However, sometimes, it also triggers OOM errors in cases where he's clearly not running out of memory. Presumably, the ulimit kicks in a little to early, i.e. before spawning all the separate processes, since in those cases there hasn't been any CP2K output whatsoever.The text was updated successfully, but these errors were encountered: