The message comes from the OS, from the dread OOM-killer, the "out of memory killer".
Years ago, operating systems took their memory allocation responsibilities seriously. When an application allocated memory, the OS allocated the memory. If there was not enough memory to satisfy the request, it was denied. Some still work that way, but Linux usually does not.
Linux is usually configured by default to treat memory allocation the way airlines treat reservations. It "honors" more allocations than it can actually supply, on the (correct) theory that not every allocation will actually be used. That works great until it doesn't.
On an airplane, overbooked passengers aren't pushed off the plane in mid-air. On Linux, that's exactly what happens: when memory is actually, really exhausted, the Grim Reaper, the OOM-killer, picks a victim. In your case, the victim is Matlab. Lucky you. Instead of a free ticket, you get a message.
To defeat the OOM-killer, change the overcommit_memory kernel setting to 2.
That solution is unfortunately a rear-guard action. The influence of Linux has led many applications to rely on overcommitted memory, and be profligate in their memory allocations. When it's not available, those applications use a lot of memory. On the other hand, it has the meritorious property of actually running programs that do start, and not killing running processes arbitrarily chosen as OOM targets.
To run Matlab reliably for the problem you have on the machine you have, you might need more RAM. As a stopgap, you could increase swap space, which increases virtual memory, which is what you need.
