There are three kernel parameters that control this.

The maximum user-stack segment size for a process is specified via the
'maxssiz' kernel parameter.  The user-stack segment is where dynamically
allocated (ie - malloc'ed) data is stored.

The maximum shared-text segment size for a process is specified via the
'maxtsiz' kernel parameter.  The shared-text segment is where executable
code is stored.

The maximum data segment size for a process is specified via the 'maxdsiz'
kernel parameter.  The data segment is where statically allocated data
(ie - arrays) is stored.

The max[std]siz parameters are fences, not consumable resources.
Therefore, you can set them to large values with no impact on the kernel.
Their sole purpose is to prevent poorly written programs from using
massive amounts of memory.

For the values of both maxssiz maxdsiz, use integers that are multiples
of 4096 bytes, the system pagesize.

More information on maxdsiz, maxssiz, and other process-related kernel
parameters is provided online (see section 5.5.3 for more info).

Another potential limiter is swap space.  The combined virtual data space
of all running processes can't exceed swap size.  Run /etc/swapinfo -t and
look at the "total" line.  That's what you have left.