This is not a memory problem, but an operating system resource problem. You are running out of native threads, i.e. how many threads the operating system will allow your JVM to use.
This is an uncommon problem, because you rarely need that many. Do you have a lot of unconditional thread spawning where the threads should but doesn't finish?
You might consider rewriting into using Callable/Runnables under the control of an Executor if at all possible. There are plenty of standard executors with various behavior which your code can easily control.
(There are many reasons why the number of threads is limited, but they vary from operating system to operating system)