[Build 117] Project stopped - Exceeded memory limit

It looks that you do not have enough memory to run SQX with the specific settings.

Please attach your SQX project.

Don't you have more RAM on your computer? If you do, configure manually at least 4-6 GB of memory for SQ, Java sometimes doesn't determine it correctly for small RAMs. 

But even with 4 GB I'm afraid you'll have problems running any more advanced features of SQ. 8 GB RAM is optimal minimum.

You can also try runing Random generation instead of Genetic evolution, it uses memory in a different way.

how much SQX can maximize RAM usage? are there any restrictions?

suppose my computer has 512GB of RAM and 8xXeon X7560 is all the memory and time will be spent?

SQ doesn't need that much memory, it needs only a certain level (usually 8 - 16 GB should be enough, depending on your setting) to work. 

Normally, if you allocate from 8  - 64 GB memory for SQ you shouldn't have any problems.

If you have computer as powerful as you described it, we can test SQ on it and possibly make some tuning ot make it work optimally there.

Java has really NO memory limitation, Java 10, 11 and 12 in their Server versions can address MULTI-TERRABYTES(!) of RAM. The newest GC in Java is optimized for especially such scenarios. Your 512GB can be used without issues. That does not mean that SQ WILL use it though, it depends on what you build and  how much memory that needs. I am building on 32 years of H1 data, genetic evolution with 512 population size, 5 islands and ~25 GB of RAM are doing fine for that really. So 512 GB would be a serious overkill if you ask me ;-)