There is a limit upto which you can increase the JBoss size. Even of you specify 8-10 GB for Jboss, it doesn't use that much.
How much memory will be used by Jboss depends on your QC server as well.
When I encountered this issue (when we upgraded to QC 11.0) multiple times, we came to know that Jboss only utilizes maximum 1 GB in case of Windiws & around 4 GB in case of Unix/ Linux Server. It doesn't matter how much you allocate. Max only this can be used per the QC design.
You can open a support ticket with HP & they will help you in resolving this issue. Further this issue was more in earlier patches of QC. If you upgrade to patch 10 or higher (patch 16 latest), this issue will be curbed to a great extent & performance of QC will be improved too.
You can also turn QC sense off, if you are not using it. It also saves some memory which can be used elsewhere.
Please note that it is all my personal experience by debugging my organization system. If you have any doubt, consult HP.