We initially thought there could be a memory leak which is causing the high memory consumption. Upon opening a TAR for this issue, Oracle support suggested that the memory consumption by the clusterware is pretty normal and there is no high memory consumption by any of Oracle CW processes. Shutting down the cluster in order to test the memory consumption doesn't help either as the memory consumption still was the same.
Then, we thought of investigating from OS point of view and we found out that the OS kernel parameter 'dbc_max_pct' was set to the default value, i.e. 50%. The general recommendation on most of the environments is 10%. Thank god, its a dyanmic parameter which can be modified without the shutdown. Upon resetting the value to 10%, the memory consumption drastically down from 99% to 35% on all the nodes.
I have also found a very good Metalink Note (68105.1 : Commonly Misconfigured HP-UX Kernel Parameters) which explains the most commonly misconfigured HP-UX kernal parameters. Dears, you are running any databases on HPUX, I strongly recommend you to have a look at this note and make changes accordingly.