The error message which was coming 'Unable to copy the '/etc/oratab' to '/tmp/oratab.
We certainly knew that the issue is either with the permission or copying the said file and our immediate step was checking the permisson and ownership on the oratab file. Everything was just fine and then we try to create a file in the /tmp directory using touch command from the Oracle user and it worked without any hassles.
We were able to create the RAC database successfully on a single node and when tried to add instance on the other node, the same issue encountered again and again.
We finally thought of running the dbca in tracing mode and when the same problem arises and when we had a look at the trace file, we couldn't stop wondering about the silly/stupid thing which Oracle was complaining.
The trace file reveals that it is looking for 'scp' executable in '/usr/local/bin' location, whereas, the 'scp' excutable was in '/usr/bin' directory. We simply created a link for 'scp' in the '/usr/local/bin' and dbca run smoothly and we were able to complete the RAC database creation.
Past three months, its been really a great working expeirence on RAC and I must say, we have faced hell a lot of issues while trying to implement RAC on HP Virtual Machines. Definately, its a great learning phase.
Guess what's next? MOVING 8 DEVELOPMENT DATABASE TO RAC from AIX to HP. Umm.. a good and taugh task ahead.
I will keep posting my experience and the issues that have faced while setting up our 16 node RAC with 8 databases.