I have my permanent account number with me but I have misplaced my card. I have applied for passport so does this number is of any use to me or should I have to apply for the PAN CARD all over again?

First and foremost, PAN Card is not a required document for applying for passport. Your PAN Number remain same and remain tagged to you. It doesn't matter if you have or not have PAN Card with you. By law, one person cannot have two PAN number. If you have lost your PAN Card, you will have to submit a 'Change Request Form' requesting for reissuing the PAN Card.  You can get the PAN Card change request form the official website of NSDL PAN Service. The form provides you the list of documents you will have to submit along with it. You will have to pay a small fee and it takes 15-20 days to get the card. It is highly recommended to get the PAN Card reissued as soon as possible as there days most financial transitions require it as mandatory original document.