The Capital is the province where the country's seat of the government lies. In Victoria 2 the capture of capital province increases War score significantly, and if it's under control of rebels (other than nationalist) for 30 days, they will enforce their demands.
The Capital can not be taken with acquire state or demand concession casus belli. The whole country has to be annexed to take its capital. It is not possible to change a nations capital but a few special cases. Russia and Italy have the decisions Return to the Third Rome and The Law of Guarantees respectively and The Ottoman Empire has the event Flight from Istanbul, which changes the capital.
A List of nations' capitals can be found at Country basic info.