All major web browsers will warn the user if they encounter a self-signed certificate. This is supposedly because self-signed certificates are less secure than certificates signed by a certificate authority (CA).
Essentially, there are three different levels of “security” when connecting to a website:
This is insane, since a self-signed certificate is unilaterally better than no certificate. Suppose a bad guy was going to do some badness—i.e. something that would trigger a browser warning—using a self-signed certificate. He could just delete the self-signed certificate and go about his business. With no certificate, there's no browser warning!
So, this has been a topic of discussion numerous times:
In every discussion (linked above and elsewhere), the main criticism of self-signed certificates is that they allow man-in-the-middle (MITM) attacks. The “authentication” mentioned above (in the three levels of security) is supposed to prevent MITM attacks. In theory, having your certificate signed by a CA allows the user to verify that you are who you say you are.
And technically, self-signed certificates do allow MITM attacks. However, this argument is specious: self-signed certificates prevent MITM attacks at least as well as CA-signed certificates.
We'll make two mild assumptions about the use of self-signed certificates.
Browsers should alert the user if a site's certificate has changed. This is called “pinning” in some places, and is the way that SSH handles authentication. There is a Firefox extension, Certificate Patrol, which gives Firefox this ability.
I call this a mild assumption, since if we're going to do anything at all to fix the web browsers, this is trivial to implement.
Under these assumptions, we'll see that the MITM criticism is invalid.
With pinning, there is exactly one opportunity for a man in the middle attack to occur. When you connect to a website for the first time ever and pin the certificate, an attacker would be able to MITM the exchange and forge the certificate. This is unavoidable, unless you have some out-of-band method to verify the certificate's fingerprint.
Note that the same attacker would have to MITM all future traffic between you and the website; otherwise, you would see the real self-signed certificate at some point, and would be warned that it had changed (from the forged one you originally saw).
To be fair, current implementations would allow for one MITM opportunity every ten years (when the certificate expires). However, this is easy to fix, either by making the certificates valid indefinitely, or by signing the new certificate with your old key.
With a CA-signed certificate, there is also exactly one opportunity for a man in the middle attack to occur.
With a CA-signed certificate, you're supposed to be protected from MITM attacks. The first time you connect to a website, you can check the signature on the certificate to see if it has a valid CA signature. This should prevent someone from forging it in-transit.
But, how do you verify the CA signature? After all, it's basically just another certificate. The usual answer is “it came with my browser, so it has to be valid.”
Really? Where did your browser come from? There are a only a few answers that you'll get to this question, all equivalent:
The first answer of course means that the browser download could have been MITMed. In the second case, you can repeat the question until you get the third answer (i.e. how did you verify Mozilla's certificate before you downloaded Firefox?).
If your browser came with your PC, that's an opportunity for a MITM attack. Acer, or HP, or whoever could easily have replaced all of your certificates with phony ones. (In practice, you only need to add an additional CA, which is relatively easy.) See the Nokia link below. This attack need not be physical, as there are plenty of examples of PCs shipping with malware on them.
You may have to go back a ways, but in every case, there's a leap of faith at the beginning where a MITM was possible.
If there was any doubt that this was possible or is occurring in Real Life, Opera and Nokia have been busted performing MITM attacks on their customers.
This belies a deeper problem with the CA infrastructure: any certificate authority can sign a “valid” certificate for any website. So the government of China can create its own certificate for bankofamerica.com, and your browser will be fine with it. Of course, the Chinese government isn't the only CA that can't be trusted.
With a pinned, self-signed certificate, the only way to compromise communication is with a one-time MITM, and this affects only a single website. With CA-signed certificates, any vulnerability anywhere in the (massive) hierarchy compromises every website on the internet. All things considered, self-signed certificates with pinning are more secure than CA-signed certificates.