This is the secret Racoon error message decoder ring.

May it help google help others struggling with racoon's "helpful" error
messages.


unknown notify message, no phase2 handle found

  racoon got an error message from the peer but was too dumb to spell it
  out.  Try running racoon with -d.  Usually, the peer was unable to
  verify the certificate.


no suitable policy found

  you did not set a policy for this IP with setkey.
  Check /etc/ipsec.conf and run it through setkey -f again.


no policy found

  same as above?  Maybe the difference is that you didn't set a policy
  at all?


trns_id mismatched: my:12 peer:3

  I got this when I tried a racoon key exchange over a NATted
  connection.  I think it means that the outer IP (the IP of the NAT
  gateway) and the inner IP (192.168.*) don't match the same policy rule
  in /etc/ipsec.conf.


invalid msg length.

  I have no idea what causes this.  It didn't appear to have any
  negative impact though.


failed to get proposal for responder

  I got this when one side was configured for blowfish and the other was
  configured for 3des in racoon.conf.


unable to get local issuer certificate

  This is an openssl error.  I got this when the certificate was
  self-signed instead of signed by a CA the peer recognizes.

  If it is signed by a CA, openssl was unable to verify.  Copy the
  CA.crt file to /etc/cert (or wherever you told racoon to look for
  certificates) and run

    ln -s CA.crt `openssl x509 -noout -hash -in CA.crt`.0


notify message must be encrypted

  This happened to me because one side proposed aes and the other 3des.


failed to get subjectAltName

  You forgot to set "my_identifier asn1dn;" in the remote section.


failed to get my CERT

  The path or filename is wrong in racoon.conf.  Use

    strace -eopen racoon -F

  to see which files racoon tries to open.  Here is an excerpt from my
  racoon.conf:

    path certificate "/etc/cert" ;
    certificate_type x509 "knuth.crt" "IPsec Server Key.pem";

  This will look for /etc/cert/knuth.crt.



(Thanks to Kurt Jaeger for this one:)

ERROR: ignore information because the message is too short

  This can happen if the system time on the client is wrong.  For Kurt
  it happened because the client was an Android phone and had its clock
  reset to 1980, but the cert was from 2011.

[Update 2014-11-22: This can also happen if the system time is correct
but the client certificate is expired, says Vadim]





I recommend using xca for the certificate stuff.  Using plain openssl is
very time consuming and in the end not worth your while.  The URL is

  http://www.hohnstaedt.de/xca.html

See also:

  http://www.ipsec-howto.org/x210.html

  http://lartc.org/howto/lartc.ipsec.html



And now a little goodie for poor Cisco users (thanks to Andreas Bogk
for this):

  %CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 145.254.62.73   failed its sanity check or is malformed

    This is Cisco's catch-all error message.
    If you increase the log level, you will get more detail, but it will
    be in the debug buffer, not in the syslog messages.
    The most probably reason for this error message is that the
    certificate is expired.

    The magic incancation is:

      conf term
      logging buffered 8192 debugging
      ^Z
      debug crypto isakmp error

    then try isakmp and look in the log with

      show log


  invalid PEM boundary

    You foolishly assumed that, since the public key is also part of the
    private key, it would be sufficient to just give IOS the private
    key.  IOS actually wants something like "BEGIN PUBLIC KEY"... "END
    PUBLIC KEY"... "BEGIN RSA PRIVATE KEY"... "END RSA PRIVATE KEY" as
    sequence.  Also, IOS will only accept keys with a PEM passphrase.
    It will ask for the passphrase on import and then save the key
    without passphrase internally.


  If you find that you cannot import PEM keys, your IOS may not have PEM
  support.  As ridiculous as this sounds, IOS 12.3(8)T has PEM support,
  but IOS 12.3(9)T does not.  Another IOS command you should know is
  "term mon", Andreas finally adds.  It makes debug output show up on
  your terminal (if it's not the console anyway).


And now, finally, something for the Windows people.  If your racoon
says:

  INFO: begin Identity Protection mode.
  INFO: received Vendor ID: MS NT5 ISAKMPOAKLEY
  ERROR: ignore information because ISAKMP-SA has not been established yet.

then Windows probably could not find a computer certificate.  If you
foolishly thought double clicking on the .p12 file would put make
certificate known to Windows, think again.  You need to do this:

  Start -> Run -> MMC
  File -> Add/Remove Snap-In
  Choose the Certificate Snap-in
  Add -> Service Account -> Local computer -> IPSEC Services -> Close, OK
  Right click on one of the certificate stores in the tree view on the
  left
  Import -> Next -> choose your key file and let Windows choose

To debug IKE on Windows, start regedit, move to

  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent

and create a new key, "Oakley".  Then, inside Oakley, create a new
DWORD, "EnableLogging".  Set this to 1.  Windows will then log Oakley
debug messages to \WINNT\Debug\Oakley.log.  Now that's what I call
obvious!