The MDN disposition reads automatic-action/MDN-sent-automatically; processed/error: decryption-failed. Or your own inbound log shows a stack trace ending in org.bouncycastle.cms.CMSException: key invalid in message. Either way, the same story: an encrypted AS2 message arrived somewhere and the receiving side could not open it.

Decryption failures are certificate problems wearing a disguise about 80% of the time. The other 20% split between algorithm mismatches and mangled payloads. Here are the five causes, how to confirm each, and the fix.

One orientation note first. Direction matters. If the error is in your logs, your partner encrypted with the wrong key or algorithm for you. If the error came back in an MDN, you encrypted wrong for them. Every cause below exists in both directions.

Cause 1: The sender encrypted with an old or wrong certificate

This is the number one cause, and it spikes around certificate rollovers. You renewed your cert, sent the new public key to your partner, and they loaded it into their system but never activated it, or activated it a day late, or loaded it for the wrong partnership. Every message they encrypt still uses the old public key. Your server, now holding only the new private key, cannot decrypt a thing.

How to confirm: Compare the timestamp of the first failure against your certificate renewal date. A match is close to proof. For certainty, dump the recipient info from a failed message with openssl cms -cmsout -print -inform DER -in message.p7m and compare the issuer and serial number against your current certificate's serial from openssl x509 -in current.cer -noout -serial. Different serial means they encrypted for a cert you no longer use.

The fix: Resend your current public certificate, get written confirmation of the exact activation timestamp, and keep the old private key installed until every partner has switched. The AS2 certificates guide covers a rollover sequence that avoids this window entirely.

Cause 2: Your certificate and private key do not match

Someone imported a renewed certificate into the keystore but kept the old private key, or restored a keystore from backup, or fat fingered an alias in OpenAS2's as2_certs.p12. The public cert your partner encrypts against and the private key your server decrypts with are from different key pairs.

How to confirm: Compare modulus hashes: openssl x509 -noout -modulus -in cert.pem | openssl md5 versus openssl rsa -noout -modulus -in key.pem | openssl md5. Different output means mismatch. In a Java keystore, check that the alias your AS2 server references holds a PrivateKeyEntry, not a trustedCertEntry, with keytool -list -keystore as2_certs.p12.

The fix: Reimport the certificate and private key together as a single PKCS#12 bundle so they cannot drift apart. Then run your cert through the free AS2 certificate checker to verify what you are actually presenting.

Cause 3: Algorithm mismatch, usually 3DES versus AES

The sender encrypts with an algorithm the receiver refuses or does not support. Legacy systems default to 3DES; hardened modern gateways reject it. Occasionally the mismatch is subtler: RSA-OAEP key wrapping against a receiver that only handles PKCS#1 v1.5.

How to confirm: The receiving side's log usually names the algorithm it could not handle, with strings like no such algorithm or an OID it does not recognize. The same openssl cms -cmsout -print dump from cause 1 shows the content encryption algorithm OID in the envelope. 1.2.840.113549.3.7 is 3DES; 2.16.840.1.101.3.4.1.42 is AES-256-CBC.

The fix: Agree on AES-256 in writing and set it explicitly on both sides. In OpenAS2 that is encrypt="aes256" on the partnership; in mendelson, pick the algorithm in the partner security settings. Do not leave either side on a default.

Cause 4: The PKCS#7 structure arrived corrupted

The envelope was fine when it left. Something between the two AS2 servers rewrote it. Usual offenders: a content inspecting proxy or WAF that modifies the body, a load balancer that truncates at a size limit, or a transfer path that treats the binary payload as text and converts line endings.

How to confirm: Compare the byte count sent against the byte count received; any difference is corruption. Parse errors that fire at the very start of the structure, like malformed content or ASN.1 sequence errors at offset 0, point to wholesale mangling rather than a key problem. Intermittent failures that correlate with message size point to truncation.

The fix: Exempt the AS2 endpoint from body inspection on the proxy or WAF, raise any request size limits above your largest payload, and confirm nothing on the path applies content transformation. Then resend the same file that failed.

Cause 5: The certificate cannot be used for encryption at all

X.509 certificates declare what they may be used for. A certificate minted for signing only, with keyUsage limited to digitalSignature, gets rejected by strict implementations when used for key encipherment. This bites teams that generate separate signing and encryption certs, then hand the partner the wrong one.

How to confirm: Run openssl x509 -in cert.pem -noout -text and read the X509v3 Key Usage extension. Encryption needs Key Encipherment present.

The fix: Issue a certificate with both digitalSignature and keyEncipherment, or make sure each partner maps the right cert to the right function. Most AS2 setups use a single dual purpose certificate for exactly this reason.

Find the cause in 60 seconds instead of 3 days

The painful part of decryption failures is that the sender and receiver each hold half the evidence. That is how a one line error turns into a week of "our cert is fine, check yours."

Skip the standoff. The free AS2 connection test sends a real encrypted message against an endpoint and grades the certificate, algorithm support, and decryption round trip separately, so you know which side owns the problem before the first email goes out. For the broader class of certificate failures beyond decryption, the guide to fixing AS2 certificate errors covers expiry, chains, and format mismatches.

Stop guessing. Start proving.