Algorithm Choices Matter More Than You Think

Every AS2 message involves two cryptographic operations: encryption (so only your partner can read it) and signing (so your partner can prove you sent it). Each operation requires you to pick an algorithm. Pick wrong, and the message fails silently, the partner rejects it, or worse, you're running production traffic on algorithms that NIST deprecated years ago.

This guide covers every algorithm you'll encounter in AS2 configurations, which ones to use, which to retire, and how to handle the inevitable mismatch errors when your settings don't match your partner's.

Encryption Algorithms

Encryption protects the message content. You encrypt with your partner's public certificate. They decrypt with their private key. The algorithm determines how the symmetric content-encryption key is generated and applied.

3DES (Triple DES / DESede)

OID: 1.2.840.113549.3.7

Key size: 168 bits (effective security: 112 bits)

Status: Deprecated. NIST disallowed 3DES for new applications after 2023.

3DES was the default encryption algorithm in AS2 for over a decade. If your AS2 configuration was set up before 2018, you're probably running 3DES. It works, but it's slow and provides weaker security than AES. The 64-bit block size makes it vulnerable to birthday attacks (Sweet32) on large message volumes.

Should you use it? No, unless a trading partner absolutely cannot support AES. If your partner insists on 3DES, document this as a security exception and push them to upgrade. Many compliance frameworks (PCI DSS, HIPAA) now flag 3DES usage.

AES-128 (AES with 128-bit key)

OID: 2.16.840.1.101.3.4.1.2 (CBC mode)

Key size: 128 bits

Status: Approved. NIST recommended through at least 2030.

AES-128 is the minimum recommended encryption for AS2 today. It's significantly faster than 3DES (hardware acceleration via AES-NI is available on every modern CPU) and provides strong security. For most AS2 use cases, AES-128 is perfectly adequate.

Should you use it? Yes. This is a solid default choice. If a partner supports AES but doesn't specify which key size, AES-128 is safe and widely compatible.

AES-192 (AES with 192-bit key)

OID: 2.16.840.1.101.3.4.1.22 (CBC mode)

Key size: 192 bits

Status: Approved. Rarely used in practice.

AES-192 exists but is uncommon in AS2 deployments. It offers slightly more security margin than AES-128 with minimal performance difference. However, most organizations jump straight from AES-128 to AES-256 if they need stronger encryption.

Should you use it? Only if a trading partner specifically requires it. Otherwise, choose AES-128 or AES-256.

AES-256 (AES with 256-bit key)

OID: 2.16.840.1.101.3.4.1.42 (CBC mode)

Key size: 256 bits

Status: Approved. Highest standard AES key size. Recommended for sensitive data.

AES-256 is what to choose when compliance requirements call for the strongest available encryption, or when your trading partner requires it. Walmart, for example, requires AES-256 for AS2 connections. The performance difference from AES-128 is negligible on modern hardware.

Should you use it? Yes, especially for pharmaceutical supply chain (DSCSA), financial data, or any partner that requires it. If you're setting up a new AS2 connection today and have no specific guidance, AES-256 is the safest bet.

Encryption Algorithm Comparison

Quick reference for decision-making:

3DES: 112-bit effective security, 64-bit block size, slow, deprecated by NIST. Avoid.
AES-128: 128-bit security, 128-bit block size, fast, NIST approved. Good default.
AES-192: 192-bit security, 128-bit block size, fast, NIST approved. Rarely needed.
AES-256: 256-bit security, 128-bit block size, fast, NIST approved. Best for sensitive data.

Signing Algorithms

Signing proves message authenticity and integrity. You sign with your private key. The partner verifies with your public certificate. The signing algorithm is a hash function applied to the message content before the cryptographic signature is generated.

SHA-1

Digest size: 160 bits

Status: Deprecated for signing since 2011. Collision attacks demonstrated in 2017 (SHAttered).

SHA-1 is broken for collision resistance. Google and CWI Amsterdam demonstrated a practical collision attack in 2017. While preimage attacks (forging a specific message) are still computationally expensive, using SHA-1 for signing new messages is indefensible from a security standpoint.

Should you use it? No. If a trading partner requires SHA-1, escalate to their security team. Many legacy AS2 systems still default to SHA-1, and you may encounter it during partner onboarding. Push for SHA-256 at minimum.

Exception: Some very old AS2 software (Cyclone Interchange, older Gentran builds) only supports SHA-1. If you're forced to use it, document the risk and push the partner to upgrade their AS2 platform.

SHA-256

Digest size: 256 bits

Status: Approved. Current industry standard for AS2 signing.

SHA-256 is the safe default. It's supported by every AS2 implementation built in the last 10 years. It provides strong collision resistance with no known practical attacks. NIST approves it through at least 2030.

Should you use it? Yes. This is the right choice for almost every AS2 deployment. If a partner doesn't specify a signing algorithm preference, use SHA-256.

SHA-384

Digest size: 384 bits

Status: Approved. Used primarily with ECDSA and 384-bit elliptic curves.

SHA-384 is a truncated version of SHA-512. It's most commonly paired with P-384 elliptic curve keys. In RSA-based AS2 signing (which is the vast majority of deployments), SHA-384 provides more security margin than SHA-256 with slightly larger signatures.

Should you use it? Only if your partner specifically requires it or you're using ECC certificates (uncommon in AS2).

SHA-512

Digest size: 512 bits

Status: Approved. Maximum security margin in the SHA-2 family.

SHA-512 is the strongest hash in the SHA-2 family. On 64-bit systems, it's actually faster than SHA-256 due to how the algorithm operates on 64-bit words. The larger digest size means slightly larger signatures, but this is negligible for AS2 message sizes.

Should you use it? If your partner supports it and you want maximum security margin, sure. But SHA-256 is sufficient for all current threat models. SHA-512 is more of a future-proofing choice.

Signing Algorithm Comparison

SHA-1: 160-bit digest, broken collision resistance, deprecated. Do not use.
SHA-256: 256-bit digest, no known attacks, NIST approved. Industry standard.
SHA-384: 384-bit digest, no known attacks, NIST approved. Niche use (ECC keys).
SHA-512: 512-bit digest, no known attacks, NIST approved. Maximum margin.

What Major Trading Partners Require

When connecting to large retailers and supply chain partners, they dictate the algorithms. Here's what several major partners specify:

Walmart

Encryption: AES-256 (CBC mode)
Signing: SHA-256
Certificate: RSA 2048-bit minimum
MDN signing: SHA-256

Amazon

Encryption: AES-128 or AES-256
Signing: SHA-256
MDN: Synchronous, signed

Common Pharmaceutical Supply Chain (DSCSA)

Encryption: AES-128 minimum, AES-256 preferred
Signing: SHA-256 minimum
Certificate: RSA 2048-bit minimum, 4096-bit recommended
TLS: 1.2 minimum

When in doubt, ask your partner for their AS2 configuration sheet. Most large trading partners have a standard document that specifies every setting, including algorithms, AS2 IDs, certificate requirements, and endpoint URLs.

How to Check Your Current Algorithm Settings

If you inherited an AS2 configuration and need to find out what algorithms are currently in use, here's where to look in common AS2 software.

OpenAS2

Check the partnership definition in config/partnerships.xml:

<partnership name="YourCompany-PartnerName">
  <!-- Encryption algorithm -->
  <attribute name="encrypt" value="aes256-cbc"/>
  
  <!-- Signing algorithm -->
  <attribute name="sign" value="sha256"/>
  
  <!-- MDN signing algorithm -->
  <attribute name="mdn_options" 
    value="signed-receipt-protocol=optional, pkcs7-signature; signed-receipt-micalg=optional, sha-256"/>
</partnership>

Mendelson AS2

Open the partner configuration dialog. The "Security" tab shows encryption and signature algorithms as dropdown menus. Common values: "AES 256", "SHA-256".

IBM Sterling Integrator

Navigate to Trading Partner > AS2 > Profile. The encryption algorithm and signing algorithm are in the partner profile settings. Sterling uses labels like "3DES", "AES128", "AES256" for encryption and "SHA1", "SHA256", "SHA384", "SHA512" for signing.

Algorithm Mismatch Errors and How to Fix Them

Algorithm mismatches are the most common source of AS2 encryption and signing failures. The message is technically valid, but the receiver can't process it because they're configured for a different algorithm than what you sent.

"Decryption failed" or "Cannot decrypt message"

You encrypted with AES-256. The partner's system is configured to expect 3DES (or vice versa). The partner's AS2 software tries to decrypt with the wrong algorithm and fails.

Fix: Confirm the encryption algorithm with your partner. Set yours to match theirs. In the outgoing message, the encryption algorithm is determined by your configuration, not negotiated. You must set it correctly.

"Signature verification failed" (algorithm-related)

You signed with SHA-256. The partner's system tries to verify with SHA-1 (or vice versa). The verification fails because the hash doesn't match.

Note: Many modern AS2 implementations auto-detect the signing algorithm from the message's MIME headers. This error is more common with older software that has a fixed expected algorithm.

Fix: Check the partner's AS2 software version. If it auto-detects, the issue may be elsewhere (wrong certificate, not algorithm). If it doesn't auto-detect, configure your signing algorithm to match what they expect.

"Unsupported algorithm" or "Unknown OID"

You configured AES-256. The partner's system is so old it doesn't support AES at all. It only handles 3DES. When it sees the AES OID in the encrypted message, it doesn't know what to do.

Fix: This is a compatibility problem. Your options:

Fall back to 3DES for this partner (document the security exception)
Ask the partner to upgrade their AS2 software to a version that supports AES
If you're under compliance requirements that mandate AES, the partner must upgrade. There's no workaround.

"MIC algorithm mismatch" in MDN

The MDN contains a MIC (Message Integrity Check) computed with a specific hash algorithm. If you requested SHA-256 but the partner computed the MIC with SHA-1, your system reports a MIC verification failure even though the message was processed correctly.

Fix: Check the signed-receipt-micalg value in your outgoing Disposition-Notification-Options header:

Disposition-Notification-Options: 
  signed-receipt-protocol=optional, pkcs7-signature; 
  signed-receipt-micalg=optional, sha-256

The keyword "optional" means you're requesting SHA-256 but the partner can use something else. Change "optional" to "required" if your system should reject MDNs that don't use SHA-256. But be aware: some AS2 implementations ignore this header entirely and always use their default algorithm.

Migration Plan: Moving from 3DES/SHA-1 to AES/SHA-256

If you're running legacy algorithms and need to upgrade, here's a practical migration approach:

Audit your current settings. List every trading partner, their encryption algorithm, and their signing algorithm. Export this from your AS2 software's partner list.
Identify partners still on legacy algorithms. Any partner using 3DES or SHA-1 needs to be contacted.
Contact partners individually. Send each partner a notice: "We are upgrading our AS2 security settings to AES-256 encryption and SHA-256 signing by [date]. Please confirm your system supports these algorithms, or let us know your preferred AES variant."
Test with each partner. After both sides update their configurations, send test messages and verify the full round-trip: message delivery, decryption, signature verification, and MDN.
Update in pairs. Change both the encryption and signing algorithm at the same time for each partner. Changing one but not the other leads to partial failures that are confusing to debug.
Keep records. Document which partners have been migrated and which are still pending. Some partners will take months to respond.

Certificate Key Size and Algorithm Interaction

Your certificate's key size interacts with your algorithm choices. A few guidelines:

RSA 1024-bit: Deprecated. NIST disallowed after 2013. If your AS2 certificate is 1024-bit, replace it immediately regardless of which algorithms you use.
RSA 2048-bit: Current minimum standard. Pairs well with AES-128/AES-256 and SHA-256. Adequate through 2030.
RSA 4096-bit: Extra margin. Recommended for DSCSA and pharmaceutical supply chain. Slightly slower for signing/verification but negligible in AS2 volumes.
ECDSA (P-256/P-384): Smaller key sizes with equivalent security. Faster operations. But AS2 support for ECC is inconsistent. Many AS2 platforms (including OpenAS2 and older IBM Sterling) only support RSA. Don't use ECC unless you've confirmed your partner supports it.

Check your certificate's key type and size:

openssl x509 -in your_as2_cert.pem -noout -text | grep -A 2 "Public Key"

The Recommended Configuration for New AS2 Connections

If you're setting up a new AS2 partnership today with no specific partner requirements, use these settings:

Encryption: AES-256-CBC
Signing: SHA-256
MDN signing: SHA-256
Certificate: RSA 2048-bit (4096-bit for pharma/DSCSA)
TLS transport: TLS 1.2 minimum, TLS 1.3 preferred
Certificate validity: 1 to 2 years maximum

These settings are compatible with every modern AS2 implementation and meet current NIST guidelines. They'll pass compliance audits for PCI DSS, HIPAA, and DSCSA without exception requests.

Validate Your Algorithm Configuration

Algorithm mismatches are configuration problems, and configuration problems are testable. Don't wait until production traffic fails to discover that your partner expects AES-256 and you're sending 3DES.

AS2 Certify validates your encryption and signing algorithm configuration as part of every connection test. It checks that both sides agree on algorithms, that the algorithms meet current security standards, and that the certificate key size is appropriate for the chosen algorithms. If you're running deprecated algorithms, the test report flags it with specific upgrade recommendations. One test catches the mismatches that would otherwise take days of back-and-forth emails to diagnose.