Component/s: Core Spec
For encryption, the core specification currently has a PMode PMode.Security.X509.Encryption.Algorithm which identifies "the encryption algorithm to be used" based on W3C XML Encryption algorithm identifiers.
XML Encryption actually distinguishes two algorithms:
xenc:EncryptedData / xenc:EncryptionMethod / @Algorithm
xenc:EncryptedKey / xenc:EncryptionMethod / @Algorithm
The value is an identifier of an algorithm used for Key Transport. XML encryption currently recommends http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p, commonly used values include http://www.w3.org/2001/04/xmlenc#rsa-1_5.
I assume the PMode parameter identifies the first use. There does not seem to be a parameter for the second one? When using WS-SecurityPolicy, it would be needed to select the correct policy, e.g. Basic128Sha256 versus Basic128Sha256Rsa15.
(This is not to promote WS-SecurityPolicy, the ebMS3 approach of directly using the W3C Signature and Encryption parameters is actually more future-proof than WS-SecurityPolicy's identifiers, just to note that some implementers of ebMS3 will use security toolkits that are configured using WS-SecurityPolicy).