You'll need to write a wrapper class that inherits from the XmlObjectSerializer. (In case you're wondering why you're still working with xml ... WCF uses Xml Infoset as its internal data model,
(Inside the ReadObject and WriteObjectContent, you use the JsonSerializer.)
Then, write an custom operation behavior that derived from DataContractSerializerOperationBehavior; Override the CreateSerializer method to return the wrapper class.
Next, write an endpoint behavior (IEndpointBehavior) that replaces the default binding's DataContractSerializerOperationBehavior with the custom operation behavior.
Finally, you can write either (or both) a custom operation behavior (IOperationBehavior) or a custom behavior extension element (BehaviorExtensionElement) so that you can configure the usage of the JsonSerializer, in code or in the configuration, respectively.
Check Marc Gravell's protocol buffers serialization implementation of the XmlObjectSerializer.