JsonMediaTypeFormatter has a public property to access the SerializerSettings, you can set ReferenceResolver to your custom class.
When the media type formatter creates the JsonSerializer instances, it simply passes the serializer settings to the json serializer. Json.NET will use the assigned ReferenceResolver and add items to it. This has two side effectes:
The resolver has to be thrad-safe and might mix references from different thread which use the media type formatter at the same time.
When serialization finishes, the resolver is not notified, so it cannot clear its internal lists. References stay in the dictionary forever and the GC will never free the memory.
This all only happens when you assign your own ReferenceResolver instance. When you leave it as default (null), a new resolver is instantiated for each serialization/deserialization. But in current implementation, ReferenceResolver is nearby useless.