Self-referencing loop problems

Jan 16, 2008 at 8:01 PM
I'm using Json.NET with the NHibernate ORM (object-relational mapper), and I'm running into a lot of self-referencing loop errors. The way NHibernate works, I've got a Product object defined, and a product table in my oracle db, and I've just got to call product = db.Load(typeof(Product, id)) to load the correct db record into my Product object. Usually JavaScriptConvert.SerializeObject(product) works fine, except when I'm doing an update; NHibernate will make a new subclass of my Product object, and add some kind of state-keeping properties to the subclass. It's these extra things that are causing the problem.

To try getting around this, I've bypassed JavaScriptConvert.Serialize(product) and written my own SerializeObject call that sets ReferenceLoopHandling to "Ignore":
private string MySerializeObject(Object value)
{
StringWriter sw = new StringWriter(CultureInfo.InvariantCulture);
JsonSerializer jsonSerializer = new JsonSerializer();
jsonSerializer.ReferenceLoopHandling = ReferenceLoopHandling.Ignore;
using (JsonWriter jsonWriter = new JsonWriter(sw))
{
jsonWriter.Formatting = Formatting.Indented;
jsonSerializer.Serialize(jsonWriter, value);
}
return sw.ToString();
}

However, now the error I'm getting is this:
System.InvalidOperationException: Method may only be called on a Type for which Type.IsGenericParameter is true.

So I guess my question is, what can I try next? Without completely re-writing the Json.NET source, I don't know how to figure out what is being added to my object, and I don't know how to remove the extras either.
Jan 16, 2008 at 8:05 PM
If it helps, here is the stack trace:

InvalidOperationException: Method may only be called on a Type for which Type.IsGenericParameter is true.
System.RuntimeType.get_DeclaringMethod() +2453102

TargetInvocationException: Exception has been thrown by the target of an invocation.
System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner) +0
System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner) +72
System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks) +296
System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +29
System.Reflection.RuntimePropertyInfo.GetValue(Object obj, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture) +55
System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index) +18
Newtonsoft.Json.Utilities.ReflectionUtils.GetMemberValue(MemberInfo member, Object target) +130
Newtonsoft.Json.JsonSerializer.WriteMemberInfoProperty(JsonWriter writer, Object value, MemberInfo member) +40
Newtonsoft.Json.JsonSerializer.SerializeObject(JsonWriter writer, Object value) +438
Newtonsoft.Json.JsonSerializer.SerializeValue(JsonWriter writer, Object value) +1455
Newtonsoft.Json.JsonSerializer.WriteMemberInfoProperty(JsonWriter writer, Object value, MemberInfo member) +285
Newtonsoft.Json.JsonSerializer.SerializeObject(JsonWriter writer, Object value) +438
Newtonsoft.Json.JsonSerializer.SerializeValue(JsonWriter writer, Object value) +1455
Newtonsoft.Json.JsonSerializer.WriteMemberInfoProperty(JsonWriter writer, Object value, MemberInfo member) +285
Newtonsoft.Json.JsonSerializer.SerializeObject(JsonWriter writer, Object value) +438
Newtonsoft.Json.JsonSerializer.SerializeValue(JsonWriter writer, Object value) +1455
Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value) +22
SRLoopTest.MySerializeObject(Object value) in c:\Inetpub\vhosts\kicdev\Tests\SRLoop.ashx:80
SRLoopTest.ProcessRequest(HttpContext context) in c:\Inetpub\vhosts\kicdev\Tests\SRLoop.ashx:56
System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +303
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +64
Coordinator
Feb 8, 2008 at 11:20 PM
Hi

I have searched for that error message and taken a look in reflector to see what could be going on but I couldn't find anything. To be able to effectively debug your issue I would really need more information about the class you're serializing.
Sep 6, 2008 at 9:26 AM
I have the same issue with class generated by the entity framework with references to other classes, I think it is easy to replicate.
Coordinator
Sep 7, 2008 at 8:37 AM
Great. Send me an example project of the problem.
Dec 17, 2008 at 1:42 PM
Edited Dec 17, 2008 at 2:10 PM
Any updates on this one?

This is happening consistently with the Entity Framework, since by default a pair of related tables in a db are generated with mirrored navigation properties.  This is really useful in C#, but in JavaScript, I'm generally treating one entity as a parent/container, and will not be navigating over the entity graph.

For clarification:
I'm also setting ReferenceLoopHandling to Ignore.  This is followed by: System.InvalidOperationException: Method may only be called on a Type for which Type.IsGenericParameter is true.


btw, great lib.
Coordinator
Dec 18, 2008 at 7:04 AM
If you send me a code example of it happening I will fix it. I have never used the Entity Framework myself.
Mar 17, 2009 at 5:23 AM
I'm encountering the same bug, during my very first "hello world" attempt at serializing and Entity that I instantiated from a LINQ to Entity query.  I'm trying to debug now bystepping through your code in changeset 32207.  It's a little surprising that you're not familiar with EF and already testing it in that arena.  Objects instantiated via EF will only increase in the upcoming weeks and months as developers start kicking the tires on EF.

I'll try to post an answer when I find it.  Regarding your request for an example, that's a little tough.  I'm using my own database schema and I don't know what is triggering the problem.  By the time I'd get my environment documented I could just as easily find why your code is in a loop.  I think the problem stems from EF's proclivity to create two-way pointers between objects as a default configuration.  That's my hunch but I'll soon see.
Coordinator
Mar 18, 2009 at 6:56 AM
I managed to recreate the issue myself and it is a tricky one to solve without a fair bit of custom work. The entities objects do some very dumb things which prevent the normal loop detection logic working.

For now serializing entities doesn't work with Json.NET. If/when it does work I'll post an update here.
Mar 20, 2009 at 6:00 PM
James,
I was able to get this to work but I had to modify your RelectionUtils class and change the GetFieldsAndProperties() method so that it wouldn't return any fields or properties that were added by EntityFramework.  After some digging, I figured out that  EF marks its "non-POCO" properties with a Browable(false) attribute. At first I thought this was a hack to Json.Net but the more I think about it it not too bad.  EF is certainly not Persistent Ignorant and adds a bunch of garbage to POCO object.  I don't know of anybody that's gonig to want to serialize this stuff (and even if you try your going to get circular references so what's the point?).  Whenever the Browsable(false) attribute is found that's a clear signal for "EF add-on junk" that I would never want to serialize.  And since this attribute is only added by EF, it wouldn't affect any other existing objects that Json.Net would serialize.

But, coupling wouldn't be cool.  Json.Net really shouldn't intrisically "know" about EF attributes any more than any other framework's attributes.  Maybe a possible solution would be to make this configurable by the callers of Json.Net.  They could passed in some sort of  "AttributesToIgnore" collection.  Then Json.Net would be uncoupled to EF but we could get those problematic EF properties out of there.

I'd be happy to help with any ideas or coding assistance if you thought this is something worthy of adding.


CODE:

    public static List<MemberInfo> GetFieldsAndProperties(Type type, BindingFlags bindingAttr)
    {
      List<MemberInfo> targetMembers = new List<MemberInfo>();

      //*************************************************
      //MWS - Be more selective about including fields.
      //      Need to eliminate fields where Browsable attribute is false
      //      because of EntityFramework non PersistenceIgnorance intrusion!!
      //targetMembers.AddRange(type.GetFields(bindingAttr));
      //targetMembers.AddRange(type.GetProperties(bindingAttr));

      BrowsableAttribute ba;
      foreach (FieldInfo i in type.GetFields(bindingAttr))
      {
          ba = Attribute.GetCustomAttribute(i,
              typeof(BrowsableAttribute), false) as BrowsableAttribute;

          if (ba == null || ba.Browsable == true)
              targetMembers.Add(i);
      }

      foreach (PropertyInfo i in type.GetProperties(bindingAttr))
      {
          ba = Attribute.GetCustomAttribute(i,
              typeof(BrowsableAttribute), false) as BrowsableAttribute;
          
          if (ba == null || ba.Browsable == true) 
              targetMembers.Add(i);               
      }
      //*************************************************




      // for some reason .NET returns multiple members when overriding a generic member on a base class
      // http://forums.msdn.microsoft.com/en-US/netfxbcl/thread/b5abbfee-e292-4a64-8907-4e3f0fb90cd9/
      // filter members to only return the override on the topmost class
      List<MemberInfo> distinctMembers = new List<MemberInfo>(targetMembers.Count);

      var groupedMembers = targetMembers.GroupBy(m => m.Name).Select(g => new { Count = g.Count(), Members = g.Cast<MemberInfo>() });
      foreach (var groupedMember in groupedMembers)
      {
        if (groupedMember.Count == 1)
          distinctMembers.Add(groupedMember.Members.First());
        else
          distinctMembers.Add(groupedMember.Members.Where(m => !IsOverridenGenericMember(m, bindingAttr)).First());
      }

      return distinctMembers;
    }

May 7, 2009 at 2:00 PM
As a kind of an exercise, I wrote an ADO.NET Entities dumper which dumps on object graph into JSON. This did not require any rewriting on the JSON part, but needed some converters and mappingresolvers.

(I wanted to attach the code here, but codeplex does not seem to have functionality for that...)

Just say so if you want to have the code.

Coordinator
May 8, 2009 at 8:50 AM

You can upload patches in the Source Code section. I would be interested to see what you have done.

May 8, 2009 at 8:27 PM

I didn't write any patches, but just a (standalone) project which uses JSON.NET for serializing ADO.NET Entityframework graphs to JSON format.

The source can be found at ftp://eludias.dyndns.org/EntitiesAsJSON.zip

Features:

  - readable database dump

  - example code on how to write JSON converters and mapping resolvers :)

Limitations:

  - ADO.NET Entityframework is slow. This project makes it slower (not yet speed optimized)

  - It only serializes, but does not yet deserialize. (this needs support from JSON)

  - It keeps all objects in memory before outputting anything; no streaming at all.

 

May 19, 2009 at 2:18 PM

eludias,

I can't find an example there :( can you post it here?


May 19, 2009 at 3:29 PM

Could you be a little bit more specific?

Does the link ftp://eludias.dyndns.org/EntitiesAsJSON.zip not work for you?

I cannot post it here, since one cannot post/upload .zip files here.

May 19, 2009 at 8:42 PM

I have downloaded the zip, linked your project to my solution, and what to do now? :) I have such code: "List<Product> products = (from prod in m_Db.ProductSet select prod).ToList(); products.ForEach(/*loading required relationships*/);" Now I need to serialize this List<Product> to JSON. What to do next with your code?

May 21, 2009 at 7:22 PM

Yeah, I know, the documentation is somewhat lacking, just like the features :)

Now you do something like:

                var json = EntitiesAsJSON.EdtJsonSerializer.Serialize(
                    // Root objects:
                    new IEnumerable[] { products },
                    // Namespaces to remove:
                    new string[] { "YourNameSpace" }
                );
                Console.WriteLine(json);

But be warned, since JSON.NET is missing the 'populate' functionality together with custom converters (the newest not released version has populate without custom converters), deserialization is currently not possible (and not written).

Rutger Nijlunsing.

Mar 21, 2010 at 7:19 AM

@westrate: why you not try using "JsonConvert" class.

 

            JsonSerializerSettings jsSettings = new JsonSerializerSettings();
            jsSettings.ReferenceLoopHandling = ReferenceLoopHandling.Ignore;

            string jsonTask = JsonConvert.SerializeObject(task, Formatting.None, jsSettings);

but can anyone help me that I dont want to serialize the reference object, how can i set this. and also if i want specific reference object or even few field from that object?

 

 

            JsonSerializerSettings jsSettings = new JsonSerializerSettings();
            jsSettings.ReferenceLoopHandling = ReferenceLoopHandling.Ignore;
            string jsonTask = JsonConvert.SerializeObject(task, Formatting.None, jsSettings);

 

 

 

 

Jul 2, 2010 at 10:23 AM

@asifrazafastian

worked fantastic for me, thank you!