Scary Error

Feb 10, 2011 at 3:01 AM

We've been getting an intermittent but serious error.  Basically, sometimes we get a deadlock on one of our transformation changes, but sometimes we don't.

It's when we're hooking up to a valueprovider's propertychanged event.  Here's the stacktrace.

   at Obtics.ObservableObjectBase.Lock()
   at Obtics.NCObservableObjectBase`1.SubscribeINC(IReceiveChangeNotification receiver)
   at Obtics.Collections.Transformations.NotifyVpcTransformation`2.Notifier..ctor(TType inItem, IValueProvider`1 valueProvider, NotifyVpcTransformation`2 owner)
   at Obtics.Collections.Transformations.NotifyVpcTransformation`2.CreateNotifier(TType item, NotifyVpcTransformation`2 owner)
   at Obtics.Collections.Transformations.NotifyVpcTransformation`2.RefreshBuffer()
   at Obtics.Collections.Transformations.NotifyVpcTransformation`2.UpdateHaveChangedListeningClients(Boolean newValue, Boolean oldValue)
   at Obtics.NCObservableObjectBase`1.set_HaveChangedListeningClients(Boolean value)
   at Obtics.NCObservableObjectBase`1.SubscribeINC(IReceiveChangeNotification receiver)
   at Obtics.NCSourcedObject`1.UpdateNeedSourceChangedListener(Boolean newValue, Boolean oldValue)
   at Obtics.NCSourcedObject`1.set_NeedSourceChangedListener(Boolean value)
   at Obtics.NCSourcedObject`1.RefreshHaveSourceChangedListener()
   at Obtics.NCSourcedObject`1.UpdateHaveChangedListeningClients(Boolean newValue, Boolean oldValue)
   at Obtics.NCObservableObjectBase`1.set_HaveChangedListeningClients(Boolean value)
   at Obtics.NCObservableObjectBase`1.SubscribeINC(IReceiveChangeNotification receiver)
   at Obtics.NCSourcedObject`1.UpdateNeedSourceChangedListener(Boolean newValue, Boolean oldValue)
   at Obtics.NCSourcedObject`1.set_NeedSourceChangedListener(Boolean value)
   at Obtics.NCSourcedObject`1.RefreshHaveSourceChangedListener()
   at Obtics.NCSourcedObject`1.UpdateHaveChangedListeningClients(Boolean newValue, Boolean oldValue)
   at Obtics.Collections.Transformations.ExtremityAggregateBase`5.UpdateHaveChangedListeningClients(Boolean newValue, Boolean oldValue)
   at Obtics.NCObservableObjectBase`1.set_HaveChangedListeningClients(Boolean value)
   at Obtics.NCObservableObjectBase`1.SubscribeINC(IReceiveChangeNotification receiver)
   at Obtics.Values.Transformations.CascadeTransformation`2.UpdateBuffer(IInternalValueProvider`1 newValue, IInternalValueProvider`1 oldValue)
   at Obtics.Values.Transformations.CascadingTransformationBase`5.set_Buffer(TItm value)
   at Obtics.Values.Transformations.CascadingTransformationBase`5.RefreshBufferCache()
   at Obtics.Values.Transformations.CascadingTransformationBase`5.UpdateHaveChangedListeningClients(Boolean newValue, Boolean oldValue)
   at Obtics.NCObservableObjectBase`1.set_HaveChangedListeningClients(Boolean value)
   at Obtics.NCObservableObjectBase`1.SubscribeINC(IReceiveChangeNotification receiver)
   at Obtics.NCSourcedObject`1.UpdateNeedSourceChangedListener(Boolean newValue, Boolean oldValue)
   at Obtics.NCSourcedObject`1.set_NeedSourceChangedListener(Boolean value)
   at Obtics.NCSourcedObject`1.RefreshHaveSourceChangedListener()
   at Obtics.NCSourcedObject`1.UpdateHaveChangedListeningClients(Boolean newValue, Boolean oldValue)
   at Obtics.NCObservableObjectBase`1.set_HaveChangedListeningClients(Boolean value)
   at Obtics.NCObservableObjectBase`1.SubscribeINC(IReceiveChangeNotification receiver)
   at Obtics.NCSourcedObject`1.UpdateNeedSourceChangedListener(Boolean newValue, Boolean oldValue)
   at Obtics.NCSourcedObject`1.set_NeedSourceChangedListener(Boolean value)
   at Obtics.NCSourcedObject`1.RefreshHaveSourceChangedListener()
   at Obtics.NCSourcedObject`1.UpdateHaveChangedListeningClients(Boolean newValue, Boolean oldValue)
   at Obtics.NCObservableObjectBase`1.set_HaveChangedListeningClients(Boolean value)
   at Obtics.NCObservableObjectBase`1.SubscribeINC(IReceiveChangeNotification receiver)
   at Obtics.NCToNPC.set_HavePropertyChangedListeningClients(Boolean value)
   at Obtics.NCToNPC.add_PropertyChanged(PropertyChangedEventHandler value)
   at Obtics.NCSourcedObjectToVP`2.add_PropertyChanged(PropertyChangedEventHandler value)

Do you have any possible thoughts what this might be related to?  Does the stack trace provide any clues that the untrained eye can't see?  We're kinda stuck.

Our only idea is to figure out a way to print out the transformation chain and see where the deadlock is coming from, but even that might not lead to the real answer.

Note that the property we're seeing this on has no cycles visible in inspecting the code, and it actually works most of the time.

Thoughts? :|

Coordinator
Feb 10, 2011 at 8:28 AM

I assume you are using multiple threads in your application?

What is the other thread (the one that is holding the lock) doing?

Does this happen in one particular expression or is it a general problem?

When a transformation 'pipeline' is queried for its value or change listeners are added (basicly any operation originating from client going (drilling) down to the source) the whole chain of transformations will get locked.

Could it be that a thread that is drilling for a value (or collection) is triggering and waiting for your deadlocked thread to attach a change listener (it could even be the same thread)? If it does it should do this in a detached maner like via a dispatcher.

It should be fine if all querying threads go from client to sources only. Deadlocks may happen if such a thread (or a thread it is waiting on) would re'enter from the top or cause a change notification to bubble up to the clients(modify a source by reading it). So obtics is threadsafe and concurrent but not re-entrant. (with the exception that on receiving a change notification you can drill for a value immediately). Naturally that is a 'should'. There might be a bug lurking.

Feb 10, 2011 at 5:26 PM

Obtics is checking that the thread ids match before failing with the deadlock.

Coordinator
Feb 11, 2011 at 9:42 AM

Do you mean that it is failing with an exception? Reporting "Deadlock - transformation is its own source"?

That points to a dependency loop. You could create it like this:

var source = new ObservableCollection<IEnumerable<int>>();
var trf = Obtics.Collections.ObservableEnumerable.Concat(source);
source.Add(trf);
((INotifyCollectionChanged)trf).CollectionChanged += (s, e) => Console.Write("changed");
Console.Write(trf.FirstOrDefault());

The thread then should be in the same object somewhere further down the stack.

Btw. It doesn't need to be a continuous trail of obtics methods in the stack. It could be interspersed with system or your own code. 

Neither does it need to be direct. Might be for example that accessing a property that is the source of a tansformation pipeline triggers a lazy initialization of an object where some part of the initialization accesses the same or part of the same pipeline.