1

In a program I'm working on, I need to do a substantial amount of real-time reflection in order to maintain a list of known "attributes" throughout the program structure (for use by a sort of "virtual programming assistant"). I came up with kind of a "cheaty" way of doing this (which I'm frankly kind of proud of), which is essentially akin to the following:

old_getattribute = obj.__getattribute__
def new_getattribute(self, attr):
    # various things...
    return old_getattribute(attr)
methods = {"__getattribute__" : new_getattribute}
obj.__class__ = type(              # Dynamically create a new class
    "%s" % obj.__class__.__name__, # whose name is _CLASS
    (obj.__class__, ),             # which subclasses CLASS
    methods)                       # and uses the new __getattribute__

Essentially, I dynamically create a new type that subclasses the original type, using a new __getattribute__ method, and then reassign the object's internal __class__ to this new dynamic type.

As amazed and happy as I am that this works (in Python >=3.6 at least), I'm still on the fence about using it. In my past python programming, any kind of modification of an object's magic methods from outside the object's class definition was pretty much forbidden, and yet here I am doing it twice in really messed up ways.

I would like to use it, as it would save me a ton of time instead of keeping track of individual objects in some "appropriate" manner and then continually checking them for changes, but I want to be sure that I'm not going to create any potential danger for the rest of the program.

So that's what I'm here to ask about. From a software architecture standpoint, how potentially dangerous is this method, and what pitfalls I may encounter getting this to work? I'm sure there are more than a few OOP/general programming principles I'm severely violating with this, but I'm willing to let those slide if this can be "managed".

  • Am I understanding correctly that you basically want to inject code into existing objects that you can't (or don't want to) change directly? Where are these objects coming from? Are they coming from an unknown source? – JimmyJames Nov 28 '18 at 19:05
  • @JimmyJames Yup thats the plan. Scary isn't it? They are indeed coming from an unknown source. What I like about this method though is that it essentially provides completely automatic, real-time tracking of changes in the object when they occur. If an object's parameter gets set or updated or deleted, I don't need to constantly check the __dict__ to see what's changed, the change is received immediately. – user3002473 Nov 28 '18 at 19:09
  • It's pretty slick but you are definitely going deep into the inner workings of Python. I only have vague concerns but I'll ponder this some more. Do all calls into this eventually pass-through to the original __getattribute__? – JimmyJames Nov 28 '18 at 19:19
  • @JimmyJames Yes, unless program execution halts inbetween the call to the new __getattribute__ and the call to the original __getattribute__. Otherwise it's essentially the same as just decorating __getattribute__ dynamically. The reason I have to do the obj.__class__ = type(...) trick is because magic method lookups are performed on the class, and not on the instance, so setting obj.__getattribute__ = decorator(obj.__getattribute__) doesn't work. – user3002473 Nov 28 '18 at 19:24
  • @JimmyJames One major concern I just realized is that my injection code might get run in another process. I'm not sure what could go wrong if an object's attributes are modified while the object's __class__ is being modified, but I can't really see any major problems that could occur. Also, as I'm the one running the injection code above, I can essentially control when it's called and ensure it's never called in a parallel process. – user3002473 Nov 28 '18 at 19:28
4

When you're using __getattribute__ you are deep in the bowel's of Python's object model. Things might work, but require a solid understanding. Also, some aspects might be implementation details of CPython and might fail on other implementations.

Things to consider:

  • __getattribute__ is insufficient to observe object modifications.
  • __getattribute__ will not be called when accessing special methods and possibly not when the object is accessed from within the Python interpreter.
  • consider how this interacts with class member access.
  • __class__ might not be assignable.
  • Changing the class breaks type(x) is Foo style checks.
  • Metaclasses might prevent you from extending the original class.
  • Specifying a __getattribute__ implies a noticeable performance overhead. This might be a bit better if you implement it in C (provided you're using CPython).

From an OOP perspective, you're no longer working within the OO paradigm: instead of sending messages to other objects, you're intercepting the language's message dispatch implementation. That's arguable fine, and even has a name: Aspect-Oriented Programming. Python is flexible enough to pull this off implicitly, as you have shown. However, it might be vastly desirable to not instrument existing objects like this, but offer an API that makes it easier to make objects observable. Such an API would perhaps implement properties that can trigger a callback on access.

If you need to observe objects not during their normal execution but for some analysis, it might actually be easier to patch the CPython runtime with an extra hook. You might find some overlap with existing debugger hooks.

  • These are interesting points, thanks! The performance overhead is definitely concerning, seeing as how I would also technically have to overhead __setattr__, __getitem__ and __setitem__, and even then I wouldn't be completely characterizing the changes in the object. For this application, I'm more or less only interested in the object's __dict__ of public values, so it might be "good enough", but the other problems you mentioned are still concerning. – user3002473 Nov 29 '18 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.