1

I process a lot of tweets in real time using python and for each tweet I need to assign it in to a specific bucket. I have about 50 buckets, each with their own rules. The majority of them are simple like so

bucket_one = ['test', 'foo']
if any(x in tweet['text'] for x in bucket_one):
  tweet['bucket_one'] = 1

So the way I designed it is by having a dictionary where the key is the bucket name and the value is a list of terms that would need to be present. This way I can have 1 generic method that would assign the tweet to a bucket by looping through the dict.

My problem is that some buckets have more complicated rules like:

if 'boo' in tweet['text'] and not any(y in tweet['text'] for y in no_words):
  tweet['bucket_two'] = 1

So as you see I have a neat and elegant way (in my opinion and please challenge me if you can think of a better way) of assigning the easy buckets however there are about 10 buckets with more complex rules that differ.

Is it ok to have 1 generic method for 40 buckets and then have an individual method for each other more complex bucket? or should I just have 50 individual methods?

1 Answer 1

4

I think a very maintainable way would be to define functions that take a tweet and return a boolean like this:

def has_terms(*terms):
    return lambda tweet: any(term in tweet['text'] for term in terms)

def complex_rule(tweet):
    return 'boo' in tweet['text'] and not any(y in tweet['text'] for y in no_words)

Then you can set up your list of rules like this:

rules = [(has_terms('test', 'foo'), 'bucket_one'),
         (complex_rule, 'bucket_two')]

Then assigning your buckets looks like this:

for rule, bucket in rules:
     if rule(tweet):
         tweet[bucket] = 1

This makes a very clean separation between the implementation of the rules, the specification of your list of rules, and the application of the list of rules to a tweet.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.