0

Although this question takes a very narrow, concrete form, I'm more interested in the conceptual underpinnings of justifying an approach, and how I can apply it to the next problem.

Setup

I'm writing a tool that takes a list of AWS accounts and optionally, for each account, a blacklist of S3 buckets. The tools goes through and does stuff for each account, but ignores the blacklisted buckets.

The tool takes a config as an input:

{
    "accounts": [
        "aws.account.1",
        "my.other.account",
        "aws.account.6"
        ],

    "accountSpecificBlacklists": {
        "aws.account.6": ["my.sqs.enqueue.bucket.high.volume", "my.sqs.audits"],
        "my.other.account": ["sqs.log.testing"]
    },

    "globalBlacklistRegex": "^(critical|restricted|confidential)\..{0,}$",

    "deliveryBucket": "sqs.food.barred",

    "otherParam": "scrubbedValue"
}

The question

The list of accounts above could either be a list as shown, or a key:val pair, with the values being the associated blacklist. The tool would then be informed of which accounts to operate on by looping through the keys of the key:val pair.

e.g.

"accountsToBlacklist": {
    "aws.account.1": [],
    "my.other.account": ["sqs.log.testing"],
    "aws.account.6": ["my.sqs.enqueue.bucket.high.volume", "my.sqs.audits"]
}

I originally had the data model configured in that way, but in the case where there aren't any blacklists, it ends up being a bunch of keys with empty lists as values. It seemed like the key:val approach had polluted the data model, but I couldn't articulate why.

Which approach would you use? Why? What are some guiding questions/principles that you use to decide on data model decisions?

1
  • Does there have to be a key:val pair in "accountsToBlacklist" where the value is empty? Feb 13, 2017 at 21:58

1 Answer 1

1

Both are valid, but each option has its pros and cons; it really depends on the situation:

The first format is more flexible, allowing you to maybe add more details in the future, like another list "accountSpecificPriority" or something similar. It is also compact WHEN you expect to only have a small number of accounts with non-empty Blacklists (when "accountSpecificBlacklists" is very small). In other words, you would use it when the presence of blacklists is an exception, not the rule. Note that in order to retrieve the blacklist for any given account will need to perform a search "accountSpecificPriority", which is a performance hit proportional to the expected size of that list.

The second format - the one using key/value pairs - is more restrictive: it would be difficult to associate other kinds of data alongside the blacklists. On the other hand, it's faster because you don't need to perform a search just to know whether an account has a blacklist associated or not: you will have this info available on hand, so the performance hit will not be there anymore. So this would be both FASTER and more COMPACT for LARGE amounts of data with a HIGH percentage of accounts having non-empty Blacklists.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.