I am currently facing a situation where I will need to obfuscate data for legal reasons (these are countrywide rules and not specific to me)

I will still need to work with teams outside this area who will still need to access applications/databases for testing/deployment and so on, however the data must be fully obfuscated.

What are your experiences of this scenario?

e.g. Who owns what stage of the process, what access do the various parties have, how strong is the encryption, how do updates occur

  • I assume that by "obfuscation" you mean "encryption"; also, there's no reason to use weak encryption when implementations of strong encryption (like AES) are readily available. Could you describe the situation better? Where is encryption required, and of what? Who needs to have access, and for what? Nov 22 '10 at 14:53
  • I didn't use the term 'encryption' as to me this implies that it will be subsequently decrypted, which is not what is required. The encryption must occur as soon as the data leaves the source system. The encryption is of sensitive client data. I'm more interested in knowning 'best practice', 'strategies' etc Nov 22 '10 at 15:52
  • 2
    "Obfuscation" is a perfectly legitimate thing to do - it means you get data as varied and crazy as in real life, but the data's been manipulated in such a way that you can't, say, tie the data back to people. Some might call it "cleaning" or "anonymising" the data. Nov 22 '10 at 23:49
  • It may be useful to identify which legal reasons you need to obfuscate the data; as your profile puts you in Europe, I suspect this may be data protection legislation which stops the export of personal data outside the EU (and a few safe-harbour countries, as well as other exceptions). There are some cases of obfuscation that wouldn't satisfy the requirements off that, for instance. Dec 30 '10 at 16:14

One large financial client we do business with has a standardized automated process for obfuscating data. We don't, so I have a few scripts where I do this by hand. The point is to leave reasonably realistic data (lengths of names, postal codes) while rendering the personally identifiable data irretrievably scrambled. Their system is far more complicated than this, but basically when production data gets copied to development and QA environments, it will be scrambled automatically. This way there is no potential for "forgetting" to do some of the scrambling.

Set them all to something test accounts use: like Password1 or 1234567.

Tax ID numbers, Social Insurance Numbers, Social Security Numbers:
Take the first 3 digits and generate random numbers for the remainder. In the US, the first 3 digits are generally assigned based on where you lived when the SSN was issued, so not all combinations of first 3 digits are valid. For EINs, take the first 2 digits, as not all combinations of first 2 digits are valid. Adjust which digits get left alone if your country uses different rules.

Hash and base64 the first and last names separately. Take the first letter of unhashed name append the hash afterwards and truncate the result to original name's length

Example: Name = "John Doe" (I am using SHA384)

So John Doe gets turned into Jnbn Dnh. It helps to keep the names the same length as that may help to point out usability issues.

If you have rules such as "names cannot have digits" then you need to remove out the base 64 values that aren't valid, also lowercasing the subsequent letters (done in sample code below).

Addresses: Street names and city names get hashed as names above do. Numbers stay the same. State and zip stays the same.

So 1313 Mockingbird Lane becomes 1313 Mvtqiwtuqrd Lzzx

Phone numbers:
Leave area code the same, generate random digits for the remaining digits.

Credit Card Numbers:
You should not be storing these at all.

Here is some sample & crude C# code for hashing and truncating (simple to display the concept)

    using System.Security.Cryptography;  
    using System.Text.RegularExpressions;   

    public string ScrambleInput(string sInput)
        string sReturn = sInput.Substring(0,1);
        string sTemp = string.Empty;
        System.Security.Cryptography.SHA384Managed Hasher = new SHA384Managed();
        System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
        byte[] buff = new byte[sInput.Length];
        buff = enc.GetBytes(sInput);
        sTemp = Convert.ToBase64String(Hasher.Hash, 0, Hasher.Hash.Length, System.Base64FormattingOptions.None);
        sTemp = sTemp.ToLower().Replace("+", "").Replace("/", "");
        sReturn += Regex.Replace(sTemp, @"\d", "");
        sReturn = sReturn.Substring(0, sInput.Length );
        return sReturn;
  • 1
    I'm not sure whether it's just the example data you give or whether it's the hash algorithm but if it preserves the initials of the individuals it's likely to be unsuitable - if combined with other data it's potentially still identifiable. I'm dubious even about keeping the lengths as in extreme cases these may be useful as part of a compound identifier. Nov 24 '10 at 9:02

First thing, as you say, this isn't encryption or possibly even obfuscation. There needs to be no possible way to retrieve the data - the legal restriction isn't on exporting the data securely, it's on exporting it at all.

Also worth noting that that will include someone outside the country accessing a database within the country (exporting a single record is still exporting data) so they can have no access to your systems via any means.

The issue with anonymising the data is it's very hard to do. You've got to scrub every personally identifiable piece of information (don't just scramble them) including names, addresses, e-mails, dates of birth, social security numbers. Anything that might be usable even if only as part of a composite needs to be removed. You also need to check comments fields, notes fields, audit fields and anything else for these same values.

When you're scrubbing the data, don't swap things. Replace each one from a random list of possible values. If you swap them the data is still there and there are risks they'll be swap and swapped back or in some other way remain readable. Remember, the restriction is on exporting the data, not making it hard to read.

Beyond this you need to understand what might be implied by data. Say it's a film rental database and the guys outside the country know from a support mail that someone has rented one particular a few times. That's a straight forward search which will likely give you a very short list. Cross reference it with another few things (it's a guy so he probably wouldn't rent that and so on) and without his name being anywhere there you've got his whole history.

Read this: http://arstechnica.com/tech-policy/news/2009/09/your-secrets-live-online-in-databases-of-ruin.ars for a better example.

You imply that the reasons the people outside the country need the database are for technical reasons so the actual data isn't important to them - just that they have something they can run up, check it works and so on.

If that's the case then for my money the best way is don't bother. Send them a test database instead with no real data in. Need more data than your test database has? Write scripts to generate it.

The scripts then need to be maintained along with the rest of the database and application, updated with each new release and so on.

If you have a skeleton database (that is with structure but no data) you can just give the other guys access to the scripts to create databases as they want (which is what I'd recommend). If you do go with a process of scrubbing the live data you probably need to maintain the scripts for doing it in the same way but have to run the process yourself - I'd personally create a test database with each release.


I don't know if there are any best practices for this sort of thing. But if I were given this task I think I'd identify what correlations yield the most data and then try to randomize those correlations.

The problem with a piece of data is not the data itself. It's the correlations that can be made. Any obfuscation scheme would do best to concentrate on randomizing those correlations so thoroughly that they couldn't be extracted again. One correlation that's independent of other pieces of data is the overall rarity of a piece of data, and that correlation should also be obscured.

For example, identify a rarity cutoff for first and last names, then replace all first and last names below that rarity cutoff with others that are just as rare, but picked from a random list of baby names or something along those lines. I might also copy some names from one place to another to change the frequencies at which they occur. For example, change all "John"s to "Avercrombie"s, and all "Hilda"s to "Mary"s.

Then I would start randomly swapping the first names with other first names in the tables. I would do the same (but independently) for last names. I would also start scrambling important relationships. For example, with a Netflix style database, the list of movies people rented. I would randomly move movies from one person's list to another until all the lists were hopelessly scrambled. Of course, if what you want is to be able to refine a prediction algorithm with 'anonymized' data, that would make the data useless, so you will have to exercise judgment in what data you did this with.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.