32

As a web developer I have very little understanding of binary data.

If I take the sentence "Hello world.", convert it to binary, and store it as binary in an SQL database, it seems like the 1s and 0s would take up more space than the letters. It seems to me like using letters would sort of be like using compression, where one symbol stands for multiple.

But is that really how it works?

Does storing plain text data take up less space than storing the equivalent message in binary?

  • 126
    You do not know the absolute minimum that every developer must know about character encoding. Fortunately the founder of this site wrote you an article. Read it before you program again. joelonsoftware.com/2003/10/08/… – Eric Lippert May 26 '17 at 22:01
  • 16
    @EricLippert A great read and I'm better off as a result thank you. – john doe May 27 '17 at 1:35
  • 4
    I recommend also utf8everywhere.org – Basile Starynkevitch May 27 '17 at 3:04
  • 2
    Being a web developer isn't a excuse to don't know how character encoding and binary data works. You really need to brush up your skills... – T. Sar - Reinstate Monica May 29 '17 at 11:46
134

Plaintext is binary.

When you write an H to a hard drive, the write head doesn't carve two vertical lines and a horizontal line into the platter, it magnetically encodes the bits 010010001 into the platter.

From there, it should be obvious that storing plain text data takes up exactly the same amount of space as storing binary data.

But plaintext is just one2 particular binary format

Plaintext can be reversibly transformed into other binary formats. One common transformation is compression which usually results in a more compact representation, meaning fewer bits used to represent the same information.

Depending on what you're using the plaintext to represent, you may be able to use different binary formats to represent the same information. This may use more space, it may use less.

For example, the numbers 5 and 1234567 could be represented in plaintext using digit characters, resulting in these bit sequences on disk3:

00110101 00000000
00110001 00110010 00110011 00110100 00110101 00110110 00110111 00000000

Alternatively, you could use 32-bit two's complement:

00000000 00000000 00000000 00000101
00000000 00010010 11010110 10000111

Which is a less compact representation of 5, but more compact representation of 1234567.

And there is a literally infinite number of other representations which would have varying levels of compactness, and flexibility, although, in practice far less than that many representations are actually used.


1 Assuming UTF-8. The exact sequence of bits for a character depends on which specific encoding you're using.

2 Or really, several formats, given the various encodings.

3 If you're wondering what those eight zeros on the ends are, well, you need some way of knowing how long the data is. The options basically boil down to a marker (I used this, via a null byte), space dedicated to storing the length (Pascal used a byte to store the length of a string), or a fixed size (used in the subsequent two's complement example).

  • 6
    One slight difference is the representation of End-of-line, which in Unix/binary takes one byte (LF) while in Windows/text takes two bytes (CR-LF). – Glenn Randers-Pehrson May 26 '17 at 17:35
  • 97
    +1 for "the write head doesn't carve two vertical lines and a horizontal line into the platter. – Tulains Córdova May 26 '17 at 18:09
  • @BaardKopperud You are right! ;) – Tulains Córdova May 26 '17 at 21:14
  • 2
    @BaardKopperud There is/was LightScribe, but that wasn't really meant for computer reading, though perhaps something like Google Goggles could read some LightScribe labels. But doing that on the actual data storage side would be pretty interesting. Reminds me of songs that have fancy graphics when run through an oscilloscope. – 8bittree May 26 '17 at 21:23
  • 2
    @TulainsCórdova Though actually, Turing machines operate on an arbitrary alphabet, so they in theory could write letters onto the tape. It just so happens we've settled on using a two-symbol alphabet. – gardenhead May 27 '17 at 4:14
15

I find this a great fun thing to think about. Binary is not 1s and 0s in the way you talk about it.

Imagine there is a quantity, I can tell you what quantity it is in many different ways:

  • Nine in English
  • Neuf in French
  • 9 in Arabic numerals
  • IX in Roman numerals
  • 1001 in Binary with Arabic numerals
  • on off off on in Binary with on/off
  • high low low high in Binary represented with voltages or levers or water levels or electric charge ... or English words 'high' and 'low'

They all represent the same thing. The point here is that binary is not 1s and 0s, that's only one way to represent of a value.

When you talk of converting a H into binary, you probably imagine seeing 10101010 on screen - but that's not "binary", that's one digit for each binary bit.

Yes, if you converted H to "binary" as people normally talk about it, and then represented that in Arabic digits and then stored it, it would take more space in the same way that converting H to aitch takes more space.

But you can see that binary is one way of representing a quantity, well by that logic saying "if I converted H to binary and represented it as high low high low high low high low then it would take 35 characters! That's even more than 10101010! But these two are both 'binary' .. so how is one bigger than the other?

The other side of this is to wonder how H is stored by a computer, and to see that H is itself just a way of representing a quantity - the same quantity 72, 01001000, or seventy two or ASCII character code H. Which is 8bittree's answer that plain text is binary, but this is me trying to show what that means.

So you get a bit pattern in a computer 01001000 and what does it mean? Anything - could be talked about as a number, as a part of a zip file, as a character, depends what the intent of the person who created it was. If you know it's supposed to be plain text, then it came from a character encoding H -> 01001000 and you look it up the other way in the character encoding table - ASCII, UTF-8, shift-jis, etc. and find the right font character and out comes a H or whatever. Or out comes the wrong character if you use a different encoding lookup than the person who created it used. This is @Eric Lippert's link.

But as I write this, and as you think about it, H is one byte and 01001000 is 8 bytes, yes that's more space. And yes it's (a representation of) binary. But it's at a higher level of abstraction than the computer is using - binary displayed in ASCII characters, where each character is represented behind the scenes with a binary bit pattern, each as big as the H alone.

12

Does storing plain text data take up less space than storing the equivalent message in binary?

No, never.

Your computer already stores the plain text data in the equivalent binary representation. Storing something as plain text versus binary just signals how the computer should interpret that identical binary stream.

It seems to me like using letters would sort of be like using compression, where one symbol stands for multiple.

That is kinda true. One character will represent more than one bit. The problem is that they're different sized things. It only takes one bit to store a 1 or a 0, but 8 bits (or more) to store a plain text character. You don't gain anything by using characters.

If anything, you can compress things the other way. After all, 8 bits is 256 different possible values, yet plain text usually is limited to letters, numbers, and a few punctuation characters. It doesn't need as many bits as it takes.

  • 3
    Well, maybe sometimes :-) Two possible cases I can think of. 1) You have a short text string which you compress. The compressed file contains some metadata, which makes the compressed file larger than the original string. 2) You have some floating point values, say 1.2. Storing as text would be 3 bytes (4 with a terminator), while storing a binary double would take 8 bytes. – jamesqf May 26 '17 at 20:44
  • 5
    The answer really depends on what you mean by 'binary.' For example, UTF-32 takes up four times as much space as ASCII, so if by 'plain text' you meant ASCII, and by 'binary' you meant UTF-32, plain text would take less space than binary. But you can reverse the definitions and get the opposite result. – David Conrad May 26 '17 at 21:13
  • 1
    @DavidConrad Well, that just skirts on the "there's no such thing as plain text". The closest thing you have is a binary file with no metadata/headers identifying the type and guessing "must be text encoded as XXX!". There has been a time when "plain text file" meant something reasonable, in a limited context, but it doesn't really anymore. The best you can get is "all the data in the file is encoded as text" in contrast with "some/all parts of the data are not encoded as text". – Luaan May 29 '17 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.