Hot answers tagged

163

systemd famously stores its log files in binary format. The main issues I have heard with it are: if the log gets corrupted it's hard to recover as it needs specialist tooling they are not human readable, so you can't use standard tools such as vi, grep, tail etc to analyse them The main reason for using a binary format (to my knowledge) was that it was ...


134

Plaintext is binary. When you write an H to a hard drive, the write head doesn't carve two vertical lines and a horizontal line into the platter, it magnetically encodes the bits 010010001 into the platter. From there, it should be obvious that storing plain text data takes up exactly the same amount of space as storing binary data. But plaintext is just ...


105

The word you are looking for is not "bit" but "symbol." "Symbol" is the word used to describe the process of mapping hardware signals (such as voltages or magnetic patterns) into logical bits. If a symbol may have 4 states, it can encode 2 bits worth of information. Of course, we aren't saying anything about the resource usage of the symbol in that ...


91

There are 28192 possible different 1K blocks. Storing them all would take 28202 bits of storage. Since the universe contains only about 1080 (or ~2266) particles, it's a safe bet that it isn't possible to store them all, and you don't have to wonder about whether it would save time or not. But there is, in fact a more interesting way of answering this. You ...


89

Why do most log files use plain text rather than a binary format? Search for the word "text" in the Unix philosophy Wikipedia article, for example you'll find statements like: McIlroy, then head of the Bell Labs CSRC (Computing Sciences Research Center), and inventor of the Unix pipe,[9] summarized the Unix philosophy as follows:[10] This is ...


49

There are a lot of debatable presumptions here. Logging has been an integral part of (almost) every job I've had. It is essential if you want any sort of visibility on the health of your applications. I doubt that it is a "fringe" use; most organizations I've been involved with consider logs very important. Storing logs as binary means you must decode ...


36

Log files are a critical part of any serious application: if the logging in the app is any good, then they let you see which key events have happened and when; what errors have occurred; and general application health that goes beyond whatever monitoring has been designed in. It's common to hear about a problem, check the application's built-in diagnostics (...


33

The way I see this, it depends on what you intend to do with the data afterwards. Based on a few simple checks you can determine which of the two data structures is better for you: Does this data have any logic associated with it? For example, is $price stored as an integer number of cents, so a product with a price of $9.99 would have price = 999 and not ...


24

You may not have a very good picture of computing technology pre-1974. Time-sharing, multi-user, systems were invented in the late fifties, but they were comparatively rare through all through the 60s. Most computers ran in batch mode, running a single program at a time, with no facilities for interacting with users other than the card reader, the line ...


21

One quarternary memory cell can store exactly as much information as 2 binary memory cells: Quaternary Binary 0 00 1 01 2 10 3 11 So if you have same number of memory cells, but they are quarternary, then you have twice as much memory. But if this quad cell takes twice as much space on a chip, then there is no benefit. ...


20

As others have already pointed out, you have 2^8192 possibilities for a 1k block. This means you would need 8192 bits to encode the address of a block if all blocks addresses are encoded with the same amount of bits, so your addresses would be 1k long. You wouldn't have gained anything except adding a layer of indirection so you wouldn't gain any performance....


15

Cloud-based file syncing is great for pictures of your cat. Cloud-based scm such as github or bitbucket are what you should use for code. They have all the advantages of cloud file storage plus nearly magical abilities to version, compare and merge source files. Personally I'm partial to bitbucket as you get free unlimited private and public repositories ...


15

I find this a great fun thing to think about. Binary is not 1s and 0s in the way you talk about it. Imagine there is a quantity, I can tell you what quantity it is in many different ways: Nine in English Neuf in French 9 in Arabic numerals IX in Roman numerals 1001 in Binary with Arabic numerals on off off on in Binary with on/off high low low high in ...


14

Punched cards served not only as an input/output medium, but also as a long-term storage device. You could run a program more than once by reloading the card deck for that program. There were no hard drives, no floppy drives, no magnetic tapes.


14

[W]hy did programmers ever write their programs on punch cards? Didn't computer screens and keyboards already exist by the time programmers used them? We use whatever i/o mechanism our computers have. In the 1960's, paper tape was common. In the 1970's (when I started programming), paper tape was being replaced by punched cards. Yes, there were machines ...


13

An approach I've used: count the number of leading 1 bits, say n. The size of the number is then 2^n bytes (including the leading 1 bits). Take the bits after the first 0 bit as an integer, and add the maximum value (plus one) that can be represented by a number using this encoding in 2^(n-1) bytes. Thus, 0 = 0b00000000 ...


13

When should a database be preferred for storing the data over storing the data in a text file? Wikipedia tells us that a database is an organized collection of data. By that measure, your text file is a database. It goes on to say: The data are typically organized to model relevant aspects of reality in a way that supports processes requiring this ...


13

Is it less efficient to store a small bit of data like your example encoded in a string than as binary? Yes. How much less? Not enough to care. Is it less efficient to store thousands of such records in a string than in binary? Oh god yes. Here's why: I can't predict the index of "account1" in the 42nd record because the fields of the previous records didn'...


13

Interpreter An intepreter will work about the way you guessed. In a simple model, it will maintain one dictionary with the variable names as dictionary keys and the variable values as dictionary value. If the language knows the concept of variables that are visible only in specific contexts, the interpreter will maintain multiple dictionaries to reflect the ...


12

Does storing plain text data take up less space than storing the equivalent message in binary? No, never. Your computer already stores the plain text data in the equivalent binary representation. Storing something as plain text versus binary just signals how the computer should interpret that identical binary stream. It seems to me like using letters ...


11

It seems to me that the simplest solution is to make your version header unambiguous and make sure that the old format can never look like it has a format header, you simply look for it. If it's not there you assume it's the old style and try to find it from the middle. There might also be things in the beginning of the old format that can clue you in. ...


10

Keep in mind, once you accumulate enough data, your simple home-grown approach will be very slow when it comes to retrieval unless you then implement some sort of indexing system. So if thats a potential issue, I'd stick with using a dbms. In addition to using a NoSQL server, you can also use an SQL database to store key pairs. There's nothing inherent ...


10

There is a whole lot of theory based around what you are trying to do. Take a look at wiki page about universal codes - there is rather exhaustive list of integer encoding methods (some of which are actually being used in practice). In data compression, a universal code for integers is a prefix code that maps the positive integers onto binary codewords ...


10

Both options are over-engineered, involving the database is inappropriate here, and the directory structure is too deep. Instead, for administrator-friendly software, I see the following patterns emerging: Paths are configured in config files, not database. Usually, you want to be able to copy the database dump to another system. Often, the database ...


10

It depends on the implementation. For example, a C compiler might maintain a symbol table during compilation. This is a rich data structure that allows pushing and popping of scopes, since each compound-statement opening brace { potentially introduces a new scope for new local variables. In addition to handling scopes coming and going, it records the ...


9

Why not do both? Database is the backing, ultimate store for the image. The public side can be a simple read from db but you can easily extend that into a read-through disk cache and also take advantage of numerous infrastructure tricks to better enhance performance. The wins here are: simpler data backup -- database backups are fun and easy, no file ...


9

The weight of history... Lets go back to the Jacquard loom, a mechanical loom invented over 200 years ago. The loom was controlled by a "chain of cards", a number of punched cards, laced together into a continuous sequence. This was a known technology and later on the US Census Bureau used a tally machine - the Hollerith machine to tally census data. ...


9

Read speed and caching is an important factor, but it's not the only factor, and perhaps not even the primary factor in selecting a filesystem's block size. Every block on your filesystem has overhead associated with it. The filesystem must track which blocks are free, which blocks belong to which files, etc. This overhead must be stored on the disk ...


9

The most generalized term for this is "distributed index", and it is indeed a quite complex topic where a lot of active research is done. The first step is a dedicated index server, i.e. one that contains no data itself but for every key it knows which server has the data for that key and forwards the request there. When even the index becomes too big for ...


Only top voted, non community-wiki answers of a minimum length are eligible