1

In this book, the author is using avro to serialize data before it is being processed by pig. What I don't understand is, why we need to do it using a serialization framework like avro? Why don't we just use json as the serialization data format? This way we can just use serializer/deserializer library in a language e.g. data.json in clojure.

closed as primarily opinion-based by gnat, Bart van Ingen Schenau, user40980, user53019, Wayne Molina Aug 5 '14 at 16:39

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

4

Rendering data into JSON can be considered a form of serialization, albeit into a very unconstrained format. A lot of the value in a serialization protocol and library is the additional structure it imposes on the wire representation.

JSON used as-is is not able to represent via any built-in way the 'type' of objects or inter-object references. In order to represent them, you need to add additional ad-hoc structure yourself like fields encoding type information, object IDs so they can be linked, and so on.

A purpose of a serialization library is to encode these things in a consistent manner onto the chosen transport format (custom binary, JSON, XML, text), such that the important aspects of the data are not lost.

When you stuff a bunch of data into a JSON file, you are adhering to an informal contract by the encoding and interpretation of the producer and consumer. Concepts such as JSON Schema may help structure such an approach, but it's often easier to use an established serialization format.

Not the answer you're looking for? Browse other questions tagged or ask your own question.