3

I am trying to log a JSON error message to Google's Cloud Logging platform. Unfortunately, the maximum size message you can log is 8000 bytes and some of the JSON documents I want to log are larger than this. Rather than just chop the JSON string off I would prefer to iteratively remove the largest 'field' (and replace it with a marker such as 'REMOVED') until the document is under the limit.

I've got something working that essentially walks through the deserialized version of the document, field by field, keeping a track of the longest value found so far. This method fails to keep track of where in the object graph this item is located but I am able to perform a simple replace on the JSON string afterwards.

Although this code does the job for what I need, I'm still interested to know if there is an existing algorithm or built-in Python functions that could be applied in this situation.

import json
import sys


class RecursiveApply:
    def __init__(self, function_to_apply):
        self.function_to_apply = function_to_apply

    def process_item(self, current, item):
        if isinstance(item, list):
            for i in item:
                current = self.process_item(current, i)
            return current
        elif isinstance(item, dict):
            for k, v in item.items():
                current = self.process_item(current, v)
            return current
        else:
            return self.function_to_apply(current, item)


class GetBigger:
    def __init__(self, ignore_value):
        self.ignore = ignore_value

    def get_bigger(self, biggest, obj):
        json_obj = json.dumps(obj)
        size = sys.getsizeof(json_obj)
        if (biggest is None or size > biggest[0]) and obj != self.ignore:
            return size, json_obj
        else:
            return biggest


class LimitJSON:
    def __init__(self, find_biggest, replace_val):
        self.find_biggest = find_biggest
        self.replace_val = replace_val

    def limit_JSON(self, max_len, json_string):
        current = None
        while len(json_string) > max_len:
            biggest_item = self.find_biggest(current, json.loads(json_string))
            if biggest_item:
                json_string = json_string.replace(biggest_item[1], '"' + REPLACE_VAL + '"')
            else:
                break
        return json_string

REPLACE_VAL = '*REMOVED*'
gb = GetBigger(REPLACE_VAL)
ra = RecursiveApply(gb.get_bigger)
lj = LimitJSON(ra.process_item, REPLACE_VAL)


original_data = ['long array item', {'item1': 'dict item', 'item 2': 'longer dict item'}, 'little item', 123123123123123123123123123]
json_string = json.dumps(original_data)
processed_json = lj.limit_JSON(100, json_string)
print(processed_json)

# RESULT ->
# ["long array item", {"item 2": "*REMOVED*", "item1": "dict item"}, "little item", "*REMOVED*"]
| improve this question | |
  • If you deserialize the JSON string into an actual first-class JSON object, you will get a tree of nodes, which you should be able to process any way you want. – Robert Harvey Mar 28 '16 at 18:35
  • ...Why do you have such huge JSON documents that you want to log? Without more description, it sounds like you should try to reduce the size of the documents logged, or break it into more reasonable pieces, or focus it so it outputs only relevant data. Of course, I could be dead wrong for your project, so take this with a grain of salt. – Fund Monica's Lawsuit Mar 28 '16 at 19:06
  • @RobertHarvey do you mean by using something like this? pypi.python.org/pypi/jsonpath-rw – samaspin Mar 29 '16 at 8:52
  • @QPaysTaxes one of the fields is an html page that the error occurred on - this could be 50kb+. I dont want my application to have to worry about writing it to disk somewhere or trying to handle it separately, it just wraps up any errors into a single event including all the info you need to debug it and pushes it out the door. This manipulation I'm doing is just a work-around in the execution environment (the app itself knows nothing about this) until I find an alternative way to move events. – samaspin Mar 29 '16 at 9:00
1

Use dict.items() to create an iterator and sorted()

dict = {"a":"abc", "b":"defg", "c":"gh"}

  jd = json.dumps(dict)
  print(jd)
  data = json.loads(jd)

  sv = sorted((len(value), key) for (key, value) in dict.items())



  for i in range(len(sv)-1,-1,-1):
     l,v=sv[i]
     print(l,v)

Produces

  4 b
  3 a
  2 c
| improve this answer | |
  • How does this solve the OPs problem? expanding on this may be useful. – esoterik May 25 '18 at 20:57
  • OP asked " I would prefer to iteratively remove the largest 'field' (and replace it with a marker such as 'REMOVED')" The solution shows how to 1) load the json object, 2) sort attributes by size, 3) iterate over the largest items first. That is what they asked to be able to do. So it helps. – RichMeister Jun 1 '18 at 22:45
  • The user interface here does not permit adding to the comment. So I'll add another. Having found the item to replace with the marker which is now in variable v, they can then set jd[v]="REMOVED" Go to the next item until the total of l is>= to what they need to remove and then run dumps. – RichMeister Jun 1 '18 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.