Using jsonfold for compact, readable JSON formatting

Using jsonfold for compact, readable JSON formatting

I've been working on a Python module called jsonfold, and I wrote an article describing the motivation and design. I'd really appreciate feedback from other Python developers.

JSON serializers tend to give us two choices: compact JSON, which is efficient but a dense wall of text that's painful to read, or pretty-printed JSON, which is readable but often wastes a lot of vertical space (a small array of numbers can turn into ten lines).

I wanted something in between. jsonfold keeps the shape of pretty-printed JSON, but folds small, simple structures back onto a single line whenever that improves readability. It works on top of your existing serializer (json, orjson, ujson, ...) - you keep using whatever you already have, and jsonfold just reformats the output.

Example 1 - Coding

import sys
import json
import jsonfold

data = {
    "_id": 123,
    "locations": [
        {"city": "Boston",   "state": "MA", "country": "USA"},
        {"city": "Seattle",  "state": "WA", "country": "USA"},
        {"city": "Montreal", "state": "QC", "country": "Canada"},
    ],
    "info": {
        "roles": ["foo", "bar", "baz"],
    },
    "name": "Alice",
}

print("===> json.dump")
json.dump(data, fp=sys.stdout)
print("")

print("===> jsonfold.dump")
jsonfold.dump(data, fp=sys.stdout)

Output

===> json.dump
{"_id": 123, "locations": [{"city": "Boston", "state": "MA", "country": "USA"}, {"city": "Seattle", "state": "WA", "country": "USA"}, {"city": "Montreal", "state": "QC", "country": "Canada"}], "info": {"roles": ["foo", "bar", "baz"]}, "name": "Alice"}

===> jsonfold.dump
{
  "_id": 123,
  "locations": [
    { "city": "Boston", "state": "MA", "country": "USA" },
    { "city": "Seattle", "state": "WA", "country": "USA" },
    { "city": "Montreal", "state": "QC", "country": "Canada" }
  ],
  "info": {
    "roles": [ "foo", "bar", "baz" ]
  },
  "name": "Alice"
}

Example 2 - Packing

Traditional pretty-printing:

{
  "states": [
    "Alabama",
    "Alaska",
    "Arizona",
    ...
    "Wyoming"
  ]
}

jsonfold output:

{
  "states": [
    "Alabama", "Alaska", "Arizona", "Arkansas", "California", "Colorado",
    "Connecticut", "Delaware", "Florida", "Georgia", "Hawaii", "Idaho", "Illinois",
    "Indiana", "Iowa", "Kansas", "Kentucky", "Louisiana", "Maine", "Maryland",
    "Massachusetts", "Michigan", "Minnesota", "Mississippi", "Missouri", "Montana",
    ...
    "West_Virginia", "Wisconsin", "Wyoming"
  ]
}

Same data, just using the available line width more effectively.

Example 3 - Grid Formatting

When an array contains repeated structures, jsonfold can align values into columns:

Traditional pretty-printing:

[
  {
    "orders": 18,
    "product": "Laptop",
    "region": "North",
    "sales": 1250
  },
  ...
  {
    "orders": 24,
    "product": "Mouse",
    "region": "East",
    "sales": 1422
  }
]

jsonfold output:

[
  { "orders": 18, "product": "Laptop",   "region": "North",     "sales": 1250 },
  { "orders": 21, "product": "Monitor",  "region": "Southwest", "sales": 1345 },
  { "orders": 17, "product": "Keyboard", "region": "West",      "sales": 1198 },
  { "orders": 24, "product": "Mouse",    "region": "East",      "sales": 1422 }
]

The module also supports:

  • Folding small arrays and objects onto a single line.
  • Joining adjacent folded objects to further reduce vertical space.
  • Compatibility APIs similar to JSON and JSON::PP.

Full documentation and examples:

PYPI: https://pypi.org/project/jsonfold/

GitHub: https://github.com/yairlenga/jsonfold/tree/main/python

I'd love to hear what the Python developers think. Has anyone else run into JSON pretty-printing pain in logs, configs, or debugging output? And are there formatting styles or options you'd want to see?

u/Yairlenga — 4 days ago
▲ 24 r/perl

I wrote JSON::JSONFold – a CPAN module for compact, readable JSON formatting

Hi everyone! This is my first post in r/perl.

I've been working on a CPAN module called JSON::JSONFold, and I wrote an article describing the motivation and design. I'd really appreciate feedback from other Perl developers.

JSON serializers tend to give us two choices: compact JSON, which is efficient but a dense wall of text that's painful to read, or pretty-printed JSON, which is readable but often wastes a lot of vertical space (a small array of numbers can turn into ten lines).

I wanted something in between. JSONFold keeps the shape of pretty-printed JSON, but folds small, simple structures back onto a single line whenever that improves readability. It works on top of your existing serializer (JSON, JSON::PP, JSON::XS, etc.) - you keep using whatever you already have, and JSONFold just reformats the output.

Example 1 - Coding

use JSON::JSONFold qw(encode_json);

my $data = {
    _id       => 123,
    locations => [
      { city => "Boston",    state => "MA", country => "USA" },
      { city => "Seattle",   state => "WA", country => "USA" },
      { city => "Montreal",  state => "QC", country => "Canada" },
    ],
    info      => {
      roles     => [ "foo", "bar", "baz" ],
    },
    name      => "Alice",
};

print encode_json($data) ;

Output

{
  "_id": 123,
  "info": { "roles": [ "foo", "bar", "baz" ] },
  "locations": [
    { "city": "Boston",   "country": "USA",    "state": "MA" },
    { "city": "Seattle",  "country": "USA",    "state": "WA" },
    { "city": "Montreal", "country": "Canada", "state": "QC" }
  ],
  "name": "Alice"
}

Example 2 - Packing

Traditional pretty-printing:

{
  "states": [
    "Alabama",
    "Alaska",
    "Arizona",
    ...
    "Wyoming"
  ]
}

JSONFold:

{
  "states": [
    "Alabama", "Alaska", "Arizona", "Arkansas", "California", "Colorado",
    "Connecticut", "Delaware", "Florida", "Georgia", ...
    "West_Virginia", "Wisconsin", "Wyoming"
  ]
}

Same data, just using the available line width more effectively.

Example 3 - Grid Formatting

When an array contains repeated structures, JSONFold can align values into columns:

Traditional pretty-printing:

[
  {
    "orders": 18,
    "product": "Laptop",
    "region": "North",
    "sales": 1250
  },
  ...
  {
    "orders": 24,
    "product": "Mouse",
    "region": "East",
    "sales": 1422
  }
]

JSONFold:

[
  { "orders": 18, "product": "Laptop",   "region": "North",     "sales": 1250 },
  { "orders": 21, "product": "Monitor",  "region": "Southwest", "sales": 1345 },
  { "orders": 17, "product": "Keyboard", "region": "West",      "sales": 1198 },
  { "orders": 24, "product": "Mouse",    "region": "East",      "sales": 1422 }
]

The module also supports:

  • Folding small arrays and objects onto a single line.
  • Joining adjacent folded objects to further reduce vertical space.
  • Compatibility APIs similar to JSON and JSON::PP.

I wrote a more detailed article covering the design, implementation, and full set of examples:

Medium: https://medium.com/p/a619c9e7c3ec

CPAN: https://metacpan.org/pod/JSON::JSONFold

GitHub: https://github.com/yairlenga/jsonfold/tree/main/perl

I'd love to hear what the Perl community thinks. Has anyone else run into JSON pretty-printing pain in logs, configs, or debugging output? And are there formatting styles or options you'd want to see?

u/Yairlenga — 6 days ago

jsonfold: Making Pretty-Printed JSON Compact and Readable

jsonfold: Making Pretty-Printed JSON Compact and Readable

Most JSON serializers give you only two choices:

  • compact machine output:{"a":{"b":{"c":"abc"}},"x":{"y":{"z":"xyz"}}}
  • or fully expanded “pretty-print”:{ "a": { "b": { "c": "abc" } }, "x": { "y": { "z": "xyz" } } }

I wanted something in between: the first is hard for humans to scan, and the second becomes extremely verbose on real-world nested data.

I experimented with a small Python formatter jsonfold that can selectively:

  • Pack lists of scalars/simple objects
  • Fold small containers back onto a single line
  • Merge multiple small containers onto a single line

One interesting implementation detail is that it works as a streaming wrapper around json.dump() output rather than reparsing JSON or building another JSON tree.

json.dump(obj, JSONFoldWriter(fp), indent=2)

So it works with fixed memory usage and linear processing time even for large documents.

Minimal Usage

Pull jsonfold.py from GitHub project

import jsonfold
import sys
data = {
    "meta": {"version": 1, "ok": True},
    "ids": [1, 2, 3, 4, 5],
    "items": [{"id": 1, "name": "alpha"}, {"id": 2, "name": "beta"}],
}
# compact can be: default, low, med, high, max
jsonfold.dump(data, sys.stdout, compact="default")

References:

Disclosure: I am the developer of jsonfold

Repository: https://github.com/yairlenga/jsonfold

Python implementation is under python directory.

Article with implementation details: Medium (no paywall): A Streaming JSON Formatter That Works With Existing Serializers

reddit.com
u/Yairlenga — 1 month ago
▲ 1 r/lowlevel+2 crossposts

I wrote about automatic enum stringifcation in C, using build-time code generation from DWARF debug info.

No manual lookup tables to build or maintain, no complex macros - just compile, extract and link.

The final binary contains plain C data structures with zero runtime dependency on DWARF libraries, or tools.

enum country_code {
    ISO3_AFG = 4,    /* Afghanistan */
    ISO3_ALB = 8,    /* Albania */
    ISO3_ATA = 10,   /* Antarctica */
    ISO3_DZA = 12,   /* Algeria */
    ...
} ;

ENUM_DESCRIBE(country3, country_code)

void foo(enum country_code c)
{
    printf("Called with C=%s\n", ENUM_LABEL_OF(country3, c)) ;
}
u/Yairlenga — 2 months ago
▲ 8 r/lowlevel+1 crossposts

I've been working on an article to describe a small performance issues with a pattern I've seen multiple times - long chain of if statements based on strcmp. This is the equivalent of switch/case on string (which is not supported in C).

bool model_ccy_lookup(const char *s, int asof, struct model_param *param)
{
    // Major Currencies
    if ( strcmp(s, "USD") == 0 || strcmp(s, "EUR") == 0 || ...) {
        ...
    // Asia-Core
    } else if ( strcmp(s, "CNY") == 0 || strcmp(s, "HKD") == 0 || ... ) {
        ...
    } else if ( ... ) {
        ...
    } else {
        ...
    }
} 

The code couldn’t be refactored into a different structure (for non-technical reasons), so I had to explore few approaches to keep the existing structure - without rewrite/reshape of the logic. I tried few tings - like memcmp, small filters, and eventually packing the strings into 32-bit values (“4CC”-style) and letting the compiler work with integer compares.

Sharing in the hope that other readers may find the ideas/process useful.

The article is on Medium (no paywall): Optimizing Chained strcmp Calls for Speed and Clarity.

I’m also trying a slightly different writing style than usual - a bit more narrative, focusing on the path (including the dead ends), not just the final result.

If you have a few minutes, I’d really appreciate feedback on two things:

  • Does the technical content hold up?
  • Is the presentation clear, or does it feel too long / indirect?

Interested to hear on other ideas/approach for this problem as well.

u/Yairlenga — 3 months ago

I've been working on a module that needed a lot of temporary memory. It didn't take long for malloc/free to show up as a performance bottleneck. Since the program was running on a modern Linux server (Red Hat, 8MB default stack), I started moving allocation to the stack - (I used VLA (Variable Length array), but it is possible to achieve the same effect with alloca, or similar).

Initially, I used fixed rules - anything below X (I used 128KB) goes to the stack, anything above - goes to the heap. I got a decent speedup as the number of malloc/free went down.

However, I noticed that I was "leaving money on the table". Since over allocation on the stack is usually non-recoverable error (SIGSEGV) - I had to be very conservative in what was placed into the stack.

I was looking for a way to make more efficient use of stack allocation. On the surface, there is no API "getRemainingStackSpace()" in Linux, standard C library, or glibc extensions. After some research, I identified a few options:

  • pthread_getattr_np → actual stack base + size for the current thread
  • getrlimit(RLIMIT_STACK) + initial stack position → rough estimate
  • Leveraging gcc/clang constructor attributes to measure stack at startup.
  • Worth mentioning /proc/self/maps provides similar information - but does not qualify as "API", so I chose not to use it.

None is perfect, but together they provide enough information for safe decision making, allowing me to write code like:

bool do_something(...)
{
    size_t need_temp = ... ;
    bool use_stack = stack_remaining() > need_temp ;
    char temp_stack[use_stack ? need_temp : 1] ;
    void *temp = use_stack ? temp_stack : malloc(need_temp) ;

    // Use temporary space via temp->

    // If malloc was used, call free
    if ( !use_stack ) free(temp) ;    
    return ... ;
}

And my stack_remaining function is:

size_t stack_remaining(void)
{
    StackAddr sp = stack_marker_addr() ;
    if ( sp < stack_low_mark ) stack_low_mark = sp ;
    return sp - stack_base - safety_margin;    
}

The article provides more details, including sample implementation, available as single file self-contained sample C program (<100 lines) that can be dropped into any project.

The solution worked for my use case, but I am curious what other developers are doing to solve similar problems:

  • Are there any API/system calls that can help with estimating available stack space ?
  • Are there other approaches to solve the specific problem of large temporary buffers ?

Disclaimers:

  • Assuming downward-growing stack, as X86/x86_64/ARM.
  • Solution tested on GCC/CLANG.
medium.com
u/Yairlenga — 3 months ago