NDJSON / JSON Lines: The Format for Streaming and Logs

The Hidden Gem of Data Formats: NDJSON

Have you ever struggled with parsing large JSON files or dealing with the limitations of traditional JSON arrays? You're not alone. We've all been there, and that's where NDJSON comes to the rescue.

Table of Contents

• What is NDJSON and how does it differ from JSON arrays?
• When to use NDJSON vs JSON arrays
• Parsing NDJSON in JavaScript, Python, and Go
• Real-world uses: logging and beyond
• Key Takeaways
• FAQ

What is NDJSON and how does it differ from JSON arrays?

NDJSON, also known as JSON Lines or JSONL, is a lightweight data format that stores each JSON object on a separate line. This simple yet powerful format has been around since 2010, but it's still underutilized in many development projects.

Unlike traditional JSON arrays, where multiple objects are stored in a single array, NDJSON stores each object as a separate line, making it easier to read, write, and parse large datasets. This format is particularly useful when dealing with streaming data, logs, or large datasets that don't fit into memory.

Here's an example of NDJSON:

{"name": "John", "age": 30}
{"name": "Jane", "age": 25}
{"name": "Bob", "age": 40}

When to use NDJSON vs JSON arrays

So, when should you use NDJSON over traditional JSON arrays? Here are some scenarios:

• Streaming data: When working with streaming data, NDJSON allows you to process each object individually, without having to load the entire dataset into memory.
• Large datasets: NDJSON is perfect for large datasets that don't fit into memory. You can process each object one by one, without having to worry about running out of memory.
• Logging: NDJSON is widely used in logging applications, where each log entry is stored as a separate JSON object.

On the other hand, JSON arrays are still a good choice when:

• Data is small: If your dataset is small and fits into memory, JSON arrays are a more concise and efficient choice.
• Data is highly structured: If your data has a fixed structure and you need to query it frequently, JSON arrays might be a better choice.

Parsing NDJSON in JavaScript, Python, and Go

Parsing NDJSON is relatively straightforward in most programming languages. Here are some examples:

JavaScript

const fs = require('fs');
const ndjson = fs.readFileSync('data.ndjson', 'utf8');
const objects = ndjson.split('\n').map(JSON.parse);
console.log(objects);

Python

import json

with open('data.ndjson', 'r') as f:
    objects = [json.loads(line) for line in f]
print(objects)

Go

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
)

func main() {
	data, err := ioutil.ReadFile("data.ndjson")
	if err != nil {
		fmt.Println(err)
		return
	}
	objects := []map[string]interface{}{}
	for _, line := range strings.Split(string(data), "\n") {
		var obj map[string]interface{}
		err = json.Unmarshal([]byte(line), &obj)
		if err != nil {
			fmt.Println(err)
			return
		}
		objects = append(objects, obj)
	}
	fmt.Println(objects)
}

Real-world uses: logging and beyond

NDJSON is widely used in logging applications, such as Apache Kafka, Logstash, and ELK Stack. It's also used in data processing pipelines, such as Apache Beam and AWS Glue.

Key Takeaways

• NDJSON is a lightweight data format that stores each JSON object on a separate line.
• Use NDJSON for streaming data, large datasets, and logging applications.
• Use JSON arrays for small datasets and highly structured data.
• NDJSON is widely supported in most programming languages.

FAQ

Q: What is the difference between NDJSON and JSONL?

A: NDJSON and JSONL are interchangeable terms that refer to the same data format.

Q: Can I use NDJSON with JSON schema validation?

A: Yes, you can use NDJSON with JSON schema validation. Each object can be validated against a schema before processing.

Q: Is NDJSON compatible with all programming languages?

A: Most programming languages support NDJSON, but some may require additional libraries or parsing logic.