How to Flatten nested JSON in Go

How to flatten nested JSON in Go

Flattening nested JSON is a common task when working with JSON data in Go. It involves transforming a nested JSON object into a single-level JSON object with dot notation keys. This is useful when working with data that needs to be processed or stored in a flat format, such as in a database or data warehouse. In this article, we will explore how to flatten nested JSON in Go, covering the basics, edge cases, common mistakes, and performance tips.

Quick Example

Here is a minimal example that flattens a nested JSON object:

package main

import (
	"encoding/json"
	"fmt"
)

func flattenJSON(data map[string]interface{}) map[string]interface{} {
	result := make(map[string]interface{})
	for k, v := range data {
		if v, ok := v.(map[string]interface{}); ok {
			flattened := flattenJSON(v)
			for subk, subv := range flattened {
				result[fmt.Sprintf("%s.%s", k, subk)] = subv
			}
		} else {
			result[k] = v
		}
	}
	return result
}

func main() {
	jsonData := `{"name":"John","address":{"street":"123 Main St","city":"Anytown"}}`
	var data map[string]interface{}
	json.Unmarshal([]byte(jsonData), &data)
	flattened := flattenJSON(data)
	fmt.Println(flattened)
}

This code defines a recursive flattenJSON function that takes a map[string]interface{} as input and returns a flattened map[string]interface{}. The main function demonstrates how to use this function with a sample JSON string.

Step-by-Step Breakdown

Let's walk through the code line by line:

func flattenJSON(data map[string]interface{}) map[string]interface{}: This defines the flattenJSON function with a single argument data of type map[string]interface{} and returns a map[string]interface{}.
result := make(map[string]interface{}): This initializes an empty map[string]interface{} to store the flattened result.
for k, v := range data { ... }: This loops through each key-value pair in the input data map.
if v, ok := v.(map[string]interface{}); ok { ... }: This checks if the value v is a map[string]interface{} using a type assertion. If true, it recursively calls flattenJSON on the nested map.
flattened := flattenJSON(v): This calls the flattenJSON function recursively on the nested map v.
for subk, subv := range flattened { ... }: This loops through each key-value pair in the flattened result.
result[fmt.Sprintf("%s.%s", k, subk)] = subv: This constructs a new key by concatenating the current key k with the sub-key subk using dot notation and assigns the corresponding value subv to the result map.
} else { result[k] = v }: If the value v is not a map[string]interface{}, it simply assigns the value to the result map with the original key k.

Handling Edge Cases

Here are some common edge cases to consider:

Empty/null input

If the input JSON is empty or null, the function should return an empty map.

func main() {
	jsonData := `{}`
	var data map[string]interface{}
	json.Unmarshal([]byte(jsonData), &data)
	flattened := flattenJSON(data)
	fmt.Println(flattened) // Output: map[]
}

Invalid input

If the input JSON is invalid, the json.Unmarshal function will return an error. We should handle this error accordingly.

func main() {
	jsonData := ` invalid json `
	var data map[string]interface{}
	err := json.Unmarshal([]byte(jsonData), &data)
	if err != nil {
		fmt.Println(err) // Output: invalid character 'i' looking for beginning of value
		return
	}
	flattened := flattenJSON(data)
	fmt.Println(flattened)
}

Large input

For large input JSON, we should consider using a streaming JSON parser to avoid loading the entire JSON into memory.

func main() {
	jsonData := `{"name":"John","address":{"street":"123 Main St","city":"Anytown"}}`
	jsonDecoder := json.NewDecoder(bytes.NewBuffer([]byte(jsonData)))
	var data map[string]interface{}
	err := jsonDecoder.Decode(&data)
	if err != nil {
		fmt.Println(err)
		return
	}
	flattened := flattenJSON(data)
	fmt.Println(flattened)
}

Unicode/special characters

The function should handle Unicode and special characters correctly.

func main() {
	jsonData := `{"name":"Jöhn","address":{"street":"123 Main St","city":"Anytown"}}`
	var data map[string]interface{}
	json.Unmarshal([]byte(jsonData), &data)
	flattened := flattenJSON(data)
	fmt.Println(flattened) // Output: map[name:Jöhn address.street:123 Main St address.city:Anytown]
}

Common Mistakes

Here are some common mistakes developers make when flattening JSON in Go:

1. Not handling recursive structures

func flattenJSON(data map[string]interface{}) map[string]interface{} {
	result := make(map[string]interface{})
	for k, v := range data {
		result[k] = v // WRONG: does not handle recursive structures
	}
	return result
}

Corrected code:

func flattenJSON(data map[string]interface{}) map[string]interface{} {
	result := make(map[string]interface{})
	for k, v := range data {
		if v, ok := v.(map[string]interface{}); ok {
			flattened := flattenJSON(v)
			for subk, subv := range flattened {
				result[fmt.Sprintf("%s.%s", k, subk)] = subv
			}
		} else {
			result[k] = v
		}
	}
	return result
}

2. Not handling invalid input

func main() {
	jsonData := ` invalid json `
	var data map[string]interface{}
	json.Unmarshal([]byte(jsonData), &data)
	flattened := flattenJSON(data) // WRONG: ignores error
	fmt.Println(flattened)
}

Corrected code:

func main() {
	jsonData := ` invalid json `
	var data map[string]interface{}
	err := json.Unmarshal([]byte(jsonData), &data)
	if err != nil {
		fmt.Println(err) // Output: invalid character 'i' looking for beginning of value
		return
	}
	flattened := flattenJSON(data)
	fmt.Println(flattened)
}

3. Not handling large input

func main() {
	jsonData := ` large json data `
	var data map[string]interface{}
	json.Unmarshal([]byte(jsonData), &data) // WRONG: loads entire JSON into memory
	flattened := flattenJSON(data)
	fmt.Println(flattened)
}

Corrected code:

func main() {
	jsonData := ` large json data `
	jsonDecoder := json.NewDecoder(bytes.NewBuffer([]byte(jsonData)))
	var data map[string]interface{}
	err := jsonDecoder.Decode(&data)
	if err != nil {
		fmt.Println(err)
		return
	}
	flattened := flattenJSON(data)
	fmt.Println(flattened)
}

Performance Tips

Here are some performance tips when flattening JSON in Go:

1. Use a streaming JSON parser

Instead of loading the entire JSON into memory, use a streaming JSON parser to process the JSON data in chunks.

jsonDecoder := json.NewDecoder(bytes.NewBuffer([]byte(jsonData)))

2. Avoid unnecessary allocations

Minimize unnecessary allocations by reusing existing maps and slices.

result := make(map[string]interface{})

3. Use efficient data structures

Use efficient data structures such as map[string]interface{} instead of interface{} to reduce memory allocation and garbage collection.

result := make(map[string]interface{})

FAQ

Q: How do I handle nested JSON arrays?

A: You can handle nested JSON arrays by using a recursive approach similar to the one used for objects.

Q: How do I handle JSON with duplicate keys?

A: You can handle JSON with duplicate keys by using a map[string]interface{} to store the values and overwriting any existing values.

Q: How do I handle JSON with null values?

A: You can handle JSON with null values by checking for nil values and handling them accordingly.

Q: How do I handle JSON with Unicode characters?

A: You can handle JSON with Unicode characters by using the encoding/json package which supports Unicode characters.

Q: How do I handle large JSON data?

A: You can handle large JSON data by using a streaming JSON parser to process the JSON data in chunks.

How to Flatten nested JSON in Go