[BUG] parseMap() doesn't handle arrays recursively, causing JSON marshal errors

[tykid16]

Summary

The parseMap() function doesn't recursively process arrays, leaving
map[interface{}]interface{}
inside arrays unconverted. This causes JSON marshaling to fail with nested data structures.

Environment

• Plugin version: latest (main branch)
• Go version: 1.23.0
• Error: json: unsupported type: map[interface {}]interface {}

Problem Description

When processing data with nested arrays containing objects, the plugin fails with a JSON marshal
error.

Example data that causes the error:

{
  "request": {
    "imp": [{
      "banner": {"w": 300, "h": 250}
    }]
  }
}

Root Cause

The current parseMap() implementation:

func parseMap(m0 map[interface{}]interface{}) map[string]interface{} {
    m1 := make(map[string]interface{})
    for k, v := range m0 {
        switch t := v.(type) {
        case []byte:
            m1[k.(string)] = string(t)
        case map[interface{}]interface{}:
            m1[k.(string)] = parseMap(t)
        default:  // Arrays pass through without processing
            m1[k.(string)] = v
        }
    }
    return m1
}

The issue: Arrays ([]interface{}) containing map[interface{}]interface{} are not processed,
causing json.Marshal() to fail when it encounters these unprocessed maps.

Proposed Fix

Add recursive array processing:

func parseMap(m0 map[interface{}]interface{}) map[string]interface{} {
    m1 := make(map[string]interface{})
    for k, v := range m0 {
        switch t := v.(type) {
        case []byte:
            m1[k.(string)] = string(t)
        case map[interface{}]interface{}:
            m1[k.(string)] = parseMap(t)
        case []interface{}:  // Add array handling
            m1[k.(string)] = parseArray(t)
        default:
            m1[k.(string)] = v
        }
    }
    return m1
}

func parseArray(a0 []interface{}) []interface{} {
    a1 := make([]interface{}, len(a0))
    for i, v := range a0 {
        switch t := v.(type) {
        case map[interface{}]interface{}:
            a1[i] = parseMap(t)
        case []interface{}:
            a1[i] = parseArray(t)
        default:
            a1[i] = v
        }
    }
    return a1
}

Impact

• Any data with arrays containing objects fails to write to BigQuery
• Common with OpenRTB bid request data and similar nested structures

I'm happy to submit a PR with this fix if you agree with the approach.

[tykid16]

Hi @agrawal-siddharth,

Thank you for maintaining this valuable plugin. I've encountered this issue in our production environment and would like to contribute a fix.

Additional Context for Reproduction

To help reproduce this issue, I've prepared a minimal test case:

1. Minimal Go Test to Demonstrate the Issue

// test_parseMap.go
package main

import (
    "encoding/json"
    "fmt"
)

// Current parseMap implementation from out_writeapi.go
func parseMap(mapInterface map[interface{}]interface{}) map[string]interface{} {
    m := make(map[string]interface{})
    for k, v := range mapInterface {
        switch t := v.(type) {
        case []byte:
            m[k.(string)] = string(t)
        case map[interface{}]interface{}:
            m[k.(string)] = parseMap(t)
        default:
            m[k.(string)] = v  // ← Arrays are not processed recursively
        }
    }
    return m
}

func main() {
    // Typical data structure from Fluent Bit (via MessagePack)
    data := map[interface{}]interface{}{
        "event_params": []interface{}{
            map[interface{}]interface{}{  // ← This map inside array causes the error
                "key": "campaign_id",
                "value": map[interface{}]interface{}{
                    "string_value": "273",
                },
            },
        },
    }

    result := parseMap(data)
    _, err := json.Marshal(result)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        // Output: Error: json: unsupported type: map[interface {}]interface {}
    }
}

2. Real-world Impact

This issue affects common BigQuery schemas including:

• OpenRTB bid request data (imp array with nested objects)
• Application logs with nested structures

3. Proposed Solution

I've prepared a fix that adds recursive array processing:

  func parseArray(arr []interface{}) []interface{} {
      result := make([]interface{}, len(arr))
      for i, v := range arr {
          switch t := v.(type) {
          case map[interface{}]interface{}:
              result[i] = parseMap(t)  // Recursively process maps in arrays
          case []interface{}:
              result[i] = parseArray(t)  // Handle nested arrays
          default:
              result[i] = v
          }
      }
      return result
  }

Would you prefer:

1. A PR with just the minimal fix to parseMap()?
2. A more comprehensive solution with additional test cases?
3. Any specific testing requirements or benchmarks?

I'm happy to adjust the implementation based on your preferences. We're currently working around this issue with a Lua filter that pre-processes the data, but a fix in the plugin
would be much more efficient.

Thank you for your time and for maintaining this important integration!

[agrawal-siddharth]

Hi, thanks for the detailed explanation and repro. I could verify your changes fix the new test case. In parseArray, I recommend also adding a case for:

case []byte: result[i] = string(t)

Are there additional key scenarios that are still missing in this logic? I am fine with adding those in future revisions.

It would be great if you could define a unit test for parseMap that validates these test cases.