Skip to content

How to rename or copy and rename nested fields? #11324

@Ants5

Description

@Ants5

`

[INPUT]
    Name                tail
    Tag                 charging-us-prod-vmlogs.*
    Path                /var/log/containers/hes-hes-2030*_charging-us-prod_*.log
    Parser              cri
    DB                  /var/fluent-bit/state/charging-us-prod-vmlogs.db
    Mem_Buf_Limit       100MB
    Skip_Long_Lines     On
    Skip_Empty_Lines    On
    Refresh_Interval    5
    Rotate_Wait         20
    storage.type        filesystem
    #Read_from_Head      off

[FILTER]
    Name                kubernetes
    Match               charging-us-prod-vmlogs.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_Tag_Prefix     charging-us-prod-vmlogs.var.log.containers.
    Merge_Log           On
    Keep_Log            On
    namespace_labels    On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Labels              Off
    Annotations         Off
[FILTER]
    Name                nest
    Match               charging-us-prod-vmlogs.*
    Operation           lift
    Nested_under        kubernetes
    Add_prefix          kubernetes.
[FILTER]
    Name                record_modifier
    Match               charging-us-prod-vmlogs.*
    Record              area   usa
    Record              type   test
    Remove_key          kubernetes.pod_id
    Remove_key          kubernetes.pod_ip
    Remove_key          kubernetes.docker_id
    Remove_key          kubernetes.container_hash
    Remove_key          kubernetes.container_image
    #Remove_key          kubernetes.container_name
    #Remove_key          kubernetes.pod_name
    #Remove_key          kubernetes.namespace_name
[FILTER]
    Name                modify
    Match               charging-us-prod-vmlogs.*
    Rename              kubernetes.container_name   service_name
    Rename              kubernetes.namespace_name   namespace
    Rename              kubernetes.pod_name   pod
    Rename              kubernetes.host   nodename
    #Set                 cluster ${kubernetes_namespace.labels.cluster}
    #Set                 cluster ${kubernetes_namespace.labels.project}
    #Rename              kubernetes_namespace.labels.cluster  cluster
    #Rename              kubernetes_namespace.labels.project  project
    #Remove              kubernetes_namespace

#[FILTER]
#    Name                nest
#    Match               charging-us-prod-vmlogs.*
#    Operation           nest
#    Wildcard            kubernetes.*
#    Nest_under          kubernetes
#    Remove_prefix       kubernetes.

[OUTPUT]
    Name                stdout
    Match               charging-us-prod-vmlogs.*
[OUTPUT]
    Name                http
    Match               charging-us-prod-vmlogs.*
    host                localhost
    port                8427
    http_user           aaaaa
    http_passwd         123123
    header              AccountID 0
    header              ProjectID 0
    uri                 /insert/jsonline?_stream_fields=project,cluster,namespace,service_name,type&_msg_field=log
    format              json_lines
    json_date_format    iso8601
    compress            gzip

`

My data collection configuration is shown above. The Kubernetes plugin has namespace_labels enabled to retrieve labels from the namespace, but the retrieved labels are in nested JSON format. How can I rename kubernetes_namespace.labels.cluster to the cluster name without unnesting the tags? My log format is as follows.

charging-us-prod-vmlogs.var.log.containers.hes-hes-2030-795697479c-ps9dc_charging-us-prod_hes-hes-2030-102360dc7fd5eaab6fe5b984d1a742884ed36bb43864328b284e174a3d8da43c.log: [[1767168962.880809955, {}], {"stream"=>"stdout", "logtag"=>"F", "log"=>"{"@timestamp":"2025-12-31T08:16:02.880Z","caller":"util/httputil.go:130","content":"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:\"[Subscription xmlns=\\\"urn:ieee:std:2030.5:ns\\\"]\\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\\n [encoding]0[/encoding]\\n [level]+S1[/level]\\n [limit]0[/limit]\\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_2030_svc/ntfy/synergy/2F266B0212F96DF43FCD4AF2E51B007D00060981[/notificationURI]\\n[/Subscription]\", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165","level":"info"}", "@timestamp"=>"2025-12-31T08:16:02.880Z", "caller"=>"util/httputil.go:130", "content"=>"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:"[Subscription xmlns=\"urn:ieee:std:2030.5:ns\"]\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\n [encoding]0[/encoding]\n [level]+S1[/level]\n [limit]0[/limit]\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_20_svc/ntfy/synergy/2F266B0212F43FCD4AF2EB007D00060981[/notificationURI]\n[/Subscription]", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165", "level"=>"info", "kubernetes_namespace"=>{"name"=>"charging-us-prod", "labels"=>{"cluster"=>"aiot-us-prod", "dynakube.internal.dynatrace.com/instance"=>"aiot-qa", "field.cattle.io/projectId"=>"p-7w2zz", "kubernetes.io/metadata.name"=>"charging-us-prod", "project"=>"charging"}}, "pod"=>"hea-hes-22230-795697479c-ps9dc", "namespace"=>"charging-us-prod", "nodename"=>"ip-11-11-11-153.us-east-2.compute.internal", "service_name"=>"hea-hes-22230", "area"=>"usa", "type"=>"test"}]

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions