Skip to content

cleanup(libsinsp): optimize dynamic_table storage#2923

Open
gnosek wants to merge 19 commits intofalcosecurity:masterfrom
gnosek:table-cleanup-remove-typeinfo
Open

cleanup(libsinsp): optimize dynamic_table storage#2923
gnosek wants to merge 19 commits intofalcosecurity:masterfrom
gnosek:table-cleanup-remove-typeinfo

Conversation

@gnosek
Copy link
Copy Markdown
Contributor

@gnosek gnosek commented Mar 31, 2026

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind test

/kind feature

/kind sync

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area API-version

/area build

/area CI

/area driver-kmod

/area driver-modern-bpf

/area libscap-engine-kmod

/area libscap-engine-modern-bpf

/area libscap-engine-nodriver

/area libscap-engine-noop

/area libscap-engine-source-plugin

/area libscap-engine-savefile

/area libscap

/area libpman

/area libsinsp

/area tests

/area proposals

Does this PR require a change in the driver versions?

/version driver-API-version-major

/version driver-API-version-minor

/version driver-API-version-patch

/version driver-SCHEMA-version-major

/version driver-SCHEMA-version-minor

/version driver-SCHEMA-version-patch

What this PR does / why we need it:

This PR does two things. Sadly, they're intertwined enough that it's simpler to submit them in the same PR.

The things are:

  1. Store dynamic field values inline, as a dynamic_field_value, which is little more than a ss_plugin_state_data (used directly to interact with the table API) and a ss_plugin_state_type marker to indicate which variant of the data is currently active. This also introduces borrowed_state_data, which is going to be used a lot more in upcoming PRs as the representation of data being exposed to the table API by the built in tables.

This avoids an allocation for every single dynamic field value (which is pointer-sized anyway; strings still get their own allocation for the actual data).

  1. Remove the type_info class. This has a few uses:
  • knowing the SS_PLUGIN_ST_* value for a particular type. This can be replaced with the type_id directly
  • knowing the type name. This becomes a free function, which is only used to generate error messages anyway (so a bit of overhead, runtime or mental, is acceptable here)
  • knowing the layout (size, constructor, destructor) for dynamic field allocation. This goes away when we introduce dynamic_field_value.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

gnosek added 19 commits March 31, 2026 10:47
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
This is a constexpr variant of typeinfo::of which only returns
the type_id, not the whole typeinfo object

Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
Signed-off-by: Grzegorz Nosek <grzegorz.nosek@sysdig.com>
@poiana
Copy link
Copy Markdown
Contributor

poiana commented Mar 31, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gnosek

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions
Copy link
Copy Markdown

Perf diff from master - unit tests

    13.65%     +1.32%  [.] std::__shared_count<(__gnu_cxx::_Lock_policy)2>::_M_get_use_count() const
    10.64%     -1.14%  [.] std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_add_ref_lock_nothrow()
     9.22%     +1.05%  [.] sinsp_threadinfo::update_main_fdtable()
     4.08%     -0.80%  [.] sinsp_thread_manager::create_thread_dependencies(std::shared_ptr<sinsp_threadinfo> const&)
    15.29%     +0.42%  [.] std::__shared_ptr<sinsp_threadinfo, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__weak_ptr<sinsp_threadinfo, (__gnu_cxx::_Lock_policy)2> const&, std::nothrow_t)
     6.94%     -0.41%  [.] std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count(std::__weak_count<(__gnu_cxx::_Lock_policy)2> const&, std::nothrow_t)
     8.08%     -0.28%  [.] sinsp_threadinfo::get_fd_table()
    11.50%     +0.25%  [.] sinsp_threadinfo::get_main_thread()
     4.47%     +0.10%  [.] thread_group_info::get_first_thread() const
     0.12%     -0.10%  [.] strlen@plt

Heap diff from master - unit tests

peak heap memory consumption: 1.28M
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Heap diff from master - scap file

peak heap memory consumption: 34.34K
peak RSS (including heaptrack overhead): 0B
total memory leaked: 0B

Benchmarks diff from master

Comparing gbench_data.json to /root/actions-runner/_work/libs/libs/build/gbench_data.json
Benchmark                                                         Time             CPU      Time Old      Time New       CPU Old       CPU New
----------------------------------------------------------------------------------------------------------------------------------------------
BM_sinsp_split_mean                                            +0.0177         +0.0178           240           245           240           244
BM_sinsp_split_median                                          +0.0179         +0.0182           240           245           240           244
BM_sinsp_split_stddev                                          +0.2274         +0.2408             2             2             2             2
BM_sinsp_split_cv                                              +0.2061         +0.2190             0             0             0             0
BM_sinsp_concatenate_paths_relative_path_mean                  -0.0730         -0.0730            76            70            76            70
BM_sinsp_concatenate_paths_relative_path_median                -0.0745         -0.0745            76            70            76            70
BM_sinsp_concatenate_paths_relative_path_stddev                +0.5811         +0.5774             0             1             0             1
BM_sinsp_concatenate_paths_relative_path_cv                    +0.7056         +0.7017             0             0             0             0
BM_sinsp_concatenate_paths_empty_path_mean                     +0.0038         +0.0038            42            43            42            43
BM_sinsp_concatenate_paths_empty_path_median                   +0.0059         +0.0060            42            43            42            43
BM_sinsp_concatenate_paths_empty_path_stddev                   +1.6513         +1.6474             0             0             0             0
BM_sinsp_concatenate_paths_empty_path_cv                       +1.6413         +1.6373             0             0             0             0
BM_sinsp_concatenate_paths_absolute_path_mean                  -0.0702         -0.0702            75            69            75            69
BM_sinsp_concatenate_paths_absolute_path_median                -0.0708         -0.0709            75            69            75            69
BM_sinsp_concatenate_paths_absolute_path_stddev                -0.0319         -0.0174             0             0             0             0
BM_sinsp_concatenate_paths_absolute_path_cv                    +0.0412         +0.0569             0             0             0             0

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 31, 2026

Codecov Report

❌ Patch coverage is 76.61290% with 58 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.90%. Comparing base (e01d603) to head (30a7ac0).

Files with missing lines Patch % Lines
userspace/libsinsp/state/extensible_struct.h 51.16% 21 Missing ⚠️
userspace/libsinsp/state/borrowed_state_data.h 38.88% 11 Missing ⚠️
userspace/libsinsp/state/table_adapters.h 65.38% 9 Missing ⚠️
userspace/libsinsp/state/type_info.h 56.25% 7 Missing ⚠️
userspace/libsinsp/plugin.h 57.14% 3 Missing ⚠️
userspace/libsinsp/plugin_table_api.cpp 40.00% 3 Missing ⚠️
userspace/libsinsp/examples/test.cpp 0.00% 2 Missing ⚠️
userspace/libsinsp/state/table.cpp 81.81% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2923      +/-   ##
==========================================
- Coverage   74.96%   74.90%   -0.07%     
==========================================
  Files         296      297       +1     
  Lines       31472    31496      +24     
  Branches     4977     4979       +2     
==========================================
- Hits        23594    23593       -1     
- Misses       7878     7903      +25     
Flag Coverage Δ
libsinsp 74.90% <76.61%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

@ekoops ekoops left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvement! Overall looks great, but could you please squash some commits? Like the replace typeinfo with raw ... ones?

Copy link
Copy Markdown
Contributor

@irozzo-1A irozzo-1A left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I spotted a potential issue but I'm not sure it's relevant

Comment on lines +61 to +63
dynamic_field_value& operator=(dynamic_field_value&& rhs) noexcept {
m_type = rhs.m_type;
m_data = rhs.m_data;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the dynamic_field_value already owns a string, we should free it before assigning isn't it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

4 participants