Skip to content

chore(logging): demote noisy per-region/per-tenant info logs to debug#2702

Open
jychp wants to merge 3 commits intomasterfrom
jychp/trusting-mestorf-1313fb
Open

chore(logging): demote noisy per-region/per-tenant info logs to debug#2702
jychp wants to merge 3 commits intomasterfrom
jychp/trusting-mestorf-1313fb

Conversation

@jychp
Copy link
Copy Markdown
Collaborator

@jychp jychp commented Apr 27, 2026

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring (no functional changes)
  • Documentation update
  • Other (please describe):

Summary

Reduce log volume in cartography by reserving INFO for sync-level start/complete and per-account recap messages, and demoting per-region / per-tenant step logs (fetch, get, transform, retrieved, ...) to DEBUG. The AGENTS.md and docs/root/agents/create-module.md examples already document this convention; this PR brings the rest of the codebase in line.

Highlights:

  • cartography/graph/job.py — "Finished job" → DEBUG (fired ~620k/day across cleanup/relationship sub-jobs).
  • cartography/intel/aws/dynamodb.py — removed the 8 "Loading DynamoDB ... into graph" calls; they duplicate the aggregate message already emitted by cartography/client/core/tx.py.
  • cartography/intel/aws/cloudtrail_management_events.py — per-region Processing/Fetching/Retrieved/Transforming/Loaded calls → DEBUG. The per-account aggregates ("Syncing N regions ...", "... sync completed successfully ...") stay at INFO.
  • cartography/intel/gcp/vertex/utils.py, cartography/intel/gcp/cloudrun/util.py — per (resource × region × project) "Found ..." → DEBUG (per-project recap logs stay INFO).
  • cartography/intel/aws/sagemaker/util.py — per-(submodule × region × account) "Syncing ..." → DEBUG.
  • Sweep: across AWS Bedrock, ECR, CloudFront, IdentityCenter, GitHub, GitLab, Microsoft Entra, Spacelift, SentinelOne, Duo, Ubuntu, Workday, WorkOS, Semgrep, JumpCloud, CVE EPSS, and cartography/util.py, all Fetching/Fetched/Retrieved/Getting/Transforming step logs → DEBUG.
  • docs/root/dev/writing-intel-modules.md — removed the redundant "Loading EMR clusters ... into graph" line from the load_emr_clusters example so docs reflect the convention.

Related issues or links

How was this tested?

  • Syntax check across the whole cartography/ tree (python3 -c \"import ast, pathlib; [ast.parse(p.read_text()) for p in pathlib.Path('cartography').rglob('*.py')]\").
  • make lint locally.
  • Manual review of the diff: every demoted call is a per-region / per-tenant step log; every remaining logger.info is a sync-level start/complete or per-account recap message.

Checklist

General

  • I have read the contributing guidelines.
  • The linter passes locally (make lint).
  • I have added/updated tests that prove my fix is effective or my feature works.

Proof of functionality

  • Screenshot showing the graph before and after changes.
  • New or updated unit/integration tests.

This PR only changes log levels and removes redundant log statements; no graph behavior changes, so no new tests or screenshots are warranted.

Notes for reviewers

  • Per-account aggregate INFO logs are intentionally preserved (e.g. CloudTrail Syncing N regions with X hour lookback period, ... sync completed successfully ...).
  • cartography/intel/aws/emr.py:80 still has a "Loading EMR ... into graph" line equivalent to the ones removed from DynamoDB; left intact in this PR to keep scope tight, but it could be removed as a follow-up since client/core/tx.py:830 already logs the same information.

Demote step-level fetch/transform/loaded logs to DEBUG across cartography
intel modules and reserve INFO for sync-level start/complete and per-account
recap messages. Removes redundant DynamoDB "Loading ... into graph" logs that
duplicate the message already emitted by client/core/tx.py.

Signed-off-by: Jeremy Chapeau <jeremy@subimage.io>
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 50 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

jychp added 2 commits April 27, 2026 12:47
Signed-off-by: Jeremy Chapeau <jeremy@subimage.io>
… logs

The logger calls demoted to debug in this PR were using f-strings, which
format the message even when the log level filters it out. Convert them
to lazy %-formatting so the message is only built when the record is
emitted.

Signed-off-by: Jeremy Chapeau <jeremy@subimage.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant