chore(logging): demote noisy per-region/per-tenant info logs to debug#2702
Open
chore(logging): demote noisy per-region/per-tenant info logs to debug#2702
Conversation
Demote step-level fetch/transform/loaded logs to DEBUG across cartography intel modules and reserve INFO for sync-level start/complete and per-account recap messages. Removes redundant DynamoDB "Loading ... into graph" logs that duplicate the message already emitted by client/core/tx.py. Signed-off-by: Jeremy Chapeau <jeremy@subimage.io>
Signed-off-by: Jeremy Chapeau <jeremy@subimage.io>
… logs The logger calls demoted to debug in this PR were using f-strings, which format the message even when the log level filters it out. Convert them to lazy %-formatting so the message is only built when the record is emitted. Signed-off-by: Jeremy Chapeau <jeremy@subimage.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type of change
Summary
Reduce log volume in cartography by reserving
INFOfor sync-level start/complete and per-account recap messages, and demoting per-region / per-tenant step logs (fetch,get,transform,retrieved, ...) toDEBUG. TheAGENTS.mdanddocs/root/agents/create-module.mdexamples already document this convention; this PR brings the rest of the codebase in line.Highlights:
cartography/graph/job.py— "Finished job" → DEBUG (fired ~620k/day across cleanup/relationship sub-jobs).cartography/intel/aws/dynamodb.py— removed the 8 "Loading DynamoDB ... into graph" calls; they duplicate the aggregate message already emitted bycartography/client/core/tx.py.cartography/intel/aws/cloudtrail_management_events.py— per-regionProcessing/Fetching/Retrieved/Transforming/Loadedcalls → DEBUG. The per-account aggregates ("Syncing N regions ...", "... sync completed successfully ...") stay at INFO.cartography/intel/gcp/vertex/utils.py,cartography/intel/gcp/cloudrun/util.py— per (resource × region × project) "Found ..." → DEBUG (per-project recap logs stay INFO).cartography/intel/aws/sagemaker/util.py— per-(submodule × region × account) "Syncing ..." → DEBUG.cartography/util.py, allFetching/Fetched/Retrieved/Getting/Transformingstep logs → DEBUG.docs/root/dev/writing-intel-modules.md— removed the redundant "Loading EMR clusters ... into graph" line from theload_emr_clustersexample so docs reflect the convention.Related issues or links
How was this tested?
cartography/tree (python3 -c \"import ast, pathlib; [ast.parse(p.read_text()) for p in pathlib.Path('cartography').rglob('*.py')]\").make lintlocally.logger.infois a sync-level start/complete or per-account recap message.Checklist
General
make lint).Proof of functionality
This PR only changes log levels and removes redundant log statements; no graph behavior changes, so no new tests or screenshots are warranted.
Notes for reviewers
Syncing N regions with X hour lookback period,... sync completed successfully ...).cartography/intel/aws/emr.py:80still has a "Loading EMR ... into graph" line equivalent to the ones removed from DynamoDB; left intact in this PR to keep scope tight, but it could be removed as a follow-up sinceclient/core/tx.py:830already logs the same information.