Huge memory consumption when multiple scripts source their peers "recursively"

[pmhahn]

For bugs with existing features

• My shellcheck version (shellcheck --version or "online"): 0.9.0 from Ubuntu 24.04 LTS
• The rule's wiki page does not already cover this (e.g. https://www.shellcheck.net/wiki/SC1090)
• I tried https://github.com/koalaman/shellcheck/releases/download/latest/shellcheck-latest.linux.x86_64.tar.xz and verified that this is still a problem on the latest commit

Here's a snippet or screenshot that shows the problem:

The Linux kernel contains some multiple test scripts written in shell, which source their peer scripts "recursively"; see
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/cpufreq?h=v6.19.11

• cpu.sh

  # protect against multiple inclusion
  if [ $FILE_CPU ]; then
  	return 0
  else
  	FILE_CPU=DONE
  fi
  source cpufreq.sh

• cpufreq.sh

  # protect against multiple inclusion
  if [ $FILE_CPUFREQ ]; then
  	return 0
  else
  	FILE_CPUFREQ=DONE
  fi
  source cpu.sh

• main.sh is the top-level script, which includes both above files plus others, which are irrelevant here.

Here's what shellcheck currently says:

Running shellcheck cpu.sh takes a long time and consumes 18 GiB of RAM:

# /usr/bin/time --quiet -f '%M\t%C' shellcheck -x -P SCRIPTDIR -a -f quiet tools/testing/selftests/cpufreq/cpu.sh
18_38_860 shellcheck -x -P SCRIPTDIR -a -f quiet tools/testing/selftests/cpufreq/cpu.sh

(time prints KiB)

Using strace I can see, that the files are opened multiple times:

# /usr/bin/strace -e t=openat -e s=none shellcheck -x -P SCRIPTDIR -a -f quiet tools/testing/selftests/cpufreq/cpu.sh | sort | uniq -c
      1 +++ exited with 1 +++
      1 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
      1 openat(AT_FDCWD, "/home/phahn/.config/shellcheckrc", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
      1 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
      1 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libffi.so.8", O_RDONLY|O_CLOEXEC) = 3
      1 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgmp.so.10", O_RDONLY|O_CLOEXEC) = 3
      1 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
     50 openat(AT_FDCWD, "tools/testing/selftests/cpufreq/cpufreq.sh", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
     51 openat(AT_FDCWD, "tools/testing/selftests/cpufreq/cpu.sh", O_RDONLY|O_NOCTTY|O_NONBLOCK) = 3
      1 openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3

(for ./shellcheck-latest/shellcheck --version v0.11.0-69-g766a836 you have to use strace -e t=open … instead of openat)

Here's what I wanted or expected to see:

• shellcheck should detect, that the same files are sources multiple times and break the loop earlier.
• shellcheck should not consume 18 GiB for checking

Currently I maintain a growing list of files, which must be ignored by shellcheck, e.g. those problematic Linux kernel files and {auto{conf,make},libtoolize} generated/provided files. It would help, if there would be command-line-options / shellcheckrc-directives to

• ignore certain file patterns
• limit the file size of files to check
• …

Maybe related issues

There are several other reports, where shellcheck takes a large amount of memory

• shellcheck runs out of memory and gets killed on qemu/docker #2428
• ShellCheck busy loops and allocates memory until it is killed by OOM killer #2721
• Shellcheck v0.8.0 runs out of memory on linux OS for big sh files #2579
• source= directive on 4000 LOC drains 64GB RAM and trigger OOM #3177
• since bash-language server update earlier today shellcheck appears to be getting stuck in a loop and eating up memory #2801

Regarding -x there is this issues

• why shellcheck -x very slow? #1063

Disclaimer

I'm not the author of the above mention Linux kernel files, but I'm required to use shellcheck for my other Linux kernel related work. As such we're running shellcheck as part of our GitLab-CI pipeline, which then also checks those files and currently fails, as the GitLab runner fails with an out-of-memory error as shellcheck runs inside a Kubernetes cluster, which limits its memory to 8 GiB maximum.

[e-kwsm]

Your snippets give SC1093 (info): This file appears to be recursively sourced. Ignoring.
But the files in https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/cpufreq?h=v6.19.11 exhaust memory, w/o SC1093.