NetBox container 4.1.0 (or newer) SSL crypto error on startup

[tjmullicani]

Current Behavior

I get the following error starting NetBox on docker using tag v4.1.0 or newer (running on Kubernetes). Image tag v4.0.11 or older works without issue.

ssl.SSLError: [CRYPTO] unknown error (_ssl.c:3076)

Expected Behavior

Container successfully runs

Docker Compose Version

kubernetes 1.30.5 helm chart version netbox-5.0.0-beta.139

Azure Linux with FIPS enabled

Docker Version

n/a

The git Revision

n/a

The git Status

n/a

Startup Command

n/a

NetBox Logs

Traceback (most recent call last):
  File "/opt/netbox/netbox/./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 382, in execute
    settings.INSTALLED_APPS
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 89, in __getattr__
    self._setup(name)
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 76, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/netbox/netbox/netbox/settings.py", line 10, in <module>
    import requests
  File "/opt/netbox/venv/lib/python3.12/site-packages/requests/__init__.py", line 164, in <module>
    from .api import delete, get, head, options, patch, post, put, request
  File "/opt/netbox/venv/lib/python3.12/site-packages/requests/api.py", line 11, in <module>
    from . import sessions
  File "/opt/netbox/venv/lib/python3.12/site-packages/requests/sessions.py", line 15, in <module>
    from .adapters import HTTPAdapter
  File "/opt/netbox/venv/lib/python3.12/site-packages/requests/adapters.py", line 80, in <module>
    _preloaded_ssl_context = create_urllib3_context()
                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 292, in create_urllib3_context
    context = SSLContext(PROTOCOL_TLS_CLIENT)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ssl.py", line 438, in __new__
    self = _SSLContext.__new__(cls, protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLError: [CRYPTO] unknown error (_ssl.c:3076)
⏳ Waiting on DB... (0s / 30s)

Content of docker-compose.override.yml

n/a

[tjmullicani]

similar error certbot/certbot#10044

[Pyromonkey83]

Have a similar issue on one of my hosts, and I'm unsure as to why. I have 1 dev environment and 3 prod environments running the same image. Works on dev, works on 2 prod environments. Third prod environment throws this error, despite the build being identical.

EDIT: Builds were not identical. FIPS mode was enabled on the third host throwing the errors, but disabled on the other prod units. Disabling FIPS mode allows the container to start.

[alexandrud]

I had a similar issue, but I also loaded some custom ROOT CAs and apparently there was a newline missing when concatenating the certs. Maybe it helps.

[timothy-mullican]

A workaround is to set the environment variable OPENSSL_FORCE_FIPS_MODE to 0. Example for the NetBox helm chart values.yaml:

...
extraEnvs:
  - name: OPENSSL_FORCE_FIPS_MODE
     value: "0"
worker:
  extraEnvs:
    - name: OPENSSL_FORCE_FIPS_MODE
       value: "0"

[timothy-mullican]

I am also attaching a work in progress Dockerfile that runs rootless NetBox on Rocky Linux 9 (unable to use RHEL UBI since it lacks the xmlsec1-devel package). It should not be affected by this SSL crypto issue.

Dockerfile.txt

[Codykilpatrick]

A workaround is to set the environment variable OPENSSL_FORCE_FIPS_MODE to 0. Example for the NetBox helm chart values.yaml:

...
extraEnvs:
  - name: OPENSSL_FORCE_FIPS_MODE
     value: "0"
worker:
  extraEnvs:
    - name: OPENSSL_FORCE_FIPS_MODE
       value: "0"

Huge save for me thank you a bunch for this.

[jo-bzh]

Thanks @timothy-mullican for sharing your rootless Netbox Dockerfile,
By any chance did you plan to opensourced it in another location ?

I am seeing something that is likely related. We are running Bottlerocket OS 1.47.0 (aws-k8s-1.32-fips). The required shared libraries are probably not in the image and not available from the host on Bottlerocket. The errors in the log look like this:
⏳ Waiting on DB... (6s / 30s)
🧬 loaded config '/etc/netbox/config/configuration.py'
🧬 loaded config '/etc/netbox/config/extra.py'
🧬 loaded config '/etc/netbox/config/logging.py'
🧬 loaded config '/etc/netbox/config/plugins.py'
Traceback (most recent call last):
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 279, in ensure_connection
self.connect()
File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 256, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/postgresql/base.py", line 332, in get_new_connection
connection = self.Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/psycopg/connection.py", line 120, in connect
raise last_ex.with_traceback(None)
psycopg.OperationalError: connection failed: connection to server at "XXX.XXX.XXX.XXX", port 5432 failed: could not create SSL context: could not load the shared library
connection to server at "XXX.XXX.XXX.XXX", port 5432 failed: could not encrypt password: could not load the shared library
fe_sendauth: error sending password authentication

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/netbox/netbox/./manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/init.py", line 442, in execute_from_command_line
utility.execute()
File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/init.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 416, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/base.py", line 460, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/commands/showmigrations.py", line 67, in handle
return self.show_list(connection, options["app_label"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/commands/showmigrations.py", line 86, in show_list
loader = MigrationLoader(connection, ignore_no_migrations=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/loader.py", line 58, in init
self.build_graph()
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/loader.py", line 235, in build_graph
self.applied_migrations = recorder.applied_migrations()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/recorder.py", line 89, in applied_migrations
if self.has_table():
^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/migrations/recorder.py", line 63, in has_table
with self.connection.cursor() as cursor:
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 320, in cursor
return self._cursor()
^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 296, in _cursor
self.ensure_connection()
File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 278, in ensure_connection
with self.wrap_database_errors:
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/utils.py", line 91, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 279, in ensure_connection
self.connect()
File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/base/base.py", line 256, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/django/db/backends/postgresql/base.py", line 332, in get_new_connection
connection = self.Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/psycopg/connection.py", line 120, in connect
raise last_ex.with_traceback(None)
django.db.utils.OperationalError: connection failed: connection to server at "XXX.XXX.XXX.XXX", port 5432 failed: could not create SSL context: could not load the shared library
connection to server at "XXX.XXX.XXX.XXX", port 5432 failed: could not encrypt password: could not load the shared library
fe_sendauth: error sending password authentication