[go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] - SSL Certificate Verification Failure in Dask Gateway Unit Tests #2744

Closed
viniciusdc opened this issue Sep 25, 2024 · 0 comments · Fixed by #2741
Closed

[BUG] - SSL Certificate Verification Failure in Dask Gateway Unit Tests #2744

viniciusdc opened this issue Sep 25, 2024 · 0 comments · Fixed by #2741
Labels
area: CI/CD 👷🏽‍♀️ area: testing ✅ Testing needs: triage 🚦 Someone needs to have a look at this issue and triage type: bug 🐛 Something isn't working

Comments

@viniciusdc
Copy link
Contributor
viniciusdc commented Sep 25, 2024 •
edited
Loading

Describe the bug

Our unit tests have begun failing due to SSL certificate verification errors when connecting to the Dask Gateway. Specifically, we encounter an SSLCertVerificationError caused by a self-signed certificate.

=================================== FAILURES ===================================
______________________________ test_dask_gateway _______________________________
....
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/dask_gateway/client.py", line 397, in _request
    resp = await session.request(method, url, json=json, **self._request_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/client.py", line [659](https://github.com/nebari-dev/nebari/actions/runs/11022572877/job/30612182623?pr=2739#step:16:660), in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 557, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 1002, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 1336, in _create_direct_connection
    raise last_exc
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 1305, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 1070, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host github-actions.nebari.dev:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')]

We've been using a monkey patch to disable SSL verification as a temporary workaround, which has allowed tests to pass so far; however, this does not work anymore.

nebari/tests/tests_deployment/utils.py

Lines 115 to 132 in b2afe1a

def monkeypatch_ssl_context():
"""
This is a workaround monkeypatch to disable ssl checking to avoid SSL
failures.
TODO: A better way to do this would be adding the Traefik's default certificate's
CA public key to the trusted certificate authorities.
"""
def create_default_context(context):
def _inner(*args, **kwargs):
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
return context
return _inner
sslcontext = ssl.create_default_context()
ssl.create_default_context = create_default_context(sslcontext)

My guess, based on a similar encounter, is that aiohttp the base lib used by dask to handle the connections has changed something internally regarding the TCPConnector default behavior (this happens internally within the requests, and is decided by a combination of parameters that dask handles based on the provided proxy schema, which in our case we were passing for jupyterhub authentication)

    async def _request(self, method, url, json=None):
        if self._session is None:
            # "unsafe" allows cookies to be set for ip addresses, which can
            # commonly serve dask-gateway deployments. Since this client is
            # only ever used with a single endpoint, there is no danger of
            # leaking cookies to a different server that happens to have the
            # same ip.
            self._session = aiohttp.ClientSession(
                cookie_jar=aiohttp.CookieJar(unsafe=True)
            )
        session = self._session

        resp = await session.request(method, url, json=json, **self._request_kwargs)

To address this adequately, I propose overriding the dask_gateway.Gateway class using self._request_kwargs object above to allow the TLS parameter to be disabled within our test context.

Expected behavior

  • unit tests should run correctly

OS and architecture in which you are running Nebari

Linux (CI/GHA)

How to Reproduce the problem?

https://github.com/nebari-dev/nebari/actions/runs/11022572877/job/30612182623?pr=2739

Command output

No response

Versions and dependencies used.

No response

Compute environment

kind

Integrations

No response

Anything else?

No response
@viniciusdc viniciusdc added type: bug 🐛 Something isn't working needs: triage 🚦 Someone needs to have a look at this issue and triage labels Sep 25, 2024
@viniciusdc viniciusdc changed the title [BUG] - <title> [BUG] - SSL Certificate Verification Failure in Dask Gateway Unit Tests Sep 25, 2024
@viniciusdc viniciusdc mentioned this issue Sep 25, 2024
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: CI/CD 👷🏽‍♀️ area: testing ✅ Testing needs: triage 🚦 Someone needs to have a look at this issue and triage type: bug 🐛 Something isn't working
Projects
🪴 Nebari Project Management
Status: Done 💪🏾
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Test implict aiohttp's TCP to HTTP connector change nebari-dev/nebari
1 participant
@viniciusdc