[go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ArgoCD Becomes Unresponsive and Frequently Disconnects from the Cluster When Kyverno is Installed (Same or Different Cluster) #20977

Open
2 of 3 tasks
hablislim opened this issue Nov 27, 2024 · 8 comments
Open
2 of 3 tasks

ArgoCD Becomes Unresponsive and Frequently Disconnects from the Cluster When Kyverno is Installed (Same or Different Cluster) #20977

hablislim opened this issue Nov 27, 2024 · 8 comments
Labels
bug Something isn't working more-information-needed Further information is requested version:2.11 Latest confirmed affected version is 2.11

Comments

@hablislim
Copy link
hablislim commented Nov 27, 2024 •
edited
Loading

argoCD version: v2.11.2+25f7504

kyverno version: 1.12.5

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug

When Kyverno is deployed on the Kubernetes cluster alongside ArgoCD (or in a separate cluster), ArgoCD becomes significantly slower and eventually disconnects from the k8s cluster. This issue affects the overall performance and usability of ArgoCD, causing interruptions in the continuous deployment process.

We set up ArgoCD following the official documentation and incorporated best practices outlined in the ArgoCD user notes. Additionally:

  • we excluded specific Kyverno-related objects by applying the following configuration:
resource.exclusions: |
  - apiGroups:
      - kyverno.io
      - reports.kyverno.io
      - wgpolicyk8s.io
    kinds:
      - ClusterPolicyReport
      - PolicyReport
      - EphemeralReport
    clusters:
      - '*'
  • Ignore diff in aggregated cluster roles
  resource.compareoptions: |
    ignoreAggregatedRoles: true
  • Enable Replace in the syncOptions in kyverno argoCD application
    syncOptions:
      - Replace=true

To Reproduce

  1. Deploy a Kubernetes cluster.
  2. Install ArgoCD on the cluster.
  3. Configure ArgoCD with kyverno recommended configuration.
  4. Install Kyverno policies on the same cluster.
  5. Observe the performance degradation in ArgoCD, including delays and frequent disconnects.

Expected behavior

ArgoCD should remain stable and perform efficiently, even when Kyverno is deployed on the same cluster. Both tools should coexist without negatively impacting each other.

Screenshots

argo-issue

Version

argocd: v2.11.2+25f7504

Logs

Failed to load live state: failed to get cluster info for "https://xxx.xxx.xxx.xxx": 
error synchronizing cache state : error getting openapi resources: 
the server is currently unable to handle the request

Related links:
ArgoCD configuration with kyverno

@hablislim hablislim added the bug Something isn't working label Nov 27, 2024
@andrii-korotkov-verkada andrii-korotkov-verkada added the version:2.13 Latest confirmed affected version is 2.13 label Nov 27, 2024
@hablislim
Copy link
Author

@andrii-korotkov-verkada please note that argocD version is v2.11.2 (I updated the description)

@andrii-korotkov-verkada andrii-korotkov-verkada added version:2.11 Latest confirmed affected version is 2.11 and removed version:2.13 Latest confirmed affected version is 2.13 labels Nov 27, 2024
@andrii-korotkov-verkada
Copy link
Contributor

Can you share some logs, like for "Reconciliation Completed"?

@andrii-korotkov-verkada andrii-korotkov-verkada added the more-information-needed Further information is requested label Nov 27, 2024
@LS80
Copy link
LS80 commented Nov 28, 2024

What is your Kyverno version? Could be this?

@bygui86
Copy link
bygui86 commented Nov 28, 2024

Hi guys
I'm working with @hablislim on the same project.

Here a screenshot about an ArgoCD App conditions while connectivity is lost
image

@andrii-korotkov-verkada actually the reconciliation doesn't happen because ArgoCD looses connection to the K8s cluster. We are 100% sure that there is no connectivity issues for following reasons:

  • before deploying Kyverno, no issues at all, ArgoCD working good 100% of time
  • kubectl commands from local and from within the ArgoCD cluster work perfectly
  • most importantly, right after removing Kyverno ArgoCD is able to re-establish the connection almost instantly

@LS80 as you can see in the description, we use Kyverno v1.12.5. I read that issue already and to avoid it we introduced the reports-server and excluded Kyverno ClusterPolicyReport, PolicyReport, EphemeralReport CRDs in ArgoCD configuration (as already reported in the description).

@andrii-korotkov-verkada andrii-korotkov-verkada removed the more-information-needed Further information is requested label Nov 28, 2024
@andrii-korotkov-verkada
Copy link
Contributor

These logs indicate to me that ArgoCD can't query URL from Kyverno. Do you have logs on Kyverno side, like whether there's some permission denied or rate limiting happening?

@andrii-korotkov-verkada andrii-korotkov-verkada added the more-information-needed Further information is requested label Nov 28, 2024
@bygui86
Copy link
bygui86 commented Nov 28, 2024

@andrii-korotkov-verkada thanks for your message.
Sorry but I'm not sure to understand. What do you mean by "ArgoCD can't query URL from Kyverno"?
Anyway we removed Kyverno after collecting some information but we found nothing relevant in the logs, we will deploy Kyverno again in a sandbox K8s cluster to see whether we can replicate the same behaviour.

@andrii-korotkov-verkada
Copy link
Contributor

I was referring to get cluster info error for the cluster. Maybe I misunderstood. Can you try querying the url normally with Kyverno being there and see what happens?

@bygui86
Copy link
bygui86 commented Nov 28, 2024

@andrii-korotkov-verkada actually here is where it becomes weird :|
With Kyverno in place and ArgoCD with connection lost, the K8s cluster is perfectly reachable via kubectl from local. No issue running commands like "kubectl get pods" or any similar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working more-information-needed Further information is requested version:2.11 Latest confirmed affected version is 2.11
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@LS80 @bygui86 @hablislim @andrii-korotkov-verkada