[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

maxBuffer length exceeded #273

Open
lomkju opened this issue Jun 10, 2020 · 20 comments
Open

maxBuffer length exceeded #273

lomkju opened this issue Jun 10, 2020 · 20 comments

Comments

@lomkju
Copy link
lomkju commented Jun 10, 2020

Hi,
I get the below error when running checkoutv2
I always need to delete the contents of the repo for all build.
_work/api

Deleting the contents of '/home/ubuntu/actions-runner/_work/api/api'
##[error]stderr maxBuffer length exceeded
@lomkju
Copy link
Author
lomkju commented Jun 10, 2020

Sounds like a permission issue. The issue below seems to be related.
#211 (comment)

@ericsciple
Copy link
Contributor

@lomkju can you enable debug logging hopefully will produce more info

@lomkju
Copy link
Author
lomkju commented Jun 11, 2020

Adding the below step before the checkout step fixed the problem. It would be a hack until this is fixed or know why this is happening? Maybe the docker image checkouts the branch as root?

    - name: cleanup #https://github.com/actions/checkout/issues/211
      run: |
        sudo chown -R $USER:$USER $GITHUB_WORKSPACE

The logs for the same. The list is huge for permissions denied just pasting the top ones.

Run actions/checkout@v2
Syncing repository: ***********
Getting Git version info
/usr/bin/git config --local --get remote.origin.url
https://github.com/*******/*******
Removing previously created refs, to avoid conflicts
Cleaning the repository
##[warning]Unable to clean or reset the repository. The repository will be recreated instead.
Deleting the contents of '/home/ubuntu/actions-runner/_work/api/api'
##[error]Command failed: rm -rf "/home/ubuntu/actions-runner/_work/api/api/storage"
rm: cannot remove '/home/ubuntu/actions-runner/_work/api/api/storage/************': Permission denied
rm: cannot remove '/home/ubuntu/actions-runner/_work/api/api/storage/l*************': Permission denied

@jokreliable
Copy link
Contributor

in my environment, the runner runs as a "action-runner" user. one of the action uses an docker instance where the user inside is root. When the docker instance generates files, those are owned as root (even from the host perspective).
We solved this by having a "trap" in the entrypoint.sh script that resets ownership when the script ends

fixperms() {
  chown -R `stat --printf=%u /github/workspace/.` /github/workspace/*
}
trap fixperms EXIT

We also see this when we try to rerun workflows (where there is a ton of output in some of those jobs). Clearing the logs before retrying helps but I'm concerned however that we might end up getting even more output from those builds, and hit that limit again.

@pixsaoul
Copy link

Hello I am having the exact same issue using game CI action on a self-hosted runner. However this workaround

Adding the below step before the checkout step fixed the problem. It would be a hack until this is fixed or know why this is happening? Maybe the docker image checkouts the branch as root?

    - name: cleanup #https://github.com/actions/checkout/issues/211
      run: |
        sudo chown -R $USER:$USER $GITHUB_WORKSPACE

The logs for the same. The list is huge for permissions denied just pasting the top ones.

Run actions/checkout@v2
Syncing repository: ***********
Getting Git version info
/usr/bin/git config --local --get remote.origin.url
https://github.com/*******/*******
Removing previously created refs, to avoid conflicts
Cleaning the repository
##[warning]Unable to clean or reset the repository. The repository will be recreated instead.
Deleting the contents of '/home/ubuntu/actions-runner/_work/api/api'
##[error]Command failed: rm -rf "/home/ubuntu/actions-runner/_work/api/api/storage"
rm: cannot remove '/home/ubuntu/actions-runner/_work/api/api/storage/************': Permission denied
rm: cannot remove '/home/ubuntu/actions-runner/_work/api/api/storage/l*************': Permission denied

Did not work for me, the cleanup is lasting forever like blocked. Has anyonje found another solution ? (cleaner, or atleast that would not block on my machine ?)

Thanks in advance !

@alexduros
Copy link

I'm facing this error as well on a self-hosted runner like @pixsaoul.
I suspect that this error is hiding the original one. Is there a way to have more insights (not only verbose mode which is not really helping) to understand how to work around this. Thx !

##[debug]Added matchers: 'checkout-git'. Problem matchers scan action output for known warning or error strings and report these inline.
Syncing repository: /
::group::Getting Git version info
Getting Git version info
Deleting the contents of '/home/ubuntu/actions-runner/_work//'
::remove-matcher owner=checkout-git,::
##[debug]Removed matchers: 'checkout-git'
Error: stderr maxBuffer length exceeded
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Run actions/checkout@v2

@na-jakobs
Copy link

+1 we are also seeing this issue

@rizvisx
Copy link
rizvisx commented Feb 28, 2022

Is there any update on this issue, I am getting it on a self-hosted runner.

Thanks !!

@jokreliable
Copy link
Contributor
jokreliable commented Feb 28, 2022

We have seen this when there are permissions issues. tracking down what generate the files with the wrong permission and making sure that the ownership is correct fixed it for us. (we had some docker container based processes running as root, which left behind some leftover temp files, which in turn prevented the following checkout action from cleaning up. not quite sure why that shows up as "maxBuffer length exceeded" (other than we had a LOT of files with that problem). We tracked down which job by going through the _diag/*.log files to identify the problematic job...

@imollm
Copy link
imollm commented Mar 18, 2022

My solution has been to add a step in pipeline executing a SSH command to delete project folder in _work folder (first step of them)

@Sunil-Jacob
Copy link

Am also facing the Error: stderr maxBuffer length exceeded
Do we have a fix for the same?

@dayananda30
Copy link
  • 1, we are also facing the same issue.

@dginovker
Copy link

Anyone have an elegant solution to this?

@jokreliable
Copy link
Contributor

We tried a few things here:

  • trap in the docker script to chown all the things when it's done back to the runner user (does not work when job gets cancelled)
  • cron job that runs and does the same (this may not work for you, depending on how your workflows are setup -- and often doesn't fix it for a bit, with a window for some jobs to land on the broken runner for a bit.
  • use ephemeral runners (this has worked really well for us, but you may not have the ability to do this).

The only thing that i can think of is that more recent runners have the option to run some script between jobs, iirc, and that might be your best bet to address the issue on your end. See here for more details: https://docs.github.com/en/actions/hosting-your-own-runners/running-scripts-before-or-after-a-job
(note that this feature is in beta at the moment...)

Ultimately, you will also need to sort out how to elevate permissions to be able to change the ownership of all the directory/ies where the checkout end up at back to the user that runs the runner...

@cniackz
Copy link
cniackz commented Aug 19, 2022

Other temporal solution I found is to delete the folder and then recreate the folder, that works for me:

  - name: cleanup #https://github.com/actions/checkout/issues/211
    run: |
      echo ${{ secrets.DEPLOY_PASSWORD }} | sudo -S rm -rf ${GITHUB_WORKSPACE}
      mkdir ${GITHUB_WORKSPACE}

@arlekinjose
Copy link

I think we the gitlab runner user should not have admin privileges using sudo, if so anyone with permissions to modify the github actions pipeline could cause damage to the self runner if he wanted. If the checkout actions is not able to remove existing files from the workspace it could be because these files are not own by the gitlab runner user. Maybe another github actions that ran previously has a defect and creates files in the folder owned by the root user.

@AndreyDodonov-EH
Copy link

FYI:
I observed that after rebasing branch in git.
As a temporary workaround I removed _work directory and restarted the runner

@damienjoldersma
Copy link

I think we the gitlab runner user should not have admin privileges using sudo, if so anyone with permissions to modify the github actions pipeline could cause damage to the self runner if he wanted. If the checkout actions is not able to remove existing files from the workspace it could be because these files are not own by the gitlab runner user. Maybe another github actions that ran previously has a defect and creates files in the folder owned by the root user.

I was using an action 'kube-tools' by stefanprodan/kube-tools@v1 and the commands that were run were in fact executed by root uid / gid outside of the docker image which I was surprised. I ended up embedding a chown -R : of my github actions user/group - but it's not pretty for sure. I will investigate another solution now that I have my own self-hosted runners, like install those tools locally and not depend on this action and the "leaking" root uid/gid like you say as a defect.

@Usman-rauf
Copy link

Just to Give permission with "sudo chown -R $USER:$USER $GITHUB_WORKSPACE " and then add aline to add node_modules with sudo " sudo rm -rf node_modules"

name: Ci/Cd Pipeline

on:
push:
branches: [ "staging" ]
pull_request:
branches: [ "staging" ]

jobs:
build:

runs-on: self-hosted

steps:
- name: cleanup #https://github.com/actions/checkout/issues/21
  run: |
    sudo chown -R $USER:$USER $GITHUB_WORKSPACE
- uses: actions/checkout@v3
- run: sudo rm -rf node_modules
- run: sudo npm i 
- run: sudo pm2 kill -a
- run: sudo pm2 start index.js

@jcook-uptycs
Copy link

Can we get the actions/checkout action to clean up at the end of the job?

The goals...

  • Not require the runner to run with root access
  • Be able to run the action on runners that checkout with and without containers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests