Laravel Worker with super-visor

Many of us struggled to setup supervisor with Laravel Worker in our docker container or web-server. If you don't then you are lucky not as me. 😉

If you are struggling to do or you want to know about what is job, how supervisor maintain the jobs inside a docker-container or web-server, how to setup all these stuffs then you are in right post i guess.

Follow my journey that let you know something informative.

Pre-requisites

• Setup laravel queue:

php artisan queue:table
php artisan migrate

After migrating, You should find 2 tables named failed_jobs & jobs. If you don't then follow the procedure again or check laravel queue for better understanding.

• Setup job batches to keep track of jobs:

php artisan queue:batches-table
php artisan migrate

You should see table named job_batches. To learn more about batch jobs, Check the beautiful documentation of batch jobs laravel

• Setup some test methods for batch jobs:

    public function testJob() {
        $path = storage_path('test');
        $files = glob("$path/user.csv"); // Parse the stored csv
        $header = [];
        foreach ($files as $key => $file) {
            $data = array_map('str_getcsv', file($file));
            if($key === 0) {
                $header = $data[0];
                unset($data[0]);
            }
        }
        $batch = Bus::batch([new TestCsvForSupervisor($header, 
        $data)->dispatch();
        return $batch->id;
    }

What we do here?
Ok, we've imported a csv with 100 thousand users information ex. email, username, address etc. TestCsvForSupervisor is the job class for handling some checks or operation like insert, update or comparing datas with any table ex. users. Simple as writing artisan command. 😉

In TestCsvForSupervisor Job Class:

    public function handle()
    {
        try{
            foreach ($this->data as $row) {
                $result = array();
                $result = User::whereName($row[0])->first() ? 'Found' : 'Not Found';
                Log::info('User checked successfully with email: '.$row[1]);
            }
        } catch (Exception $err) {
            Log::info('Error occured with message: '.$err);
        }
    }

Tough one, I have 20 thousand users information in my users table and what I've done here? Compare csv data with users table data and store the checking result in log (You might find it clueless but it's just a test case actually not good test case😒). Then dispatch the job to store the pending jobs to be done in table.

Setup supervisor

Now it's time for setup supervisor into docker container where you can follow 2 steps:

1. You can access docker bash and install supervisor by command scripts (It is for local testing not for global deployment or team collaboration)
2. You can integrate the command scripts in Dockerfile.

Before digging into both please let me inform you some details about supervisor

Supervisor is basically linux based daemon that runs in the background. You have to configure your laravel app so that supervisor know about background jobs. It monitors the executing or executable jobs and if for any interference the jobs accidentally stop ex. server down after the restarting the supervisor automatically starts the job from where it stops and monitor the failed, pending and success jobs through this time and can deliver us jobs report.

Install from docker bash:

sudo apt-get install supervisor

Install from Dockerfile using Daemon

In your app/docker-conf/php/Dockerfile, Write the following script:

RUN apt-get install -y supervisor

Then, build and start the docker:

docker compose build
docker compose up -d // -d for running the docker in background

I will suggest you to follow the docker daemon-based (step-2) supervisor because it can be used globally. So, we will be continuing with that.

In your root directory make file named laravel-worker.conf and write and save the file:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /app/artisan queue:work // you may use extra attributes like --sleep=3
autostart=true
autorestart=true
# stopasgroup=true
# killasgroup=true
;user=
numprocs=8
redirect_stderr=true
stdout_logfile=/app/storage/logs/worker.log​
stopwaitsecs = 100

In the docker bash, you will find file named supervisord.conf in /etc/supervisor/ directory. Overwrite the last line of the supervisord.conf file by following script and save it:

[include]
files = /etc/supervisor/conf.d/*.conf /app/laravel-worker.conf

After that, Execute the following commands to initiate supervisor:

# Start the service
  service supervisor start

# Read the new config from supervisord file
  supervisorctl reread

# Activate the configuration
  supervisorctl update

# Start queue command for running jobs
  supervisorctl start laravel-worker:* or laravel-worker

# Check the status of new config operation
  supervisorctl status​

Congrats 🥳, You've just did a great job not just laravel-job but the real automated jobs using supervisor.

You may find the job list with status and unique id from job_batches table after starting supervisor like below shot:

There is one interesting fact about supervisor. Do you know what language used to develop supervisor?

It's Python 🐍😲

         Thanks for reading with so much patience. ❤️