Pulsar docker image on Azure

I wanted to quick set up Apache Pulsar on Azure. Long time ago this didn't go well but I stumbled on Kesque helm chart. Thanks to them it was quite easy. However Microsoft tends to charge quite high for VMs in Azure Kubernetes Service. Now for a startup it gets too high and we don't need that high throughput nor that kind of stability. However if you ever worked with messaging systems like Pulsar, they love to fail if there's too much traffic and VMs are not high-end or they are low in quantity. Also, using storage with AKS would be too much of a price hit so using File Share with Azure Container Instance was best overall solution for us for development environment.

Apache Pulsar Docker image customisation for JWT

This step is something I wanted but it's not required to make it work on Azure nor anywhere else.

Apache Pulsar has a nice Docker image. However it does not come with predefined way of using JWT or some other security. In order to do so I've created custom image that changes /pulsar/conf/standalone.conf to use JWT. This would make it easier for automating development to use simple JWT. First I generated public and private key, then used private key to generate JWTs for superuser, and then stored superuser JWT in container so brokers can communicate. This also included adding such lines to standalone.conf. If you forget to set these they will make your Pulsar not list clusters let alone tenants, namespaces, and such. Example of lines required to be changed

# line number about 352
# Enable authentication
authenticationEnabled=true

# line number about 358
# Autentication provider name list, which is comma separated list of class names
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken

#line number about 370
superUserRoles=superuser
#line number about 374-375
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=file:///pulsar/superuser.jwt

# line 384
tokenPublicKey=file:///pulsar/my-public.key

Notice the difference between org.apache.pulsar.broker.authentication.AuthenticationProviderToken for authenticationProviders and org.apache.pulsar.client.impl.auth.AuthenticationToken for borkerAuth. Also tokenPublicKey must be set for Pulsar to be able to verify those JWTs

First time I put the provider as the client so it didn't work and took about 20 min to figure out :D.

I also modified client.conf

#line 35
authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
#line 42
authParams=file:///pulsar/superuser.jwt

which is used by tools in bin folder, and websocket.conf same settings as in standalone (maybe different lines).

Next, the docker image. I used official docker image as base and added lines to copy .jwt and .conf files.

FROM apachepulsar/pulsar:2.6.1
WORKDIR /pulsar
COPY my-private.key my-private.key
COPY my-public.key my-public.key
COPY superuser.jwt superuser.jwt
COPY client.conf conf/client.conf
COPY websocket.conf conf/websocket.conf
COPY standalone.conf conf/standalone.conf
CMD [ "/pulsar/bin/pulsar", "standalone", "-nfw", "-nss" ]

1. Public and private key have been generated before and copied to local machine. You can find in Pulsar official docs how to do it.
2. Client, websocket, and standalone conf are modified to use authorisation and need to be copied. I couldn't get something like RUN sed -i "s|xxx|xxx|g" xxx.conf" to work - it would not modify files. I guess it is because it doesn't see the files when building an image and it actually runs command at that point. So I went for fallback method which is copy existing files, modify then push back.
3. And finally, strip startup of parameters as much as possible so start Pulsar with the image. Parameters -nfw -nss are to avoid using functions feature.

Azure Container Instance

Assuming you already know how to do it, first create File Share. We need it so pulsar can store /pulsar/data directory in it in to be able to keep data in cases of restarts, failures, and such. I still haven't figured out how to connect Blob (if possible) so I'm mounting File Shares.

At the time of writing, Azure didn't support Container Instances to be created with the mount through Web interface so back to terminal:

az container create --resource-group RC_NAME --name CONTAINER_NAME \
--image YOUR_DOCKER_IMAGE_REPO/IMAGE_TAG \
--registry-username IMAGE_REPO_USERNAME \
--registry-password IMAGE_REPO_PASSWORD \
--dns-name-label SUBDOMAIN_PART_OF_URL_YOU_WANT \
--ports 8080 6650 \
--azure-file-volume-account-name STORAGE_ACC_NAME \
--azure-file-volume-account-key STORAGE_ACC_KEY \
--azure-file-volume-share-name pulsar \
--azure-file-volume-mount-path /pulsar/data \
--cpu 4 --memory 4

If your reading this article, your probably fine with 4 core / 4 gig ram.

Mounting /pulsar container path would lead to errors and your container wouldn't start. If you need more than 1 path like /pulsar/conf and /pulsar/data you would need to go through -templates. Reason it fails on /pulsar mount is that, if you haven't guessed it on first look, Pulsar would look into empty file share and wouldn't be able to discover /pulsar/bin/pulsar script and thus report error that file is not found for latest line in Dockerfile.

Should work

After couple of seconds / minutes, you should see your container instance up and running.

If you hit any problems call Azure support team, they "always help" XD.