Background processing with Node.js

Understanding the Cloud Run function

• The function starts by importing several dependencies like Firestore and Translation. The global Firestore and Translation clients are initialized so they can be reused between function invocations. That way, you don't have to initialize new clients for every function invocation, which would slow down execution.

  const Firestore = require('@google-cloud/firestore');
  const {Translate} = require('@google-cloud/translate').v2;
  
  const firestore = new Firestore();
  const translate = new Translate();

• The Translation API translates the string to the language you selected.

  const [
    translated,
    {
      data: {translations},
    },
  ] = await translate.translate(original, language);
  const originalLanguage = translations[0].detectedSourceLanguage;
  console.log(
    `Translated ${original} in ${originalLanguage} to ${translated} in ${language}.`
  );

• The Cloud Run function parses the Pub/Sub message to get the text to translate and the desired target language.

  Then, the app requests a translation and stores the result in Firestore.

  // translate translates the given message and stores the result in Firestore.
  // Triggered by Pub/Sub message.
  exports.translate = async (pubSubEvent) => {
    const {language, original} = JSON.parse(
      Buffer.from(pubSubEvent.data, 'base64').toString()
    );
  
    const [
      translated,
      {
        data: {translations},
      },
    ] = await translate.translate(original, language);
    const originalLanguage = translations[0].detectedSourceLanguage;
    console.log(
      `Translated ${original} in ${originalLanguage} to ${translated} in ${language}.`
    );
  
    // Store translation in firestore.
    await firestore.collection('translations').doc().set({
      language,
      original,
      translated,
      originalLanguage,
    });
  };

Deploying the Cloud Run function

• In the same directory as the translate.js file, deploy the Cloud Run function with a Pub/Sub trigger:

  gcloud functions deploy translate --runtime nodejs10 --trigger-topic translate

Understanding the app

There are two main components for the web app:

• A Node.js HTTP server to handle web requests. The server has the following two endpoints:
  • /: Lists all of the existing translations and shows a form users can submit to request new translations.
  • /request-translation: Form submissions are sent to this endpoint, which publishes the request to Pub/Sub to be translated asynchronously.
• An HTML template that is filled in with the existing translations by the Node.js server.

The HTTP server

• In the server directory, app.js starts by setting up the app and registering HTTP handlers:

  
  // This app is an HTTP app that displays all previous translations
  // (stored in Firestore) and has a form to request new translations. On form
  // submission, the request is sent to Pub/Sub to be processed in the background.
  
  // TOPIC_NAME is the Pub/Sub topic to publish requests to. The Cloud Function to
  // process translation requests should be subscribed to this topic.
  const TOPIC_NAME = 'translate';
  
  const express = require('express');
  const bodyParser = require('body-parser');
  const {PubSub} = require('@google-cloud/pubsub');
  const {Firestore} = require('@google-cloud/firestore');
  
  const app = express();
  const port = process.env.PORT || 8080;
  
  const firestore = new Firestore();
  
  const pubsub = new PubSub();
  const topic = pubsub.topic(TOPIC_NAME);
  
  // Use handlebars.js for templating.
  app.set('views', __dirname);
  app.set('view engine', 'html');
  app.engine('html', require('hbs').__express);
  
  app.use(bodyParser.urlencoded({extended: true}));
  
  app.get('/', index);
  app.post('/request-translation', requestTranslation);
  app.listen(port, () => console.log(`Listening on port ${port}!`));
  

• The index handler (/) gets all existing translations from Firestore and fills an HTML template with the list:

  
  // index lists the current translations.
  async function index(req, res) {
    const translations = [];
    const querySnapshot = await firestore.collection('translations').get();
    querySnapshot.forEach((doc) => {
      console.log(doc.id, ' => ', doc.data());
      translations.push(doc.data());
    });
  
    res.render('index', {translations});
  }
  

• New translations are requested by submitting an HTML form. The request translation handler, registered at /request-translation, parses the form submission, validates the request, and publishes a message to Pub/Sub:

  
  // requestTranslation parses the request, validates it, and sends it to Pub/Sub.
  function requestTranslation(req, res) {
    const language = req.body.lang;
    const original = req.body.v;
  
    const acceptableLanguages = ['de', 'en', 'es', 'fr', 'ja', 'sw'];
    if (!acceptableLanguages.includes(language)) {
      throw new Error(`Invalid language ${language}`);
    }
  
    console.log(`Translation requested: ${original} -> ${language}`);
  
    const buffer = Buffer.from(JSON.stringify({language, original}));
    topic.publish(buffer);
    res.sendStatus(200);
  }

The HTML template

The HTML template is the basis for the HTML page shown to the user so they can see previous translations and request new ones. The template is filled in by the HTTP server with the list of existing translations.

• The <head> element of the HTML template includes metadata, style sheets, and JavaScript for the page:

  <html>
  
  <head>
      <meta charset="UTF-8">
      <meta name="viewport" content="width=device-width, initial-scale=1.0">
      <title>Translations</title>
  
      <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
      <link rel="stylesheet" href="https://code.getmdl.io/1.3.0/material.indigo-pink.min.css">
      <script defer src="https://code.getmdl.io/1.3.0/material.min.js"></script>
      <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
      <script>
          $(document).ready(function() {
              $("#translate-form").submit(function(e) {
                  e.preventDefault();
                  // Get value, make sure it's not empty.
                  if ($("#v").val() == "") {
                      return;
                  }
                  $.ajax({
                      type: "POST",
                      url: "/request-translation",
                      data: $(this).serialize(),
                      success: function(data) {
                          // Show snackbar.
                          console.log(data);
                          var notification = document.querySelector('.mdl-js-snackbar');
                          $("#snackbar").removeClass("mdl-color--red-100");
                          $("#snackbar").addClass("mdl-color--green-100");
                          notification.MaterialSnackbar.showSnackbar({
                              message: 'Translation requested'
                          });
                      },
                      error: function(data) {
                          // Show snackbar.
                          console.log("Error requesting translation");
                          var notification = document.querySelector('.mdl-js-snackbar');
                          $("#snackbar").removeClass("mdl-color--green-100");
                          $("#snackbar").addClass("mdl-color--red-100");
                          notification.MaterialSnackbar.showSnackbar({
                              message: 'Translation request failed'
                          });
                      }
                  });
              });
          });
      </script>
      <style>
          .lang {
              width: 50px;
          }
          .translate-form {
              display: inline;
          }
      </style>
  </head>

  The page pulls in Material Design Lite (MDL) CSS and JavaScript assets. MDL lets you add a Material Design look and feel to your websites.

  The page uses JQuery to wait for the document to finish loading and set a form submission handler. Whenever the request translation form is submitted, the page does minimal form validation to check that the value isn't empty, and then sends an asynchronous request to the /request-translation endpoint.

  Finally, an MDL snackbar appears to indicate if the request was successful or if it encountered an error.

• The HTML body of the page uses an MDL layout and several MDL components to display a list of translations and a form to request additional translations:

  <body>
      <div class="mdl-layout mdl-js-layout mdl-layout--fixed-header">
          <header class="mdl-layout__header">
              <div class="mdl-layout__header-row">
                  <!-- Title -->
                  <span class="mdl-layout-title">Translate with Background Processing</span>
              </div>
          </header>
          <main class="mdl-layout__content">
              <div class="page-content">
                  <div class="mdl-grid">
                  <div class="mdl-cell mdl-cell--1-col"></div>
                      <div class="mdl-cell mdl-cell--3-col">
                          <form id="translate-form" class="translate-form">
                              <div class="mdl-textfield mdl-js-textfield mdl-textfield--floating-label">
                                  <input class="mdl-textfield__input" type="text" id="v" name="v">
                                  <label class="mdl-textfield__label" for="v">Text to translate...</label>
                              </div>
                              <select class="mdl-textfield__input lang" name="lang">
                                  <option value="de">de</option>
                                  <option value="en">en</option>
                                  <option value="es">es</option>
                                  <option value="fr">fr</option>
                                  <option value="ja">ja</option>
                                  <option value="sw">sw</option>
                              </select>
                              <button class="mdl-button mdl-js-button mdl-button--raised mdl-button--accent" type="submit"
                                  name="submit">Submit</button>
                          </form>
                      </div>
                      <div class="mdl-cell mdl-cell--8-col">
                          <table class="mdl-data-table mdl-js-data-table mdl-shadow--2dp">
                              <thead>
                                  <tr>
                                      <th class="mdl-data-table__cell--non-numeric"><strong>Original</strong></th>
                                      <th class="mdl-data-table__cell--non-numeric"><strong>Translation</strong></th>
                                  </tr>
                              </thead>
                              <tbody>
                                  {{#each translations}}
                                  <tr>
                                      <td class="mdl-data-table__cell--non-numeric">
                                          <span class="mdl-chip mdl-color--primary">
                                              <span class="mdl-chip__text mdl-color-text--white">{{ originalLanguage }} </span>
                                          </span>
                                      {{ original }}
                                      </td>
                                      <td class="mdl-data-table__cell--non-numeric">
                                          <span class="mdl-chip mdl-color--accent">
                                              <span class="mdl-chip__text mdl-color-text--white">{{ language }} </span>
                                          </span>
                                          {{ translated }}
                                      </td>
                                  </tr>
                                  {{/each}}
                              </tbody>
                          </table>
                          <br/>
                          <button class="mdl-button mdl-js-button mdl-button--raised" type="button" 
                              Refresh
                          </button>
                      </div>
                  </div>
              </div>
              <div aria-live="assertive" aria-atomic="true" aria-relevant="text" class="mdl-snackbar mdl-js-snackbar" id="snackbar">
                  <div class="mdl-snackbar__text mdl-color-text--black"></div>
                  <button type="button" class="mdl-snackbar__action"></button>
              </div>
          </main>
      </div>
  </body>
  
  </html>

Deploying the web app

You can use the App Engine standard environment to build and deploy an app that runs reliably under heavy load and with large amounts of data.

This tutorial uses the App Engine standard environment to deploy the HTTP frontend.

The app.yaml configures the App Engine app:

runtime: nodejs10

• From the same directory as the app.yaml file, deploy your app to the App Engine standard environment:

  gcloud app deploy

Testing the app

After you've deployed the Cloud Run function and App Engine app, try requesting a translation.

1. To view the app in your browser,enter the following URL:

   https://PROJECT_ID.REGION_ID.r.appspot.com

   Replace the following:

   • PROJECT_ID: Your Google Cloud project ID
   • REGION_ID: A code that App Engine assigns to your app

   There is a page with an empty list of translations and a form to request new translations.

2. In the Text to translate field, enter some text to translate, for example, Hello, World.
3. Select a language from the drop-down list that you want to translate the text to.
4. Click Submit.
5. To refresh the page, click Refresh refresh. There is a new row in the translation list. If you don't see a translation, wait a few more seconds and try again. If you still don't see a translation, see the next section about debugging the app.

Debugging the app

If you cannot connect to your App Engine app or don't see new translations, check the following:

1. Check that the gcloud deploy commands successfully completed and didn't output any errors. If there were errors, fix them, and try deploying the Cloud Run function and the App Engine app again.

2. In the Google Cloud console, go to the Logs Viewer page.

   Go to Logs Viewer page
   1. In the Recently selected resources drop-down list, click GAE Application, and then click All module_id. You see a list of requests from when you visited your app. If you don't see a list of requests, confirm you selected All module_id from the drop-down list. If you see error messages printed to the Google Cloud console, check that your app's code matches the code in the section about understanding the app.
   2. In the Recently selected resources drop-down list, click Cloud Function, and then click All function name. You see a function listed for each requested translation. If not, check that the Cloud Run function and App Engine app are using the same Pub/Sub topic:
      • In the background/server/app.js file, check that the TOPIC_NAME constant is "translate".
      • When you deploy the Cloud Run function, be sure to include the --trigger-topic=translate flag.