Ignacio Le Fluk for This Dot

Posted on Aug 15, 2019 • Edited on Sep 12, 2019

Schematics: Building blocks

Intro

One of the most powerful tools that the Angular ecosystem provides us as developers is the Angular CLI. Have you ever wondered how is that components, services, and many others are created every time you hit the ng generate command? The CLI makes use under the hood of Schematics and specifically the default schematics collection @schematics/angular. In this post, we'll explore a few key concepts when working with schematics, understand its basic operations and build our own schematic collection from scratch using the schematics CLI.

The Tree

Schematics let us operate on a virtual filesystem, called a Tree. when running a schematic we'll be able to stage a set of transforms to it (create, update or remove files), and finally apply (or not) those changes.

Creating our first schematic

To start creating our first schematic, we'll start by installing the Schematics CLI, which will help us in scaffolding our schematic collection.

// install CLI
npm install -g @angular-devkit/schematics-cli

//create collection
schematics blank my-collection // or schematics blank --name=my-collection

The schematics command works in two ways depending on where it's used. If you're not in a schematics project folder, it will create a new one with the basic structure and install all its dependencies, otherwise, it will add a new schematic to the collection.
Let's take a look at our project structure.

collection.json contains our schematics collection information, exposes the schematics that the collection will have and links to the proper factory methods. There's more configuration that can be done inside this file, like adding aliases, but we'll get to that later.

{
  "$schema": "../node_modules/@angular-devkit/schematics/collection-schema.json",
  "schematics": {
    "my-collection": {
      "description": "A blank schematic.",
      "factory": "./my-collection/index#myCollection"
    }
  }
}

The default schematic when creating a new collection has the same name as the collection. A schematic should export at least one Rule factory function. Our new schematic has a short description and a path to the file that contains the schematic follow by a hashtag and the factory function to use when it's called.

// src/my-collection/index.ts
import { Rule, SchematicContext, Tree } from "@angular-devkit/schematics";

export function myCollection(_options: any): Rule {
  return (tree: Tree, _context: SchematicContext) => {
    return tree;
  };
}

index.ts exports a single function, a Rule factory function. A Rule is a function that given a Tree and a SchematicContext, will return a new Tree.
In the example above, no transformation is being applied.

You can export more than a single function from a single file, you can also export a single default function and omit the last part of the factory path on collection.json.

Building and running

Before we can run a schematic we need to build the collection. To build and run our schematic we'll use two commands.

// build
npm run build

// run
schematics .:my-collection

First, we build our application using npm run build and then we run the schematic using the Schematics CLI. We are telling the CLI to run the my-collection in the current collection folder (schematics \<path\>:\<schematic-name\>).

Remember to compile your collection before testing it. I recommend running the command with the watch flag while developing. (npm run build -- --watch)

Logging

At the moment our schematic is not doing anything. Logging information can provide useful information to the user or can help us while debugging. Let's change our schematic to log some information.

// src/my-collection/index.ts
export function myCollection(_options: any): Rule {
  return (tree: Tree, context: SchematicContext) => {
    context.logger.info('Info message');
    context.logger.warn('Warn message');
    context.logger.error('Error message');

    return tree;
  };
}

If we run our schematic now (remember to build first), we'll get a nice colored output.

Creating files

Now that we know how to log some information we should start modifying our tree. We'll start by creating a schematic to add a new file.

schematics blank create-file

Our new schematic has been added to the collection, you'll notice that a new folder is created and collection.jsonis modified to include the new schematic. We can now update the createFile method to modify the tree.

// src/create-file/index.ts
import { Rule, SchematicContext, Tree } from "@angular-devkit/schematics";

export function createFile(_options: any): Rule {
  return (tree: Tree, _context: SchematicContext) => {
    tree.create("test.ts", "File created from schematic!");
    return tree;
  };
}

Our rule is taking a tree, then creating a new file test.ts at the root of that tree and then returning the modified tree.

Build and run.

schematics .:create-file

Success? If you look inside your current folder you won't be able to find the expected file. That's because our schematic will run by default in debug mode when called from a relative path. This means it will not make modifications to the file system. To make changes add the --debug=false option to the command. Try again. test.js is finally created with the desired content in it. Delete the file before going forward.

If we run the command again, it will fail because create will not overwrite a file that exists.

Arguments and schemas

Our schematic is very limited at the moment. It will always create the same file no matter what. Wouldn't it be better if we could pass some arguments to it? In order to do that we will define a schema. Let's create a schema.json file inside our create-file schematic folder. We will also create an interface schema.ts that matches our declared arguments in the .json file.

{
  "$schema": "http://json-schema.org/schema",
  "id": "my-collection-create-file",
  "title": "Creates a file using the given path",
  "type": "object",
  "properties": {
    "path": {
      "type": "string",
      "description": "The path of the file to create."
    }
  },
  "required": ["path"]
}

// src/create-file/schema.ts
export interface CreateFileOptions{
  path:string;
}

We declared a new argument named path and make it required on schema.json.
Let's add the schema and interface to our schematic function and our collection schema.

import { Rule, SchematicContext, Tree } from '@angular-devkit/schematics';
import { CreateFileOptions } from './schema';

export function createFile(options: CreateFileOptions): Rule {
  return (tree: Tree, _context: SchematicContext) => {
    tree.create(options.path, "File created from schematic!");
    return tree;
  };
}

{
  "$schema": "../node_modules/@angular-devkit/schematics/collection-schema.json",
  "schematics": {
    "my-collection": {
      "description": "A blank schematic.",
      "factory": "./my-collection/index#myCollection"
    },
    "create-file": {
      "description": "A blank schematic.",
      "factory": "./create-file/index#createFile",
      "schema": "./create-file/schema.json"
    }
  }
}

By creating an interface we've added type inference to the options argument of our schematic factory function. schema property must be added to the schematic declaration in collection.json with the path to the schema file.
This schema will tell the schematic CLI to fail if the path argument is not entered.

Let's build it and run it.

schematics .:create-file

If we run our schematic without passing a path argument, it will fail.

Let's run it again passing an argument.

schematics .:create-file --path=test-path.ts

Even though the schema and interface are not required. They provide useful validation, parsing and type checking. The ng generate command will also read the schema and display the available arguments when running with the --help flag.

Prompting the user and aliases

It might be difficult to remember all the arguments that a schematic can get. We can make our schematic more user-friendly by prompting for required arguments. We can make our commands shorter too by adding aliases to our schematics. We will do that by adding some properties to our schemas.

First, lest's add x-prompt to the path property in schema.json. The user will be prompted with its value if the argument is missing.

{
  "$schema": "http://json-schema.org/schema",
  "id": "my-collection-create-file",
  "title": "Creates a file using the given path",
  "type": "object",
  "properties": {
    "path": {
      "type": "string",
      "description": "The path of the file to create.",
      "x-prompt": "Enter the file path:",
    }
  },
  "required": ["path"]
}

To create an alias, add the aliases property to the schematic in collection.json

{
  "$schema": "../node_modules/@angular-devkit/schematics/collection-schema.json",
  "schematics": {
    "my-collection": {
      "description": "A blank schematic.",
      "factory": "./my-collection/index#myCollection"
    },
    "create-file": {
      "description": "A blank schematic.",
      "factory": "./create-file/index#createFile",
      "schema": "./create-file/schema.json",
      "aliases": ["cf"]
    }
  }
}

Now we can run our schematic using the new alias cf and if we forget to add the path argument, we will be prompted for it.

schematics .:cf

Templates

Passing arguments to change the path of the file to create is fine, but our content is always the same. Passing the content as an argument wouldn't be practical as it can get very complex. Fortunately, we can create templates when dealing with this kind of content. Templates are nothing more than placeholder files that will be copied, moved or modified in our tree.
Let's create a new schematic and apply some of the concepts that we've seen before.

schematics blank create-from-template

collection.json (partial)

{
  "$schema": "../node_modules/@angular-devkit/schematics/collection-schema.json",
  "schematics": { 
    "create-from-template": {
      "description": "A blank schematic.",
      "factory": "./create-from-template/index#createFromTemplate",
      "schema": "./create-from-template/schema.json",
      "aliases": ["cft"]
    }
  }
}

// src/create-from-template/schema.ts
export interface CreateFromTemplateOptions {
  folder: string;
}

// src/create-from-template/schema.json
{
  "$schema": "http://json-schema.org/schema",
  "id": "my-collection-create-from-template",
  "title": "Creates files in the given folder",
  "type": "object",
  "properties": {
    "folder": {
      "type": "string",
      "description": "The destination folder of the files to create.",
      "x-prompt":"Enter the destination folder:"
    }
  },
  "required": ["folder"]
}

To create our templates we must create a /files folder inside our schematic folder, and place the files to be copied.

You can use any folder name as long as it is ignored by the compiler. /files is ignored by default

We've added two files inside our folders.
Now we can make use of them inside our schematic.

// src/create-from-template/index.ts
import {
  Rule,
  SchematicContext,
  Tree,
  Source,
  url,
  mergeWith,
  move,
  apply
} from "@angular-devkit/schematics";
import { CreateFromTemplateOptions } from "./schema";
import { normalize } from "@angular-devkit/core";

export function createFromTemplate(options: CreateFromTemplateOptions): Rule {
  return (tree: Tree, context: SchematicContext) => {
    const source: Source = url("./files");
    const transformedSource: Source = apply(source, [
      move(normalize(options.folder))
    ]);

    return mergeWith(transformedSource)(tree, context);
  };
}

There's a lot going on in here. First, we are reading from our files folder using the url function that is returning a Source. Then we are applying a set of rules, to each one of the files from the source. In this example, we are moving our files from the root level, inside the folder given as an argument. Finally, we are merging the modified source with the initial tree.

Build and run.

Our files have been copied from our /files folder to /my-folder

Dynamic content

Let's take a step back and think of real examples of schematics. When we create a component using the Angular CLI, a set of files and a folder is created. Those files change their name and content depending on the input that the user gives. How can we achieve something similar?
We'll make use of the template function provided angular-devkit/schematics and apply it to our source.

// ...imports
export function createFromTemplate(options: CreateFromTemplateOptions): Rule {
  return (tree: Tree, context: SchematicContext) => {
// ...
    const transformedSource: Source = apply(source, [
      template({
        filaname: options.folder,
        ...strings // dasherize, classify, camelize, etc
      }),
      move(normalize(folder))
    ]);

    return mergeWith(transformedSource)(tree, context);
  };
}

template will take an object as an argument and make all of its properties available to filenames and the templates. We are passing the unformatted folder name and a set of methods for transforming strings.
To test this we'll create two new files.

// files/__filename@dasherize__.ts
export class <%= classify(filename) %> {
    constructor(){}
}

<!-- files/__filename@dasherize__.html -->
<ul>
  <li><%= dasherize(filename) %></li>
  <li><%= camelize(filename) %></li>
  <li><%= capitalize(filename) %></li>
  <li><%= underscore(filename) %></li>
</ul>

Look at those filenames. __ is the default start and end delimiter, @ will pass the argument (before the symbol) to a function (after the symbol). In this example, the filename will be passed as an argument to the dasherize function and the value returned will be followed by the file extension.
Inside our templates, we will continue using our string functions and the filename property.
Once again... build and run.

schematics .:cft --folder=very-complexFolder_name

Our filenames have been formatted the way we wanted to (dasherized). Let's take a look at the content of the files. (You need to run the command with --debug=false to actually make changes and be able to see them)

<!-- very-complex-flder-name.html -->
<ul>
  <li>very-complex-folder-name</li>
  <li>veryComplexFolderName</li>
  <li>Very-complexFolder_name</li>
  <li>very_complex_folder_name</li>
</ul

// very-complex-folder-name.ts
export class VeryComplexFolderName {
    constructor(){}
}

Our schematic it's starting to look much more useful!

Deleting files

Next on the list is deleting files.

Let's add a new schematic.

schematics blank --name=remove-file

collection.json

{
  "$schema": "../node_modules/@angular-devkit/schematics/collection-schema.json",
  "schematics": {
    "remove-file": {
      "description": "Removes a file",
      "factory": "./remove-file/index#removeFile",
      "schema": "./remove-file/schema.json",
      "aliases": ["rm"]
    }
  }
}

schema.json

{
  "$schema": "http://json-schema.org/schema",
  "id": "my-collection-remove-file",
  "title": "Deletes a file using the given path",
  "type": "object",
  "properties": {
    "path": {
      "type": "string",
      "description": "The path of the file to remove.",
      "x-prompt":"Enter the file path:"
    }
  },
  "required": ["path"]
}

// src/remove-file/schema.ts
export interface RemoveFileOptions {
  path: string;
}

// src/remove-file/index.ts
import { Rule, SchematicContext, Tree } from '@angular-devkit/schematics';
import { RemoveFileOptions } from './schema';

export function removeFile(options: RemoveFileOptions): Rule {
  return (tree: Tree, _context: SchematicContext) => {
    tree.delete(options.path);
    return tree;
  };
}

Before running the schematic make sure the file you'll try to delete exists. Remember that by default we'll be running in debug mode so no changes will be applied.

schematics .:rm --path=src/collection.json

Updating files

Deleting files seemed a lot simpler than creating files. I left file updates last because (in my opinion) it involves the most complex operations, depending on what kind of files and how you're updating them. It can be as simple as adding a few lines at the top (or the bottom), or as complex as having to use the typescript language AST (Abstract Syntax Tree) to determine where and how to perform the update.

Let's create a new schematic.

schematics blank overwrite-file

// src/overwrite-file
export function overwriteFile(options: OverwriteFileOptions): Rule {
  return (tree: Tree, _context: SchematicContext) => {
    const buffer = tree.read(options.path);
    const content = buffer ? buffer.toString() : '';
    const comment = `// ¯\_(ツ)_/¯\n`;
    if(!content.includes(comment)){
      tree.overwrite(options.path, comment + content)
    }
    return tree;
  };
}

I've omitted the schema creation and collection update as it is similar to what we've been doing with the rest of schematics.
Our function reads a file from a path given by the user. Converts that buffer to a string, check if a comment has been already added, and prepend it if not. Then we overwrite the file with the updated content. We could add some checks to avoid adding a comment to a .json file (and making it invalid) but that's out of the scope of this tutorial. This is not the only way to update a file.

schematics blank update-recorder

export function updateRecorder(options: RecorderOptions): Rule {
  return (tree: Tree, _context: SchematicContext) => {

    const comment = '// ᕙ(⇀‸↼‶)ᕗ\n';

    const updateRecorder: UpdateRecorder = tree.beginUpdate(options.path);

    updateRecorder.insertLeft(0, comment);
    updateRecorder.insertLeft(0, comment);
    updateRecorder.insertLeft(0, comment);
    updateRecorder.insertLeft(0, comment);

    tree.commitUpdate(updateRecorder);

    return tree;
  };
}

This is doing something similar to our previous schematic, but it works a little bit different. First, we read our file and then we start updating inserting values (string or Buffer) to the left or right of a given position. The changes are only applied to the tree after committing them.
The interesting part here is the position given to insert values and how to determine where do we want to make modifications. In this example, it doesn't matter that much because we are adding comments at the very start, but now we will see a more complex scenario. Don't forget to build and run!

Using the typescript AST

Let's say we want to read a typescript file, get the first interface declared on it and add a property at the beginning and another one at the end. We could try to read everything as text and find the right characters. We can make use of the TypeScript AST to navigate our files, not thinking in characters, but in nodes that have a meaning.

schematics blank ts-ast

import { Rule, SchematicContext, Tree, SchematicsException } from '@angular-devkit/schematics';
import * as ts from 'typescript';

export function tsAst(options: TsAstOptions): Rule {
  return (tree: Tree, _context: SchematicContext) => {
    const buffer = tree.read(options.path);
    if(!buffer){
      throw new SchematicsException(`File ${options.path} not found.`);
    } 

    const source = ts.createSourceFile(options.path, buffer.toString(), ts.ScriptTarget.Latest, true);
    const nodes = getSourceNodes(source);

    const interfaceDeclaration = nodes.find(n=>n.kind === ts.SyntaxKind.InterfaceDeclaration);
    if(!interfaceDeclaration){
      throw new SchematicsException(`No Interface found`);
    }

    const [openBrace, closeBrace] = [
      interfaceDeclaration!.getChildren().find(n=>n.kind===ts.SyntaxKind.OpenBraceToken),
      interfaceDeclaration!.getChildren().slice().reverse().find(n=>n.kind===ts.SyntaxKind.CloseBraceToken),
    ]

    const text = interfaceDeclaration!.getText();
    let indentation;
    const matches = text.match(/\r?\n\s*/);
    if (matches && matches.length > 0) {
      indentation = matches[0]
    } else {
      indentation= ''
    }

    const recorder = tree.beginUpdate(options.path);
    recorder.insertRight(openBrace!.end, `${indentation}first: string;`);
    recorder.insertLeft(closeBrace!.pos, `${indentation}last: string;`);
    tree.commitUpdate(recorder);
    return tree;
  };
}


// taken from angular schematics. returns an array of Nodes
function getSourceNodes(sourceFile: ts.SourceFile): ts.Node[] {
  const nodes: ts.Node[] = [sourceFile];
  const result = [];

  while (nodes.length > 0) {
    const node = nodes.shift();

    if (node) {
      result.push(node);
      if (node.getChildCount(sourceFile) >= 0) {
        nodes.unshift(...node.getChildren());
      }
    }
  }

  return result;
}

Let's go through this file step by step.
First we make sure the file exists, otherwise, we throw an error.
Then using the typescript compiler, we read the file and get all the nodes from the AST.
Then we look for the first node of type InterfaceDeclaration. The list of types is very extense, but this one will return the whole declaration of the interface, until the closing brace. With every node, we get the start and end positions. (Do you remember the insertLeft and insertRight methods?)
We are not there yet, we can get more information from this isolated node. We'll start by getting its child nodes and get the first occurrence of an opening brace ({) and the last closing brace (}). These nodes also contain a start and an end position. Whitespace has no 'meaning' in our syntax tree, so we'll take a different approach. We'll get the text from our interface declaration and get the white space as a way to determine the indentation. This is for aesthetic reasons and readability, a new line would work.
Now it's time to start our recording, insert to the right of our opening brace and to the left of our closing brace. Remember that no changes have been committed until we call the commitUpdate method. Even if we inserted something after the opening brace, our closing brace position is still the same as before and we can safely insert to the left of it.

Create a test file and run the schematic to test it.

interface TestInterface {
  aProperty: string;
}

After running the schematic you should have an updated file similar to this.

interface TestInterface {
  first:string;
  aProperty: string;
  last:string;
}

Success!

Final words

Event though schematics are widely used in the Angular ecosystem they are not limited to it. In fact, everything we've done so far has been used outside an angular project.
In the next part of this series, we will learn about tasks and how to test schematics, extend them and how to run a sequence of schematics.
In Part 3, we'll create a 'real world' schematic collection that will add TailwindCSS to an Angular project.
You can find the final code in this repository

References

Top comments (3)

sundarlearns • Dec 1 '19

Hello @flakolefluk , I have been going through lots of articles and references about Schematics for past 1 week includes all the links you provided in your references. But today only I got chance to read through this post. Its very simple and summaries everything. Great Job! Keep it up.