Enhance TypeScript Code Samples with Compiler Diagnostic

Build-time Type Annotation and Syntax Highlighting using Twoslash and Shiki

  • Published at 
  • 10+ mins reading time

The main reason I call my blog project Paper is because I want it to be a pleasant reading platform. I enjoy my kindle but reading from real book and actual paper feels like a better experience for me. When I built my blog platform, I kept thinking how to improve digital reading experience to compensate for lack of paper-like feeling for both hand and eyes.

1 of the thing that I always wanted to do is adding some sort of annotation in the code block to show where the error/warning occurs. As my blog content is primarily about software engineering and technical topic, it would be great if I can provide additional context directly on the code itself.

When I came across Andrew Branch's amazing blog post I thought wow someone finally did it! The only thing that I didn't like is the fact that the blog ships with TypeScript compiler to make the code samples editable and fully interactive.

It's not something that I need. I just want to surface type annotation and error information in my code samples.

Fast forward a few months, Orta open sourced Twoslash as part of new TypeScript website built using Gatsby. It's like a dream come true. I can use Twoslash to generate type annotation in build time. No need to ship TypeScript compiler in client-side!

Twoslash

Using Twoslash by itself is pretty straightforward, you can import twoslasher from @typescript/twoslash, call it with the code that you want to typecheck and pass an extension in the second argument.

import { twoslasher } from '@typescript/twoslash';
const result = twoslasher(code, extension);
// do something with these
result.errors;
result.staticQuickInfos;

That's basically it.

The hard part is integrating with the build system, adding external types and syntax highlighting as we'll see later.

The important things from result are errors and staticQuickInfos which list the error and type information respectively alongside line and column information. These two properties then can be used to create annotation in HTML.

Error Handling

By default Twoslash will throw an error if your code sample is not type safe. This is really useful to make sure that your code examples are error-free. I caught few from my previous articles.

If you want to show/annotate error you have to explicitly define it in your code block by adding comment hint with expected error code, like this:

// @errors: 2339

If you have multiple expected error codes, you can separate them by space:

// @errors: 2339 2340

You might be wondering how to know the error code. The answer is actually pretty simple: you let Twoslash fail when you first write your code samples. From there you decide whether the error is expected to be displayed to user or it's the one you need to fix.

You can see the error code directly in the error message.

After you add the comment instead of throwing error in build time, you can use result.errors to display the error in the frontend.

Custom compilerOptions

If you want more customization in how your code is typechecked you can set different compiler options. When using typescript you can use tsconfig.json but here you'll need to add this manually in every code block using comment hint. It uses @ prefix similar to error handling above.

For example, if you want to disable noImplicitAny check, you do this by adding // @noImplicitAny: false comment at the top of your code:

// @noImplicitAny: false
// your actual code starts here
const a: Record<string, string> = {};

Hiding All the Cruft

If you combine all the comment hints above you might end up with a long code block that user most certainly don't care about.

To solve this problem, Twoslash provides a method to remove specific part of code and it will return the clean code using ---cut--- comment:

// @errors: 2339
// @noImplicityAny: false
// inline type definition that you don't want to show
// to user
type A = Record<string, string>;
// ---cut---
// your code here
const a: A = {};

When the code above is passed through Twoslash, alsongside with error information and type annotation, you also get clean code inside results.code. You should use this property to make sure that the code sample you show to the reader is free of Twoslash specific comment or inline annotations.

After you run it through Twoslash, you'll get something like this:

// your code here
const a: A = {};

Framework Integration

If you're using Gatsby, you can use gatsby-remark-shiki-twoslash and it works for typical scenario. There's a few things that's missing from the implementation though like dark/light mode support, integration with external types, and some minor UX adjustment.

You can read the rest of the article on how I implement these. I'm not using Gatsby for my blog but fortunately I'm using the same markdown parser: Remark. My implementation is basically a fork of that module specifically customized to suit my needs.

Dark/Light Mode Support

As the compilation happen on build time, it's also the best time to move the syntax higlighting logic to build time as well. This way we don't ship any JS in client-side just to highlight a piece of code. This also means that the dark/light mode should be supported in build time as well.

The new TypeScript website uses Shiki as syntax highlighter which is really good because it's based on VSCode TextMate which means any VSCode theme will works! Unfortunately, using Shiki means you can only use single theme in your blog.

Fortunately I able to work around this because I created my own vscode theme with dark and light mode support. This means the difference betweeen dark and light theme is how the syntax token maps to specific color pallete.

Here I'm using loadTheme and pass it to getHighlighter to highlight the code with my own theme.

import { loadTheme, getHighlighter } from 'shiki';
async function highlight(code: string, language: string) {
  const themePath = require.resolve(
    '@pveyes/aperture/themes/Large-Aperture.json'
  );
  const theme = await loadTheme(themePath);
  const highlighter = await getHighlighter({ theme });
  const tokens = highlighter.codeToThemedTokens(code, language);
  return generateHTMLFromTokens(tokens);
}

Shiki can return highlighted code string directly using highlighter.codeToHtml but I decided to use highlighter.codeToThemedToken instead because I want to build my own custom HTML to support dark/light mode.

If you use highlighter.codeToHtml you'll get something like <span style="color: #000"> which works for basic syntax highlighting, but because the color is static, it can't be switched in runtime. The solution is to do reverse mapping and use CSS variables (custom properties) instead.

This is why I said I was lucky to own my syntax highlighter. I added 2 JSON file containing key value pair of the color and where it's used, and publish them to NPM so I can use it as module.

The JSON looks something like this

{
  "text": "#3d383d",
  "function": "#0091cd",
  "string": "#689b1f"
}

Then, I import the JSON, do reverse lookup based on the token colors, and replace them with CSS variables

// @ts-ignore
import pallete from '@pveyes/aperture/themes/pallete-dark.json';
function generateHTMLFromTokens(tokens: Token[]): string {
  let html = '';
  tokens.forEach((token) => {
    const cssVar = replaceColorToCSSVariable(token.color);
    html += `<span style="color: ${cssVar}">${token.content}</span>`;
  });
  return html;
}
const colors = Object.keys(pallete);
function replaceColorToCSSVariable(color: string) {
  const key = colors.find((key) => pallete[key] === color);
  if (!key) {
    // something wrong
  }
  // prefix the variable with syntax-
  return `--var(syntax-${key})`;
}

In client, I use the same JSON and extract them as CSS variables in the :root selector.

import { createGlobalStyle } from 'styled-components';
// @ts-ignore
import lightPallete from '@pveyes/aperture/themes/pallete-light.json';
// @ts-ignore
import darkPallete from '@pveyes/aperture/themes/pallete-dark.json';
export default createGlobalStyle`
  :root {
    ${extractToCSSVariables(lightPallete)}
  }
  @media (prefers-color-scheme: dark) {
    :root {
      ${extractToCSSVariables(darkPallete)}
    }
  }
`;
function extractToCSSVariables(pallete: Record<string, string>) {
  return Object.keys(pallete)
    .map((key) => {
      const value = pallete[key];
      return `--syntax-${key}: ${value};`;
    })
    .join('\n');
}

That's not the actual implementation because I intentionally left out part to implement proper dark mode toggle. That's for another article, but you get the idea. I know it's a lot of moving parts just to add syntax highlighting support, but the result is pretty impresive. I was able to shave 17kB of client-side JS bundle while at the same time improving the syntax highlighting capability!

It highlights code better compared to other client-side syntax highlighter that I've tried so far.

Markdown Integration

In gatsby-remark-shiki-twoslash the convention to enable typecheck is by adding twoslash metadata to your code block.

```ts twoslash
// code
const x = 4;
```

Personally I don't like it as I need to annotate almost all my code samples. Another consideration was that I want most of my code sample to be properly typechecked to prevent typo or any other error.

This means I have to change the default to be always typecheck and only skip if it find certains metadata.

I decided to use ignore metadata as it reads better.

```ts ignore
// will not be typechecked
```

Like I said previously, I'm using Remark to parse markdown files to HTML. I won't explain too much about it here, think of it like Babel for markdown files. It operates on AST level, and you can create plugin to add your own transformation.

To implement the validation above, we can create new plugin by using visitor to code element which contains our code samples. The visitor callback will get executed every time it finds code node.

We can then check both node.lang and node.meta whether a code block can be typechecked and contains a metadata or not.

import visit from 'unist-util-visit';
export default function plugin(options) {
  return transform;
  function transform(node: Node, options: any) {
    visit(node, 'code', visitor);
    return node;
  }
}
function visitor(node: RichNode) {
  if (!node.lang) {
    return;
  }
  // run typecheck by default and use ```ts ignore to explicitly ignore
  const isTypescript = node.lang === 'ts' || node.lang === 'tsx';
  const disableTypecheck = node.meta?.includes('ignore');
  if (disableTypecheck || !isTypescript) {
    // only highlight
  }
  // typecheck, then highlight
}

External Types Integration

By default Twoslash doesn't do anything to imported modules. It doesn't try to read types from package.json types/typings fields, or even @types dependency. All of these must be done manually.

This means if you run Twoslash using twoslash(code, extension) and your code contains import to 3rd party npm modules, you'll get error, like this:

import _ from 'lodash';
Cannot find module 'lodash' or its corresponding type declarations.

To fix this, you have to run Twoslash using different signature by including more type declarations into the fsMap.

fsMap is a virtual file system that the typescript compiler runs inside Twoslash. You can use createDefaultMapFromNodeModules that contains all default standard type definitions from typescript, like dom, etc.

import ts from 'typescript';
import { twoslasher } from '@typescript/twoslash';
import { createDefaultMapFromNodeModules } from '@typescript/vfs';
const fsMap = createDefaultMapFromNodeModules({
  target: ts.ScriptTarget.ES2015,
});
const result = twoslasher(code, extension, { fsMap });

After that, we'll need to add manually all external .d.ts declarations one by one. I decided to use regex to get all import declaration and use 2 heuristic to add external types definition: included typings and external typings.

function getAllImportedModules(code: string): Array<string> {
  return code
    .split('\n')
    .filter((line) => {
      return /^import.*from(.*)/.test(line);
    })
    .map((importLine) => {
      return importLine.replace(/^import.*from ('|")(.*)('|");?/, '$2').trim();
    });
}

For any heuristic that we use, assuming we have a mod packagge, we have to store the type definition inside /node-modules/@types/mod/index.d.ts in the virtual file system.

This means we have to follow typescript convention on module naming. For scoped package, we're using scope__name instead of package name, for example @pveyes/use-less becomes /node-modules/@types/pveyes__useless/index.d.ts and so on.

const typedModulePath = mod.replace('@', '').replace('/', '__');

The first heuristic is done by using types/typings property from package.json. So, assuming mod is the name of imported module I can read its package.json, check whether they have type definition included in the published package

try {
  // try reading from included types
  const pkgJsonPath = require.resolve(`${mod}/package.json`);
  const pkg = require(pkgJsonPath);
  const types = pkg.types || pkg.typings;
  if (types) {
    const dtsPath = path.join(pkgJsonPath, '..', types);
    const dts = fs.readFileSync(dtsPath, { encoding: 'utf-8' });
    fsMap.set(`/node_modules/@types/${typedModulePath}/index.d.ts`, dts);
  }
} catch (err) {
  // ignore error
}

The 2nd one is by checking @types/* dependency. We can assume that the type definitions are provided in index.d.ts file:

try {
  // try reading from @types/*/index.d.ts dependency
  const atTypesDtsPath = require.resolve(
    `@types/${typedModulePath}/index.d.ts`
  );
  const atTypesDts = fs.readFileSync(atTypesDtsPath, { encoding: 'utf-8' });
  fsMap.set(`/node_modules/@types/${typedModulePath}/index.d.ts`, atTypesDts);
} catch (err) {
  // ignore error
}

With fsMap populated with external types, Twoslash now works as you'd normally expect.

Other UX improvements

My gatsby-remark-shiki-twoslash fork that I made also have additional features that I won't explain in detail here because it will be too long.

  • Syntax Highlight support for type annotation.

This is done by re-running the highlighting function to the provided type annotation before serializing to HTML. Hover around this code block:

import React from 'react';
type Something = Record<string, any>;
const [state, setState] = React.useState<Something>({ a: '1' });
  • Hover & non hover support for error message.

These piece of code contains an error that you can inspect in desktop by hovering over the code with red squiggly lines.

In mobile where there's no hover state, you can see the error directly below the problematic line.

function square(x: number) {
  return x * 2;
}
square('not a number');
Argument of type 'string' is not assignable to parameter of type 'number'.

This neat trick is implemented using hover media feature

  • Container-width aware tooltip

Following VSCode behavior, when hovering type declaration, the tooltip message is rendered inline as wide as its content unless it's wider than the container.

Try highlighting .replace method below and compare it to the str declaration:

const str = 'string';
const newStringWithLongName = str.replace('test', '');

The great thing about Twoslash is that it already does much of the heavy lifting for you and gatsby-remark-shiki-twoslash is a really solid reference. I'm confident you can build this yourself that's suited to your liking. Feel free to ping me on twitter if you're stuck.

Happy hacking!

Categorized under

Webmentions

If you think this article is helpful
© 2023 Fatih Kalifa. All rights reserved.