lezer/website
1
0
Fork 0
Code Issues Pull requests Activity

Trouble Building Website #8

New issue
Closed
opened 2021-10-26 01:11:36 +02:00 by ashtonsix · 2 comments
ashtonsix commented 2021-10-26 01:11:36 +02:00 (Migrated from github.com)
Copy link

Hi, Lezer looks amazing! Thank you for building it!

Last week I began work on a tool for creating programming languages, and selected Lezer for generating parsers.

I have found the reference manual to be very useful, but a bit long. And so I cloned this repository with the intention of adding a table of contents to it, which would let me jump straight to the relevant class when I wanted to look something up instead of scrolling around a bunch.

When I ran the documented build steps:

npm install
node src/index.js

I encountered this error:

ENOENT: no such file or directory, open '/home/ashtonsix/workspaces/lezer/website/node_modules/@lezer/common/CHANGELOG.md'

I realised this error occured because what was on NPM didn't match what was on GitHub, and npm install was getting the wrong stuff. I remediated the error like so:

cd ..
git clone https://github.com/lezer-parser/common.git
cd website
npm link ../common

I then had to repeat this for @lezer/lr and @lezer/generator. I believe either the NPM packages should be modified to include the dependencies for the website, or this link step should be documented.

After linking the dependencies and re-running node src/index.js I encountered this error:

/home/ashtonsix/workspaces/lezer/website/src/mapdir.js:33
    throw e
    ^

Error: Unknown type 'Parser'
    at linkType (/home/ashtonsix/workspaces/lezer/website/node_modules/builddocs/src/builddocs.js:137:15)
    at Object.eval [as type] (eval at evaluate (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:193:10), <anonymous>:134:13)
    at Mold.dispatch (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:42:26)
    at Object.eval [as class] (eval at evaluate (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:193:10), <anonymous>:31:12)
    at Mold.dispatch (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:42:26)
    at Object.eval [as item] (eval at evaluate (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:193:10), <anonymous>:13:12)
    at /home/ashtonsix/workspaces/lezer/website/node_modules/builddocs/src/builddocs.js:53:24
    at String.replace (<anonymous>)
    at exports.build (/home/ashtonsix/workspaces/lezer/website/node_modules/builddocs/src/builddocs.js:52:16)
    at /home/ashtonsix/workspaces/lezer/website/src/buildref.js:20:16

I attached a debugger to node_modules/builddocs/src/builddocs.js:137 to take a closer look:

debug session

Here are the relevant variables as they're defined at the breakpoint:

link = undefined

type = {
  type: "Parser",
  typeSource: "node_modules/@lezer/common/dist/parse.d.ts",
}

config = {
  name: "lr",
  anchorPrefix: "lr.",
  main: "/home/ashtonsix/workspaces/lezer/lr/src/README.md",
  allowUnresolvedTypes: false,
  breakAt: 45,
  imports: [
    type => {
      let sibling = type.typeSource && modules.find(m => type.typeSource.startsWith(m.relative))
      if (sibling) return "#" + sibling.name + "." + type.type
    },
  ],
}

env = {
  prefix: "lr.",
  linkType: function(type) {
    let link = maybeLinkType(config, items, type)
    if (!link && link !== false && !config.allowUnresolvedTypes) {
      debugger
      throw new Error("Unknown type '" + type.type + "'" + (type.loc ? " at " + type.loc.file + ":" + type.loc.line : ""))
    }
    
    return link
  },
  hasDescription: function(type) {
    if (type.description) return true
    if (type.properties) for (let prop in type.properties)
      if (env.hasDescription(type.properties[prop])) return true
    if (type.params) for (let i = 0; i < type.params.length; i++)
      if (env.hasDescription(type.params[i])) return true
    if (type.returns && type.returns.description) return true
    return false
  },
  breakType: function(type) {
    return config.breakAt != null && typeLen(type) >= config.breakAt
  },
  processType: function(type) {
    return (config.processType && config.processType(type)) || type
  },
}

items = {
  LRParser: {
    kind: "class",
    id: "LRParser",
    description: "A parser holds the parse tables for a given grammar, as generated\nby `lezer-generator`.",
    loc: {
      file: "src/parse.ts",
      line: 604,
      column: 0,
    },
    type: "class",
    extends: {
      type: "Parser",
      typeSource: "node_modules/@lezer/common/dist/parse.d.ts",
    },
    instanceProperties: {
      nodeSet: {
        kind: "property",
        id: "LRParser.nodeSet",
        description: "The nodes used in the trees emitted by this parser.",
        loc: {
          file: "src/parse.ts",
          line: 653,
          column: 2,
        },
        readonly: true,
        type: "NodeSet",
        typeSource: "node_modules/@lezer/common/dist/tree.d.ts",
      },
      configure: {
        kind: "method",
        id: "LRParser.configure",
        description: "Configure the parser. Returns a new parser instance that has the\ngiven settings modified. Settings not provided in `config` are\nkept from the original parser.",
        loc: {
          file: "src/parse.ts",
          line: 800,
          column: 2,
        },
        type: "Function",
        signatures: [
          {
            type: "function",
            params: [
              {
                id: "LRParser.configure^config",
                kind: "parameter",
                type: "ParserConfig",
                typeSource: "src/parse.ts",
                loc: {
                  file: "src/parse.ts",
                  line: 803,
                  column: 12,
                },
                name: "config",
              },
            ],
            returns: {
              type: "LRParser",
              typeSource: "src/parse.ts",
            },
          },
        ],
      },
      getName: {
        kind: "method",
        id: "LRParser.getName",
        description: "Returns the name associated with a given term. This will only\nwork for all terms when the parser was generated with the\n`--names` option. By default, only the names of tagged terms are\nstored.",
        loc: {
          file: "src/parse.ts",
          line: 832,
          column: 2,
        },
        type: "Function",
        signatures: [
          {
            type: "function",
            params: [
              {
                id: "LRParser.getName^term",
                kind: "parameter",
                type: "number",
                loc: {
                  file: "src/parse.ts",
                  line: 836,
                  column: 10,
                },
                name: "term",
              },
            ],
            returns: {
              type: "string",
            },
          },
        ],
      },
      topNode: {
        kind: "property",
        id: "LRParser.topNode",
        description: "The type of top node produced by the parser.",
        loc: {
          file: "src/parse.ts",
          line: 844,
          column: 2,
        },
        readonly: true,
        type: "NodeType",
        typeSource: "node_modules/@lezer/common/dist/tree.d.ts",
      },
    },
  },
  ParserConfig: {
    kind: "interface",
    id: "ParserConfig",
    description: "Configuration options when\n[reconfiguring](#lr.LRParser.configure) a parser.",
    loc: {
      file: "src/parse.ts",
      line: 576,
      column: 0,
    },
    type: "interface",
    properties: {
      props: {
        kind: "property",
        id: "ParserConfig.props",
        description: "Node prop values to add to the parser's node set.",
        loc: {
          file: "src/parse.ts",
          line: 579,
          column: 2,
        },
        optional: true,
        type: "ReadonlyArray",
        typeArgs: [
          {
            type: "NodePropSource",
            typeSource: "node_modules/@lezer/common/dist/tree.d.ts",
          },
        ],
      },
      top: {
        kind: "property",
        id: "ParserConfig.top",
        description: "The name of the `@top` declaration to parse from. If not\nspecified, the first top rule declaration in the grammar is\nused.",
        loc: {
          file: "src/parse.ts",
          line: 581,
          column: 2,
        },
        optional: true,
        type: "string",
      },
      dialect: {
        kind: "property",
        id: "ParserConfig.dialect",
        description: "A space-separated string of dialects to enable.",
        loc: {
          file: "src/parse.ts",
          line: 585,
          column: 2,
        },
        optional: true,
        type: "string",
      },
      tokenizers: {
        kind: "property",
        id: "ParserConfig.tokenizers",
        description: "Replace the given external tokenizers with new ones.",
        loc: {
          file: "src/parse.ts",
          line: 587,
          column: 2,
        },
        optional: true,
        type: "Array",
        typeArgs: [
          {
            type: "Object",
            properties: {
              from: {
                kind: "property",
                id: "ParserConfig.tokenizers.from",
                loc: {
                  file: "src/parse.ts",
                  line: 588,
                  column: 16,
                },
                type: "ExternalTokenizer",
                typeSource: "src/token.ts",
              },
              to: {
                kind: "property",
                id: "ParserConfig.tokenizers.to",
                loc: {
                  file: "src/parse.ts",
                  line: 588,
                  column: 41,
                },
                type: "ExternalTokenizer",
                typeSource: "src/token.ts",
              },
            },
          },
        ],
      },
      contextTracker: {
        kind: "property",
        id: "ParserConfig.contextTracker",
        description: "Replace the context tracker with a new one.",
        loc: {
          file: "src/parse.ts",
          line: 589,
          column: 2,
        },
        optional: true,
        type: "ContextTracker",
        typeSource: "src/parse.ts",
        typeArgs: [
          {
            type: "any",
          },
        ],
      },
      strict: {
        kind: "property",
        id: "ParserConfig.strict",
        description: "When true, the parser will raise an exception, rather than run\nits error-recovery strategies, when the input doesn't match the\ngrammar.",
        loc: {
          file: "src/parse.ts",
          line: 591,
          column: 2,
        },
        optional: true,
        type: "boolean",
      },
      wrap: {
        kind: "property",
        id: "ParserConfig.wrap",
        description: "Add a wrapper, which can extend parses created by this parser\nwith additional logic (usually used to add\n[mixed-language](#common.parseMixed) parsing).",
        loc: {
          file: "src/parse.ts",
          line: 595,
          column: 2,
        },
        optional: true,
        type: "ParseWrapper",
        typeSource: "node_modules/@lezer/common/dist/parse.d.ts",
      },
      bufferLength: {
        kind: "property",
        id: "ParserConfig.bufferLength",
        description: "The maximum length of the TreeBuffers generated in the output\ntree. Defaults to 1024.",
        loc: {
          file: "src/parse.ts",
          line: 599,
          column: 2,
        },
        optional: true,
        type: "number",
      },
    },
  },
  ContextTracker: {
    kind: "class",
    id: "ContextTracker",
    description: "Context trackers are used to track stateful context (such as\nindentation in the Python grammar, or parent elements in the XML\ngrammar) needed by external tokenizers. You declare them in a\ngrammar file as `@context exportName from \"module\"`.\n\nContext values should be immutable, and can be updated (replaced)\non shift or reduce actions.\n\nThe export used in a `@context` declaration should be of this\ntype.",
    loc: {
      file: "src/parse.ts",
      line: 500,
      column: 0,
    },
    type: "class",
    construct: {
      kind: "constructor",
      id: "ContextTracker.constructor",
      type: "Function",
      description: "Define a context tracker.",
      loc: {
        file: "src/parse.ts",
        line: 524,
        column: 2,
      },
      signatures: [
        {
          type: "constructor",
          params: [
            {
              id: "ContextTracker.constructor^spec",
              kind: "parameter",
              type: "Object",
              properties: {
                start: {
                  kind: "property",
                  id: "ContextTracker.constructor^spec.start",
                  description: "The initial value of the context at the start of the parse.",
                  loc: {
                    file: "src/parse.ts",
                    line: 526,
                    column: 4,
                  },
                  type: "T",
                  typeParamSource: "ContextTracker^T",
                },
                shift: {
                  kind: "method",
                  id: "ContextTracker.constructor^spec.shift",
                  description: "Update the context when the parser executes a\n[shift](https://en.wikipedia.org/wiki/LR_parser#Shift_and_reduce_actions)\naction.",
                  loc: {
                    file: "src/parse.ts",
                    line: 528,
                    column: 4,
                  },
                  optional: true,
                  type: "Function",
                  signatures: [
                    {
                      type: "function",
                      params: [
                        {
                          id: "ContextTracker.constructor^spec.shift^context",
                          kind: "parameter",
                          type: "T",
                          typeParamSource: "ContextTracker^T",
                          loc: {
                            file: "src/parse.ts",
                            line: 531,
                            column: 11,
                          },
                          name: "context",
                        },
                        {
                          id: "ContextTracker.constructor^spec.shift^term",
                          kind: "parameter",
                          type: "number",
                          loc: {
                            file: "src/parse.ts",
                            line: 531,
                            column: 23,
                          },
                          name: "term",
                        },
                        {
                          id: "ContextTracker.constructor^spec.shift^stack",
                          kind: "parameter",
                          type: "Stack",
                          typeSource: "src/stack.ts",
                          loc: {
                            file: "src/parse.ts",
                            line: 531,
                            column: 37,
                          },
                          name: "stack",
                        },
                        {
                          id: "ContextTracker.constructor^spec.shift^input",
                          kind: "parameter",
                          type: "InputStream",
                          typeSource: "src/token.ts",
                          loc: {
                            file: "src/parse.ts",
                            line: 531,
                            column: 51,
                          },
                          name: "input",
                        },
                      ],
                      returns: {
                        type: "T",
                        typeParamSource: "ContextTracker^T",
                      },
                    },
                  ],
                },
                reduce: {
                  kind: "method",
                  id: "ContextTracker.constructor^spec.reduce",
                  description: "Update the context when the parser executes a reduce action.",
                  loc: {
                    file: "src/parse.ts",
                    line: 532,
                    column: 4,
                  },
                  optional: true,
                  type: "Function",
                  signatures: [
                    {
                      type: "function",
                      params: [
                        {
                          id: "ContextTracker.constructor^spec.reduce^context",
                          kind: "parameter",
                          type: "T",
                          typeParamSource: "ContextTracker^T",
                          loc: {
                            file: "src/parse.ts",
                            line: 533,
                            column: 12,
                          },
                          name: "context",
                        },
                        {
                          id: "ContextTracker.constructor^spec.reduce^term",
                          kind: "parameter",
                          type: "number",
                          loc: {
                            file: "src/parse.ts",
                            line: 533,
                            column: 24,
                          },
                          name: "term",
                        },
                        {
                          id: "ContextTracker.constructor^spec.reduce^stack",
                          kind: "parameter",
                          type: "Stack",
                          typeSource: "src/stack.ts",
                          loc: {
                            file: "src/parse.ts",
                            line: 533,
                            column: 38,
                          },
                          name: "stack",
                        },
                        {
                          id: "ContextTracker.constructor^spec.reduce^input",
                          kind: "parameter",
                          type: "InputStream",
                          typeSource: "src/token.ts",
                          loc: {
                            file: "src/parse.ts",
                            line: 533,
                            column: 52,
                          },
                          name: "input",
                        },
                      ],
                      returns: {
                        type: "T",
                        typeParamSource: "ContextTracker^T",
                      },
                    },
                  ],
                },
                reuse: {
                  kind: "method",
                  id: "ContextTracker.constructor^spec.reuse",
                  description: "Update the context when the parser reuses a node from a tree\nfragment.",
                  loc: {
                    file: "src/parse.ts",
                    line: 534,
                    column: 4,
                  },
                  optional: true,
                  type: "Function",
                  signatures: [
                    {
                      type: "function",
                      params: [
                        {
                          id: "ContextTracker.constructor^spec.reuse^context",
                          kind: "parameter",
                          type: "T",
                          typeParamSource: "ContextTracker^T",
                          loc: {
                            file: "src/parse.ts",
                            line: 536,
                            column: 11,
                          },
                          name: "context",
                        },
                        {
                          id: "ContextTracker.constructor^spec.reuse^node",
                          kind: "parameter",
                          type: "Tree",
                          typeSource: "node_modules/@lezer/common/dist/tree.d.ts",
                          loc: {
                            file: "src/parse.ts",
                            line: 536,
                            column: 23,
                          },
                          name: "node",
                        },
                        {
                          id: "ContextTracker.constructor^spec.reuse^stack",
                          kind: "parameter",
                          type: "Stack",
                          typeSource: "src/stack.ts",
                          loc: {
                            file: "src/parse.ts",
                            line: 536,
                            column: 35,
                          },
                          name: "stack",
                        },
                        {
                          id: "ContextTracker.constructor^spec.reuse^input",
                          kind: "parameter",
                          type: "InputStream",
                          typeSource: "src/token.ts",
                          loc: {
                            file: "src/parse.ts",
                            line: 536,
                            column: 49,
                          },
                          name: "input",
                        },
                      ],
                      returns: {
                        type: "T",
                        typeParamSource: "ContextTracker^T",
                      },
                    },
                  ],
                },
                hash: {
                  kind: "method",
                  id: "ContextTracker.constructor^spec.hash",
                  description: "Reduce a context value to a number (for cheap storage and\ncomparison). Only needed for strict contexts.",
                  loc: {
                    file: "src/parse.ts",
                    line: 537,
                    column: 4,
                  },
                  optional: true,
                  type: "Function",
                  signatures: [
                    {
                      type: "function",
                      params: [
                        {
                          id: "ContextTracker.constructor^spec.hash^context",
                          kind: "parameter",
                          type: "T",
                          typeParamSource: "ContextTracker^T",
                          loc: {
                            file: "src/parse.ts",
                            line: 539,
                            column: 10,
                          },
                          name: "context",
                        },
                      ],
                      returns: {
                        type: "number",
                      },
                    },
                  ],
                },
                strict: {
                  kind: "property",
                  id: "ContextTracker.constructor^spec.strict",
                  description: "By default, nodes can only be reused during incremental\nparsing if they were created in the same context as the one in\nwhich they are reused. Set this to false to disable that\ncheck (and the overhead of storing the hashes).",
                  loc: {
                    file: "src/parse.ts",
                    line: 540,
                    column: 4,
                  },
                  optional: true,
                  type: "boolean",
                },
              },
              loc: {
                file: "src/parse.ts",
                line: 525,
                column: 14,
              },
              name: "spec",
            },
          ],
        },
      ],
    },
    typeParams: [
      {
        type: "typeparam",
        kind: "typeparam",
        name: "T",
        id: "ContextTracker^T",
        loc: {
          file: "src/parse.ts",
          line: 510,
          column: 28,
        },
      },
    ],
  },
  InputStream: {
    kind: "class",
    id: "InputStream",
    description: "[Tokenizers](#lr.ExternalTokenizer) interact with the input\nthrough this interface. It presents the input as a stream of\ncharacters, tracking lookahead and hiding the complexity of\n[ranges](#common.Parser.parse^ranges) from tokenizer code.",
    loc: {
      file: "src/token.ts",
      line: 16,
      column: 0,
    },
    type: "class",
    instanceProperties: {
      next: {
        kind: "property",
        id: "InputStream.next",
        description: "The character code of the next code unit in the input, or -1\nwhen the stream is at the end of the input.",
        loc: {
          file: "src/token.ts",
          line: 31,
          column: 2,
        },
        type: "number",
      },
      pos: {
        kind: "property",
        id: "InputStream.pos",
        description: "The current position of the stream. Note that, due to parses\nbeing able to cover non-contiguous\n[ranges](#common.Parser.startParse), advancing the stream does\nnot always mean its position moves a single unit.",
        loc: {
          file: "src/token.ts",
          line: 38,
          column: 2,
        },
        type: "number",
      },
      resolveOffset: {
        kind: "method",
        id: "InputStream.resolveOffset",
        loc: {
          file: "src/token.ts",
          line: 63,
          column: 2,
        },
        type: "Function",
        signatures: [
          {
            type: "function",
            params: [
              {
                id: "InputStream.resolveOffset^offset",
                kind: "parameter",
                type: "number",
                loc: {
                  file: "src/token.ts",
                  line: 63,
                  column: 16,
                },
                name: "offset",
              },
              {
                id: "InputStream.resolveOffset^assoc",
                kind: "parameter",
                type: "union",
                typeArgs: [
                  {
                    type: "-1",
                  },
                  {
                    type: "1",
                  },
                ],
                loc: {
                  file: "src/token.ts",
                  line: 63,
                  column: 32,
                },
                name: "assoc",
              },
            ],
            returns: {
              type: "union",
              typeArgs: [
                {
                  type: "number",
                },
                {
                  type: "null",
                },
              ],
            },
          },
        ],
      },
      peek: {
        kind: "method",
        id: "InputStream.peek",
        description: "Look at a code unit near the stream position. `.peek(0)` equals\n`.next`, `.peek(-1)` gives you the previous character, and so\non.\n\nNote that looking around during tokenizing creates dependencies\non potentially far-away content, which may reduce the\neffectiveness incremental parsing—when looking forward—or even\ncause invalid reparses when looking backward more than 25 code\nunits, since the library does not track lookbehind.",
        loc: {
          file: "src/token.ts",
          line: 81,
          column: 2,
        },
        type: "Function",
        signatures: [
          {
            type: "function",
            params: [
              {
                id: "InputStream.peek^offset",
                kind: "parameter",
                type: "number",
                loc: {
                  file: "src/token.ts",
                  line: 90,
                  column: 7,
                },
                name: "offset",
              },
            ],
            returns: {
              type: "number",
            },
          },
        ],
      },
      acceptToken: {
        kind: "method",
        id: "InputStream.acceptToken",
        description: "Accept a token. By default, the end of the token is set to the\ncurrent stream position, but you can pass an offset (relative to\nthe stream position) to change that.",
        loc: {
          file: "src/token.ts",
          line: 113,
          column: 2,
        },
        type: "Function",
        signatures: [
          {
            type: "function",
            params: [
              {
                id: "InputStream.acceptToken^token",
                kind: "parameter",
                type: "number",
                loc: {
                  file: "src/token.ts",
                  line: 116,
                  column: 14,
                },
                name: "token",
              },
              {
                id: "InputStream.acceptToken^endOffset",
                kind: "parameter",
                type: "number",
                loc: {
                  file: "src/token.ts",
                  line: 116,
                  column: 29,
                },
                default: "0",
                optional: true,
                name: "endOffset",
              },
            ],
          },
        ],
      },
      advance: {
        kind: "method",
        id: "InputStream.advance",
        description: "Move the stream forward N (defaults to 1) code units. Returns\nthe new value of [`next`](#lr.InputStream.next).",
        loc: {
          file: "src/token.ts",
          line: 147,
          column: 2,
        },
        type: "Function",
        signatures: [
          {
            type: "function",
            params: [
              {
                id: "InputStream.advance^n",
                kind: "parameter",
                type: "number",
                loc: {
                  file: "src/token.ts",
                  line: 149,
                  column: 10,
                },
                default: "1",
                optional: true,
                name: "n",
              },
            ],
            returns: {
              type: "number",
            },
          },
        ],
      },
    },
  },
  ExternalTokenizer: {
    kind: "class",
    id: "ExternalTokenizer",
    description: "`@external tokens` declarations in the grammar should resolve to\nan instance of this class.",
    loc: {
      file: "src/token.ts",
      line: 253,
      column: 0,
    },
    type: "class",
    implements: [
      {
        type: "interface",
      },
    ],
    construct: {
      kind: "constructor",
      id: "ExternalTokenizer.constructor",
      type: "Function",
      description: "Create a tokenizer. The first argument is the function that,\ngiven an input stream, scans for the types of tokens it\nrecognizes at the stream's position, and calls\n[`acceptToken`](#lr.InputStream.acceptToken) when it finds\none.",
      loc: {
        file: "src/token.ts",
        line: 263,
        column: 2,
      },
      signatures: [
        {
          type: "constructor",
          params: [
            {
              id: "ExternalTokenizer.constructor^token",
              kind: "parameter",
              type: "Function",
              signatures: [
                {
                  type: "function",
                  params: [
                    {
                      id: "ExternalTokenizer.constructor^token^input",
                      kind: "parameter",
                      type: "InputStream",
                      typeSource: "src/token.ts",
                      loc: {
                        file: "src/token.ts",
                        line: 270,
                        column: 21,
                      },
                      name: "input",
                    },
                    {
                      id: "ExternalTokenizer.constructor^token^stack",
                      kind: "parameter",
                      type: "Stack",
                      typeSource: "src/stack.ts",
                      loc: {
                        file: "src/token.ts",
                        line: 270,
                        column: 41,
                      },
                      name: "stack",
                    },
                  ],
                },
              ],
              loc: {
                file: "src/token.ts",
                line: 269,
                column: 4,
              },
              name: "token",
            },
            {
              id: "ExternalTokenizer.constructor^options",
              kind: "parameter",
              type: "interface",
              properties: {
                contextual: {
                  kind: "property",
                  id: "ExternalTokenizer.constructor^options.contextual",
                  description: "When set to true, mark this tokenizer as depending on the\ncurrent parse stack, which prevents its result from being cached\nbetween parser actions at the same positions.",
                  loc: {
                    file: "src/token.ts",
                    line: 237,
                    column: 2,
                  },
                  optional: true,
                  type: "boolean",
                },
                fallback: {
                  kind: "property",
                  id: "ExternalTokenizer.constructor^options.fallback",
                  description: "By defaults, when a tokenizer returns a token, that prevents\ntokenizers with lower precedence from even running. When\n`fallback` is true, the tokenizer is allowed to run when a\nprevious tokenizer returned a token that didn't match any of the\ncurrent state's actions.",
                  loc: {
                    file: "src/token.ts",
                    line: 241,
                    column: 2,
                  },
                  optional: true,
                  type: "boolean",
                },
                extend: {
                  kind: "property",
                  id: "ExternalTokenizer.constructor^options.extend",
                  description: "When set to true, tokenizing will not stop after this tokenizer\nhas produced a token. (But it will still fail to reach this one\nif a higher-precedence tokenizer produced a token.)",
                  loc: {
                    file: "src/token.ts",
                    line: 247,
                    column: 2,
                  },
                  optional: true,
                  type: "boolean",
                },
              },
              loc: {
                file: "src/token.ts",
                line: 271,
                column: 4,
              },
              default: "{}",
              optional: true,
              name: "options",
            },
          ],
        },
      ],
    },
  },
  Stack: {
    kind: "class",
    id: "Stack",
    description: "A parse stack. These are used internally by the parser to track\nparsing progress. They also provide some properties and methods\nthat external code such as a tokenizer can use to get information\nabout the parse state.",
    loc: {
      file: "src/stack.ts",
      line: 5,
      column: 0,
    },
    type: "class",
    instanceProperties: {
      pos: {
        kind: "property",
        id: "Stack.pos",
        description: "The input position up to which this stack has parsed.",
        loc: {
          file: "src/stack.ts",
          line: 25,
          column: 4,
        },
        type: "number",
      },
      context: {
        kind: "property",
        id: "Stack.context",
        description: "The stack's current [context](#lr.ContextTracker) value, if\nany. Its type will depend on the context tracker's type\nparameter, or it will be `null` if there is no context\ntracker.",
        loc: {
          file: "src/stack.ts",
          line: 67,
          column: 2,
        },
        readonly: true,
        type: "any",
      },
      useNode: {
        kind: "method",
        id: "Stack.useNode",
        loc: {
          file: "src/stack.ts",
          line: 188,
          column: 2,
        },
        type: "Function",
        signatures: [
          {
            type: "function",
            params: [
              {
                id: "Stack.useNode^value",
                kind: "parameter",
                type: "Tree",
                typeSource: "node_modules/@lezer/common/dist/tree.d.ts",
                loc: {
                  file: "src/stack.ts",
                  line: 189,
                  column: 10,
                },
                name: "value",
              },
              {
                id: "Stack.useNode^next",
                kind: "parameter",
                type: "number",
                loc: {
                  file: "src/stack.ts",
                  line: 189,
                  column: 23,
                },
                name: "next",
              },
            ],
          },
        ],
      },
      canShift: {
        kind: "method",
        id: "Stack.canShift",
        description: "Check if the given term would be able to be shifted (optionally\nafter some reductions) on this stack. This can be useful for\nexternal tokenizers that want to make sure they only provide a\ngiven token when it applies.",
        loc: {
          file: "src/stack.ts",
          line: 233,
          column: 2,
        },
        type: "Function",
        signatures: [
          {
            type: "function",
            params: [
              {
                id: "Stack.canShift^term",
                kind: "parameter",
                type: "number",
                loc: {
                  file: "src/stack.ts",
                  line: 237,
                  column: 11,
                },
                name: "term",
              },
            ],
            returns: {
              type: "boolean",
            },
          },
        ],
      },
      parser: {
        kind: "property",
        id: "Stack.parser",
        description: "Get the parser used by this stack.",
        loc: {
          file: "src/stack.ts",
          line: 330,
          column: 2,
        },
        readonly: true,
        type: "LRParser",
        typeSource: "src/parse.ts",
      },
      dialectEnabled: {
        kind: "method",
        id: "Stack.dialectEnabled",
        description: "Test whether a given dialect (by numeric ID, as exported from\nthe terms file) is enabled.",
        loc: {
          file: "src/stack.ts",
          line: 333,
          column: 2,
        },
        type: "Function",
        signatures: [
          {
            type: "function",
            params: [
              {
                id: "Stack.dialectEnabled^dialectID",
                kind: "parameter",
                type: "number",
                loc: {
                  file: "src/stack.ts",
                  line: 335,
                  column: 17,
                },
                name: "dialectID",
              },
            ],
            returns: {
              type: "boolean",
            },
          },
        ],
      },
    },
  },
}

It appears that maybeLinkType expects items.Parser to be defined, but it is not. It is interesting to note, however, that items.LRParser is defined. When I looked at the changelog for @lezer/lr, I noticed version 0.15.0 changed the name of Parser to LRParser and I believe this change broke the build process for the website.

At this point, I decided to report my findings rather than continue. Hope this helps, and that we can get this fixed. 🙂

Hi, Lezer looks amazing! Thank you for building it! Last week I began work on a tool for creating programming languages, and selected Lezer for generating parsers. I have found the reference manual to be very useful, but a bit long. And so I cloned this repository with the intention of adding a table of contents to it, which would let me jump straight to the relevant class when I wanted to look something up instead of scrolling around a bunch. When I ran the documented build steps: ```sh npm install node src/index.js ``` I encountered this error: ```txt ENOENT: no such file or directory, open '/home/ashtonsix/workspaces/lezer/website/node_modules/@lezer/common/CHANGELOG.md' ``` I realised this error occured because what was on NPM didn't match what was on GitHub, and `npm install` was getting the wrong stuff. I remediated the error like so: ```sh cd .. git clone https://github.com/lezer-parser/common.git cd website npm link ../common ``` I then had to repeat this for `@lezer/lr` and `@lezer/generator`. I believe either the NPM packages should be modified to include the dependencies for the website, or this link step should be documented. After linking the dependencies and re-running `node src/index.js` I encountered this error: ``` /home/ashtonsix/workspaces/lezer/website/src/mapdir.js:33 throw e ^ Error: Unknown type 'Parser' at linkType (/home/ashtonsix/workspaces/lezer/website/node_modules/builddocs/src/builddocs.js:137:15) at Object.eval [as type] (eval at evaluate (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:193:10), <anonymous>:134:13) at Mold.dispatch (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:42:26) at Object.eval [as class] (eval at evaluate (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:193:10), <anonymous>:31:12) at Mold.dispatch (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:42:26) at Object.eval [as item] (eval at evaluate (/home/ashtonsix/workspaces/lezer/website/node_modules/mold-template/mold.js:193:10), <anonymous>:13:12) at /home/ashtonsix/workspaces/lezer/website/node_modules/builddocs/src/builddocs.js:53:24 at String.replace (<anonymous>) at exports.build (/home/ashtonsix/workspaces/lezer/website/node_modules/builddocs/src/builddocs.js:52:16) at /home/ashtonsix/workspaces/lezer/website/src/buildref.js:20:16 ``` I attached a debugger to `node_modules/builddocs/src/builddocs.js:137` to take a closer look: ![debug session](https://user-images.githubusercontent.com/6740947/138781820-b27e5849-4531-40a7-9cbe-2200de7dcc95.png) Here are the relevant variables as they're defined at the breakpoint: <details> ```js link = undefined type = { type: "Parser", typeSource: "node_modules/@lezer/common/dist/parse.d.ts", } config = { name: "lr", anchorPrefix: "lr.", main: "/home/ashtonsix/workspaces/lezer/lr/src/README.md", allowUnresolvedTypes: false, breakAt: 45, imports: [ type => { let sibling = type.typeSource && modules.find(m => type.typeSource.startsWith(m.relative)) if (sibling) return "#" + sibling.name + "." + type.type }, ], } env = { prefix: "lr.", linkType: function(type) { let link = maybeLinkType(config, items, type) if (!link && link !== false && !config.allowUnresolvedTypes) { debugger throw new Error("Unknown type '" + type.type + "'" + (type.loc ? " at " + type.loc.file + ":" + type.loc.line : "")) } return link }, hasDescription: function(type) { if (type.description) return true if (type.properties) for (let prop in type.properties) if (env.hasDescription(type.properties[prop])) return true if (type.params) for (let i = 0; i < type.params.length; i++) if (env.hasDescription(type.params[i])) return true if (type.returns && type.returns.description) return true return false }, breakType: function(type) { return config.breakAt != null && typeLen(type) >= config.breakAt }, processType: function(type) { return (config.processType && config.processType(type)) || type }, } items = { LRParser: { kind: "class", id: "LRParser", description: "A parser holds the parse tables for a given grammar, as generated\nby `lezer-generator`.", loc: { file: "src/parse.ts", line: 604, column: 0, }, type: "class", extends: { type: "Parser", typeSource: "node_modules/@lezer/common/dist/parse.d.ts", }, instanceProperties: { nodeSet: { kind: "property", id: "LRParser.nodeSet", description: "The nodes used in the trees emitted by this parser.", loc: { file: "src/parse.ts", line: 653, column: 2, }, readonly: true, type: "NodeSet", typeSource: "node_modules/@lezer/common/dist/tree.d.ts", }, configure: { kind: "method", id: "LRParser.configure", description: "Configure the parser. Returns a new parser instance that has the\ngiven settings modified. Settings not provided in `config` are\nkept from the original parser.", loc: { file: "src/parse.ts", line: 800, column: 2, }, type: "Function", signatures: [ { type: "function", params: [ { id: "LRParser.configure^config", kind: "parameter", type: "ParserConfig", typeSource: "src/parse.ts", loc: { file: "src/parse.ts", line: 803, column: 12, }, name: "config", }, ], returns: { type: "LRParser", typeSource: "src/parse.ts", }, }, ], }, getName: { kind: "method", id: "LRParser.getName", description: "Returns the name associated with a given term. This will only\nwork for all terms when the parser was generated with the\n`--names` option. By default, only the names of tagged terms are\nstored.", loc: { file: "src/parse.ts", line: 832, column: 2, }, type: "Function", signatures: [ { type: "function", params: [ { id: "LRParser.getName^term", kind: "parameter", type: "number", loc: { file: "src/parse.ts", line: 836, column: 10, }, name: "term", }, ], returns: { type: "string", }, }, ], }, topNode: { kind: "property", id: "LRParser.topNode", description: "The type of top node produced by the parser.", loc: { file: "src/parse.ts", line: 844, column: 2, }, readonly: true, type: "NodeType", typeSource: "node_modules/@lezer/common/dist/tree.d.ts", }, }, }, ParserConfig: { kind: "interface", id: "ParserConfig", description: "Configuration options when\n[reconfiguring](#lr.LRParser.configure) a parser.", loc: { file: "src/parse.ts", line: 576, column: 0, }, type: "interface", properties: { props: { kind: "property", id: "ParserConfig.props", description: "Node prop values to add to the parser's node set.", loc: { file: "src/parse.ts", line: 579, column: 2, }, optional: true, type: "ReadonlyArray", typeArgs: [ { type: "NodePropSource", typeSource: "node_modules/@lezer/common/dist/tree.d.ts", }, ], }, top: { kind: "property", id: "ParserConfig.top", description: "The name of the `@top` declaration to parse from. If not\nspecified, the first top rule declaration in the grammar is\nused.", loc: { file: "src/parse.ts", line: 581, column: 2, }, optional: true, type: "string", }, dialect: { kind: "property", id: "ParserConfig.dialect", description: "A space-separated string of dialects to enable.", loc: { file: "src/parse.ts", line: 585, column: 2, }, optional: true, type: "string", }, tokenizers: { kind: "property", id: "ParserConfig.tokenizers", description: "Replace the given external tokenizers with new ones.", loc: { file: "src/parse.ts", line: 587, column: 2, }, optional: true, type: "Array", typeArgs: [ { type: "Object", properties: { from: { kind: "property", id: "ParserConfig.tokenizers.from", loc: { file: "src/parse.ts", line: 588, column: 16, }, type: "ExternalTokenizer", typeSource: "src/token.ts", }, to: { kind: "property", id: "ParserConfig.tokenizers.to", loc: { file: "src/parse.ts", line: 588, column: 41, }, type: "ExternalTokenizer", typeSource: "src/token.ts", }, }, }, ], }, contextTracker: { kind: "property", id: "ParserConfig.contextTracker", description: "Replace the context tracker with a new one.", loc: { file: "src/parse.ts", line: 589, column: 2, }, optional: true, type: "ContextTracker", typeSource: "src/parse.ts", typeArgs: [ { type: "any", }, ], }, strict: { kind: "property", id: "ParserConfig.strict", description: "When true, the parser will raise an exception, rather than run\nits error-recovery strategies, when the input doesn't match the\ngrammar.", loc: { file: "src/parse.ts", line: 591, column: 2, }, optional: true, type: "boolean", }, wrap: { kind: "property", id: "ParserConfig.wrap", description: "Add a wrapper, which can extend parses created by this parser\nwith additional logic (usually used to add\n[mixed-language](#common.parseMixed) parsing).", loc: { file: "src/parse.ts", line: 595, column: 2, }, optional: true, type: "ParseWrapper", typeSource: "node_modules/@lezer/common/dist/parse.d.ts", }, bufferLength: { kind: "property", id: "ParserConfig.bufferLength", description: "The maximum length of the TreeBuffers generated in the output\ntree. Defaults to 1024.", loc: { file: "src/parse.ts", line: 599, column: 2, }, optional: true, type: "number", }, }, }, ContextTracker: { kind: "class", id: "ContextTracker", description: "Context trackers are used to track stateful context (such as\nindentation in the Python grammar, or parent elements in the XML\ngrammar) needed by external tokenizers. You declare them in a\ngrammar file as `@context exportName from \"module\"`.\n\nContext values should be immutable, and can be updated (replaced)\non shift or reduce actions.\n\nThe export used in a `@context` declaration should be of this\ntype.", loc: { file: "src/parse.ts", line: 500, column: 0, }, type: "class", construct: { kind: "constructor", id: "ContextTracker.constructor", type: "Function", description: "Define a context tracker.", loc: { file: "src/parse.ts", line: 524, column: 2, }, signatures: [ { type: "constructor", params: [ { id: "ContextTracker.constructor^spec", kind: "parameter", type: "Object", properties: { start: { kind: "property", id: "ContextTracker.constructor^spec.start", description: "The initial value of the context at the start of the parse.", loc: { file: "src/parse.ts", line: 526, column: 4, }, type: "T", typeParamSource: "ContextTracker^T", }, shift: { kind: "method", id: "ContextTracker.constructor^spec.shift", description: "Update the context when the parser executes a\n[shift](https://en.wikipedia.org/wiki/LR_parser#Shift_and_reduce_actions)\naction.", loc: { file: "src/parse.ts", line: 528, column: 4, }, optional: true, type: "Function", signatures: [ { type: "function", params: [ { id: "ContextTracker.constructor^spec.shift^context", kind: "parameter", type: "T", typeParamSource: "ContextTracker^T", loc: { file: "src/parse.ts", line: 531, column: 11, }, name: "context", }, { id: "ContextTracker.constructor^spec.shift^term", kind: "parameter", type: "number", loc: { file: "src/parse.ts", line: 531, column: 23, }, name: "term", }, { id: "ContextTracker.constructor^spec.shift^stack", kind: "parameter", type: "Stack", typeSource: "src/stack.ts", loc: { file: "src/parse.ts", line: 531, column: 37, }, name: "stack", }, { id: "ContextTracker.constructor^spec.shift^input", kind: "parameter", type: "InputStream", typeSource: "src/token.ts", loc: { file: "src/parse.ts", line: 531, column: 51, }, name: "input", }, ], returns: { type: "T", typeParamSource: "ContextTracker^T", }, }, ], }, reduce: { kind: "method", id: "ContextTracker.constructor^spec.reduce", description: "Update the context when the parser executes a reduce action.", loc: { file: "src/parse.ts", line: 532, column: 4, }, optional: true, type: "Function", signatures: [ { type: "function", params: [ { id: "ContextTracker.constructor^spec.reduce^context", kind: "parameter", type: "T", typeParamSource: "ContextTracker^T", loc: { file: "src/parse.ts", line: 533, column: 12, }, name: "context", }, { id: "ContextTracker.constructor^spec.reduce^term", kind: "parameter", type: "number", loc: { file: "src/parse.ts", line: 533, column: 24, }, name: "term", }, { id: "ContextTracker.constructor^spec.reduce^stack", kind: "parameter", type: "Stack", typeSource: "src/stack.ts", loc: { file: "src/parse.ts", line: 533, column: 38, }, name: "stack", }, { id: "ContextTracker.constructor^spec.reduce^input", kind: "parameter", type: "InputStream", typeSource: "src/token.ts", loc: { file: "src/parse.ts", line: 533, column: 52, }, name: "input", }, ], returns: { type: "T", typeParamSource: "ContextTracker^T", }, }, ], }, reuse: { kind: "method", id: "ContextTracker.constructor^spec.reuse", description: "Update the context when the parser reuses a node from a tree\nfragment.", loc: { file: "src/parse.ts", line: 534, column: 4, }, optional: true, type: "Function", signatures: [ { type: "function", params: [ { id: "ContextTracker.constructor^spec.reuse^context", kind: "parameter", type: "T", typeParamSource: "ContextTracker^T", loc: { file: "src/parse.ts", line: 536, column: 11, }, name: "context", }, { id: "ContextTracker.constructor^spec.reuse^node", kind: "parameter", type: "Tree", typeSource: "node_modules/@lezer/common/dist/tree.d.ts", loc: { file: "src/parse.ts", line: 536, column: 23, }, name: "node", }, { id: "ContextTracker.constructor^spec.reuse^stack", kind: "parameter", type: "Stack", typeSource: "src/stack.ts", loc: { file: "src/parse.ts", line: 536, column: 35, }, name: "stack", }, { id: "ContextTracker.constructor^spec.reuse^input", kind: "parameter", type: "InputStream", typeSource: "src/token.ts", loc: { file: "src/parse.ts", line: 536, column: 49, }, name: "input", }, ], returns: { type: "T", typeParamSource: "ContextTracker^T", }, }, ], }, hash: { kind: "method", id: "ContextTracker.constructor^spec.hash", description: "Reduce a context value to a number (for cheap storage and\ncomparison). Only needed for strict contexts.", loc: { file: "src/parse.ts", line: 537, column: 4, }, optional: true, type: "Function", signatures: [ { type: "function", params: [ { id: "ContextTracker.constructor^spec.hash^context", kind: "parameter", type: "T", typeParamSource: "ContextTracker^T", loc: { file: "src/parse.ts", line: 539, column: 10, }, name: "context", }, ], returns: { type: "number", }, }, ], }, strict: { kind: "property", id: "ContextTracker.constructor^spec.strict", description: "By default, nodes can only be reused during incremental\nparsing if they were created in the same context as the one in\nwhich they are reused. Set this to false to disable that\ncheck (and the overhead of storing the hashes).", loc: { file: "src/parse.ts", line: 540, column: 4, }, optional: true, type: "boolean", }, }, loc: { file: "src/parse.ts", line: 525, column: 14, }, name: "spec", }, ], }, ], }, typeParams: [ { type: "typeparam", kind: "typeparam", name: "T", id: "ContextTracker^T", loc: { file: "src/parse.ts", line: 510, column: 28, }, }, ], }, InputStream: { kind: "class", id: "InputStream", description: "[Tokenizers](#lr.ExternalTokenizer) interact with the input\nthrough this interface. It presents the input as a stream of\ncharacters, tracking lookahead and hiding the complexity of\n[ranges](#common.Parser.parse^ranges) from tokenizer code.", loc: { file: "src/token.ts", line: 16, column: 0, }, type: "class", instanceProperties: { next: { kind: "property", id: "InputStream.next", description: "The character code of the next code unit in the input, or -1\nwhen the stream is at the end of the input.", loc: { file: "src/token.ts", line: 31, column: 2, }, type: "number", }, pos: { kind: "property", id: "InputStream.pos", description: "The current position of the stream. Note that, due to parses\nbeing able to cover non-contiguous\n[ranges](#common.Parser.startParse), advancing the stream does\nnot always mean its position moves a single unit.", loc: { file: "src/token.ts", line: 38, column: 2, }, type: "number", }, resolveOffset: { kind: "method", id: "InputStream.resolveOffset", loc: { file: "src/token.ts", line: 63, column: 2, }, type: "Function", signatures: [ { type: "function", params: [ { id: "InputStream.resolveOffset^offset", kind: "parameter", type: "number", loc: { file: "src/token.ts", line: 63, column: 16, }, name: "offset", }, { id: "InputStream.resolveOffset^assoc", kind: "parameter", type: "union", typeArgs: [ { type: "-1", }, { type: "1", }, ], loc: { file: "src/token.ts", line: 63, column: 32, }, name: "assoc", }, ], returns: { type: "union", typeArgs: [ { type: "number", }, { type: "null", }, ], }, }, ], }, peek: { kind: "method", id: "InputStream.peek", description: "Look at a code unit near the stream position. `.peek(0)` equals\n`.next`, `.peek(-1)` gives you the previous character, and so\non.\n\nNote that looking around during tokenizing creates dependencies\non potentially far-away content, which may reduce the\neffectiveness incremental parsing—when looking forward—or even\ncause invalid reparses when looking backward more than 25 code\nunits, since the library does not track lookbehind.", loc: { file: "src/token.ts", line: 81, column: 2, }, type: "Function", signatures: [ { type: "function", params: [ { id: "InputStream.peek^offset", kind: "parameter", type: "number", loc: { file: "src/token.ts", line: 90, column: 7, }, name: "offset", }, ], returns: { type: "number", }, }, ], }, acceptToken: { kind: "method", id: "InputStream.acceptToken", description: "Accept a token. By default, the end of the token is set to the\ncurrent stream position, but you can pass an offset (relative to\nthe stream position) to change that.", loc: { file: "src/token.ts", line: 113, column: 2, }, type: "Function", signatures: [ { type: "function", params: [ { id: "InputStream.acceptToken^token", kind: "parameter", type: "number", loc: { file: "src/token.ts", line: 116, column: 14, }, name: "token", }, { id: "InputStream.acceptToken^endOffset", kind: "parameter", type: "number", loc: { file: "src/token.ts", line: 116, column: 29, }, default: "0", optional: true, name: "endOffset", }, ], }, ], }, advance: { kind: "method", id: "InputStream.advance", description: "Move the stream forward N (defaults to 1) code units. Returns\nthe new value of [`next`](#lr.InputStream.next).", loc: { file: "src/token.ts", line: 147, column: 2, }, type: "Function", signatures: [ { type: "function", params: [ { id: "InputStream.advance^n", kind: "parameter", type: "number", loc: { file: "src/token.ts", line: 149, column: 10, }, default: "1", optional: true, name: "n", }, ], returns: { type: "number", }, }, ], }, }, }, ExternalTokenizer: { kind: "class", id: "ExternalTokenizer", description: "`@external tokens` declarations in the grammar should resolve to\nan instance of this class.", loc: { file: "src/token.ts", line: 253, column: 0, }, type: "class", implements: [ { type: "interface", }, ], construct: { kind: "constructor", id: "ExternalTokenizer.constructor", type: "Function", description: "Create a tokenizer. The first argument is the function that,\ngiven an input stream, scans for the types of tokens it\nrecognizes at the stream's position, and calls\n[`acceptToken`](#lr.InputStream.acceptToken) when it finds\none.", loc: { file: "src/token.ts", line: 263, column: 2, }, signatures: [ { type: "constructor", params: [ { id: "ExternalTokenizer.constructor^token", kind: "parameter", type: "Function", signatures: [ { type: "function", params: [ { id: "ExternalTokenizer.constructor^token^input", kind: "parameter", type: "InputStream", typeSource: "src/token.ts", loc: { file: "src/token.ts", line: 270, column: 21, }, name: "input", }, { id: "ExternalTokenizer.constructor^token^stack", kind: "parameter", type: "Stack", typeSource: "src/stack.ts", loc: { file: "src/token.ts", line: 270, column: 41, }, name: "stack", }, ], }, ], loc: { file: "src/token.ts", line: 269, column: 4, }, name: "token", }, { id: "ExternalTokenizer.constructor^options", kind: "parameter", type: "interface", properties: { contextual: { kind: "property", id: "ExternalTokenizer.constructor^options.contextual", description: "When set to true, mark this tokenizer as depending on the\ncurrent parse stack, which prevents its result from being cached\nbetween parser actions at the same positions.", loc: { file: "src/token.ts", line: 237, column: 2, }, optional: true, type: "boolean", }, fallback: { kind: "property", id: "ExternalTokenizer.constructor^options.fallback", description: "By defaults, when a tokenizer returns a token, that prevents\ntokenizers with lower precedence from even running. When\n`fallback` is true, the tokenizer is allowed to run when a\nprevious tokenizer returned a token that didn't match any of the\ncurrent state's actions.", loc: { file: "src/token.ts", line: 241, column: 2, }, optional: true, type: "boolean", }, extend: { kind: "property", id: "ExternalTokenizer.constructor^options.extend", description: "When set to true, tokenizing will not stop after this tokenizer\nhas produced a token. (But it will still fail to reach this one\nif a higher-precedence tokenizer produced a token.)", loc: { file: "src/token.ts", line: 247, column: 2, }, optional: true, type: "boolean", }, }, loc: { file: "src/token.ts", line: 271, column: 4, }, default: "{}", optional: true, name: "options", }, ], }, ], }, }, Stack: { kind: "class", id: "Stack", description: "A parse stack. These are used internally by the parser to track\nparsing progress. They also provide some properties and methods\nthat external code such as a tokenizer can use to get information\nabout the parse state.", loc: { file: "src/stack.ts", line: 5, column: 0, }, type: "class", instanceProperties: { pos: { kind: "property", id: "Stack.pos", description: "The input position up to which this stack has parsed.", loc: { file: "src/stack.ts", line: 25, column: 4, }, type: "number", }, context: { kind: "property", id: "Stack.context", description: "The stack's current [context](#lr.ContextTracker) value, if\nany. Its type will depend on the context tracker's type\nparameter, or it will be `null` if there is no context\ntracker.", loc: { file: "src/stack.ts", line: 67, column: 2, }, readonly: true, type: "any", }, useNode: { kind: "method", id: "Stack.useNode", loc: { file: "src/stack.ts", line: 188, column: 2, }, type: "Function", signatures: [ { type: "function", params: [ { id: "Stack.useNode^value", kind: "parameter", type: "Tree", typeSource: "node_modules/@lezer/common/dist/tree.d.ts", loc: { file: "src/stack.ts", line: 189, column: 10, }, name: "value", }, { id: "Stack.useNode^next", kind: "parameter", type: "number", loc: { file: "src/stack.ts", line: 189, column: 23, }, name: "next", }, ], }, ], }, canShift: { kind: "method", id: "Stack.canShift", description: "Check if the given term would be able to be shifted (optionally\nafter some reductions) on this stack. This can be useful for\nexternal tokenizers that want to make sure they only provide a\ngiven token when it applies.", loc: { file: "src/stack.ts", line: 233, column: 2, }, type: "Function", signatures: [ { type: "function", params: [ { id: "Stack.canShift^term", kind: "parameter", type: "number", loc: { file: "src/stack.ts", line: 237, column: 11, }, name: "term", }, ], returns: { type: "boolean", }, }, ], }, parser: { kind: "property", id: "Stack.parser", description: "Get the parser used by this stack.", loc: { file: "src/stack.ts", line: 330, column: 2, }, readonly: true, type: "LRParser", typeSource: "src/parse.ts", }, dialectEnabled: { kind: "method", id: "Stack.dialectEnabled", description: "Test whether a given dialect (by numeric ID, as exported from\nthe terms file) is enabled.", loc: { file: "src/stack.ts", line: 333, column: 2, }, type: "Function", signatures: [ { type: "function", params: [ { id: "Stack.dialectEnabled^dialectID", kind: "parameter", type: "number", loc: { file: "src/stack.ts", line: 335, column: 17, }, name: "dialectID", }, ], returns: { type: "boolean", }, }, ], }, }, }, } ``` </details> It appears that `maybeLinkType` expects `items.Parser` to be defined, but it is not. It is interesting to note, however, that `items.LRParser` is defined. When I looked at the changelog for @lezer/lr, I noticed version 0.15.0 changed the name of `Parser` to `LRParser` and I believe this change broke the build process for the website. At this point, I decided to report my findings rather than continue. Hope this helps, and that we can get this fixed. 🙂
marijnh commented 2021-10-26 09:26:35 +02:00 (Migrated from github.com)
Copy link

Oh, right, good point about the npm packages not having the change log or the sources.

Parser is now defined in @lezer/common, and the files build fine on my machine. Could be some kind of pathname confusion—are you on Windows by any chance?

Oh, right, good point about the npm packages not having the change log or the sources. `Parser` is now defined in `@lezer/common`, and the files build fine on my machine. Could be some kind of pathname confusion—are you on Windows by any chance?
ashtonsix commented 2021-10-26 18:41:24 +02:00 (Migrated from github.com)
Copy link

It was indeed pathname confusion! Not because I'm using Windows (I'm on Linux) but because of how npm link works. This command creates a symbolic link, and these links are currently poorly handled by buildref.

This check in particular yields false negatives:

type.typeSource.startsWith(m.relative)

For the following cases:

('node_modules/@lezer/common/dist/parse.d.ts').startsWith('../common') // === false
('../lr/node_modules/@lezer/common/dist/tree.d.ts').startsWith('../common') // === false

I fixed the issue on my computer like so (using fs.realpathSync):

diff --git a/src/buildref.js b/src/buildref.js
index d8802fb..90fe28e 100644
--- a/src/buildref.js
+++ b/src/buildref.js
@@ -1,13 +1,14 @@
 const {gatherMany} = require("getdocs-ts")
 const {build} = require("builddocs")
 const {join, relative} = require("path")
+const {realpathSync} = require("fs")
 
 function Mod(name) {
   this.name = name
   let dir = require.resolve("@lezer/" + name)
   this.base = dir.replace(/[\\\/]dist[\\\/][^\\\/]*$/, "")
   this.main = join(join(this.base, "src"), "index.ts")
-  this.relative = relative(process.cwd(), this.base) + "/"
+  this.path = realpathSync(relative(process.cwd(), this.base) + "/")
 }
 
 exports.buildRef = function buildRef() {
@@ -24,10 +25,25 @@ exports.buildRef = function buildRef() {
         allowUnresolvedTypes: false,
         breakAt: 45,
         imports: [type => {
-          let sibling = type.typeSource && modules.find(m => type.typeSource.startsWith(m.relative))
+          let sibling = type.typeSource && modules.find(m => realpathSync(type.typeSource).startsWith(m.path))
           if (sibling) return "#" + sibling.name + "." + type.type
         }]
       }, moduleItems[i])
     }
   })
 }

With this change the check passes for the relevant cases:

('/home/ashtonsix/workspaces/lezer/common/dist/parse.d.ts')
  .startsWith('/home/ashtonsix/workspaces/lezer/common') // === true
('/home/ashtonsix/workspaces/lezer/lr/node_modules/@lezer/common/dist/tree.d.ts')
  .startsWith('/home/ashtonsix/workspaces/lezer/lr') // === true

I was then able to run node src/index.js without issue. I have created a pull request for this change.

It was indeed pathname confusion! Not because I'm using Windows (I'm on Linux) but because of how `npm link` works. This command creates a symbolic link, and these links are currently poorly handled by `buildref`. This check in particular yields false negatives: ```js type.typeSource.startsWith(m.relative) ``` For the following cases: ```js ('node_modules/@lezer/common/dist/parse.d.ts').startsWith('../common') // === false ('../lr/node_modules/@lezer/common/dist/tree.d.ts').startsWith('../common') // === false ``` I fixed the issue on my computer like so (using `fs.realpathSync`): ```diff diff --git a/src/buildref.js b/src/buildref.js index d8802fb..90fe28e 100644 --- a/src/buildref.js +++ b/src/buildref.js @@ -1,13 +1,14 @@ const {gatherMany} = require("getdocs-ts") const {build} = require("builddocs") const {join, relative} = require("path") +const {realpathSync} = require("fs") function Mod(name) { this.name = name let dir = require.resolve("@lezer/" + name) this.base = dir.replace(/[\\\/]dist[\\\/][^\\\/]*$/, "") this.main = join(join(this.base, "src"), "index.ts") - this.relative = relative(process.cwd(), this.base) + "/" + this.path = realpathSync(relative(process.cwd(), this.base) + "/") } exports.buildRef = function buildRef() { @@ -24,10 +25,25 @@ exports.buildRef = function buildRef() { allowUnresolvedTypes: false, breakAt: 45, imports: [type => { - let sibling = type.typeSource && modules.find(m => type.typeSource.startsWith(m.relative)) + let sibling = type.typeSource && modules.find(m => realpathSync(type.typeSource).startsWith(m.path)) if (sibling) return "#" + sibling.name + "." + type.type }] }, moduleItems[i]) } }) } ``` With this change the check passes for the relevant cases: ```js ('/home/ashtonsix/workspaces/lezer/common/dist/parse.d.ts') .startsWith('/home/ashtonsix/workspaces/lezer/common') // === true ('/home/ashtonsix/workspaces/lezer/lr/node_modules/@lezer/common/dist/tree.d.ts') .startsWith('/home/ashtonsix/workspaces/lezer/lr') // === true ``` I was then able to run `node src/index.js` without issue. I have created a pull request for this change.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lezer/website#8
No description provided.