The catalog and language views

Every tool that shows code to a person needs the same things: the types of a program and of its libraries, by namespace; their members, including inherited ones with the right type arguments; a search by name; and a way to write a type in the reader's language. Larva provides these once, in the Larva.Hir.Services project, over HIR, so that every language on Larva gets larva types, larva show, larva search, completion, hover, signature help and "go to definition" into libraries without writing them again.

The project is language-neutral. A language contributes two things: a LanguageView subclass (how it writes types, signatures and whole types) and an IDocumentAnalysis (what is at each position of a document it has analyzed). Antril's are AntrilView and AntrilAnalysis, inside its language pack.

 HIR assemblies (.lva / .dll)          program compiled in memory
          │                                     │
          └───────────────┬─────────────────────┘
                          ▼
                    HirCatalog  ── types by namespace, members, inheritance, search
                     │       │
          LanguageView       LanguageService ◄── IDocumentAnalysis (the language's binder)
          (hir, antril…)     completion, hover, definition, signatures
             │                       │
   larva types / show / search    larva lsp (through the pack's lsp.* requests)

HirCatalog

HirCatalog.Create(assemblies) builds a catalog from any set of HIR assemblies: a program and the libraries it references, or one linked assembly. It does not link them. Each assembly keeps its own tokens, and the catalog resolves a reference (a TypeRef or MemberRef) by name in the assembly it names, when it is asked to. Building a catalog is cheap: the design document measured 0.4 ms to read System.Core and build its catalog (method bodies are not decoded; see The .lva format).

What Create builds:

Global types

A HirType in an assembly uses that assembly's tokens: T3 in System.Core and T3 in the program are different types. The catalog gives views and editors one numbering for everything: Globalize(type, assembly) rewrites every named type as a definition token whose index is the type's CatalogType.Id (a type no assembly has becomes HirCatalog.Unknown). Every type a CatalogType or CatalogMember exposes (BaseType, InterfaceTypes, ValueType, Parameters) is global, so a view writes types from any assembly without knowing where they came from, and TypeOf(global) returns the CatalogType a global type names.

Types and members

A CatalogType wraps a TypeDef with what tools ask about it: Kind (adds Module for the $Module class that holds a namespace's functions), Base, Interfaces, BaseChain(), AllInterfaces(), IsExtension and Extended, Primitive, Doc, Location (a file, line and column, when the assembly has its sources' positions).

A type is hidden when it is Generated (closures, describers, other types a frontend made) or its name starts with $ (except $Module). Hidden types are in AllTypes but never in listings, search or completion.

Declared lists the members a person sees, as CatalogMembers of one of these kinds:

Accessors are not members of their own: Member(MemberKind.Method, i) on a getter returns its property. Generated fields and methods, and names starting with $ (other than operators and conversions), are left out.

Members(arguments) gives the members of a value of the type, with inheritance: its own, then each base's not overridden or hidden (same name and parameters, CatalogMember.Key), each as a MemberView that carries its declaring type's arguments seen from here. For class B : A<str>, the inherited A<T>.Get() : T comes back as Get() : str. Constructors are not inherited; an interface also gets the members of the interfaces it extends.

Other queries: DerivedFrom(type) (direct subclasses), Implementers(interface), ExtensionsIn(namespace) and ExtensionsOf(type) (the ext blocks that apply to a type, exact or by a generic constraint), NamespacesIn(namespace), IsNamespace(name).

Finding a type

Find(fullName) accepts what people type:

Search

Search(query, limit) scores every visible type and member name against the query, ignoring case:

A member scores one more than a type with the same match. Hits are sorted by score, then by shorter name, then by full name, and cut at the limit (larva search shows 40). Without nonPublic (and larva search does not pass it), only public types and public or inherited members are searched.

Language views

A LanguageView is how one language writes what a catalog holds. Its abstract members:

And overridable ones with defaults: ExtensionSignature (an extension method as its receiver sees it), Local and Parameter (for hovers and signature help), Keywords (what completion offers), KindWord, and Listing(catalog, namespace) (a namespace's namespaces, types and module functions, one line each).

DecompileOptions has three switches: Bodies (method bodies, or declarations only), NonPublic (private and internal members too) and Docs (documentation comments). GenericNames gives type parameters their declared names (T, K), since in HIR they are only indices.

LanguageViews is a small registry by name or extension. It holds HirView from the start; a language may register its own. LanguageViews.Get("hir") and Get(".hir") both find the HIR view.

The HIR view

HirView is the language-neutral view: every type fully qualified, every member labelled with its kind, no syntax sugar. It writes declarations only (it ignores Bodies); the full text form with bodies is HirPrinter's (see Larva HIR).

$ larva types --lang hir --lib Geometry.dll Geometry
Geometry
  public struct Geometry.Point
  public class Geometry.Shapes : System.Any
$ larva show --lang hir --lib Geometry.dll Geometry.Point
// A point on a plane.
public struct Geometry.Point
{
    field X: f64
    field Y: f64
    constructor Point(x: f64, y: f64)
    // How far the point is from the origin.
    method Length() -> f64
}

The same view on another language's types shows what HIR keeps of them. Here Antril's enum Color { Red, Green, Blue } with the ToString the frontend synthesizes, and a generic class whose T : number is an interface constraint:

$ larva show --lang hir Store.Counter Store.Color Stack.ant
public class Store.Counter<T> : System.Any
{
    field total: T
    constructor Counter()
    method Add(value: T) -> void
    method Each(values: T[], keep: func<T> : bool) -> i32
    property Total: T
}

public enum Store.Color
{
    static enumvalue Red: Store.Color
    static enumvalue Green: Store.Color
    static enumvalue Blue: Store.Color
    method ToString() -> System.String<u8>
}

A language's view: decompiling

A language's view writes HIR back as that language's source, whatever language the HIR came from. That is decompiling, and it is how Larva shows a library without its sources: System.Core ships as a compiled .dll with no .ant file, yet larva show System.Console and the editor's "go to definition" show it as Antril. The design document reports System.Core decompiled to Antril (54 types at the time, in about 19 ms) and compiled again to the same types and members.

AntrilView writes Antril's own forms (str, func<A> : R, Name(i32 x) : R, public: sections, into namespace) and its body printer rebuilds some of the sugar the frontend expanded: + on strings instead of String.Concat calls, list[i] for indexers, : base(…), method groups. Other sugar stays expanded. The library from Larva HIR, decompiled:

$ larva show Geometry.Shapes Geometry.ant
// Geometry.Shapes — Program (decompiled from Larva HIR)

into namespace Geometry
{
	public class Shapes
	{
		public:
			static Farthest(Point[] points) : Point
			{
				Point best = points[0];
				{
					Point[] _items1 = points;
					i32 _index2 = 0;
					loop (_index2 < _items1.Length)
					{
						Point p = _items1[_index2];
						{
							if (p.Length() > best.Length())
								best = p;
						}
						_index2++;
					}
				}
				return best;
			}
	}
}

The source said loop (Point p in points). HIR only has the index loop the frontend made of it, so that is what comes back; the hidden locals are renamed from $items1 to _items1 so the result is valid source. The header line names the assembly (Program: the sources given on the command line).

With --no-bodies, a type reads like a reference page:

$ larva show --no-bodies --lib Geometry.dll Geometry.Shapes
// Geometry.Shapes — Geometry (decompiled from Larva HIR)

into namespace Geometry
{
	public class Shapes
	{
		public:
			static Farthest(Point[] points) : Point;
	}
}

The same machinery writes source for other tools: larva new writes a template's HIR as a project's sources, and larva bind --source writes bindings, both through the pack's source request (see Language packs).

larva types, show and search

TypeCommands.Run (in Larva.Cli) prepares a catalog and a view; CatalogCommands.Run (in Larva.Hir.Services) does the command. The steps:

1. Arguments. A file of an installed language is a source; a .lva, or a .dll with a .lva section, is a library (as is --lib X.dll); a .queen adds its project's sources and the libraries it names; anything else is a word (a namespace, type names, a query). --no-bodies and --public set DecompileOptions; --core picks another System.Core.
2. The view. --lang hir: Larva's own view. --lang <language>: that pack's view. Without --lang: the language of the sources given, else the only installed language, else HIR. An unknown language stops with "no view for the language klingon (hir; installed: Antril (.ant))".
3. The assemblies. System.Core and the libraries, as file references. If sources were given, their pack compiles them in memory into an assembly named Program, placed first.
4. The command. With the HIR view, larva reads the assemblies itself, builds a HirCatalog and runs CatalogCommands.Run with HirView. With a language's view, it sends the pack a catalog request (command, words, options, the assemblies); the pack builds the same catalog, runs the same CatalogCommands.Run with its own view, and returns {output, error, exit}.

So the logic of the three commands exists once, and a language only changes how things are written.

types

Without a namespace, every namespace that has types, each as a Listing; with one, that namespace's listing:

$ larva types --lib Geometry.dll Geometry
Geometry
  public value Point
  public class Shapes

A listing starts with the namespace's inner namespaces:

$ larva types --lang hir
System
  namespace System.Diagnostics
  namespace System.IO
  namespace System.Net
  namespace System.Reactive
  namespace System.Reflection
  namespace System.Runtime
  namespace System.Text
  namespace System.Threading
  public class System.Activator : System.Any
  public class System.Any
  public class System.ArgumentError : System.Exception
…

show

Each word is a type (decompiled), or a namespace (its listing). A name that is neither fails with exit code 1 and, if the search finds types with a similar name, suggests up to five:

$ larva show Geometry.Poin --lib Geometry.dll
larva show: no type `Geometry.Poin`
  did you mean: Geometry.Point

Without any word, larva show says "which type? (larva show System.List)" and exits with 2.

search

The query is the words joined by spaces. Types print as Namespace: header, members as Type.signature, in the view's syntax:

$ larva search --lib Geometry.dll farth
Geometry.Shapes.static Farthest(Point[] points) : Point
$ larva search --lang hir --lib Geometry.dll farth
Geometry.Shapes.static method Farthest(points: Geometry.Point[]) -> Geometry.Point
$ larva search sb
System: public final class StringBuilder
System.IO.BinaryReader.ReadSByte() : i8
System.IO.StringWriter.GetStringBuilder() : StringBuilder
System.Text.JsonValue.bool AsBool { get; }

sb finds StringBuilder by its word starts (score 8), ReadSByte and GetStringBuilder the same way, as members (9), and AsBool because it contains sB (10 + 1).

Editor services

LanguageService turns a language's analysis of one document into editor features, through the catalog and the language's view. The language implements IDocumentAnalysis:

And LanguageService does the rest the same way for every language:

• Completion (Complete). Members of a type with inheritance, visible from the type the code is in (Visible: public everywhere; internal and friendly in the same assembly; inherited in the type and what derives from it; private in the type itself or, for a namespace's functions, in the same namespace of the same assembly; members of an interface are always visible). Overloads collapse into one item with a count. Types of namespaces the document does not import are offered too, with the namespace to import added when chosen. Items are ranked by kind (locals and parameters first, then fields and properties, methods, functions, constants, type parameters, classes, structs, interfaces, enums, constructors, types to import, namespaces, keywords last), then by how well they match the prefix (the search scores), then by name. The view writes each item's detail line.
• Hover: the view's Local, Header or Signature with the documentation comment.
• Definition: where a local or a member is declared, from the HIR's source locations.
• Signature help: each candidate's Signature, the parameter ranges inside it found with the view's Parameter, and the active overload (the one with as many parameters as arguments so far, else the fewest more).

Semantic tokens use the standard kinds of the language server protocol plus four of Larva's own, each declared with a standard parent so that any theme colors it: Field (property), Exception and Attribute (class), Bitflags (enum). Semantic.Of(type) and Semantic.Of(member) give a catalog entity's kind; the modifiers are declaration, static, readonly and defaultLibrary (from another assembly than the program).

Decompiled types in the editor

When "go to definition" lands in a library, the pack answers lsp.definition with a decompiled type instead of a file. Its view writes the type with every name wrapped in invisible marks that record what the name is; the marks are removed at the end and become two lists: tokens (offset, length, kind, modifiers: the editor colors the read-only file with them) and links (offset, length, the type and member the name stands for: clicking it decompiles that one in turn). The answer also says where to put the cursor: the declaration of the member asked for, by its overload key. larva lsp writes the text to a read-only file and serves it. The larva/decompile request decompiles any type of the catalog on demand, through the language's view or (with "language": "hir") the HIR view. See The language server.

Speed

The design document's measurements when the services were introduced: building the catalog of System.Core 0.4 ms, analyzing a 35-line document 1.5 ms, a 1,300-line one 5.5 ms, a completion list 0.07 ms. All of it works on metadata and on the document being edited; library bodies are only decoded when a library type is decompiled with bodies.