Esta página aún no está traducida al español. Leer la versión en inglés
The .lva format
The binary form of a Larva HIR assembly - header, string and type tables, metadata, lazily read method bodies, resources and language data - and the PE container that makes a Larva library a .dll.
En esta página
A .lva file is one HIR assembly in binary form. It is what a frontend returns from a compile request, what larva lib writes, what a Hive package carries and what the HIR linker reads. A Larva library .dll is the same bytes wrapped in a minimal Windows PE image, in a section named .lva (see The .dll container).
The code is in Larva.Hir/Encoding/:
| File | What it holds |
|---|---|
LvaWriter |
LvaFormat (magic, Version, Write, Save, Read, Load), the tag enums, and LvaWriter. |
LvaReader |
LvaReader and TokenMap (the mapping the linker reads through). |
Bytes |
ByteSink and ByteSource: LEB128 and zigzag integers, big integers, UTF-8 strings; HirFormatException. |
LvaContainer |
The PE .dll wrapper: ToDll, Unwrap, IsAssembly. |
Design goals
The format was built for three things, in this order:
- Referencing a library costs only its metadata. Method bodies sit in their own block at the end and each method records where its body is. A reader decodes the declarations and leaves every body as bytes until something asks for it. The design document measured a synthetic library of 4,000 types and 40,000 methods (5.9 MB): about 200 ms to write, 30 to 50 ms to read its metadata (the cost of referencing it), about 130 ms to read every body as well.
- Small, with shared tables. Every string is stored once in a string table and every distinct type once in a type table; the rest of the file refers to them by index. Integers are variable-length.
- An exact round trip. Writing an assembly and reading it back gives the same HIR. The text form with types on every expression prints two assemblies the same exactly when they are the same, which is how a round trip is compared. A compiled System.Core loads in tens of milliseconds, where compiling its sources took 854 ms in the design document's measurement.
Integers and strings
Every number in the file is one of these encodings (ByteSink writes them, ByteSource reads them):
| Encoding | Used for | Form |
|---|---|---|
UInt |
counts, indices, lengths, flags | Unsigned LEB128: 7 bits per byte, low bits first, high bit set on every byte but the last. 5 is 05; 195 is c3 01. At most 10 bytes; a count above 2^31-1 is "count out of range". |
Int |
signed values: type tokens, member type indices, labels, -1 sentinels |
Zigzag, then LEB128: 0→00, -1→01, 1→02, -3→05. Small negatives stay one byte. |
Big |
integer constants, float bit patterns | A byte 0 then an Int when the value fits 64 bits; else a byte 1, a UInt length, and the two's-complement little-endian bytes. |
Bool, Byte |
flags, tags, enum values | One byte. |
Utf8 |
the string table only | A UInt byte length, then UTF-8. |
Str / OptStr |
every string elsewhere | A UInt index into the string table; the optional form stores index + 1, 0 for none. |
Type / OptType |
every type elsewhere | A UInt index into the type table; optional: index + 1, 0 for none. |
The file
offset ┌──────────────────────────────────────────────────────────────┐
0 │ "LVA" (4C 56 41) magic, 3 bytes │ header
3 │ UInt version = 5 │
├──────────────────────────────────────────────────────────────┤
│ UInt string count │ string table
│ Utf8 string 0, string 1, … (first use order) │
├──────────────────────────────────────────────────────────────┤
│ UInt type count │ type table
│ entry 0, entry 1, … (a type's parts always come before it) │
├──────────────────────────────────────────────────────────────┤
│ UInt metadata length │ metadata
│ assembly, references, typerefs, memberrefs, files, │
│ attributes, resources, entry point, type definitions, │
│ [language data] │
├──────────────────────────────────────────────────────────────┤
│ UInt bodies length │ bodies
│ body, body, … (methods point here: offset + length) │
└──────────────────────────────────────────────────────────────┘
| Part | Contents | Read |
|---|---|---|
| Header | LVA, then the format version as a UInt. |
First; any other magic is "not a Larva HIR assembly (.lva)", any other version is refused. |
| String table | Every string of the assembly once: names, namespaces, file paths, string constants, documentation. | Whole, up front. |
| Type table | Every distinct HirType once, each entry built from earlier ones. |
Whole, up front. |
| Metadata | The assembly's declarations. Its length is stored, so a reader can find the bodies without decoding it. | Whole, up front. |
| Bodies | Method bodies, one after another. | One body at a time, the first time a method's Body is used. |
The writer produces the metadata and bodies first, into separate buffers, interning strings and types as it meets them; then it writes the tables, followed by the two buffers. So the string table is in the order of first use: index 0 is always the assembly's name.
Versions
LvaFormat.Version is 5. A reader accepts exactly its own version and throws format version N; this Larva reads version 5 for any other. The design document records the steps: version 2 added documentation comments and the positions of fields and properties; version 5 added GenericParam.Constructors and FixedSize (T : new(…) and T : fixed(N)).
Other things the version guards:
- Language packs declare
hir: 5in their manifest and answerhirininfo; a pack of another format is not loaded (see Language packs). - A compiled System.Core is checked by its first four bytes before use:
L,V,Aand the version byte (one byte while the version is below 128). A System.Core of another format is reported with "reinstall Larva" instead of being misread. - Cached libraries compiled from sources are keyed by the pack, its library's hash and the format version, so a new format recompiles them on their own.
Language data was added without a new version: it is optional and comes last in the metadata, and a reader that does not know it stops before it (see Language data).
The type table
Each entry starts with a TypeTag byte:
| Tag | Value | Then |
|---|---|---|
Primitive |
0 | Byte primitive kind |
Named |
1 | Int type token, then UInt count and the argument type indices |
Array |
2 | element, UInt rank |
InlineArray |
3 | element, UInt count, Bool has-sizeof, [sizeof type] |
Nullable |
4 | inner |
Reference |
5 | target, Bool mutable |
Pointer |
6 | target, Bool mutable |
Func |
7 | parameter type indices (counted), result |
NativeFunc |
8 | parameter type indices (counted), result |
GenericParam |
9 | Byte owner (0 type, 1 method), UInt index |
Pack |
10 | pattern |
Primitive kinds: Void 0, Bool 1, Char 2, I8 3, I16 4, I32 5, I64 6, I128 7, I256 8, U8 9, U16 10, U32 11, U64 12, U128 13, U256 14, F16 15, F32 16, F64 17, F128 18, F256 19.
A named type stores its raw token as an Int: a definition is its index (≥ 0), a reference is ~index (negative). Because the writer always adds a type's parts before the type, a reader builds the table in one pass, each entry from entries it already has.
The metadata
In order:
| # | Field | Encoding |
|---|---|---|
| 1 | Assembly name | Str |
| 2 | Version | UInt count, then Int parts |
| 3 | References | UInt count; each: Str name, version (count + Ints) |
| 4 | Type references | UInt count; each: UInt assembly index, Str namespace, Str name, UInt arity |
| 5 | Member references | UInt count; each: Byte kind (field, method, property, event), Int parent type token, Str name, UInt generic arity, parameter types (counted), result Type |
| 6 | Files | UInt count; each Str |
| 7 | Assembly attributes | see below |
| 8 | Resources | UInt count; each: Str name, UInt length, the bytes |
| 9 | Entry point | Bool present, then a member token |
| 10 | Types | UInt count; each a type definition |
| 11 | Language data (optional) | UInt count; each: Str name, UInt length, the bytes |
A member token is an Int type index (-1 for a member reference) followed by a UInt member index.
An attribute list is a UInt count; each attribute is its constructor's member token, a counted list of constants, and a counted list of named arguments (Str name, constant).
A source location is a UInt file index plus one (0: unknown), then, when known, UInt line and UInt column.
Type definitions
Str namespace Str name Byte kind UInt flags Byte access
Int declaring type generic parameters OptType base types: interfaces OptType underlying
UInt field count fields…
UInt method count methods…
UInt property count properties…
UInt event count events…
UInt operator count operators…
attributes location OptStr doc
| Item | Encoding |
|---|---|
| Generic parameter | Str name, Bool variadic, UInt constraint kinds, types, one-of types, operators (UInt count; each Str symbol, operand types, result), converts-to types, converts-from types, constructors (UInt count; each a type list), UInt fixed size, OptType default |
| Field | Str name, Type, Byte flags, Byte access, optional constant (Bool + constant), attributes, location, OptStr doc |
| Method | Str name, Byte kind, UInt flags, Byte access, generic parameters, parameters (UInt count; each Str name, Type, Byte flags, optional default constant, attributes), result Type, OptStr extern name, OptStr library, attributes, location, OptStr doc, then the body reference |
| Body reference | UInt body length plus one (0: no body), then UInt offset from the start of the bodies block |
| Property | Str name, Type, index types, Int getter, Int setter (method indices, -1 none), Byte flags, Byte access, attributes, location, OptStr doc |
| Event | Str name, Type, Int adder, Int remover, Byte flags, Byte access, attributes |
| Operator | Str symbol, Byte fixity, UInt method index, Int precedence, Byte associativity |
Constants
A constant is a ConstantTag byte and its value:
| Tag | Value | Then |
|---|---|---|
Integer |
0 | Byte primitive, Big value |
Float |
1 | Byte primitive, Big IEEE bit pattern |
Bool |
2 | Bool |
Char |
3 | UInt code point |
String |
4 | Str |
Null |
5 | nothing |
Type |
6 | Type |
Enum |
7 | Type, Big value |
Array |
8 | element Type, UInt count, constants |
EnumText |
9 | Type, Big value, Str text (an enum based on str) |
EnumText was added after Enum with its own tag, so files with the older tag read the same.
Bodies
A body is its locals, then its block:
UInt local count { Str name, Type, Bool mutable } … statement (always a Block)
Statements and expressions are a pre-order tree: a tag byte, the node's fields, its children inline. Every statement ends with its source location. Every expression starts with its tag and its result Type, then its fields:
StmtTag |
Value | StmtTag |
Value |
|---|---|---|---|
Block |
0 | Return |
11 |
Local |
1 | Throw |
12 |
Expr |
2 | Try |
13 |
If |
3 | Using |
14 |
While |
4 | Delete |
15 |
DoWhile |
5 | Region |
16 |
For |
6 | Unsafe |
17 |
Foreach |
7 | Yield |
18 |
Switch |
8 | Checked |
19 |
Break |
9 | ||
Continue |
10 |
ExprTag |
Value | ExprTag |
Value | ExprTag |
Value |
|---|---|---|---|---|---|
Literal |
0 | Assign |
14 | TypeOf |
26 |
Local |
1 | Convert |
15 | SizeOf |
27 |
Param |
2 | Is |
16 | Default |
28 |
This |
3 | As |
17 | With |
29 |
Field |
4 | Switch |
18 | Range |
30 |
Property |
5 | Conditional |
19 | Interpolated |
31 |
Element |
6 | Coalesce |
20 | Borrow |
32 |
Call |
7 | ConditionalAccess |
21 | AddressOf |
33 |
Invoke |
8 | Receiver |
22 | Deref |
34 |
New |
9 | Lambda |
23 | Block |
35 |
NewArray |
10 | MethodRef |
24 | ArrayLength |
36 |
ArrayLiteral |
11 | Await |
25 | Checked |
37 |
Unary |
12 | Resource |
38 | ||
Binary |
13 |
Pattern tags: Discard 0, Constant 1, Relational 2, Range 3, Type 4, Binding 5, And 6, Or 7, Not 8, Property 9, Positional 10, List 11.
Some node layouts, as LvaWriter writes them (optional parts are a Bool then the part):
| Node | Fields after the tag (and, for expressions, the type) |
|---|---|
Return |
optional expression |
If |
condition, then-statement, optional else-statement |
While |
condition, body, Int label |
For |
initializer statements (counted), optional condition, step expressions (counted), body, Int label |
Try |
body, catches (UInt count; each optional type, Int local, optional filter, block), optional finally |
Call |
optional receiver, member token, type arguments (counted), arguments (counted), Bool non-virtual |
Field |
optional instance, member token, Bool has type arguments, [type arguments] |
Binary |
Byte operator, left, right |
Assign |
target, value, Byte compound operator plus one (0: plain =) |
Convert |
value, Byte conversion kind, optional method token |
Lambda |
parameter locals (counted UInts), body statement, Bool async |
Resource |
Str resource name |
Binary operators: Add 0, Subtract 1, Multiply 2, Divide 3, Remainder 4, Power 5, BitAnd 6, BitOr 7, BitXor 8, ShiftLeft 9, ShiftRight 10, Equal 11, NotEqual 12, Less 13, LessEqual 14, Greater 15, GreaterEqual 16, And 17, Or 18.
Reading bodies later
When LvaReader meets a method's body reference, it checks that the range is inside the file ("method body out of range") and attaches a loader to the method (MethodDef.LoadBodyLater). The first read of MethodDef.Body decodes the body from the file's bytes and drops the loader. The reader keeps the whole file in memory for this, and the string and type tables it built.
The HIR linker reads each input a second time through a TokenMap, so the bodies it decodes later come out already linked: every type token, member token and file index mapped into the merged assembly. Lowering then reads bodies on several threads; since all member references were resolved before, those threads only look things up.
Resources
Resources are stored raw inside the metadata (field 8): a name in the string table, a length, and the bytes. They are part of the program: the HIR linker concatenates the resources of every input, and a body reads one with a Resource expression that names it (by name, not index, so linking renumbers nothing; the last resource of a name wins). Before resources were stored this way, generated code carried them as byte literals in source, which was slow to parse for large files; now the parser never sees the data.
Language data
HirAssembly.LanguageData is a list of name and bytes that a language keeps with its libraries for its own compiler. It is written at the very end of the metadata, after the types, and only when the list is not empty. A reader knows it is there because the metadata continues past the types: it compares its position with the metadata's stored length. That is why the format stayed at version 5 when it was added.
Language data is never linked into a program. Antril uses it for variadic templates (antril.template:<file>): the text of each template declaration, which a program that references the library parses and expands for the sizes it uses.
A real file, byte by byte
This is the whole of Tiny.ant:
into namespace Tiny
{
public class Calc
{
public:
static Twice(i32 n) : i32 => n * 2;
}
}
larva lib -o Tiny.lva Tiny.ant writes 3,192 bytes:
larva lib -o Tiny.lva Tiny.ant
Tiny.lva: 1 types (frontend 1718 ms, write 2 ms; total 1728 ms)
Most of them are the type references the Antril frontend keeps to System.Core (211 of them, for every core type it may need), and their names in the string table. The start and the end of the file:
00000000: 4c56 4105 c301 0454 696e 790b 5379 7374 LVA....Tiny.Syst
00000010: 656d 2e43 6f72 6506 5379 7374 656d 0454 em.Core.System.T
00000020: 7970 6506 5374 7269 6e67 0341 6e79 0542 ype.String.Any.B
…
00000840: 6c61 7276 612d 6869 722f 5469 6e79 2e61 larva-hir/Tiny.a
00000850: 6e74 0443 616c 6305 5477 6963 6501 6e02 nt.Calc.Twice.n.
00000860: 0105 0000 05fb 0700 0202 0001 0100 d301 ................
00000870: 0002 0300 0002 0401 0002 0500 0002 0601 ................
…
00000c20: 02be 010f 0002 bd01 1000 02be 0110 0001 ................
00000c30: bf01 0000 0001 00c0 0100 0000 0100 0100 ................
00000c40: 0000 01c1 0100 0100 0001 c201 0100 0000 ................
00000c50: 0100 0000 0106 0b00 1600 0000 0000 0103 ................
00000c60: 0f00 1500 0001 0b01 0d01 0202 0100 0001 ................
00000c70: 0005 0004 0106 2100 ......!.
Decoded:
| Offset | Bytes | Meaning |
|---|---|---|
000 |
4c 56 41 |
Magic LVA. |
003 |
05 |
Version 5. |
004 |
c3 01 |
195 strings. |
006 |
04 "Tiny" |
String 0: the assembly's name (also the namespace). |
00b |
0b "System.Core" |
String 1. Then System (2), Type (3), String (4), Any (5)… |
83a |
17 "docs-larva-hir/Tiny.ant" |
String 191: the source file, as folder and file. |
852 |
04 "Calc", 05 "Twice", 01 "n" |
Strings 192, 193, 194. |
85f |
02 |
2 types in the type table. |
860 |
01 05 00 |
Type 0: Named, token zigzag 5 = -3 = ~2, so TR2; no arguments. |
863 |
00 05 |
Type 1: Primitive, I32. |
865 |
fb 07 |
Metadata: 1,019 bytes. |
867 |
00 |
Assembly name: string 0, Tiny. |
868 |
02 02 00 |
Version: 2 parts, 1 and 0. |
86b |
01 01 00 |
1 reference: System.Core, no version. |
86e |
d3 01 |
211 type references. |
870 |
00 02 03 00 |
TR0: assembly 0, System, Type, arity 0. Then TR1 00 02 04 01 (String, arity 1), TR2 00 02 05 00 (Any)… |
c2e |
00 |
No member references. |
c2f |
01 bf 01 |
1 file: string 191. |
c32 |
00 00 00 |
No assembly attributes, no resources, no entry point. |
c35 |
01 |
1 type definition. |
c36 |
00 c0 01 |
Namespace Tiny (string 0), name Calc (192). |
c39 |
00 00 00 |
Kind Class, no flags, Public. |
c3c |
01 |
Declaring type: zigzag 1 = -1, none. |
c3d |
00 01 00 00 |
No generics; base type 0 + 1 (TR2, System.Any); no interfaces; no underlying type. |
c41 |
00 01 |
No fields; 1 method. |
c43 |
c1 01 00 01 00 |
Method Twice (193), kind Normal, flags Static, Public. |
c48 |
00 01 c2 01 01 00 00 00 |
No generics; 1 parameter: n (194), type 1 (i32), no flags, no default, no attributes. |
c50 |
01 00 00 00 |
Result type 1 (i32); no extern name, no library, no attributes. |
c54 |
01 06 0b |
Location: file 0, line 6, column 11. |
c57 |
00 |
No documentation. |
c58 |
16 00 |
Body: 22 - 1 = 21 bytes, at offset 0 of the bodies block. |
c5a |
00 00 00 00 |
No properties, events, operators or type attributes. |
c5e |
01 03 0f 00 |
Type location: file 0, line 3, column 15; no documentation. End of metadata. |
c62 |
15 |
Bodies: 21 bytes. |
c63 |
00 |
Body of Twice: no locals. |
c64 |
00 01 |
Block with 1 statement. |
c66 |
0b 01 |
Return, with a value: |
c68 |
0d 01 02 |
Binary of type 1 (i32), operator Multiply: |
c6b |
02 01 00 |
left: Param of type i32, parameter 0 (n); |
c6e |
00 01 00 05 00 04 |
right: Literal of type i32: constant Integer, I32, small Big, zigzag 4 = 2. |
c74 |
01 06 21 |
The return's location: file 0, line 6, column 33. |
c77 |
00 |
The block's location: unknown. End of file (0xc78 = 3,192 bytes). |
The text form of the same assembly is shown in Larva HIR.
The .dll container
Larva libraries ship as .dll files: System.Core is System.Core.dll in an installation, larva lib -o Name.dll and larva bind write .dlls, and a .queen project names them with assembly:. Such a file is a valid Windows PE image (x64, a DLL with no code and no entry point) whose one section, .lva, holds the assembly's bytes exactly as a .lva file has them. Shipping libraries in the platform's own library format means they look like libraries to the system and to users, carry the right extension, and need no new file type; Larva's readers take either form.
LvaContainer.ToDll builds it; LvaFormat.Save uses it whenever the path ends in .dll, and larva lib does the same. (larva build --dll is different: it builds a native library for C, with [Export] functions.)
0x000 ┌─────────────────────────────────┐
│ DOS header "MZ", e_lfanew = 0x80│
0x040 │ DOS stub: "This is a Larva │
│ library.\r\r\n$" │
0x080 ├─────────────────────────────────┤
│ "PE\0\0" + COFF header │ machine 0x8664 (x64), 1 section,
│ │ characteristics 0x2022 (image, large address, DLL)
0x098 │ optional header (PE32+, 240 B) │ image base 0x180000000, alignment 0x1000 / 0x200,
│ │ subsystem 2, NX compatible, 16 empty data directories
0x188 │ section header ".lva" │ virtual size = the assembly's exact length,
│ │ raw offset 0x200, flags 0x40000040 (initialized data, readable)
0x200 ├─────────────────────────────────┤
│ the .lva bytes ("LVA" 05 …) │
│ padded to a multiple of 0x200 │
└─────────────────────────────────┘
The headers of Tiny.dll (4,096 bytes: 512 of headers and the 3,192 bytes of Tiny.lva, padded):
00000000: 4d5a 9000 0300 0000 0400 0000 ffff 0000 MZ..............
00000010: b800 0000 0000 0000 4000 0000 0000 0000 ........@.......
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000030: 0000 0000 0000 0000 0000 0000 8000 0000 ................
00000040: 0e1f ba0e 00b4 09cd 21b8 014c cd21 5468 ........!..L.!Th
00000050: 6973 2069 7320 6120 4c61 7276 6120 6c69 is is a Larva li
00000060: 6272 6172 792e 0d0d 0a24 0000 0000 0000 brary....$......
00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000080: 5045 0000 6486 0100 0000 0000 0000 0000 PE..d...........
00000090: 0000 0000 f000 2220 0b02 0e00 0000 0000 ......" ........
…
00000180: 0000 0000 0000 0000 2e6c 7661 0000 0000 .........lva....
00000190: 780c 0000 0010 0000 000e 0000 0002 0000 x...............
000001a0: 0000 0000 0000 0000 0000 0000 4000 0040 ............@..@
…
00000200: 4c56 4105 c301 0454 696e 790b 5379 7374 LVA....Tiny.Syst
The section header at 0x188 reads: name .lva, virtual size 0xc78 (3,192, the assembly's exact length), virtual address 0x1000, raw size 0xe00, raw offset 0x200. From offset 0x200 the bytes are those of Tiny.lva.
Reading back (LvaContainer.Unwrap): if the file starts with MZ and e_lfanew points at PE\0\0, the section table is scanned for .lva and the section's bytes (raw offset, virtual size) are the assembly; a section that runs past the end of the file is "a Larva library whose .lva section is cut short". Anything else is taken as a plain .lva. LvaReader.Read always unwraps first, so the linker, larva bind, the language server and the type commands accept both forms.
LvaContainer.IsAssembly(path) tells a Larva library from any other file by reading at most the first 4 KB: it starts with LVA, or it is a PE image with a .lva section. The project loader makes the same decision by name and header (QueenProject.IsAssembly: any .lva, or a .dll whose first kilobyte shows a .lva section; a native DLL has none). That is how larva build, larva lib and the type commands know which inputs are libraries to reference and which are sources to compile, and how larva bind refuses to write over a .dll that is not a Larva library.
Errors
A damaged or foreign file raises HirFormatException with one of these messages:
| Message | Cause |
|---|---|
not a Larva HIR assembly (.lva) |
No LVA magic (after unwrapping a PE). |
format version N; this Larva reads version 5 |
Another format. |
unexpected end of file |
A read past the end. |
integer too long, count out of range |
A malformed LEB128 number. |
string index out of range, type index out of range |
A reference past a table. |
unknown type tag N (and constant, statement, expression, pattern) |
A tag this reader does not know. |
expected a block |
A body or a try part that is not a block. |
method body out of range |
A body reference outside the file. |
a Larva library whose .lva section is cut short |
A truncated .dll. |