From 7fbb381221c42403f80ac913d33357abb1620839 Mon Sep 17 00:00:00 2001 From: Kam Date: Tue, 20 Oct 2026 12:12:27 +1100 Subject: [PATCH] feat: read a received email's MAP part and release 0.3.2 extractMessage() and `mailschema message ` find an email's one MAP part from its original bytes, with the rules of the Ruby package's Message.extract. Enum and const errors name their allowed values, and the README says where contracts live. Prepared from mailschema/mailschema@1b41186. --- README.md | 14 +- UPSTREAM.md | 2 +- bin/mailschema.js | 23 +- dist/core/artifacts.js | 14 +- dist/core/index.d.ts | 1 + dist/core/index.js | 1 + dist/core/message.d.ts | 27 +++ dist/core/message.js | 262 ++++++++++++++++++++++ package-lock.json | 4 +- package.json | 2 +- src/core/artifacts.ts | 15 +- src/core/index.ts | 1 + src/core/message.ts | 286 +++++++++++++++++++++++++ test/fixtures/examples/publication.eml | 56 +++++ test/package.test.mjs | 3 + 15 files changed, 693 insertions(+), 18 deletions(-) create mode 100644 dist/core/message.d.ts create mode 100644 dist/core/message.js create mode 100644 src/core/message.ts create mode 100644 test/fixtures/examples/publication.eml diff --git a/README.md b/README.md index 6d99c7e..7cbe055 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [Mail Action Protocol](https://mailschema.org) 0.3 for JavaScript. It parses and checks MAP descriptions, type contracts and Registry implementation records, and computes the RFC 8785 digests that identify contracts. The same code runs the MailSchema specification's own checks. It has no Node.js-only dependencies, so it also runs in browsers and workers. ```sh -npm install mailschema@0.3.1 +npm install mailschema@0.3.2 ``` ## Check a description against its contract @@ -20,6 +20,8 @@ if (errors.length) throw new Error(errors.join('\n')); `parseDescription` reads the bytes as MAP JSON (strict I-JSON within 64 KiB) and checks the core rules, or throws `InvalidDocument` with every reason. Obtain contracts from a catalogue you configure and pin them by digest; a message never chooses one. +The published types and their contracts are in the Registry, [`https://mailschema.org/registry/map-0.3.json`](https://mailschema.org/registry/map-0.3.json): each type's `contractHref` is its contract under `https://mailschema.org/artifacts/map-0.3/contracts/`, and its Registry page (`href`) links an example description of the type. + ## API - `parse(input, maxBytes)`, `canonicalize(value)`, `digest(value)`: MAP JSON and RFC 8785 digests. @@ -27,6 +29,7 @@ if (errors.length) throw new Error(errors.join('\n')); - `new Contract(value)`, `Contract.parse(input)`: a checked contract with `digest`, `operation(id)`, `detailsErrors(details)`, `inputErrors(operationId, input)` and `descriptionErrors(description)`. - `contractErrors(value)`: why a value is not a contract, including schema keywords, formats and patterns every implementation can enforce alike. - `implementationErrors(value)`, `parseImplementation(input)`: Registry implementation records. +- `extractMessage(input)`: the one MAP part of a received email, read from its original bytes. It returns `{ status: 'valid', description, bytes }`, `{ status: 'invalid', reason }` or `{ status: 'none' }`, and never throws for untrusted input. - `PROFILE`, `CONTEXT`, `DESCRIPTION_MEDIA_TYPE`, `EFFECTS`, `CAPABILITY_KINDS` and the size limits. The profile record, context and schemas are exported byte for byte: `mailschema/profile.json`, `mailschema/context.jsonld`, `mailschema/core.schema.json`, `mailschema/contract.schema.json` and `mailschema/implementation.schema.json`. @@ -34,12 +37,13 @@ The profile record, context and schemas are exported byte for byte: `mailschema/ ## Command line ```sh -npx mailschema@0.3.1 contract campaign-send-approval-0.1.json -npx mailschema@0.3.1 description description.json --contract campaign-send-approval-0.1.json -npx mailschema@0.3.1 implementation record.json +npx mailschema@0.3.2 contract campaign-send-approval-0.1.json +npx mailschema@0.3.2 description description.json --contract campaign-send-approval-0.1.json +npx mailschema@0.3.2 implementation record.json +npx mailschema@0.3.2 message approval.eml > description.json ``` -Each command first prints a line naming what it checked. `contract` prints `Valid type contract ` and, on the second line, the contract's digest. `description` prints `Valid MAP 0.3 description <@id>` and, on the second line, the description's digest; with `--contract`, it first checks that the description uses that contract as it allows. `implementation` prints `Valid implementation record for `. A digest is `sha-256:` and the SHA-256 of the document's RFC 8785 canonical JSON, the value `digest()` returns; the description's digest is not the contract digest it names. An invalid file prints every reason on standard error and exits with status 1. Nothing is uploaded or changed. +Each command first prints a line naming what it checked. `contract` prints `Valid type contract ` and, on the second line, the contract's digest. `description` prints `Valid MAP 0.3 description <@id>` and, on the second line, the description's digest; with `--contract`, it first checks that the description uses that contract as it allows. `implementation` prints `Valid implementation record for `. `message` reads a received email (a saved `.eml`), writes its one MAP description to standard output exactly as the MAP part carries it, and prints the `description` lines on standard error. A digest is `sha-256:` and the SHA-256 of the document's RFC 8785 canonical JSON, the value `digest()` returns; the description's digest is not the contract digest it names. An invalid file prints every reason on standard error and exits with status 1. Nothing is uploaded or changed. Checking does not authenticate a message, establish a service's authority or perform an action. Those belong to the implementation, under the [specification](https://mailschema.org/specification/core). diff --git a/UPSTREAM.md b/UPSTREAM.md index 39b7f7f..63f61c4 100644 --- a/UPSTREAM.md +++ b/UPSTREAM.md @@ -1,6 +1,6 @@ # Release provenance -Version `0.3.1` derives from [`mailschema/mailschema@67f9544`](https://github.com/mailschema/mailschema/commit/67f95447f6f2e640af8f515d6844f58d1b1d1fa2). Its sources, tests and bundled files are prepared from that commit by `npm run packages:prepare`. +Version `0.3.2` derives from [`mailschema/mailschema@1b41186`](https://github.com/mailschema/mailschema/commit/1b41186a0c843cd12a574038fa9e7bcfca913765). Its sources, tests and bundled files are prepared from that commit by `npm run packages:prepare`. The bundled MAP 0.3 artifacts are exact copies of that commit's files and do not independently define MAP: diff --git a/bin/mailschema.js b/bin/mailschema.js index b0f677a..1c61d53 100755 --- a/bin/mailschema.js +++ b/bin/mailschema.js @@ -5,6 +5,7 @@ import { Contract, CONTRACT_MAX_BYTES, digest, + extractMessage, InvalidDocument, parseDescription, parseImplementation, @@ -15,17 +16,23 @@ const help = `Mail Action Protocol 0.3 checks mailschema contract mailschema description [--contract ] mailschema implementation + mailschema message contract checks a type contract and prints its digest. description checks a description and, with --contract, that it uses that contract as it allows, then prints the description's digest. implementation checks a Registry -implementation record. A digest is the SHA-256 of the document's RFC 8785 -canonical JSON. Nothing is uploaded or changed.`; +implementation record. message finds a received email's one MAP part, writes +its description to standard output exactly as the part carries it, and prints +the description's digest on standard error. A digest is the SHA-256 of the +document's RFC 8785 canonical JSON. Nothing is uploaded or changed.`; -async function read(path) { +// A whole email, attachments included, is far larger than any MAP document. +const MESSAGE_MAX_BYTES = 26214400; + +async function read(path, maxBytes = CONTRACT_MAX_BYTES, kind = 'a JSON file') { const info = await stat(path); - if (!info.isFile() || info.size > CONTRACT_MAX_BYTES) - throw new Error(`Expected a JSON file of at most ${CONTRACT_MAX_BYTES} bytes.`); + if (!info.isFile() || info.size > maxBytes) + throw new Error(`Expected ${kind} of at most ${maxBytes} bytes.`); return new Uint8Array(await readFile(path)); } @@ -51,6 +58,12 @@ try { if (values.contract) report(Contract.parse(await read(values.contract)).descriptionErrors(description)); console.log(`Valid MAP 0.3 description ${description['@id']}\n${digest(description)}`); + } else if (command === 'message') { + const result = extractMessage(await read(path, MESSAGE_MAX_BYTES, 'an email file')); + if (result.status === 'none') throw new Error('The message has no MAP 0.3 part.'); + if (result.status === 'invalid') throw new Error(`The message's MAP part cannot be used: ${result.reason}.`); + process.stdout.write(result.bytes); + console.error(`Valid MAP 0.3 description ${result.description['@id']}\n${digest(result.description)}`); } else if (command === 'implementation') { const record = parseImplementation(await read(path)); console.log(`Valid implementation record for ${record.service}`); diff --git a/dist/core/artifacts.js b/dist/core/artifacts.js index 619ce36..6ca25fe 100644 --- a/dist/core/artifacts.js +++ b/dist/core/artifacts.js @@ -26,5 +26,15 @@ export function createValidator() { ajv.removeKeyword(keyword); return ajv; } -/** Each validation error as its JSON Pointer and message. @internal */ -export const schemaErrors = (errors, prefix = '') => (errors ?? []).map((error) => `${prefix + error.instancePath || '/'}: ${error.message}`); +/** + * Each validation error as its JSON Pointer and message. An enum or const error names the + * values it allows, so the fix is in the message. @internal + */ +export const schemaErrors = (errors, prefix = '') => (errors ?? []).map((error) => `${prefix + error.instancePath || '/'}: ${schemaMessage(error)}`); +function schemaMessage(error) { + if (error.keyword === 'enum') + return `must be one of ${error.params.allowedValues.map((value) => JSON.stringify(value)).join(', ')}`; + if (error.keyword === 'const') + return `must be ${JSON.stringify(error.params.allowedValue)}`; + return error.message; +} diff --git a/dist/core/index.d.ts b/dist/core/index.d.ts index 2b64971..aa164f1 100644 --- a/dist/core/index.d.ts +++ b/dist/core/index.d.ts @@ -4,4 +4,5 @@ export { sha256 } from './sha256.ts'; export { descriptionErrors, parseDescription, type Description } from './description.ts'; export { CAPABILITY_KINDS, Contract, contractErrors, type CapabilityKind, type ContractDocument, type Operation, } from './contract.ts'; export { implementationErrors, parseImplementation, type ImplementationRecord, } from './implementation.ts'; +export { extractMessage, type MessageProblem, type MessageResult } from './message.ts'; export { unportablePattern } from './patterns.ts'; diff --git a/dist/core/index.js b/dist/core/index.js index 427d5b2..cbf524d 100644 --- a/dist/core/index.js +++ b/dist/core/index.js @@ -6,4 +6,5 @@ export { sha256 } from "./sha256.js"; export { descriptionErrors, parseDescription } from "./description.js"; export { CAPABILITY_KINDS, Contract, contractErrors, } from "./contract.js"; export { implementationErrors, parseImplementation, } from "./implementation.js"; +export { extractMessage } from "./message.js"; export { unportablePattern } from "./patterns.js"; diff --git a/dist/core/message.d.ts b/dist/core/message.d.ts new file mode 100644 index 0000000..f9ebc5c --- /dev/null +++ b/dist/core/message.d.ts @@ -0,0 +1,27 @@ +import { type Description } from './description.ts'; +/** + * Why a message's MAP part cannot be used: more than one designated part, a header that + * readers could read differently, a transfer encoding other than base64 or quoted-printable, + * a part not labelled exactly with this profile, a part outside a multipart/related entity + * whose root is readable, an invalid description, or a message that cannot be read. + */ +export type MessageProblem = 'multiple_parts' | 'ambiguous_headers' | 'transfer_encoding' | 'media_type' | 'not_partial_representation' | 'invalid_description' | 'unreadable'; +/** What a message carries: no MAP part, an unusable one and why, or its description. */ +export type MessageResult = { + status: 'none'; +} | { + status: 'invalid'; + reason: MessageProblem; +} | { + status: 'valid'; + description: Description; + bytes: Uint8Array; +}; +/** + * The MAP part of a received email (Core, "Email representation"), read from its original + * bytes: the one designated description of this profile, in a multipart/related entity whose + * root is the readable content it describes. A description inside an attachment or an + * attached message is never read. It never throws for untrusted input. `bytes` are the + * description exactly as the part carries it, after its transfer encoding is removed. + */ +export declare function extractMessage(input: string | Uint8Array): MessageResult; diff --git a/dist/core/message.js b/dist/core/message.js new file mode 100644 index 0000000..2ab681d --- /dev/null +++ b/dist/core/message.js @@ -0,0 +1,262 @@ +import { PROFILE } from "./artifacts.js"; +import { parseDescription } from "./description.js"; +import { InvalidDocument } from "./json.js"; +const READABLE = new Set(['text/plain', 'text/html']); +const TRANSFER_ENCODINGS = new Set(['base64', 'quoted-printable']); +const SINGLE_FIELDS = ['content-type', 'content-purpose', 'content-transfer-encoding']; +const FIELD = /^([!-9;-~]+):[ \t]*([\s\S]*)$/; +const PARAMETER = /^\s*([!#$%&'*+.^_`|~0-9A-Za-z-]+)\s*=/; +/** + * The MAP part of a received email (Core, "Email representation"), read from its original + * bytes: the one designated description of this profile, in a multipart/related entity whose + * root is the readable content it describes. A description inside an attachment or an + * attached message is never read. It never throws for untrusted input. `bytes` are the + * description exactly as the part carries it, after its transfer encoding is removed. + */ +export function extractMessage(input) { + try { + const raw = typeof input === 'string' ? input : binary(input); + const designated = entities(entity(raw)).filter(([part]) => isDesignated(part)); + if (!designated.length) + return { status: 'none' }; + if (designated.length > 1) + return invalid('multiple_parts'); + const [part, parent] = designated[0]; + const reason = problem(part, parent); + if (reason) + return invalid(reason); + const bytes = decoded(part); + try { + return { status: 'valid', description: parseDescription(bytes), bytes }; + } + catch (error) { + if (error instanceof InvalidDocument) + return invalid('invalid_description'); + throw error; + } + } + catch { + return invalid('unreadable'); + } +} +function invalid(reason) { + return { status: 'invalid', reason }; +} +// Every part with its parent, at any depth. An attachment or an attached message is a leaf, +// so nothing inside one is read. +function entities(parent) { + return parent.parts.flatMap((part) => attached(part) || part.mimeType === 'message/rfc822' ? [] : [[part, parent], ...entities(part)]); +} +// A designated part labelled with this profile, judged from its original header lines, +// before a parser picks one of repeated fields or parameters. +function isDesignated(part) { + const purposes = values(part, 'content-purpose'); + if (!purposes.some((value) => value.trim().toLowerCase() === 'machine-readable')) + return false; + const types = values(part, 'content-type'); + return types.some((value) => value.toLowerCase().includes('application/ld+json')) && types.some((value) => value.includes(PROFILE)); +} +function problem(part, parent) { + if (!unambiguous(part)) + return 'ambiguous_headers'; + if (!TRANSFER_ENCODINGS.has(first(part, 'content-transfer-encoding').trim().toLowerCase())) + return 'transfer_encoding'; + if (part.mimeType !== 'application/ld+json' || part.parameters.get('profile') !== PROFILE) + return 'media_type'; + if (!partialRepresentation(parent)) + return 'not_partial_representation'; + return undefined; +} +// Each field that decides how the part is read appears once, with each parameter once. +function unambiguous(part) { + if (!part.fields) + return false; + const counts = new Map(); + for (const [name] of part.fields) + counts.set(name, (counts.get(name) ?? 0) + 1); + return SINGLE_FIELDS.every((name) => counts.get(name) === 1) && uniqueParameters(first(part, 'content-type')); +} +// A multipart/related entity with one unambiguous Content-Type, whose root (its start +// parameter's part, or else its first) is readable. +function partialRepresentation(parent) { + if (parent.mimeType !== 'multipart/related') + return false; + const types = values(parent, 'content-type'); + if (types.length !== 1 || !uniqueParameters(types[0])) + return false; + const start = parent.parameters.get('start') ?? ''; + const roots = start ? parent.parts.filter((part) => first(part, 'content-id').trim() === start) : parent.parts.slice(0, 1); + return roots.length === 1 && readable(roots[0]); +} +function readable(part) { + if (attached(part)) + return false; + if (part.mimeType === 'multipart/alternative') + return part.parts.some(readable); + return READABLE.has(part.mimeType) && binary(decoded(part)).trim().length > 0; +} +function attached(part) { + return first(part, 'content-disposition').trim().toLowerCase().startsWith('attachment'); +} +function values(part, name) { + return (part.fields ?? []).filter(([field]) => field === name).map(([, value]) => value); +} +function first(part, name) { + return values(part, name)[0] ?? ''; +} +// One entity from its raw bytes (a binary string, one character per byte): its header, +// Content-Type and, for a multipart entity, its parts. +function entity(raw) { + const fields = headerFields(raw); + const body = split(raw)?.[1] ?? ''; + const contentType = (fields ?? []).find(([name]) => name === 'content-type')?.[1] ?? 'text/plain'; + const [type, ...rest] = segments(contentType) ?? [contentType]; + const mimeType = type.trim().toLowerCase(); + const parameters = new Map(); + for (const segment of rest) { + const name = PARAMETER.exec(segment)?.[1]; + if (!name) + continue; + const key = name.toLowerCase(); + if (!parameters.has(key)) + parameters.set(key, unquote(segment.slice(segment.indexOf('=') + 1).trim())); + } + const boundary = parameters.get('boundary'); + const parts = mimeType.startsWith('multipart/') && boundary ? multipart(body, boundary).map(entity) : []; + return { fields, mimeType, parameters, body, parts }; +} +// The raw bytes of each body part between the boundary's delimiter lines; the preamble and +// epilogue are not parts. +function multipart(body, boundary) { + const delimiter = `--${boundary}`; + const parts = []; + let current = null; + // The line break before a delimiter line belongs to the delimiter, not to the part. + const close = () => { + if (current) + parts.push(current.join('').replace(/\r?\n$/, '')); + }; + for (const line of body.split(/(?<=\n)/)) { + const content = line.replace(/\r?\n$/, '').replace(/[ \t]+$/, ''); + if (content === `${delimiter}--`) { + close(); + return parts; + } + if (content === delimiter) { + close(); + current = []; + } + else + current?.push(line); + } + close(); + return parts; +} +// An entity's header block and body, split at the first empty line; a part that begins with +// an empty line has no header fields. Null when there is no empty line. +function split(raw) { + const leading = /^\r?\n/.exec(raw); + if (leading) + return ['', raw.slice(leading[0].length)]; + const separator = /\r?\n\r?\n/.exec(raw); + return separator ? [raw.slice(0, separator.index), raw.slice(separator.index + separator[0].length)] : null; +} +// The unfolded fields of an entity's header block as [lowercase name, value] pairs, or null +// when it is malformed. Lines end with CRLF or LF. +function headerFields(raw) { + const parts = split(raw); + if (!parts) + return null; + const lines = parts[0] ? parts[0].split(/\r?\n/) : []; + const fields = []; + for (const line of lines) { + if (/[\r\n]/.test(line)) + return null; + if (line.startsWith(' ') || line.startsWith('\t')) { + if (!fields.length) + return null; + fields[fields.length - 1][1] += ` ${line.trim()}`; + } + else { + const match = FIELD.exec(line); + if (!match) + return null; + fields.push([match[1].toLowerCase(), match[2]]); + } + } + return fields; +} +// Whether every parameter of a Content-Type value has its own name. RFC 2231 sections and +// charset markers belong to the name they extend, so profile and profile* are one name. +function uniqueParameters(value) { + const parts = segments(value); + if (!parts) + return false; + const names = []; + for (const segment of parts.slice(1)) { + const name = PARAMETER.exec(segment)?.[1]; + if (!name) + return false; + names.push(name.toLowerCase().replace(/\*.*$/, '')); + } + return new Set(names).size === names.length; +} +// The value split at each semicolon outside a quoted string, or null when a quoted string or +// an escape is left open. +function segments(value) { + const parts = ['']; + let quoted = false; + let escaped = false; + for (const char of value) { + if (char === ';' && !quoted) { + parts.push(''); + continue; + } + if (escaped) + escaped = false; + else if (quoted && char === '\\') + escaped = true; + else if (char === '"') + quoted = !quoted; + parts[parts.length - 1] += char; + } + return quoted || escaped ? null : parts; +} +function unquote(value) { + return value.startsWith('"') && value.endsWith('"') && value.length >= 2 + ? value.slice(1, -1).replace(/\\(.)/g, '$1') + : value; +} +// The part's body with its transfer encoding removed. +function decoded(part) { + const encoding = first(part, 'content-transfer-encoding').trim().toLowerCase(); + if (encoding === 'base64') + return base64(part.body); + if (encoding === 'quoted-printable') + return quotedPrintable(part.body); + return bytesOf(part.body); +} +// Characters outside the alphabet are skipped, as mail readers do, so a damaged body reads as +// an invalid description rather than an unreadable message. +function base64(body) { + let clean = body.replace(/[^A-Za-z0-9+/]/g, ''); + if (clean.length % 4 === 1) + clean = clean.slice(0, -1); + return bytesOf(atob(clean.padEnd(Math.ceil(clean.length / 4) * 4, '='))); +} +function quotedPrintable(body) { + const text = body.replace(/[ \t]+(?=\r?\n)/g, '').replace(/=\r?\n/g, ''); + return bytesOf(text.replace(/=([0-9A-Fa-f]{2})/g, (_match, hex) => String.fromCharCode(parseInt(hex, 16)))); +} +function binary(bytes) { + let text = ''; + for (let index = 0; index < bytes.length; index += 0x8000) + text += String.fromCharCode(...bytes.subarray(index, index + 0x8000)); + return text; +} +function bytesOf(text) { + const bytes = new Uint8Array(text.length); + for (let index = 0; index < text.length; index += 1) + bytes[index] = text.charCodeAt(index) & 0xff; + return bytes; +} diff --git a/package-lock.json b/package-lock.json index b2474fa..192b662 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "mailschema", - "version": "0.3.1", + "version": "0.3.2", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "mailschema", - "version": "0.3.1", + "version": "0.3.2", "license": "MIT", "dependencies": { "ajv": "8.20.0", diff --git a/package.json b/package.json index 324a637..9f696f5 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "mailschema", - "version": "0.3.1", + "version": "0.3.2", "description": "Mail Action Protocol 0.3: parse and check descriptions, type contracts and implementation records", "type": "module", "license": "MIT", diff --git a/src/core/artifacts.ts b/src/core/artifacts.ts index c652a56..9285dc7 100644 --- a/src/core/artifacts.ts +++ b/src/core/artifacts.ts @@ -29,6 +29,17 @@ export function createValidator() { return ajv; } -/** Each validation error as its JSON Pointer and message. @internal */ +/** + * Each validation error as its JSON Pointer and message. An enum or const error names the + * values it allows, so the fix is in the message. @internal + */ export const schemaErrors = (errors: ErrorObject[] | null | undefined, prefix = '') => - (errors ?? []).map((error) => `${prefix + error.instancePath || '/'}: ${error.message}`); + (errors ?? []).map((error) => `${prefix + error.instancePath || '/'}: ${schemaMessage(error)}`); + +function schemaMessage(error: ErrorObject): string | undefined { + if (error.keyword === 'enum') + return `must be one of ${(error.params as { allowedValues: unknown[] }).allowedValues.map((value) => JSON.stringify(value)).join(', ')}`; + if (error.keyword === 'const') + return `must be ${JSON.stringify((error.params as { allowedValue: unknown }).allowedValue)}`; + return error.message; +} diff --git a/src/core/index.ts b/src/core/index.ts index 44cded9..f4d3dfd 100644 --- a/src/core/index.ts +++ b/src/core/index.ts @@ -28,4 +28,5 @@ export { parseImplementation, type ImplementationRecord, } from './implementation.ts'; +export { extractMessage, type MessageProblem, type MessageResult } from './message.ts'; export { unportablePattern } from './patterns.ts'; diff --git a/src/core/message.ts b/src/core/message.ts new file mode 100644 index 0000000..284b02c --- /dev/null +++ b/src/core/message.ts @@ -0,0 +1,286 @@ +import { PROFILE } from './artifacts.ts'; +import { parseDescription, type Description } from './description.ts'; +import { InvalidDocument } from './json.ts'; + +/** + * Why a message's MAP part cannot be used: more than one designated part, a header that + * readers could read differently, a transfer encoding other than base64 or quoted-printable, + * a part not labelled exactly with this profile, a part outside a multipart/related entity + * whose root is readable, an invalid description, or a message that cannot be read. + */ +export type MessageProblem = + | 'multiple_parts' + | 'ambiguous_headers' + | 'transfer_encoding' + | 'media_type' + | 'not_partial_representation' + | 'invalid_description' + | 'unreadable'; + +/** What a message carries: no MAP part, an unusable one and why, or its description. */ +export type MessageResult = + | { status: 'none' } + | { status: 'invalid'; reason: MessageProblem } + | { status: 'valid'; description: Description; bytes: Uint8Array }; + +const READABLE = new Set(['text/plain', 'text/html']); +const TRANSFER_ENCODINGS = new Set(['base64', 'quoted-printable']); +const SINGLE_FIELDS = ['content-type', 'content-purpose', 'content-transfer-encoding']; +const FIELD = /^([!-9;-~]+):[ \t]*([\s\S]*)$/; +const PARAMETER = /^\s*([!#$%&'*+.^_`|~0-9A-Za-z-]+)\s*=/; + +type Field = [name: string, value: string]; + +interface Entity { + /** The unfolded header fields, or null when the header block is malformed. */ + fields: Field[] | null; + mimeType: string; + parameters: Map; + body: string; + parts: Entity[]; +} + +/** + * The MAP part of a received email (Core, "Email representation"), read from its original + * bytes: the one designated description of this profile, in a multipart/related entity whose + * root is the readable content it describes. A description inside an attachment or an + * attached message is never read. It never throws for untrusted input. `bytes` are the + * description exactly as the part carries it, after its transfer encoding is removed. + */ +export function extractMessage(input: string | Uint8Array): MessageResult { + try { + const raw = typeof input === 'string' ? input : binary(input); + const designated = entities(entity(raw)).filter(([part]) => isDesignated(part)); + if (!designated.length) return { status: 'none' }; + if (designated.length > 1) return invalid('multiple_parts'); + + const [part, parent] = designated[0]; + const reason = problem(part, parent); + if (reason) return invalid(reason); + + const bytes = decoded(part); + try { + return { status: 'valid', description: parseDescription(bytes), bytes }; + } catch (error) { + if (error instanceof InvalidDocument) return invalid('invalid_description'); + throw error; + } + } catch { + return invalid('unreadable'); + } +} + +function invalid(reason: MessageProblem): MessageResult { + return { status: 'invalid', reason }; +} + +// Every part with its parent, at any depth. An attachment or an attached message is a leaf, +// so nothing inside one is read. +function entities(parent: Entity): Array<[Entity, Entity]> { + return parent.parts.flatMap((part) => + attached(part) || part.mimeType === 'message/rfc822' ? [] : [[part, parent] as [Entity, Entity], ...entities(part)], + ); +} + +// A designated part labelled with this profile, judged from its original header lines, +// before a parser picks one of repeated fields or parameters. +function isDesignated(part: Entity): boolean { + const purposes = values(part, 'content-purpose'); + if (!purposes.some((value) => value.trim().toLowerCase() === 'machine-readable')) return false; + const types = values(part, 'content-type'); + return types.some((value) => value.toLowerCase().includes('application/ld+json')) && types.some((value) => value.includes(PROFILE)); +} + +function problem(part: Entity, parent: Entity): MessageProblem | undefined { + if (!unambiguous(part)) return 'ambiguous_headers'; + if (!TRANSFER_ENCODINGS.has(first(part, 'content-transfer-encoding').trim().toLowerCase())) return 'transfer_encoding'; + if (part.mimeType !== 'application/ld+json' || part.parameters.get('profile') !== PROFILE) return 'media_type'; + if (!partialRepresentation(parent)) return 'not_partial_representation'; + return undefined; +} + +// Each field that decides how the part is read appears once, with each parameter once. +function unambiguous(part: Entity): boolean { + if (!part.fields) return false; + const counts = new Map(); + for (const [name] of part.fields) counts.set(name, (counts.get(name) ?? 0) + 1); + return SINGLE_FIELDS.every((name) => counts.get(name) === 1) && uniqueParameters(first(part, 'content-type')); +} + +// A multipart/related entity with one unambiguous Content-Type, whose root (its start +// parameter's part, or else its first) is readable. +function partialRepresentation(parent: Entity): boolean { + if (parent.mimeType !== 'multipart/related') return false; + const types = values(parent, 'content-type'); + if (types.length !== 1 || !uniqueParameters(types[0])) return false; + + const start = parent.parameters.get('start') ?? ''; + const roots = start ? parent.parts.filter((part) => first(part, 'content-id').trim() === start) : parent.parts.slice(0, 1); + return roots.length === 1 && readable(roots[0]); +} + +function readable(part: Entity): boolean { + if (attached(part)) return false; + if (part.mimeType === 'multipart/alternative') return part.parts.some(readable); + return READABLE.has(part.mimeType) && binary(decoded(part)).trim().length > 0; +} + +function attached(part: Entity): boolean { + return first(part, 'content-disposition').trim().toLowerCase().startsWith('attachment'); +} + +function values(part: Entity, name: string): string[] { + return (part.fields ?? []).filter(([field]) => field === name).map(([, value]) => value); +} + +function first(part: Entity, name: string): string { + return values(part, name)[0] ?? ''; +} + +// One entity from its raw bytes (a binary string, one character per byte): its header, +// Content-Type and, for a multipart entity, its parts. +function entity(raw: string): Entity { + const fields = headerFields(raw); + const body = split(raw)?.[1] ?? ''; + const contentType = (fields ?? []).find(([name]) => name === 'content-type')?.[1] ?? 'text/plain'; + const [type, ...rest] = segments(contentType) ?? [contentType]; + const mimeType = type.trim().toLowerCase(); + const parameters = new Map(); + for (const segment of rest) { + const name = PARAMETER.exec(segment)?.[1]; + if (!name) continue; + const key = name.toLowerCase(); + if (!parameters.has(key)) parameters.set(key, unquote(segment.slice(segment.indexOf('=') + 1).trim())); + } + const boundary = parameters.get('boundary'); + const parts = mimeType.startsWith('multipart/') && boundary ? multipart(body, boundary).map(entity) : []; + return { fields, mimeType, parameters, body, parts }; +} + +// The raw bytes of each body part between the boundary's delimiter lines; the preamble and +// epilogue are not parts. +function multipart(body: string, boundary: string): string[] { + const delimiter = `--${boundary}`; + const parts: string[] = []; + let current: string[] | null = null; + // The line break before a delimiter line belongs to the delimiter, not to the part. + const close = () => { + if (current) parts.push(current.join('').replace(/\r?\n$/, '')); + }; + for (const line of body.split(/(?<=\n)/)) { + const content = line.replace(/\r?\n$/, '').replace(/[ \t]+$/, ''); + if (content === `${delimiter}--`) { + close(); + return parts; + } + if (content === delimiter) { + close(); + current = []; + } else current?.push(line); + } + close(); + return parts; +} + +// An entity's header block and body, split at the first empty line; a part that begins with +// an empty line has no header fields. Null when there is no empty line. +function split(raw: string): [header: string, body: string] | null { + const leading = /^\r?\n/.exec(raw); + if (leading) return ['', raw.slice(leading[0].length)]; + const separator = /\r?\n\r?\n/.exec(raw); + return separator ? [raw.slice(0, separator.index), raw.slice(separator.index + separator[0].length)] : null; +} + +// The unfolded fields of an entity's header block as [lowercase name, value] pairs, or null +// when it is malformed. Lines end with CRLF or LF. +function headerFields(raw: string): Field[] | null { + const parts = split(raw); + if (!parts) return null; + const lines = parts[0] ? parts[0].split(/\r?\n/) : []; + const fields: Field[] = []; + for (const line of lines) { + if (/[\r\n]/.test(line)) return null; + if (line.startsWith(' ') || line.startsWith('\t')) { + if (!fields.length) return null; + fields[fields.length - 1][1] += ` ${line.trim()}`; + } else { + const match = FIELD.exec(line); + if (!match) return null; + fields.push([match[1].toLowerCase(), match[2]]); + } + } + return fields; +} + +// Whether every parameter of a Content-Type value has its own name. RFC 2231 sections and +// charset markers belong to the name they extend, so profile and profile* are one name. +function uniqueParameters(value: string): boolean { + const parts = segments(value); + if (!parts) return false; + const names: string[] = []; + for (const segment of parts.slice(1)) { + const name = PARAMETER.exec(segment)?.[1]; + if (!name) return false; + names.push(name.toLowerCase().replace(/\*.*$/, '')); + } + return new Set(names).size === names.length; +} + +// The value split at each semicolon outside a quoted string, or null when a quoted string or +// an escape is left open. +function segments(value: string): string[] | null { + const parts = ['']; + let quoted = false; + let escaped = false; + for (const char of value) { + if (char === ';' && !quoted) { + parts.push(''); + continue; + } + if (escaped) escaped = false; + else if (quoted && char === '\\') escaped = true; + else if (char === '"') quoted = !quoted; + parts[parts.length - 1] += char; + } + return quoted || escaped ? null : parts; +} + +function unquote(value: string): string { + return value.startsWith('"') && value.endsWith('"') && value.length >= 2 + ? value.slice(1, -1).replace(/\\(.)/g, '$1') + : value; +} + +// The part's body with its transfer encoding removed. +function decoded(part: Entity): Uint8Array { + const encoding = first(part, 'content-transfer-encoding').trim().toLowerCase(); + if (encoding === 'base64') return base64(part.body); + if (encoding === 'quoted-printable') return quotedPrintable(part.body); + return bytesOf(part.body); +} + +// Characters outside the alphabet are skipped, as mail readers do, so a damaged body reads as +// an invalid description rather than an unreadable message. +function base64(body: string): Uint8Array { + let clean = body.replace(/[^A-Za-z0-9+/]/g, ''); + if (clean.length % 4 === 1) clean = clean.slice(0, -1); + return bytesOf(atob(clean.padEnd(Math.ceil(clean.length / 4) * 4, '='))); +} + +function quotedPrintable(body: string): Uint8Array { + const text = body.replace(/[ \t]+(?=\r?\n)/g, '').replace(/=\r?\n/g, ''); + return bytesOf(text.replace(/=([0-9A-Fa-f]{2})/g, (_match, hex: string) => String.fromCharCode(parseInt(hex, 16)))); +} + +function binary(bytes: Uint8Array): string { + let text = ''; + for (let index = 0; index < bytes.length; index += 0x8000) + text += String.fromCharCode(...bytes.subarray(index, index + 0x8000)); + return text; +} + +function bytesOf(text: string): Uint8Array { + const bytes = new Uint8Array(text.length); + for (let index = 0; index < text.length; index += 1) bytes[index] = text.charCodeAt(index) & 0xff; + return bytes; +} diff --git a/test/fixtures/examples/publication.eml b/test/fixtures/examples/publication.eml new file mode 100644 index 0000000..07b3064 --- /dev/null +++ b/test/fixtures/examples/publication.eml @@ -0,0 +1,56 @@ +From: Documentation +To: owner@example.net +Subject: Publish the getting started guide +Date: Fri, 02 Oct 2026 08:00:00 +0000 +Message-ID: +MIME-Version: 1.0 +Content-Type: multipart/related; type="text/plain"; + boundary="map-publication-example"; start="" + +--map-publication-example +Content-Type: text/plain; charset=utf-8 +Content-Transfer-Encoding: 7bit +Content-ID: + +Please review Getting started (r4). +Approval authorizes publication at: +https://docs.service.example/guide (public) +Schedule: immediate. +You can approve, decline or request changes. +Review the proposal: https://service.example/review/p7 + +--map-publication-example +Content-Type: application/ld+json; + profile="https://mailschema.org/profiles/map/0.3" +Content-Purpose: Machine-readable +Content-Transfer-Encoding: base64 + +ewogICJAY29udGV4dCI6ICJodHRwczovL21haWxzY2hlbWEub3JnL2NvbnRleHRzL21hcC0wLjMu +anNvbmxkIiwKICAiQHR5cGUiOiAiTWFpbEFjdGlvbiIsCiAgInByb2ZpbGUiOiAiaHR0cHM6Ly9t +YWlsc2NoZW1hLm9yZy9wcm9maWxlcy9tYXAvMC4zIiwKICAidHlwZSI6IHsKICAgICJpZCI6ICJo +dHRwczovL21haWxzY2hlbWEub3JnL3R5cGVzL3B1YmxpY2F0aW9uLWFwcHJvdmFsIiwKICAgICJ2 +ZXJzaW9uIjogIjAuMSIsCiAgICAiY29udHJhY3REaWdlc3QiOiAic2hhLTI1NjpkZTkxYTQ5ZGU0 +Y2JhOTEyMDliZTI1MzQ1NmU3YzU2NzA5NzkxZmZmODc3YTA2MDU2MGQ0ZWFkYTBjYzk0NjhjIgog +IH0sCiAgIkBpZCI6ICJ1cm46dXVpZDowMTk5YTZjMi00YjFlLTdjM2EtOWQyZS01ZjhhMWIyYzNk +NDEiLAogICJpc3N1ZWRBdCI6ICIyMDI2LTEwLTAyVDA4OjAwOjAwWiIsCiAgImV4cGlyZXNBdCI6 +ICIyMDI2LTEwLTAzVDA4OjAwOjAwWiIsCiAgImluTGFuZ3VhZ2UiOiAiZW4iLAogICJzZXJ2aWNl +IjogewogICAgImlkIjogImh0dHBzOi8vc2VydmljZS5leGFtcGxlIiwKICAgICJ0ZW5hbnQiOiAi +dXJuOmV4YW1wbGU6dGVuYW50OjciCiAgfSwKICAicmVjaXBpZW50IjogIm93bmVyQGV4YW1wbGUu +bmV0IiwKICAic3ViamVjdCI6IHsKICAgICJpZCI6ICJ1cm46ZXhhbXBsZTpvYmplY3Q6NDIiLAog +ICAgInRpdGxlIjogIlB1Ymxpc2ggdGhlIGdldHRpbmcgc3RhcnRlZCBndWlkZSIKICB9LAogICJ0 +ZXJtcyI6IHsKICAgICJpZCI6ICJ1cm46ZXhhbXBsZTpwcm9wb3NhbDpwNyIsCiAgICAidmVyc2lv +biI6ICJwNy1yNCIKICB9LAogICJodW1hbiI6IHsKICAgICJ1cmwiOiAiaHR0cHM6Ly9zZXJ2aWNl +LmV4YW1wbGUvcmV2aWV3L3A3IgogIH0sCiAgImRldGFpbHMiOiB7CiAgICAic3VtbWFyeSI6ICJQ +dWJsaXNoIHRoZSBhcHByb3ZlZCBnZXR0aW5nIHN0YXJ0ZWQgZ3VpZGUuIiwKICAgICJyZXF1ZXN0 +ZXIiOiAiRG9jdW1lbnRhdGlvbiBlZGl0b3IiLAogICAgImNvbnRlbnQiOiB7CiAgICAgICJpZCI6 +ICJ1cm46ZXhhbXBsZTpjb250ZW50OjQyIiwKICAgICAgInJldmlzaW9uIjogInI0IiwKICAgICAg +InRpdGxlIjogIkdldHRpbmcgc3RhcnRlZCIsCiAgICAgICJkaWdlc3QiOiAic2hhLTI1Njo0ZmZk +ZjUwNGM0NGE3M2EwNTljMDljOWNiMzAzNjJjMDRmNTkzYjNhZjQwNWI4M2JjMDg0YmIwNDUwNjdm +OTVjIgogICAgfSwKICAgICJkZXN0aW5hdGlvbnMiOiBbCiAgICAgIHsKICAgICAgICAiaWQiOiAi +aHR0cHM6Ly9kb2NzLnNlcnZpY2UuZXhhbXBsZS9ndWlkZSIsCiAgICAgICAgImFjY291bnQiOiAi +dXJuOmV4YW1wbGU6YWNjb3VudDo3IiwKICAgICAgICAidmlzaWJpbGl0eSI6ICJwdWJsaWMiCiAg +ICAgIH0KICAgIF0sCiAgICAic2NoZWR1bGUiOiB7CiAgICAgICJtb2RlIjogImltbWVkaWF0ZSIK +ICAgIH0KICB9LAogICJvcGVyYXRpb25zIjogWwogICAgewogICAgICAiaWQiOiAiYXBwcm92ZSIK +ICAgIH0sCiAgICB7CiAgICAgICJpZCI6ICJkZWNsaW5lIgogICAgfSwKICAgIHsKICAgICAgImlk +IjogInJlcXVlc3QtY2hhbmdlcyIKICAgIH0KICBdCn0K +--map-publication-example-- diff --git a/test/package.test.mjs b/test/package.test.mjs index f60ecc5..11693a0 100644 --- a/test/package.test.mjs +++ b/test/package.test.mjs @@ -98,7 +98,10 @@ test('checks files from the command line and prints their digests', async () => assert.equal(run('description', description), output, slug); assert.equal(run('description', description, '--contract', contract), output, slug); } + const extracted = execFileSync(process.execPath, [cli, 'message', path('examples/publication.eml')], { stdio: 'pipe' }); + assert.deepEqual(parseDescription(extracted), await load('examples/publication-approval.json')); const campaign = path('examples/campaign-send-approval.json'); + assert.throws(() => run('message', campaign)); assert.throws(() => run('contract', campaign)); assert.throws(() => run('description', campaign, '--contract', path('contracts/publication-approval-0.1.json')),