Skip to content

Commit 7fb8511

Browse files
committed
Lossy WebP animations: unchanged pixels within a tolerance, in 8x8 blocks
Optimized lossy frames made each exactly unchanged pixel transparent. Where most pixels change a little (cross-fades), only scattered pixels stay the same: the alpha masks were speckled and could cost more than the color data they saved, so optimized files got larger than unoptimized ones (reported: 8.9 MB instead of 7.0 MB). Like libwebp's WebPAnimEncoder (QualityToMaxDiff, FlattenSimilarBlocks), lossy frames now - count pixels whose channels changed by up to a tolerance from the quality (3 at quality 85, 1 at 100) as unchanged, for the frame rectangle too (compared with what's shown, so errors don't add up); - make unchanged pixels transparent only in whole 8x8 blocks (colored like the average of what's shown), and are sparse whenever a block is. Six cross-faded radar-like animations at quality 85: 2% larger to 13% smaller than before, 7-46% smaller than unoptimized, PSNR 0.1 dB below unoptimized (before: 0.8-1 dB). A layer whose color changes by 1-2 per frame: 16 kB instead of 158 kB (unoptimized: 246 kB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BsU2GDnECphmYTjLB5UeZv
1 parent 5bb9034 commit 7fb8511

5 files changed

Lines changed: 163 additions & 16 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,14 @@
22

33
## 3.7.1
44

5-
### Fixes
5+
### Improvements
6+
7+
- Optimized lossy WebP animations are smaller: like libwebp's, they leave out pixels that changed by up to a tolerance
8+
from the quality (3 per channel at quality 85, 1 at 100), and make unchanged pixels transparent only in whole 8x8
9+
blocks. Making each unchanged pixel transparent gave speckled alpha masks where most pixels change a little
10+
(cross-fades), which could cost more than they saved. On six cross-faded radar-like animations at quality 85 the
11+
files are 2% larger to 13% smaller than before and 7-46% smaller than unoptimized ones, at 0.1 dB PSNR below
12+
unoptimized ones (before: 0.8-1 dB below); where nearly every pixel changes a little, they're 10x smaller.
613

714
- Lossless WebPs, and lossy ones with alpha (most optimized animations), could be undecodable since 3.7.0 (#82): a
815
prefix group whose code had a single symbol of 256 or more (a length code) took a bit per symbol instead of none.

‎src/animation.ts‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,8 @@ export interface AnimationEncodeOptions {
5959
export interface WebPAnimationEncodeOptions extends AnimationEncodeOptions {
6060
/**
6161
* Lossy frames (0-100, like libwebp's quality: 75 is its default), much smaller than lossless ones. Without it the
62-
* frames are lossless
62+
* frames are lossless. Optimized lossy frames, like libwebp's, leave out 8x8 blocks that only changed a little (by
63+
* up to 3 per channel at quality 85, 1 at 100)
6364
*/
6465
quality?: number;
6566
/** How much effort lossy frames take (0-6), see {@link WebPEncodeOptions.effort}. Default: `2` */
@@ -296,7 +297,8 @@ export class Animation extends Array<Frame> {
296297
if (!Number.isInteger(effort) || effort < 0 || effort > 6) throw new RangeError(`effort must be an integer 0-6, not ${effort}`);
297298
}
298299

299-
const frames = optimize ? this.__optimized__({ replace: true, even: true, alpha: 'frame' }) : this.__webpFrames__();
300+
const tolerance = quality === undefined ? undefined : webp.lossyTolerance(quality);
301+
const frames = optimize ? this.__optimized__({ replace: true, even: true, alpha: 'frame', tolerance }) : this.__webpFrames__();
300302
return webp.encodeAnimation(frames, this.width, this.height, toPlays(this.loopCount), exifData(resolveMetadata(metadata)), quality, effort);
301303
}
302304

‎src/codecs/webp.ts‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -379,6 +379,16 @@ export interface WebPEncodeFrame {
379379
pixels: Uint8Array;
380380
}
381381

382+
/**
383+
* How much a color channel of a lossy animation's pixel may change for it to count as unchanged (libwebp's
384+
* QualityToMaxDiff: 1 at quality 100, 3 at 85, 31 at 0)
385+
* @param quality 0-100
386+
*/
387+
export function lossyTolerance(quality: number): number {
388+
const value = Math.sqrt(quality / 100);
389+
return Math.max(1, Math.round(31 * (1 - value) + value));
390+
}
391+
382392
/**
383393
* Encodes frames into an animated WebP, lossless or (with a quality) lossy. Frame offsets have to be even (the format
384394
* stores them halved)

‎src/util/optimize.ts‎

Lines changed: 97 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,12 @@ export interface OptimizeOptions {
5858
* transparent, others the background color)
5959
*/
6060
cover?: boolean;
61+
/**
62+
* For lossy frames (WebP): how much a color channel may change (at full alpha) for a pixel to count as unchanged,
63+
* like libwebp's lossy animations. Unchanged pixels then only become transparent in whole 8x8 blocks (the alpha of
64+
* scattered unchanged pixels costs more than leaving them as they are). Needs `replace`
65+
*/
66+
tolerance?: number;
6167
}
6268

6369
// what transparent pixels make an opaque frame cost: an alpha channel, 4 channels instead of 3 (or a palette entry
@@ -69,14 +75,14 @@ const ALPHA_COST = 4 / 3;
6975
* @param width The canvas width
7076
* @param height The canvas height
7177
*/
72-
export function optimize(frames: InputFrame[], width: number, height: number, { replace, even, alpha, palette, cover }: OptimizeOptions): OptimizedFrame[] {
78+
export function optimize(frames: InputFrame[], width: number, height: number, { replace, even, alpha, palette, cover, tolerance }: OptimizeOptions): OptimizedFrame[] {
7379
const size = width * height * 4;
7480
// the input animation's canvas, and what the optimized animation shows after its last frame (and its disposal)
7581
const canvas = new Uint8Array(size), canvas32 = new Int32Array(canvas.buffer);
7682
const shown = new Uint8Array(size), shown32 = new Int32Array(shown.buffer);
7783
const output: OptimizedFrame[] = [];
7884

79-
const state: State = { canvas32, shown32, width, replace };
85+
const state: State = { canvas32, shown32, width, replace, tolerance };
8086

8187
// for a shared pixel format: whether the animation is opaque when all frames are stored dense, and the changes
8288
// from pixel to pixel all dense and as chosen
@@ -94,7 +100,7 @@ export function optimize(frames: InputFrame[], width: number, height: number, {
94100
return;
95101
}
96102
if (even) changed = evenArea(changed);
97-
let area = sparse(state, changed);
103+
let area = tolerance === undefined ? sparse(state, changed) : sparseBlocks(state, changed);
98104
let blend = true;
99105
if (area.unblendable && replace) blend = false;
100106
else if (area.unblendable) {
@@ -115,25 +121,31 @@ export function optimize(frames: InputFrame[], width: number, height: number, {
115121
// compresses better when the unchanged pixels are scattered (e.g. in videos): choose the one with fewer
116122
// changes from pixel to pixel, which estimates the compressed size well and costs a lot less than compressing
117123
// both. GIFs blend the dense area too (its unchanged pixels are the same or transparent in both), the others
118-
// replace the area with it
124+
// replace the area with it. Lossy frames are sparse whenever a block is unchanged (like libwebp's)
119125
const { x, y, width: w, height: h } = changed;
120126
const pixels = area.pixels;
121127
if (!output.length && (w !== width || h !== height)) opaque = false;
122128
if (!area.denseOpaque) opaque = false;
123129
const sparseCost = alpha === 'frame' && area.denseOpaque ? area.sparseChanges * ALPHA_COST : area.sparseChanges;
124-
const dense = !blend || area.denseChanges < sparseCost
130+
const dense = !blend || (tolerance !== undefined ? !area.unchangedBlocks : area.denseChanges < sparseCost
125131
|| (palette !== undefined && colors(new Int32Array(pixels.buffer), w, { x: 0, y: 0, width: w, height: h }, palette + 1) > palette
126-
&& colors(canvas32, width, changed, palette + 1) <= palette);
132+
&& colors(canvas32, width, changed, palette + 1) <= palette));
127133
denseTotal += area.denseChanges;
128134
chosenTotal += dense ? area.denseChanges : area.sparseChanges;
129135
if (dense) {
130136
for (let row = 0; row < h; row++) pixels.set(canvas.subarray(((y + row) * width + x) * 4, ((y + row) * width + x + w) * 4), row * w * 4);
131137
if (replace) blend = false;
132138
}
133-
for (let row = 0; row < h; row++) {
134-
const start = ((y + row) * width + x) * 4;
135-
shown.set(canvas.subarray(start, start + w * 4), start);
136-
}
139+
// (lossy frames keep showing the pixels of unchanged blocks as they were, with tolerance they can differ)
140+
if (!dense && tolerance !== undefined) {
141+
const pixels32 = new Int32Array(pixels.buffer);
142+
for (let row = 0, k = 0; row < h; row++)
143+
for (let p = (y + row) * width + x, end = p + w; p < end; p++, k++) if (pixels32[k] & ALPHA) shown32[p] = pixels32[k];
144+
} else
145+
for (let row = 0; row < h; row++) {
146+
const start = ((y + row) * width + x) * 4;
147+
shown.set(canvas.subarray(start, start + w * 4), start);
148+
}
137149
output.push({ x, y, width: w, height: h, duration, dispose: false, blend, pixels });
138150
};
139151

@@ -221,6 +233,18 @@ interface State {
221233
shown32: Int32Array;
222234
width: number;
223235
replace: boolean;
236+
tolerance?: number;
237+
}
238+
239+
/**
240+
* Whether two pixels (their offsets in RGBA bytes) look the same within a tolerance, like libwebp's PixelsAreSimilar:
241+
* the same alpha, and each color channel differs by at most the tolerance at full alpha
242+
*/
243+
function similar(a: Uint8Array, i: number, b: Uint8Array, j: number, tolerance: number): boolean {
244+
const alpha = a[i + 3];
245+
if (alpha !== b[j + 3]) return false;
246+
const limit = tolerance * 255;
247+
return Math.abs(a[i] - b[j]) * alpha <= limit && Math.abs(a[i + 1] - b[j + 1]) * alpha <= limit && Math.abs(a[i + 2] - b[j + 2]) * alpha <= limit;
224248
}
225249

226250
// the area grown to even offsets (WebP stores them halved)
@@ -236,6 +260,65 @@ interface SparseArea {
236260
unblendable: Rect | null;
237261
/** whether the area as it is has no transparent pixels */
238262
denseOpaque: boolean;
263+
/** lossy frames: how many blocks are unchanged (and transparent in the pixels) */
264+
unchangedBlocks?: number;
265+
}
266+
267+
// the size of the blocks lossy frames make transparent (libwebp's)
268+
const BLOCK = 8;
269+
270+
/**
271+
* For lossy frames: the area with the 8x8 blocks (from its top left) that didn't change, within the tolerance,
272+
* transparent (their color is the average of what's shown, for the encoder's prediction). The other blocks are as they
273+
* are, they can't be blended where they aren't opaque, unless transparent in both
274+
*/
275+
function sparseBlocks({ canvas32, shown32, width, tolerance = 0 }: State, { x, y, width: w, height: h }: Rect): SparseArea {
276+
const canvas = new Uint8Array(canvas32.buffer), shown = new Uint8Array(shown32.buffer);
277+
const pixels = new Uint8Array(w * h * 4), pixels32 = new Int32Array(pixels.buffer);
278+
let left = w, right = -1, top = h, bottom = -1, translucent = 0, unchangedBlocks = 0;
279+
for (let by = 0; by < h; by += BLOCK)
280+
for (let bx = 0; bx < w; bx += BLOCK) {
281+
const bw = Math.min(BLOCK, w - bx), bh = Math.min(BLOCK, h - by);
282+
let unchanged = true;
283+
for (let row = by; row < by + bh && unchanged; row++)
284+
for (let p = (y + row) * width + x + bx, end = p + bw; p < end; p++) {
285+
const value = canvas32[p], before = shown32[p];
286+
if (value !== before && ((value | before) & ALPHA) !== 0 && !similar(canvas, p * 4, shown, p * 4, tolerance)) {
287+
unchanged = false;
288+
break;
289+
}
290+
}
291+
292+
if (unchanged) {
293+
unchangedBlocks++;
294+
let r = 0, g = 0, b = 0;
295+
for (let row = by; row < by + bh; row++)
296+
for (let p = ((y + row) * width + x + bx) * 4, end = p + bw * 4; p < end; p += 4) {
297+
r += shown[p]; g += shown[p + 1]; b += shown[p + 2];
298+
}
299+
const n = bw * bh;
300+
r = r / n | 0; g = g / n | 0; b = b / n | 0;
301+
for (let row = by; row < by + bh; row++)
302+
for (let k = ((row * w) + bx) * 4, end = k + bw * 4; k < end; k += 4) {
303+
pixels[k] = r; pixels[k + 1] = g; pixels[k + 2] = b;
304+
}
305+
}
306+
for (let row = by; row < by + bh; row++)
307+
for (let i = bx, k = row * w + bx, p = (y + row) * width + x + bx; i < bx + bw; i++, k++, p++) {
308+
const value = canvas32[p];
309+
translucent |= ~value & ALPHA;
310+
if (unchanged) continue;
311+
pixels32[k] = value;
312+
if ((value & ALPHA) !== ALPHA && ((value | shown32[p]) & ALPHA) !== 0) {
313+
if (i < left) left = i;
314+
if (i > right) right = i;
315+
if (row < top) top = row;
316+
if (row > bottom) bottom = row;
317+
}
318+
}
319+
}
320+
const unblendable: Rect | null = right < 0 ? null : { x: x + left, y: y + top, width: right - left + 1, height: bottom - top + 1 };
321+
return { pixels, sparseChanges: 0, denseChanges: 0, unblendable, denseOpaque: translucent === 0, unchangedBlocks };
239322
}
240323

241324
/**
@@ -279,14 +362,15 @@ function sparse({ canvas32, shown32, width, replace }: State, { x, y, width: w,
279362
}
280363

281364
/**
282-
* The bounding box of the changed pixels within an area. Searches from the edges inwards, so it stops early when much
365+
* The bounding box of the changed pixels (within the tolerance) within an area. Searches from the edges inwards, so it stops early when much
283366
* changed
284367
*/
285-
function bounds({ canvas32, shown32, width }: State, area: Rect): Rect | null {
368+
function bounds({ canvas32, shown32, width, tolerance }: State, area: Rect): Rect | null {
286369
const x0 = area.x, x1 = area.x + area.width, y0 = area.y, y1 = area.y + area.height;
370+
const canvas = new Uint8Array(canvas32.buffer), shown = new Uint8Array(shown32.buffer);
287371
const test = (p: number): boolean => {
288372
const c = canvas32[p], s = shown32[p];
289-
return c !== s && ((c | s) & ALPHA) !== 0;
373+
return c !== s && ((c | s) & ALPHA) !== 0 && (tolerance === undefined || !similar(canvas, p * 4, shown, p * 4, tolerance));
290374
};
291375
const rowHas = (y: number, from: number, to: number): number => {
292376
for (let x = from, p = y * width + from; x < to; x++, p++) if (test(p)) return x;

‎tests/animation-optimize.js‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,4 +162,48 @@ const disc = (size, color) => new Image(size, size).fill((x, y) => {
162162
panic(`screen recording: ${format} is ${size} bytes, the changes aren't stored alone`);
163163
}
164164
}
165+
166+
// lossy WebP: pixels that change by up to the tolerance count as unchanged, and unchanged pixels only become
167+
// transparent in whole 8x8 blocks. A translucent layer over a textured map changes its color a little each frame
168+
// (most pixels change by 1-2, scattered ones not at all), a box moves and one in the far corner changes its color
169+
{
170+
let seed = 5;
171+
const random = () => (seed = Math.imul(seed, 1103515245) + 12345 >>> 0) / 2 ** 32;
172+
const map = new Image(100, 60).fill((x, y) => {
173+
const g = 150 + 40 * Math.sin(x / 9) * Math.cos(y / 13) + (random() - 0.5) * 30;
174+
return Image.rgbaToColor(g | 0, g * 0.9 | 0, g * 0.8 | 0, 255);
175+
});
176+
const animation = new Animation(Array.from({length: 8}, (_, i) => {
177+
const layer = new Image(100, 60).fill(x => x < 80 ? Image.rgbaToColor(i % 4, 180 + i * 7 % 5, 250 + i * 3 % 4, 150) : 0);
178+
return Frame.from(map.clone().composite(layer).drawBox(5 + i * 10, 20, 12, 12, 0xff0000ff)
179+
.drawBox(90, 50, 6, 6, Image.rgbaToColor(0, i * 30, 0, 255)), 50);
180+
}));
181+
182+
const tolerance = 3;
183+
const frames = animation.__optimized__({replace: true, even: true, alpha: 'frame', tolerance});
184+
if (frames.length !== animation.length) panic(`lossy WebP: ${frames.length} frames instead of ${animation.length}, the box moves in each`);
185+
for (const [i, f] of frames.entries()) {
186+
if (i && !f.blend) panic(`lossy WebP: frame ${i} isn't sparse`);
187+
// the transparent pixels of each block (from the frame's top left): all or none
188+
for (let by = 0; by < f.height; by += 8)
189+
for (let bx = 0; bx < f.width; bx += 8) {
190+
let transparent = 0, count = 0;
191+
for (let y = by; y < Math.min(by + 8, f.height); y++)
192+
for (let x = bx; x < Math.min(bx + 8, f.width); x++, count++) if (!f.pixels[(y * f.width + x) * 4 + 3]) transparent++;
193+
if (transparent && transparent !== count) panic(`lossy WebP: frame ${i} has a partly transparent block at ${bx},${by}`);
194+
}
195+
}
196+
// what the frames show is within the tolerance of the animation
197+
const png = await import('../dist/codecs/png.js');
198+
const shown = png.compositeAnimation({width: 100, height: 60, frames: frames.map(f => ({...f, dispose: f.dispose ? 1 : 0}))});
199+
shown.forEach(({pixels}, i) => {
200+
const expected = animation[i].bitmap;
201+
for (let k = 0; k < pixels.length; k++)
202+
if (Math.abs(pixels[k] - expected[k]) > tolerance) panic(`lossy WebP: frame ${i} differs by ${Math.abs(pixels[k] - expected[k])} at pixel ${k >> 2}`);
203+
});
204+
// and it's smaller than without optimizing (storing the scattered unchanged pixels as transparent made it larger)
205+
const optimized = await animation.encodeWebP({quality: 85}), plain = await animation.encodeWebP({quality: 85, optimize: false});
206+
if (optimized.length >= plain.length / 2) panic(`lossy WebP: ${optimized.length} bytes optimized, ${plain.length} without`);
207+
if ((await Animation.decode(optimized)).length !== animation.length) panic('lossy WebP: the optimized animation doesn\'t decode');
208+
}
165209
})();

0 commit comments

Comments
 (0)