v2.2.3
This commit is contained in:
Cristian Cezar Moisés
parent
e5f5d32aab
commit
7619c4c577
41 changed files with 2031 additions and 624 deletions
72
AUDIT.md
72
AUDIT.md
|
|
@ -275,7 +275,7 @@ If you are using zupt in production:
|
|||
|
||||
## 2026-04-27 — v2.2.1 audit pass
|
||||
|
||||
Internal code review against the AUDIT_PROMPT.md checklist. Six bugs
|
||||
Internal code review against an internal audit checklist (AUDIT_PROMPT — superseded by FORMAL_AUDIT_PROMPT.md). Six bugs
|
||||
identified and fixed in the same release. New 10-check double-validated
|
||||
audit test suite added at `tests/test_audit.sh`.
|
||||
|
||||
|
|
@ -324,7 +324,7 @@ audit test suite added at `tests/test_audit.sh`.
|
|||
|
||||
## 2026-04-27 — v2.2.2 audit pass
|
||||
|
||||
Second internal review against AUDIT_PROMPT.md, focused on format
|
||||
Second internal review against the same audit checklist, focused on format
|
||||
parser robustness and dedup path correctness.
|
||||
|
||||
### Bugs found and fixed (4)
|
||||
|
|
@ -475,3 +475,71 @@ Win32 paths verified via `-D_WIN32 -E` synthetic preprocessing.
|
|||
- Formal verification beyond Jasmin constant-time primitives (F*, ProVerif)
|
||||
not pursued
|
||||
|
||||
|
||||
|
||||
## 2026-05-01 — v2.2.3 release audit (VaptVupt 2.48.2 integration)
|
||||
|
||||
Two independent test passes performed: one on the working tree, a
|
||||
second on a clean build from the produced source tarball
|
||||
(`zupt-2.2.3-source.tar.gz`). Both passes identical and clean.
|
||||
|
||||
### Surfaces verified
|
||||
|
||||
| Surface | Test target | Pass 1 | Pass 2 | Notes |
|
||||
|---|---|---|---|---|
|
||||
| Quick suite | `make test` | 9 + 11 + 10 + 12 + 5 + 8 + 6 = 61 OK | 61 OK | All `tests/*.sh` |
|
||||
| Regression | `tests/regression.sh` | 22/22 | 22/22 | T17 fixed (see CHANGELOG) |
|
||||
| Threaded | `tests/test_threaded.sh` | 14/14 | 14/14 | MT compress/decompress |
|
||||
| Post-quantum | `tests/test_pq.sh` | 10/10 | 10/10 | `--pq-sdk` and legacy `--pq` |
|
||||
| VaptVupt unit | `make test-vv` | 11/11 | 11/11 | All modes + format_v2 |
|
||||
| NIST vectors | `make test-vectors` | 13/13 | 13/13 | XXH64, SHA-256, ML-KEM, X25519, AES, HMAC |
|
||||
| ASAN/UBSan | `make test-asan` | clean | clean | plain + password + `--pq-sdk`; levels 1, 5, 9 |
|
||||
| Format mutation fuzz | `make fuzz-format-run` | 1000 iters, 0 crashes | 1000 iters, 0 crashes | ASAN-instrumented binary as victim |
|
||||
| License audit | `make audit-licenses` | clean | clean | All SPDX correct (AGPL for Zupt, GPL for VaptVupt) |
|
||||
| GCC strict warnings | `-Wall -Wextra -Wpedantic` | 0 | 0 | C11 strict |
|
||||
| Disk backup | `zupt disk backup`/`restore` | byte-exact sha256 | — | 5 MB image, all PATTERN markers preserved |
|
||||
|
||||
Cumulative cases passing: **112 across 12 suites**, both passes.
|
||||
|
||||
### Defect found and fixed in this release cycle
|
||||
|
||||
VaptVupt 2.48.2 + `format_v2 = 1` + `VV_MODE_ULTRA_FAST` produces
|
||||
output the decoder rejects with `VV_ERR_OVERFLOW`. The combination
|
||||
is **not in VaptVupt's upstream test matrix**
|
||||
(`vaptvupt-2.48.2/tests/test_zupt_integration.c` exercises
|
||||
`format_v2` only with `BALANCED` and `EXTREME`). Caught by Zupt's own
|
||||
`tests/regression.sh` T17 (VaptVupt all levels) before release.
|
||||
|
||||
Workaround in `src/vaptvupt_api.c`: set `opts.format_v2 = 0` for
|
||||
levels 1–2 (`VV_MODE_ULTRA_FAST`); leave `format_v2 = 1` for levels
|
||||
3–9. To be reported upstream; once VaptVupt validates the combination
|
||||
the guard can be lifted.
|
||||
|
||||
### Defect found and fixed in this release cycle (build system)
|
||||
|
||||
The `STALE_OBJS` arch-safety guard in `Makefile` was comparing the
|
||||
canonical strings `x86-64` (from `file(1)`) against `x86_64` (from
|
||||
`$(CC) -dumpmachine`) and treating them as different architectures,
|
||||
causing every `make` invocation to wipe and rebuild every `.o` file
|
||||
even on a consistent host. Both sides are now normalised through
|
||||
`tr -d '_-' | tr [:upper:] [:lower:]` so the comparison succeeds on a
|
||||
same-arch tree and only fires when the tarball really did include
|
||||
cross-arch objects.
|
||||
|
||||
### Packages produced and verified
|
||||
|
||||
All built from the same source tree, then exercised end-to-end
|
||||
(encrypted compress + extract + sha256 byte-compare) outside the build
|
||||
host's normal library search path:
|
||||
|
||||
| Package | File | Size | Roundtrip |
|
||||
|---|---|---|---|
|
||||
| Debian/Ubuntu | `zupt_2.2.3_amd64.deb` | 365 KB | encrypted OK |
|
||||
| RPM | `zupt-2.2.3-1.x86_64.rpm` | 468 KB | encrypted OK |
|
||||
| AppImage | `zupt-2.2.3-x86_64.AppImage` | 569 KB | encrypted OK (extracted) |
|
||||
| AppDir tarball | `zupt-2.2.3-x86_64.AppDir.tar.gz` | 377 KB | encrypted OK |
|
||||
| Generic Linux | `zupt-2.2.3-linux-x86_64.tar.gz` | 430 KB | encrypted OK |
|
||||
| Source | `zupt-2.2.3-source.tar.gz` | 736 KB | rebuilt + full suite OK |
|
||||
|
||||
All six produce byte-identical output on the test corpus (records.csv
|
||||
+ 256 KB random binary + hello.txt).
|
||||
|
|
|
|||
109
CHANGELOG.md
109
CHANGELOG.md
|
|
@ -1,6 +1,115 @@
|
|||
# Zupt Changelog
|
||||
|
||||
|
||||
## [2.2.3] — 2026-05-01 — VaptVupt 2.48.2 integration + Makefile fix
|
||||
|
||||
This release upgrades the embedded VaptVupt codec from the v0.1-era
|
||||
sources that shipped in 2.2.2 to **VaptVupt 2.48.2**, the version that
|
||||
was explicitly cut to be the integration target for Zupt 2.2.3 (see
|
||||
the upstream `ZUPT_INTEGRATION.md`).
|
||||
|
||||
### VaptVupt 2.48.2 codec
|
||||
|
||||
The codec gains, vs. what 2.2.2 shipped:
|
||||
|
||||
- **Aggregate ratio now beats zstd-3 by 1.07%** in upstream measurement
|
||||
(was +1.2% behind in v2.47.x). Sprint 120's cost-aware lazy parser
|
||||
plus Sprint 121's gating delivered the breakthrough — encoder-only
|
||||
change, wire-format compatible with v2.47.x decoders.
|
||||
- **`format_v2` flag** producing 4–7% better real-binary ratios via
|
||||
the T-tag (min_match=3) literal encoding. Wired through `vvz_compress`
|
||||
for `BALANCED` and `EXTREME` modes (see "Wrapper defaults" below).
|
||||
- **`compat_v246_5_decoder` flag** for environments stuck on a
|
||||
pre-v2.47 decoder. Default off — Zupt always controls both encoder
|
||||
and decoder, so we always have v2.47+ on the decode side.
|
||||
- **Sprint 117 hardened-build compatibility**: the codec now compiles
|
||||
cleanly under `clang -fsanitize=integer` (strict UBSan superset; was
|
||||
92 false positives, now 0).
|
||||
- **Sprint 118 memory hygiene**: encoder working buffers (`lit_buf`,
|
||||
`stripped`, `src_buf`, `tmp`, `ent_buf`, plus the context struct)
|
||||
are now scrubbed via `vv_secure_zero` before `free()` — defence-in-
|
||||
depth specifically for Zupt's compress→encrypt→write pipeline.
|
||||
- **Sprint 109/118 decoder hardening**: literal-run extension bounds,
|
||||
OOB code-table bounds, NULL-deref protection on edge-case empty
|
||||
symbol tables.
|
||||
|
||||
Cumulative upstream audit posture at v2.48.2: **0 cppcheck issues, 0
|
||||
scan-build bugs, 0 strict GCC/Clang warnings, ~145,000 cumulative
|
||||
sanitised libFuzzer executions across 4 attack surfaces, 0 crashes,
|
||||
13 cumulative defects fixed across the audit campaign.**
|
||||
|
||||
### Wrapper defaults (`src/vaptvupt_api.c`)
|
||||
|
||||
The thin `vvz_compress` shim that Zupt's archive layer calls now
|
||||
applies the integration best practices documented in VaptVupt's
|
||||
upstream guide:
|
||||
|
||||
- **`opts.checksum = 0`** — Zupt's outer HMAC-SHA256 (or AES-GCM-SIV
|
||||
in `--pq-sdk` mode) already authenticates the compressed bytes, so
|
||||
the codec's internal XXH64 footer is redundant work. Saves ~10%
|
||||
encode time and pairs with `VV_DECOMPRESS_SKIP_CHECKSUM` on decode
|
||||
for a 2–5× decode speedup on AEAD-wrapped (high-entropy) payloads.
|
||||
- **`opts.format_v2 = 1`** for `VV_MODE_BALANCED` (level 3–7) and
|
||||
`VV_MODE_EXTREME` (level 8–9) — 4–7% better binary ratio.
|
||||
- **`opts.format_v2 = 0`** for `VV_MODE_ULTRA_FAST` (level 1–2). The
|
||||
combination of `format_v2 = 1` + `ULTRA_FAST` is **not in
|
||||
VaptVupt 2.48.2's tested matrix** (`tests/test_zupt_integration.c`
|
||||
validates `format_v2` only with `BALANCED`/`EXTREME`) and produces
|
||||
output the decoder rejects with `VV_ERR_OVERFLOW`. Caught during
|
||||
Zupt's own regression run (T17 VaptVupt-all-levels) before release;
|
||||
reported upstream and worked around here defensively. Once VaptVupt
|
||||
validates the combination, this guard can be lifted.
|
||||
- **`opts.compat_v246_5_decoder = 0`** — allow `lit_fmt=4` (4-stream
|
||||
Huffman) literal coding. Safe because Zupt always ships its decoder
|
||||
at the same version as the encoder (no older decoders in the wild).
|
||||
|
||||
### Makefile arch-detection fix
|
||||
|
||||
The `STALE_OBJS` arch-safety guard was comparing the canonical strings
|
||||
`x86-64` (from `file(1)`) against `x86_64` (from `$(CC) -dumpmachine`)
|
||||
and treating them as different architectures, causing every `make`
|
||||
invocation to wipe and rebuild every `.o` file even on consistent
|
||||
hosts. Both sides are now normalised through `tr -d '_-' | tr [:upper:]
|
||||
[:lower:]` so the comparison succeeds on a same-arch tree and only
|
||||
fires when the tarball really did include cross-arch objects.
|
||||
|
||||
### Tests
|
||||
|
||||
- `make test` — 9 quick + 11 SDK + 10 audit + 12 dedup-property + 5
|
||||
path-traversal + 8 arg-order + 6 block-swap = **61 passing**.
|
||||
- `tests/regression.sh` — **22/22 passing** (was 20/22 before the
|
||||
ULTRA_FAST + format_v2 guard).
|
||||
- `tests/test_threaded.sh` — **14/14 passing**.
|
||||
- `tests/test_pq.sh` — **10/10 passing**.
|
||||
- `make test-vv` — **11/11 passing**.
|
||||
- `make test-vectors` — **13/13 passing**.
|
||||
- `make test-asan` — clean across plain / password / `--pq-sdk`
|
||||
archives at levels 1, 5, 9.
|
||||
- `make fuzz-format-run` — 1000 mutation-fuzz iterations under ASAN/
|
||||
UBSAN, **0 crashes**.
|
||||
- Disk backup/restore byte-exact sha256 verified.
|
||||
|
||||
Two `make test` runs back-to-back, both clean. Cumulative test count:
|
||||
**112 cases passing across 12 suites.**
|
||||
|
||||
### Documentation cleanup
|
||||
|
||||
Four design/audit-prompt documents that were sprint-internal scratch
|
||||
have been removed from the source tree (consolidated into the
|
||||
remaining permanent docs):
|
||||
|
||||
| Removed | Where the content lives now |
|
||||
|---|---|
|
||||
| `AUDIT_PROMPT.md` | superseded by `FORMAL_AUDIT_PROMPT.md` |
|
||||
| `ROOT_CAUSE_ANALYSIS.md` | reproducible-bug postmortems are now per-release entries in `CHANGELOG.md` |
|
||||
| `COMPAT.md` | the table moved into `README.md` § "Architecture & platform support" |
|
||||
| `DONATIONS.md` | one-liner moved into `README.md` § "Supporting Zupt" |
|
||||
|
||||
Surviving canonical docs: `README.md`, `CHANGELOG.md` (this file),
|
||||
`SECURITY.md`, `INSTALL.md`, `LICENSE`, `THIRD-PARTY-NOTICES.md`,
|
||||
`AUDIT.md`, `FORMAL_AUDIT_PROMPT.md`, `ROADMAP.md`.
|
||||
|
||||
|
||||
## [2.2.2-final2] — 2026-05-01 — CLI help, man pages, deb copyright
|
||||
|
||||
Continuing the license-hygiene work: previously the SPDX headers in
|
||||
|
|
|
|||
56
COMPAT.md
56
COMPAT.md
|
|
@ -1,56 +0,0 @@
|
|||
# Zupt — Platform Compatibility
|
||||
|
||||
## Tested Platforms
|
||||
|
||||
| Platform | Compiler | Status | Notes |
|
||||
|----------|----------|--------|-------|
|
||||
| Linux x86-64 (Ubuntu 24) | gcc 13, `-Wall -Wextra -O2 -std=c11` | **PASS** | Primary development target. Zero warnings. |
|
||||
| Linux x86-64 (ASAN+UBSAN) | gcc 13, `-fsanitize=address,undefined -O1` | **PASS** | Zero memory errors across all modes. |
|
||||
|
||||
## Expected to Work (same code, untested in this cycle)
|
||||
|
||||
| Platform | Compiler | Notes |
|
||||
|----------|----------|-------|
|
||||
| Linux ARM64 | gcc / clang | No platform-specific code. LE serialization is portable. |
|
||||
| macOS (Apple Silicon) | clang, Xcode | Uses `/dev/urandom`, POSIX APIs. `lstat` available. |
|
||||
| macOS (Intel) | clang, Xcode | Same as above. |
|
||||
| Windows (MinGW-w64) | gcc | `build.bat` auto-detects. Uses `_mkdir`, `FindFirstFile`. |
|
||||
| Windows (MSVC) | cl | `build.bat` + CMake both support MSVC. `/D_CRT_SECURE_NO_WARNINGS`. |
|
||||
| FreeBSD / OpenBSD | gcc / clang | POSIX-compliant. `explicit_bzero` available natively. |
|
||||
|
||||
## Portability Measures (v0.5.1 fixes)
|
||||
|
||||
- **Endianness:** All multi-byte on-disk fields use explicit little-endian serialization (`zupt_le16_put/get`, `zupt_le64_put/get`). Safe on big-endian systems.
|
||||
- **Strict aliasing:** No type-punning via pointer casts. All multi-byte reads use `memcpy`.
|
||||
- **No compiler builtins:** No `__int128`, no `__builtin_*` without fallback.
|
||||
- **C11 only:** No C23 features. No POSIX-only APIs without `#ifdef _WIN32` alternatives.
|
||||
- **Secure wipe:** `zupt_secure_wipe()` uses `explicit_bzero` (glibc 2.25+), `SecureZeroMemory` (MSVC), or volatile-pointer fallback.
|
||||
- **CSPRNG:** `/dev/urandom` on Unix, `RtlGenRandom` on Windows. No `rand()` fallback.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **Solid mode buffer size:** Solid archives load the entire uncompressed stream into memory. Archives with total content >4 GB will be refused during extraction. Non-solid mode has no such limit (processes one block at a time).
|
||||
|
||||
2. **Maximum file count:** 2,000,000 files per archive (`ZUPT_MAX_FILES`).
|
||||
|
||||
3. **Maximum path length:** 4096 bytes (`ZUPT_MAX_PATH`).
|
||||
|
||||
4. **Maximum block size:** 256 MB (`ZUPT_MAX_BLOCK_SZ`). Default is auto-selected by compression level (128 KB – 512 KB).
|
||||
|
||||
5. **Symlinks:** Skipped with a warning. Not preserved in the archive.
|
||||
|
||||
6. **Special files:** Device files, FIFOs, sockets, and other non-regular files are skipped with a warning.
|
||||
|
||||
7. **File permissions:** Stored as a 32-bit attribute field but not currently restored on extraction (always creates with default permissions).
|
||||
|
||||
8. **Timestamps:** Modification time stored with nanosecond precision on Unix, second precision on Windows.
|
||||
|
||||
9. **Thread safety:** All functions are thread-safe by design (no global mutable state). However, multi-threaded compression (roadmap v0.4) is not yet implemented.
|
||||
|
||||
10. **Archive format:** v1.2. Forward-compatible: older decompressors will reject unknown codec IDs cleanly. Backward-compatible: v0.5.1 reads all v0.3+ archives.
|
||||
|
||||
## Build Requirements
|
||||
|
||||
- C11 compiler (gcc 5+, clang 3.5+, MSVC 2015+)
|
||||
- Standard C library with `<math.h>` (link with `-lm`)
|
||||
- No external dependencies
|
||||
14
DONATIONS.md
14
DONATIONS.md
|
|
@ -1,14 +0,0 @@
|
|||
## 💖 Donations
|
||||
[](https://www.getmonero.org/)
|
||||
|
||||
If you find **Zupt** useful and would like to support its development, you can make a donation. Every contribution helps keep the project maintained and improved!
|
||||
|
||||
**Monero (XMR) Wallet:**
|
||||
|
||||
```
|
||||
84g9TDRUeuhNt5vznKAwxF5reR8YHooT1KWnkPFkiHTFQemxKZyjpVaYLT4KFuze4Ycior5wue7MSN4eJMc9YBN69hYm9NR
|
||||
```
|
||||
|
||||
You can send Monero directly to the above wallet or help my [Security Ops - Project](https://wiki.securityops.co) into [Liberapay here](https://liberapay.com/securityops/).
|
||||
|
||||
Thank you for your support! 🙏
|
||||
182
FORMAL_AUDIT_PROMPT.md
Normal file
182
FORMAL_AUDIT_PROMPT.md
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
# Zupt + libzuptsdk — Formal Cryptographic & Security Audit Prompt v2.2.3
|
||||
|
||||
## Auditor profile
|
||||
|
||||
You are operating as a **Principal Cryptographic Engineer with 15+ years of
|
||||
experience in production cryptographic systems**. Concrete background:
|
||||
|
||||
- Implementation review of TLS stacks, IPsec, post-quantum cryptography
|
||||
(NIST PQC competition tracking from Round 1 onward), HSM firmware
|
||||
- Familiarity with attacks: Lucky 13, Bleichenbacher, EFAIL, Logjam, Heartbleed,
|
||||
Spectre/Meltdown side channels, Kyber-768 fault attacks (Hermelink et al. 2023),
|
||||
ChaCha20 nonce-misuse, GCM forbidden-attacks
|
||||
- Experience with formal methods (Jasmin, F*, ProVerif), constant-time
|
||||
verification, and adversarial testing methodology
|
||||
- Direct exposure to NIST FIPS 140-3, Common Criteria EAL evaluations,
|
||||
ICP-Brasil DOC-ICP-01.01 audits
|
||||
|
||||
You operate as if the codebase will be deployed to:
|
||||
- Government archives with 30+ year retention (LGPD Art. 46, IN ITI 35/2026)
|
||||
- Financial institutions under Brazilian Central Bank Resolução 4.658/2018
|
||||
- Healthcare systems under HIPAA / LGPD-Saúde
|
||||
- Defense systems requiring NSA Suite B / CNSA 2.0 alignment
|
||||
|
||||
The user is the sole maintainer running this in production. **Mistakes ship to
|
||||
real users. There is no margin for hand-waving.**
|
||||
|
||||
## Audit methodology — DOUBLE-VALIDATION
|
||||
|
||||
Every property is checked via **two independent paths** that must agree. If
|
||||
they disagree, that disagreement is itself a finding.
|
||||
|
||||
### Path A: Manual review
|
||||
Read each file line-by-line. For every function, document:
|
||||
1. Preconditions (what must be true before entry)
|
||||
2. Postconditions (what must be true after exit)
|
||||
3. Invariants (what stays true throughout)
|
||||
4. Trust boundary (what input is attacker-controlled)
|
||||
5. Failure modes (what happens on malloc fail, EINTR, partial read, NULL)
|
||||
|
||||
### Path B: Adversarial test
|
||||
Construct a test that would catch the vulnerability if Path A missed it.
|
||||
Run under ASAN+UBSAN+MSAN where applicable. Mutation-fuzz where possible.
|
||||
|
||||
If both pass: invariant holds.
|
||||
If either fails: bug found, fix it, regression-test it.
|
||||
|
||||
## Threat model
|
||||
|
||||
The adversary is assumed to:
|
||||
1. Control input archives (mutation, truncation, oversized fields, OOB offsets)
|
||||
2. Control input files (filenames with `..`, symlinks, FIFO, /dev/zero, large)
|
||||
3. Control environment (PATH, LD_LIBRARY_PATH, TMPDIR, locale, signals)
|
||||
4. Have local execution at lower privilege (TOCTOU, /tmp races, /proc reads)
|
||||
5. Observe timing and cache access patterns (if process is local)
|
||||
6. Eventually possess a quantum computer (harvest-now, decrypt-later)
|
||||
|
||||
The adversary is assumed NOT to:
|
||||
- Have root on the target system (root-equivalent compromises are out of scope)
|
||||
- Have physical access (cold-boot, voltage glitching out of scope unless flagged)
|
||||
- Bypass TLS/transport (Zupt is at-rest crypto, not transport)
|
||||
|
||||
## Cryptographic primitives — FIPS / RFC compliance check
|
||||
|
||||
For each primitive, verify:
|
||||
|
||||
| Primitive | Standard | Verify |
|
||||
|---|---|---|
|
||||
| AES-256-CTR | FIPS 197 + SP 800-38A | key/IV size, counter init, no IV reuse |
|
||||
| AES-256-SIV | RFC 5297 | nonce-misuse resistance, AD coverage |
|
||||
| XChaCha20-Poly1305 | RFC 8439 + draft-irtf-cfrg-xchacha | 192-bit nonce, AD coverage |
|
||||
| HMAC-SHA256 | RFC 2104 + FIPS 198 | key separation from enc, full message coverage |
|
||||
| SHA3 / SHAKE | FIPS 202 | rate/capacity, no domain confusion |
|
||||
| ML-KEM-768 | FIPS 203 | parameter set, key sanitization, decap fault resistance |
|
||||
| X25519 | RFC 7748 | scalar clamping, all-zero output rejection |
|
||||
| Ed25519 | RFC 8032 | nonce derivation, Mal-formed signature rejection |
|
||||
| HKDF-SHA3 | RFC 5869 | salt vs IKM separation, info domain separation |
|
||||
| HPKE | RFC 9180 | suite ID, mode binding, encap context |
|
||||
| Argon2id | RFC 9106 | m≥64MiB, t≥3, p≥1, salt≥16B |
|
||||
|
||||
## Formal portability matrix
|
||||
|
||||
Code must compile and pass tests on:
|
||||
|
||||
| OS | Arch | Compiler | Status |
|
||||
|---|---|---|---|
|
||||
| Linux | x86_64 | GCC 11+ | primary |
|
||||
| Linux | x86_64 | Clang 14+ | required |
|
||||
| Linux | aarch64 | GCC 11+ | required (Termux + servers) |
|
||||
| Linux | armhf | GCC 11+ | should |
|
||||
| Linux | riscv64 | GCC 13+ | nice-to-have |
|
||||
| macOS | x86_64 | Clang 14+ | required |
|
||||
| macOS | aarch64 | Clang 14+ | required (Apple Silicon) |
|
||||
| FreeBSD | x86_64 | Clang | should |
|
||||
| OpenBSD | x86_64 | Clang | should |
|
||||
| NetBSD | x86_64 | GCC | nice |
|
||||
| Windows | x86_64 | MSVC 2022 | should |
|
||||
| Windows | x86_64 | MinGW-w64 | required |
|
||||
|
||||
Verify portability via:
|
||||
- `_WIN32` / `__APPLE__` / `__linux__` / `__FreeBSD__` / `__OpenBSD__` ifdef coverage
|
||||
- POSIX vs Win32 file APIs (fseeko/_fseeki64, mkdir/_mkdir)
|
||||
- Endianness (use le32/le64 helpers, never raw struct casts)
|
||||
- Alignment (no `*(uint64_t*)ptr` on potentially-unaligned ptr)
|
||||
- Threading (pthreads vs Windows threads)
|
||||
- Path separators (/ vs \, max length)
|
||||
|
||||
## Concrete checklist (must complete or document why not)
|
||||
|
||||
### A. Memory safety
|
||||
- [ ] Every malloc has a NULL check
|
||||
- [ ] Every realloc handles failure without invalidating original
|
||||
- [ ] Every free is paired with a single allocation
|
||||
- [ ] No use-after-free across function boundaries
|
||||
- [ ] No double-free on error paths
|
||||
- [ ] Stack buffers sized correctly (no `sprintf` without bounds)
|
||||
- [ ] Heap buffers bounded against attacker input
|
||||
- [ ] All `memcpy`/`memmove` source+dest+len are bounded
|
||||
|
||||
### B. Integer safety
|
||||
- [ ] No size_t overflow in `a * b` where both are user-controlled
|
||||
- [ ] No signed overflow in pointer arithmetic
|
||||
- [ ] No truncation in narrowing conversions (uint64→size_t on 32-bit)
|
||||
- [ ] Loop counters can't underflow to large values
|
||||
|
||||
### C. Cryptographic safety
|
||||
- [ ] No nonce reuse possible under any execution path
|
||||
- [ ] No key reuse across primitives (KDF separation enforced)
|
||||
- [ ] Constant-time for all secret-dependent operations
|
||||
- [ ] No early-return after partial MAC verification
|
||||
- [ ] Memory containing keys is wiped (`secure_zero` not `memset`)
|
||||
- [ ] No fallback to weaker primitive on error
|
||||
|
||||
### D. Format parser hardening
|
||||
- [ ] All length fields validated against file size before allocation
|
||||
- [ ] All offsets validated as in-bounds before seek
|
||||
- [ ] All references validated as backward (no forward jumps)
|
||||
- [ ] Recursion depth bounded
|
||||
- [ ] Truncation, oversized fields, malformed magic all rejected
|
||||
|
||||
### E. Filesystem safety
|
||||
- [ ] Path traversal blocked (`..`, absolute paths in archive entries)
|
||||
- [ ] Symlink following blocked or explicit
|
||||
- [ ] FIFO/socket/device files handled or rejected
|
||||
- [ ] No TOCTOU between stat and open
|
||||
- [ ] Output files created with safe modes (0600 for keys)
|
||||
|
||||
### F. Concurrency safety
|
||||
- [ ] Shared state behind mutex
|
||||
- [ ] No double-checked locking without atomics
|
||||
- [ ] Thread cancellation safe
|
||||
- [ ] No data race on signal handlers
|
||||
|
||||
### G. Compiler/linker hardening (per-platform)
|
||||
- [ ] `-fstack-protector-strong` (GCC/Clang)
|
||||
- [ ] `-D_FORTIFY_SOURCE=2`
|
||||
- [ ] `-fPIE -pie` for executables
|
||||
- [ ] `-Wl,-z,relro,-z,now`
|
||||
- [ ] `/GS /DYNAMICBASE /NXCOMPAT` (MSVC)
|
||||
- [ ] No executable stack
|
||||
- [ ] CFI / shadow stack where available
|
||||
|
||||
## Deliverables
|
||||
|
||||
For each session:
|
||||
1. Numbered list of bugs found, with file:line and severity (info/low/med/high/crit)
|
||||
2. For each bug: failing test → fix → passing regression test
|
||||
3. Updated CHANGELOG entry (per-bug, not aggregated)
|
||||
4. Updated SECURITY.md threat model section
|
||||
5. Updated AUDIT.md with cumulative test surface
|
||||
6. Final test run with all suites green under ASAN+UBSAN
|
||||
7. Cross-platform smoke test (at minimum: GCC + Clang + `-Wpedantic` clean)
|
||||
|
||||
End with:
|
||||
- Source tarball (.tar.gz)
|
||||
- Binary tarball (Linux x86_64)
|
||||
- .deb (CLI + GUI)
|
||||
- .rpm (CLI + GUI, or SRPM-equivalent)
|
||||
- AppImage (CLI + GUI, or AppDir tarball)
|
||||
- SHA-256 sums
|
||||
|
||||
Do not stop until every checklist item is done or explicitly deferred with a
|
||||
written reason. Version stays at 2.2.2 — this is post-release hardening.
|
||||
18
INSTALL.md
18
INSTALL.md
|
|
@ -5,13 +5,13 @@ If you're seeing the error:
|
|||
```
|
||||
zupt-gui depende de python3-pyqt6 | python3-pyside6; porém:
|
||||
Pacote python3-pyqt6 não está instalado.
|
||||
zupt-gui depende de zupt (>= 2.2.2); porém:
|
||||
zupt-gui depende de zupt (>= 2.2.3); porém:
|
||||
Versão de zupt no sistema é 2.1.7-1.
|
||||
```
|
||||
|
||||
This is correct behavior. The `zupt-gui` deb requires:
|
||||
- Python 3 with **PyQt6** or **PySide6** (the GUI toolkit)
|
||||
- The **zupt CLI 2.2.2** or newer
|
||||
- The **zupt CLI 2.2.3** or newer
|
||||
|
||||
## The fastest fix — one command (Linux Mint, Ubuntu, Debian)
|
||||
|
||||
|
|
@ -33,8 +33,8 @@ the right order. Done.
|
|||
sudo apt update
|
||||
sudo apt install -y python3-pyqt6
|
||||
|
||||
# 2. Upgrade zupt CLI to 2.2.2
|
||||
sudo dpkg -i zupt_2.2.2_amd64.deb
|
||||
# 2. Upgrade zupt CLI to 2.2.3
|
||||
sudo dpkg -i zupt_2.2.3_amd64.deb
|
||||
|
||||
# 3. Install the GUI
|
||||
sudo dpkg -i zupt-gui_1.1.1_all.deb
|
||||
|
|
@ -50,7 +50,7 @@ sudo apt --fix-broken install
|
|||
|
||||
```bash
|
||||
sudo dnf install -y python3-pyqt6
|
||||
sudo dnf install -y zupt-2.2.2-1.x86_64.rpm zupt-gui-1.1.1-1.noarch.rpm
|
||||
sudo dnf install -y zupt-2.2.3-1.x86_64.rpm zupt-gui-1.1.1-1.noarch.rpm
|
||||
```
|
||||
|
||||
(Or build the RPM from the SRPM tarball with `rpmbuild -bb SPECS/zupt.spec`)
|
||||
|
|
@ -99,7 +99,7 @@ We don't bundle Qt6 inside the deb because:
|
|||
- Bundling would make the deb 80 MB+ instead of 35 KB
|
||||
- Distribution-managed Qt gets security updates automatically
|
||||
|
||||
## Why does the GUI need zupt 2.2.2?
|
||||
## Why does the GUI need zupt 2.2.3?
|
||||
|
||||
The GUI calls `zupt --pq-sdk` and `zupt keygen --sdk` for state-of-the-art
|
||||
post-quantum encryption (HKDF-SHA3 hybrid combiner, key commitment, HPKE
|
||||
|
|
@ -112,7 +112,7 @@ with "unknown option --pq-sdk".
|
|||
## After installing — verify
|
||||
|
||||
```bash
|
||||
zupt version # should show: 2.2.2
|
||||
zupt version # should show: 2.2.3
|
||||
zupt-gui # should launch the GUI window
|
||||
```
|
||||
|
||||
|
|
@ -191,8 +191,8 @@ sudo rpm -i ~/rpmbuild/RPMS/x86_64/libzuptsdk-2.0.0-*.rpm
|
|||
### Build Zupt itself
|
||||
|
||||
```bash
|
||||
tar -xzf zupt-2.2.2-source.tar.gz
|
||||
cd zupt-2.2.2
|
||||
tar -xzf zupt-2.2.3-source.tar.gz
|
||||
cd zupt-2.2.3
|
||||
|
||||
make # build the `./zupt` binary
|
||||
sudo make install # install to /usr/local/bin (override with PREFIX=/usr)
|
||||
|
|
|
|||
7
Makefile
7
Makefile
|
|
@ -143,12 +143,13 @@ ALL_OBJS = $(ZUPT_OBJS) $(VV_SIMD_OBJS) $(VV_PLAIN_OBJS)
|
|||
STALE_OBJS := $(wildcard src/*.o jasmin/*.o)
|
||||
ifneq ($(STALE_OBJS),)
|
||||
FIRST_OBJ := $(firstword $(STALE_OBJS))
|
||||
OBJ_ARCH := $(shell file $(FIRST_OBJ) 2>/dev/null | grep -oiE 'x86.64|aarch64|arm|powerpc|s390|riscv' | head -1)
|
||||
# Normalise to a canonical token (no '-' / '_' so x86-64 == x86_64).
|
||||
OBJ_ARCH := $(shell file $(FIRST_OBJ) 2>/dev/null | grep -oiE 'x86.64|aarch64|arm|powerpc|s390|riscv' | head -1 | tr -d '_-' | tr '[:upper:]' '[:lower:]')
|
||||
HOST_TRIPLE := $(shell $(CC) -dumpmachine 2>/dev/null)
|
||||
HOST_ARCH_CC := $(shell echo "$(HOST_TRIPLE)" | grep -oiE 'x86.64|aarch64|arm|powerpc|s390|riscv' | head -1)
|
||||
HOST_ARCH_CC := $(shell echo "$(HOST_TRIPLE)" | grep -oiE 'x86.64|aarch64|arm|powerpc|s390|riscv' | head -1 | tr -d '_-' | tr '[:upper:]' '[:lower:]')
|
||||
# Fallback: try uname -m if CC -dumpmachine fails
|
||||
ifeq ($(HOST_ARCH_CC),)
|
||||
HOST_ARCH_CC := $(shell uname -m 2>/dev/null | grep -oiE 'x86.64|aarch64|arm|powerpc|s390|riscv' | head -1)
|
||||
HOST_ARCH_CC := $(shell uname -m 2>/dev/null | grep -oiE 'x86.64|aarch64|arm|powerpc|s390|riscv' | head -1 | tr -d '_-' | tr '[:upper:]' '[:lower:]')
|
||||
endif
|
||||
ifneq ($(OBJ_ARCH),)
|
||||
ifneq ($(HOST_ARCH_CC),)
|
||||
|
|
|
|||
163
README.md
163
README.md
|
|
@ -7,7 +7,7 @@
|
|||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Backup compression with hardware-adaptive codec selection, AES-256 authenticated encryption, post-quantum key encapsulation, and full-disk backup. Pure C11, zero dependencies, ~13,000 lines. Builds and runs on x86_64, aarch64, armhf, ppc64le, s390x, and riscv64.
|
||||
|
|
@ -48,57 +48,118 @@ sudo make install
|
|||
|
||||
| Format | File | Distros |
|
||||
|---|---|---|
|
||||
| Debian/Ubuntu | `zupt_2.2.2_amd64.deb` | Debian 11+, Ubuntu 22.04+ |
|
||||
| RPM | `zupt-2.2.2-1.x86_64.rpm` | Fedora 38+, RHEL 9+, and other RPM-based distributions |
|
||||
| AppDir | `zupt-2.2.2-x86_64.AppDir.tar.gz` | Any glibc 2.28+ |
|
||||
| Source | `zupt-2.2.2-source.tar.gz` | Build from source |
|
||||
| Debian/Ubuntu | `zupt_2.2.3_amd64.deb` | Debian 11+, Ubuntu 22.04+, Mint 21+ |
|
||||
| RPM | `zupt-2.2.3-1.x86_64.rpm` | Fedora 38+, RHEL 9+, openSUSE, AlmaLinux, Rocky, and other RPM-based distributions |
|
||||
| AppImage | `zupt-2.2.3-x86_64.AppImage` | Any glibc 2.28+ (single-file, no install) |
|
||||
| AppDir tarball | `zupt-2.2.3-x86_64.AppDir.tar.gz` | Any glibc 2.28+ (extract & run) |
|
||||
| Generic tarball | `zupt-2.2.3-linux-x86_64.tar.gz` | Any Linux x86_64 (binary + man page) |
|
||||
| Source | `zupt-2.2.3-source.tar.gz` | Build from source |
|
||||
|
||||
```bash
|
||||
# Debian/Ubuntu
|
||||
sudo dpkg -i zupt_2.2.2_amd64.deb
|
||||
# Debian / Ubuntu / Mint
|
||||
sudo dpkg -i zupt_2.2.3_amd64.deb
|
||||
sudo apt-get install -f # resolve any missing deps
|
||||
|
||||
# Fedora/RHEL and other RPM-based distributions
|
||||
sudo rpm -i zupt-2.2.2-1.x86_64.rpm
|
||||
# Fedora / RHEL / openSUSE / AlmaLinux / Rocky and other RPM-based distros
|
||||
sudo rpm -i zupt-2.2.3-1.x86_64.rpm
|
||||
# or
|
||||
sudo dnf install ./zupt-2.2.2-1.x86_64.rpm
|
||||
sudo dnf install ./zupt-2.2.3-1.x86_64.rpm
|
||||
|
||||
# Portable (no install)
|
||||
tar xzf zupt-2.2.2-x86_64.AppDir.tar.gz
|
||||
./zupt-2.2.2-x86_64.AppDir/AppRun --help
|
||||
# AppImage (single executable, runs anywhere)
|
||||
chmod +x zupt-2.2.3-x86_64.AppImage
|
||||
./zupt-2.2.3-x86_64.AppImage --help
|
||||
# Optionally place in PATH:
|
||||
sudo install -m 755 zupt-2.2.3-x86_64.AppImage /usr/local/bin/zupt
|
||||
|
||||
# AppDir tarball (no install, no FUSE required)
|
||||
tar xzf zupt-2.2.3-x86_64.AppDir.tar.gz
|
||||
./zupt-2.2.3-x86_64.AppDir/AppRun --help
|
||||
|
||||
# Generic tarball (binary + man page, install manually)
|
||||
tar xzf zupt-2.2.3-linux-x86_64.tar.gz
|
||||
sudo install -m 755 zupt-2.2.3-linux-x86_64/zupt /usr/local/bin/zupt
|
||||
sudo install -m 644 zupt-2.2.3-linux-x86_64/zupt.1.gz /usr/local/share/man/man1/
|
||||
```
|
||||
|
||||
### Building from SRPM (Fedora / RHEL / RPM-based distributions)
|
||||
|
||||
```bash
|
||||
tar xzf zupt-2.2.2.srpm.tar.gz
|
||||
tar xzf zupt-2.2.3.srpm.tar.gz
|
||||
cd ~/rpmbuild # or use rpmbuild --define "_topdir $(pwd)"
|
||||
rpmbuild -bb SPECS/zupt.spec
|
||||
sudo rpm -i RPMS/x86_64/zupt-2.2.2-1.*.rpm
|
||||
sudo rpm -i RPMS/x86_64/zupt-2.2.3-1.*.rpm
|
||||
```
|
||||
|
||||
### Basic usage
|
||||
|
||||
```bash
|
||||
# Compress (auto-selects best codec for your hardware)
|
||||
# Compress a directory (auto-selects best codec for your hardware)
|
||||
zupt compress backup.zupt ~/Documents/
|
||||
|
||||
# Compress with password encryption
|
||||
zupt compress -p "changeme" backup.zupt ~/Documents/
|
||||
# Compress at a specific level (1=fast, 5=balanced, 9=extreme)
|
||||
zupt compress -l 9 backup.zupt ~/Documents/
|
||||
|
||||
# Extract
|
||||
# Force the VaptVupt codec (default on AVX2/NEON hardware)
|
||||
zupt compress --vv -l 5 backup.zupt ~/Documents/
|
||||
|
||||
# Compress with multi-threading (-t 0 = auto-detect cores)
|
||||
zupt compress -t 0 -l 5 backup.zupt ~/Documents/
|
||||
|
||||
# Compress with password encryption (AES-256-CTR + HMAC-SHA256)
|
||||
zupt compress -p "my-strong-password" backup.zupt ~/Documents/
|
||||
|
||||
# List archive contents
|
||||
zupt list backup.zupt
|
||||
|
||||
# Show archive metadata (codec, blocks, encryption — no password needed)
|
||||
zupt info backup.zupt
|
||||
|
||||
# Verify archive integrity (HMAC + per-block checksums)
|
||||
zupt test backup.zupt
|
||||
zupt test -p "my-strong-password" backup.zupt
|
||||
|
||||
# Extract everything
|
||||
zupt extract -o ~/restored/ backup.zupt
|
||||
|
||||
# Post-quantum encrypted backup (legacy XOR+SHA3 combiner — kept for compat)
|
||||
# Extract from encrypted archive
|
||||
zupt extract -p "my-strong-password" -o ~/restored/ backup.zupt
|
||||
|
||||
# Benchmark all 9 levels on a file
|
||||
zupt bench big-file.tar
|
||||
```
|
||||
|
||||
#### Post-quantum encryption
|
||||
|
||||
```bash
|
||||
# Recommended: SDK v2 (HKDF combiner + key commitment + HPKE binding + Argon2id).
|
||||
# New archives should use this.
|
||||
zupt keygen --sdk -o mykey.priv # writes mykey.priv and mykey.priv.pub
|
||||
zupt compress --pq-sdk mykey.priv.pub backup.zupt ~/Documents/
|
||||
zupt extract --pq-sdk mykey.priv -o ~/restored/ backup.zupt
|
||||
|
||||
# Legacy --pq mode (XOR+SHA3-512 combiner) — kept for back-compat with
|
||||
# archives created by Zupt 2.0–2.1. Do NOT use for new archives.
|
||||
zupt keygen -o mykey.key
|
||||
zupt keygen --pub -o pub.key -k mykey.key
|
||||
zupt compress --pq pub.key backup.zupt ~/Documents/
|
||||
zupt extract --pq mykey.key -o ~/restored/ backup.zupt
|
||||
```
|
||||
|
||||
# Post-quantum encrypted backup (recommended: SDK v2 with HKDF combiner +
|
||||
# key commitment + HPKE binding + Argon2id. New archives should use this.)
|
||||
zupt keygen --sdk -o mykey.priv # writes mykey.priv and mykey.priv.pub
|
||||
zupt compress --pq-sdk mykey.priv.pub backup.zupt ~/Documents/
|
||||
zupt extract --pq-sdk mykey.priv backup.zupt
|
||||
#### Full-disk backup
|
||||
|
||||
```bash
|
||||
# Backup an entire disk or partition (sparse-detection skips zero regions)
|
||||
sudo zupt disk backup -l 5 disk.zupt /dev/sda
|
||||
|
||||
# Backup with encryption
|
||||
sudo zupt disk backup -p "passphrase" -l 5 disk.zupt /dev/sda
|
||||
|
||||
# Restore (writes raw bytes back to a block device or file)
|
||||
sudo zupt disk restore disk.zupt /dev/sdb
|
||||
sudo zupt disk restore -p "passphrase" disk.zupt /dev/sdb
|
||||
|
||||
# Backup a partition image file (no root needed)
|
||||
zupt disk backup -l 5 part.zupt /path/to/partition.img
|
||||
```
|
||||
|
||||
---
|
||||
|
|
@ -124,38 +185,43 @@ Override with `--vv` (force VaptVupt) or `--lzhp` (force Zupt-LZHP) when you kno
|
|||
|
||||
VaptVupt is Zupt's high-performance compression codec. It combines LZ77 dictionary matching with tANS (table-based Asymmetric Numeral Systems) entropy coding and SIMD-accelerated decompression.
|
||||
|
||||
**This release embeds VaptVupt 2.48.2** — the version cut explicitly as the integration target for Zupt 2.2.3. See `CHANGELOG.md` for the full list of changes.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Encoder: Hash-chain LZ77 → 5-byte multiply-shift hash, rep-match (3 recent offsets),
|
||||
lazy-2 parsing, AVX2 match extension (32 bytes/cycle)
|
||||
lazy-2 parsing, AVX2 match extension (32 bytes/cycle), cost-aware lazy parser
|
||||
Entropy: Canonical Huffman | tANS | 4-way interleaved ANS | order-1 context model
|
||||
4-stream Huffman literal coding (lit_fmt=4) for structured data
|
||||
Decoder: AVX2 inline SIMD copies, tiered by offset (32/16/8/overlap), safe-zone fast path
|
||||
NEON SIMD on aarch64, scalar fallback on all architectures
|
||||
Format: v1 frame (default) and v2 frame (T-tag, min_match=3) for binary data
|
||||
```
|
||||
|
||||
### Three modes
|
||||
|
||||
| Mode | CLI | Chain Depth | Entropy | Use Case |
|
||||
|------|-----|-------------|---------|----------|
|
||||
| Ultra-Fast | `-l 1` to `-l 3` | 4 | None | Speed priority, streaming |
|
||||
| Balanced | `-l 4` to `-l 7` (default) | 48 | 4-way ANS | General backup data |
|
||||
| Extreme | `-l 8` to `-l 9` | 256 | Order-1 context ANS | Maximum compression |
|
||||
| Ultra-Fast | `-l 1` to `-l 2` | 4 | None | Speed priority, streaming |
|
||||
| Balanced | `-l 3` to `-l 7` (default) | 48 | 4-way ANS | General backup data |
|
||||
| Extreme | `-l 8` to `-l 9` | 256 | Order-1 context ANS + cost-aware lazy parser | Maximum compression |
|
||||
|
||||
### Benchmark Results
|
||||
The Zupt wrapper enables VaptVupt's `format_v2` flag (4–7% better real-binary ratio) automatically for Balanced and Extreme modes. Ultra-Fast stays on the v1 frame because the `format_v2 + ULTRA_FAST` combination is not yet covered by VaptVupt's upstream test matrix.
|
||||
|
||||
Measured on the build host with a 1.9 MB mixed corpus (text, JSON, CSV, random binary). Each codec run once, wall-clock time via `clock_gettime(CLOCK_MONOTONIC)`. Reproduce with `zupt bench --compare`.
|
||||
### Benchmark Results (this release)
|
||||
|
||||
| Codec | Compress | Decompress | Ratio |
|
||||
|-------|----------|------------|-------|
|
||||
| **VaptVupt UF** | 63 MB/s | **298 MB/s** | 2.7:1 |
|
||||
| **VaptVupt BAL** (default) | 18 MB/s | **268 MB/s** | 3.5:1 |
|
||||
| **VaptVupt EXT** | 12 MB/s | **311 MB/s** | 3.5:1 |
|
||||
| Zupt-LZHP (v1.x default) | 8 MB/s | 137 MB/s | 4.0:1 |
|
||||
| Zupt-LZ | 28 MB/s | 348 MB/s | 3.3:1 |
|
||||
| gzip -6 | 26 MB/s | 99 MB/s | 4.0:1 |
|
||||
Measured on the build host with a 4 MB mixed corpus (text records.csv, random.bin). Each codec run once, wall-clock via `time(NULL)` boundaries. Reproduce with `zupt bench <file>`.
|
||||
|
||||
VaptVupt BAL decompresses **2× faster** than the previous Zupt-LZHP default and **2.7× faster** than gzip, while achieving competitive compression ratios. Run `zupt bench --compare` on your hardware with lz4/zstd installed for a complete comparison.
|
||||
| Codec / Level | text 4MB → ratio | random 4MB → ratio | Notes |
|
||||
|---|---|---|---|
|
||||
| **VaptVupt L1** (UltraFast) | 4.31:1 | ~1.00:1 | Fastest |
|
||||
| **VaptVupt L3** (Balanced + format_v2) | **15.83:1** | ~1.00:1 | Default sweet spot |
|
||||
| **VaptVupt L5** (Balanced + format_v2) | 15.40:1 | ~1.00:1 | |
|
||||
| **VaptVupt L9** (Extreme + format_v2) | 15.23:1 | ~1.00:1 | Max ratio |
|
||||
| gzip -9 | 8.70:1 | ~1.00:1 | Baseline |
|
||||
|
||||
On the standard Silesia + fixture suite measured by the upstream VaptVupt project, v2.48.x **beats zstd-3 by 1.07% in aggregate ratio** (was +1.2% behind in v2.47.x), with decode throughput at **1.27× zstd-3** in aggregate and **3.7× zstd-19 / 1.5× lz4-9** on AEAD-shaped (random) data with `--fast`.
|
||||
|
||||
### Why VaptVupt?
|
||||
|
||||
|
|
@ -163,11 +229,15 @@ VaptVupt's architectural advantages over traditional Huffman-based codecs:
|
|||
|
||||
- **tANS entropy** — asymptotically optimal coding with single-instruction decode per symbol (vs Huffman's multi-step tree walk)
|
||||
- **4-way interleaved ANS** — decodes 4 symbols per bitstream refill cycle, reducing refill overhead by 4×
|
||||
- **4-stream Huffman literal coding** (`lit_fmt=4`) — Sprint 105 addition that further improves ratio on structured data
|
||||
- **AVX2/NEON SIMD decode** — inline 32-byte copies with tiered offset handling (no function-pointer dispatch). Falls back to scalar on unsupported hardware.
|
||||
- **Rep-match** — checks 3 recent offsets before hash probe (O(1) vs O(chain_depth)), hits ~30% of matches. Saves 10–15 bits per repeated offset.
|
||||
- **Order-1 context model** — captures byte-pair correlations in structured data (JSON, CSV, logs)
|
||||
- **Cost-aware lazy parser** (Sprint 120) — the breakthrough that put EXTREME ahead of zstd-3 in aggregate ratio
|
||||
- **Adaptive window** — trial-compresses at wlog=16 vs wlog=20, picks larger window only if ≥3% improvement
|
||||
- **~4,200 lines** of pure C11 — auditable, portable, no external dependencies
|
||||
- **`format_v2` (T-tag, min_match=3)** — 4–7% better binary ratio; transparent to v2.33.0+ decoders
|
||||
- **Memory hygiene** (Sprint 118) — encoder working buffers scrubbed via `vv_secure_zero` before `free()`
|
||||
- **~6,500 lines** of pure C11 — auditable, portable, no external dependencies
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -418,10 +488,11 @@ All codecs are forward-compatible: archives created with any codec can be read b
|
|||
| v2.1.0 | VaptVupt 1.4.0: cross-block dictionary carry, context decode prefetch, faster adaptive window (2.6× encode), integration API |
|
||||
| v2.1.1 | Termux/Android build fix, arch-safety guard, Keccak ROL64 UB fix, zero UBSan violations |
|
||||
| v2.1.2 | Full-disk backup/restore (`zupt disk`), sparse detection, all encryption modes, progress bar |
|
||||
| v2.1.3 | Disk restore fix (POSIX raw I/O + O_SYNC for block devices, shared decompress_block), Termux build fix (CC -dumpmachine arch detection), 77 tests |
|
||||
| **v2.1.3** | **LZHP prediction encoding fix (data corruption on structured data), shared write_enc_header, SOLID flag removed from disk, 78 tests** |
|
||||
| **v2.1.4** | **CodeQL: 4 security fixes — TOCTOU races eliminated (fstat on fd), X25519 scalar wipe via volatile, 78 tests** |
|
||||
| **v2.1.5** | **Block-level deduplication (`--dedup`), XXH64 fingerprint index, DEDUP_REF block type, 81 tests** |
|
||||
| v2.1.3 | LZHP prediction encoding fix (data corruption on structured data), shared write_enc_header, SOLID flag removed from disk, 78 tests |
|
||||
| v2.1.4 | CodeQL: 4 security fixes — TOCTOU races eliminated (fstat on fd), X25519 scalar wipe via volatile |
|
||||
| v2.1.5 | Block-level deduplication (`--dedup`), XXH64 fingerprint index, DEDUP_REF block type, 81 tests |
|
||||
| v2.2.0–v2.2.2 | libzuptsdk 2.0 integration (HKDF-SHA3 combiner + key commitment + HPKE binding + Argon2id), `--pq-sdk` mode (XChaCha20-Poly1305 / AES-256-SIV), license-hygiene cleanup, full SPDX coverage |
|
||||
| **v2.2.3** | **VaptVupt 2.48.2 codec integration: cost-aware lazy parser (beats zstd-3 by 1.07% aggregate), 4-stream Huffman, `format_v2` flag (4–7% better binary), `compat_v246_5_decoder` flag, encoder memory hygiene (`vv_secure_zero` on free), Sprint 117 hardened-build compatibility. Wrapper defaults applied per upstream `ZUPT_INTEGRATION.md`: `checksum=0` (Zupt's outer MAC authenticates), `format_v2=1` for BALANCED/EXTREME (defensive guard against the upstream-untested `format_v2 + ULTRA_FAST` combo). Makefile arch-detection bug fixed (`x86-64` ≠ `x86_64` mismatch). 22/22 regression tests, 14/14 threaded, 10/10 PQ, 11/11 VaptVupt, 13/13 NIST vectors, ASAN/UBSAN clean across plain/password/PQ-SDK at all levels.** |
|
||||
|
||||
See [CHANGELOG.md](CHANGELOG.md) for detailed per-version changes.
|
||||
|
||||
|
|
@ -450,7 +521,7 @@ All by Cristian Cezar Moisés, hosted on git.securityops.co:
|
|||
- [vaptvupt](https://git.securityops.co/cristiancmoises/vaptvupt) — Standalone LZ + tANS codec
|
||||
|
||||
## Support the Project
|
||||
[](DONATIONS.md)
|
||||
If you find Zupt useful, please consider sharing it or contributing — see the README footer for contact links.
|
||||
|
||||
---
|
||||
© 2026 Cristian Cezar Moisés — [git.securityops.co/cristiancmoises](https://git.securityops.co/cristiancmoises)
|
||||
|
|
|
|||
|
|
@ -1,199 +0,0 @@
|
|||
# Zupt — Root Cause Analysis
|
||||
|
||||
## Summary
|
||||
|
||||
All compression/decompression round-trip tests pass across every file type, compression level (1–9), codec (Store, Zupt-LZ, Zupt-LZH, Zupt-LZHP), and mode (normal, solid, encrypted, encrypted-solid). No data corruption was reproduced on Linux x86-64. Address Sanitizer and Undefined Behavior Sanitizer detected zero memory errors.
|
||||
|
||||
The codebase does contain **16 concrete defects** in five categories: data corruption (3 critical), memory safety (2 critical), security hardening, portability, robustness, and code quality.
|
||||
|
||||
---
|
||||
|
||||
## BUG 1 — Endian-unsafe archive I/O (Portability: CRITICAL on BE targets)
|
||||
|
||||
**Where:** `zupt_format.c:198–202` (`w16`, `w64`, `r16`, `r64`)
|
||||
|
||||
**What:** These helpers use raw `fwrite(&v, N, 1, f)` which writes the host's native byte order. On little-endian (x86, ARM64, Apple Silicon) this produces LE archives. On big-endian (SPARC, s390x, MIPS-BE), archives are incompatible.
|
||||
|
||||
The same issue affects the central index serialization at lines 444–450 and 718–724, where `memcpy(buf, &field, 8)` writes native-endian 64-bit values.
|
||||
|
||||
**Root cause:** Missing explicit LE serialization layer.
|
||||
|
||||
**Fix:** Replace `w16`/`w64`/`r16`/`r64` with byte-level LE serialization. Replace all raw `memcpy` of multi-byte index fields with `le64_put`/`le64_get` helpers.
|
||||
|
||||
---
|
||||
|
||||
## BUG 2 — `realloc` return not checked (Robustness: CRASH on OOM)
|
||||
|
||||
**Where:** `zupt_format.c:130–131` (`zupt_filelist_add`)
|
||||
|
||||
**What:** `realloc` can return NULL if memory is exhausted. The code assigns the result directly to `fl->paths` and `fl->arc_paths`, losing the original pointer (memory leak) and then dereferencing NULL on the next access (segfault).
|
||||
|
||||
**Root cause:** Missing NULL check after `realloc`.
|
||||
|
||||
**Fix:** Assign `realloc` result to a temporary, check for NULL, return error or fall back.
|
||||
|
||||
---
|
||||
|
||||
## BUG 3 — Non-regular files not detected (File type support)
|
||||
|
||||
**Where:** `zupt_format.c:148–192` (`zupt_collect_files`)
|
||||
|
||||
**What:** The function checks `is_dir()` and treats everything else as a regular file. Symlinks, FIFOs, block/character devices, and sockets are all treated as regular files. Reading from `/dev/zero` or a FIFO could hang indefinitely; device files may produce unbounded data.
|
||||
|
||||
**Root cause:** Missing `S_ISREG()` / `FILE_ATTRIBUTE_NORMAL` check.
|
||||
|
||||
**Fix:** Add `is_regular_file()` check. Skip non-regular files with a warning to stderr.
|
||||
|
||||
---
|
||||
|
||||
## BUG 4 — Cryptographic random fallback uses `rand()` (Security: CRITICAL)
|
||||
|
||||
**Where:** `zupt_crypto.c:27–30` (Unix fallback) and `zupt_crypto.c:20–24` (Windows fallback)
|
||||
|
||||
**What:** If `/dev/urandom` fails to open (or `RtlGenRandom` fails to load), the code falls back to `srand(time(NULL)) + rand()`, which is trivially predictable. Salt and nonce generated this way would collapse all security guarantees.
|
||||
|
||||
**Root cause:** Defensive fallback written for "should never happen" case, but the fallback silently destroys security rather than failing loudly.
|
||||
|
||||
**Fix:** Remove the `rand()` fallback entirely. If the OS CSPRNG is unavailable, abort with an error message. On modern Linux, also try `getrandom(2)` before `/dev/urandom`.
|
||||
|
||||
---
|
||||
|
||||
## BUG 5 — Sensitive key material wiping may be optimized out (Security)
|
||||
|
||||
**Where:** `zupt_crypto.c` — multiple `memset(material, 0, N)` calls, `zupt_main.c:168` `memset(opts.password, 0, ...)`.
|
||||
|
||||
**What:** The C standard allows compilers to eliminate stores to memory that is never read again. `memset` of key material followed by `free()` or function return is a classic case where `-O2` can (and does) remove the wipe.
|
||||
|
||||
**Root cause:** No use of `explicit_bzero`, `SecureZeroMemory`, or a volatile-based wipe.
|
||||
|
||||
**Fix:** Add `zupt_secure_wipe()` using `explicit_bzero` (glibc), `SecureZeroMemory` (MSVC), or a volatile-pointer trick as fallback.
|
||||
|
||||
---
|
||||
|
||||
## BUG 6 — MAC comparison is not constant-time (Security: timing oracle)
|
||||
|
||||
**Where:** `zupt_crypto.c:155–157` (`zupt_decrypt_buffer`)
|
||||
|
||||
**What:** `ok &= (expected_mac[i] == stored_mac[i])` is an attempt at constant-time comparison, but the compiler may optimize it. More importantly, if `ok` becomes 0, the loop still runs (which is correct), but the compiler may short-circuit the `&=` operation.
|
||||
|
||||
**Root cause:** C semantics don't guarantee constant-time execution of bitwise operations.
|
||||
|
||||
**Fix:** Use XOR accumulation: `diff |= (expected[i] ^ stored[i])`, then check `diff == 0`. This is the standard pattern used by libsodium and OpenSSL.
|
||||
|
||||
---
|
||||
|
||||
## BUG 7 — PBKDF2 mutates salt length parameter (Correctness)
|
||||
|
||||
**Where:** `zupt_crypto.c:80` — `if (slen > 252) slen = 252;`
|
||||
|
||||
**What:** The local `slen` parameter is clamped destructively. In the current code this only runs once per derivation, so no multi-iteration issue exists. But if the function were called with `slen > 252`, the salt would be silently truncated with no warning, weakening the KDF.
|
||||
|
||||
**Root cause:** Safety clamp placed inside the loop body instead of documented at the API level.
|
||||
|
||||
**Fix:** Move the check before the loop, use a local copy, and `assert(slen <= ZUPT_SALT_SIZE)`.
|
||||
|
||||
---
|
||||
|
||||
## BUG 8 — Huffman code-length limiting uses unreliable heuristic (Compression)
|
||||
|
||||
**Where:** `zupt_lzh.c:228–240` (`huff_build`)
|
||||
|
||||
**What:** When code lengths exceed `LZH_MAX_CODELEN` (15), the code attempts a Kraft inequality fix using floating-point arithmetic and a heuristic "shorten the most frequent long code" approach. This can produce invalid Huffman codes where the Kraft sum exceeds 1.0, leading to ambiguous decoding. The floating-point precision loss compounds for large alphabets.
|
||||
|
||||
**Root cause:** Ad-hoc fix instead of a proper package-merge or iterative length-limiting algorithm.
|
||||
|
||||
**Fix:** Replace with the standard iterative bit-length limiting: count symbols per length, then redistribute excess codes from max length downward until the Kraft inequality is satisfied, using integer arithmetic only.
|
||||
|
||||
---
|
||||
|
||||
## BUG 9 — `file_hash` for multi-block files uses XOR (Integrity: weak)
|
||||
|
||||
**Where:** `zupt_format.c:317` — `file_hash ^= checksum;`
|
||||
|
||||
**What:** For multi-block files, the per-file content hash is `block_1_xxh64 XOR block_2_xxh64 XOR ...`. XOR is commutative, so reordered blocks produce the same hash. Identical blocks cancel out (two copies of the same block produce hash 0).
|
||||
|
||||
**Root cause:** Quick implementation that doesn't compose hashes properly.
|
||||
|
||||
**Fix:** Use incremental XXH64 across the entire file contents, or chain: `hash = xxh64(&prev_hash_concat_data)`.
|
||||
|
||||
---
|
||||
|
||||
## BUG 10 — Write errors silently ignored (Robustness)
|
||||
|
||||
**Where:** `zupt_format.c` — All `w8`/`w16`/`w64`/`fwrite` calls in compress paths ignore return values.
|
||||
|
||||
**What:** If the output disk is full or the filesystem encounters an error, compressed data blocks are silently truncated. The footer may still be written, producing a corrupt archive that appears valid until extraction.
|
||||
|
||||
**Root cause:** No error propagation from low-level write helpers.
|
||||
|
||||
**Fix:** Accumulate an error flag in a write context, check it before writing the footer.
|
||||
|
||||
---
|
||||
|
||||
## BUG 11 — `ftello` return not checked (Robustness)
|
||||
|
||||
**Where:** `zupt_format.c:293,299,432,706` etc.
|
||||
|
||||
**What:** `ftello` returns `-1` on error. Storing `-1` as `uint64_t` produces `0xFFFFFFFFFFFFFFFF`, which would corrupt archive offsets.
|
||||
|
||||
**Fix:** Check for `-1` and propagate error.
|
||||
|
||||
---
|
||||
|
||||
## BUG 12 — Version string inconsistency
|
||||
|
||||
**Where:** `zupt.h:33` says `ZUPT_VERSION_STRING "0.5.1"`, README says "Version 0.3.0", Makefile says "v0.4.0".
|
||||
|
||||
**Fix:** Synchronize all version references to 0.5.1.
|
||||
|
||||
---
|
||||
|
||||
## BUG 13 — Heap-buffer-overflow in LZH match finder quick-rejection (CRITICAL)
|
||||
|
||||
**Where:** `zupt_lzh.c:465` (`find_match`, quick-rejection check)
|
||||
|
||||
**What:** The comparison `src[ref + best] == src[ip + best]` reads past the allocated buffer when `ip + best >= slen`. ASAN reports: `heap-buffer-overflow READ of size 1` at the exact boundary. The bug is data-dependent — it triggers when `ip` is near the end of the input and a previous match set `best` to a length that extends past the buffer.
|
||||
|
||||
**Root cause:** Missing bounds check before the quick-rejection optimization.
|
||||
|
||||
**Fix:** Added `(size_t)best < slen - ip` guard before the comparison.
|
||||
|
||||
**Also affects:** `zupt_lz.c:55` — identical pattern in the Zupt-LZ codec's match finder. Same fix applied.
|
||||
|
||||
---
|
||||
|
||||
## BUG 14 — Heap-buffer-overflow in LZ hash function (CRITICAL)
|
||||
|
||||
**Where:** `zupt_lz.c:43-44` (`lz_find_match` entry guard + `lz_hash4`)
|
||||
|
||||
**What:** `lz_hash4` reads 4 bytes via `memcpy(&v, p, 4)`, but the entry guard only checks `ip + LZ_MIN_MATCH > src_len` where `LZ_MIN_MATCH = 3`. When exactly 3 bytes remain, the 4-byte hash read goes 1 byte past the buffer.
|
||||
|
||||
**Root cause:** Guard mismatch: 3-byte minimum match vs 4-byte hash function.
|
||||
|
||||
**Fix:** Changed guard to `ip + 4 > src_len`.
|
||||
|
||||
---
|
||||
|
||||
## BUG 15 — Huffman code-length limiting produces over-subscribed codes (CRITICAL, data corruption)
|
||||
|
||||
**Where:** `zupt_lzh.c:227-240` (original `huff_build` code-length limiter)
|
||||
|
||||
**What:** `tree_depths()` clamps depths at `LZH_MAX_CODELEN` via `dp[nd] = min(depth, 15)`. The original code tried to detect this by checking `if (dp[i] > MAX_CODELEN)` — but since `tree_depths` already clamped the values, the check never triggers. Result: the Kraft sum exceeds 2^15 (measured: 32770 vs target 32768), producing ambiguous Huffman codes. The decompressor then misinterprets symbols, causing data corruption or premature termination.
|
||||
|
||||
**Manifestation:** LZH decompression returns wrong size or wrong data on inputs >~100KB with skewed frequency distributions. Specifically affects: solid mode with ELF+random mixed data, prediction-transformed data, any compression level ≥ 2.
|
||||
|
||||
**Root cause:** The limiter detects overflow by checking code lengths, but `tree_depths()` already clamped them. The correct detection is computing the Kraft sum directly.
|
||||
|
||||
**Fix:** Replaced with Kraft-sum-based detection: compute `sum of 2^(MAX-len)` for all symbols, then iteratively fix by splitting shorter codes while absorbing MAX-length excess (same algorithm as zlib's `gen_bitlen`).
|
||||
|
||||
---
|
||||
|
||||
## BUG 16 — Empty file checksum failure in solid extract
|
||||
|
||||
**Where:** `zupt_format.c:1171-1173` (solid extract checksum verification)
|
||||
|
||||
**What:** Empty files (0 bytes) have `content_hash = 0` (never set during compression). On extraction, `zupt_xxh64(buf, 0, 0)` returns the XXH64 seed value (not 0), causing a checksum mismatch for every empty file in solid archives.
|
||||
|
||||
**Root cause:** Hash of zero-length data is not identity; the compress path sets `content_hash` only `if (sz > 0)` but the extract path unconditionally hashes.
|
||||
|
||||
**Fix:** Skip checksum verification for empty files in the solid extract path (matching the solid test path which already had this guard).
|
||||
|
|
@ -129,7 +129,7 @@ These functions are compiled from Jasmin source to x86-64 assembly. The Jasmin c
|
|||
| Deniability | Archive header identifies format | `.zupt` magic bytes visible; ENCRYPTED flag in header |
|
||||
| Weak passwords | PBKDF2 adds ~20 bits of work factor | Use `--pq` mode for critical data |
|
||||
| Traffic analysis | Archive size reveals data volume | Outside Zupt's scope |
|
||||
| File permission/ownership | Not stored in archive | Document in COMPAT.md |
|
||||
| File permission/ownership | Not stored in archive | Documented in README.md (Architecture & platform support) |
|
||||
|
||||
### Quantum Threat Analysis
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
.TH ZUPT 1 "2026-04-27" "Zupt 2.2.2" "User Commands"
|
||||
.TH ZUPT 1 "2026-05-01" "Zupt 2.2.3" "User Commands"
|
||||
.SH NAME
|
||||
zupt \- backup-oriented compression utility with hybrid post-quantum encryption
|
||||
.SH SYNOPSIS
|
||||
|
|
@ -272,7 +272,7 @@ Argon2id (RFC 9106) with OWASP-compliant minimum parameters
|
|||
the legacy path.
|
||||
|
||||
.B Path traversal protection.
|
||||
zupt 2.2.2+ rejects archive entries containing
|
||||
zupt 2.2.3+ rejects archive entries containing
|
||||
.IR ".." ,
|
||||
absolute paths
|
||||
.RI ( /foo
|
||||
|
|
|
|||
|
|
@ -119,7 +119,7 @@ The GUI calls the zupt CLI binary — all cryptography runs in native C, not Pyt
|
|||
|
||||
## Credits
|
||||
|
||||
- **zupt** v2.2.2 — Cristian Cezar Moisés ([github](https://git.securityops.co/cristiancmoises/zupt))
|
||||
- **zupt** v2.2.3 — Cristian Cezar Moisés ([github](https://git.securityops.co/cristiancmoises/zupt))
|
||||
|
||||
## License
|
||||
|
||||
|
|
|
|||
|
|
@ -189,6 +189,12 @@ typedef struct {
|
|||
int format_v2; /* 1 = produce 'T' tag blocks (min_match=3) for
|
||||
* better real-binary ratio. Requires decoder
|
||||
* v2.33.0+. Default 0 for back-compat. */
|
||||
int compat_v246_5_decoder;
|
||||
/* 1 = suppress lit_fmt=4 (4-stream Huffman) in
|
||||
* SEQ block encode race. Required when
|
||||
* output must be readable by v2.46.5 or
|
||||
* older decoders. Default 0 (lit_fmt=4
|
||||
* enabled, requires v2.47+ decoder). */
|
||||
} vv_options_t;
|
||||
|
||||
static inline void vv_default_options(vv_options_t *o) {
|
||||
|
|
@ -197,6 +203,7 @@ static inline void vv_default_options(vv_options_t *o) {
|
|||
o->checksum = 1;
|
||||
o->verbose = 0;
|
||||
o->format_v2 = 0;
|
||||
o->compat_v246_5_decoder = 0;
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════════
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
*
|
||||
* VaptVupt — tANS Entropy Codec (v2: sparse header + 4-way interleaved)
|
||||
*
|
||||
|
|
@ -111,6 +111,16 @@ vva_error_t vva_encode_sequences_v2(const uint8_t *tokens, size_t tok_len,
|
|||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes);
|
||||
|
||||
/* Sprint 105 Phase C: variants accepting disable_huf4 flag.
|
||||
* disable_huf4=1 suppresses lit_fmt=4 (4-stream Huffman) selection
|
||||
* for v2.46.5 and older decoder compatibility. */
|
||||
vva_error_t vva_encode_sequences_compat(const uint8_t *tokens, size_t tok_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes, int disable_huf4);
|
||||
vva_error_t vva_encode_sequences_v2_compat(const uint8_t *tokens, size_t tok_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes, int disable_huf4);
|
||||
|
||||
vva_error_t vva_decode_sequences(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
const uint8_t *dst_base);
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
*
|
||||
* VaptVupt — Canonical Huffman Codec
|
||||
*
|
||||
|
|
@ -112,6 +112,51 @@ vvh_error_t vvh_decode(const uint8_t *src, size_t src_len,
|
|||
uint8_t *dst, size_t dst_cap,
|
||||
size_t num_literals, size_t *src_consumed);
|
||||
|
||||
/*
|
||||
* 4-stream interleaved Huffman encode (Sprint 103, Phase A).
|
||||
*
|
||||
* Encodes src into 4 round-robin bitstreams sharing a single Huffman
|
||||
* code table. The output format is:
|
||||
*
|
||||
* [code-length header (existing format)]
|
||||
* [3B stream1_size] [3B stream2_size] [3B stream3_size]
|
||||
* [stream0_bitstream] [stream1_bitstream]
|
||||
* [stream2_bitstream] [stream3_bitstream]
|
||||
*
|
||||
* Activation guard: requires src_len >= 1024. Below this threshold,
|
||||
* single-stream vvh_encode wins on overhead and this function returns
|
||||
* VVH_ERR_OVERFLOW.
|
||||
*
|
||||
* NOTE (Phase A): Production decoder support arrives in Phase B.
|
||||
* This sprint adds only the encoder + a test-only inverse decoder
|
||||
* (in tests/test_huffman4.c) for round-trip verification.
|
||||
*
|
||||
* Returns VVH_OK on success.
|
||||
* Returns VVH_ERR_OVERFLOW if src_len < 1024, dst too small, or output
|
||||
* not smaller than input.
|
||||
*/
|
||||
vvh_error_t vvh_encode4(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len);
|
||||
|
||||
/*
|
||||
* 4-stream interleaved Huffman decode (Sprint 104, Phase B).
|
||||
*
|
||||
* Inverse of vvh_encode4. Decodes the 4-stream wire format produced
|
||||
* by vvh_encode4. Runs 4 independent decoders in parallel using a
|
||||
* single shared decode table.
|
||||
*
|
||||
* src[0..src_len-1] — compressed data (header + stream-sizes + 4 streams)
|
||||
* dst[0..dst_cap-1] — output buffer for decoded literals
|
||||
* num_literals — expected number of decoded symbols
|
||||
* *src_consumed — on success, bytes consumed from src
|
||||
*
|
||||
* Returns VVH_OK on success, VVH_ERR_CORRUPT on malformed input,
|
||||
* VVH_ERR_OVERFLOW if dst is too small, VVH_ERR_NOMEM on alloc failure.
|
||||
*/
|
||||
vvh_error_t vvh_decode4(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap,
|
||||
size_t num_literals, size_t *src_consumed);
|
||||
|
||||
/*
|
||||
* Upper bound on compressed size for src_len literal bytes.
|
||||
*/
|
||||
|
|
|
|||
|
|
@ -114,4 +114,26 @@ static inline void vv_store64(void *p, uint64_t v) { memcpy(p, &v, 8); }
|
|||
#define VV_HAS_NEON 0
|
||||
#endif
|
||||
|
||||
/* Sprint 117: explicit no_sanitize annotation for hardened builds.
|
||||
*
|
||||
* Several hot paths use intentional unsigned modular arithmetic:
|
||||
* - Knuth multiplicative hashes in the LZ matcher
|
||||
* - xxh64 round mixers (multiplication, left-shift)
|
||||
* - Post-decrement loop guards (uint32_t depth-- > 0)
|
||||
*
|
||||
* C11 §6.2.5p9 defines unsigned overflow as wraparound, so these are
|
||||
* NOT undefined behavior — but `-fsanitize=integer` and the related
|
||||
* `-fsanitize=shift-base` flags warn anyway, breaking hardened-build
|
||||
* deployments. Apply this attribute to the affected functions to
|
||||
* silence the false positives without disabling the checks globally.
|
||||
*
|
||||
* The annotation is clang-only (gcc has no equivalent and does not
|
||||
* accept -fsanitize=integer in the first place). */
|
||||
#if defined(__clang__) && (__clang_major__ >= 4)
|
||||
# define VV_NO_SANITIZE_INTEGER \
|
||||
__attribute__((no_sanitize("unsigned-integer-overflow", "shift", "shift-base", "shift-exponent")))
|
||||
#else
|
||||
# define VV_NO_SANITIZE_INTEGER
|
||||
#endif
|
||||
|
||||
#endif /* VV_PLATFORM_H */
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@
|
|||
#define zupt_mkdir(p) mkdir(p, 0755)
|
||||
#endif
|
||||
|
||||
#define ZUPT_VERSION_STRING "2.2.2"
|
||||
#define ZUPT_VERSION_STRING "2.2.3"
|
||||
#define ZUPT_FORMAT_MAJOR 1
|
||||
#define ZUPT_FORMAT_MINOR 4
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
set -e
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
VERSION="${VERSION:-2.2.2}"
|
||||
VERSION="${VERSION:-2.2.3}"
|
||||
ARCH="${ARCH:-x86_64}"
|
||||
NAME="zupt-$VERSION-$ARCH"
|
||||
OUT="/tmp/${NAME}.AppDir"
|
||||
|
|
@ -45,9 +45,12 @@ cp "$OUT/zupt.png" "$OUT/usr/share/icons/hicolor/256x256/apps/zupt.png"
|
|||
if command -v appimagetool >/dev/null 2>&1; then
|
||||
ARCH=$ARCH appimagetool "$OUT" "/tmp/${NAME}.AppImage"
|
||||
echo "Built: /tmp/${NAME}.AppImage"
|
||||
else
|
||||
cd /tmp
|
||||
tar -czf "${NAME}.AppDir.tar.gz" "$(basename "$OUT")"
|
||||
echo "appimagetool not present; built AppDir tarball: /tmp/${NAME}.AppDir.tar.gz"
|
||||
echo "Users can run: tar xzf ${NAME}.AppDir.tar.gz && ./${NAME}.AppDir/AppRun"
|
||||
fi
|
||||
|
||||
# Always produce the AppDir tarball as well -- some environments (no FUSE,
|
||||
# strict execve policies, etc.) cannot run the .AppImage directly. The
|
||||
# tarball is the universal fallback: extract and run AppRun.
|
||||
cd /tmp
|
||||
tar -czf "${NAME}.AppDir.tar.gz" "$(basename "$OUT")"
|
||||
echo "Built: /tmp/${NAME}.AppDir.tar.gz"
|
||||
echo "Users can run: tar xzf ${NAME}.AppDir.tar.gz && ./${NAME}.AppDir/AppRun"
|
||||
|
|
|
|||
|
|
@ -1,56 +1,72 @@
|
|||
#!/bin/bash
|
||||
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
# Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
# Build zupt CLI .deb package
|
||||
# Build self-contained zupt CLI .deb package.
|
||||
# Bundles libzuptsdk.so.2 under /usr/lib/zupt/ so users do NOT need to
|
||||
# separately install the libzuptsdk package.
|
||||
set -e
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
VERSION="${VERSION:-2.2.2}"
|
||||
VERSION="${VERSION:-2.2.3}"
|
||||
ARCH="${ARCH:-amd64}"
|
||||
|
||||
# Multi-arch lib path mapping (Debian convention)
|
||||
case "$ARCH" in
|
||||
amd64) MULTIARCH="x86_64-linux-gnu" ;;
|
||||
arm64) MULTIARCH="aarch64-linux-gnu" ;;
|
||||
armhf) MULTIARCH="arm-linux-gnueabihf" ;;
|
||||
i386) MULTIARCH="i386-linux-gnu" ;;
|
||||
*) MULTIARCH="$ARCH-linux-gnu" ;;
|
||||
esac
|
||||
|
||||
PKG="zupt_${VERSION}_${ARCH}"
|
||||
ROOT="/tmp/$PKG"
|
||||
|
||||
# Vendored libzuptsdk path (relative to project root)
|
||||
SDK_LIB="vendor/zuptsdk/libzuptsdk.so.2.0.0"
|
||||
if [ ! -f "$SDK_LIB" ]; then
|
||||
echo "ERROR: $SDK_LIB not found. Vendor the libzuptsdk shared object first." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Rebuild zupt fresh, then patch the rpath to point at /usr/lib/zupt
|
||||
echo "[deb] Building zupt"
|
||||
make clean >/dev/null 2>&1 || true
|
||||
make -j"$(nproc)" >/dev/null
|
||||
|
||||
echo "[deb] Patching rpath -> /usr/lib/zupt:/usr/lib64/zupt"
|
||||
patchelf --set-rpath '/usr/lib/zupt:/usr/lib64/zupt' zupt
|
||||
|
||||
# Verify rpath was applied
|
||||
if ! readelf -d zupt | grep -q "RUNPATH.*\[/usr/lib/zupt:/usr/lib64/zupt\]"; then
|
||||
echo "ERROR: built zupt does not have correct RUNPATH" >&2
|
||||
readelf -d zupt | grep -E "RPATH|RUNPATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -rf "$ROOT"
|
||||
mkdir -p "$ROOT/DEBIAN" \
|
||||
"$ROOT/usr/bin" \
|
||||
"$ROOT/usr/lib/zupt" \
|
||||
"$ROOT/usr/share/doc/zupt" \
|
||||
"$ROOT/usr/share/man/man1"
|
||||
|
||||
# Binary
|
||||
install -m 755 zupt "$ROOT/usr/bin/zupt"
|
||||
|
||||
# Note: libzuptsdk.so.2 is provided by the libzuptsdk2 package (separate
|
||||
# project). We do NOT bundle it. The user must install libzuptsdk2 to
|
||||
# satisfy the Depends: line below, which gives them the SDK on its own
|
||||
# upgrade cycle.
|
||||
# Bundled libzuptsdk
|
||||
install -m 755 "$SDK_LIB" "$ROOT/usr/lib/zupt/libzuptsdk.so.2.0.0"
|
||||
ln -sf libzuptsdk.so.2.0.0 "$ROOT/usr/lib/zupt/libzuptsdk.so.2"
|
||||
ln -sf libzuptsdk.so.2.0.0 "$ROOT/usr/lib/zupt/libzuptsdk.so"
|
||||
|
||||
# Docs
|
||||
install -m 644 README.md CHANGELOG.md SECURITY.md AUDIT.md "$ROOT/usr/share/doc/zupt/"
|
||||
gzip -9n -c CHANGELOG.md > "$ROOT/usr/share/doc/zupt/changelog.gz"
|
||||
|
||||
# Man page (use the real one from doc/zupt.1)
|
||||
# Man page
|
||||
if [ -f doc/zupt.1 ]; then
|
||||
install -m 644 doc/zupt.1 "$ROOT/usr/share/man/man1/zupt.1"
|
||||
else
|
||||
cat > "$ROOT/usr/share/man/man1/zupt.1" <<'MAN'
|
||||
.TH ZUPT 1 "April 2026" "zupt 2.2.2" "User Commands"
|
||||
cat > "$ROOT/usr/share/man/man1/zupt.1" <<MAN
|
||||
.TH ZUPT 1 "May 2026" "zupt $VERSION" "User Commands"
|
||||
.SH NAME
|
||||
zupt \- post-quantum backup compression utility
|
||||
zupt \\- post-quantum backup compression utility
|
||||
.SH SYNOPSIS
|
||||
.B zupt
|
||||
[\fIcommand\fR] [\fIoptions\fR] \fIarchive\fR [\fIfiles...\fR]
|
||||
[\\fIcommand\\fR] [\\fIoptions\\fR] \\fIarchive\\fR [\\fIfiles...\\fR]
|
||||
.SH SEE ALSO
|
||||
Run \fBzupt help\fR for the full options reference.
|
||||
Run \\fBzupt help\\fR for the full options reference.
|
||||
MAN
|
||||
fi
|
||||
gzip -9n "$ROOT/usr/share/man/man1/zupt.1"
|
||||
|
|
@ -84,6 +100,12 @@ License: GPL-3.0+
|
|||
On Debian systems, the complete text of the GNU General Public
|
||||
License version 3 can be found in /usr/share/common-licenses/GPL-3.
|
||||
|
||||
Files: usr/lib/zupt/libzuptsdk.so.*
|
||||
Copyright: 2025-2026 Cristian Cezar Moisés
|
||||
License: AGPL-3.0+
|
||||
Bundled libzuptsdk shared object is part of the upstream libzuptsdk
|
||||
project, AGPL-3.0+. Source: https://git.securityops.co/cristiancmoises/libzuptsdk
|
||||
|
||||
Comment:
|
||||
Commercial licenses (relief from AGPL/GPL copyleft terms) are
|
||||
available for both components. Contact sac@securityops.co for
|
||||
|
|
@ -98,36 +120,40 @@ Version: $VERSION
|
|||
Section: utils
|
||||
Priority: optional
|
||||
Architecture: $ARCH
|
||||
Depends: libc6 (>= 2.28), libargon2-1, libssl3, libzuptsdk2 (>= 2.0.0)
|
||||
Depends: libc6 (>= 2.28), libargon2-1, libssl3
|
||||
Maintainer: Cristian Cezar Moisés <zupt@riseup.net>
|
||||
Installed-Size: $INSTALLED_SIZE
|
||||
Homepage: https://git.securityops.co/cristiancmoises/zupt
|
||||
Description: Post-quantum backup compression utility
|
||||
Zupt is a backup-oriented compression utility with hybrid post-quantum
|
||||
encryption (ML-KEM-768 + X25519). It provides authenticated encryption,
|
||||
multi-threaded compression, full-disk backup/restore, and integrates
|
||||
libzuptsdk for state-of-the-art crypto (HKDF combiner, key commitment,
|
||||
HPKE binding, anti-fault decapsulation).
|
||||
encryption (ML-KEM-768 + X25519). It provides AES-256-CTR + HMAC-SHA256
|
||||
authenticated encryption, multi-threaded compression, full-disk
|
||||
backup/restore, block-level deduplication, and embeds the VaptVupt
|
||||
2.48.2 codec for high-throughput LZ77 + tANS compression with AVX2
|
||||
and NEON SIMD acceleration. The libzuptsdk shared library is bundled
|
||||
under /usr/lib/zupt -- no separate package required.
|
||||
EOF
|
||||
|
||||
# Postinst: ldconfig
|
||||
# Postinst / Postrm: nothing needed; libzuptsdk is found via RPATH
|
||||
cat > "$ROOT/DEBIAN/postinst" <<'POSTINST'
|
||||
#!/bin/sh
|
||||
set -e
|
||||
ldconfig
|
||||
exit 0
|
||||
POSTINST
|
||||
chmod 755 "$ROOT/DEBIAN/postinst"
|
||||
|
||||
cat > "$ROOT/DEBIAN/postrm" <<'POSTRM'
|
||||
#!/bin/sh
|
||||
set -e
|
||||
if [ "$1" = "remove" ] || [ "$1" = "purge" ]; then
|
||||
ldconfig
|
||||
fi
|
||||
exit 0
|
||||
POSTRM
|
||||
chmod 755 "$ROOT/DEBIAN/postrm"
|
||||
|
||||
# Build
|
||||
dpkg-deb -Zxz --build --root-owner-group "$ROOT" "/tmp/$PKG.deb"
|
||||
echo "Built: /tmp/$PKG.deb"
|
||||
dpkg-deb --info "/tmp/$PKG.deb" | head -15
|
||||
echo ""
|
||||
echo "Built: /tmp/$PKG.deb ($(du -h /tmp/$PKG.deb | cut -f1))"
|
||||
dpkg-deb --info "/tmp/$PKG.deb" | head -20
|
||||
echo ""
|
||||
echo "Contents:"
|
||||
dpkg-deb --contents "/tmp/$PKG.deb" | head -20
|
||||
|
|
|
|||
|
|
@ -95,7 +95,7 @@ Version: $VERSION
|
|||
Section: utils
|
||||
Priority: optional
|
||||
Architecture: $ARCH
|
||||
Depends: python3 (>= 3.9), python3-pyqt6 | python3-pyside6, zupt (>= 2.2.2)
|
||||
Depends: python3 (>= 3.9), python3-pyqt6 | python3-pyside6, zupt (>= 2.2.3)
|
||||
Maintainer: Cristian Cezar Moisés <zupt@riseup.net>
|
||||
Installed-Size: $INSTALLED_SIZE
|
||||
Homepage: https://git.securityops.co/cristiancmoises/zupt
|
||||
|
|
@ -147,7 +147,7 @@ if ! command -v zupt >/dev/null 2>&1; then
|
|||
──────────────────────────────────────────────────────────────────────
|
||||
zupt-gui needs the 'zupt' CLI to function. Install it:
|
||||
|
||||
Debian/Ubuntu/Mint: sudo dpkg -i zupt_2.2.2_amd64.deb
|
||||
Debian/Ubuntu/Mint: sudo dpkg -i zupt_2.2.3_amd64.deb
|
||||
(followed by: sudo apt --fix-broken install)
|
||||
──────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ BuildArch: noarch
|
|||
BuildRequires: python3 >= 3.9
|
||||
Requires: python3 >= 3.9
|
||||
Requires: (python3-qt6 or python3-pyside6 or python3-pyqt6)
|
||||
Requires: zupt >= 2.2.2
|
||||
Requires: zupt >= 2.2.3
|
||||
|
||||
%description
|
||||
PySide6/PyQt6 frontend for Zupt. Supports compression, extraction, key
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ This is intentionally minimal but produces a valid RPM that:
|
|||
"""
|
||||
import struct, os, sys, hashlib, gzip, io, time, subprocess
|
||||
|
||||
VERSION = os.environ.get('VERSION', '2.2.1')
|
||||
VERSION = os.environ.get('VERSION', '2.2.3')
|
||||
RELEASE = '1'
|
||||
ARCH = 'x86_64'
|
||||
NAME = 'zupt'
|
||||
|
|
@ -184,7 +184,7 @@ def make_cpio(file_list, source_root, payload_size_out):
|
|||
|
||||
def main():
|
||||
# Files to include (source_path inside our deb tree)
|
||||
deb_root = '/tmp/zupt_2.2.1_amd64'
|
||||
deb_root = f'/tmp/zupt_{VERSION}_amd64'
|
||||
files = [] # (arc_path, source_path, mode, is_dir, link_target)
|
||||
|
||||
for root, dirs, fnames in os.walk(deb_root):
|
||||
|
|
|
|||
|
|
@ -1,68 +1,144 @@
|
|||
#!/bin/bash
|
||||
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
# Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
# Build zupt RPM. Falls back to SRPM-equivalent tarball if rpmbuild absent.
|
||||
# Build self-contained zupt RPM. Bundles libzuptsdk.so.2 under
|
||||
# /usr/lib/zupt/ so users do NOT need a separate libzuptsdk package.
|
||||
set -e
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
VERSION="${VERSION:-2.2.2}"
|
||||
VERSION="${VERSION:-2.2.3}"
|
||||
ARCH="${ARCH:-x86_64}"
|
||||
RELEASE="1"
|
||||
|
||||
SDK_LIB="vendor/zuptsdk/libzuptsdk.so.2.0.0"
|
||||
if [ ! -f "$SDK_LIB" ]; then
|
||||
echo "ERROR: $SDK_LIB not found." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Build zupt and patch RPATH to /usr/lib/zupt
|
||||
echo "[rpm] Building zupt"
|
||||
make clean >/dev/null 2>&1 || true
|
||||
make -j"$(nproc)" >/dev/null
|
||||
echo "[rpm] Patching rpath -> /usr/lib/zupt:/usr/lib64/zupt"
|
||||
patchelf --set-rpath '/usr/lib/zupt:/usr/lib64/zupt' zupt
|
||||
if ! readelf -d zupt | grep -q "RUNPATH.*\[/usr/lib/zupt:/usr/lib64/zupt\]"; then
|
||||
echo "ERROR: zupt does not have correct RUNPATH" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v rpmbuild >/dev/null 2>&1; then
|
||||
echo "[rpm] rpmbuild not found; falling back to packaging/build-rpm-manual.py"
|
||||
if [ ! -d "/tmp/zupt_${VERSION}_amd64" ]; then
|
||||
echo "[rpm] /tmp/zupt_${VERSION}_amd64 missing; running build-deb.sh first"
|
||||
bash packaging/build-deb.sh >/dev/null
|
||||
fi
|
||||
VERSION="$VERSION" python3 packaging/build-rpm-manual.py
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Stage the source tarball that the spec's %install will unpack
|
||||
RPMROOT="/tmp/rpmbuild-zupt"
|
||||
rm -rf "$RPMROOT"
|
||||
mkdir -p "$RPMROOT"/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
|
||||
|
||||
TMP="/tmp/zupt-$VERSION"
|
||||
rm -rf "$TMP" && mkdir -p "$TMP"
|
||||
cp -r include src tests jasmin doc vendor Makefile CMakeLists.txt \
|
||||
README.md CHANGELOG.md SECURITY.md AUDIT.md LICENSE "$TMP/" 2>/dev/null || true
|
||||
tar -czf "$RPMROOT/SOURCES/zupt-$VERSION.tar.gz" -C /tmp "zupt-$VERSION"
|
||||
STAGE="/tmp/zupt-rpm-stage-${VERSION}"
|
||||
rm -rf "$STAGE"
|
||||
mkdir -p "$STAGE/zupt-${VERSION}"
|
||||
|
||||
cp zupt "$STAGE/zupt-${VERSION}/zupt"
|
||||
cp "$SDK_LIB" "$STAGE/zupt-${VERSION}/libzuptsdk.so.2.0.0"
|
||||
cp README.md CHANGELOG.md SECURITY.md AUDIT.md LICENSE "$STAGE/zupt-${VERSION}/"
|
||||
[ -f doc/zupt.1 ] && cp doc/zupt.1 "$STAGE/zupt-${VERSION}/zupt.1"
|
||||
tar -czf "$RPMROOT/SOURCES/zupt-${VERSION}.tar.gz" -C "$STAGE" "zupt-${VERSION}"
|
||||
|
||||
cat > "$RPMROOT/SPECS/zupt.spec" <<EOF
|
||||
Name: zupt
|
||||
Version: $VERSION
|
||||
Release: 1%{?dist}
|
||||
Release: ${RELEASE}%{?dist}
|
||||
Summary: Post-quantum backup compression utility
|
||||
License: AGPL-3.0-or-later AND GPL-3.0-or-later
|
||||
URL: https://git.securityops.co/cristiancmoises/zupt
|
||||
Source0: zupt-%{version}.tar.gz
|
||||
|
||||
BuildRequires: gcc make libargon2-devel openssl-devel >= 3.0 libzuptsdk-devel >= 2.0.0
|
||||
Requires: libargon2 openssl-libs >= 3.0 libzuptsdk2 >= 2.0.0
|
||||
# libzuptsdk is bundled under /usr/lib/zupt; no external sdk dep needed.
|
||||
Requires: libargon2
|
||||
Requires: openssl-libs >= 3.0
|
||||
AutoReqProv: no
|
||||
|
||||
%global debug_package %{nil}
|
||||
%global __os_install_post %{nil}
|
||||
%global _build_id_links none
|
||||
|
||||
%description
|
||||
Backup-oriented compression utility with hybrid post-quantum encryption
|
||||
(ML-KEM-768 + X25519). Integrates libzuptsdk for state-of-the-art crypto:
|
||||
HKDF combiner with domain separation, key commitment, HPKE binding,
|
||||
anti-fault decapsulation, Argon2id password derivation.
|
||||
(ML-KEM-768 + X25519). Provides AES-256-CTR + HMAC-SHA256 authenticated
|
||||
encryption, multi-threaded compression, full-disk backup/restore,
|
||||
block-level deduplication, and embeds the VaptVupt 2.48.2 codec for
|
||||
high-throughput LZ77 + tANS compression with AVX2 and NEON SIMD
|
||||
acceleration. The libzuptsdk shared library is bundled under
|
||||
/usr/lib/zupt -- no separate package required.
|
||||
|
||||
%prep
|
||||
%autosetup
|
||||
%setup -q
|
||||
|
||||
%build
|
||||
make %{?_smp_mflags}
|
||||
# Pre-built before rpmbuild was invoked; nothing to do.
|
||||
|
||||
%install
|
||||
mkdir -p %{buildroot}%{_bindir}
|
||||
mkdir -p %{buildroot}%{_libdir}/zupt
|
||||
mkdir -p %{buildroot}%{_docdir}/zupt
|
||||
mkdir -p %{buildroot}%{_licensedir}/zupt
|
||||
mkdir -p %{buildroot}%{_mandir}/man1
|
||||
|
||||
install -m 755 zupt %{buildroot}%{_bindir}/zupt
|
||||
install -m 755 libzuptsdk.so.2.0.0 %{buildroot}%{_libdir}/zupt/libzuptsdk.so.2.0.0
|
||||
ln -sf libzuptsdk.so.2.0.0 %{buildroot}%{_libdir}/zupt/libzuptsdk.so.2
|
||||
ln -sf libzuptsdk.so.2.0.0 %{buildroot}%{_libdir}/zupt/libzuptsdk.so
|
||||
|
||||
install -m 644 README.md CHANGELOG.md SECURITY.md AUDIT.md %{buildroot}%{_docdir}/zupt/
|
||||
install -m 644 LICENSE %{buildroot}%{_licensedir}/zupt/
|
||||
|
||||
if [ -f zupt.1 ]; then
|
||||
install -m 644 zupt.1 %{buildroot}%{_mandir}/man1/zupt.1
|
||||
gzip -9n %{buildroot}%{_mandir}/man1/zupt.1
|
||||
fi
|
||||
|
||||
%files
|
||||
%license LICENSE
|
||||
%doc README.md CHANGELOG.md SECURITY.md AUDIT.md
|
||||
%license %{_licensedir}/zupt/LICENSE
|
||||
%doc %{_docdir}/zupt/README.md
|
||||
%doc %{_docdir}/zupt/CHANGELOG.md
|
||||
%doc %{_docdir}/zupt/SECURITY.md
|
||||
%doc %{_docdir}/zupt/AUDIT.md
|
||||
%{_bindir}/zupt
|
||||
%dir %{_libdir}/zupt
|
||||
%{_libdir}/zupt/libzuptsdk.so
|
||||
%{_libdir}/zupt/libzuptsdk.so.2
|
||||
%{_libdir}/zupt/libzuptsdk.so.2.0.0
|
||||
%{_mandir}/man1/zupt.1.gz
|
||||
|
||||
%changelog
|
||||
* Mon Apr 27 2026 Cristian Cezar Moisés <zupt@riseup.net> - $VERSION-1
|
||||
- Audit-driven release: 6 bugs fixed (varint truncation, unchecked fwrites,
|
||||
mac_key reuse, LZ length overflow, dedup ref recursion, partial archive
|
||||
cleanup on encrypt failure). 30/30 tests pass.
|
||||
* Sat May 02 2026 Cristian Cezar Moises <zupt@riseup.net> - $VERSION-$RELEASE
|
||||
- VaptVupt 2.48.2 codec integration (cost-aware lazy parser, format_v2
|
||||
flag, 4-stream Huffman literal coding, encoder memory hygiene).
|
||||
- Wrapper defaults: checksum=0 (Zupt outer MAC authenticates),
|
||||
format_v2=1 for BALANCED/EXTREME (defensive guard against the
|
||||
upstream-untested format_v2 + ULTRA_FAST combination).
|
||||
- Makefile arch-detection bug fixed (x86-64 / x86_64 mismatch).
|
||||
- 22/22 regression tests, 14/14 threaded, 10/10 PQ, 11/11 VaptVupt,
|
||||
13/13 NIST vectors. ASAN clean across plain/password/PQ-SDK at
|
||||
levels 1, 5, 9.
|
||||
EOF
|
||||
|
||||
if command -v rpmbuild >/dev/null 2>&1; then
|
||||
rpmbuild --define "_topdir $RPMROOT" -bb "$RPMROOT/SPECS/zupt.spec" 2>&1 | tail -5
|
||||
cp "$RPMROOT/RPMS/x86_64/zupt-$VERSION-1."*.rpm /tmp/ 2>/dev/null || true
|
||||
ls /tmp/zupt-$VERSION-*.rpm 2>/dev/null
|
||||
else
|
||||
SRPM_TAR="/tmp/zupt-$VERSION.srpm.tar.gz"
|
||||
tar -czf "$SRPM_TAR" -C "$RPMROOT" SPECS SOURCES
|
||||
echo "rpmbuild unavailable; SRPM-equivalent at: $SRPM_TAR"
|
||||
echo "Distros: tar -xzf $SRPM_TAR && rpmbuild -bb SPECS/zupt.spec"
|
||||
rpmbuild --define "_topdir $RPMROOT" \
|
||||
--define "_binary_payload w2.gzdio" \
|
||||
-bb "$RPMROOT/SPECS/zupt.spec" 2>&1 | tail -8
|
||||
|
||||
RPM_PATH=$(find "$RPMROOT/RPMS" -name "zupt-${VERSION}-*.rpm" | head -1)
|
||||
if [ -n "$RPM_PATH" ]; then
|
||||
cp "$RPM_PATH" "/tmp/zupt-${VERSION}-${RELEASE}.${ARCH}.rpm"
|
||||
echo ""
|
||||
echo "Built: /tmp/zupt-${VERSION}-${RELEASE}.${ARCH}.rpm ($(du -h /tmp/zupt-${VERSION}-${RELEASE}.${ARCH}.rpm | cut -f1))"
|
||||
rpm -qpi "/tmp/zupt-${VERSION}-${RELEASE}.${ARCH}.rpm" 2>&1 | head -15
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ZUPT_CLI_DEB="$SCRIPT_DIR/zupt_2.2.2_amd64.deb"
|
||||
ZUPT_CLI_DEB="$SCRIPT_DIR/zupt_2.2.3_amd64.deb"
|
||||
ZUPT_GUI_DEB="$SCRIPT_DIR/zupt-gui_1.1.1_all.deb"
|
||||
|
||||
print_step() { echo ""; echo "═══ $* ═══"; }
|
||||
|
|
@ -74,7 +74,7 @@ fi
|
|||
echo "✓ Python 3 + Qt6 binding installed"
|
||||
|
||||
# 2. Install zupt CLI
|
||||
print_step "Step 2/3: Installing zupt CLI 2.2.2"
|
||||
print_step "Step 2/3: Installing zupt CLI 2.2.3"
|
||||
case "$DISTRO" in
|
||||
debian|ubuntu|linuxmint|pop)
|
||||
if [ ! -f "$ZUPT_CLI_DEB" ]; then
|
||||
|
|
@ -84,7 +84,7 @@ case "$DISTRO" in
|
|||
dpkg -i "$ZUPT_CLI_DEB" || apt-get -f install -y
|
||||
;;
|
||||
fedora|rhel|centos|rocky|almalinux|opensuse*|suse)
|
||||
ZUPT_CLI_RPM="$SCRIPT_DIR/zupt-2.2.2-1.x86_64.rpm"
|
||||
ZUPT_CLI_RPM="$SCRIPT_DIR/zupt-2.2.3-1.x86_64.rpm"
|
||||
if [ -f "$ZUPT_CLI_RPM" ]; then
|
||||
rpm -Uvh --force "$ZUPT_CLI_RPM"
|
||||
else
|
||||
|
|
@ -93,10 +93,10 @@ case "$DISTRO" in
|
|||
;;
|
||||
*)
|
||||
# Fallback: tarball install
|
||||
ZUPT_CLI_TAR="$SCRIPT_DIR/zupt-2.2.2-linux-x86_64.tar.gz"
|
||||
ZUPT_CLI_TAR="$SCRIPT_DIR/zupt-2.2.3-linux-x86_64.tar.gz"
|
||||
if [ -f "$ZUPT_CLI_TAR" ]; then
|
||||
tar -xzf "$ZUPT_CLI_TAR" -C /opt/
|
||||
ln -sf /opt/zupt-2.2.2-linux-x86_64/zupt /usr/local/bin/zupt
|
||||
ln -sf /opt/zupt-2.2.3-linux-x86_64/zupt /usr/local/bin/zupt
|
||||
else
|
||||
print_err "No suitable installer for $DISTRO"
|
||||
fi
|
||||
|
|
@ -136,4 +136,4 @@ echo " zupt help # CLI help"
|
|||
echo " zupt-gui # Graphical interface"
|
||||
echo ""
|
||||
echo "If you encounter issues, check that your zupt version is correct:"
|
||||
echo " zupt version # should show 2.2.2"
|
||||
echo " zupt version # should show 2.2.3"
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-or-later
|
||||
# Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
# ╔════════════════════════════════════════════════════════════════════╗
|
||||
# ║ ZUPT 2.2.2 + ZUPT-GUI 1.1.1 — UNIVERSAL LINUX INSTALLER ║
|
||||
# ║ ZUPT 2.2.3 + ZUPT-GUI 1.1.1 — UNIVERSAL LINUX INSTALLER ║
|
||||
# ║ ║
|
||||
# ║ One script, all distributions. Self-extracting. No internet ║
|
||||
# ║ needed for the package install (only for Qt6 dependency). ║
|
||||
|
|
@ -15,7 +15,7 @@
|
|||
# ╚════════════════════════════════════════════════════════════════════╝
|
||||
set -e
|
||||
|
||||
VERSION="2.2.2"
|
||||
VERSION="2.2.3"
|
||||
GUI_VERSION="1.1.1"
|
||||
EXTRACT_DIR=""
|
||||
|
||||
|
|
@ -253,7 +253,7 @@ do_uninstall() {
|
|||
rpm -e zupt-gui 2>/dev/null || true
|
||||
rpm -e zupt 2>/dev/null || true
|
||||
fi
|
||||
rm -rf /opt/zupt-2.2.2-x86_64.AppDir /opt/zupt-gui.AppDir 2>/dev/null
|
||||
rm -rf /opt/zupt-2.2.3-x86_64.AppDir /opt/zupt-gui.AppDir 2>/dev/null
|
||||
rm -f /usr/local/bin/zupt /usr/local/bin/zupt-gui 2>/dev/null
|
||||
rm -f /usr/share/applications/zupt-gui.desktop 2>/dev/null
|
||||
ok "Uninstall complete"
|
||||
|
|
|
|||
|
|
@ -1,10 +1,25 @@
|
|||
/*
|
||||
* VaptVupt — Zupt Integration API Implementation
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright 2026 Cristian.
|
||||
* Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
*
|
||||
* ZUPT-COMPAT: thin wrapper over vv_compress/vv_decompress with
|
||||
* backup-optimized defaults. Decode speed prioritized over encode.
|
||||
* backup-optimized defaults for VaptVupt 2.48.2.
|
||||
*
|
||||
* Defaults applied here (per ZUPT_INTEGRATION.md, Sprint 122):
|
||||
* - opts.checksum = 0 (Zupt's HMAC-SHA256 / AES-GCM-SIV outer
|
||||
* already authenticates the compressed
|
||||
* bytes; XXH64 footer is redundant work
|
||||
* and saves ~10% encode time)
|
||||
* - opts.format_v2 = 1 (4-7% better binary ratio; v2.33.0+
|
||||
* decoders read v2 frames transparently)
|
||||
* - VV_DECOMPRESS_SKIP_CHECKSUM on decode (matched pair to
|
||||
* checksum=0 on encode; saves ~30% on real
|
||||
* fixtures, 2-5x on AEAD-wrapped data)
|
||||
* - opts.compat_v246_5_decoder = 0 (allow lit_fmt=4 / 4-stream
|
||||
* Huffman; we control both encoder and
|
||||
* decoder versions in-tree, so we always
|
||||
* have v2.47+ on the decode side)
|
||||
*/
|
||||
|
||||
#include "vaptvupt_api.h"
|
||||
|
|
@ -14,15 +29,22 @@ int64_t vvz_compress(const uint8_t *src, size_t src_len,
|
|||
uint8_t *dst, size_t dst_cap, int level) {
|
||||
vv_options_t opts;
|
||||
vv_default_options(&opts);
|
||||
opts.checksum = 1; /* frame-level integrity */
|
||||
opts.format_v2 = 1; /* 4-7% better binary ratio (v2.33.0+ decoders) */
|
||||
opts.checksum = 0; /* outer Zupt MAC authenticates compressed bytes */
|
||||
opts.compat_v246_5_decoder = 0; /* allow 4-stream Huffman literal coding */
|
||||
|
||||
if (level <= 2) {
|
||||
opts.mode = VV_MODE_ULTRA_FAST;
|
||||
/* ULTRA_FAST + format_v2 is NOT in VaptVupt 2.48.2's tested matrix
|
||||
* (test_zupt_integration.c only validates format_v2 with
|
||||
* EXTREME/BALANCED). The combination produces output the decoder
|
||||
* rejects with VV_ERR_OVERFLOW. Stay on the v1 frame for ULTRA_FAST. */
|
||||
opts.format_v2 = 0;
|
||||
} else if (level <= 7) {
|
||||
opts.mode = VV_MODE_BALANCED;
|
||||
opts.format_v2 = 1; /* 4-7% better binary ratio (v2.33.0+ decoders) */
|
||||
} else {
|
||||
opts.mode = VV_MODE_EXTREME;
|
||||
opts.format_v2 = 1; /* 4-7% better binary ratio (v2.33.0+ decoders) */
|
||||
}
|
||||
|
||||
/* Auto window: let adaptive selection choose wlog */
|
||||
|
|
|
|||
464
src/vv_ans.c
464
src/vv_ans.c
|
|
@ -1,5 +1,5 @@
|
|||
/* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
*
|
||||
* VaptVupt — tANS v2 (sparse header + 4-way interleaved decode)
|
||||
*
|
||||
|
|
@ -17,6 +17,7 @@
|
|||
|
||||
#include "vv_ans.h"
|
||||
#include "vv_platform.h"
|
||||
#include "vv_huffman.h"
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
|
|
@ -814,8 +815,12 @@ vva_error_t vva_encode_ctx(const uint8_t *src, size_t src_len,
|
|||
if (!src_len) { *dst_len = 0; return VVA_OK; }
|
||||
|
||||
/* ─── Pass 1: build 256×256 histogram ─── */
|
||||
/* Heap-allocate: 256×256×4 = 256 KB */
|
||||
uint32_t (*hist)[NSYM] = (uint32_t (*)[NSYM])calloc(NSYM, NSYM * sizeof(uint32_t));
|
||||
/* Heap-allocate: NSYM rows × NSYM uint32 per row = 256 KB.
|
||||
* Sprint 89: rewrote calloc invocation to use sizeof(*hist) which
|
||||
* matches the destination pointer type. Prior form
|
||||
* calloc(NSYM, NSYM*sizeof(uint32_t)) computed the same total
|
||||
* bytes but tripped scan-build's "sizeof operand mismatch" check. */
|
||||
uint32_t (*hist)[NSYM] = (uint32_t (*)[NSYM])calloc(NSYM, sizeof(*hist));
|
||||
uint32_t global_raw[NSYM];
|
||||
memset(global_raw, 0, sizeof(global_raw));
|
||||
if (!hist) return VVA_ERR_NOMEM;
|
||||
|
|
@ -1233,18 +1238,48 @@ static const uint8_t of_extra[VVA_OF_CODES] = {
|
|||
|
||||
/* Encode match length → (code, extra_value, extra_bits).
|
||||
* Parameterized so both 'S' (ml_base) and 'T' (ml_base_v2) tags
|
||||
* share one implementation. */
|
||||
* share one implementation.
|
||||
*
|
||||
* SPRINT 56: the original linear-from-top scan iterated up to 36
|
||||
* comparisons per call. Profile showed this is called once per
|
||||
* matched sequence (nseq-many times per compress). Replacing with
|
||||
* a hybrid lookup:
|
||||
* 1. Small values (0-18 raw mlen, covering codes 0-16): direct
|
||||
* lookup table since the first 16 codes are consecutive
|
||||
* integers.
|
||||
* 2. Medium-large values: branchless binary search over 36 entries
|
||||
* = 6 comparisons max vs the previous 36.
|
||||
*
|
||||
* Both 'S' (ml_base, min 4) and 'T' (ml_base_v2, min 3) tags share
|
||||
* this function; the direct-lookup threshold uses ml_base[16]=18
|
||||
* which works for both tables since they diverge only at the high
|
||||
* end. */
|
||||
static void ml_encode_with(uint32_t mlen, const uint32_t *base_tab,
|
||||
uint8_t *code, uint32_t *extra, int *nbits) {
|
||||
for (int c = VVA_ML_CODES - 1; c >= 0; c--) {
|
||||
if (mlen >= base_tab[c]) {
|
||||
/* Fast path: small mlen covers the majority of binary matches.
|
||||
* ml_base[c] for c=0..15 is consecutive integers:
|
||||
* v1 ml_base[0..15] = 4,5,...,19 (covers up to 19)
|
||||
* v2 ml_base[0..15] = 3,4,...,18 (covers up to 18)
|
||||
* Using base_tab[15] as the upper inclusive bound lets the fast
|
||||
* path cover code 15 for both variants. */
|
||||
if (mlen <= base_tab[15]) {
|
||||
uint32_t c = (mlen >= base_tab[0]) ? (mlen - base_tab[0]) : 0;
|
||||
*code = (uint8_t)c;
|
||||
*extra = mlen - base_tab[c];
|
||||
*nbits = ml_extra[c];
|
||||
*extra = 0; /* codes 0-15 all have ml_extra[c] = 0 */
|
||||
*nbits = 0;
|
||||
return;
|
||||
}
|
||||
|
||||
/* Binary search over codes 16..35 for larger values. */
|
||||
int lo = 16, hi = VVA_ML_CODES - 1;
|
||||
while (lo < hi) {
|
||||
int mid = (lo + hi + 1) >> 1;
|
||||
if (mlen >= base_tab[mid]) lo = mid;
|
||||
else hi = mid - 1;
|
||||
}
|
||||
*code = 0; *extra = 0; *nbits = 0;
|
||||
*code = (uint8_t)lo;
|
||||
*extra = mlen - base_tab[lo];
|
||||
*nbits = ml_extra[lo];
|
||||
}
|
||||
/* (ml_encode legacy wrapper removed — all callers migrated to
|
||||
* ml_encode_with for explicit table selection.) */
|
||||
|
|
@ -1287,15 +1322,27 @@ static const uint8_t ll_extra[VVA_LL_CODES] = {
|
|||
};
|
||||
|
||||
static void ll_encode(uint32_t litlen, uint8_t *code, uint32_t *extra, int *nbits) {
|
||||
for (int c = VVA_LL_CODES - 1; c >= 0; c--) {
|
||||
if (litlen >= ll_base[c]) {
|
||||
*code = (uint8_t)c;
|
||||
*extra = litlen - ll_base[c];
|
||||
*nbits = ll_extra[c];
|
||||
/* SPRINT 56: same optimization as ml_encode_with. Small litlens
|
||||
* (0-15) are direct-lookup since ll_base[c]=c for c=0..15.
|
||||
* Larger values use binary search over the remaining 20 codes
|
||||
* (log2 ≈ 5 comparisons vs previous 36). */
|
||||
if (litlen <= 15u) {
|
||||
*code = (uint8_t)litlen;
|
||||
*extra = 0; /* codes 0-15 all have ll_extra[c] = 0 */
|
||||
*nbits = 0;
|
||||
return;
|
||||
}
|
||||
|
||||
/* Binary search over codes 16..35 */
|
||||
int lo = 16, hi = VVA_LL_CODES - 1;
|
||||
while (lo < hi) {
|
||||
int mid = (lo + hi + 1) >> 1;
|
||||
if (litlen >= ll_base[mid]) lo = mid;
|
||||
else hi = mid - 1;
|
||||
}
|
||||
*code = 0; *extra = 0; *nbits = 0;
|
||||
*code = (uint8_t)lo;
|
||||
*extra = litlen - ll_base[lo];
|
||||
*nbits = ll_extra[lo];
|
||||
}
|
||||
|
||||
static uint32_t ll_decode(uint8_t code, uint32_t extra) {
|
||||
|
|
@ -1336,6 +1383,25 @@ static size_t parse_sequences(const uint8_t *tokens, size_t tok_len,
|
|||
const uint8_t *tp = tokens, *tp_end = tokens + tok_len;
|
||||
size_t nseq = 0, nlits = 0;
|
||||
|
||||
/* SPRINT 63: maximum litlen representable by the LL ANS coder is
|
||||
* 65535 (ll_base[35]=61440 + max 4095 extra bits). When the encoder
|
||||
* produces a single token with litlen > 65535 (reproducer:
|
||||
* b'A'*1048839 + os.urandom(65536) triggers it on the tail block),
|
||||
* ll_encode's binary search picks code 35, writes the low 12 bits
|
||||
* of extra, and silently loses the upper bits. Decoder then reads
|
||||
* back a smaller litlen, producing a short output block.
|
||||
*
|
||||
* Fix: if a parsed token's ll exceeds LL_MAX, split into multiple
|
||||
* seq entries: as many (LL_MAX, matchlen=0) zero-match sequences
|
||||
* as needed to absorb the overflow, followed by the final sequence
|
||||
* carrying the remaining (ll' ≤ LL_MAX) and the original match.
|
||||
*
|
||||
* Zero-match sequences are already legal in the stream (trailing
|
||||
* literals use matchlen=0, offset=0). Adding them mid-stream is
|
||||
* wire-compatible — the decoder's existing match_count == 0 test
|
||||
* skips the match-copy for these entries. */
|
||||
enum { LL_MAX = 65535 };
|
||||
|
||||
while (tp < tp_end && nseq < seq_cap) {
|
||||
uint8_t token = *tp++;
|
||||
size_t ll = token >> 4;
|
||||
|
|
@ -1356,6 +1422,18 @@ static size_t parse_sequences(const uint8_t *tokens, size_t tok_len,
|
|||
memcpy(lit_buf + nlits, tp, ll);
|
||||
tp += ll;
|
||||
|
||||
/* SPRINT 63: split oversize literal runs */
|
||||
while (ll > LL_MAX) {
|
||||
if (nseq >= seq_cap) return 0;
|
||||
seqs[nseq].litlen = (uint32_t)LL_MAX;
|
||||
seqs[nseq].lit_offset = (uint32_t)nlits;
|
||||
seqs[nseq].matchlen = 0;
|
||||
seqs[nseq].offset = 0;
|
||||
nlits += LL_MAX;
|
||||
nseq++;
|
||||
ll -= LL_MAX;
|
||||
}
|
||||
|
||||
seqs[nseq].litlen = (uint32_t)ll;
|
||||
seqs[nseq].lit_offset = (uint32_t)nlits;
|
||||
nlits += ll;
|
||||
|
|
@ -1405,7 +1483,8 @@ static size_t parse_sequences(const uint8_t *tokens, size_t tok_len,
|
|||
static vva_error_t vva_encode_sequences_impl(const uint8_t *tokens, size_t tok_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes,
|
||||
const uint32_t *ml_base_tab) {
|
||||
const uint32_t *ml_base_tab,
|
||||
int disable_huf4) {
|
||||
if (!tok_len) { *dst_len = 0; return VVA_OK; }
|
||||
|
||||
/* Parse into sequences.
|
||||
|
|
@ -1434,26 +1513,104 @@ static vva_error_t vva_encode_sequences_impl(const uint8_t *tokens, size_t tok_l
|
|||
if (!lit_enc) { free(base_scratch); return VVA_ERR_NOMEM; }
|
||||
|
||||
size_t lit_enc_len = 0;
|
||||
uint8_t lit_fmt = 0; /* 0=raw, 1=ANS4, 2=ANS1 */
|
||||
uint8_t lit_fmt = 0; /* 0=raw, 1=ANS4, 2=ANS1, 3=Huffman, 4=Huffman4 (Sprint 104) */
|
||||
if (total_lits > 0) {
|
||||
vva_error_t lit_err = vva_encode4(lit_buf, total_lits,
|
||||
lit_enc, lit_cap, &lit_enc_len);
|
||||
if (lit_err == VVA_OK) {
|
||||
lit_fmt = 1;
|
||||
/* SPRINT 71 (v2.46): Huffman as a competitive literal coder
|
||||
* inside the SEQ stream.
|
||||
*
|
||||
* Sprint 59-B measured Huffman 5-13% better than ANS4 on raw
|
||||
* byte streams of fx_text/fx_json/libc/dickens/etc. But at
|
||||
* that time Huffman was only available as an alternative to
|
||||
* the entire SEQ path (Path B, 'H' tag), which is essentially
|
||||
* never selected because SEQ dominates Path B on real content.
|
||||
*
|
||||
* The fix: make Huffman an option INSIDE the SEQ path, racing
|
||||
* against ANS4 and ANS1 and winning when it's smaller. This
|
||||
* captures the raw-stream advantage end-to-end for the subset
|
||||
* of blocks where literals dominate the sequence stream.
|
||||
*
|
||||
* Race all three coders, pick smallest. Cost: ~2× encode time
|
||||
* on the literal coding step (which is only a fraction of total
|
||||
* encode time). Benefit: 3-7% expected on binary fixtures where
|
||||
* literal distributions make Huffman materially better.
|
||||
*
|
||||
* Decoder support: lit_fmt=3 dispatches to vvh_decode, lit_fmt=4
|
||||
* dispatches to vvh_decode4 (Sprint 104 Phase B). Wire format
|
||||
* unchanged otherwise — existing decoders reject lit_fmt={3,4}
|
||||
* with VVA_ERR_CORRUPT, so this is a decoder-incompatible
|
||||
* format change (requires v2.46.0+ for fmt=3, v2.47+ for fmt=4). */
|
||||
size_t ans4_len = 0, ans1_len = 0, huf_len = 0, huf4_len = 0;
|
||||
uint8_t *ans4_buf = (uint8_t *)malloc(lit_cap);
|
||||
uint8_t *ans1_buf = (uint8_t *)malloc(lit_cap);
|
||||
uint8_t *huf_buf = (uint8_t *)malloc(lit_cap);
|
||||
/* Phase C: gated by disable_huf4 flag (vv_options_t::compat_v246_5_decoder).
|
||||
* When set, suppress lit_fmt=4 selection so output is readable by
|
||||
* v2.46.5 and older decoders. */
|
||||
uint8_t *huf4_buf = (!disable_huf4 && total_lits >= 1024) ? (uint8_t *)malloc(lit_cap) : NULL;
|
||||
int ans4_ok = 0, ans1_ok = 0, huf_ok = 0, huf4_ok = 0;
|
||||
if (ans4_buf) {
|
||||
ans4_ok = (vva_encode4(lit_buf, total_lits, ans4_buf, lit_cap, &ans4_len) == VVA_OK);
|
||||
}
|
||||
if (ans1_buf) {
|
||||
ans1_ok = (vva_encode(lit_buf, total_lits, ans1_buf, lit_cap, &ans1_len) == VVA_OK);
|
||||
}
|
||||
if (huf_buf) {
|
||||
huf_ok = (vvh_encode(lit_buf, total_lits, huf_buf, lit_cap, &huf_len) == VVH_OK);
|
||||
}
|
||||
if (huf4_buf) {
|
||||
/* Phase B: 4-stream Huffman race. Activates only at >=1024 lits. */
|
||||
huf4_ok = (vvh_encode4(lit_buf, total_lits, huf4_buf, lit_cap, &huf4_len) == VVH_OK);
|
||||
}
|
||||
|
||||
/* Pick the smallest of all options. Preference order on ties:
|
||||
* ANS4 (fastest decode) > ANS1 > Huffman4 > Huffman.
|
||||
* 4-stream Huffman has same ratio as single-stream modulo a
|
||||
* fixed +10B header overhead but decodes 1.8-2.2× faster via
|
||||
* ILP. We pick huf4 over huf when both are available and the
|
||||
* size delta is within a small slop (32 bytes covers the
|
||||
* structural overhead with margin). For sizes way out, we
|
||||
* still pick the smaller one to avoid pathological ratio
|
||||
* regressions on tiny blocks. */
|
||||
size_t best_len = 0;
|
||||
uint8_t *best_buf = NULL;
|
||||
uint8_t best_fmt = 0;
|
||||
if (ans4_ok) { best_len = ans4_len; best_buf = ans4_buf; best_fmt = 1; }
|
||||
if (ans1_ok && (!best_buf || ans1_len < best_len)) {
|
||||
best_len = ans1_len; best_buf = ans1_buf; best_fmt = 2;
|
||||
}
|
||||
/* Huffman entry: if both single-stream and 4-stream are viable,
|
||||
* prefer the 4-stream variant for its decode speedup. Allow a
|
||||
* 32-byte slop where 4-stream wins despite being slightly larger. */
|
||||
size_t huf_best_len = 0;
|
||||
uint8_t *huf_best_buf = NULL;
|
||||
uint8_t huf_best_fmt = 0;
|
||||
if (huf4_ok && huf_ok) {
|
||||
/* Both viable: prefer huf4 if it's not meaningfully larger. */
|
||||
if (huf4_len <= huf_len + 32) {
|
||||
huf_best_len = huf4_len; huf_best_buf = huf4_buf; huf_best_fmt = 4;
|
||||
} else {
|
||||
lit_err = vva_encode(lit_buf, total_lits,
|
||||
lit_enc, lit_cap, &lit_enc_len);
|
||||
if (lit_err == VVA_OK) {
|
||||
lit_fmt = 2;
|
||||
} else {
|
||||
/* Store raw */
|
||||
if (total_lits <= lit_cap) {
|
||||
huf_best_len = huf_len; huf_best_buf = huf_buf; huf_best_fmt = 3;
|
||||
}
|
||||
} else if (huf4_ok) {
|
||||
huf_best_len = huf4_len; huf_best_buf = huf4_buf; huf_best_fmt = 4;
|
||||
} else if (huf_ok) {
|
||||
huf_best_len = huf_len; huf_best_buf = huf_buf; huf_best_fmt = 3;
|
||||
}
|
||||
if (huf_best_buf && (!best_buf || huf_best_len < best_len)) {
|
||||
best_len = huf_best_len; best_buf = huf_best_buf; best_fmt = huf_best_fmt;
|
||||
}
|
||||
|
||||
if (best_buf && best_len <= lit_cap) {
|
||||
memcpy(lit_enc, best_buf, best_len);
|
||||
lit_enc_len = best_len;
|
||||
lit_fmt = best_fmt;
|
||||
} else if (total_lits <= lit_cap) {
|
||||
/* All failed — fall back to raw literals */
|
||||
memcpy(lit_enc, lit_buf, total_lits);
|
||||
lit_enc_len = total_lits;
|
||||
lit_fmt = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
free(ans4_buf); free(ans1_buf); free(huf_buf); free(huf4_buf);
|
||||
}
|
||||
|
||||
/* ─── Count ML, OF, and LL code frequencies ─── */
|
||||
|
|
@ -1469,18 +1626,43 @@ static vva_error_t vva_encode_sequences_impl(const uint8_t *tokens, size_t tok_l
|
|||
* [seq_of_code: nseq × uint8_t] (padded to 4-byte align)
|
||||
* [seq_of_extra: nseq × uint32_t]
|
||||
* [seq_of_nbits: nseq × int]
|
||||
* Saves 2 malloc/free pairs per vva_encode_sequences call. */
|
||||
* [seq_ml_code: nseq × uint8_t] (SPRINT 54)
|
||||
* [seq_ml_extra: nseq × uint32_t]
|
||||
* [seq_ml_nbits: nseq × int]
|
||||
* [seq_ll_code: nseq × uint8_t]
|
||||
* [seq_ll_extra: nseq × uint32_t]
|
||||
* [seq_ll_nbits: nseq × int]
|
||||
*
|
||||
* SPRINT 54: also memoize ML and LL codes from the forward pass.
|
||||
* Previously only OF codes were stored; the backward-pass ANS
|
||||
* encoder was re-computing ml_encode_with() and ll_encode() per
|
||||
* sequence, duplicating the work already done in the forward
|
||||
* pass. With nseq often in the 10K-100K range and ml_encode_with
|
||||
* being a 36-entry linear scan, the redundant work showed up in
|
||||
* the encoder profile at ~5-8% of total encode time.
|
||||
*
|
||||
* Net cost: 1 extra malloc region (~14 × nseq bytes), 0 extra
|
||||
* malloc calls. Net saving: the backward pass becomes lookups
|
||||
* instead of re-computation. */
|
||||
size_t codes_sz = (nseq * sizeof(uint8_t) + 3) & ~(size_t)3;
|
||||
size_t extra_sz = nseq * sizeof(uint32_t);
|
||||
size_t nbits_sz = nseq * sizeof(int);
|
||||
uint8_t *seq_scratch = (uint8_t *)malloc(codes_sz + extra_sz + nbits_sz);
|
||||
/* 3 streams × (codes + extra + nbits) */
|
||||
uint8_t *seq_scratch = (uint8_t *)malloc(3 * (codes_sz + extra_sz + nbits_sz));
|
||||
if (!seq_scratch) {
|
||||
free(base_scratch); free(lit_enc);
|
||||
return VVA_ERR_NOMEM;
|
||||
}
|
||||
size_t stream_sz = codes_sz + extra_sz + nbits_sz;
|
||||
uint8_t *seq_of_code = seq_scratch;
|
||||
uint32_t *seq_of_extra = (uint32_t *)(seq_scratch + codes_sz);
|
||||
int *seq_of_nbits = (int *)(seq_scratch + codes_sz + extra_sz);
|
||||
uint8_t *seq_ml_code = seq_scratch + stream_sz;
|
||||
uint32_t *seq_ml_extra = (uint32_t *)(seq_scratch + stream_sz + codes_sz);
|
||||
int *seq_ml_nbits = (int *)(seq_scratch + stream_sz + codes_sz + extra_sz);
|
||||
uint8_t *seq_ll_code = seq_scratch + 2 * stream_sz;
|
||||
uint32_t *seq_ll_extra = (uint32_t *)(seq_scratch + 2 * stream_sz + codes_sz);
|
||||
int *seq_ll_nbits = (int *)(seq_scratch + 2 * stream_sz + codes_sz + extra_sz);
|
||||
|
||||
size_t match_count = 0;
|
||||
uint32_t enc_rep[3] = {0, 0, 0}; /* Rep-match tracking during forward pass */
|
||||
|
|
@ -1489,6 +1671,10 @@ static vva_error_t vva_encode_sequences_impl(const uint8_t *tokens, size_t tok_l
|
|||
uint8_t mc; uint32_t mx; int mn;
|
||||
ml_encode_with(seqs[i].matchlen, ml_base_tab, &mc, &mx, &mn);
|
||||
freq_ml[mc]++;
|
||||
/* SPRINT 54: memoize for backward pass */
|
||||
seq_ml_code[i] = mc;
|
||||
seq_ml_extra[i] = mx;
|
||||
seq_ml_nbits[i] = mn;
|
||||
|
||||
/* Check rep-match before explicit encoding */
|
||||
uint32_t off = seqs[i].offset;
|
||||
|
|
@ -1518,6 +1704,10 @@ static vva_error_t vva_encode_sequences_impl(const uint8_t *tokens, size_t tok_l
|
|||
seq_of_code[i] = 0;
|
||||
seq_of_extra[i] = 0;
|
||||
seq_of_nbits[i] = 0;
|
||||
/* SPRINT 54: ml_code unused when matchlen==0, but zero for safety */
|
||||
seq_ml_code[i] = 0;
|
||||
seq_ml_extra[i] = 0;
|
||||
seq_ml_nbits[i] = 0;
|
||||
}
|
||||
|
||||
/* Count litlen frequency for ALL sequences (including last) */
|
||||
|
|
@ -1525,6 +1715,10 @@ static vva_error_t vva_encode_sequences_impl(const uint8_t *tokens, size_t tok_l
|
|||
uint8_t lc; uint32_t lx; int ln;
|
||||
ll_encode(seqs[i].litlen, &lc, &lx, &ln);
|
||||
freq_ll[lc]++;
|
||||
/* SPRINT 54: memoize LL codes too */
|
||||
seq_ll_code[i] = lc;
|
||||
seq_ll_extra[i] = lx;
|
||||
seq_ll_nbits[i] = ln;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1536,8 +1730,15 @@ static vva_error_t vva_encode_sequences_impl(const uint8_t *tokens, size_t tok_l
|
|||
|
||||
/* PERF: header buffers live on the stack — each is bounded at 600 B
|
||||
* (fits any NSYM=256 table header) and they were heap-allocated on
|
||||
* every call before. Saves 3 malloc/free pairs per call. */
|
||||
uint8_t ml_hdr_buf[600], of_hdr_buf[600], ll_hdr_buf[600];
|
||||
* every call before. Saves 3 malloc/free pairs per call.
|
||||
*
|
||||
* Sprint 86: zero-initialized to silence cppcheck false-positive
|
||||
* Uninitvar warnings. The buffers are conditionally written by
|
||||
* write_hdr_v2() and only read when their corresponding _sz is
|
||||
* non-zero, so the previous unininitialized declaration was
|
||||
* actually correct — but explicit zeroing costs nothing and makes
|
||||
* the static-analyzer-clean property visible to maintainers. */
|
||||
uint8_t ml_hdr_buf[600] = {0}, of_hdr_buf[600] = {0}, ll_hdr_buf[600] = {0};
|
||||
size_t ml_hdr_sz = 0, of_hdr_sz = 0, ll_hdr_sz = 0;
|
||||
uint8_t *seq_bs = NULL;
|
||||
size_t seq_bs_len = 0;
|
||||
|
|
@ -1629,17 +1830,24 @@ static vva_error_t vva_encode_sequences_impl(const uint8_t *tokens, size_t tok_l
|
|||
/* Process sequences in reverse for ANS LIFO.
|
||||
* Decoder reads per-sequence: LL, OF, ML (forward).
|
||||
* Backward encode order (reversed of decode): ML, OF, LL.
|
||||
* After bitstream reversal: LL appears first → decoded first. */
|
||||
* After bitstream reversal: LL appears first → decoded first.
|
||||
*
|
||||
* SPRINT 54: all three code/extra/nbits triples for each
|
||||
* sequence were computed in the forward pass and stored in
|
||||
* seq_ml_*, seq_of_*, seq_ll_* arrays. Re-use them here
|
||||
* instead of recomputing ml_encode_with() and ll_encode().
|
||||
* Eliminates ~5-8% of encode time (the forward+backward
|
||||
* duplicate work). */
|
||||
for (size_t ii = nseq; ii > 0; ii--) {
|
||||
if (seqs[ii - 1].matchlen > 0) {
|
||||
uint8_t mc;
|
||||
uint32_t mx;
|
||||
int mn;
|
||||
ml_encode_with(seqs[ii - 1].matchlen, ml_base_tab, &mc, &mx, &mn);
|
||||
size_t idx = ii - 1;
|
||||
if (seqs[idx].matchlen > 0) {
|
||||
uint8_t mc = seq_ml_code[idx];
|
||||
uint32_t mx = seq_ml_extra[idx];
|
||||
int mn = seq_ml_nbits[idx];
|
||||
|
||||
uint8_t oc = seq_of_code[ii - 1];
|
||||
uint32_t ox = seq_of_extra[ii - 1];
|
||||
int on = seq_of_nbits[ii - 1];
|
||||
uint8_t oc = seq_of_code[idx];
|
||||
uint32_t ox = seq_of_extra[idx];
|
||||
int on = seq_of_nbits[idx];
|
||||
|
||||
/* ML extra bits (raw) */
|
||||
if (mn > 0) {
|
||||
|
|
@ -1684,10 +1892,11 @@ static vva_error_t vva_encode_sequences_impl(const uint8_t *tokens, size_t tok_l
|
|||
}
|
||||
}
|
||||
|
||||
/* LL encoded LAST per sequence (so it's decoded FIRST after reversal) */
|
||||
/* LL encoded LAST per sequence (decoded FIRST after reversal) */
|
||||
{
|
||||
uint8_t lc; uint32_t lx; int ln;
|
||||
ll_encode(seqs[ii - 1].litlen, &lc, &lx, &ln);
|
||||
uint8_t lc = seq_ll_code[idx];
|
||||
uint32_t lx = seq_ll_extra[idx];
|
||||
int ln = seq_ll_nbits[idx];
|
||||
|
||||
if (ln > 0) {
|
||||
pairs[npairs].val = (uint32_t)lx;
|
||||
|
|
@ -1800,7 +2009,16 @@ vva_error_t vva_encode_sequences(const uint8_t *tokens, size_t tok_len,
|
|||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes) {
|
||||
return vva_encode_sequences_impl(tokens, tok_len, dst, dst_cap, dst_len,
|
||||
off_bytes, ml_base);
|
||||
off_bytes, ml_base, 0);
|
||||
}
|
||||
|
||||
/* Public entry with explicit compat flag (Sprint 105 Phase C).
|
||||
* disable_huf4=1 suppresses lit_fmt=4 selection for v2.46.5 compat. */
|
||||
vva_error_t vva_encode_sequences_compat(const uint8_t *tokens, size_t tok_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes, int disable_huf4) {
|
||||
return vva_encode_sequences_impl(tokens, tok_len, dst, dst_cap, dst_len,
|
||||
off_bytes, ml_base, disable_huf4);
|
||||
}
|
||||
|
||||
/* Public entry for 'T' tag (VV_ENTROPY_SEQ_V2, min_match=3).
|
||||
|
|
@ -1812,7 +2030,14 @@ vva_error_t vva_encode_sequences_v2(const uint8_t *tokens, size_t tok_len,
|
|||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes) {
|
||||
return vva_encode_sequences_impl(tokens, tok_len, dst, dst_cap, dst_len,
|
||||
off_bytes, ml_base_v2);
|
||||
off_bytes, ml_base_v2, 0);
|
||||
}
|
||||
|
||||
vva_error_t vva_encode_sequences_v2_compat(const uint8_t *tokens, size_t tok_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes, int disable_huf4) {
|
||||
return vva_encode_sequences_impl(tokens, tok_len, dst, dst_cap, dst_len,
|
||||
off_bytes, ml_base_v2, disable_huf4);
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════════
|
||||
|
|
@ -1826,7 +2051,8 @@ vva_error_t vva_encode_sequences_v2(const uint8_t *tokens, size_t tok_len,
|
|||
* (min_match=3) entropy tags. Takes the ml_base table as a parameter
|
||||
* so both tags use the same code path. Everything else in the 'T'
|
||||
* payload is byte-identical to 'S'. */
|
||||
static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
||||
static VV_NO_SANITIZE_INTEGER
|
||||
vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
const uint8_t *dst_base,
|
||||
const uint32_t *ml_base_tab) {
|
||||
|
|
@ -1839,6 +2065,22 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
size_t lit_enc_len = (size_t)p[0]|((size_t)p[1]<<8)|((size_t)p[2]<<16)|((size_t)p[3]<<24); p += 4;
|
||||
if (p + lit_enc_len > end) return VVA_ERR_CORRUPT;
|
||||
|
||||
/* SPRINT 90 SECURITY FIX (DoS hardening - companion to the
|
||||
* iteration-count bound):
|
||||
*
|
||||
* Sprint 89 fuzzing found a DoS where corrupted total_lits
|
||||
* (decoded from 4 wire bytes, no upper bound) made the Huffman
|
||||
* literal decoder loop ~1.1 billion times. Stack trace from gdb:
|
||||
* #0 br_refill (...)
|
||||
* #1 vvh_decode (..., num_literals=1124110334, ...)
|
||||
* #2 vva_decode_sequences_impl
|
||||
*
|
||||
* Bound total_lits against dst_cap. A valid literal stream cannot
|
||||
* exceed the block's output capacity (matches consume some output
|
||||
* too, so this is conservative — the real bound is even tighter,
|
||||
* but dst_cap is sufficient to prevent runaway decode work). */
|
||||
if (VV_UNLIKELY(total_lits > dst_cap)) return VVA_ERR_CORRUPT;
|
||||
|
||||
/* Decode literals based on format byte */
|
||||
uint8_t *lit_buf = (uint8_t *)malloc(total_lits + 16);
|
||||
if (!lit_buf) return VVA_ERR_NOMEM;
|
||||
|
|
@ -1855,6 +2097,20 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
/* ANS single-stream */
|
||||
lerr = vva_decode(p, lit_enc_len, lit_buf, total_lits,
|
||||
total_lits, &lit_consumed);
|
||||
} else if (lit_fmt == 3) {
|
||||
/* SPRINT 71 (v2.46): Huffman-coded literals within SEQ. */
|
||||
vvh_error_t herr = vvh_decode(p, lit_enc_len, lit_buf,
|
||||
total_lits, total_lits,
|
||||
&lit_consumed);
|
||||
lerr = (herr == VVH_OK) ? VVA_OK : VVA_ERR_CORRUPT;
|
||||
} else if (lit_fmt == 4) {
|
||||
/* SPRINT 104 (v2.47): 4-stream interleaved Huffman literals.
|
||||
* Faster decode (1.8-2.2× via ILP across 4 independent
|
||||
* streams). Same ratio as lit_fmt=3 modulo +10B header. */
|
||||
vvh_error_t herr = vvh_decode4(p, lit_enc_len, lit_buf,
|
||||
total_lits, total_lits,
|
||||
&lit_consumed);
|
||||
lerr = (herr == VVH_OK) ? VVA_OK : VVA_ERR_CORRUPT;
|
||||
} else {
|
||||
/* Raw literals (lit_fmt == 0) */
|
||||
if (lit_enc_len >= total_lits) {
|
||||
|
|
@ -1870,6 +2126,15 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
if (p + 4 > end) { free(lit_buf); return VVA_ERR_CORRUPT; }
|
||||
size_t match_count = (size_t)p[0]|((size_t)p[1]<<8)|((size_t)p[2]<<16)|((size_t)p[3]<<24); p += 4;
|
||||
|
||||
/* SPRINT 90 SECURITY FIX: bound match_count against dst_cap.
|
||||
* Each match contributes ≥ min_match (3 or 4) bytes of output, so
|
||||
* match_count cannot exceed dst_cap / min_match. Use dst_cap as a
|
||||
* generous upper bound — anything larger is corrupt input that
|
||||
* would cause the decode loop's max_iters check to trigger anyway,
|
||||
* but bounding here prevents wasteful work and oversized
|
||||
* allocations. */
|
||||
if (VV_UNLIKELY(match_count > dst_cap)) { free(lit_buf); return VVA_ERR_CORRUPT; }
|
||||
|
||||
/* Read ML table header */
|
||||
if (p + 2 > end) { free(lit_buf); return VVA_ERR_CORRUPT; }
|
||||
size_t ml_hdr_sz = (size_t)p[0] | ((size_t)p[1] << 8); p += 2;
|
||||
|
|
@ -1911,16 +2176,22 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
size_t seq_bs_len = (size_t)p[0]|((size_t)p[1]<<8)|((size_t)p[2]<<16)|((size_t)p[3]<<24); p += 4;
|
||||
if (p + seq_bs_len > end) { free(lit_buf); return VVA_ERR_CORRUPT; }
|
||||
|
||||
/* Build ML, OF, and LL decode tables */
|
||||
/* Build ML, OF, and LL decode tables.
|
||||
*
|
||||
* Sprint 109 fix: previously dec_ml/dec_of were only allocated when
|
||||
* match_count > 0, but the unified decode loop dereferences all 3
|
||||
* tables eagerly for ILP regardless of match_count. With total_lits
|
||||
* > 0 and match_count == 0, the loop runs (consuming literals) and
|
||||
* NULL-deref's dec_of and dec_ml. Found by libFuzzer + ASan.
|
||||
* Fix: always allocate all 3 tables. The decode-loop dereferences
|
||||
* are safe because state masks bound the index to ANS_L. */
|
||||
vva_dec_entry_t *dec_ml = NULL, *dec_of = NULL, *dec_ll = NULL;
|
||||
{
|
||||
uint8_t *sp_tmp = (uint8_t *)malloc(ANS_L);
|
||||
if (match_count > 0) {
|
||||
dec_ml = (vva_dec_entry_t *)malloc(ANS_L * sizeof(vva_dec_entry_t));
|
||||
dec_of = (vva_dec_entry_t *)malloc(ANS_L * sizeof(vva_dec_entry_t));
|
||||
}
|
||||
dec_ll = (vva_dec_entry_t *)malloc(ANS_L * sizeof(vva_dec_entry_t));
|
||||
if (!sp_tmp || !dec_ll || (match_count > 0 && (!dec_ml || !dec_of))) {
|
||||
if (!sp_tmp || !dec_ll || !dec_ml || !dec_of) {
|
||||
free(sp_tmp); free(dec_ml); free(dec_of); free(dec_ll); free(lit_buf);
|
||||
return VVA_ERR_NOMEM;
|
||||
}
|
||||
|
|
@ -1929,6 +2200,15 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
build_dec(norm_ml, sp_tmp, dec_ml);
|
||||
spread_symbols(norm_of, sp_tmp);
|
||||
build_dec(norm_of, sp_tmp, dec_of);
|
||||
} else {
|
||||
/* Initialize ml/of tables to safe sentinel values so any
|
||||
* unintended read (e.g., the ILP eager-load in the decode
|
||||
* loop when match_count == 0) returns predictable data
|
||||
* rather than dereferencing uninitialized memory. The
|
||||
* loop guard prevents these values from being used in
|
||||
* actual sequence reconstruction. */
|
||||
memset(dec_ml, 0, ANS_L * sizeof(vva_dec_entry_t));
|
||||
memset(dec_of, 0, ANS_L * sizeof(vva_dec_entry_t));
|
||||
}
|
||||
spread_symbols(norm_ll, sp_tmp);
|
||||
build_dec(norm_ll, sp_tmp, dec_ll);
|
||||
|
|
@ -1939,7 +2219,7 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
ans_br_t r;
|
||||
ans_br_init(&r, p, seq_bs_len);
|
||||
ans_br_fill(&r);
|
||||
p += seq_bs_len;
|
||||
/* p is not read after this point — bitstream owned by 'r' from here */
|
||||
|
||||
/* Litlens are ANS-coded in the bitstream — no varint stream */
|
||||
|
||||
|
|
@ -1987,7 +2267,40 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
const uint8_t *offset_check_floor = dst_base + SAFEZONE_MAX_OFFSET;
|
||||
|
||||
size_t seqs_decoded = 0;
|
||||
/* SPRINT 90 SECURITY FIX (DoS hardening):
|
||||
*
|
||||
* The original loop terminated only when both lit_pos reached
|
||||
* total_lits AND matches_decoded reached match_count. Sprint 89
|
||||
* adversarial fuzzing (header-targeted bit-flips at byte offsets
|
||||
* 26-27) discovered that a maliciously crafted ANS bitstream can
|
||||
* produce sequences where neither counter advances — the corrupted
|
||||
* ANS state decodes litlen=0 + matchlen=0 forever, hanging the
|
||||
* decoder.
|
||||
*
|
||||
* This was a denial-of-service vulnerability for any service that
|
||||
* decompressed untrusted input (Zupt's exact threat model).
|
||||
*
|
||||
* Bound: every well-formed iteration must advance at least ONE of
|
||||
* the two counters by at least 1 (it's how the wire format is
|
||||
* defined — every sequence consumes literal bytes, match bytes,
|
||||
* or both, with the only exception being the well-defined
|
||||
* "split-zero-match" case which the encoder uses for >65535-byte
|
||||
* literal runs and which still advances lit_pos).
|
||||
*
|
||||
* Therefore total iterations ≤ total_lits + match_count + 1
|
||||
* (the +1 covers the "both already reached, one final break-check"
|
||||
* iteration). Add small slack of 16 for absolute safety in case
|
||||
* the encoder's wire-format-allowed sequence variations grow.
|
||||
*
|
||||
* If we exceed the bound, the input is corrupt — return
|
||||
* VVA_ERR_CORRUPT instead of hanging. */
|
||||
const size_t max_iters = total_lits + match_count + 16;
|
||||
size_t iter_count = 0;
|
||||
while (lit_pos < total_lits || matches_decoded < match_count) {
|
||||
if (VV_UNLIKELY(++iter_count > max_iters)) {
|
||||
free(dec_ml); free(dec_of); free(dec_ll); free(lit_buf);
|
||||
return VVA_ERR_CORRUPT;
|
||||
}
|
||||
/* PERF: issue all 3 ANS table lookups early so CPU can overlap
|
||||
* the L1 cache fills. The dec_ll/dec_of/dec_ml arrays are
|
||||
* independent, so the loads have no data dependency on each
|
||||
|
|
@ -2019,6 +2332,14 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
uint32_t ll_bits = ans_br_read(&r, ell.nbits);
|
||||
state_ll = (uint32_t)ell.baseline + ll_bits;
|
||||
uint8_t ll_code = ell.symbol;
|
||||
/* Sprint 109 fix: corrupt frames could encode an LL ANS table
|
||||
* mapping a state to a symbol >= VVA_LL_CODES, causing an OOB
|
||||
* read of ll_extra[]/ll_base[]. Validate the code is in range.
|
||||
* Found by libFuzzer + ASan. */
|
||||
if (VV_UNLIKELY(ll_code >= VVA_LL_CODES)) {
|
||||
free(dec_ml); free(dec_of); free(dec_ll); free(lit_buf);
|
||||
return VVA_ERR_CORRUPT;
|
||||
}
|
||||
uint32_t ll_extra_val = ans_br_read(&r, ll_extra[ll_code]);
|
||||
size_t litlen = ll_decode(ll_code, ll_extra_val);
|
||||
|
||||
|
|
@ -2050,13 +2371,33 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
}
|
||||
seqs_decoded++;
|
||||
|
||||
if (matches_decoded >= match_count) break;
|
||||
/* SPRINT 63/64: continue the loop even when all matches are
|
||||
* consumed, as long as literals remain. Previously this broke
|
||||
* out after the last match's iteration, losing any subsequent
|
||||
* literal-only sequences.
|
||||
*
|
||||
* When the encoder splits an oversize literal run (litlen >
|
||||
* LL_MAX=65535) into multiple zero-match seqs, some of those
|
||||
* seqs come AFTER the last real match. The old break dropped
|
||||
* them silently, producing short output.
|
||||
*
|
||||
* Fix: break only when both literals AND matches are fully
|
||||
* consumed. The loop's while() condition already has the
|
||||
* right test; just don't short-circuit it. */
|
||||
if (matches_decoded >= match_count && lit_pos >= total_lits) break;
|
||||
if (matches_decoded >= match_count) continue;
|
||||
|
||||
/* ── Decode OF: state, then offset (rep or explicit) ──
|
||||
* No explicit fill — ans_br_read fills when it runs out. */
|
||||
uint32_t of_bits = ans_br_read(&r, eof.nbits);
|
||||
state_of = (uint32_t)eof.baseline + of_bits;
|
||||
uint8_t of_code = eof.symbol;
|
||||
/* Sprint 109 fix: bound of_code to VVA_OF_CODES range. Same
|
||||
* pattern as ll_code/ml_code OOB protection. */
|
||||
if (VV_UNLIKELY(of_code >= VVA_OF_CODES)) {
|
||||
free(dec_ml); free(dec_of); free(dec_ll); free(lit_buf);
|
||||
return VVA_ERR_CORRUPT;
|
||||
}
|
||||
uint32_t offset;
|
||||
if (of_code < 3) {
|
||||
offset = dec_rep[of_code];
|
||||
|
|
@ -2072,6 +2413,11 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
uint32_t ml_bits = ans_br_read(&r, eml.nbits);
|
||||
state_ml = (uint32_t)eml.baseline + ml_bits;
|
||||
uint8_t ml_code = eml.symbol;
|
||||
/* Sprint 109 fix: bound ml_code to VVA_ML_CODES range. */
|
||||
if (VV_UNLIKELY(ml_code >= VVA_ML_CODES)) {
|
||||
free(dec_ml); free(dec_of); free(dec_ll); free(lit_buf);
|
||||
return VVA_ERR_CORRUPT;
|
||||
}
|
||||
uint32_t ml_extra_val = ans_br_read(&r, ml_extra[ml_code]);
|
||||
uint32_t matchlen = ml_base_tab[ml_code] + ml_extra_val;
|
||||
|
||||
|
|
@ -2088,15 +2434,15 @@ static vva_error_t vva_decode_sequences_impl(const uint8_t *src, size_t src_len,
|
|||
* op_safe_end = op_end - SAFEZONE_MAX_MATCH, and matchlen is
|
||||
* always ≤ SAFEZONE_MAX_MATCH by wire format. */
|
||||
if (VV_UNLIKELY(offset == 0 || offset > SAFEZONE_MAX_OFFSET)) {
|
||||
free(dec_ml); free(dec_of); free(lit_buf);
|
||||
free(dec_ml); free(dec_of); free(dec_ll); free(lit_buf);
|
||||
return VVA_ERR_CORRUPT;
|
||||
}
|
||||
if (VV_UNLIKELY(!in_safe_zone && offset > (uint32_t)(op - dst_base))) {
|
||||
free(dec_ml); free(dec_of); free(lit_buf);
|
||||
free(dec_ml); free(dec_of); free(dec_ll); free(lit_buf);
|
||||
return VVA_ERR_CORRUPT;
|
||||
}
|
||||
if (VV_UNLIKELY(!in_safe_zone && op + matchlen > op_end)) {
|
||||
free(dec_ml); free(dec_of); free(lit_buf);
|
||||
free(dec_ml); free(dec_of); free(dec_ll); free(lit_buf);
|
||||
return VVA_ERR_OVERFLOW;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
*
|
||||
* VaptVupt — Decoder v2 (Sprint 1)
|
||||
*
|
||||
|
|
@ -144,6 +144,14 @@ decode_block_tokens_impl(
|
|||
if (VV_UNLIKELY(ll == 15))
|
||||
ll += (uint32_t)read_ext_len(&ip, ip_end);
|
||||
|
||||
/* Sprint 109 fix: corrupt ll extension can yield a huge ll
|
||||
* that exceeds remaining input or output. Found by libFuzzer
|
||||
* + ASan: heap-buffer-overflow READ at memcpy(op,ip,ll) when
|
||||
* ll > ip_end - ip. Cheap bounds check after ll has its final
|
||||
* value (post-extension-length read if any). */
|
||||
if (VV_UNLIKELY((size_t)(ip_end - ip) < ll || (size_t)(op_end - op) < ll))
|
||||
return -1;
|
||||
|
||||
if (VV_LIKELY(ll <= 14 && ip + ll + 2 <= ip_end)) {
|
||||
uint16_t off_raw;
|
||||
memcpy(&off_raw, ip + ll, 2);
|
||||
|
|
@ -195,6 +203,14 @@ decode_block_tokens_impl(
|
|||
if (VV_UNLIKELY(ll == 15))
|
||||
ll += (uint32_t)read_ext_len(&ip, ip_end);
|
||||
|
||||
/* Sprint 109 fix: corrupt ll extension can yield a huge ll
|
||||
* that exceeds remaining input or output. Found by libFuzzer
|
||||
* + ASan: heap-buffer-overflow READ at memcpy(op,ip,ll) when
|
||||
* ll > ip_end - ip. Cheap bounds check after ll has its final
|
||||
* value (post-extension-length read if any). */
|
||||
if (VV_UNLIKELY((size_t)(ip_end - ip) < ll || (size_t)(op_end - op) < ll))
|
||||
return -1;
|
||||
|
||||
if (VV_LIKELY(ll <= 14 && ip + ll + 2 <= ip_end)) {
|
||||
uint16_t off_raw;
|
||||
memcpy(&off_raw, ip + ll, 2);
|
||||
|
|
|
|||
458
src/vv_encoder.c
458
src/vv_encoder.c
|
|
@ -1,5 +1,5 @@
|
|||
/* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
*
|
||||
* VaptVupt — Encoder v2 (Sprint 1)
|
||||
*
|
||||
|
|
@ -28,6 +28,53 @@
|
|||
#define VV_ENC_AVX2 0
|
||||
#endif
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════════
|
||||
* SECURE MEMORY ZERO (Sprint 117 — defense in depth)
|
||||
*
|
||||
* Compiler-resistant memset that the optimizer cannot eliminate
|
||||
* even when followed by free(). Used to scrub plaintext-derived
|
||||
* working buffers (literals, stripped tokens, ANS scratch) before
|
||||
* release back to the heap allocator. Without this, plaintext
|
||||
* fragments persist in the heap free-list and may be observable
|
||||
* through later allocations or memory disclosure.
|
||||
*
|
||||
* Implementation strategy:
|
||||
* - Prefer `explicit_bzero` (BSD/glibc 2.25+, guaranteed-secure)
|
||||
* - Fall back to `memset_explicit` (C23)
|
||||
* - Last resort: volatile-pointer memset (compiler cannot
|
||||
* prove the writes are dead)
|
||||
* ═══════════════════════════════════════════════════════════════ */
|
||||
|
||||
#if defined(__GLIBC__) && (__GLIBC__ > 2 || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 25))
|
||||
# define VV_HAS_EXPLICIT_BZERO 1
|
||||
#elif defined(__APPLE__) || defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__NetBSD__)
|
||||
# define VV_HAS_EXPLICIT_BZERO 1
|
||||
#else
|
||||
# define VV_HAS_EXPLICIT_BZERO 0
|
||||
#endif
|
||||
|
||||
#if VV_HAS_EXPLICIT_BZERO
|
||||
/* Forward-declare so we don't have to enable _DEFAULT_SOURCE globally
|
||||
* (which would conflict with the strict _POSIX_C_SOURCE=199309L the
|
||||
* project sets). The symbol is in libc on supported platforms. */
|
||||
extern void explicit_bzero(void *s, size_t n);
|
||||
#endif
|
||||
|
||||
static inline void vv_secure_zero(void *buf, size_t len) {
|
||||
if (!buf || !len) return;
|
||||
#if VV_HAS_EXPLICIT_BZERO
|
||||
explicit_bzero(buf, len);
|
||||
#else
|
||||
/* Volatile pointer prevents the compiler from concluding the
|
||||
* memset is dead and eliminating it. The volatile read of `p`
|
||||
* each iteration forces the writes to be observable. */
|
||||
volatile unsigned char *p = (volatile unsigned char *)buf;
|
||||
while (len--) *p++ = 0;
|
||||
#endif
|
||||
}
|
||||
|
||||
/* Sprint 117: VV_NO_SANITIZE_INTEGER is provided by include/vv_platform.h. */
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════════
|
||||
* VARINT WRITER
|
||||
* ═══════════════════════════════════════════════════════════════ */
|
||||
|
|
@ -46,7 +93,7 @@ static inline size_t write_varint(uint8_t *dst, size_t val) {
|
|||
* widening to an 8-byte load that over-reads the buffer.
|
||||
* ═══════════════════════════════════════════════════════════════ */
|
||||
|
||||
static inline uint32_t hash5(const uint8_t *p) {
|
||||
static inline VV_NO_SANITIZE_INTEGER uint32_t hash5(const uint8_t *p) {
|
||||
uint32_t lo;
|
||||
memcpy(&lo, p, 4);
|
||||
uint64_t v = (uint64_t)lo | ((uint64_t)p[4] << 32);
|
||||
|
|
@ -55,14 +102,14 @@ static inline uint32_t hash5(const uint8_t *p) {
|
|||
}
|
||||
|
||||
/* 4-byte hash for positions near end of buffer */
|
||||
static inline uint32_t hash4(const uint8_t *p) {
|
||||
static inline VV_NO_SANITIZE_INTEGER uint32_t hash4(const uint8_t *p) {
|
||||
uint32_t v;
|
||||
memcpy(&v, p, 4);
|
||||
return (v * 2654435761u) >> (32 - VV_HC_BITS);
|
||||
}
|
||||
|
||||
/* Safe hash: picks 5-byte or 4-byte depending on remaining bytes */
|
||||
static inline uint32_t hash_safe(const uint8_t *p, int32_t remain) {
|
||||
static inline VV_NO_SANITIZE_INTEGER uint32_t hash_safe(const uint8_t *p, int32_t remain) {
|
||||
return (remain >= 5) ? hash5(p) : hash4(p);
|
||||
}
|
||||
|
||||
|
|
@ -76,6 +123,31 @@ static inline uint32_t hash_safe(const uint8_t *p, int32_t remain) {
|
|||
static inline int32_t extend_match(const uint8_t *a, const uint8_t *b,
|
||||
int32_t max_len) {
|
||||
int32_t len = 0;
|
||||
|
||||
/* SPRINT 55: 8-byte fast-path check first. On binary content,
|
||||
* most matches extend 0-12 bytes past the initial 4-byte compare
|
||||
* (chain_match_ex already verified 4 bytes before calling). The
|
||||
* AVX2 loop's 32-byte minimum overshoots for these common short
|
||||
* matches, wasting a load and movemask on bytes we don't need.
|
||||
*
|
||||
* Check 8 bytes via scalar xor-ctz first: this resolves the
|
||||
* common case in 2-3 uops. On binary fixtures (bash, libc.so.6,
|
||||
* python3 extreme+format-v2), measurement shows ~60-75% of
|
||||
* extend_match calls return len ≤ 8.
|
||||
*
|
||||
* Falls through to AVX2 when the 8-byte window fully matches
|
||||
* and max_len is ≥ 32, so long-match ratio is preserved. */
|
||||
if (max_len >= 8) {
|
||||
uint64_t va, vb;
|
||||
memcpy(&va, a, 8);
|
||||
memcpy(&vb, b, 8);
|
||||
uint64_t xor_ab = va ^ vb;
|
||||
if (xor_ab) {
|
||||
/* Little-endian: byte at position k differs iff bit k*8 set */
|
||||
return __builtin_ctzll(xor_ab) >> 3;
|
||||
}
|
||||
len = 8;
|
||||
}
|
||||
#if VV_ENC_AVX2
|
||||
while (len + 32 <= max_len) {
|
||||
__m256i va = _mm256_loadu_si256((const __m256i *)(a + len));
|
||||
|
|
@ -98,7 +170,7 @@ static inline int32_t extend_match(const uint8_t *a, const uint8_t *b,
|
|||
#define VV_HC4_BITS 16
|
||||
#define VV_HC4_SIZE (1u << VV_HC4_BITS)
|
||||
|
||||
static inline uint32_t hash4_short(const uint8_t *p) {
|
||||
static inline VV_NO_SANITIZE_INTEGER uint32_t hash4_short(const uint8_t *p) {
|
||||
uint32_t v;
|
||||
memcpy(&v, p, 4);
|
||||
return (v * 2654435761u) >> (32 - VV_HC4_BITS);
|
||||
|
|
@ -120,7 +192,7 @@ static inline uint32_t hash4_short(const uint8_t *p) {
|
|||
#define VV_HC3_BITS 14
|
||||
#define VV_HC3_SIZE (1u << VV_HC3_BITS)
|
||||
|
||||
static inline uint32_t hash3_short(const uint8_t *p) {
|
||||
static inline VV_NO_SANITIZE_INTEGER uint32_t hash3_short(const uint8_t *p) {
|
||||
uint32_t v = (uint32_t)p[0] | ((uint32_t)p[1] << 8) | ((uint32_t)p[2] << 16);
|
||||
return (v * 2654435761u) >> (32 - VV_HC3_BITS);
|
||||
}
|
||||
|
|
@ -143,16 +215,40 @@ typedef struct {
|
|||
* extra bits only reaches 65534). */
|
||||
} matcher_t;
|
||||
|
||||
static void matcher_init(matcher_t *m, uint32_t window_log, uint32_t depth) {
|
||||
/* SPRINT 93 audit: returns 1 on success, 0 on allocation failure.
|
||||
* Callers MUST check the return value — on failure m is left in a
|
||||
* partially-initialized state with all pointers either valid or NULL,
|
||||
* safe to pass to matcher_free for cleanup.
|
||||
*
|
||||
* Prior to Sprint 93 this function was void-returning with unchecked
|
||||
* mallocs. If allocation failed, the immediately-following memset on
|
||||
* the NULL pointer would crash. Sprint 92 audit identified this as a
|
||||
* real defect. The fix tolerates allocator failure cleanly. */
|
||||
static void matcher_free(matcher_t *m); /* fwd decl for cleanup-on-failure */
|
||||
static int matcher_init(matcher_t *m, uint32_t window_log, uint32_t depth) {
|
||||
uint32_t wsz = 1u << window_log;
|
||||
/* Initialize ALL pointers to NULL first so matcher_free is safe to
|
||||
* call on partial-failure paths. */
|
||||
m->table = NULL;
|
||||
m->chain = NULL;
|
||||
m->table4 = NULL;
|
||||
m->hash4_chain = NULL;
|
||||
m->table3 = NULL;
|
||||
m->hash3_chain = NULL;
|
||||
|
||||
m->table = (int32_t *)malloc(VV_HC_SIZE * sizeof(int32_t));
|
||||
m->chain = (int32_t *)malloc(wsz * sizeof(int32_t));
|
||||
m->table4 = (int32_t *)malloc(VV_HC4_SIZE * sizeof(int32_t));
|
||||
m->hash4_chain = (int32_t *)malloc(wsz * sizeof(int32_t));
|
||||
if (!m->table || !m->chain || !m->table4 || !m->hash4_chain) {
|
||||
matcher_free(m);
|
||||
/* Re-NULL after free so caller's matcher_free is also safe */
|
||||
m->table = m->chain = m->table4 = m->hash4_chain = NULL;
|
||||
m->table3 = m->hash3_chain = NULL;
|
||||
return 0;
|
||||
}
|
||||
/* hash3 tables allocated lazily only when use_hash3 is enabled.
|
||||
* On v1 path (the default), they stay NULL and cost nothing. */
|
||||
m->table3 = NULL;
|
||||
m->hash3_chain = NULL;
|
||||
/* PERF: only the table arrays need to be cleared. chain/hash4_chain
|
||||
* are only read via table entries (which are now -1), so stale
|
||||
* data in them is unreachable. See matcher_reset for rationale. */
|
||||
|
|
@ -165,6 +261,7 @@ static void matcher_init(matcher_t *m, uint32_t window_log, uint32_t depth) {
|
|||
m->use_hash4 = 0; /* Disabled by default — enabled adaptively for binary */
|
||||
m->use_hash3 = 0; /* Disabled by default — enabled for format v2 */
|
||||
m->max_match = VV_MAX_MATCH; /* v1 default, see matcher_set_format_v2 */
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* Apply format v2 matcher constraints. Must be called whenever the
|
||||
|
|
@ -296,7 +393,8 @@ static inline int32_t try_rep_match(const matcher_t *m, const uint8_t *data,
|
|||
/* ─── Hash chain match: uses 5-byte hash, searches up to chain_depth.
|
||||
* If use_hash4 is nonzero AND hash5 finds nothing, fall back to hash4
|
||||
* chain for binary/struct coverage. ─── */
|
||||
static int32_t chain_match_ex(const matcher_t *m, const uint8_t *data,
|
||||
static VV_NO_SANITIZE_INTEGER int32_t
|
||||
chain_match_ex(const matcher_t *m, const uint8_t *data,
|
||||
int32_t pos, int32_t end, int32_t *best_off,
|
||||
int use_hash4) {
|
||||
/* Early exit: we need at least 3 bytes for hash3 probe, 4 for
|
||||
|
|
@ -319,27 +417,58 @@ static int32_t chain_match_ex(const matcher_t *m, const uint8_t *data,
|
|||
memcpy(&pos4, data + pos, 4);
|
||||
|
||||
/* Primary hash5 chain traversal.
|
||||
* PERF: prefetch the next chain slot 2 iterations ahead. Chain
|
||||
* entries are random-access through m->chain[ref & mask] and
|
||||
* typically miss L1 on binary-like data. A speculative L1 prefetch
|
||||
* issued 2 links ahead gives the CPU enough time to hide the
|
||||
* DRAM latency behind the match-compare work. */
|
||||
*
|
||||
* SPRINT 55: 4-way software-pipelined chain walk. Chain traversal
|
||||
* is a linked list — each next_ref depends on the previous chain
|
||||
* load. This serializes iterations at memory-latency speed (~10
|
||||
* ns per cache miss on binary data with poor hash5 locality).
|
||||
*
|
||||
* By walking the chain 4 links ahead and prefetching ALL of the
|
||||
* candidate data arrays AND the next chain slots speculatively,
|
||||
* we keep 4+ outstanding memory operations in flight per core.
|
||||
* The CPU's out-of-order engine then overlaps the 4 L1 fills,
|
||||
* effectively quadrupling match-test throughput on cache-miss-
|
||||
* bound workloads (bash, libc, python3).
|
||||
*
|
||||
* Measured effect: +8-15% encode on binary, ~neutral on text
|
||||
* (text already has good locality — fewer cache misses to hide).
|
||||
*
|
||||
* Safety: the prefetch is speculative ONLY. The actual chain walk
|
||||
* still respects the ref validity check before any load. A
|
||||
* prefetched ref that turns out to be out-of-range or cycles
|
||||
* back just results in a harmless L1 pollution — no OOB read, no
|
||||
* data-flow dependency on the prefetched value.
|
||||
*/
|
||||
uint32_t h = hash_safe(data + pos, end - pos);
|
||||
int32_t ref = m->table[h];
|
||||
uint32_t depth = m->chain_depth;
|
||||
uint32_t chain_mask = m->chain_mask;
|
||||
int32_t *chain_arr = m->chain;
|
||||
|
||||
/* Seed the pipeline: prefetch the source side of next candidate */
|
||||
/* Pipeline priming: look 4 chain entries ahead. If chain is
|
||||
* short, the prefetches become no-ops (chain entries below limit
|
||||
* just return -1 or an expired position). */
|
||||
if (ref >= limit && ref < pos) {
|
||||
__builtin_prefetch(data + ref, 0, 0);
|
||||
int32_t r1 = chain_arr[ref & chain_mask];
|
||||
if (r1 >= limit && r1 < pos) {
|
||||
__builtin_prefetch(data + r1, 0, 0);
|
||||
__builtin_prefetch(&chain_arr[r1 & chain_mask], 0, 0);
|
||||
int32_t r2 = chain_arr[r1 & chain_mask];
|
||||
if (r2 >= limit && r2 < pos) {
|
||||
__builtin_prefetch(data + r2, 0, 0);
|
||||
__builtin_prefetch(&chain_arr[r2 & chain_mask], 0, 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
while (ref >= 0 && ref >= limit && ref < pos && depth-- > 0) {
|
||||
int32_t next_ref = m->chain[ref & m->chain_mask];
|
||||
/* Prefetch: next chain traversal's candidate data bytes */
|
||||
int32_t next_ref = chain_arr[ref & chain_mask];
|
||||
/* Prefetch the link 2-3 iterations ahead so the linked-list
|
||||
* chain of loads can overlap with match-compare work */
|
||||
if (next_ref >= limit && next_ref < pos) {
|
||||
__builtin_prefetch(data + next_ref, 0, 0);
|
||||
/* Also prefetch the chain entry after next, for 2-ahead cover */
|
||||
__builtin_prefetch(&m->chain[next_ref & m->chain_mask], 0, 0);
|
||||
__builtin_prefetch(&chain_arr[next_ref & chain_mask], 0, 0);
|
||||
}
|
||||
|
||||
uint32_t b;
|
||||
|
|
@ -567,7 +696,37 @@ static size_t compress_block(const uint8_t *src, size_t start_pos, size_t block_
|
|||
}
|
||||
|
||||
/* ─── Step 3: Lazy evaluation (balanced + extreme) ─── */
|
||||
if (mode >= VV_MODE_BALANCED && mlen >= min_match &&
|
||||
/* Sprint 121: gate lazy probing on `mlen < 8`. Counterintuitive
|
||||
* but empirically validated: the cost-aware lazy decision
|
||||
* (added Sprint 120) makes WORSE choices when the current match
|
||||
* is already moderately long.
|
||||
*
|
||||
* Why: cost-aware lazy compares per-byte costs of competing
|
||||
* matches. When mlen ≥ 8, the current match's per-byte cost is
|
||||
* already low (≈1.5–3 bits/byte for typical offsets). A lazy
|
||||
* probe at pos+1 finding a slightly longer match at a different
|
||||
* offset triggers a shift, paying 1 literal but only marginally
|
||||
* improving per-byte cost. The literal cost dominates the small
|
||||
* per-byte gain, AND the cost-model approximation accumulates
|
||||
* error that biases toward shifting.
|
||||
*
|
||||
* Empirical sweep on the 8-fixture suite (aggregate Δ vs zstd-3):
|
||||
* no gate (lazy always): −0.130% (v2.48.0 / Sprint 120)
|
||||
* mlen < 16: −0.252%
|
||||
* mlen < 12: −0.386%
|
||||
* mlen < 9: −0.776%
|
||||
* mlen < 8: −1.070% (THIS)
|
||||
* mlen < 7: −1.282%
|
||||
* mlen < 6: −1.860%
|
||||
* mlen < 5: −2.044% (best aggregate, but
|
||||
* fx_json regresses 8.5pp)
|
||||
* no lazy (mlen < 4): −0.921%
|
||||
*
|
||||
* The mlen<5 setting wins aggregate but breaks fx_json from
|
||||
* −2.49% to +6.04% — unacceptable per-fixture regression.
|
||||
* mlen<8 is the safe optimum: improves every fixture vs v2.48.0
|
||||
* with no regressions. */
|
||||
if (mode >= VV_MODE_BALANCED && mlen >= min_match && mlen < 8 &&
|
||||
pos + 1 < end - min_match) {
|
||||
/* Check pos+1 */
|
||||
matcher_insert(m, src, pos, end);
|
||||
|
|
@ -577,54 +736,81 @@ static size_t compress_block(const uint8_t *src, size_t start_pos, size_t block_
|
|||
/* Also check rep at pos+1 */
|
||||
int32_t nri = -1;
|
||||
int32_t nrl = try_rep_match(m, src, pos + 1, end, &nri);
|
||||
if (nrl > nlen) { nlen = nrl; noff = (int32_t)m->rep[nri]; }
|
||||
if (nrl > nlen && nri >= 0) { nlen = nrl; noff = (int32_t)m->rep[nri]; }
|
||||
|
||||
/* standard */int32_t lazy_gain = 2;
|
||||
if (nlen > mlen + lazy_gain) {
|
||||
/* pos+1 is significantly better: emit literal, shift */
|
||||
/* SPRINT 119: cost-aware lazy decision (closes the +1.2%
|
||||
* ratio gap to zstd-3 — see CHANGELOG.md, Sprint 120).
|
||||
*
|
||||
* Old code used `nlen > mlen + 2` which ignores offset cost.
|
||||
* This made vv prefer shorter matches at far offsets over
|
||||
* longer matches at near offsets. zstd-3 produces ~11% fewer
|
||||
* sequences on dickens (1.22M vs 1.37M) by accounting for
|
||||
* offset cost when choosing between competing matches.
|
||||
*
|
||||
* Cost model:
|
||||
* match_bits(off, len) ≈ 10 + log2(off) + ml_extra(len)
|
||||
* - 10 covers ML/OF/LL ANS code values (avg ~3 bits each)
|
||||
* - log2(off) is the OF extra-bit cost (info-theoretic min)
|
||||
* - ml_extra is small for short matches (0 for len ≤ 19)
|
||||
* literal_bits ≈ 6 (4-stream Huffman avg on text)
|
||||
*
|
||||
* Decide on B (shift) over A (emit current match) when:
|
||||
* (literal_bits + match_bits(noff, nlen)) / (nlen + 1)
|
||||
* < match_bits(moff, mlen) / mlen
|
||||
*
|
||||
* Rep matches have offset cost ≈ 1 bit (rep code, no extras),
|
||||
* so they're heavily favored regardless of length. */
|
||||
if (nlen >= min_match) {
|
||||
/* Approx log2(off) — clamp to 1 for rep candidates and
|
||||
* to >= 1 generally to avoid div-by-zero quirks. */
|
||||
int log2_moff = 0; uint32_t mo = (uint32_t)moff;
|
||||
while (mo > 1) { mo >>= 1; log2_moff++; }
|
||||
int log2_noff = 0; uint32_t no = (uint32_t)noff;
|
||||
while (no > 1) { no >>= 1; log2_noff++; }
|
||||
|
||||
/* Rep matches use 0 extra bits but do consume an OF code
|
||||
* slot. Approximate them as cost 2 bits regardless of
|
||||
* the actual offset value.
|
||||
*
|
||||
* Sprint 121: per-mode constant. Extreme mode (deep
|
||||
* chain search) optimizes at +14; balanced mode
|
||||
* (shallow chain) optimizes at +18 because the lazy
|
||||
* candidates from a depth-24 search are noisier and
|
||||
* benefit from less aggressive shifting. */
|
||||
int cost_const = (mode >= VV_MODE_EXTREME) ? 14 : 18;
|
||||
int moff_bits = (rep_idx >= 0) ? 2 : (cost_const + log2_moff);
|
||||
int noff_bits = (nri >= 0) ? 2 : (cost_const + log2_noff);
|
||||
|
||||
/* Cross-multiply to avoid floating-point in hot path:
|
||||
* (literal_bits + noff_bits) * mlen < moff_bits * (nlen + 1) */
|
||||
int literal_bits = 6;
|
||||
int lhs = (literal_bits + noff_bits) * mlen;
|
||||
int rhs = moff_bits * (nlen + 1);
|
||||
if (lhs < rhs) {
|
||||
pos++;
|
||||
mlen = nlen; moff = noff;
|
||||
rep_idx = nri; /* may have shifted from explicit→rep or vice versa */
|
||||
|
||||
/* Lazy-2 disabled in v2.24.0.
|
||||
/* SPRINT 121: cost-aware lazy-2 was tested and rejected.
|
||||
* Measured Δ vs lazy-1-only (gate mlen<8 in both cases):
|
||||
* fx_text: neutral
|
||||
* fx_json: −0.003% (negligible)
|
||||
* fx_source: +0.022%
|
||||
* bash: −0.052%
|
||||
* dickens: +0.912% ← significant regression
|
||||
* xml: +0.086%
|
||||
* sao: +0.874% ← significant regression
|
||||
* x-ray: +0.361%
|
||||
* AGGREGATE: +0.601% (worse)
|
||||
*
|
||||
* Previously, after a lazy-1 shift to pos+1, this block
|
||||
* would try another shift to pos+2 if n2len > mlen + 1.
|
||||
* That was a classic "greedy past the peak" bug:
|
||||
* The shift cascade dominates: after a successful
|
||||
* lazy-1 shift, a second probe at the new pos+1
|
||||
* tends to find marginally-longer matches and
|
||||
* shifts again, eating literals faster than the
|
||||
* cost model accounts for. The cost-model error
|
||||
* compounds with each shift.
|
||||
*
|
||||
* Empirical results on 50KB English-text corpus:
|
||||
* balanced: 11,279 bytes
|
||||
* extreme + lazy-2 (+1): 12,157 bytes (+8% vs balanced)
|
||||
* extreme + lazy-2 (+2): 12,157 bytes (threshold didn't matter)
|
||||
* extreme + lazy-2 (+4): 11,860 bytes (still worse)
|
||||
* extreme, lazy-2 off: 10,718 bytes (5% better!)
|
||||
*
|
||||
* On 4 text corpora tested, lazy-2 cost an aggregate
|
||||
* 2,463 bytes vs disabled. On 2 JSON corpora, it saved
|
||||
* 213 bytes. Net-bytes-across-inputs: disabled wins by
|
||||
* ~10×. Disabling produces the strictly correct
|
||||
* "extreme >= balanced ratio" contract on all tested
|
||||
* inputs except JSON, where the regression is tiny
|
||||
* (<5%) and offset-encoding-cost-aware parsing would
|
||||
* be the proper fix (future work).
|
||||
*
|
||||
* Root cause: lazy-2's break-even model doesn't
|
||||
* account for the offset-extra-bits encoding cost of
|
||||
* the shifted-to match. On text, deeper chain search
|
||||
* finds long matches at far offsets whose extra-bits
|
||||
* cost exceeds the gain from the extra match length. */
|
||||
(void)lazy_gain; /* still used in lazy-1 above */
|
||||
if (0) {
|
||||
/* dead code — reference for future cost-aware work */
|
||||
matcher_insert(m, src, pos, end);
|
||||
int32_t n2off = 0;
|
||||
int32_t n2len = chain_match(m, src, pos + 1, end, &n2off);
|
||||
int32_t n2ri = -1;
|
||||
int32_t n2rl = try_rep_match(m, src, pos + 1, end, &n2ri);
|
||||
if (n2rl > n2len) { n2len = n2rl; n2off = (int32_t)m->rep[n2ri]; }
|
||||
if (n2len > mlen + lazy_gain) {
|
||||
pos++;
|
||||
mlen = n2len; moff = n2off;
|
||||
}
|
||||
* Lazy-1 captures the available benefit cleanly. */
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -771,7 +957,8 @@ static size_t emit_block(const uint8_t *src, size_t block_start, size_t braw,
|
|||
uint8_t *tmp, size_t tcap,
|
||||
uint8_t *lit_buf, size_t lit_cap,
|
||||
uint8_t *stripped, uint8_t *ent_buf, size_t ent_cap,
|
||||
uint8_t *dst, size_t dst_cap, int min_match) {
|
||||
uint8_t *dst, size_t dst_cap, int min_match,
|
||||
int compat_v246_5) {
|
||||
uint8_t *op = dst;
|
||||
|
||||
size_t csz = compress_block(src, block_start, braw, tmp, tcap, m, mode, min_match);
|
||||
|
|
@ -797,9 +984,11 @@ static size_t emit_block(const uint8_t *src, size_t block_start, size_t braw,
|
|||
* contain 3-byte matches that v1 encode_sequences cannot
|
||||
* represent correctly. */
|
||||
int use_v2 = (min_match < (int)VV_MIN_MATCH);
|
||||
/* Sprint 105 Phase C: thread compat flag through to SEQ encoder. */
|
||||
int dis_huf4 = compat_v246_5;
|
||||
vva_error_t serr = use_v2
|
||||
? vva_encode_sequences_v2(tmp, csz, ent_buf, ent_cap, &seq_len, off_bytes)
|
||||
: vva_encode_sequences(tmp, csz, ent_buf, ent_cap, &seq_len, off_bytes);
|
||||
? vva_encode_sequences_v2_compat(tmp, csz, ent_buf, ent_cap, &seq_len, off_bytes, dis_huf4)
|
||||
: vva_encode_sequences_compat(tmp, csz, ent_buf, ent_cap, &seq_len, off_bytes, dis_huf4);
|
||||
if (serr == VVA_OK) {
|
||||
seq_block_sz = 4 + 3 + 1 + seq_len;
|
||||
seq_valid = 1;
|
||||
|
|
@ -837,7 +1026,36 @@ static size_t emit_block(const uint8_t *src, size_t block_start, size_t braw,
|
|||
lit_count = extract_literals(tmp, csz, lit_buf, lit_cap,
|
||||
stripped, &stripped_len, off_bytes);
|
||||
if (lit_count > 0) {
|
||||
if (mode >= VV_MODE_BALANCED && lit_count >= 4096) {
|
||||
/* SPRINT 53: skip the expensive CTX (order-1 context)
|
||||
* path when sequence coding is already winning by a
|
||||
* big margin. Profile data across 7 fixtures (text,
|
||||
* json, source, 4 ELF binaries) showed CTX wins 0/16
|
||||
* attempts — the CTX coder has never actually beaten
|
||||
* SEQ on these workloads, but burned 20% of encode
|
||||
* time building per-context ANS tables that were
|
||||
* always discarded.
|
||||
*
|
||||
* Heuristic: skip CTX when seq_block_sz already does
|
||||
* better than 2:1 compression (seq_block_sz < braw/2).
|
||||
* Path A (SEQ) essentially never loses to Path B (CTX)
|
||||
* when the LZ matcher found strong matches. CTX only
|
||||
* matters for low-redundancy data where SEQ produces
|
||||
* close-to-raw output — exactly the case where
|
||||
* seq_block_sz ≥ braw/2.
|
||||
*
|
||||
* Falls back to ANS4 / ANS as literal coders in the
|
||||
* unchanged code below. These are ~10× cheaper than
|
||||
* CTX to build. Net encode-time savings measured in
|
||||
* SPRINT 53 CHANGELOG entry.
|
||||
*
|
||||
* Security/correctness: this is purely an encoder
|
||||
* heuristic. Decoder is unchanged. Output wire format
|
||||
* still meets spec. Worst case on a pathological
|
||||
* input where CTX would have won: we produce slightly
|
||||
* larger output via ANS4 or ANS. Ratio gate guards
|
||||
* against any real regression. */
|
||||
int skip_ctx = seq_valid && seq_block_sz < (braw * 4 / 5);
|
||||
if (!skip_ctx && mode >= VV_MODE_BALANCED && lit_count >= 4096) {
|
||||
vva_error_t aerr = vva_encode_ctx(lit_buf, lit_count,
|
||||
ent_buf2, ent_cap2, &ent_len);
|
||||
if (aerr == VVA_OK) ent_tag = VV_ENTROPY_CTX;
|
||||
|
|
@ -934,10 +1152,21 @@ size_t vv_compress_bound(size_t src_len) {
|
|||
int64_t vv_compress(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap,
|
||||
const vv_options_t *opts) {
|
||||
if (!src || !dst || !opts) return VV_ERR_PARAM;
|
||||
/* SPRINT 95 audit: accept NULL opts (fall back to defaults) for
|
||||
* consistency with vv_cstream_create. Also accept src_len=0
|
||||
* (an empty frame is a valid thing to produce — some streaming
|
||||
* protocols rely on it as a flush marker). */
|
||||
if (!dst) return VV_ERR_PARAM;
|
||||
if (src_len > 0 && !src) return VV_ERR_PARAM;
|
||||
if (dst_cap < sizeof(vv_frame_header_t) + sizeof(vv_frame_footer_t) + 16)
|
||||
return VV_ERR_OVERFLOW;
|
||||
|
||||
vv_options_t local_opts;
|
||||
if (!opts) {
|
||||
vv_default_options(&local_opts);
|
||||
opts = &local_opts;
|
||||
}
|
||||
|
||||
uint8_t wlog = opts->window_log;
|
||||
uint32_t depth;
|
||||
if (wlog == 0) {
|
||||
|
|
@ -967,13 +1196,21 @@ int64_t vv_compress(const uint8_t *src, size_t src_len,
|
|||
size_t trial_cap = trial_len + trial_len / 255 + 1024;
|
||||
uint8_t *trial_buf = (uint8_t *)malloc(trial_cap);
|
||||
if (trial_buf) {
|
||||
matcher_t m16; matcher_init(&m16, 16, 4);
|
||||
size_t sz16 = compress_block(src, 0, trial_len, trial_buf, trial_cap, &m16, VV_MODE_ULTRA_FAST, VV_MIN_MATCH);
|
||||
matcher_t m16;
|
||||
size_t sz16 = 0, sz20 = 0;
|
||||
/* SPRINT 93 audit: matcher_init can fail; if it does, skip
|
||||
* the trial (this path is a perf-tuning probe — falling
|
||||
* back to default wlog is safe). */
|
||||
if (matcher_init(&m16, 16, 4)) {
|
||||
sz16 = compress_block(src, 0, trial_len, trial_buf, trial_cap, &m16, VV_MODE_ULTRA_FAST, VV_MIN_MATCH);
|
||||
matcher_free(&m16);
|
||||
}
|
||||
|
||||
matcher_t m20; matcher_init(&m20, 20, 4);
|
||||
size_t sz20 = compress_block(src, 0, trial_len, trial_buf, trial_cap, &m20, VV_MODE_ULTRA_FAST, VV_MIN_MATCH);
|
||||
matcher_t m20;
|
||||
if (matcher_init(&m20, 20, 4)) {
|
||||
sz20 = compress_block(src, 0, trial_len, trial_buf, trial_cap, &m20, VV_MODE_ULTRA_FAST, VV_MIN_MATCH);
|
||||
matcher_free(&m20);
|
||||
}
|
||||
|
||||
free(trial_buf);
|
||||
if (sz20 > 0 && sz16 > 0 && sz20 < (sz16 * 97 / 100)) wlog = 20;
|
||||
|
|
@ -983,6 +1220,15 @@ int64_t vv_compress(const uint8_t *src, size_t src_len,
|
|||
}
|
||||
}
|
||||
|
||||
/* SPRINT 67: size-based wlog override. The trial above often
|
||||
* misses wins that only become visible past the 128 KB trial
|
||||
* boundary (long-range refs in multi-MB files). Override to
|
||||
* wlog=18 for files ≥ 3 MB when the trial left wlog at 16. */
|
||||
if (opts->window_log == 0 && opts->mode >= VV_MODE_BALANCED &&
|
||||
wlog == 16 && src_len >= 3145728) {
|
||||
wlog = 18;
|
||||
}
|
||||
|
||||
/* Frame header */
|
||||
uint8_t *op = dst;
|
||||
vv_frame_header_t fh;
|
||||
|
|
@ -997,7 +1243,10 @@ int64_t vv_compress(const uint8_t *src, size_t src_len,
|
|||
|
||||
/* Matcher */
|
||||
matcher_t m;
|
||||
matcher_init(&m, wlog, depth);
|
||||
/* SPRINT 93 audit: handle allocation failure cleanly */
|
||||
if (!matcher_init(&m, wlog, depth)) {
|
||||
return VV_ERR_NOMEM;
|
||||
}
|
||||
m.use_hash4 = (uint8_t)enable_hash4; /* From fused adaptive-window trial */
|
||||
/* Format v2 cap applies to EVERY match emitted from this matcher,
|
||||
* not just those produced via hash3. Set unconditionally when
|
||||
|
|
@ -1065,7 +1314,8 @@ int64_t vv_compress(const uint8_t *src, size_t src_len,
|
|||
size_t written = emit_block(src, block_start, braw, last, &m, opts->mode, wlog,
|
||||
tmp, tcap, lit_buf, lit_cap,
|
||||
stripped, ent_buf, ent_cap,
|
||||
op, dst_cap - (size_t)(op - dst), min_match);
|
||||
op, dst_cap - (size_t)(op - dst), min_match,
|
||||
opts->compat_v246_5_decoder);
|
||||
if (written == 0) {
|
||||
free(lit_buf); free(stripped); free(ent_buf);
|
||||
free(tmp); matcher_free(&m);
|
||||
|
|
@ -1075,6 +1325,12 @@ int64_t vv_compress(const uint8_t *src, size_t src_len,
|
|||
ip += braw; remaining -= braw;
|
||||
}
|
||||
|
||||
/* Sprint 117: scrub plaintext-derived working buffers before free
|
||||
* to prevent heap-residue leak (defense in depth). */
|
||||
vv_secure_zero(tmp, tcap);
|
||||
if (lit_buf) vv_secure_zero(lit_buf, lit_cap);
|
||||
if (stripped) vv_secure_zero(stripped, tcap);
|
||||
if (ent_buf) vv_secure_zero(ent_buf, ent_cap);
|
||||
free(lit_buf); free(stripped); free(ent_buf);
|
||||
free(tmp);
|
||||
|
||||
|
|
@ -1154,16 +1410,26 @@ vv_cstream_t *vv_cstream_create(const vv_options_t *opts) {
|
|||
default: depth = 24;
|
||||
}
|
||||
|
||||
matcher_init(&ctx->m, wlog, depth);
|
||||
/* SPRINT 93 audit: matcher_init can fail; cstream returns NULL
|
||||
* on any allocation error per public API contract. */
|
||||
if (!matcher_init(&ctx->m, wlog, depth)) {
|
||||
free(ctx);
|
||||
return NULL;
|
||||
}
|
||||
/* Format v2 matchlen cap applies to every match — set whenever
|
||||
* streaming opts has format_v2 on, not just when hash3 fires. */
|
||||
if (opts->format_v2) {
|
||||
* streaming opts has format_v2 on, not just when hash3 fires.
|
||||
*
|
||||
* Sprint 89 audit: read from ctx->opts (populated above with either
|
||||
* the caller's opts or default values) rather than the raw opts
|
||||
* pointer, which can be NULL when caller wants defaults. The prior
|
||||
* code dereferenced NULL when called as vv_cstream_create(NULL). */
|
||||
if (ctx->opts.format_v2) {
|
||||
matcher_set_format_v2(&ctx->m);
|
||||
}
|
||||
/* SPRINT 45: enable hash3 for format v2 streaming. Must free
|
||||
* ctx before returning NULL — callers use NULL-check semantics
|
||||
* here, not error codes. */
|
||||
if (opts->format_v2) {
|
||||
if (ctx->opts.format_v2) {
|
||||
if (!matcher_enable_hash3(&ctx->m)) {
|
||||
matcher_free(&ctx->m);
|
||||
free(ctx);
|
||||
|
|
@ -1198,9 +1464,20 @@ vv_cstream_t *vv_cstream_create(const vv_options_t *opts) {
|
|||
|
||||
void vv_cstream_destroy(vv_cstream_t *ctx) {
|
||||
if (!ctx) return;
|
||||
/* Sprint 117: zero plaintext-derived working buffers before free.
|
||||
* lit_buf and stripped contain literal bytes from the input; src_buf
|
||||
* holds raw input. tmp/ent_buf may contain compressed-but-not-yet-
|
||||
* encrypted output. All are scrubbed to prevent heap-residue leak. */
|
||||
if (ctx->tmp) vv_secure_zero(ctx->tmp, ctx->tcap);
|
||||
if (ctx->lit_buf) vv_secure_zero(ctx->lit_buf, ctx->lit_cap);
|
||||
if (ctx->stripped) vv_secure_zero(ctx->stripped, ctx->lit_cap);
|
||||
if (ctx->ent_buf) vv_secure_zero(ctx->ent_buf, ctx->ent_cap);
|
||||
if (ctx->src_buf) vv_secure_zero(ctx->src_buf, ctx->src_cap);
|
||||
free(ctx->tmp); free(ctx->lit_buf); free(ctx->stripped); free(ctx->ent_buf);
|
||||
free(ctx->src_buf);
|
||||
matcher_free(&ctx->m);
|
||||
/* Scrub the context itself in case it held sensitive options */
|
||||
vv_secure_zero(ctx, sizeof(*ctx));
|
||||
free(ctx);
|
||||
}
|
||||
|
||||
|
|
@ -1337,7 +1614,8 @@ int vv_cstream_compress_chunk(vv_cstream_t *ctx,
|
|||
ctx->tmp, ctx->tcap,
|
||||
ctx->lit_buf, ctx->lit_cap,
|
||||
ctx->stripped, ctx->ent_buf, ctx->ent_cap,
|
||||
op, cap_left, stream_min_match);
|
||||
op, cap_left, stream_min_match,
|
||||
ctx->opts.compat_v246_5_decoder);
|
||||
if (block_sz == 0) return VV_ERR_OVERFLOW;
|
||||
op += block_sz; cap_left -= block_sz;
|
||||
}
|
||||
|
|
@ -1349,7 +1627,8 @@ int vv_cstream_compress_chunk(vv_cstream_t *ctx,
|
|||
ff.checksum = vv_xxh64_finalize(&ctx->cks);
|
||||
ff.footer_magic = 0x56564E44u;
|
||||
memcpy(op, &ff, sizeof(ff));
|
||||
op += sizeof(ff); cap_left -= sizeof(ff);
|
||||
op += sizeof(ff);
|
||||
/* cap_left no longer read — function returns immediately below */
|
||||
}
|
||||
|
||||
*written = (size_t)(op - dst);
|
||||
|
|
@ -1388,7 +1667,12 @@ typedef struct {
|
|||
typedef struct {
|
||||
mt_task_t *tasks;
|
||||
size_t ntasks;
|
||||
volatile size_t next_task;
|
||||
/* SPRINT 98 audit: removed redundant `volatile`. next_task is
|
||||
* protected by the mutex below, which provides full memory
|
||||
* ordering. `volatile` was misleading — it doesn't provide
|
||||
* synchronization, only prevents compiler reordering, and the
|
||||
* mutex already prevents both. */
|
||||
size_t next_task;
|
||||
pthread_mutex_t mutex;
|
||||
} mt_pool_t;
|
||||
|
||||
|
|
@ -1411,7 +1695,15 @@ int64_t vv_compress_mt(const uint8_t *src, size_t src_len,
|
|||
const vv_options_t *opts,
|
||||
unsigned int nthreads,
|
||||
size_t chunk_size) {
|
||||
if (!src || !dst || !opts) return VV_ERR_PARAM;
|
||||
/* SPRINT 95 audit: same as vv_compress — accept NULL opts and
|
||||
* src_len=0 for API consistency. */
|
||||
if (!dst) return VV_ERR_PARAM;
|
||||
if (src_len > 0 && !src) return VV_ERR_PARAM;
|
||||
vv_options_t local_opts;
|
||||
if (!opts) {
|
||||
vv_default_options(&local_opts);
|
||||
opts = &local_opts;
|
||||
}
|
||||
if (chunk_size == 0) chunk_size = 4 * 1024 * 1024; /* 4 MB default */
|
||||
if (chunk_size < VV_MAX_BLOCK_SIZE) chunk_size = VV_MAX_BLOCK_SIZE;
|
||||
|
||||
|
|
|
|||
248
src/vv_huffman.c
248
src/vv_huffman.c
|
|
@ -1,5 +1,5 @@
|
|||
/* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
*
|
||||
* VaptVupt — Canonical Huffman Codec Implementation
|
||||
*
|
||||
|
|
@ -461,6 +461,111 @@ vvh_error_t vvh_encode(const uint8_t *src, size_t src_len,
|
|||
return VVH_OK;
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════════
|
||||
* 4-STREAM INTERLEAVED ENCODE (Sprint 103, Phase A)
|
||||
*
|
||||
* Splits input into 4 round-robin streams sharing a single Huffman
|
||||
* table. The decoder runs 4 independent decoders in parallel,
|
||||
* gaining instruction-level parallelism (1.8-2.2× decode throughput).
|
||||
*
|
||||
* Wire format (after the standard code-length header):
|
||||
* [3B stream1_size] [3B stream2_size] [3B stream3_size]
|
||||
* [stream0_bitstream] [stream1_bitstream] [stream2_bitstream] [stream3_bitstream]
|
||||
*
|
||||
* Stream0's size is implicit: total - 9 - hdr_sz - s1 - s2 - s3.
|
||||
* Each stream is byte-aligned at its start (clean entry for decoder).
|
||||
*
|
||||
* Activation guard: src_len >= 1024. Below this, single-stream wins
|
||||
* on overhead (9-byte stream-size header + per-stream alignment slop
|
||||
* dominates).
|
||||
*
|
||||
* See CHANGELOG.md (Sprint 105) for the design rationale.
|
||||
* ═══════════════════════════════════════════════════════════════ */
|
||||
|
||||
#define VVH4_STREAM_HDR_SZ 9 /* 3 bytes × 3 stream sizes */
|
||||
#define VVH4_MIN_LITERALS 1024
|
||||
|
||||
vvh_error_t vvh_encode4(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len) {
|
||||
/* Activation guard: 4-stream is only profitable above 1024 lits. */
|
||||
if (src_len < VVH4_MIN_LITERALS) return VVH_ERR_OVERFLOW;
|
||||
|
||||
/* Need at least: code-length header (~129B) + 9B stream-hdr +
|
||||
* 4 streams of nonzero size. Conservative lower bound. */
|
||||
if (dst_cap < 200) return VVH_ERR_OVERFLOW;
|
||||
|
||||
/* ─── 1. Count frequencies (single shared table) ─── */
|
||||
uint32_t freq[VVH_SYMBOLS];
|
||||
memset(freq, 0, sizeof(freq));
|
||||
for (size_t i = 0; i < src_len; i++)
|
||||
freq[src[i]]++;
|
||||
|
||||
/* ─── 2. Build encode table (shared across all 4 streams) ─── */
|
||||
vvh_enc_table_t enc;
|
||||
build_enc_table(freq, &enc);
|
||||
|
||||
/* ─── 3. Write code-length header ─── */
|
||||
size_t hdr_sz = write_header(enc.lengths, dst, dst_cap);
|
||||
if (hdr_sz == 0) return VVH_ERR_OVERFLOW;
|
||||
|
||||
/* ─── 4. Reserve 9 bytes for stream-size header (backpatched) ─── */
|
||||
if (hdr_sz + VVH4_STREAM_HDR_SZ >= dst_cap) return VVH_ERR_OVERFLOW;
|
||||
uint8_t *stream_hdr = dst + hdr_sz;
|
||||
size_t streams_offset = hdr_sz + VVH4_STREAM_HDR_SZ;
|
||||
|
||||
/* ─── 5. Encode each stream into the dst buffer ─── */
|
||||
/* Per-stream symbol counts for round-robin distribution:
|
||||
* stream0 gets indices 0, 4, 8, ..., (src_len + 3) / 4 symbols
|
||||
* stream1 gets indices 1, 5, 9, ..., (src_len + 2) / 4 symbols
|
||||
* stream2 gets indices 2, 6, 10, ..., (src_len + 1) / 4 symbols
|
||||
* stream3 gets indices 3, 7, 11, ..., src_len / 4 symbols
|
||||
*/
|
||||
size_t cur_off = streams_offset;
|
||||
size_t stream_sizes[4];
|
||||
|
||||
for (int s = 0; s < 4; s++) {
|
||||
if (cur_off >= dst_cap) return VVH_ERR_OVERFLOW;
|
||||
bw_t w;
|
||||
bw_init(&w, dst + cur_off, dst_cap - cur_off);
|
||||
|
||||
/* Round-robin: encode symbols at indices s, s+4, s+8, ... */
|
||||
for (size_t i = (size_t)s; i < src_len; i += 4) {
|
||||
uint8_t sym = src[i];
|
||||
/* enc.lengths[sym] could be 0 only if the symbol never
|
||||
* appeared in input — but we just counted and it did, so
|
||||
* length > 0 for every symbol we encode. Defensive check
|
||||
* for static analyzer happiness: */
|
||||
if (enc.lengths[sym] == 0) return VVH_ERR_CORRUPT;
|
||||
bw_add(&w, enc.codes[sym], enc.lengths[sym]);
|
||||
}
|
||||
|
||||
size_t sz = bw_flush(&w);
|
||||
stream_sizes[s] = sz;
|
||||
cur_off += sz;
|
||||
}
|
||||
|
||||
/* ─── 6. Backpatch stream-size header (3 bytes per stream, LE) ─── */
|
||||
/* Stream 0 size is implicit; encode streams 1, 2, 3 here.
|
||||
* Each size is stored as 24-bit little-endian (max 16 MB / stream
|
||||
* — far above any realistic literal-block size). */
|
||||
for (int s = 1; s <= 3; s++) {
|
||||
size_t sz = stream_sizes[s];
|
||||
if (sz > 0xFFFFFF) return VVH_ERR_OVERFLOW; /* >16MB stream */
|
||||
uint8_t *p = stream_hdr + (s - 1) * 3;
|
||||
p[0] = (uint8_t)(sz & 0xFF);
|
||||
p[1] = (uint8_t)((sz >> 8) & 0xFF);
|
||||
p[2] = (uint8_t)((sz >> 16) & 0xFF);
|
||||
}
|
||||
|
||||
size_t total = cur_off;
|
||||
|
||||
/* Incompressible guard: same convention as vvh_encode. */
|
||||
if (total >= src_len) return VVH_ERR_OVERFLOW;
|
||||
|
||||
*dst_len = total;
|
||||
return VVH_OK;
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════════
|
||||
* DECODE
|
||||
*
|
||||
|
|
@ -556,3 +661,142 @@ vvh_error_t vvh_decode(const uint8_t *src, size_t src_len,
|
|||
free(dec);
|
||||
return VVH_OK;
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════════
|
||||
* 4-STREAM INTERLEAVED DECODE (Sprint 104, Phase B)
|
||||
*
|
||||
* Production decoder for the wire format produced by vvh_encode4.
|
||||
* Runs 4 independent decoders in parallel using a single shared
|
||||
* decode table. Each iteration of the hot loop performs 4 lookups
|
||||
* with no inter-decoder data dependencies, allowing the OoO engine
|
||||
* to pipeline them.
|
||||
*
|
||||
* Wire format expected (see vvh_encode4):
|
||||
* [code-length header] [3B s1] [3B s2] [3B s3]
|
||||
* [stream0_data] [stream1_data] [stream2_data] [stream3_data]
|
||||
*
|
||||
* Stream0's size is implicit. All four streams share one Huffman
|
||||
* table built from the code-length header.
|
||||
* ═══════════════════════════════════════════════════════════════ */
|
||||
|
||||
vvh_error_t vvh_decode4(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap,
|
||||
size_t num_literals, size_t *src_consumed) {
|
||||
if (num_literals == 0) {
|
||||
*src_consumed = 0;
|
||||
return VVH_OK;
|
||||
}
|
||||
if (num_literals > dst_cap) return VVH_ERR_OVERFLOW;
|
||||
|
||||
/* ─── 1. Read code-length header ─── */
|
||||
uint8_t lengths[VVH_SYMBOLS];
|
||||
size_t hdr_sz = read_header(src, src_len, lengths);
|
||||
if (hdr_sz == 0) return VVH_ERR_CORRUPT;
|
||||
|
||||
/* Validate at least one nonzero length */
|
||||
int has_sym = 0;
|
||||
for (int i = 0; i < VVH_SYMBOLS; i++)
|
||||
if (lengths[i] > 0) { has_sym = 1; break; }
|
||||
if (!has_sym) return VVH_ERR_CORRUPT;
|
||||
|
||||
/* ─── 2. Read 9-byte stream-size header ─── */
|
||||
if (hdr_sz + VVH4_STREAM_HDR_SZ > src_len) return VVH_ERR_CORRUPT;
|
||||
const uint8_t *sh = src + hdr_sz;
|
||||
size_t s1 = (size_t)sh[0] | ((size_t)sh[1] << 8) | ((size_t)sh[2] << 16);
|
||||
size_t s2 = (size_t)sh[3] | ((size_t)sh[4] << 8) | ((size_t)sh[5] << 16);
|
||||
size_t s3 = (size_t)sh[6] | ((size_t)sh[7] << 8) | ((size_t)sh[8] << 16);
|
||||
|
||||
/* ─── 3. Validate stream sizes (DoS-resistant bounds checks) ─── */
|
||||
size_t streams_off = hdr_sz + VVH4_STREAM_HDR_SZ;
|
||||
if (streams_off > src_len) return VVH_ERR_CORRUPT;
|
||||
size_t streams_total = src_len - streams_off;
|
||||
/* Overflow-safe check: s1 + s2 + s3 <= streams_total */
|
||||
if (s1 > streams_total) return VVH_ERR_CORRUPT;
|
||||
if (s2 > streams_total - s1) return VVH_ERR_CORRUPT;
|
||||
if (s3 > streams_total - s1 - s2) return VVH_ERR_CORRUPT;
|
||||
size_t s0 = streams_total - s1 - s2 - s3;
|
||||
/* All streams must be non-zero unless num_literals < 4 (degenerate) */
|
||||
if (num_literals >= 4) {
|
||||
if (s0 == 0 || s1 == 0 || s2 == 0 || s3 == 0) return VVH_ERR_CORRUPT;
|
||||
}
|
||||
|
||||
/* ─── 4. Build decode table (shared across all 4 streams) ─── */
|
||||
vvh_dec_table_t *dec = (vvh_dec_table_t *)malloc(sizeof(vvh_dec_table_t));
|
||||
if (!dec) return VVH_ERR_NOMEM;
|
||||
build_dec_table(lengths, dec);
|
||||
|
||||
/* ─── 5. Initialize 4 independent bitstream readers ─── */
|
||||
br_t r0, r1, r2, r3;
|
||||
br_init(&r0, src + streams_off, s0);
|
||||
br_init(&r1, src + streams_off + s0, s1);
|
||||
br_init(&r2, src + streams_off + s0 + s1, s2);
|
||||
br_init(&r3, src + streams_off + s0 + s1 + s2, s3);
|
||||
br_refill(&r0); br_refill(&r1); br_refill(&r2); br_refill(&r3);
|
||||
|
||||
/* ─── 6. Per-stream symbol counts (round-robin) ─── */
|
||||
/* num_literals = 4*Q + R where R in {0,1,2,3}.
|
||||
* stream 0 decodes Q + (R >= 1) symbols
|
||||
* stream 1 decodes Q + (R >= 2) symbols
|
||||
* stream 2 decodes Q + (R >= 3) symbols
|
||||
* stream 3 decodes Q symbols */
|
||||
size_t Q = num_literals / 4;
|
||||
|
||||
/* Helper: decode one symbol. Inlined manually below for ILP. */
|
||||
#define DEC_ONE(R, OUT) do { \
|
||||
if ((R).nbits < VVH_MAX_CODE_LEN) br_refill(&(R)); \
|
||||
uint32_t peek = br_peek(&(R), VVH_DECODE_BITS); \
|
||||
uint32_t entry = dec->table[peek]; \
|
||||
int sym = (int)(entry & 0xFF); \
|
||||
int len = (int)((entry >> 8) & 0xF); \
|
||||
if (VV_LIKELY(len > 0)) { \
|
||||
br_consume(&(R), len); \
|
||||
(OUT) = (uint8_t)sym; \
|
||||
} else { \
|
||||
int found = 0; \
|
||||
for (int s = 0; s < dec->slow_count; s++) { \
|
||||
int slen = dec->slow_len[s]; \
|
||||
uint32_t mask = (1u << slen) - 1; \
|
||||
if ((br_peek(&(R), slen) & mask) == dec->slow_code[s]) { \
|
||||
br_consume(&(R), slen); \
|
||||
(OUT) = dec->slow_sym[s]; \
|
||||
found = 1; \
|
||||
break; \
|
||||
} \
|
||||
} \
|
||||
if (!found) { free(dec); return VVH_ERR_CORRUPT; } \
|
||||
} \
|
||||
} while (0)
|
||||
|
||||
/* ─── 7. Hot loop: decode 4 symbols per iteration ─── */
|
||||
/* Each iteration's 4 decodes are fully independent — different
|
||||
* readers, different table peeks, different output positions.
|
||||
* Modern OoO engines can pipeline 4 independent decode chains
|
||||
* achieving ~1.8-2.2× speedup over single-stream. */
|
||||
size_t out_idx = 0;
|
||||
for (size_t i = 0; i < Q; i++) {
|
||||
uint8_t y0, y1, y2, y3;
|
||||
DEC_ONE(r0, y0);
|
||||
DEC_ONE(r1, y1);
|
||||
DEC_ONE(r2, y2);
|
||||
DEC_ONE(r3, y3);
|
||||
dst[out_idx + 0] = y0;
|
||||
dst[out_idx + 1] = y1;
|
||||
dst[out_idx + 2] = y2;
|
||||
dst[out_idx + 3] = y3;
|
||||
out_idx += 4;
|
||||
}
|
||||
|
||||
/* ─── 8. Tail (handle remaining 0-3 symbols) ─── */
|
||||
size_t tail = num_literals - Q * 4;
|
||||
if (tail >= 1) { uint8_t y; DEC_ONE(r0, y); dst[out_idx++] = y; }
|
||||
if (tail >= 2) { uint8_t y; DEC_ONE(r1, y); dst[out_idx++] = y; }
|
||||
if (tail >= 3) { uint8_t y; DEC_ONE(r2, y); dst[out_idx++] = y; }
|
||||
|
||||
#undef DEC_ONE
|
||||
|
||||
/* Total bytes consumed: header + stream-size header + all 4 streams */
|
||||
*src_consumed = streams_off + s0 + s1 + s2 + s3;
|
||||
|
||||
free(dec);
|
||||
return VVH_OK;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
/* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
*
|
||||
* VaptVupt — SIMD-accelerated copy routines
|
||||
*
|
||||
|
|
@ -19,7 +19,7 @@
|
|||
* SCALAR FALLBACK (always compiled)
|
||||
* ═══════════════════════════════════════════════════════════════ */
|
||||
|
||||
static void copy_fast_scalar(uint8_t *dst, const uint8_t *src, size_t n) {
|
||||
static void __attribute__((unused)) copy_fast_scalar(uint8_t *dst, const uint8_t *src, size_t n) {
|
||||
memcpy(dst, src, n);
|
||||
}
|
||||
|
||||
|
|
@ -181,38 +181,62 @@ static copy_fast_fn g_copy_fast = NULL;
|
|||
static copy_match_fn g_copy_match = NULL;
|
||||
|
||||
static void vv_init_simd(void) {
|
||||
if (g_copy_fast) return; /* Already initialized */
|
||||
/* SPRINT 98 audit: thread-safe lazy init using GCC/Clang atomic
|
||||
* builtins. The previous implementation had a benign-but-UB data
|
||||
* race: two threads could both observe NULL and both write to
|
||||
* g_copy_fast/g_copy_match. The writes were idempotent (always
|
||||
* the same CPU-feature pointer), so it never caused incorrect
|
||||
* behavior on x86, but per C11 it was UB. On weakly-ordered
|
||||
* architectures (ARM, POWER) the race could become observable.
|
||||
*
|
||||
* Atomic loads with ACQUIRE pair with atomic stores with RELEASE
|
||||
* to give a proper happens-before relationship. Multiple threads
|
||||
* may still race into the body, but each store is atomic and any
|
||||
* subsequent reader sees a consistent value. */
|
||||
if (__atomic_load_n(&g_copy_fast, __ATOMIC_ACQUIRE) &&
|
||||
__atomic_load_n(&g_copy_match, __ATOMIC_ACQUIRE)) return;
|
||||
|
||||
copy_fast_fn fast;
|
||||
copy_match_fn match;
|
||||
|
||||
#if defined(__x86_64__) || defined(_M_X64)
|
||||
#ifdef __AVX2__
|
||||
if (vv_has_avx2()) {
|
||||
g_copy_fast = copy_fast_avx2;
|
||||
g_copy_match = copy_match_avx2;
|
||||
return;
|
||||
}
|
||||
fast = copy_fast_avx2;
|
||||
match = copy_match_avx2;
|
||||
} else
|
||||
#endif
|
||||
{
|
||||
/* SSE2 is baseline on all x86-64 — no runtime check needed */
|
||||
g_copy_fast = copy_fast_sse2;
|
||||
g_copy_match = copy_match_sse2;
|
||||
return;
|
||||
fast = copy_fast_sse2;
|
||||
match = copy_match_sse2;
|
||||
}
|
||||
#elif defined(__aarch64__) && defined(__ARM_NEON)
|
||||
fast = copy_fast_neon;
|
||||
match = copy_match_neon;
|
||||
#else
|
||||
fast = copy_fast_scalar;
|
||||
match = copy_match_scalar;
|
||||
#endif
|
||||
|
||||
#if defined(__aarch64__) && defined(__ARM_NEON)
|
||||
g_copy_fast = copy_fast_neon;
|
||||
g_copy_match = copy_match_neon;
|
||||
return;
|
||||
#endif
|
||||
|
||||
g_copy_fast = copy_fast_scalar;
|
||||
g_copy_match = copy_match_scalar;
|
||||
__atomic_store_n(&g_copy_fast, fast, __ATOMIC_RELEASE);
|
||||
__atomic_store_n(&g_copy_match, match, __ATOMIC_RELEASE);
|
||||
}
|
||||
|
||||
void vv_copy_fast(uint8_t *dst, const uint8_t *src, size_t n) {
|
||||
if (!g_copy_fast) vv_init_simd();
|
||||
g_copy_fast(dst, src, n);
|
||||
copy_fast_fn fn = __atomic_load_n(&g_copy_fast, __ATOMIC_ACQUIRE);
|
||||
if (!fn) {
|
||||
vv_init_simd();
|
||||
fn = __atomic_load_n(&g_copy_fast, __ATOMIC_ACQUIRE);
|
||||
}
|
||||
fn(dst, src, n);
|
||||
}
|
||||
|
||||
void vv_copy_match(uint8_t *dst, uint32_t offset, size_t length) {
|
||||
if (!g_copy_match) vv_init_simd();
|
||||
g_copy_match(dst, offset, length);
|
||||
copy_match_fn fn = __atomic_load_n(&g_copy_match, __ATOMIC_ACQUIRE);
|
||||
if (!fn) {
|
||||
vv_init_simd();
|
||||
fn = __atomic_load_n(&g_copy_match, __ATOMIC_ACQUIRE);
|
||||
}
|
||||
fn(dst, offset, length);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,11 +1,12 @@
|
|||
/* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2025-2026 Cristian Cezar Moisés
|
||||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
*
|
||||
* VaptVupt — XXH64 checksum (simplified, standalone)
|
||||
* Based on xxHash by Yann Collet. Public domain.
|
||||
*/
|
||||
|
||||
#include "vaptvupt.h"
|
||||
#include "vv_platform.h" /* VV_NO_SANITIZE_INTEGER */
|
||||
#include <string.h>
|
||||
|
||||
#define XXH_PRIME64_1 0x9E3779B185EBCA87ULL
|
||||
|
|
@ -14,22 +15,47 @@
|
|||
#define XXH_PRIME64_4 0x85EBCA77C2B2AE63ULL
|
||||
#define XXH_PRIME64_5 0x27D4EB2F165667C5ULL
|
||||
|
||||
static inline uint64_t xxh_rotl64(uint64_t x, int r) { return (x << r) | (x >> (64 - r)); }
|
||||
/* Sprint 117: VV_NO_SANITIZE_INTEGER is provided by include/vv_platform.h.
|
||||
* xxh64 relies on intentional unsigned modular arithmetic (overflow
|
||||
* and shift-out-of-range bits in the round mixer); the annotation
|
||||
* silences -fsanitize=integer false positives in hardened builds. */
|
||||
|
||||
static inline uint64_t xxh_round(uint64_t acc, uint64_t input) {
|
||||
/* Sprint 117: prefer the compiler builtin which lowers to a single
|
||||
* rotate instruction and does NOT trigger UBSan's shift-base check.
|
||||
* Falls back to the manual shift expression on older compilers,
|
||||
* still annotated with no_sanitize for hardened builds. */
|
||||
#if defined(__clang__) && __has_builtin(__builtin_rotateleft64)
|
||||
# define XXH_ROTL64(x, r) __builtin_rotateleft64((x), (r))
|
||||
#elif defined(__GNUC__) && (__GNUC__ >= 7)
|
||||
/* GCC 7+ recognizes the manual idiom as a rotate */
|
||||
# define XXH_ROTL64(x, r) (((x) << (r)) | ((x) >> (64 - (r))))
|
||||
#else
|
||||
# define XXH_ROTL64(x, r) (((x) << (r)) | ((x) >> (64 - (r))))
|
||||
#endif
|
||||
|
||||
#if defined(__clang__)
|
||||
__attribute__((noinline))
|
||||
#endif
|
||||
static VV_NO_SANITIZE_INTEGER
|
||||
uint64_t xxh_rotl64(uint64_t x, int r) { return XXH_ROTL64(x, r); }
|
||||
|
||||
static inline VV_NO_SANITIZE_INTEGER
|
||||
uint64_t xxh_round(uint64_t acc, uint64_t input) {
|
||||
acc += input * XXH_PRIME64_2;
|
||||
acc = xxh_rotl64(acc, 31);
|
||||
acc *= XXH_PRIME64_1;
|
||||
return acc;
|
||||
}
|
||||
|
||||
static inline uint64_t xxh_merge_round(uint64_t acc, uint64_t val) {
|
||||
static inline VV_NO_SANITIZE_INTEGER
|
||||
uint64_t xxh_merge_round(uint64_t acc, uint64_t val) {
|
||||
val = xxh_round(0, val);
|
||||
acc ^= val;
|
||||
acc = acc * XXH_PRIME64_1 + XXH_PRIME64_4;
|
||||
return acc;
|
||||
}
|
||||
|
||||
VV_NO_SANITIZE_INTEGER
|
||||
uint64_t vv_xxh64(const void *data, size_t len, uint64_t seed) {
|
||||
const uint8_t *p = (const uint8_t *)data;
|
||||
const uint8_t *end = p + len;
|
||||
|
|
@ -97,6 +123,7 @@ void vv_xxh64_init(vv_xxh64_state_t *s, uint64_t seed) {
|
|||
s->seed = seed;
|
||||
}
|
||||
|
||||
VV_NO_SANITIZE_INTEGER
|
||||
void vv_xxh64_update(vv_xxh64_state_t *s, const void *data, size_t len) {
|
||||
const uint8_t *p = (const uint8_t *)data;
|
||||
const uint8_t *end = p + len;
|
||||
|
|
@ -137,6 +164,7 @@ void vv_xxh64_update(vv_xxh64_state_t *s, const void *data, size_t len) {
|
|||
}
|
||||
}
|
||||
|
||||
VV_NO_SANITIZE_INTEGER
|
||||
uint64_t vv_xxh64_finalize(const vv_xxh64_state_t *s) {
|
||||
uint64_t h64;
|
||||
|
||||
|
|
|
|||
Binary file not shown.
7
vendor/zuptsdk/include/vaptvupt.h
vendored
7
vendor/zuptsdk/include/vaptvupt.h
vendored
|
|
@ -189,6 +189,12 @@ typedef struct {
|
|||
int format_v2; /* 1 = produce 'T' tag blocks (min_match=3) for
|
||||
* better real-binary ratio. Requires decoder
|
||||
* v2.33.0+. Default 0 for back-compat. */
|
||||
int compat_v246_5_decoder;
|
||||
/* 1 = suppress lit_fmt=4 (4-stream Huffman) in
|
||||
* SEQ block encode race. Required when
|
||||
* output must be readable by v2.46.5 or
|
||||
* older decoders. Default 0 (lit_fmt=4
|
||||
* enabled, requires v2.47+ decoder). */
|
||||
} vv_options_t;
|
||||
|
||||
static inline void vv_default_options(vv_options_t *o) {
|
||||
|
|
@ -197,6 +203,7 @@ static inline void vv_default_options(vv_options_t *o) {
|
|||
o->checksum = 1;
|
||||
o->verbose = 0;
|
||||
o->format_v2 = 0;
|
||||
o->compat_v246_5_decoder = 0;
|
||||
}
|
||||
|
||||
/* ═══════════════════════════════════════════════════════════════
|
||||
|
|
|
|||
11
vendor/zuptsdk/include/vv_ans.h
vendored
11
vendor/zuptsdk/include/vv_ans.h
vendored
|
|
@ -1,6 +1,5 @@
|
|||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2026 Cristian Cezar Moisés
|
||||
*
|
||||
* VaptVupt — tANS Entropy Codec (v2: sparse header + 4-way interleaved)
|
||||
*
|
||||
|
|
@ -112,6 +111,16 @@ vva_error_t vva_encode_sequences_v2(const uint8_t *tokens, size_t tok_len,
|
|||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes);
|
||||
|
||||
/* Sprint 105 Phase C: variants accepting disable_huf4 flag.
|
||||
* disable_huf4=1 suppresses lit_fmt=4 (4-stream Huffman) selection
|
||||
* for v2.46.5 and older decoder compatibility. */
|
||||
vva_error_t vva_encode_sequences_compat(const uint8_t *tokens, size_t tok_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes, int disable_huf4);
|
||||
vva_error_t vva_encode_sequences_v2_compat(const uint8_t *tokens, size_t tok_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
int off_bytes, int disable_huf4);
|
||||
|
||||
vva_error_t vva_decode_sequences(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len,
|
||||
const uint8_t *dst_base);
|
||||
|
|
|
|||
46
vendor/zuptsdk/include/vv_huffman.h
vendored
46
vendor/zuptsdk/include/vv_huffman.h
vendored
|
|
@ -1,6 +1,5 @@
|
|||
/*
|
||||
* SPDX-License-Identifier: GPL-3.0-or-later
|
||||
* Copyright (c) 2026 Cristian Cezar Moisés
|
||||
*
|
||||
* VaptVupt — Canonical Huffman Codec
|
||||
*
|
||||
|
|
@ -113,6 +112,51 @@ vvh_error_t vvh_decode(const uint8_t *src, size_t src_len,
|
|||
uint8_t *dst, size_t dst_cap,
|
||||
size_t num_literals, size_t *src_consumed);
|
||||
|
||||
/*
|
||||
* 4-stream interleaved Huffman encode (Sprint 103, Phase A).
|
||||
*
|
||||
* Encodes src into 4 round-robin bitstreams sharing a single Huffman
|
||||
* code table. The output format is:
|
||||
*
|
||||
* [code-length header (existing format)]
|
||||
* [3B stream1_size] [3B stream2_size] [3B stream3_size]
|
||||
* [stream0_bitstream] [stream1_bitstream]
|
||||
* [stream2_bitstream] [stream3_bitstream]
|
||||
*
|
||||
* Activation guard: requires src_len >= 1024. Below this threshold,
|
||||
* single-stream vvh_encode wins on overhead and this function returns
|
||||
* VVH_ERR_OVERFLOW.
|
||||
*
|
||||
* NOTE (Phase A): Production decoder support arrives in Phase B.
|
||||
* This sprint adds only the encoder + a test-only inverse decoder
|
||||
* (in tests/test_huffman4.c) for round-trip verification.
|
||||
*
|
||||
* Returns VVH_OK on success.
|
||||
* Returns VVH_ERR_OVERFLOW if src_len < 1024, dst too small, or output
|
||||
* not smaller than input.
|
||||
*/
|
||||
vvh_error_t vvh_encode4(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap, size_t *dst_len);
|
||||
|
||||
/*
|
||||
* 4-stream interleaved Huffman decode (Sprint 104, Phase B).
|
||||
*
|
||||
* Inverse of vvh_encode4. Decodes the 4-stream wire format produced
|
||||
* by vvh_encode4. Runs 4 independent decoders in parallel using a
|
||||
* single shared decode table.
|
||||
*
|
||||
* src[0..src_len-1] — compressed data (header + stream-sizes + 4 streams)
|
||||
* dst[0..dst_cap-1] — output buffer for decoded literals
|
||||
* num_literals — expected number of decoded symbols
|
||||
* *src_consumed — on success, bytes consumed from src
|
||||
*
|
||||
* Returns VVH_OK on success, VVH_ERR_CORRUPT on malformed input,
|
||||
* VVH_ERR_OVERFLOW if dst is too small, VVH_ERR_NOMEM on alloc failure.
|
||||
*/
|
||||
vvh_error_t vvh_decode4(const uint8_t *src, size_t src_len,
|
||||
uint8_t *dst, size_t dst_cap,
|
||||
size_t num_literals, size_t *src_consumed);
|
||||
|
||||
/*
|
||||
* Upper bound on compressed size for src_len literal bytes.
|
||||
*/
|
||||
|
|
|
|||
22
vendor/zuptsdk/include/vv_platform.h
vendored
22
vendor/zuptsdk/include/vv_platform.h
vendored
|
|
@ -114,4 +114,26 @@ static inline void vv_store64(void *p, uint64_t v) { memcpy(p, &v, 8); }
|
|||
#define VV_HAS_NEON 0
|
||||
#endif
|
||||
|
||||
/* Sprint 117: explicit no_sanitize annotation for hardened builds.
|
||||
*
|
||||
* Several hot paths use intentional unsigned modular arithmetic:
|
||||
* - Knuth multiplicative hashes in the LZ matcher
|
||||
* - xxh64 round mixers (multiplication, left-shift)
|
||||
* - Post-decrement loop guards (uint32_t depth-- > 0)
|
||||
*
|
||||
* C11 §6.2.5p9 defines unsigned overflow as wraparound, so these are
|
||||
* NOT undefined behavior — but `-fsanitize=integer` and the related
|
||||
* `-fsanitize=shift-base` flags warn anyway, breaking hardened-build
|
||||
* deployments. Apply this attribute to the affected functions to
|
||||
* silence the false positives without disabling the checks globally.
|
||||
*
|
||||
* The annotation is clang-only (gcc has no equivalent and does not
|
||||
* accept -fsanitize=integer in the first place). */
|
||||
#if defined(__clang__) && (__clang_major__ >= 4)
|
||||
# define VV_NO_SANITIZE_INTEGER \
|
||||
__attribute__((no_sanitize("unsigned-integer-overflow", "shift", "shift-base", "shift-exponent")))
|
||||
#else
|
||||
# define VV_NO_SANITIZE_INTEGER
|
||||
#endif
|
||||
|
||||
#endif /* VV_PLATFORM_H */
|
||||
|
|
|
|||
2
vendor/zuptsdk/include/zupt.h
vendored
2
vendor/zuptsdk/include/zupt.h
vendored
|
|
@ -30,7 +30,7 @@
|
|||
#define zupt_mkdir(p) mkdir(p, 0755)
|
||||
#endif
|
||||
|
||||
#define ZUPT_VERSION_STRING "2.2.2"
|
||||
#define ZUPT_VERSION_STRING "2.2.3"
|
||||
#define ZUPT_FORMAT_MAJOR 1
|
||||
#define ZUPT_FORMAT_MINOR 4
|
||||
|
||||
|
|
|
|||
1
vendor/zuptsdk/libzuptsdk.so
vendored
Symbolic link
1
vendor/zuptsdk/libzuptsdk.so
vendored
Symbolic link
|
|
@ -0,0 +1 @@
|
|||
libzuptsdk.so.2.0.0
|
||||
1
vendor/zuptsdk/libzuptsdk.so.2
vendored
Symbolic link
1
vendor/zuptsdk/libzuptsdk.so.2
vendored
Symbolic link
|
|
@ -0,0 +1 @@
|
|||
libzuptsdk.so.2.0.0
|
||||
BIN
vendor/zuptsdk/libzuptsdk.so.2.0.0
vendored
Executable file
BIN
vendor/zuptsdk/libzuptsdk.so.2.0.0
vendored
Executable file
Binary file not shown.
Loading…
Reference in a new issue