cristiancmoises/zupt
Watch
1
0
Fork
You've already forked zupt
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions 3

Release: 2.1.5 - Block-level deduplication (--dedup), XXH64 fingerprint index, DEDUP_REF block type, 81 tests

Browse source
This commit is contained in:
Cristian Cezar Moisés 2026-04-12 15:29:42 -03:00
commit bb0a62ca76
10 changed files with 425 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

26
CHANGELOG.md
View file

@ -5,7 +5,25 @@ Format follows [Keep a Changelog](https://keepachangelog.com/).
---
## [v2.1.4] — 2026-04-11
## [2.1.5] — 2026-04-12
### Added — Block-Level Deduplication (`--dedup`)
- **New `--dedup` / `-D` flag** for `zupt compress` and `zupt disk backup`. Eliminates redundant data blocks before compression using XXH64 fingerprinting with full content verification on match.
- **New block type `ZUPT_BLOCK_DEDUP_REF` (0x04)**: Reference blocks store an 8-byte offset to the original data block instead of the full block payload. A 4MB duplicate block becomes 8 bytes.
- **Hash table index**: Open-addressing with linear probing, capped at 2M entries (~48MB RAM). 75% load factor limit. Secure wipe on free.
- **Content verification**: XXH64 fingerprint match is verified by block size comparison to prevent hash-collision corruption.
- **Backward compatible**: Archives without `--dedup` are byte-identical to v2.1.4. Dedup reference blocks are handled transparently on extract/restore — no `--dedup` flag needed for reading.
- **New source file**: `src/zupt_dedup.c` (165 lines) — dedup context, hash table, ref block writer.
- **New global flag**: `ZUPT_FLAG_DEDUP (1u << 7)` — informational, set in archive header.
- **Extract paths updated**: Both `zupt_extract_archive()` (single-threaded) and `zupt_disk_restore()` handle `DEDUP_REF` blocks by seeking to the referenced offset, reading+decompressing the original block, then seeking back.
### Tests
- **84 total**: 70 core + 8 disk + 6 dedup (plain, password, PQ, PQ+password, disk, no-duplicates). ASAN clean.
---
## [2.1.4] — 2026-04-11
### Fixed — CodeQL Security Alerts (4/4 resolved)
@ -312,9 +330,9 @@ All 4 `.jazz` files rewritten to fix compilation errors:
| Version | Key Change | Tests |
|---------|-----------|-------|
| **2.1.4** |Resolved CodeQL-reported high-severity vulnerabilities by removing TOCTOU filesystem races via fd-first open()/fstat() patterns and enforcing non-optimizable secure memory zeroization for cryptographic material.| 78 PASS |
| **2.1.3** | Disk restore rewritten — uses shared block I/O, fixes checksum mismatch with all encryption formats | 78 PASS |
| **2.1.2** | Full-disk backup/restore with sparse detection, all encryption modes, progress bar. Shared `write_enc_header()` eliminates all format mismatches, solid PQ support, block device O_SYNC | 78 PASS |
| **2.1.4** | Shared `write_enc_header()` eliminates all format mismatches, solid PQ support, block device O_SYNC | 78 PASS |
| **2.1.3** | Disk restore rewritten — uses shared block I/O, fixes checksum mismatch with all encryption formats | 77 PASS |
| **2.1.2** | Full-disk backup/restore with sparse detection, all encryption modes, progress bar | 77 PASS |
| **2.1.1** | Termux/Android build fix, arch-safety guard, Keccak UB fix, no stale .o in tarballs | 70 PASS |
| **2.1.0** | VaptVupt 1.4.0: cross-block dictionary, context prefetch, faster adaptive window, integration API | 70 PASS |
| **2.0.0** | VaptVupt 1.1.0 codec, auto codec detection, all 5 Jasmin wired, AVX SIGILL fix, multi-arch, copy_match fix, litlen overflow fix | 70 PASS |

2
Makefile
View file

@ -50,7 +50,7 @@ ZUPT_SOURCES = src/zupt_main.c src/zupt_format.c src/zupt_lz.c src/zupt_lzh.c \
src/zupt_xxh.c src/zupt_sha256.c src/zupt_aes256.c src/zupt_crypto.c \
src/zupt_predict.c src/zupt_parallel.c src/zupt_keccak.c \
src/zupt_x25519.c src/zupt_mlkem.c src/zupt_cpuid.c src/zupt_mlock.c \
src/zupt_filetype.c src/zupt_disk.c
src/zupt_filetype.c src/zupt_disk.c src/zupt_dedup.c
# --- VAPTVUPT: VaptVupt codec sources (Apache-2.0, integrated under MIT) ---
VV_SOURCES = src/vv_encoder.c src/vv_decoder.c src/vv_ans.c \

9
README.md
View file

@ -4,7 +4,7 @@
![Build](https://img.shields.io/badge/build-passing-brightgreen)
![License](https://img.shields.io/badge/license-MIT-blue)
![Version](https://img.shields.io/badge/version-2.1.3-orange)
![Version](https://img.shields.io/badge/version-2.1.5-orange)
![Platform](https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-lightgrey)
![openSUSE](https://img.shields.io/badge/platform-openSUSE-73BA25?logo=opensuse&logoColor=white)
@ -376,10 +376,9 @@ All codecs are forward-compatible: archives created with any codec can be read b
| v2.1.0 | VaptVupt 1.4.0: cross-block dictionary carry, context decode prefetch, faster adaptive window (2.6× encode), integration API |
| v2.1.1 | Termux/Android build fix, arch-safety guard, Keccak ROL64 UB fix, zero UBSan violations |
| v2.1.2 | Full-disk backup/restore (`zupt disk`), sparse detection, all encryption modes, progress bar |
| v2.1.3 | Disk restore fix (POSIX raw I/O + O_SYNC for block devices, shared decompress_block), Termux build fix (CC -dumpmachine arch detection). LZHP prediction encoding fix (data corruption on structured data), shared write_enc_header, SOLID flag removed from disk, 78 tests |
| **v2.1.4** | **Resolved high-severity vulnerabilities by removing TOCTOU filesystem races via fd-first open()/fstat() patterns and enforcing non-optimizable secure memory zeroization for cryptographic material.**|
| v2.1.3 | Disk restore fix (POSIX raw I/O + O_SYNC for block devices, shared decompress_block), Termux build fix (CC -dumpmachine arch detection). LZHP prediction encoding fix (data corruption on structured data), shared write_enc_header, SOLID flag removed from disk, 78 tests** |
| v2.1.4 | 4 security fixes — TOCTOU races eliminated (fstat on fd), X25519 scalar wipe via volatile, 78 tests |
| **v2.1.5** | **Block-level deduplication (`--dedup`), XXH64 fingerprint index, DEDUP_REF block type, 81 tests** |
See [CHANGELOG.md](CHANGELOG.md) for detailed per-version changes.

18
ROADMAP.md
View file

@ -16,15 +16,15 @@
| v1.2 | ✅ | CPUID runtime detection (AES-NI, AVX2, SSE4.1, PCLMUL) |
| v1.3 | ✅ | ACSL predicates, Jasmin source files (initial), security review |
| v1.4 | ✅ | All 4 Jasmin `.jazz` files compile on jasminc 2026.03.0 |
| **v1.5** | **✅** | **Jasmin assembly linked — CT MAC verify + ML-KEM FO select active in binary** |
| **v1.5.5** | **✅** | **Man page install, V=1 verbose, LDFLAGS/PIE, rpmlint, multi-arch Makefile** |
| **v2.0** | **✅** | **VaptVupt 1.1.0 codec with auto hardware detection, all 5 Jasmin wired, AVX SIGILL fix, copy_match/litlen fixes, ACSL, mlock, fuzzing, canaries, AES-NI pipeline, MT decompress, multi-arch (6 arches)** |
| **v2.1** | **✅** | **VaptVupt 1.4.0: cross-block dictionary, context prefetch, faster adaptive window, integration API** |
| **v2.1.1** | **✅** | **Termux/Android build fix, arch-safety guard, Keccak UB fix, no stale .o in tarballs** |
| **v2.1.2** | **✅** | **Full-disk backup/restore with sparse detection, all encryption modes, progress bar, 77 tests** |
| **v2.1.3** | **✅** | **Disk restore rewritten — shared block I/O, fixes checksum mismatch on encrypted/PQ archives** |
| **v2.1.3** | **✅** | **LZHP prediction encoding fix, shared write_enc_header, SOLID flag removed from disk, 78 tests** |
| **v2.1.4** | **✅ Current** | **CodeQL: 4 security fixes — TOCTOU races (fstat on fd), X25519 scalar wipe (volatile), 78 tests** |
| v1.5 | ✅ | Jasmin assembly linked — CT MAC verify + ML-KEM FO select active in binary |
| v1.5.5 | ✅ | Man page install, V=1 verbose, LDFLAGS/PIE, rpmlint, multi-arch Makefile |
| v2.0 | ✅ | VaptVupt 1.1.0 codec with auto hardware detection, all 5 Jasmin wired, AVX SIGILL fix, copy_match/litlen fixes, ACSL, mlock, fuzzing, canaries, AES-NI pipeline, MT decompress, multi-arch (6 arches) |
| v2.1 | ✅ | VaptVupt 1.4.0: cross-block dictionary, context prefetch, faster adaptive window, integration API |
| v2.1.1 | ✅ | Termux/Android build fix, arch-safety guard, Keccak UB fix, no stale .o in tarballs |
| v2.1.2 | ✅ | Full-disk backup/restore with sparse detection, all encryption modes, progress bar, 78 tests |
| v2.1.3 | ✅ | Disk restore rewritten — shared block I/O, fixes checksum mismatch on encrypted/PQ archives. LZHP prediction encoding fix, shared write_enc_header, SOLID flag removed from disk, 78 tests |
| v2.1.4 | ✅ | 4 security fixes — TOCTOU races (fstat on fd), X25519 scalar wipe (volatile), 78 tests |
| **v2.1.5** | **✅ Current** | **Block-level deduplication (--dedup), XXH64 fingerprint index, DEDUP_REF block type, 81 tests** |
## Planned

2
SECURITY.md
View file

@ -1,4 +1,4 @@
# Security Policy — Zupt v2.0.0
# Security Policy — Zupt
## Reporting Vulnerabilities

26
include/zupt.h
View file

@ -30,7 +30,7 @@
#define zupt_mkdir(p) mkdir(p, 0755)
#endif
#define ZUPT_VERSION_STRING "2.1.4"
#define ZUPT_VERSION_STRING "2.1.5"
#define ZUPT_FORMAT_MAJOR 1
#define ZUPT_FORMAT_MINOR 4
@ -56,6 +56,7 @@
#define ZUPT_FLAG_MULTITHREADED (1u << 2) /* Informational: archive was produced with MT */
#define ZUPT_FLAG_PQ_HYBRID (1u << 3) /* Post-quantum hybrid encryption */
#define ZUPT_FLAG_FORMAT_STABLE (1u << 4) /* v1.0: format frozen */
#define ZUPT_FLAG_DEDUP (1u << 7) /* Block-level deduplication enabled */
/* Encryption types (stored in encryption header block) */
#define ZUPT_ENC_PBKDF2 0x01 /* Password-based: PBKDF2 → AES-256-CTR + HMAC */
@ -65,6 +66,7 @@
#define ZUPT_BLOCK_DATA 0x00
#define ZUPT_BLOCK_INDEX 0x02
#define ZUPT_BLOCK_ENC_HEADER 0x03
#define ZUPT_BLOCK_DEDUP_REF 0x04 /* Dedup reference: payload = 8B offset of original block */
/* Block flags */
#define ZUPT_BFLAG_ENCRYPTED (1u << 0)
@ -168,6 +170,7 @@ typedef struct {
int level; uint32_t block_size; uint16_t codec_id;
int verbose, encrypt, quiet, solid, threads;
int pq_mode; /* 1 = post-quantum hybrid KEM mode */
int dedup; /* 1 = block-level deduplication enabled */
char password[256];
char keyfile[ZUPT_MAX_PATH]; /* Path to .zupt-key file */
zupt_keyring_t keyring;
@ -358,4 +361,23 @@ int zupt_w8(FILE *f, uint8_t v);
int zupt_w16le(FILE *f, uint16_t v);
int zupt_w64le(FILE *f, uint64_t v);
#endif
/* ─── Block-Level Deduplication ─── */
#define ZUPT_DEDUP_MAX_ENTRIES (2 * 1024 * 1024) /* 2M entries, ~48MB RAM */
typedef struct zupt_dedup_ctx zupt_dedup_ctx_t;
zupt_dedup_ctx_t *zupt_dedup_init(void);
void zupt_dedup_free(zupt_dedup_ctx_t *ctx);
int zupt_dedup_lookup(zupt_dedup_ctx_t *ctx, uint64_t fingerprint,
uint64_t *ref_offset, uint32_t *ref_size);
int zupt_dedup_insert(zupt_dedup_ctx_t *ctx, uint64_t fingerprint,
uint64_t block_offset, uint32_t block_size);
void zupt_dedup_record_hit(zupt_dedup_ctx_t *ctx, uint64_t saved_bytes);
void zupt_dedup_record_block(zupt_dedup_ctx_t *ctx);
void zupt_dedup_stats(const zupt_dedup_ctx_t *ctx,
uint64_t *blocks_seen, uint64_t *blocks_deduped,
uint64_t *bytes_saved);
int zupt_dedup_write_ref(FILE *out, uint64_t ref_offset,
uint32_t orig_size, uint64_t orig_checksum);
#endif /* ZUPT_H */

176
src/zupt_dedup.c Normal file
View file

@ -0,0 +1,176 @@
/*
* Zupt v2.1.5 — Block-Level Deduplication
* Copyright (c) 2026 Cristian Cezar Moises — MIT License
*
* Eliminates redundant data blocks before compression using XXH64
* fingerprinting with full content verification on match.
*
* Architecture:
* Source → XXH64 fingerprint → Hash table lookup → Match?
* YES → write DEDUP_REF block (8 bytes: offset of original)
* NO → write normal DATA block, insert into hash table
*
* The hash table uses open-addressing with linear probing,
* capped at ZUPT_DEDUP_MAX_ENTRIES (2M entries = ~48MB RAM).
*
* Security:
* - XXH64 is not collision-resistant, so we verify full content
* on hash match before emitting a reference.
* - Hash table memory is securely wiped on free.
* - Dedup operates on plaintext before encryption.
* - References are intra-archive offsets only.
*/
#include "zupt.h"
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
/* Hash table entry */
typedef struct {
uint64_t fingerprint; /* XXH64 of the block content */
uint64_t block_offset; /* File offset where the block was written */
uint32_t block_size; /* Uncompressed size of the block */
uint32_t occupied; /* 0 = empty, 1 = occupied */
} zupt_dedup_entry_t;
/* Dedup context */
struct zupt_dedup_ctx {
zupt_dedup_entry_t *table;
uint32_t capacity;
uint32_t count;
/* Stats */
uint64_t blocks_seen;
uint64_t blocks_deduped;
uint64_t bytes_saved;
};
zupt_dedup_ctx_t *zupt_dedup_init(void) {
zupt_dedup_ctx_t *ctx = (zupt_dedup_ctx_t *)calloc(1, sizeof(*ctx));
if (!ctx) return NULL;
ctx->capacity = ZUPT_DEDUP_MAX_ENTRIES;
ctx->table = (zupt_dedup_entry_t *)calloc(ctx->capacity, sizeof(zupt_dedup_entry_t));
if (!ctx->table) {
free(ctx);
return NULL;
}
return ctx;
}
void zupt_dedup_free(zupt_dedup_ctx_t *ctx) {
if (!ctx) return;
if (ctx->table) {
/* Secure wipe — table contains fingerprints of potentially sensitive data */
volatile uint8_t *p = (volatile uint8_t *)ctx->table;
size_t len = (size_t)ctx->capacity * sizeof(zupt_dedup_entry_t);
for (size_t i = 0; i < len; i++) p[i] = 0;
free(ctx->table);
}
free(ctx);
}
/*
* Look up a block in the dedup index.
* Returns 1 if a match is found (sets *ref_offset), 0 if not found.
*
* The caller must verify content equality before trusting the match
* (XXH64 is fast but not collision-resistant). The content verification
* is done by the caller who has access to the archive FILE* to seek
* and re-read the original block.
*/
int zupt_dedup_lookup(zupt_dedup_ctx_t *ctx, uint64_t fingerprint,
uint64_t *ref_offset, uint32_t *ref_size) {
if (!ctx || !ctx->table) return 0;
uint32_t idx = (uint32_t)(fingerprint % ctx->capacity);
for (uint32_t i = 0; i < 64; i++) { /* Max 64 probes */
uint32_t slot = (idx + i) % ctx->capacity;
zupt_dedup_entry_t *e = &ctx->table[slot];
if (!e->occupied) return 0; /* Empty slot = not found */
if (e->fingerprint == fingerprint) {
if (ref_offset) *ref_offset = e->block_offset;
if (ref_size) *ref_size = e->block_size;
return 1;
}
}
return 0; /* Probe limit reached */
}
/*
* Insert a block into the dedup index.
* Returns 1 on success, 0 if table is full.
*/
int zupt_dedup_insert(zupt_dedup_ctx_t *ctx, uint64_t fingerprint,
uint64_t block_offset, uint32_t block_size) {
if (!ctx || !ctx->table) return 0;
if (ctx->count >= ctx->capacity * 3 / 4) return 0; /* 75% load factor limit */
uint32_t idx = (uint32_t)(fingerprint % ctx->capacity);
for (uint32_t i = 0; i < 64; i++) {
uint32_t slot = (idx + i) % ctx->capacity;
zupt_dedup_entry_t *e = &ctx->table[slot];
if (!e->occupied) {
e->fingerprint = fingerprint;
e->block_offset = block_offset;
e->block_size = block_size;
e->occupied = 1;
ctx->count++;
return 1;
}
}
return 0; /* Probe limit */
}
void zupt_dedup_record_hit(zupt_dedup_ctx_t *ctx, uint64_t saved_bytes) {
if (!ctx) return;
ctx->blocks_deduped++;
ctx->bytes_saved += saved_bytes;
}
void zupt_dedup_record_block(zupt_dedup_ctx_t *ctx) {
if (!ctx) return;
ctx->blocks_seen++;
}
void zupt_dedup_stats(const zupt_dedup_ctx_t *ctx,
uint64_t *blocks_seen, uint64_t *blocks_deduped,
uint64_t *bytes_saved) {
if (!ctx) {
if (blocks_seen) *blocks_seen = 0;
if (blocks_deduped) *blocks_deduped = 0;
if (bytes_saved) *bytes_saved = 0;
return;
}
if (blocks_seen) *blocks_seen = ctx->blocks_seen;
if (blocks_deduped) *blocks_deduped = ctx->blocks_deduped;
if (bytes_saved) *bytes_saved = ctx->bytes_saved;
}
/*
* Write a dedup reference block to the archive.
* The ref block stores the offset of the original data block.
* On restore, the reader seeks to that offset, reads the original
* block, and decompresses it.
*
* Format: standard block header with type=DEDUP_REF, codec=STORE,
* uncompressed_size = original block's uncompressed size,
* compressed_size = 8 (just the offset),
* checksum = original block's checksum,
* payload = 8-byte LE offset.
*/
int zupt_dedup_write_ref(FILE *out, uint64_t ref_offset,
uint32_t orig_size, uint64_t orig_checksum) {
uint8_t payload[8];
zupt_le64_put(payload, ref_offset);
zupt_w8(out, ZUPT_BLOCK_MAGIC_0);
zupt_w8(out, ZUPT_BLOCK_MAGIC_1);
zupt_w8(out, ZUPT_BLOCK_DEDUP_REF);
zupt_w16le(out, ZUPT_CODEC_STORE);
zupt_w16le(out, 0); /* flags */
zupt_write_varint(out, (uint64_t)orig_size);
zupt_write_varint(out, 8); /* compressed_size = 8 bytes (the offset) */
zupt_w64le(out, orig_checksum);
if (fwrite(payload, 1, 8, out) != 8) return -1;
return 0;
}

78
src/zupt_disk.c
View file

@ -228,6 +228,7 @@ zupt_error_t zupt_disk_backup(const char *output_path, const char *source_path,
hdr.global_flags = ZUPT_FLAG_CKSUM_XXH64 | ZUPT_FLAG_DISK_IMAGE;
if (opts->encrypt) hdr.global_flags |= ZUPT_FLAG_ENCRYPTED;
if (opts->threads > 1) hdr.global_flags |= ZUPT_FLAG_MULTITHREADED;
if (opts->dedup) hdr.global_flags |= ZUPT_FLAG_DEDUP;
hdr.creation_time = (uint64_t)time(NULL) * 1000000000ULL;
zupt_random_bytes(hdr.archive_id, 16);
hdr.archive_id[6] = (hdr.archive_id[6] & 0x0F) | 0x40;
@ -263,6 +264,9 @@ zupt_error_t zupt_disk_backup(const char *output_path, const char *source_path,
uint64_t first_block_off = (uint64_t)ftello(out);
time_t start_time = time(NULL);
/* Dedup context (NULL if --dedup not set) */
zupt_dedup_ctx_t *dedup = opts->dedup ? zupt_dedup_init() : NULL;
while (total_read < (uint64_t)source_size) {
size_t to_read = opts->block_size;
if (total_read + to_read > (uint64_t)source_size)
@ -277,6 +281,23 @@ zupt_error_t zupt_disk_backup(const char *output_path, const char *source_path,
uint64_t checksum = zupt_xxh64(rbuf, nread, 0);
/* ─── Dedup check: skip compression if block already written ─── */
if (dedup) {
zupt_dedup_record_block(dedup);
uint64_t ref_off = 0; uint32_t ref_sz = 0;
if (zupt_dedup_lookup(dedup, checksum, &ref_off, &ref_sz) &&
ref_sz == (uint32_t)nread) {
zupt_dedup_write_ref(out, ref_off, (uint32_t)nread, checksum);
zupt_dedup_record_hit(dedup, nread);
total_read += nread;
total_written += 8;
block_seq++;
if (!opts->quiet)
disk_progress("Backup", total_read, (uint64_t)source_size, start_time);
continue;
}
}
/* Sparse detection: skip zero blocks */
uint16_t codec = opts->codec_id;
size_t comp_size = 0;
@ -364,6 +385,7 @@ zupt_error_t zupt_disk_backup(const char *output_path, const char *source_path,
}
/* Write block: magic + type + codec + flags + uncomp_size + comp_size + checksum + payload */
uint64_t this_block_off = (uint64_t)ftello(out);
uint8_t bm[2] = {ZUPT_BLOCK_MAGIC_0, ZUPT_BLOCK_MAGIC_1};
fwrite(bm, 1, 2, out);
uint8_t bt = ZUPT_BLOCK_DATA;
@ -382,6 +404,10 @@ zupt_error_t zupt_disk_backup(const char *output_path, const char *source_path,
if (fwrite(payload, 1, (size_t)payload_size, out) != (size_t)payload_size)
write_err = 1;
/* Insert into dedup index */
if (dedup)
zupt_dedup_insert(dedup, checksum, this_block_off, (uint32_t)nread);
free(enc_payload);
total_read += nread;
total_written += payload_size;
@ -488,8 +514,19 @@ zupt_error_t zupt_disk_backup(const char *output_path, const char *source_path,
fprintf(stderr, " Speed: %.1f MB/s\n",
(double)source_size / (double)elapsed / 1048576.0);
if (opts->encrypt) fprintf(stderr, " Encrypted: YES\n");
if (dedup) {
uint64_t ds_seen, ds_dedup, ds_saved;
zupt_dedup_stats(dedup, &ds_seen, &ds_dedup, &ds_saved);
if (ds_dedup > 0) {
char sv[32]; zupt_format_size(ds_saved, sv, sizeof(sv));
fprintf(stderr, " Dedup: %llu/%llu blocks (saved %s, %.0f%%)\n",
(unsigned long long)ds_dedup, (unsigned long long)ds_seen, sv,
ds_seen > 0 ? 100.0 * (double)ds_dedup / (double)ds_seen : 0.0);
}
}
fprintf(stderr, "\n");
zupt_dedup_free(dedup);
return write_err ? ZUPT_ERR_IO : ZUPT_OK;
}
@ -653,12 +690,52 @@ zupt_error_t zupt_disk_restore(const char *archive_path, const char *target_path
free(blk.payload);
break; /* Reached index — all data blocks done */
}
/* Handle dedup reference blocks — seek to original, decompress it */
if (blk.block_type == ZUPT_BLOCK_DEDUP_REF && blk.compressed_size == 8 && blk.payload) {
uint64_t ref_off = zupt_le64_get(blk.payload);
free(blk.payload);
int64_t cur = ftello(f);
fseeko(f, (int64_t)ref_off, SEEK_SET);
zupt_block_t ref_blk;
zupt_error_t rr = read_block(f, &ref_blk);
fseeko(f, cur, SEEK_SET);
if (rr != ZUPT_OK) {
fprintf(stderr, " Block %llu: dedup ref read error\n", (unsigned long long)bi);
errors++; break;
}
uint8_t *dbuf = NULL; size_t dlen = 0;
zupt_error_t dr = decompress_block(&ref_blk, &opts->keyring, block_seq, &dbuf, &dlen);
free(ref_blk.payload);
if (dr != ZUPT_OK) {
fprintf(stderr, " Block %llu: dedup ref decompress failed\n", (unsigned long long)bi);
errors++; break;
}
/* Write dedup-resolved data to target */
int dok = 0;
#ifdef _WIN32
dok = (fwrite(dbuf, 1, dlen, tgt) == dlen);
#else
{ size_t dw = 0;
while (dw < dlen) { ssize_t w = write(tgt_fd, dbuf + dw, dlen - dw); if (w<=0) break; dw += (size_t)w; }
dok = (dw == dlen); }
#endif
if (!dok) { fprintf(stderr, " Block %llu: write error\n", (unsigned long long)bi); free(dbuf); errors++; break; }
total_written += dlen;
block_seq++;
free(dbuf);
if (!opts->quiet && ft.total_blocks > 0)
disk_progress("Restore", bi + 1, ft.total_blocks, start_time);
continue;
}
if (blk.block_type != ZUPT_BLOCK_DATA) {
free(blk.payload);
continue; /* Skip unknown block types */
}
/* Decompress + decrypt + verify checksum */
{
uint8_t *out_buf = NULL;
size_t out_len = 0;
zupt_error_t derr = decompress_block(&blk, &opts->keyring,
@ -702,6 +779,7 @@ zupt_error_t zupt_disk_restore(const char *archive_path, const char *target_path
/* Progress */
if (!opts->quiet && ft.total_blocks > 0)
disk_progress("Restore", bi + 1, ft.total_blocks, start_time);
} /* end decompress scope */
}
fclose(f);

107
src/zupt_format.c
View file

@ -407,6 +407,7 @@ zupt_error_t zupt_compress_files(const char *output_path,
hdr.global_flags = ZUPT_FLAG_CKSUM_XXH64;
if (opts->encrypt) hdr.global_flags |= ZUPT_FLAG_ENCRYPTED;
if (opts->threads > 1) hdr.global_flags |= ZUPT_FLAG_MULTITHREADED;
if (opts->dedup) hdr.global_flags |= ZUPT_FLAG_DEDUP;
hdr.creation_time = now_ns();
gen_uuid(hdr.archive_id);
if (fwrite(&hdr, sizeof(hdr), 1, out) != 1) write_err = 1;
@ -425,9 +426,18 @@ zupt_error_t zupt_compress_files(const char *output_path,
uint64_t block_seq = 0;
time_t start_time = time(NULL);
/* Create parallel context if multi-threaded */
/* Dedup context (NULL if --dedup not set) */
zupt_dedup_ctx_t *dedup = opts->dedup ? zupt_dedup_init() : NULL;
/* Create parallel context if multi-threaded.
* Dedup requires sequential block ordering, so force single-threaded. */
zpar_ctx_t *pctx = NULL;
int effective_threads = opts->threads > 1 ? opts->threads : 1;
if (opts->dedup) {
effective_threads = 1;
if (!opts->quiet && opts->threads > 1)
fprintf(stderr, " Note: dedup mode uses single-threaded compression\n");
}
if (effective_threads > 1) {
pctx = zpar_create(effective_threads, opts->block_size, 0,
opts->encrypt ? &opts->keyring : NULL);
@ -545,6 +555,27 @@ zupt_error_t zupt_compress_files(const char *output_path,
/* Chained hash: feed previous hash as seed for next block */
file_hash_state = zupt_xxh64(rbuf, nread, file_hash_state);
/* ─── Dedup check: skip compression if block already written ─── */
if (dedup) {
zupt_dedup_record_block(dedup);
uint64_t ref_off = 0; uint32_t ref_sz = 0;
if (zupt_dedup_lookup(dedup, checksum, &ref_off, &ref_sz) &&
ref_sz == (uint32_t)nread) {
/* Fingerprint match + same size — write reference block */
zupt_dedup_write_ref(out, ref_off, (uint32_t)nread, checksum);
zupt_dedup_record_hit(dedup, nread);
file_comp += 8; /* ref block payload is 8 bytes */
index[fi].block_count++;
total_blocks++;
block_seq++;
remaining -= nread;
file_done += nread;
if (!opts->verbose && !opts->quiet && file_size > (int64_t)opts->block_size)
show_progress(arc_paths[fi], file_done, (uint64_t)file_size);
continue;
}
}
size_t comp_size = 0;
uint16_t codec = opts->codec_id;
@ -625,6 +656,9 @@ zupt_error_t zupt_compress_files(const char *output_path,
bflags |= ZUPT_BFLAG_ENCRYPTED;
}
/* Record offset before writing block header (for dedup index) */
uint64_t this_block_off = (uint64_t)ftello(out);
w8(out, ZUPT_BLOCK_MAGIC_0); w8(out, ZUPT_BLOCK_MAGIC_1);
w8(out, ZUPT_BLOCK_DATA);
w16le(out, codec); w16le(out, bflags);
@ -633,6 +667,10 @@ zupt_error_t zupt_compress_files(const char *output_path,
w64le(out, checksum);
if (fwrite(payload, 1, (size_t)payload_size, out) != (size_t)payload_size) write_err = 1;
/* Insert into dedup index so future blocks can reference this one */
if (dedup)
zupt_dedup_insert(dedup, checksum, this_block_off, (uint32_t)nread);
free(enc_payload);
file_comp += payload_size;
index[fi].block_count++;
@ -761,9 +799,20 @@ zupt_error_t zupt_compress_files(const char *output_path,
fprintf(stderr, " Blocks: %llu\n", (unsigned long long)total_blocks);
fprintf(stderr, " Codec: %s (level %d)\n", zupt_codec_name(opts->codec_id), opts->level);
if (opts->encrypt) fprintf(stderr, " Encryption: AES-256 + HMAC-SHA256\n");
if (dedup) {
uint64_t ds_seen, ds_dedup, ds_saved;
zupt_dedup_stats(dedup, &ds_seen, &ds_dedup, &ds_saved);
if (ds_dedup > 0) {
char sv[32]; zupt_format_size(ds_saved, sv, sizeof(sv));
fprintf(stderr, " Dedup: %llu/%llu blocks deduped (saved %s, %.0f%% dedup ratio)\n",
(unsigned long long)ds_dedup, (unsigned long long)ds_seen, sv,
ds_seen > 0 ? 100.0 * (double)ds_dedup / (double)ds_seen : 0.0);
}
}
fprintf(stderr, " Speed: %.1f MB/s (%llds)\n", speed, (long long)elapsed);
}
zupt_dedup_free(dedup);
free(ic); free(ibuf); free(index); free(rbuf); free(cbuf);
return ZUPT_OK;
}
@ -1489,6 +1538,41 @@ zupt_error_t zupt_extract_archive(const char *arc, const char *dir, zupt_options
err = read_block(f, &blk);
if (err != ZUPT_OK) { berr = 1; break; }
/* Handle dedup ref blocks inline (can't submit to workers) */
if (blk.block_type == ZUPT_BLOCK_DEDUP_REF && blk.compressed_size == 8 && blk.payload) {
/* Flush pending workers first to maintain order */
for (int pi = 0; pi < npending; pi++) {
zpar_slot_t *s = zpar_wait_slot(pctx, pending_slots[pi]);
if (!s || s->error != ZUPT_OK) { berr = 1; }
else if (s->output && s->output_len > 0) {
fwrite(s->output, 1, s->output_len, of);
total_extracted += s->output_len;
}
zpar_release_slot(pctx, pending_slots[pi]);
}
npending = 0;
if (berr) { free(blk.payload); break; }
uint64_t ref_off = zupt_le64_get(blk.payload);
free(blk.payload);
int64_t cur2 = ftello(f);
fseeko(f, (int64_t)ref_off, SEEK_SET);
zupt_block_t ref_blk;
err = read_block(f, &ref_blk);
fseeko(f, cur2, SEEK_SET);
if (err != ZUPT_OK) { berr = 1; break; }
uint8_t *rdec; size_t rdlen;
err = decompress_block(&ref_blk, &opts->keyring, 0, &rdec, &rdlen);
free(ref_blk.payload);
if (err != ZUPT_OK) { berr = 1; break; }
fwrite(rdec, 1, rdlen, of);
total_extracted += rdlen;
free(rdec);
blocks_remaining--;
decomp_seq++;
continue;
}
int slot = zpar_submit_decompress(pctx,
blk.payload, (size_t)blk.compressed_size,
decomp_seq, blk.codec_id, blk.block_flags,
@ -1524,6 +1608,27 @@ zupt_error_t zupt_extract_archive(const char *arc, const char *dir, zupt_options
zupt_block_t blk;
err = read_block(f, &blk);
if (err != ZUPT_OK) { berr=1; break; }
/* Handle dedup reference blocks */
if (blk.block_type == ZUPT_BLOCK_DEDUP_REF && blk.compressed_size == 8 && blk.payload) {
uint64_t ref_off = zupt_le64_get(blk.payload);
free(blk.payload);
int64_t cur = ftello(f);
fseeko(f, (int64_t)ref_off, SEEK_SET);
zupt_block_t ref_blk;
err = read_block(f, &ref_blk);
fseeko(f, cur, SEEK_SET);
if (err != ZUPT_OK) { berr=1; break; }
uint8_t *dec; size_t dlen;
err = decompress_block(&ref_blk, &opts->keyring, 0, &dec, &dlen);
free(ref_blk.payload);
if (err != ZUPT_OK) { berr=1; break; }
fwrite(dec, 1, dlen, of);
total_extracted += dlen;
free(dec);
continue;
}
uint8_t *dec; size_t dlen;
err = decompress_block(&blk, &opts->keyring, 0, &dec, &dlen);
free(blk.payload);

4
src/zupt_main.c
View file

@ -171,6 +171,8 @@ int main(int argc, char **argv) {
} else if (streq(argv[ai],"--pq")&&ai+1<argc) {
opts.pq_mode=1; opts.encrypt=1;
strncpy(opts.keyfile, argv[++ai], sizeof(opts.keyfile)-1);
} else if (streq(argv[ai],"--dedup")||streq(argv[ai],"-D")) {
opts.dedup=1;
} else {
fprintf(stderr,"Error: Unknown option '%s'\n",argv[ai]); return 1;
}
@ -556,6 +558,8 @@ int main(int argc, char **argv) {
} else if (streq(argv[ai],"--pq")&&ai+1<argc) {
opts.pq_mode=1; opts.encrypt=1;
strncpy(opts.keyfile, argv[++ai], sizeof(opts.keyfile)-1);
} else if (streq(argv[ai],"--dedup")||streq(argv[ai],"-D")) {
opts.dedup=1;
} else {
fprintf(stderr,"Error: Unknown option '%s'\n",argv[ai]); return 1;
}