oss/json-archive
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: transparently append to compressed archives

Browse source 
When appending to a compressed archive (gzip, brotli, zstd), the tool
now handles compression automatically. Since some compression formats don't
support appending to compressed files in place, we write a new
compressed file with all the data and atomically rename it to replace
the original (assuming there is enough space on that filesystem).

This means you can work with compressed archives the same way as
uncompressed ones. Point the tool at your .json.gz file and append
values. No manual decompression/recompression needed.
This commit is contained in:
nobody 2025-11-30 17:09:44 -08:00
commit 2ab1c31993
Signed by: GrocerPublishAgent
GPG key ID: 43B1C298CDDE181C
34 changed files with 4747 additions and 1099 deletions
Download patch file Download diff file Expand all files Collapse all files

56
tests/compression-integration/run_zstd_test.sh Executable file
View file

@ -0,0 +1,56 @@
#!/usr/bin/env bash
#
# Test zstd compression workflow:
# 1. Create archive from first state file
# 2. Compress with zstd
# 3. Append remaining state files to the compressed archive
# 4. Decompress and show info
#
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_DIR="$(dirname "$(dirname "$SCRIPT_DIR")")"
BINARY="$PROJECT_DIR/target/debug/json-archive"
DATA_DIR="$SCRIPT_DIR/data"
OUT_DIR="$SCRIPT_DIR/out/zstd"
echo "=== Zstd Compression Test ==="
# Setup
rm -rf "$OUT_DIR"
mkdir -p "$OUT_DIR"
# Create initial archive from first state file
echo "Creating archive from state_1.json..."
"$BINARY" "$DATA_DIR/state_1.json" -o "$OUT_DIR/test.json.archive"
# Compress with zstd
echo "Compressing with zstd..."
zstd "$OUT_DIR/test.json.archive"
rm "$OUT_DIR/test.json.archive"
ls -la "$OUT_DIR/"
# Append remaining files to compressed archive
for i in $(seq 2 9); do
echo "Appending state_$i.json to compressed archive..."
"$BINARY" "$OUT_DIR/test.json.archive.zst" "$DATA_DIR/state_$i.json"
done
# Show info on the result
echo ""
echo "Final archive info:"
"$BINARY" info "$OUT_DIR/test.json.archive.zst"
# Decompress for manual inspection
echo ""
echo "Decompressing for comparison..."
zstd -d -k "$OUT_DIR/test.json.archive.zst"
echo ""
echo "Decompressed archive info:"
"$BINARY" info "$OUT_DIR/test.json.archive"
echo ""
echo "Files in $OUT_DIR:"
ls -la "$OUT_DIR/"