oss/json-archive
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
json-archive/tests/compression-integration/README.md
nobody 2ab1c31993
 feat: transparently append to compressed archives 
When appending to a compressed archive (gzip, brotli, zstd), the tool
now handles compression automatically. Since some compression formats don't
support appending to compressed files in place, we write a new
compressed file with all the data and atomically rename it to replace
the original (assuming there is enough space on that filesystem).

This means you can work with compressed archives the same way as
uncompressed ones. Point the tool at your .json.gz file and append
values. No manual decompression/recompression needed.
2025-11-30 17:09:49 -08:00

2.1 KiB
Raw Blame History

Compression Integration Tests

Manual integration tests for compressed archive functionality.

These scripts exercise the tool's ability to:

  1. Read archives that were compressed by external programs (gzip, brotli, zstd)
  2. Append new observations to compressed archives
  3. Produce correct results whether reading compressed or uncompressed

Scripts

generate_state.py <n>

Generates a JSON state file with n items in each array. Output goes to stdout.

./generate_state.py 3
# Output: {"colors":["color_1","color_2","color_3"],"numbers":["number_1","number_2","number_3"],"animals":["animal_1","animal_2","animal_3"]}

generate_state_files.py <count> <output_dir>

Generates a series of state files (state_1.json through state_N.json) with progressively more items.

./generate_state_files.py 9 ./data
# Creates: data/state_1.json, data/state_2.json, ... data/state_9.json

run_gzip_test.sh

Tests the gzip compression workflow:

  1. Create archive from first state file
  2. Compress with gzip
  3. Append remaining 8 state files to the compressed archive
  4. Decompress and inspect

run_brotli_test.sh

Same workflow but with brotli compression.

run_zstd_test.sh

Same workflow but with zstd compression.

run_all.sh

Runs all compression tests in sequence.

validate.sh (optional)

Smoke test to verify the final state matches expectations.

Usage

cd tests/compression-integration

# Run all tests (generates data, builds, runs all compression formats)
./run_all.sh

# Or run individual steps:
./generate_state_files.py 9 ./data
./run_gzip_test.sh
./run_brotli_test.sh
./run_zstd_test.sh

# Optional: validate outputs match
./validate.sh

What to look for

After running the tests, you can manually verify:

  1. The compressed archives were created
  2. Appending to compressed archives worked (check file sizes grew)
  3. The info command shows the same observation count for compressed and decompressed versions
  4. The state command returns the same final state

Dependencies

  • gzip (usually pre-installed)
  • brotli (brew install brotli)
  • zstd (brew install zstd)