nobody
2ab1c31993
When appending to a compressed archive (gzip, brotli, zstd), the tool now handles compression automatically. Since some compression formats don't support appending to compressed files in place, we write a new compressed file with all the data and atomically rename it to replace the original (assuming there is enough space on that filesystem). This means you can work with compressed archives the same way as uncompressed ones. Point the tool at your .json.gz file and append values. No manual decompression/recompression needed.
78 lines
2.1 KiB
Markdown
78 lines
2.1 KiB
Markdown
# Compression Integration Tests
|
|
|
|
Manual integration tests for compressed archive functionality.
|
|
|
|
These scripts exercise the tool's ability to:
|
|
1. Read archives that were compressed by external programs (gzip, brotli, zstd)
|
|
2. Append new observations to compressed archives
|
|
3. Produce correct results whether reading compressed or uncompressed
|
|
|
|
## Scripts
|
|
|
|
### `generate_state.py <n>`
|
|
Generates a JSON state file with `n` items in each array. Output goes to stdout.
|
|
|
|
```bash
|
|
./generate_state.py 3
|
|
# Output: {"colors":["color_1","color_2","color_3"],"numbers":["number_1","number_2","number_3"],"animals":["animal_1","animal_2","animal_3"]}
|
|
```
|
|
|
|
### `generate_state_files.py <count> <output_dir>`
|
|
Generates a series of state files (state_1.json through state_N.json) with progressively more items.
|
|
|
|
```bash
|
|
./generate_state_files.py 9 ./data
|
|
# Creates: data/state_1.json, data/state_2.json, ... data/state_9.json
|
|
```
|
|
|
|
### `run_gzip_test.sh`
|
|
Tests the gzip compression workflow:
|
|
1. Create archive from first state file
|
|
2. Compress with gzip
|
|
3. Append remaining 8 state files to the compressed archive
|
|
4. Decompress and inspect
|
|
|
|
### `run_brotli_test.sh`
|
|
Same workflow but with brotli compression.
|
|
|
|
### `run_zstd_test.sh`
|
|
Same workflow but with zstd compression.
|
|
|
|
### `run_all.sh`
|
|
Runs all compression tests in sequence.
|
|
|
|
### `validate.sh` (optional)
|
|
Smoke test to verify the final state matches expectations.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
cd tests/compression-integration
|
|
|
|
# Run all tests (generates data, builds, runs all compression formats)
|
|
./run_all.sh
|
|
|
|
# Or run individual steps:
|
|
./generate_state_files.py 9 ./data
|
|
./run_gzip_test.sh
|
|
./run_brotli_test.sh
|
|
./run_zstd_test.sh
|
|
|
|
# Optional: validate outputs match
|
|
./validate.sh
|
|
```
|
|
|
|
## What to look for
|
|
|
|
After running the tests, you can manually verify:
|
|
|
|
1. The compressed archives were created
|
|
2. Appending to compressed archives worked (check file sizes grew)
|
|
3. The `info` command shows the same observation count for compressed and decompressed versions
|
|
4. The `state` command returns the same final state
|
|
|
|
## Dependencies
|
|
|
|
- gzip (usually pre-installed)
|
|
- brotli (`brew install brotli`)
|
|
- zstd (`brew install zstd`)
|