feat: transparently append to compressed archives
When appending to a compressed archive (gzip, brotli, zstd), the tool now handles compression automatically. Since some compression formats don't support appending to compressed files in place, we write a new compressed file with all the data and atomically rename it to replace the original (assuming there is enough space on that filesystem). This means you can work with compressed archives the same way as uncompressed ones. Point the tool at your .json.gz file and append values. No manual decompression/recompression needed.
This commit is contained in:
parent
da0fed29de
commit
2ab1c31993
34 changed files with 4747 additions and 1099 deletions
33
CLAUDE.md
33
CLAUDE.md
|
|
@ -1,6 +1,37 @@
|
|||
## Running Tests
|
||||
|
||||
### Unit tests
|
||||
|
||||
```bash
|
||||
cargo test # Run without compression support
|
||||
cargo test --features compression # Run with compression support
|
||||
```
|
||||
|
||||
### Integration tests
|
||||
|
||||
The compression integration tests verify appending to compressed archives (gzip, brotli, zstd):
|
||||
|
||||
```bash
|
||||
cd tests/compression-integration
|
||||
./run_all.sh # Run all compression tests
|
||||
./run_gzip_test.sh # Run only gzip test
|
||||
./run_brotli_test.sh # Run only brotli test
|
||||
./run_zstd_test.sh # Run only zstd test
|
||||
```
|
||||
|
||||
These tests:
|
||||
1. Create an uncompressed archive from the first state file
|
||||
2. Compress it with the respective tool (gzip/brotli/zstd)
|
||||
3. Append additional state files to the compressed archive
|
||||
4. Verify the archive can be read and shows the correct observation count
|
||||
|
||||
Requirements: Python 3 (for test data generation), gzip, brotli, zstd command-line tools.
|
||||
|
||||
### Fuzz testing
|
||||
|
||||
To begin fuzzing, run: `cargo fuzz run <fuzz target name>`
|
||||
|
||||
The source code for a fuzz target by default lives in `fuzz/fuzz_targets/<fuzz target name>.rs`.
|
||||
The source code for a fuzz target by default lives in `fuzz/fuzz_targets/<fuzz target name>.rs`.
|
||||
|
||||
Each fuzz target is a Rust program that is given random data and tests a crate (in this case, json-archive). Use `cargo fuzz list` to view the list of all existing fuzz targets:
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue