@itk-wasm/dam

Data Archive Manager (dam): a tool to prepare and download source code repository testing, example, and website asset data.

Data is stored in .tar.gz archives and can be hosted at multiple, redundant https URLs. A simpler, more flexible alternative to Git LFS, etc.

Features

• Download and unpack testing and example data .tar.gz archives
• CLI for NPM scripts
• Verify with an expected hash compatible with web3.storage
• Cache archive downloads based on the expected hash
• Support for multiple storage urls
• Automatic retries

Installation

npm install -g @itk-wasm/dam

Usage

Store your data in a directory. For example:

> mkdir -p test/data
> touch test/data/file1.txt
> touch test/data/file2.txt

Pack the current version of the data into a compressed archive:

> dam pack test/data test/data.tar.gz
CID: bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy

Upload the test/data.tar.gz file to hosting services such as web3.storage, GitHub Releases, AWS s3, a local http server, etc. For web3.storage, upload with the w3 CLI:

> npm install --global @web3-storage/w3
> w3 token
> w3 put --name dam-test-data --no-wrap test/data.tar.gz
# Packed 1 file (0.0MB)
# bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy
⁂ Stored 1 file
⁂ https://w3s.link/ipfs/bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy

Note the --no-wrap flag.

In your NPM scripts, add

> dam download -v test/data test/data.tar.gz bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy https://w3s.link/ipfs/bafkreibetv3747ysuplkqkwlomeaao3nk3uak7wtzgrgmrtvmah5jdyawy

If the data is stored at multiple locations, add all URLs to the end of the download subcommand.

The first time this command is called the data will be downloaded and unpacked.

When data updates are required, repeat the pack / upload process, and update the content identifer (CID) hash.