sitegui/ingest-many-json-files
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
35 commits 1 branch 0 tags 434 KiB
Find a file
Exact
sitegui f104176c21 Measurements
2026-04-03 10:34:45 +02:00
local-s3 Generate larger files 2026-04-02 16:11:42 +02:00
rust Measurements 2026-04-03 10:34:45 +02:00
spark Measurements 2026-04-03 10:34:45 +02:00
.gitignore Improve some logs 2026-03-27 19:39:41 +01:00
README.md Measurements 2026-04-03 10:34:45 +02:00
rust-toolchain CLI to generate a bunch of files 2026-03-22 20:59:01 +01:00

README.md

Ingest many json files

WIP Results

tally uv run main.py --host http://192.168.1.15:9000
tally uv run main.py --host http://192.168.1.15:9000 --partitions 16
tally uv run main.py --host http://192.168.1.15:9000 --partitions 2

tally ./target/release/rust ingest --host http://192.168.1.15:9000 --target-size-mib 128
tally ./target/release/rust ingest --host http://192.168.1.15:9000 --target-size-mib 1024
Test Time (s) Mem (GiB) Parquet files
Spark 47.0 - 313
Spark 16 partitions 53.6 - 16
Spark 2 partitions 110.4 - 2
Rust target 128 MiB 21.3 1.4 12
Rust target 1 GiB 28.6 3.9 2