this post was submitted on 21 May 2024
63 points (98.5% liked)

Rust

5989 readers
53 users here now

Welcome to the Rust community! This is a place to discuss about the Rust programming language.

Wormhole

!performance@programming.dev

Credits

  • The icon is a modified version of the official rust logo (changing the colors to a gradient and black background)

founded 1 year ago
MODERATORS
snowe@programming.dev
Ategon@programming.dev
EdTheLegendary@programming.dev
kahnclusions@programming.dev
torcherist@programming.dev
63
I just released my first OSS library! Introducing Aqueducts, a framework to build ETL pipelines using rust (github.com)
submitted 5 months ago by kato@programming.dev to c/rust@programming.dev
8 comments fedilink hide all child comments
 

This is my first try at anything open source so any feedback is welcome :)

you are viewing a single comment's thread
view the rest of the comments
[โ€“] sem@lemmy.ml 4 points 5 months ago (1 children)

Is it a wrapper on top of datafusion, but with an ability to define transformations in yaml? I mean it looks really cool, I'm just trying to understand the usecase

[โ€“] kato@programming.dev 5 points 5 months ago

Basically yes. The usecases I have found so far at work is to build an API around this to dynamically register automatic reports for data analysts, clients and non devs. In general this also greatly speeds up dev time for any ETL that we need to deploy (am part of a data engineering team). Another usecase I found is that using the CLI tool we can create run books for our SRE team to run queries for debugging/data validation purposes. I think we'll find more as we go but another part of it was to simplify working with datafusion and deltalake as their APIs expose a lot of lower level stuff.