We want to make Rust the best data engineering platform, with tools that span from development to production.
Below is just part of my imagination, hopefully we will be able to realize them some day!
sed for streams
Since we've talked about shells a lot, it will be great if we have an equivalent to
sed that allow us to filter, transform and combine structured messages with a scripting language.
Inter Process Communication
Right now pipe is the only mechanism for connecting processors together, which is copy-heavy. For some high-throughput scenario, sharded memory could fair better.
Inter Host Communication
Right now the only way to connect across host is to rely on a streaming server. In a containerized environment, it might make sense to allow processors to communicate privately via TCP streams.
We welcome you to join our Discussions if you have ideas or insights!