Skip to main content

69 posts tagged with "news"

View All Tags

What's new in SeaStreamer 0.3

Β· 5 min read
Chris Tsang
SeaQL Team

πŸŽ‰ We are pleased to release SeaStreamer 0.3.x!

File Backend​

A major addition in SeaStreamer 0.3 is the file backend. It implements the same high-level MPMC API, enabling streaming to and from files. There are different use cases. For example, it can be used to dump data from Redis / Kafka and process them locally, or as an intermediate file format for storage or transport.

The SeaStreamer File format, .ss is pretty simple. It's very much like .ndjson, but binary. The file format is designed with the following goals:

  1. Binary data support without encoding overheads
  2. Efficiency in rewinding / seeking through a large dump
  3. Streaming-friendliness - File can be truncated without losing integrity

Let me explain in details.

First of all, SeaStreamer File is a container format. It only concerns the message stream and framing, not the payload. It's designed to be paired with a binary message format like Protobuf or BSON.

Encode-free​

JSON and CSV are great plain text file formats, but they are not binary friendly. Usually, to encode binary data, one would use base64. It therefore imposes an expensive encoding / decoding overhead. In a binary protocol, delimiters are frequently used to signal message boundaries. As a consequence, byte stuffing is needed to escape the bytes.

In SeaStreamer, we want to avoid the encoding overhead entirely. The payload should be written to disk verbatim. So the file format revolves around constructing message frames and placing checksums to ensure that data is interpreted correctly.

Efficient seek​

A delimiter-based protocol has an advantage: the byte stream can be randomly sought, and we always have no trouble reading the next message.

Since SeaStreamer does not rely on delimiters, we can't easily align to message frames after a random seek. We solve this problem by placing beacons in a regular interval at fixed locations throughout the file. E.g. say the beacon interval is 1024, there will be a beacon at the 1024th byte, the 2048th, and so on. Then, every time we want to seek to a random location, we'd seek to the closest N * 1024 byte and read from there.

These beacons also double as indices: they contain summaries of the individual streams. So given a particular stream key and sequence number (or timestamp) to search for, SeaStreamer can quickly locate the message just by reading the beacons. It doesn't matter if the stream's messages are sparse!

Streaming-friendliness​

It should always be safe to truncate files. It should be relatively easy to split a file into chunks. We should be able to tell if the data is corrupted.

SeaStreamer achieves this by computing a checksum for every message, and also the running checksum of the checksums for each stream. It's not enforced right now, but in theory we can detect if any messages are missing from a stream.

Summary​

This file format is also easy to implement in different languages, as we just made an (experimental) reader in Typescript.

That's it! If you are interested, you can go and take a look at the format description.

Redis Backend​

Redis Streams are underrated! They have high throughput and concurrency, and are best suited for non-persistent stream processing near or on the same host as the application.

The obstacle is probably in library support. Redis Streams' API is rather low level, and there aren't many high-level libraries to help with programming, as opposed to Kafka, which has versatile official programming libraries.

The pitfall is, it's not easy to maximize concurrency with the raw Redis API. To start, you'd need to pipeline XADD commands. You'd also need to time and batch XACKs so that it does not block reads and computation. And of course you want to separate the reads and writes on different threads.

SeaStreamer breaks these obstacles for you and offers a Kafka-like API experience!

Benchmark​

In 0.3, we have done some optimizations to improve the throughput of the Redis and File backend. We set our initial benchmark at 100k messages per second, which hopefully we can further improve over time.

Our micro benchmark involves a simple program producing or consuming 100k messages, where each message has a payload of 256 bytes.

For Redis, it's running on the same computer in Docker. On my not-very-impressive laptop with a 10th Gen Intel Core i7, the numbers are somewhat around:

Producer​

redis    0.5s
stdio 0.5s
file 0.5s

Consumer​

redis    1.0s
stdio 1.0s
file 1.1s

It practically means that we are comfortably in the realm of producing 100k messages per second, but are just about able to consume 100k messages in 1 second. Suggestions to performance improvements are welcome!

Community​

SeaQL.org is an independent open-source organization run by passionate ️developers. If you like our projects, please star ⭐ and share our repositories. If you feel generous, a small donation via GitHub Sponsor will be greatly appreciated, and goes a long way towards sustaining the organization 🚒.

SeaStreamer is a community driven project. We welcome you to participate, contribute and together build for Rust's future πŸ¦€.

Announcing SeaORM 0.12

Β· 11 min read
SeaQL Team
Chris Tsang
SeaORM 0.12 Banner

πŸŽ‰ We are pleased to announce SeaORM 0.12 today!

We still remember the time when we first introduced SeaORM to the Rust community two years ago. We set out a goal to enable developers to build asynchronous database-driven applications in Rust.

Today, many open-source projects, a handful of startups and many more closed-source projects are using SeaORM. Thank you all who participated and contributed in the making!

SeaORM Star History

New Features πŸŒŸβ€‹

🧭 Seaography: GraphQL integration (preview)​

Seaography example

Seaography is a GraphQL framework built on top of SeaORM. In 0.12, Seaography integration is built into sea-orm. Seaography allows you to build GraphQL resolvers quickly. With just a few commands, you can launch a GraphQL server from SeaORM entities!

While Seaography development is still in an early stage, it is especially useful in prototyping and building internal-use admin panels.

Read the documentation to learn more.

Added macro DerivePartialModel​

#1597 Now you can easily perform custom select to query only the columns you needed

#[derive(DerivePartialModel, FromQueryResult)]
#[sea_orm(entity = "Cake")]
struct PartialCake {
name: String,
#[sea_orm(
from_expr = r#"SimpleExpr::FunctionCall(Func::upper(Expr::col((Cake, cake::Column::Name))))"#
)]
name_upper: String,
}

assert_eq!(
cake::Entity::find()
.into_partial_model::<PartialCake>()
.into_statement(DbBackend::Sqlite)
.to_string(),
r#"SELECT "cake"."name", UPPER("cake"."name") AS "name_upper" FROM "cake""#
);

Added Select::find_with_linked​

#1728, #1743 Similar to find_with_related, you can now select related entities and consolidate the models.

// Consider the following link
pub struct BakedForCustomer;

impl Linked for BakedForCustomer {
type FromEntity = Entity;

type ToEntity = super::customer::Entity;

fn link(&self) -> Vec<RelationDef> {
vec![
super::cakes_bakers::Relation::Baker.def().rev(),
super::cakes_bakers::Relation::Cake.def(),
super::lineitem::Relation::Cake.def().rev(),
super::lineitem::Relation::Order.def(),
super::order::Relation::Customer.def(),
]
}
}

let res: Vec<(baker::Model, Vec<customer::Model>)> = Baker::find()
.find_with_linked(baker::BakedForCustomer)
.order_by_asc(baker::Column::Id)
.all(db)
.await?

Added DeriveValueType derive macro for custom wrapper types​

#1720 So now you can use newtypes easily.

#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "custom_value_type")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub number: Integer,
// Postgres only
pub str_vec: StringVec,
}

#[derive(Clone, Debug, PartialEq, Eq, DeriveValueType)]
pub struct Integer(i32);

#[derive(Clone, Debug, PartialEq, Eq, DeriveValueType)]
pub struct StringVec(pub Vec<String>);

Which saves you the boilerplate of:

impl std::convert::From<StringVec> for Value { .. }

impl TryGetable for StringVec {
fn try_get_by<I: ColIdx>(res: &QueryResult, idx: I)
-> Result<Self, TryGetError> { .. }
}

impl ValueType for StringVec {
fn try_from(v: Value) -> Result<Self, ValueTypeErr> { .. }

fn type_name() -> String { "StringVec".to_owned() }

fn array_type() -> ArrayType { ArrayType::String }

fn column_type() -> ColumnType { ColumnType::String(None) }
}

Enhancements πŸ†™β€‹

#1433 Chained AND / OR join ON condition​

Added more macro attributes to DeriveRelation

// Entity file

#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {
// By default, it's `JOIN `fruit` ON `cake`.`id` = `fruit`.`cake_id` AND `fruit`.`name` LIKE '%tropical%'`
#[sea_orm(
has_many = "super::fruit::Entity",
on_condition = r#"super::fruit::Column::Name.like("%tropical%")"#
)]
TropicalFruit,
// Specify `condition_type = "any"` to override it, now it becomes
// `JOIN `fruit` ON `cake`.`id` = `fruit`.`cake_id` OR `fruit`.`name` LIKE '%tropical%'`
#[sea_orm(
has_many = "super::fruit::Entity",
on_condition = r#"super::fruit::Column::Name.like("%tropical%")"#
condition_type = "any",
)]
OrTropicalFruit,
}

#1508 Supports entity with composite primary key of arity 12​

#[derive(Clone, Debug, PartialEq, DeriveEntityModel)]
#[sea_orm(table_name = "primary_key_of_12")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false)]
pub id_1: String,
...
#[sea_orm(primary_key, auto_increment = false)]
pub id_12: bool,
}

#1677 Added UpdateMany::exec_with_returning()​

let models: Vec<Model> = Entity::update_many()
.col_expr(Column::Values, Expr::expr(..))
.exec_with_returning(db)
.await?;

#1511 Added MigratorTrait::migration_table_name() method to configure the name of migration table​

#[async_trait::async_trait]
impl MigratorTrait for Migrator {
// Override the name of migration table
fn migration_table_name() -> sea_orm::DynIden {
Alias::new("override_migration_table_name").into_iden()
}
...
}

#1707 Added DbErr::sql_err() method to parse common database errors​

assert!(matches!(
cake.into_active_model().insert(db).await
.expect_err("Insert a row with duplicated primary key")
.sql_err(),
Some(SqlErr::UniqueConstraintViolation(_))
));

assert!(matches!(
fk_cake.insert(db).await
.expect_err("Insert a row with invalid foreign key")
.sql_err(),
Some(SqlErr::ForeignKeyConstraintViolation(_))
));

#1737 Introduced new ConnAcquireErr​

enum DbErr {
ConnectionAcquire(ConnAcquireErr),
..
}

enum ConnAcquireErr {
Timeout,
ConnectionClosed,
}

#1627 Added DatabaseConnection::ping()​

|db: DatabaseConnection| {
assert!(db.ping().await.is_ok());
db.clone().close().await;
assert!(matches!(db.ping().await, Err(DbErr::ConnectionAcquire)));
}

#1708 Added TryInsert that does not panic on empty inserts​

// now, you can do:
let res = Bakery::insert_many(std::iter::empty())
.on_empty_do_nothing()
.exec(db)
.await;

assert!(matches!(res, Ok(TryInsertResult::Empty)));

#1712 Insert on conflict do nothing to return Ok​

let on = OnConflict::column(Column::Id).do_nothing().to_owned();

// Existing behaviour
let res = Entity::insert_many([..]).on_conflict(on).exec(db).await;
assert!(matches!(res, Err(DbErr::RecordNotInserted)));

// New API; now you can:
let res =
Entity::insert_many([..]).on_conflict(on).do_nothing().exec(db).await;
assert!(matches!(res, Ok(TryInsertResult::Conflicted)));

#1740, #1755 Replacing sea_query::Iden with sea_orm::DeriveIden​

To provide a more consistent interface, sea-query/derive is no longer enabled by sea-orm, as such, Iden no longer works as a derive macro (it's still a trait).

// then:

#[derive(Iden)]
#[iden = "category"]
pub struct CategoryEnum;

#[derive(Iden)]
pub enum Tea {
Table,
#[iden = "AfternoonTea"]
EverydayTea,
}

// now:

#[derive(DeriveIden)]
#[sea_orm(iden = "category")]
pub struct CategoryEnum;

#[derive(DeriveIden)]
pub enum Tea {
Table,
#[sea_orm(iden = "AfternoonTea")]
EverydayTea,
}

New Release Train Ferry πŸš’β€‹

It's been the 12th release of SeaORM! Initially, a major version was released every month. It gradually became 2 to 3 months, and now, it's been 6 months since the last major release. As our userbase grew and some are already using SeaORM in production, we understand the importance of having a stable API surface and feature set.

That's why we are committed to:

  1. Reviewing breaking changes with strict scrutiny
  2. Expanding our test suite to cover all features of our library
  3. Never remove features, and consider deprecation carefully

Today, the architecture of SeaORM is pretty solid and stable, and with the 0.12 release where we paid back a lot of technical debt, we will be able to deliver new features and enhancements without breaking. As our major dependency SQLx is not 1.0 yet, technically we cannot be 1.0.

We are still advancing rapidly, and we will always make a new release as soon as SQLx makes a new release, so that you can upgrade everything at once. As a result, the next major release of SeaORM will come out 6 months from now, or when SQLx makes a new release, whichever is earlier.

Community Survey πŸ“β€‹

SeaQL is an independent open-source organization. Our goal is to enable developers to build data intensive applications in Rust. If you are using SeaORM, please participate in the SeaQL Community Survey!

By completing this survey, you will help us gather insights into how you, the developer, are using our libraries and identify means to improve your developer experience. We will also publish an annual survey report to summarize our findings.

If you are a happy user of SeaORM, consider writing us a testimonial!

A big thank to DigitalOcean who sponsored our server hosting, and JetBrains who sponsored our IDE, and every sponsor on GitHub Sponsor!

If you feel generous, a small donation will be greatly appreciated, and goes a long way towards sustaining the organization.

A big shout out to our sponsors πŸ˜‡:

Shane Sveller

Γ‰mile Fugulin

Afonso Barracha

Jacob Trueb

Natsuki Ikeguchi

Marlon Mueller-Soppart

KallyDev

Dean Sheather

Manfred Lee

Roland GorΓ‘cz

IceApinan

René Klačan

Unnamed Sponsor

What's Next for SeaORM? ⛡​

Open-source project is a never-ending work, and we are actively looking for ways to sustain the project. You can support our endeavour by starring & sharing our repositories and becoming a sponsor.

We are considering multiple directions to generate revenue for the organization. If you have any suggestion, or want to join or collaborate with us, please contact us via hello[at]sea-ql.org.

Thank you for your support, and together we can make open-source sustainable.

Introducing SeaStreamer 🌊

Β· 7 min read
Chris Tsang
SeaQL Team

We are pleased to introduce SeaStreamer to the Rust community today. SeaStreamer is a stream processing toolkit to help you build stream processors in Rust.

At SeaQL we want to make Rust the best programming platform for data engineering. Where SeaORM is the essential tool for working with SQL databases, SeaStreamer aims to be your essential toolkit for working with streams.

Currently SeaStreamer provides integration with Kafka and Redis.

Let's have a quick tour of SeaStreamer.

High level async API​

  • High level async API that supports both async-std and tokio
  • Mutex-free implementation1: concurrency achieved by message passing
  • A comprehensive type system that guides/restricts you with the API

Below is a basic Kafka consumer:

#[tokio::main]
async fn main() -> Result<()> {
env_logger::init();

let stream: StreamUrl = "kafka://streamer.sea-ql.org:9092/my_stream".parse()?;
let streamer = KafkaStreamer::connect(stream.streamer(), Default::default()).await?;
let mut options = KafkaConsumerOptions::new(ConsumerMode::RealTime);
options.set_auto_offset_reset(AutoOffsetReset::Earliest);
let consumer = streamer
.create_consumer(stream.stream_keys(), options)
.await?;

loop {
let mess = consumer.next().await?;
println!("{}", mess.message().as_str()?);
}
}

Consumer::stream() returns an object that implements the Stream trait, which allows you to do neat things:

let items = consumer
.stream()
.take(num)
.map(process_message)
.collect::<Vec<_>>()
.await

Trait-based abstract interface​

All SeaStreamer backends implement a common abstract interface, offering you a familiar API. Below is a basic Redis consumer, which is nearly the same as the previous example:

#[tokio::main]
async fn main() -> Result<()> {
env_logger::init();

let stream: StreamUrl = "redis://localhost:6379/my_stream".parse()?;
let streamer = RedisStreamer::connect(stream.streamer(), Default::default()).await?;
let mut options = RedisConsumerOptions::new(ConsumerMode::RealTime);
options.set_auto_stream_reset(AutoStreamReset::Earliest);
let consumer = streamer
.create_consumer(stream.stream_keys(), options)
.await?;

loop {
let mess = consumer.next().await?;
println!("{}", mess.message().as_str()?);
}
}

Redis Streams Support​

SeaStreamer Redis provides a Kafka-like stream semantics:

  • Non-group streaming with AutoStreamReset option
  • Consumer-group-based streaming with auto-ack and/or auto-commit
  • Load balancing among consumers with automatic failover
  • Seek/rewind to point in time

You don't have to call XADD, XREAD, XACK, etc... anymore!

Enum-based generic interface​

The trait-based API requires you to designate the concrete Streamer type for monomorphization, otherwise the code cannot compile.

Akin to how SeaORM implements runtime-polymorphism, SeaStreamer provides a enum-based generic streamer, in which the backend is selected on runtime.

Here is an illustration (full example):

// sea-streamer-socket
pub struct SeaConsumer {
backend: SeaConsumerBackend,
}

enum SeaConsumerBackend {
#[cfg(feature = "backend-kafka")]
Kafka(KafkaConsumer),
#[cfg(feature = "backend-redis")]
Redis(RedisConsumer),
#[cfg(feature = "backend-stdio")]
Stdio(StdioConsumer),
}

// Your code
let uri: StreamerUri = "kafka://localhost:9092".parse()?; // or
let uri: StreamerUri = "redis://localhost:6379".parse()?; // or
let uri: StreamerUri = "stdio://".parse()?;

// SeaStreamer will be backed by Kafka, Redis or Stdio depending on the URI
let streamer = SeaStreamer::connect(uri, Default::default()).await?;

// Set backend-specific options
let mut options = SeaConsumerOptions::new(ConsumerMode::Resumable);
options.set_kafka_consumer_options(|options: &mut KafkaConsumerOptions| { .. });
options.set_redis_consumer_options(|options: &mut RedisConsumerOptions| { .. });
let mut consumer: SeaConsumer = streamer.create_consumer(stream_keys, options).await?;

// You can still retrieve the concrete type
let kafka: Option<&mut KafkaConsumer> = consumer.get_kafka();
let redis: Option<&mut RedisConsumer> = consumer.get_redis();

So you can "write once, stream anywhere"!

Good old unix pipe​

In SeaStreamer, stdin & stdout can be used as stream source and sink.

Say you are developing some processors to transform a stream in several stages:

./processor_1 --input kafka://localhost:9092/input --output kafka://localhost:9092/stage_1 &
./processor_2 --input kafka://localhost:9092/stage_1 --output kafka://localhost:9092/stage_2 &
./processor_3 --input kafka://localhost:9092/stage_2 --output kafka://localhost:9092/output &

It would be great if we can simply pipe the processors together right?

With SeaStreamer, you can do the following:

./processor_1 --input kafka://localhost:9092/input --output stdio:///stream |
./processor_2 --input stdio:///stream --output stdio:///stream |
./processor_3 --input stdio:///stream --output kafka://localhost:9092/output

All without recompiling the stream processors! Now, you can develop locally with the comfort of using |, >, < and your favourite unix program in the shell.

Testable​

SeaStreamer encourages you to write tests at all levels:

  • You can execute tests involving several stream processors in the same OS process
  • You can execute tests involving several OS processes by connecting them with pipes
  • You can execute tests involving several stream processors with Redis / Kafka

All against the same piece of code! Let SeaStreamer take away the boilerplate and mocking facility from your codebase.

Below is an example of intra-process testing, which can be run with cargo test without any dependency or side-effects:

let stream = StreamKey::new("test")?;
let mut options = StdioConnectOptions::default();
options.set_loopback(true); // messages produced will be feed back to consumers
let streamer = StdioStreamer::connect(StreamerUri::zero(), options).await?;
let producer = streamer.create_producer(stream.clone(), Default::default()).await?;
let mut consumer = streamer.create_consumer(&[stream.clone()], Default::default()).await?;

for i in 0..5 {
let mess = format!("{}", i);
producer.send(mess)?;
}

let seq = collect(&mut consumer, 5).await;
assert_eq!(seq, [0, 1, 2, 3, 4]);

Getting started​

If you are eager to get started with SeaStreamer, you can checkout our set of examples:

  • consumer: A basic consumer
  • producer: A basic producer
  • processor: A basic stream processor
  • resumable: A resumable stream processor that continues from where it left off
  • buffered: An advanced stream processor with internal buffering and batch processing
  • blocking: An advanced stream processor for handling blocking / CPU-bound tasks

Read the official documentation to learn more.

Roadmap​

A few major components we plan to develop:

  • File Backend
  • Redis Cluster

We welcome you to join our Discussions if you have thoughts or ideas!

People​

SeaStreamer is designed and developed by the same mind who brought you SeaORM:

Chris Tsang

Community​

SeaQL.org is an independent open-source organization run by passionate ️developers. If you like our projects, please star ⭐ and share our repositories. If you feel generous, a small donation via GitHub Sponsor will be greatly appreciated, and goes a long way towards sustaining the organization 🚒.

SeaStreamer is a community driven project. We welcome you to participate, contribute and together build for Rust's future πŸ¦€.

Footnotes​

  1. except sea-streamer-stdio, but only contends on consumer add/drop ↩

What's new in SeaORM 0.11.0

Β· 13 min read
SeaQL Team
Chris Tsang

πŸŽ‰ We are pleased to release SeaORM 0.11.0!

Data Loader​

[#1443, #1238] The LoaderTrait provides an API to load related entities in batches.

Consider this one to many relation:

let cake_with_fruits: Vec<(cake::Model, Vec<fruit::Model>)> = Cake::find()
.find_with_related(Fruit)
.all(db)
.await?;

The generated SQL is:

SELECT
"cake"."id" AS "A_id",
"cake"."name" AS "A_name",
"fruit"."id" AS "B_id",
"fruit"."name" AS "B_name",
"fruit"."cake_id" AS "B_cake_id"
FROM "cake"
LEFT JOIN "fruit" ON "cake"."id" = "fruit"."cake_id"
ORDER BY "cake"."id" ASC

The 1 side's (Cake) data will be duplicated. If N is a large number, this would results in more data being transferred over the wire. Using the Loader would ensure each model is transferred only once.

The following loads the same data as above, but with two queries:

let cakes: Vec<cake::Model> = Cake::find().all(db).await?;
let fruits: Vec<Vec<fruit::Model>> = cakes.load_many(Fruit, db).await?;

for (cake, fruits) in cakes.into_iter().zip(fruits.into_iter()) { .. }
SELECT "cake"."id", "cake"."name" FROM "cake"
SELECT "fruit"."id", "fruit"."name", "fruit"."cake_id" FROM "fruit" WHERE "fruit"."cake_id" IN (..)

You can even apply filters on the related entity:

let fruits_in_stock: Vec<Vec<fruit::Model>> = cakes.load_many(
fruit::Entity::find().filter(fruit::Column::Stock.gt(0i32))
db
).await?;
SELECT "fruit"."id", "fruit"."name", "fruit"."cake_id" FROM "fruit"
WHERE "fruit"."stock" > 0 AND "fruit"."cake_id" IN (..)

To learn more, read the relation docs.

Transaction Isolation Level and Access Mode​

[#1230] The transaction_with_config and begin_with_config allows you to specify the IsolationLevel and AccessMode.

For now, they are only implemented for MySQL and Postgres. In order to align their semantic difference, MySQL will execute SET TRANSACTION commands before begin transaction, while Postgres will execute SET TRANSACTION commands after begin transaction.

db.transaction_with_config::<_, _, DbErr>(
|txn| { ... },
Some(IsolationLevel::ReadCommitted),
Some(AccessMode::ReadOnly),
)
.await?;

let transaction = db
.begin_with_config(IsolationLevel::ReadCommitted, AccessMode::ReadOnly)
.await?;

To learn more, read the transaction docs.

Cast Column Type on Select and Save​

[#1304] If you need to select a column as one type but save it into the database as another, you can specify the select_as and the save_as attributes to perform the casting. A typical use case is selecting a column of type citext (case-insensitive text) as String in Rust and saving it into the database as citext. One should define the model field as below:

#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "ci_table")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
#[sea_orm(select_as = "text", save_as = "citext")]
pub case_insensitive_text: String
}

#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}

impl ActiveModelBehavior for ActiveModel {}

Changes to ActiveModelBehavior​

[#1328, #1145] The methods of ActiveModelBehavior now have Connection as an additional parameter. It enables you to perform database operations, for example, logging the changes made to the existing model or validating the data before inserting it.

#[async_trait]
impl ActiveModelBehavior for ActiveModel {
/// Create a new ActiveModel with default values. Also used by `Default::default()`.
fn new() -> Self {
Self {
uuid: Set(Uuid::new_v4()),
..ActiveModelTrait::default()
}
}

/// Will be triggered before insert / update
async fn before_save<C>(self, db: &C, insert: bool) -> Result<Self, DbErr>
where
C: ConnectionTrait,
{
// Logging changes
edit_log::ActiveModel {
action: Set("before_save".into()),
values: Set(serde_json::json!(model)),
..Default::default()
}
.insert(db)
.await?;

Ok(self)
}
}

To learn more, read the entity docs.

Execute Unprepared SQL Statement​

[#1327] You can execute an unprepared SQL statement with ConnectionTrait::execute_unprepared.

// Use `execute_unprepared` if the SQL statement doesn't have value bindings
db.execute_unprepared(
"CREATE TABLE `cake` (
`id` int NOT NULL AUTO_INCREMENT PRIMARY KEY,
`name` varchar(255) NOT NULL
)"
)
.await?;

// Construct a `Statement` if the SQL contains value bindings
let stmt = Statement::from_sql_and_values(
manager.get_database_backend(),
r#"INSERT INTO `cake` (`name`) VALUES (?)"#,
["Cheese Cake".into()]
);
db.execute(stmt).await?;

Select Into Tuple​

[#1311] You can select a tuple (or single value) with the into_tuple method.

let res: Vec<(String, i64)> = cake::Entity::find()
.select_only()
.column(cake::Column::Name)
.column(cake::Column::Id.count())
.group_by(cake::Column::Name)
.into_tuple()
.all(&db)
.await?;

Atomic Migration​

[#1379] Migration will be executed in Postgres atomically that means migration scripts will be executed inside a transaction. Changes done to the database will be rolled back if the migration failed. However, atomic migration is not supported in MySQL and SQLite.

You can start a transaction inside each migration to perform operations like seeding sample data for a newly created table.

Types Support​

  • [#1325] Support various UUID formats that are available in uuid::fmt module
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "uuid_fmt")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub uuid: Uuid,
pub uuid_braced: uuid::fmt::Braced,
pub uuid_hyphenated: uuid::fmt::Hyphenated,
pub uuid_simple: uuid::fmt::Simple,
pub uuid_urn: uuid::fmt::Urn,
}
  • [#1210] Support vector of enum for Postgres
#[derive(Debug, Clone, PartialEq, Eq, EnumIter, DeriveActiveEnum)]
#[sea_orm(rs_type = "String", db_type = "Enum", enum_name = "tea")]
pub enum Tea {
#[sea_orm(string_value = "EverydayTea")]
EverydayTea,
#[sea_orm(string_value = "BreakfastTea")]
BreakfastTea,
}

#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "enum_vec")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub teas: Vec<Tea>,
pub teas_opt: Option<Vec<Tea>>,
}
  • [#1414] Support ActiveEnum field as primary key
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "enum_primary_key")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false)]
pub id: Tea,
pub category: Option<Category>,
pub color: Option<Color>,
}

Opt-in Unstable Internal APIs​

By enabling sea-orm-internal feature you opt-in unstable internal APIs including:

Breaking Changes​

  • [#1366] sea-query has been upgraded to 0.28.x, which comes with some improvements and breaking changes. Please follow the release notes for more details

  • [#1420] sea-orm-cli: generate entity command enable --universal-time flag by default

  • [#1425] Added RecordNotInserted and RecordNotUpdated to DbErr

  • [#1327] Added ConnectionTrait::execute_unprepared method

  • [#1311] The required method of TryGetable changed:

// then
fn try_get(res: &QueryResult, pre: &str, col: &str) -> Result<Self, TryGetError>;
// now; ColIdx can be `&str` or `usize`
fn try_get_by<I: ColIdx>(res: &QueryResult, index: I) -> Result<Self, TryGetError>;

So if you implemented it yourself:

impl TryGetable for XXX {
- fn try_get(res: &QueryResult, pre: &str, col: &str) -> Result<Self, TryGetError> {
+ fn try_get_by<I: sea_orm::ColIdx>(res: &QueryResult, idx: I) -> Result<Self, TryGetError> {
- let value: YYY = res.try_get(pre, col).map_err(TryGetError::DbErr)?;
+ let value: YYY = res.try_get_by(idx).map_err(TryGetError::DbErr)?;
..
}
}
  • [#1328] The ActiveModelBehavior trait becomes async trait. If you overridden the default ActiveModelBehavior implementation:
#[async_trait::async_trait]
impl ActiveModelBehavior for ActiveModel {
async fn before_save<C>(self, db: &C, insert: bool) -> Result<Self, DbErr>
where
C: ConnectionTrait,
{
// ...
}

// ...
}
  • [#1425] DbErr::RecordNotFound("None of the database rows are affected") is moved to a dedicated error variant DbErr::RecordNotUpdated
let res = Update::one(cake::ActiveModel {
name: Set("Cheese Cake".to_owned()),
..model.into_active_model()
})
.exec(&db)
.await;

// then
assert_eq!(
res,
Err(DbErr::RecordNotFound(
"None of the database rows are affected".to_owned()
))
);

// now
assert_eq!(res, Err(DbErr::RecordNotUpdated));
  • [#1395] sea_orm::ColumnType was replaced by sea_query::ColumnType
    • Method ColumnType::def was moved to ColumnTypeTrait
    • ColumnType::Binary becomes a tuple variant which takes in additional option sea_query::BlobSize
    • ColumnType::Custom takes a sea_query::DynIden instead of String and thus a new method custom is added (note the lowercase)
// Compact Entity
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "fruit")]
pub struct Model {
- #[sea_orm(column_type = r#"Custom("citext".to_owned())"#)]
+ #[sea_orm(column_type = r#"custom("citext")"#)]
pub column: String,
}
// Expanded Entity
impl ColumnTrait for Column {
type EntityName = Entity;

fn def(&self) -> ColumnDef {
match self {
- Self::Column => ColumnType::Custom("citext".to_owned()).def(),
+ Self::Column => ColumnType::custom("citext").def(),
}
}
}

SeaORM Enhancements​

  • [#1256] Refactor schema module to expose functions for database alteration
  • [#1346] Generate compact entity with #[sea_orm(column_type = "JsonBinary")] macro attribute
  • MockDatabase::append_exec_results(), MockDatabase::append_query_results(), MockDatabase::append_exec_errors() and MockDatabase::append_query_errors() [#1367] take any types implemented IntoIterator trait
  • [#1362] find_by_id and delete_by_id take any Into primary key value
  • [#1410] QuerySelect::offset and QuerySelect::limit takes in Into<Option<u64>> where None would reset them
  • [#1236] Added DatabaseConnection::close
  • [#1381] Added is_null getter for ColumnDef
  • [#1177] Added ActiveValue::reset to convert Unchanged into Set
  • [#1415] Added QueryTrait::apply_if to optionally apply a filter
  • Added the sea-orm-internal feature flag to expose some SQLx types
    • [#1297] Added DatabaseConnection::get_*_connection_pool() for accessing the inner SQLx connection pool
    • [#1434] Re-exporting SQLx errors

CLI Enhancements​

  • [#846, #1186, #1318] Generate #[serde(skip_deserializing)] for primary key columns
  • [#1171, #1320] Generate #[serde(skip)] for hidden columns
  • [#1124, #1321] Generate entity with extra derives and attributes for model struct

Integration Examples​

SeaORM plays well with the other crates in the async ecosystem. We maintain an array of example projects for building REST, GraphQL and gRPC services. More examples wanted!

Our GitHub Sponsor profile is up! SeaQL.org is an independent open-source organization run by passionate developers. If you enjoy using SeaORM, please star and share our repositories. If you feel generous, a small donation will be greatly appreciated, and goes a long way towards sustaining the project.

A big shout out to our sponsors πŸ˜‡:

Afonso Barracha

Γ‰mile Fugulin

Dean Sheather

Shane Sveller

Sakti Dwi Cahyono

Nick Price

Roland GorΓ‘cz

Henrik Giesel

Jacob Trueb

Naoki Ikeguchi

Manfred Lee

Marcus Buffett

efrain2007

What's Next?​

SeaQL is a community driven project. We welcome you to participate, contribute and build together for Rust's future.

Here is the roadmap for SeaORM 0.12.x.

SeaORM FAQ.02

Β· 2 min read
Chris Tsang
SeaQL Team

FAQ.02 Why the empty enum Relation {} is needed even if an Entity has no relations?​

Consider the following example Post Entity:

use sea_orm::entity::prelude::*;

#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
#[sea_orm(table_name = "posts")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub title: String,
pub text: String,
}

#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}

impl ActiveModelBehavior for ActiveModel {}

The two lines for defining Relation is quite unnecessary right?

To explain the problem, let's dive slightly deeper into the macro-expanded entity:

The DeriveRelation macro simply implements the RelationTrait:

impl RelationTrait for Relation {
fn def(&self) -> RelationDef {
match self {
_ => unreachable!()
}
}
}

Which in turn is needed by EntityTrait as an associated type:

impl EntityTrait for Entity {
type Relation = Relation;
...
}

It would be ideal if, when the user does not specify this associated type, the library automatically fills in a stub to satisfy the type system?

Turns out, there is such a feature in Rust! It is an unstable feature called associated_type_defaults.

Basically, it allows trait definitions to specify a default associated type, allowing it to be elided:

// only compiles in nightly
trait EntityTrait {
type Relation: Relation = EmptyRelation;
}

Due to our commitment to stable Rust, this may not land in SeaORM very soon. When it is stabilized, do remind us to implement this feature to get rid of those two lines!

Internship @ SeaQL

Β· 4 min read
SeaQL Team
Chris Tsang

SeaQL.org offer internships tailored to university students. In fact, it will be the 3rd cohort in 2023.

The internships normally take place during summer and winter semester breaks. During the internship period, you will work on a project dedicatedly and publish the project’s outcome at the end.

The striking aspect of our mode of operation is it covers the entire lifecycle of software development, from Design ➑️ Implementation ➑️ Testing ➑️ Delivery. You will be amazed of how much you can achieve in such a short period of time!

To date StarfishQL ✴️ and Seaography 🧭 are great projects our team has created in the past year. We pride ourselves on careful planning, consistent execution, and pragmatic approach in software engineering. I spend a huge amount of time on idea evaluation: if the scope of the project is too small, it will be uninteresting, but if it is too large, it will fail to be delivered.

Fellow undergraduates, here are a few good reasons why you should participate in internships at an open-source organization like SeaQL:

  1. A tangible showcase on CV: open-source work is published, inspectable and has real-world impact. We will also ensure that it has good branding, graphics, and visibility.
  2. Not driven by a business process, we do not compromise on quality of work. We do not have a proprietary development process, so it’s all open-source tools with transferable skills.
  3. You will contribute to the community and will interact with people across the world. Collaboration on open source is the best thing humanity ever invented. You will only believe me when you have experienced it first-hand.
  4. Because you are the driver of the project you work on, it allows you to uncover something more about yourself, in particular - abilities and discipline: you always have had under/over-estimated yourself in one aspect or another.

Here are several things you are going to learn:

  1. "Thinking > Programming": the more time you spend on thinking beforehand, the better the code you are going to write. And the more time you spend on reviewing afterwards, the better the code is going to be.
  2. How to organize a codebase. Make good use of the Rust type system to craft a modular, testable codebase.
  3. Test automation. Every SeaQL project is continuously tested, and this is an integral part of our engineering process.
  4. Documentation. Our software aims to provide good documentation that is comprehensive, easy to follow, and fun to read.
  5. Performance tuning. Depending on the project, we might do some benchmarking and optimization. But in general we put you off from writing code that creates unnecessary overhead.

We were a mentor organization in GSoC 2022 and may be in 2023 (update: we were not accepted into GSoC 2023). We also offer internships outside of GSoC. So, what are the requirements when you become a contributor?

  1. Be passionate. You must show your passion in open-source and software engineering, so a good GitHub profile with some participation is needed.
  2. Be dedicated. This is a full-time job. While being fully remote and flexible on hours, you must have no other commitment or duties during the stipulated internship period.
  3. Be open-minded. You should listen carefully to your mentors and act on their advice accordingly.
  4. Write more. Communicate your thoughts and progress on all channels in an organized manner.

Don’t just listen to me though. Here is what our past interns says:

Be well-prepared for your upcoming career in this technology industry! Follow us on GitHub and Twitter now, and stay tuned for future announcements.

Call for Contributors and Reviewers πŸ“’

Β· 5 min read
Chris Tsang
SeaQL Founder

We are calling for contributors and reviewers for SeaQL projects πŸ“’!

The SeaQL userbase has been steadily growing in the past year, and it’s a pleasure for us to have helped individuals and start-ups to build their projects in Rust. However, the volume of questions, issues and pull requests is nearly saturating our core members’ capacity.

But again, thank you everyone for participating in the community!

If your project depends on SeaQL and you want to help us, here are some suggestions (if you have not already, star all our repositories and follow us on Twitter):

  1. Financial Contribution. You can sponsor us on GitHub and those will be used to cover our expenses. As a courtesy, we listen to our sponsors for their needs and use cases, and we also communicate our organizational development from time-to-time.
  2. Code Contribution. Opening a PR with us is always appreciated! To get started, you can go through our issue trackers and pick one to handle. If you are thinking of developing a substantial feature, start with drafting a "Proposal & Implementation Plan" (PIP).
  3. Knowledge Contribution. There are various formats of knowledge sharing: tutorial, cookbook, QnA and Discord. You can open PRs to our documentation repositories or publish on your own. We will be happy to list it in our learning resources section. Keep an eye on our GitHub Discussions and Discord and help others where you can!
  4. Code Review. This is an important process of our engineering. Right now, only 3 of our core members serve as reviewers. Non-core members can also become reviewers and I invite you to become one!

Now, I’d like to outline our review policy: for maturing projects, each PR merged has to be approved by at least two reviewers and one of them must be a core member; self-review allowed. Here are some examples:

  • A core member opened a PR, another core member approved βœ…
  • A core member opened a PR, a reviewer approved βœ…
  • A reviewer opened a PR, a core member approved βœ…
  • A reviewer opened a PR, another reviewer approved β›”
  • A contributor opened a PR, 2 core members approved βœ…
  • A contributor opened a PR, a core member and a reviewer approved βœ…
  • A contributor opened a PR, 2 reviewers approved β›”

In a nutshell, at least two pairs of trusted eyes should have gone through each PR.

What are the criteria when reviewing a PR?​

The following questions should all be answered yes.

  1. Implementation, documentation and tests
    1. Is the implementation easy to follow (have meaningful variable and function names)?
    2. Is there sufficient document to the API?
    3. Are there adequate tests covering various cases?
  2. API design
    1. Is the API self-documenting so users can understand its use easily?
    2. Is the API style consistent with our existing API?
    3. Does the API made reasonable use of the type system to enforce constraints?
    4. Are the failure paths and error messages clear?
    5. Are all breaking changes justified and documented?
  3. Functionality
    1. Does the feature make sense in computer science terms?
    2. Does the feature actually work with all our supported backends?
    3. Are all caveats discussed and eliminated / documented?
  4. Architecture
    1. Does it fit with the existing architecture of our codebase?
    2. Is it not going to create technical debt / maintenance burden?
    3. Does it not break abstraction?

1, 2 & 3 are fairly objective and factual, however the answers to 4 probably require some discussion and debate. If a consensus cannot be made, @tyt2y3 will make the final verdict.

Who are the current reviewers?​

As of today, SeaQL has 3 core members who are also reviewers:

Chris Tsang

Founder. Maintains all projects.

Billy Chan

Founding member. Co-maintainer of SeaORM and Seaography.

Ivan Krivosheev

Joined in 2022. Co-maintainer of SeaQuery.

How to become a reviewer?​

We are going to invite a few contributors we worked closely with, but you can also volunteer – the requirement is: you have made substantial code contribution to our projects, and has shown familiarity with our engineering practices.

Over time, when you have made significant contribution to our organization, you can also become a core member.

Let’s build for Rust's future together πŸ¦€β€‹

What's new in SeaQuery 0.28.0

Β· 6 min read
SeaQL Team
Ivan Krivosheev

πŸŽ‰ We are pleased to release SeaQuery 0.28.0! Here are some feature highlights 🌟:

New IdenStatic trait for static identifier​

[#508] Representing a identifier with &'static str. The IdenStatic trait looks like this:

pub trait IdenStatic: Iden + Copy + 'static {
fn as_str(&self) -> &'static str;
}

You can derive it easily for your existing Iden. Just changing the #[derive(Iden)] into #[derive(IdenStatic)].

#[derive(IdenStatic)]
enum User {
Table,
Id,
FirstName,
LastName,
#[iden = "_email"]
Email,
}

assert_eq!(User::Email.as_str(), "_email");

New PgExpr and SqliteExpr traits for backend specific expressions​

[#519] Postgres specific and SQLite specific expressions are being moved into its corresponding trait. You need to import the trait into scope before construct the expression with those backend specific methods.

// Importing `PgExpr` trait before constructing Postgres expression
use sea_query::{extension::postgres::PgExpr, tests_cfg::*, *};

let query = Query::select()
.columns([Font::Name, Font::Variant, Font::Language])
.from(Font::Table)
.and_where(Expr::val("a").concatenate("b").concat("c").concat("d"))
.to_owned();

assert_eq!(
query.to_string(PostgresQueryBuilder),
r#"SELECT "name", "variant", "language" FROM "font" WHERE 'a' || 'b' || 'c' || 'd'"#
);
// Importing `SqliteExpr` trait before constructing SQLite expression
use sea_query::{extension::sqlite::SqliteExpr, tests_cfg::*, *};

let query = Query::select()
.column(Font::Name)
.from(Font::Table)
.and_where(Expr::col(Font::Name).matches("a"))
.to_owned();

assert_eq!(
query.to_string(SqliteQueryBuilder),
r#"SELECT "name" FROM "font" WHERE "name" MATCH 'a'"#
);

Bug Fixes​

// given
let (statement, values) = sea_query::Query::select()
.column(Glyph::Id)
.from(Glyph::Table)
.cond_where(Cond::any()
.add(Cond::all()) // empty all() => TRUE
.add(Cond::any()) // empty any() => FALSE
)
.build(sea_query::MysqlQueryBuilder);

// old behavior
assert_eq!(statement, r#"SELECT `id` FROM `glyph`"#);

// new behavior
assert_eq!(
statement,
r#"SELECT `id` FROM `glyph` WHERE (TRUE) OR (FALSE)"#
);

// a complex example
let (statement, values) = Query::select()
.column(Glyph::Id)
.from(Glyph::Table)
.cond_where(
Cond::all()
.add(Cond::all().not())
.add(Cond::any().not())
.not(),
)
.build(MysqlQueryBuilder);

assert_eq!(
statement,
r#"SELECT `id` FROM `glyph` WHERE NOT ((NOT TRUE) AND (NOT FALSE))"#
);

Breaking Changes​

  • [#535] MSRV is up to 1.62
# Make sure you're running SeaQuery with Rust 1.62+ πŸ¦€
$ rustup update
  • [#492] ColumnType::Array definition changed from Array(SeaRc<Box<ColumnType>>) to Array(SeaRc<ColumnType>)
  • [#475] Func::* now returns FunctionCall instead of SimpleExpr
  • [#475] Func::coalesce now accepts IntoIterator<Item = SimpleExpr> instead of IntoIterator<Item = Into<SimpleExpr>
  • [#475] Removed Expr::arg and Expr::args - these functions are no longer needed
  • [#507] Moved all Postgres specific operators to PgBinOper
  • [#476] Expr methods used to accepts Into<Value> now accepts Into<SimpleExpr>
  • [#476] Expr::is_in, Expr::is_not_in now accepts Into<SimpleExpr> instead of Into<Value> and convert it to SimpleExpr::Tuple instead of SimpleExpr::Values
  • [#475] Expr::expr now accepts Into<SimpleExpr> instead of SimpleExpr
  • [#519] Moved Postgres specific Expr methods to new trait PgExpr
  • [#528] Expr::equals now accepts C: IntoColumnRef instead of T: IntoIden, C: IntoIden
use sea_query::{*, tests_cfg::*};

let query = Query::select()
.columns([Char::Character, Char::SizeW, Char::SizeH])
.from(Char::Table)
.and_where(
Expr::col((Char::Table, Char::FontId))
- .equals(Font::Table, Font::Id)
+ .equals((Font::Table, Font::Id))
)
.to_owned();

assert_eq!(
query.to_string(MysqlQueryBuilder),
r#"SELECT `character`, `size_w`, `size_h` FROM `character` WHERE `character`.`font_id` = `font`.`id`"#
);
  • [#525] Removed integer and date time column types' display length / precision option

API Additions​

  • [#475] Added SelectStatement::from_function
use sea_query::{tests_cfg::*, *};

let query = Query::select()
.column(ColumnRef::Asterisk)
.from_function(Func::random(), Alias::new("func"))
.to_owned();

assert_eq!(
query.to_string(MysqlQueryBuilder),
r#"SELECT * FROM RAND() AS `func`"#
);
  • [#486] Added binary operators from the Postgres pg_trgm extension
use sea_query::extension::postgres::PgBinOper;

assert_eq!(
Query::select()
.expr(Expr::col(Font::Name).binary(PgBinOper::WordSimilarity, Expr::value("serif")))
.from(Font::Table)
.to_string(PostgresQueryBuilder),
r#"SELECT "name" <% 'serif' FROM "font""#
);
  • [#473] Added ILIKE and NOT ILIKE operators
  • [#510] Added the mul and div methods for SimpleExpr
  • [#513] Added the MATCH, -> and ->> operators for SQLite
use sea_query::extension::sqlite::SqliteBinOper;

assert_eq!(
Query::select()
.column(Char::Character)
.from(Char::Table)
.and_where(Expr::col(Char::Character).binary(SqliteBinOper::Match, Expr::val("test")))
.build(SqliteQueryBuilder),
(
r#"SELECT "character" FROM "character" WHERE "character" MATCH ?"#.to_owned(),
Values(vec!["test".into()])
)
);
  • [#497] Added the FULL OUTER JOIN
  • [#530] Added PgFunc::get_random_uuid
  • [#528] Added SimpleExpr::eq, SimpleExpr::ne, Expr::not_equals
  • [#529] Added PgFunc::starts_with
  • [#535] Added Expr::custom_keyword and SimpleExpr::not
use sea_query::*;

let query = Query::select()
.expr(Expr::custom_keyword(Alias::new("test")))
.to_owned();

assert_eq!(query.to_string(MysqlQueryBuilder), r#"SELECT test"#);
assert_eq!(query.to_string(PostgresQueryBuilder), r#"SELECT test"#);
assert_eq!(query.to_string(SqliteQueryBuilder), r#"SELECT test"#);
  • [#539] Added SimpleExpr::like, SimpleExpr::not_like and Expr::cast_as
  • [#532] Added support for NULLS NOT DISTINCT clause for Postgres
  • [#531] Added Expr::cust_with_expr and Expr::cust_with_exprs
use sea_query::{tests_cfg::*, *};

let query = Query::select()
.expr(Expr::cust_with_expr("data @? ($1::JSONPATH)", "hello"))
.to_owned();

assert_eq!(
query.to_string(PostgresQueryBuilder),
r#"SELECT data @? ('hello'::JSONPATH)"#
);
  • [#538] Added support for converting &String to Value

Miscellaneous Enhancements​

  • [#475] New struct FunctionCall which hold function and arguments
  • [#503] Support BigDecimal, IpNetwork and MacAddress for sea-query-postgres
  • [#511] Made value::with_array module public and therefore making NotU8 trait public
  • [#524] Drop the Sized requirement on implementers of SchemaBuilders

Integration Examples​

SeaQuery plays well with the other crates in the rust ecosystem.

Community​

SeaQL is a community driven project. We welcome you to participate, contribute and together build for Rust's future.

What's new in Seaography 0.3.0

Β· 4 min read
SeaQL Team
Panagiotis Karatakis

πŸŽ‰ We are pleased to release Seaography 0.3.0! Here are some feature highlights 🌟:

Dependency Upgrade​

[#93] We have upgraded a major dependency:

You might need to upgrade the corresponding dependency in your application as well.

Support Self Referencing Relation​

[#99] You can now query self referencing models and the inverse of it.

Self referencing relation should be added to the Relation enum, note that the belongs_to attribute must be belongs_to = "Entity".

use sea_orm::entity::prelude::*;

#[derive(
Clone, Debug, PartialEq, DeriveEntityModel,
async_graphql::SimpleObject, seaography::macros::Filter,
)]
#[sea_orm(table_name = "staff")]
#[graphql(complex)]
#[graphql(name = "Staff")]
pub struct Model {
#[sea_orm(primary_key)]
pub staff_id: i32,
pub first_name: String,
pub last_name: String,
pub reports_to_id: Option<i32>,
}

#[derive(
Copy, Clone, Debug, EnumIter, DeriveRelation,
seaography::macros::RelationsCompact
)]
pub enum Relation {
#[sea_orm(
belongs_to = "Entity",
from = "Column::ReportsToId",
to = "Column::StaffId",
)]
SelfRef,
}

impl ActiveModelBehavior for ActiveModel {}

Then, you can query the related models in GraphQL.

{
staff {
nodes {
firstName
reportsToId
selfRefReverse {
staffId
firstName
}
selfRef {
staffId
firstName
}
}
}
}

The resulting JSON

{
"staff": {
"nodes": [
{
"firstName": "Mike",
"reportsToId": null,
"selfRefReverse": [
{
"staffId": 2,
"firstName": "Jon"
}
],
"selfRef": null
},
{
"firstName": "Jon",
"reportsToId": 1,
"selfRefReverse": null,
"selfRef": {
"staffId": 1,
"firstName": "Mike"
}
}
]
}
}

Web Framework Generator​

[#74] You can generate seaography project with either Actix or Poem as the web server.

CLI Generator Option​

Run seaography-cli to generate seaography code with Actix or Poem as the web framework.

# The command take three arguments, generating project with Poem web framework by default
seaography-cli <DATABASE_URL> <CRATE_NAME> <DESTINATION>

# Generating project with Actix web framework
seaography-cli -f actix <DATABASE_URL> <CRATE_NAME> <DESTINATION>

# MySQL
seaography-cli mysql://root:root@localhost/sakila seaography-mysql-example examples/mysql
# PostgreSQL
seaography-cli postgres://root:root@localhost/sakila seaography-postgres-example examples/postgres
# SQLite
seaography-cli sqlite://examples/sqlite/sakila.db seaography-sqlite-example examples/sqliteql

Actix​

use async_graphql::{
dataloader::DataLoader,
http::{playground_source, GraphQLPlaygroundConfig},
EmptyMutation, EmptySubscription, Schema,
};
use async_graphql_actix_web::{GraphQLRequest, GraphQLResponse};
use sea_orm::Database;
use seaography_example_project::*;
// ...

async fn graphql_playground() -> Result<HttpResponse> {
Ok(HttpResponse::Ok()
.content_type("text/html; charset=utf-8")
.body(
playground_source(GraphQLPlaygroundConfig::new("http://localhost:8000"))
))
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
// ...

let database = Database::connect(db_url).await.unwrap();
let orm_dataloader: DataLoader<OrmDataloader> = DataLoader::new(
OrmDataloader {
db: database.clone(),
},
tokio::spawn,
);

let schema = Schema::build(QueryRoot, EmptyMutation, EmptySubscription)
.data(database)
.data(orm_dataloader)
.finish();

let app = App::new()
.app_data(Data::new(schema.clone()))
.service(web::resource("/").guard(guard::Post()).to(index))
.service(web::resource("/").guard(guard::Get()).to(graphql_playground));

HttpServer::new(app)
.bind("127.0.0.1:8000")?
.run()
.await
}

Poem​

use async_graphql::{
dataloader::DataLoader,
http::{playground_source, GraphQLPlaygroundConfig},
EmptyMutation, EmptySubscription, Schema,
};
use async_graphql_poem::GraphQL;
use poem::{handler, listener::TcpListener, web::Html, IntoResponse, Route, Server};
use sea_orm::Database;
use seaography_example_project::*;
// ...

#[handler]
async fn graphql_playground() -> impl IntoResponse {
Html(playground_source(GraphQLPlaygroundConfig::new("/")))
}

#[tokio::main]
async fn main() {
// ...

let database = Database::connect(db_url).await.unwrap();
let orm_dataloader: DataLoader<OrmDataloader> = DataLoader::new(
OrmDataloader { db: database.clone() },
tokio::spawn,
);

let schema = Schema::build(QueryRoot, EmptyMutation, EmptySubscription)
.data(database)
.data(orm_dataloader)
.finish();

let app = Route::new()
.at("/", get(graphql_playground)
.post(GraphQL::new(schema)));

Server::new(TcpListener::bind("0.0.0.0:8000"))
.run(app)
.await
.unwrap();
}

[#84] Filtering, sorting and paginating related 1-to-many queries. Note that the pagination is work-in-progress, currently it is in memory pagination.

For example, find all inactive customers, include their address, and their payments with amount greater than 7 ordered by amount the second result. You can execute the query below at our GraphQL playground.

{
customer(
filters: { active: { eq: 0 } }
pagination: { cursor: { limit: 3, cursor: "Int[3]:271" } }
) {
nodes {
customerId
lastName
email
address {
address
}
payment(
filters: { amount: { gt: "7" } }
orderBy: { amount: ASC }
pagination: { pages: { limit: 1, page: 1 } }
) {
nodes {
paymentId
amount
}
pages
current
pageInfo {
hasPreviousPage
hasNextPage
}
}
}
pageInfo {
hasPreviousPage
hasNextPage
endCursor
}
}
}

Integration Examples​

We have the following examples for you, alongside with the SQL scripts to initialize the database.

Community​

SeaQL is a community driven project. We welcome you to participate, contribute and together build for Rust's future.

What's new in SeaORM 0.10.x

Β· 10 min read
SeaQL Team
Chris Tsang

πŸŽ‰ We are pleased to release SeaORM 0.10.0!

Rust 1.65​

The long-anticipated Rust 1.65 has been released! Generic associated types (GATs) must be the hottest newly-stabilized feature.

How is GAT useful to SeaORM? Let's take a look at the following:

trait StreamTrait<'a>: Send + Sync {
type Stream: Stream<Item = Result<QueryResult, DbErr>> + Send;

fn stream(
&'a self,
stmt: Statement,
) -> Pin<Box<dyn Future<Output = Result<Self::Stream, DbErr>> + 'a + Send>>;
}

You can see that the Future has a lifetime 'a, but as a side effect the lifetime is tied to StreamTrait.

With GAT, the lifetime can be elided:

trait StreamTrait: Send + Sync {
type Stream<'a>: Stream<Item = Result<QueryResult, DbErr>> + Send
where
Self: 'a;

fn stream<'a>(
&'a self,
stmt: Statement,
) -> Pin<Box<dyn Future<Output = Result<Self::Stream<'a>, DbErr>> + 'a + Send>>;
}

What benefit does it bring in practice? Consider you have a function that accepts a generic ConnectionTrait and calls stream():

async fn processor<'a, C>(conn: &'a C) -> Result<...>
where C: ConnectionTrait + StreamTrait<'a> {...}

The fact that the lifetime of the connection is tied to the stream can create confusion to the compiler, most likely when you are making transactions:

async fn do_transaction<C>(conn: &C) -> Result<...>
where C: ConnectionTrait + TransactionTrait
{
let txn = conn.begin().await?;
processor(&txn).await?;
txn.commit().await?;
}

But now, with the lifetime of the stream elided, it's much easier to work on streams inside transactions because the two lifetimes are now distinct and the stream's lifetime is implicit:

async fn processor<C>(conn: &C) -> Result<...>
where C: ConnectionTrait + StreamTrait {...}

Big thanks to @nappa85 for the contribution.


Below are some feature highlights 🌟:

Support Array Data Types in Postgres​

[#1132] Support model field of type Vec<T>. (by @hf29h8sh321, @ikrivosheev, @tyt2y3, @billy1624)

You can define a vector of types that are already supported by SeaORM in the model.

#[derive(Clone, Debug, PartialEq, DeriveEntityModel)]
#[sea_orm(table_name = "collection")]
pub struct Model {
#[sea_orm(primary_key)]
pub id: i32,
pub integers: Vec<i32>,
pub integers_opt: Option<Vec<i32>>,
pub floats: Vec<f32>,
pub doubles: Vec<f64>,
pub strings: Vec<String>,
}

Keep in mind that you need to enable the postgres-array feature and this is a Postgres only feature.

sea-orm = { version = "0.10", features = ["postgres-array", ...] }

Better Error Types​

[#750, #1002] Error types with parsable database specific error. (by @mohs8421, @tyt2y3)

let mud_cake = cake::ActiveModel {
id: Set(1),
name: Set("Moldy Cake".to_owned()),
price: Set(dec!(10.25)),
gluten_free: Set(false),
serial: Set(Uuid::new_v4()),
bakery_id: Set(None),
};

// Insert a new cake with its primary key (`id` column) set to 1.
let cake = mud_cake.save(db).await.expect("could not insert cake");

// Insert the same row again and it failed
// because primary key of each row should be unique.
let error: DbErr = cake
.into_active_model()
.insert(db)
.await
.expect_err("inserting should fail due to duplicate primary key");

match error {
DbErr::Exec(RuntimeErr::SqlxError(error)) => match error {
Error::Database(e) => {
// We check the error code thrown by the database (MySQL in this case),
// `23000` means `ER_DUP_KEY`: we have a duplicate key in the table.
assert_eq!(e.code().unwrap(), "23000");
}
_ => panic!("Unexpected sqlx-error kind"),
},
_ => panic!("Unexpected Error kind"),
}

Run Migration on Any Postgres Schema​

[#1056] By default migration will be run on the public schema, you can now override it when running migration on the CLI or programmatically. (by @MattGson, @nahuakang, @billy1624)

For CLI, you can specify the target schema with -s / --database_schema option:

  • via sea-orm-cli: sea-orm-cli migrate -u postgres://root:root@localhost/database -s my_schema
  • via SeaORM migrator: cargo run -- -u postgres://root:root@localhost/database -s my_schema

You can also run the migration on the target schema programmatically:

let connect_options = ConnectOptions::new("postgres://root:root@localhost/database".into())
.set_schema_search_path("my_schema".into()) // Override the default schema
.to_owned();

let db = Database::connect(connect_options).await?

migration::Migrator::up(&db, None).await?;

Breaking Changes​

enum ColumnType {
// then
Enum(String, Vec<String>)

// now
Enum {
/// Name of enum
name: DynIden,
/// Variants of enum
variants: Vec<DynIden>,
}
...
}
  • A new method array_type was added to ValueType:
impl sea_orm::sea_query::ValueType for MyType {
fn array_type() -> sea_orm::sea_query::ArrayType {
sea_orm::sea_query::ArrayType::TypeName
}
...
}
  • ActiveEnum::name() changed return type to DynIden:
#[derive(Debug, Iden)]
#[iden = "category"]
pub struct CategoryEnum;

impl ActiveEnum for Category {
// then
fn name() -> String {
"category".to_owned()
}

// now
fn name() -> DynIden {
SeaRc::new(CategoryEnum)
}
...
}

SeaORM Enhancements​

CLI Enhancements​

Please check here for the complete changelog.

Integration Examples​

SeaORM plays well with the other crates in the async ecosystem. We maintain an array of example projects for building REST, GraphQL and gRPC services. More examples wanted!

Our GitHub Sponsor profile is up! If you feel generous, a small donation will be greatly appreciated.

A big shout out to our sponsors πŸ˜‡:

Γ‰mile Fugulin

Dean Sheather

Shane Sveller

Sakti Dwi Cahyono

Henrik Giesel

Jacob Trueb

Marcus Buffett

Unnamed Sponsor

Unnamed Sponsor

Community​

SeaQL is a community driven project. We welcome you to participate, contribute and together build for Rust's future.

Here is the roadmap for SeaORM 0.11.x.

Toggle Stacked Download Graph in crates.io

Β· 2 min read
Billy Chan
SeaQL Team

Not long ago we opened a PR "Toggle stacked download graph #5010" resolving Convert download chart from stacked chart to regular chart #3876 for crates.io.

What's it all about?

Problem​

The download graph on crates.io used to be a stacked graph. With download count of older versions stack on top of newer versions. You might misinterpret the numbers. Consider this, at the first glance, it seems that version 0.9.2 has 1,500+ downloads on Nov 7. But in fact, it has only 237 downloads that day because the graph is showing the cumulative downloads.

crates.io Stacked Download Graph

This makes it hard to compare the download trend of different versions over time. Why this is important? You may ask. It's important to observe the adoption rate of newer version upon release. This paints a general picture if existing users are upgrading to newer version or not.

Solution​

The idea is simple but effective: having a dropdown to toggle between stacked and unstacked download graph. With this, one can switch between both display mode, comparing the download trend of different version and observing the most download version in the past 90 days are straightforward and intuitive.

crates.io Unstacked Download Graph

Conclusion​

This is a great tool for us to gauge the adoption rate of our new releases and we highly encourage user upgrading to newer release that contains feature updates and bug fixes.

What's new in SeaQuery 0.27.0

Β· 7 min read
SeaQL Team
Ivan Krivosheev

πŸŽ‰ We are pleased to release SeaQuery 0.27.0! Here are some feature highlights 🌟:

Dependency Upgrade​

[#356] We have upgraded a major dependency:

  • Upgrade sqlx to 0.6.1

You might need to upgrade the corresponding dependency in your application as well.

Drivers support​

We have reworked the way drivers work in SeaQuery: priori to 0.27.0, users have to invoke the sea_query_driver_* macros. Now each driver sqlx, postgres & rusqlite has their own supporting crate, which integrates tightly with the corresponding libraries. Checkout our integration examples below for more details.

[#383] Deprecate sea-query-driver in favour of sea-query-binder

[#422] Rusqlite support is moved to sea-query-rusqlite

[#433] Postgres support is moved to sea-query-postgres

// before
sea_query::sea_query_driver_postgres!();
use sea_query_driver_postgres::{bind_query, bind_query_as};

let (sql, values) = Query::select()
.from(Character::Table)
.expr(Func::count(Expr::col(Character::Id)))
.build(PostgresQueryBuilder);

let row = bind_query(sqlx::query(&sql), &values)
.fetch_one(&mut pool)
.await
.unwrap();

// now
use sea_query_binder::SqlxBinder;

let (sql, values) = Query::select()
.from(Character::Table)
.expr(Func::count(Expr::col(Character::Id)))
.build_sqlx(PostgresQueryBuilder);

let row = sqlx::query_with(&sql, values)
.fetch_one(&mut pool)
.await
.unwrap();

// You can now make use of SQLx's `query_as_with` nicely:
let rows = sqlx::query_as_with::<_, StructWithFromRow, _>(&sql, values)
.fetch_all(&mut pool)
.await
.unwrap();

Support sub-query operators: EXISTS, ALL, ANY, SOME​

[#118] Added sub-query operators: EXISTS, ALL, ANY, SOME

let query = Query::select()
.column(Char::Id)
.from(Char::Table)
.and_where(
Expr::col(Char::Id)
.eq(
Expr::any(
Query::select().column(Char::Id).from(Char::Table).take()
)
)
)
.to_owned();

assert_eq!(
query.to_string(MysqlQueryBuilder),
r#"SELECT `id` FROM `character` WHERE `id` = ANY(SELECT `id` FROM `character`)"#
);
assert_eq!(
query.to_string(PostgresQueryBuilder),
r#"SELECT "id" FROM "character" WHERE "id" = ANY(SELECT "id" FROM "character")"#
);

Support ON CONFLICT WHERE​

[#366] Added support to ON CONFLICT WHERE

let query = Query::insert()
.into_table(Glyph::Table)
.columns([Glyph::Aspect, Glyph::Image])
.values_panic(vec![
2.into(),
3.into(),
])
.on_conflict(
OnConflict::column(Glyph::Id)
.update_expr((Glyph::Image, Expr::val(1).add(2)))
.target_and_where(Expr::tbl(Glyph::Table, Glyph::Aspect).is_null())
.to_owned()
)
.to_owned();

assert_eq!(
query.to_string(MysqlQueryBuilder),
r#"INSERT INTO `glyph` (`aspect`, `image`) VALUES (2, 3) ON DUPLICATE KEY UPDATE `image` = 1 + 2"#
);
assert_eq!(
query.to_string(PostgresQueryBuilder),
r#"INSERT INTO "glyph" ("aspect", "image") VALUES (2, 3) ON CONFLICT ("id") WHERE "glyph"."aspect" IS NULL DO UPDATE SET "image" = 1 + 2"#
);
assert_eq!(
query.to_string(SqliteQueryBuilder),
r#"INSERT INTO "glyph" ("aspect", "image") VALUES (2, 3) ON CONFLICT ("id") WHERE "glyph"."aspect" IS NULL DO UPDATE SET "image" = 1 + 2"#
);

Changed cond_where chaining semantics​

[#414] Changed cond_where chaining semantics

// Before: will extend current Condition
assert_eq!(
Query::select()
.cond_where(any![Expr::col(Glyph::Id).eq(1), Expr::col(Glyph::Id).eq(2)])
.cond_where(Expr::col(Glyph::Id).eq(3))
.to_owned()
.to_string(PostgresQueryBuilder),
r#"SELECT WHERE "id" = 1 OR "id" = 2 OR "id" = 3"#
);
// Before: confusing, since it depends on the order of invocation:
assert_eq!(
Query::select()
.cond_where(Expr::col(Glyph::Id).eq(3))
.cond_where(any![Expr::col(Glyph::Id).eq(1), Expr::col(Glyph::Id).eq(2)])
.to_owned()
.to_string(PostgresQueryBuilder),
r#"SELECT WHERE "id" = 3 AND ("id" = 1 OR "id" = 2)"#
);
// Now: will always conjoin with `AND`
assert_eq!(
Query::select()
.cond_where(Expr::col(Glyph::Id).eq(1))
.cond_where(any![Expr::col(Glyph::Id).eq(2), Expr::col(Glyph::Id).eq(3)])
.to_owned()
.to_string(PostgresQueryBuilder),
r#"SELECT WHERE "id" = 1 AND ("id" = 2 OR "id" = 3)"#
);
// Now: so they are now equivalent
assert_eq!(
Query::select()
.cond_where(any![Expr::col(Glyph::Id).eq(2), Expr::col(Glyph::Id).eq(3)])
.cond_where(Expr::col(Glyph::Id).eq(1))
.to_owned()
.to_string(PostgresQueryBuilder),
r#"SELECT WHERE ("id" = 2 OR "id" = 3) AND "id" = 1"#
);

Added OnConflict::value and OnConflict::values​

[#451] Implementation From<T> for any Into<Value> into SimpleExpr

// Before: notice the tuple
OnConflict::column(Glyph::Id).update_expr((Glyph::Image, Expr::val(1).add(2)))
// After: it accepts `Value` as well as `SimpleExpr`
OnConflict::column(Glyph::Id).value(Glyph::Image, Expr::val(1).add(2))

Improvement to ColumnDef::default​

[#347] ColumnDef::default now accepts Into<SimpleExpr> instead Into<Value>

// Now we can write:
ColumnDef::new(Char::FontId)
.timestamp()
.default(Expr::current_timestamp())

Breaking Changes​

  • [#386] Changed in_tuples interface to accept IntoValueTuple
  • [#320] Removed deprecated methods
  • [#440] CURRENT_TIMESTAMP changed from being a function to keyword
  • [#375] Update SQLite boolean type from integer to boolean`
  • [#451] Deprecated OnConflict::update_value, OnConflict::update_values, OnConflict::update_expr, OnConflict::update_exprs
  • [#451] Deprecated InsertStatement::exprs, InsertStatement::exprs_panic
  • [#451] Deprecated UpdateStatement::col_expr, UpdateStatement::value_expr, UpdateStatement::exprs
  • [#451] UpdateStatement::value now accept Into<SimpleExpr> instead of Into<Value>
  • [#451] Expr::case, CaseStatement::case and CaseStatement::finally now accepts Into<SimpleExpr> instead of Into<Expr>
  • [#460] InsertStatement::values, UpdateStatement::values now accepts IntoIterator<Item = SimpleExpr> instead of IntoIterator<Item = Value>
  • [#409] Use native api from SQLx for SQLite to work with time
  • [#435] Changed type of ColumnType::Enum from (String, Vec<String>) to Enum { name: DynIden, variants: Vec<DynIden>}

Miscellaneous Enhancements​

  • [#336] Added support one dimension Postgres array for SQLx
  • [#373] Support CROSS JOIN
  • [#457] Added support DROP COLUMN for SQLite
  • [#466] Added YEAR, BIT and VARBIT types
  • [#338] Handle Postgres schema name for schema statements
  • [#418] Added %, << and >> binary operators
  • [#329] Added RAND function
  • [#425] Implements Display for Value
  • [#427] Added INTERSECT and EXCEPT to UnionType
  • [#448] OrderedStatement::order_by_customs, OrderedStatement::order_by_columns, OverStatement::partition_by_customs, OverStatement::partition_by_columns now accepts IntoIterator<Item = T> instead of Vec<T>
  • [#452] TableAlterStatement::rename_column, TableAlterStatement::drop_column, ColumnDef::new, ColumnDef::new_with_type now accepts IntoIden instead of Iden
  • [#426] Cleanup IndexBuilder trait methods
  • [#436] Introduce SqlWriter trait
  • [#448] Remove unneeded vec! from examples

Bug Fixes​

  • [#449] distinct_on properly handles ColumnRef
  • [#461] Removed ON for DROP INDEX for SQLite
  • [#468] Change datetime string format to include microseconds
  • [#452] ALTER TABLE for PosgreSQL with UNIQUE constraint

Integration Examples​

SeaQuery plays well with the other crates in the rust ecosystem.

Community​

SeaQL is a community driven project. We welcome you to participate, contribute and together build for Rust's future.