Skip to content

Add wasm32-unknown-unknown compilation target support for matrix_sdk_sqlite#6329

Open
NoManColonies wants to merge 32 commits into
matrix-org:mainfrom
NoManColonies:main
Open

Add wasm32-unknown-unknown compilation target support for matrix_sdk_sqlite#6329
NoManColonies wants to merge 32 commits into
matrix-org:mainfrom
NoManColonies:main

Conversation

@NoManColonies
Copy link
Copy Markdown

@NoManColonies NoManColonies commented Mar 20, 2026

Before starting, I apologize for any inconvenience caused by my non-native English.

This PR adds support for compiling matrix_sdk_sqlite to the wasm32-unknown-unknown target, enabling use in browser and similar WASM environments.

Currently, matrix_sdk in WASM is limited to:

  • in-memory state store/event cache, or
  • IndexedDB-backed state store with in-memory event cache

This means persistent event/media caching is not available without a custom implementation.
This PR introduces SQLite-backed persistence for WASM as an additional store option.

Summary of changes

  • Bump rusqlite to 0.38 (adds wasm32-unknown-unknown support via sqlite-wasm-rs)
    • cache feature now required for prepared/cached statements
    • fallible_uint feature required for unsigned integer casting
  • Add wasm32-wasi-vfs feature for WASM targets to enable a browser-compatible VFS
    • Default backend: OPFS
    • Other VFS implementation is split into a separate PR
  • Add web-time dependency for media retention policy serialization
  • Gate Tokio fs feature and deadpool-sync behind non-WASM targets
  • Replace Send with SendOutsideWasm where required for non-Send WASM environments
  • Use #[cfg_attr(...)] instead of unconditional #[async_trait] for target-specific behavior
  • Replace SyncWrapper<T> with RefCell<T> in the WASM deadpool adapter
    • SyncWrapper<T> requires Send bound
    • RefCell<T> provides equivalent interior mutability without Send bound

Notes for reviewers

  • Default VFS backend is OPFS; other VFS implementation is split into a separate PR
  • No public API changes beyond documented feature additions
    • js feature is needed in order to compile to WASM target
  • Changes are largely conditional on target_family = "wasm"

  • I've documented the public API Changes in the appropriate CHANGELOG.md files
  • This PR was made with the help of AI
    • This PR description was machine translated/assisted

Signed-off-by: Geng Kongpop [email protected]

@NoManColonies NoManColonies changed the title Pull changes from upstream Added wasm32-unknown-unknown compilation support target for matrix_sdk_sqlite Mar 20, 2026
@NoManColonies NoManColonies changed the title Added wasm32-unknown-unknown compilation support target for matrix_sdk_sqlite Added wasm32-unknown-unknown compilation target support for matrix_sdk_sqlite Mar 20, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 20, 2026

Codecov Report

❌ Patch coverage is 75.75758% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.83%. Comparing base (8b762b0) to head (5a38eff).
⚠️ Report is 123 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
crates/matrix-sdk-sqlite/src/test_utils.rs 0.00% 2 Missing and 1 partial ⚠️
crates/matrix-sdk-sqlite/src/lib.rs 66.66% 0 Missing and 2 partials ⚠️
crates/matrix-sdk-sqlite/src/connection/default.rs 66.66% 0 Missing and 1 partial ⚠️
crates/matrix-sdk-sqlite/src/crypto_store.rs 90.90% 0 Missing and 1 partial ⚠️
crates/matrix-sdk-sqlite/src/state_store.rs 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6329      +/-   ##
==========================================
+ Coverage   89.82%   89.83%   +0.01%     
==========================================
  Files         378      379       +1     
  Lines      104337   104330       -7     
  Branches   104337   104330       -7     
==========================================
+ Hits        93719    93729      +10     
+ Misses       7016     7007       -9     
+ Partials     3602     3594       -8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Mar 20, 2026

Merging this PR will degrade performance by 49.4%

❌ 1 regressed benchmark
✅ 49 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation Restore session [memory store] 112.8 ms 222.9 ms -49.4%

Comparing NoManColonies:main (5a38eff) with main (dab87b7)

Open in CodSpeed

@NoManColonies NoManColonies marked this pull request as ready for review March 23, 2026 10:06
@NoManColonies NoManColonies requested a review from a team as a code owner March 23, 2026 10:06
@NoManColonies NoManColonies requested review from poljar and removed request for a team March 23, 2026 10:06
@richvdh richvdh self-requested a review March 25, 2026 12:12
Copy link
Copy Markdown
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks very much for submitting this. We're quite excited by it, as it opens the door to things like being able to use the wasm bindings for matrix-sdk-crypto in node.js (matrix-org/matrix-sdk-crypto-wasm#168), and maybe even dropping support for indexeddb altogether.

Some questions, other than those below:

  • How were you able to test/build this? I tried things like cargo check -p matrix-sdk-sqlite --target wasm32-unknown-unknown and wasm-pack test --chrome --headless crates/matrix-sdk-sqlite, and got build errors.

    Would you be able to add support to cargo xtask ci for checking and testing matrix-sdk-sqlite with the wasm target? I think you'll need to add a new entry to WasmFeatureSet.

  • Would you be able to break the changes down into smaller commits, to make them easier to review?

Comment thread crates/matrix-sdk-sqlite/src/connection.rs Outdated
Comment thread crates/matrix-sdk-sqlite/src/connection/default.rs
Comment thread crates/matrix-sdk-sqlite/src/crypto_store.rs Outdated
@richvdh richvdh changed the title Added wasm32-unknown-unknown compilation target support for matrix_sdk_sqlite Add wasm32-unknown-unknown compilation target support for matrix_sdk_sqlite Mar 25, 2026
@NoManColonies NoManColonies requested a review from a team as a code owner March 26, 2026 04:20
@richvdh
Copy link
Copy Markdown
Member

richvdh commented Mar 26, 2026

@NoManColonies I assume you're still working on this. Once it's ready for review, please rebase your changes on main and ping me.

@NoManColonies
Copy link
Copy Markdown
Author

NoManColonies commented Mar 28, 2026

@richvdh

Thanks very much for submitting this. We're quite excited by it, as it opens the door to things like being able to use the wasm bindings for matrix-sdk-crypto in node.js (matrix-org/matrix-sdk-crypto-wasm#168), and maybe even dropping support for indexeddb altogether.

Some questions, other than those below:

  • How were you able to test/build this? I tried things like cargo check -p matrix-sdk-sqlite --target wasm32-unknown-unknown and wasm-pack test --chrome --headless crates/matrix-sdk-sqlite, and got build errors.
    Would you be able to add support to cargo xtask ci for checking and testing matrix-sdk-sqlite with the wasm target? I think you'll need to add a new entry to WasmFeatureSet.
  • Would you be able to break the changes down into smaller commits, to make them easier to review?

Thank you very much for taking your time to review this PR and apologize for the late reply.
To address your questions:

  • How were you able to test/build this? I tried things like cargo check -p matrix-sdk-sqlite --target wasm32-unknown-unknown and wasm-pack test --chrome --headless crates/matrix-sdk-sqlite, and got build errors.

I initially tested the prototype in a temporary downstream project and I seem to have forgotten to properly configure the feature requirements in the upstream crate, my apologies. You should be able to build and test with the following commands:

  • Cargo check: cargo check -p matrix-sdk-sqlite --target wasm32-unknown-unknown --features js
  • Wasm-pack test: wasm-pack test --chrome --headless crates/matrix-sdk-sqlite --features js --release
    • Currently, the OPFS backend has a harmless debug assertion that triggers at the end of test. So you may need to run the test under release profile

Would you be able to add support to cargo xtask ci for checking and testing matrix-sdk-sqlite with the wasm target? I think you'll need to add a new entry to WasmFeatureSet.

I’ve added tests as requested, but current VFS backend (OPFS) require dedicated worker context and is therefore not supported in a Node.js environment without polyfill. I have made a separate PR adding alternative backends that should be usable in Node.js.

  • Would you be able to break the changes down into smaller commits, to make them easier to review?

Apologies for the large initial commit, it was initially composed of many small commit that didn’t compile. I saw a note in CONTRIBUTING.md that requires all commits to compile successfully, and all initial changes were required in order to compile under WASM targets. If you'd prefer, I can break the initial commit down into a smaller one.

Since your previous comments, I have made the following changes:

  • I have refactored associated type of Manager trait in src/connection.rs to use a wrapper type around RefCell<T> and keeping the interface similar to SyncWrapper<T> in order to avoid most of conditional compilation. If you prefer a WASM connection manager entirely in a separate module, please let me know. I initially did not go this route, as it requires a lot of duplicate code in the connection manager that is largely the same as non-WASM targets WASM specific connection manager implementation has also been moved to a separate module.
  • I have created a utility function setup_db_fs in src/utils.rs for setting up filesystem under different compilation targets in order to avoid duplicate code in each module.
  • I have added tests for cargo xtask ci to cover the matrix-sdk-sqlite crate when compiled for wasm targets.
    • The provided test vector isn’t compatible with SQLite compiled for wasm, so I dumped the database contents and converted them using online tool. The wasm tests now use this test vector instead of the previously provided one.
  • I have refactored VFS registration to avoid global installation which is harder to test and could limit downstream usage of SQLite.

If you have additional requests, please let know.

@NoManColonies NoManColonies marked this pull request as ready for review March 28, 2026 17:18
@richvdh richvdh self-requested a review March 30, 2026 10:23
Geng Kongpop added 6 commits March 31, 2026 00:54
chore: address breaking changes from rusqlite

feat: support alternative v/fs for wasm environment

feat: support non-send future for wasm environment

fix: solve incorrect error transformation

feat: WIP support connection pool in wasm environment

fix: use full-path method call for wasm environment + other fixes for
wasm

fix: more wasm related fixes

feat: use RefCell to simulate interior-mutability in wasm environment

fix: use implicit borrow instead of explicit call to `as_ref()`

fix: more wasm related fixes

fix: resolve ambiguity between lock guard and ref cell implicit
reference + more wasm fixes

fix: resolve wasm related lifetime issues

chore: turn on serde feature for web time when compiled as wasm32 target

chore: ran cargo check to verify Cargo.lock file

chore: fix comment formatting
Copy link
Copy Markdown
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks very much for the updates. I've taken a more detailed look. Generally it looks great, but I have some comments.

Comment thread crates/matrix-sdk-sqlite/src/connection/wasm.rs Outdated
Comment thread crates/matrix-sdk-sqlite/src/connection/wasm.rs Outdated
Comment on lines +18 to +21
//! Similar to the one implemented in `crate::connection::default`,
//! we do not implement connection recycling here. Mostly due to
//! [`managed::Manager`] trait expecting a future output with `Send`
//! bound which is not available in WASM environment.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand this comment. What does "do not implement connection recycling" mean? In what sense does Manager expect a future with Send?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I still don't understand this, but will return to it once we've resolved the other thread about recycle.

Comment thread crates/matrix-sdk-sqlite/src/connection/wasm.rs Outdated
Comment thread crates/matrix-sdk-sqlite/src/utils.rs Outdated
Comment thread xtask/src/ci.rs
Comment thread .github/workflows/ci.yml Outdated
Comment thread xtask/src/ci.rs
Comment on lines +553 to +556
// SQLite WASM test suites has to be ran in release mode due to
// a harmless debug assertion when closing database.
//
// Ref: https://github.com/Spxg/sqlite-wasm-rs/blob/master/crates/sqlite-wasm-vfs/src/sahpool.rs#L672
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does this happen? The assertion is "close without open", but doesn't the database file get opened?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not entirely sure why this happens. I don’t have deep insight into SQLite internals or the VFS implementation, but testing in a real browser environment seems to work as expected. Creating and opening a database in OPFS behaves correctly.

I’m not sure why this assertion ("close without open") is being triggered in this case. It may be worth checking with the author of https://github.com/Spxg/sqlite-wasm-rs for more insight.

Comment thread .github/workflows/ci.yml Outdated
Comment thread crates/matrix-sdk-sqlite/src/connection/default.rs
@richvdh
Copy link
Copy Markdown
Member

richvdh commented Apr 13, 2026

@NoManColonies are you still working on this?

@NoManColonies
Copy link
Copy Markdown
Author

NoManColonies commented Apr 13, 2026

@NoManColonies are you still working on this?

@richvdh Apologies for the delay. I got a bit caught up with some IRL stuff. I've made the following changes as requested:

  • Updated doc comments for better clarity
  • Moved the shared re-exports inside the connection module to mod.rs, and refactored manager initialization to avoid having to call setup_db_fs in multiple places
  • Moved wasm-specific code from utils.rs to wasm.rs, and renamed the wrapper type as suggested
  • Removed the forced runner in the wasm CI, as it turned out not to be necessary
    The unit tests already configure the wasm-bindgen test runner, so there’s no need to set it again in CI
  • Added test utilities for managing temporary directories, reducing the need for conditional compilation within unit tests

@richvdh richvdh self-requested a review April 14, 2026 10:45
@richvdh
Copy link
Copy Markdown
Member

richvdh commented Apr 14, 2026

@NoManColonies no need to apologise for the delay: I am happy to know you are still working on this!

However, I see you have force-pushed since my previous review, which means there is no way for me to see what has changed. As https://github.com/matrix-org/matrix-rust-sdk/blob/main/CONTRIBUTING.md says: "Do not force push after a review has started."

In order for me to see what has changed since 4bd5f07, please:

  1. Reset your branch to 4bd5f07, and force-push once more.
  2. Optionally, merge the latest main into your branch.
  3. Make your new changes
  4. Push your new changes to github.

@NoManColonies
Copy link
Copy Markdown
Author

@NoManColonies no need to apologise for the delay: I am happy to know you are still working on this!

However, I see you have force-pushed since my previous review, which means there is no way for me to see what has changed. As https://github.com/matrix-org/matrix-rust-sdk/blob/main/CONTRIBUTING.md says: "Do not force push after a review has started."

In order for me to see what has changed since 4bd5f07, please:

  1. Reset your branch to 4bd5f07, and force-push once more.
  2. Optionally, merge the latest main into your branch.
  3. Make your new changes
  4. Push your new changes to github.

Apologies for that. I’ve reset the branch back to 4bd5f07 and re-applied the changes. Please let me know if anything looks off.

@richvdh
Copy link
Copy Markdown
Member

richvdh commented Apr 23, 2026

@NoManColonies sorry for taking a long time to get back to this. It's on my to-do list, I promise I'll take a look soon.

@richvdh
Copy link
Copy Markdown
Member

richvdh commented May 13, 2026

I promise I'll take a look soon.

... for some value of "soon" ...

Copy link
Copy Markdown
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NoManColonies thanks for making all these changes; I think it is going in the right direction.

I've made a few requests for changes, but, to be honest, I'm finding it a bit hard to keep track of and I think it might be a good idea to make some separate PRs which we can land separately, to reduce the size of this one.

In particular, we could start with PRs for the following:

  • Change connection::Manager::new() to call create_dir_all, with the corresponding changes to build_pool_of_connections
  • Change crypto_store::tests::get_test_db to take the database bytes, rather than the filename

}
}

#[cfg(all(target_family = "wasm", target_os = "unknown"))]
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can omit this, given the whole module is only built for wasm. Likewise below.

Comment on lines +79 to +83
// We cannot implement connection recycling
// at the moment, due to
// `managed::Manager::recycle` expecting
// a future with `Send` bound which is not
// available in WASM environments.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not entirely understanding this. What would be involved in implementing recycling? You mention that we can't call ConnectionWrapper::interact, but why would we want do so?

Comment on lines +90 to +91
/// Wrapper object for providing interior mutability similar to [`SyncWrapper`]
/// without `Send` requirement.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Wrapper object for providing interior mutability similar to [`SyncWrapper`]
/// without `Send` requirement.
/// Wrapper object, providing interior mutability similar to [`SyncWrapper`],
/// but without requiring that the wrapped type implements `Send`.

}

/// Configure VFS name using provided path.
pub fn get_vfs_name(path: &Path) -> String {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be private now

Suggested change
pub fn get_vfs_name(path: &Path) -> String {
fn get_vfs_name(path: &Path) -> String {

}
}

/// Configure VFS name using provided path.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Configure VFS name using provided path.
/// Construct a suitable VFS name, representing the given native path.


use deadpool::managed::{self, Metrics};
use rusqlite::OpenFlags;
use sqlite_wasm_vfs::sahpool::{OpfsSAHPoolCfgBuilder, OpfsSAHPoolUtil, install};
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest giving install a better name:

Suggested change
use sqlite_wasm_vfs::sahpool::{OpfsSAHPoolCfgBuilder, OpfsSAHPoolUtil, install};
use sqlite_wasm_vfs::sahpool::{OpfsSAHPoolCfgBuilder, OpfsSAHPoolUtil, install as install_opfs_sahpool};

(or just refer to it as sqlite_wasm_vfs::sahpool::install)

/// tool.
///
/// Subsequence call to this function will simply return the management tool
/// without installing vfs.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// without installing vfs.
/// Register a [SQLite VFS], using [OPFS] and the provided path.
///
/// Returns a management utility (primarily useful for testing).
///
/// Subsequent calls to this function will simply return the management tool
/// without re-registering the VFS.
///
/// [SQLite VFS]: https://sqlite.org/vfs.html
/// [OPFS]: https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system

Also, maybe the function should be called register_vfs or install_vfs ?

.vfs_name(&get_vfs_name(path))
.directory(path.to_string_lossy().as_ref())
.build();
// Avoid global installation, due to being harder to test.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Avoid global installation, due to being harder to test.
// Avoid installing as the default VFS, due to being harder to test.

///
/// Subsequence call to this function will simply return the management tool
/// without installing vfs.
pub async fn setup_vfs(path: &Path) -> Result<OpfsSAHPoolUtil, OpenStoreError> {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe this should return OpfsSAHError ?

// Load test database source during compile-time, since we do not have
// access to file system during runtime for WASM targets.
let TestDb { _dir: _, database } = get_test_db(
include_bytes!("../../../testing/data/storage/matrix-sdk-crypto.wasm.db"),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these files probably should not be called *.wasm.db now that they are used for both wasm and native builds.

Comment on lines +22 to +23
- Support compiling to `wasm32-unknown-unknown` target + bump [rusqlite](https://github.com/rusqlite/rusqlite/releases/tag/v0.38.0) version to `0.38`
([#6329](https://github.com/matrix-org/matrix-rust-sdk/pull/6329))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We switched to a towncrier-based setup for changelogs. Sorry for the inconvenience but please move this to the changelog.d folder as a Towncrier fragment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants