From 97c57a81d6df5cc2a5f2c16a43a68946a8f13cf7 Mon Sep 17 00:00:00 2001 From: alemi Date: Fri, 13 Sep 2024 22:55:32 +0200 Subject: [PATCH] docs: moved around documentation on building and using Co-authored-by: zaaarf --- dist/README.md | 46 +++++++++++++++++++++++++++++ dist/java/README.md | 5 +++- dist/js/README.md | 18 ------------ dist/lua/README.md | 19 ------------ dist/py/README.md | 14 --------- src/ffi/mod.rs | 71 +++++++++++++++++++++++++-------------------- 6 files changed, 90 insertions(+), 83 deletions(-) create mode 100644 dist/README.md delete mode 100644 dist/js/README.md delete mode 100644 dist/lua/README.md delete mode 100644 dist/py/README.md diff --git a/dist/README.md b/dist/README.md new file mode 100644 index 0000000..317e696 --- /dev/null +++ b/dist/README.md @@ -0,0 +1,46 @@ +# Compiling and Distributing FFI-compatible binaries +`codemp` aims to target as many platforms as possible, while remaining maintainable and performant. + +To guarantee this, it can compile to a bare rust lib but also 4 different FFI-compatible shared objects: JavaScript, Python, Lua, Java. + +> We also plan to offer bare C bindings for every other language which can do C interop, but it's not our top priority right now. + +To compile the bare FFI-compatible shared object, just `cargo build --release --features=`, replacing `` with either `js`, `py`, `java`, `lua` or `luajit`. +In most languages, just importing the resulting shared object will work, however refer to each language's section below for more in-depth information. + +## JavaScript +To build a npm package, `napi-cli` must first be installed: `npm install napi-cli`. + +You can then `npx napi build` in the project root to compile the native extension and create the type annotations (`index.d.ts`). +A package.json is provided for publishing, but will require some tweaking. + +## Python +To distribute the native extension we can leverage python wheels. It will be necessary to build the relevant wheels with [`maturin`](https://github.com/PyO3/maturin). +After installing with `pip install maturin`, run `maturin build` to obtain an `import`able package and installable wheels. + +## Lua +Built Lua bindings are valid lua modules and require no extra steps to be used. + +## Java +`codemp`'s Java bindings are implemented using the [JNI](https://docs.oracle.com/javase/8/docs/technotes/guides/jni/). + +On the Rust side, all Java-related code is gated behind the `java` feature, and is implemented using [`jni`](https://github.com/jni-rs/jni-rs) crate. + +Unlike other supported languages, Java is statically typed and requires knowing all foreign function types at compile time. +This means that, to use `codemp` through the JNI, all functions who need to be called must also be declared on the Java side, marked as `native`. + +Thus, we also provide pre-made Java glue code, wrapping all native calls and defining classes to hold `codemp` types. + +The Java bindings have no known major quirk. However, here are a list of facts that are useful to know when developing with these: + +* Memory management is entirely delegated to the JVM's garbage collector. + * A more elegant solution than `Object.finalize()`, who is deprecated in newer Java versions, may be coming eventually. +* Exceptions coming from the native side have generally been made checked to imitate Rust's philosophy with `Result`. + * `JNIException`s are however unchecked: there is nothing you can do to recover from them, as they usually represent a severe error in the glue code. If they arise, it's probably a bug. + +### Using +`codemp` **will be available soon** as an artifact on [Maven Central](https://mvnrepository.com) + +### Building +This is a [Gradle](https://gradle.org/) project: building requires having both Gradle and Cargo installed, as well as the JDK (any non-abandoned version). +Once you have all the requirements, building is as simple as running `gradle build`: the output is going to be a JAR under `build/libs`, which you can import into your classpath with your IDE of choice. diff --git a/dist/java/README.md b/dist/java/README.md index 6cf16e0..6ba24e4 100644 --- a/dist/java/README.md +++ b/dist/java/README.md @@ -2,7 +2,10 @@ `codemp`'s Java bindings are implemented using the [JNI](https://docs.oracle.com/javase/8/docs/technotes/guides/jni/). On the Rust side, all Java-related code is gated behind the `java` feature, and is implemented using[`jni-rs`](https://github.com/jni-rs/jni-rs). -Unlike other languages, Java requires glue code on both sides: as a result, a Java component is necessary. +Unlike other supported languages, Java is statically typed and requires knowing all foreign function types at compile time. +This means that, to use `codemp` from Java, all functions which will be used must be declared (as `native`), making using our Java binding without extra glue extremely tedious. + +We provide glue code also on the Java side, wrapping all native calls and defining classes to hold `codemp` types. ## Building This is a [Gradle](https://gradle.org/) project: building requires having both Gradle and Cargo installed, as well as the JDK (any non-abandoned version). diff --git a/dist/js/README.md b/dist/js/README.md deleted file mode 100644 index 570e186..0000000 --- a/dist/js/README.md +++ /dev/null @@ -1,18 +0,0 @@ -# JavaScript bindings -NodeJS allows directly `require`ing properly formed shared objects, so the glue can live mostly on the Rust side. - -Our JavaScript glue is built with [`napi`](https://napi.rs). - -To get a usable shared object just `cargo build --release --features=js`, however preparing a proper javascript package to be included as dependency requires more steps. - -## `npm` - -`codemp` is directly available on `npm` as [`codemp`](https://npmjs.org/package/codemp). - -## Building - -To build a node package, `napi-cli` must first be installed: `npm install napi-cli`. - -You can then `npx napi build` in the project root to compile the native extension and create the type annotations (`index.d.ts`). -A package.json is provided for publishing, but will require some tweaking. - diff --git a/dist/lua/README.md b/dist/lua/README.md deleted file mode 100644 index c7342a7..0000000 --- a/dist/lua/README.md +++ /dev/null @@ -1,19 +0,0 @@ -# Lua bindings -Lua allows directly `require`ing properly constructed shared objects, so glue code can live completely on the Rust side. - -The Lua-compatible wrappers are built with [`mlua`](https://github.com/mlua-rs/mlua). - -To build, just `cargo build --release --features=lua` and rename the resulting `libcodemp.so` / `codemp.dll` / `codemp.dylib` in `codemp_native.so/dll/dylib`. -This is important because Lua looks up the constructor symbol based on filename. - -Type hints are provided in `annotations.lua`, just include them in your language server: `---@module 'annotations'`. - -## LuaRocks -`codemp` is available as a rock on [LuaRocks](https://luarocks.org/modules/alemi/codemp) - -## Manual bundling -LuaRocks compiles from source, which only works if have the rust toolchain available. To provide a reasonable NeoVim experience, we provide pre-built binaries. - -> Download latest build and annotations from [here](https://codemp.dev/releases/lua/) - -You will need a loader file to provide annotations: you can use provided `codemp.lua` diff --git a/dist/py/README.md b/dist/py/README.md deleted file mode 100644 index a0e19f3..0000000 --- a/dist/py/README.md +++ /dev/null @@ -1,14 +0,0 @@ -# Python bindings -Python allows directly `import`ing properly formed shared objects, so the glue can live mostly on the Rust side. - -Our Python glue is built with [`PyO3`](https://pyo3.rs). - -To get a usable shared object just `cargo build --release --features=python`, however preparing a proper python package to be included as dependency requires more steps. - -## `PyPI` - -`codemp` is directly available on `PyPI` as [`codemp`](https://pypi.org/project/codemp). - -## Building -To distribute the native extension we can leverage python wheels. It will be necessary to build the relevant wheels with [`maturin`](https://github.com/PyO3/maturin). -After installing with `pip install maturin`, run `maturin build` to obtain an `import`able package. diff --git a/src/ffi/mod.rs b/src/ffi/mod.rs index 0cd53e1..f9c4ae4 100644 --- a/src/ffi/mod.rs +++ b/src/ffi/mod.rs @@ -1,38 +1,47 @@ -//! # FFI -//! The glue code for FFI (Foreign Function Interface) in various languages, each gated behind -//! a feature flag. -//! -//! For all except Java, the resulting shared object is ready to use, but external packages are -//! available to simplify dependency management and provide type hints in editor. -//! -//! ## Lua -//! Using [mlua](https://docs.rs/mlua) it's possible to map almost perfectly the entirety of `codemp` API. -//! Notable outliers are functions that receive `codemp` objects: these instead receive arguments -//! to build the object instead (such as [`crate::api::Controller::send`]) -//! -//! Note that async operations are carried out on a [tokio] current_thread runtime, so it is -//! necessary to drive it. A separate driver thread can be spawned with `spawn_runtime_driver` -//! function. -//! -//! To work with callbacks, the main Lua thread must periodically stop and poll for callbacks via -//! `poll_callback`, otherwise those will never run. This is necessary to allow safe concurrent -//! access to the global Lua state, so minimize callback execution time as much as possible. -//! -//! ## Python -//! Using [pyo3](https://docs.rs/pyo3) it's possible to map perfectly the entirety of `codemp` API. -//! Async operations run on a dedicated [tokio] runtime -//! +//! # Foreign Function Interface +//! `codemp` aims to be available as a library from as many programming languages as possible. +//! To achieve this, we rely on Foreign Function Interface. +//! //! ## JavaScript -//! Using [napi](https://docs.rs/napi) it's possible to map perfectly the entirety of `codemp` API. -//! Async operations run on a dedicated [tokio] runtime and the result is sent back to main thread +//! Our JavaScript glue is built with [`napi`](https://napi.rs). //! +//! All async operations are handled on a separate tokio runtime, automatically managed by `napi`. +//! Callbacks are safely scheduled to be called on the main loop thread. +//! +//! ## Python +//! Our Python glue is built with [`PyO3`](https://pyo3.rs). +//! +//! All async operations return a `Promise`, which can we `.wait()`-ed to block and get the return +//! value. The `Future` itself is run on a `tokio` runtime in a dedicated thread, which must be +//! stared with `codemp.init()` before doing any async operations. +//! +//! ## Lua +//! Our Lua glue is built with [`mlua`](https://github.com/mlua-rs/mlua). +//! +//! Lua bindings run all async code on a current thread tokio runtime, which should be driven with +//! a dedicated thread. +//! +//! All async functions will return a `Promise`, which can be `:await()`-ed to block and get the +//! return value. +//! +//! Note as Lua uses filename to locate entrypoint symbol, so shared object can't just have any name. +//! Accepted filenames are `libcodemp.___`, `codemp.___`, `codemp_native.___`, `codemp_lua.___` (extension depends on your platform: `so` on linux, `dll` on windows, `dylib` on macos). +//! Type hints are provided in `dist/lua/annotations.lua`, just include them in your language server: `---@module 'annotations'`. +//! +//! `codemp` is available as a rock on [LuaRocks](https://luarocks.org/modules/alemi/codemp), +//! however LuaRocks compiles from source and will require having `cargo` installed. +//! We provide pre-built binaries at [codemp.dev/releases/lua](https://codemp.dev/releases/lua/). +//! **Please do not rely on this link, as our built binaries will likely move somewhere else soon!**. +//! //! ## Java -//! Since for java it is necessary to deal with the JNI and no complete FFI library is available, -//! java glue directly writes JNI functions leveraging [jni](https://docs.rs/jni) rust bindings. +//! Our Java glue is built with [`jni`](https://github.com/jni-rs/jni-rs). //! -//! To have a runnable `jar`, some extra Java code must be compiled (available under `dist/java`) -//! and bundled together with the shared object. Such extra wrapper provides classes and methods -//! loading the native extension and invoking the underlying native functions. +//! Memory management is entirely delegated to the JVM's garbage collector. +//! A more elegant solution than `Object.finalize()`, who is deprecated in newer Java versions, may be coming eventually. +//! +//! Exceptions coming from the native side have generally been made checked to imitate Rust's philosophy with `Result`. +//! `JNIException`s are however unchecked: there is nothing you can do to recover from them, as they usually represent a severe error in the glue code. If they arise, it's probably a bug. +//! /// java bindings, built with [jni] #[cfg(feature = "java")]