JyNI – Jython Native Interface

JyNI is a compatibility layer with the goal to enable Jython to use native CPython extensions like NumPy or SciPy. This way we aim to enable scientific Python code to run on Jython. Since Java is quite present in industry and Python is more present in science, JyNI will be an important step to lower the costs of using scientific code in industrial environments.

Our philosophy is to integrate JyNI with Jython and CPython extensions as seamlessly as possible. Therefore, JyNI works without any recompilation of Jython or the desired CPython extensions. It neither requires a customized Jython version, nor customized versions of the CPython extensions. Simply put JyNI on the classpath and Jython will “magically” load native extensions as far as the required C-API is implemented.

News

2020-03-21 – Jython 2.7.2 released

Jython 2.7.2 has been released. We thank all developers and contributors who created this new release. JyNI 2.7-alpha.5 runs with Jython 2.7.2 out of the box.

2018-07-01 – JyNI student project at Diamond Light Source

We deeply appreciate this summer’s JyNI student project created and hosted by Diamond Light Source. For twelve weeks, Calum Freeman will work on improving JyNI. Potential goals are completion of PyFile API, iterator support, buffer protocol, general bugfixes, documentation, a dev-guide and improving the test suite.

2017-09-21 – JyNI 2.7-alpha.5 released

JyNI finally runs on Windows from JyNI 2.7-alpha.5 onwards. The new release is mostly based on this year’s Google Summer of Code project. We once again thank Google for this wonderful program! Read the full story of issues, solutions, setbacks and tweaks on the GSoC blog.

2017-07-01 – Jython 2.7.1 released

The long-awaited release of Jython 2.7.1 finally became reality. Thanks to all developers and contributors who made this possible!

2017-06-14 – JyNI 2.7-alpha.4 released

NumPy on Jython becomes reality with JyNI 2.7-alpha.4. For discussion of NumPy support see Issue 2. JyNI 2.7-alpha.4 requires Jython 2.7.1 (RC2 and RC3 are also workable).

2017-05-04 – GSoC 2017: Project for Windows support accepted

Windows support for JyNI is coming with our brand new Google Summer of Code project. Follow progress on the GSoC blog. Completion of that project will bring JyNI close to a beta release.

2016-07-14 – Proceedings of the 8th European Conference on Python in Science (EuroSciPy 2015) released

Part of the proceedings is a new JyNI paper, “Garbage Collection in JyNI – How to bridge Mark/Sweep and Reference Counting GC” (see unshortened abstract). It explains JyNI’s GC mechanism in detail and also contains lots of roadmap information, general implementation overview and several examples. E.g. it features a rich example of how a Python framework (even if based on C extensions) can be (almost) seamlessly embedded into Java-code. Also take a look at the official proceedings page.

2016-02-16 – JyNI 2.7-alpha.3 released

Right in time for Jython 2.7.1 we are proud to release JyNI 2.7-alpha.3. It ultimately performs CPython-compatible garbage collection on native objects and is capable to utilize the ctypes extension. Check it out here. (Note that it requires Jython 2.7.1 beta 3.)

2015-08-28 – JyNI-talk at EuroSciPy 2015 in Cambridge

JyNI was presented at EuroSciPy 2015 in Cambridge, UK: JyNI – Prepare Jython for scientific code. The talk is also available online.

2015-08-17 – JyNI license changed to LGPL

JyNI is now licensed under the LGPL. We made this decision, because LGPL is an OSI-approved license and on the other hand, the difference between LGPL and GPL with classpath-exception only impacts statical binding, which is hardly a use-case for JyNI anyway.

2015-08-11 – JyNI was presented at the JVM Language Summit at Oracle campus in Santa Clara

The JVM Language Summit is an annual event where language designers, compiler writers, runtime engineers and JVM core developers discuss the future of the JVM and the Java programming language. The JyNI presentation is available here.

2015-08-02 – Important note: JyNI’s repository version now requires Jython >2.7.0.

So until Jython 2.7.1 is released (presumably in November 2015) you must build Jython from its latest repository version. See JyNI-readme for a quick-guide.

2015-05-22 – Kick-off for Google Summer of Code JyNI-project

Implementing garbage collection support was successfully filed as a GSoC project for summer 2015. Blog available here.

2015-04-22 – lwn.net reports about the JyNI-session at the Python Language Summit

JyNI was presented at the Python Language Summit on April the 8th at PyCon 2015. See report at lwn.net.

2015-04-13/14/15/16 – PyCon coding sprints: Come sprint with us on JyNI.

Learn about Python’s C-extension API, see Jython in action and more. Also newbies are welcome, but you should be able to roughly read C-code. We will work closely with Jython sprinters, so feel encouraged to take a look at Jython sprints too. See PyCon sprints page.

2015-03-18 – JyNI 2.7-alpha.2.3 establishes compatibility with Jython 2.7 beta 4

Apart from this it has minor internal improvements, but nothing notable for users. Note that it won’t work with earlier Jython versions than 2.7 beta 4. Get it…

2014-09-06 – Support for Jython 2.7 beta 3 provided (JyNI 2.7-alpha.2.2 released)

alpha.2.2 is essentially identical to alpha.2.1, but brings compatibility to Jython 2.7 beta 3, which was broken because of some minor API changes. Note that alpha.2.2 definitely won’t work with Jython versions prior to 2.7 beta 3. (One can still use alpha.2.1 for such versions.) Get it here…

2014-05-01 – Proceedings of the 6th European Conference on Python in Science (EuroSciPy 2013) released

Feel free to read the JyNI paper. It presents rich information around JyNI along with examples, an overview how the implementation works and what the next steps will be. The complete proceedings are available here.

2014-03-29 – JyNI supports Tkinter (JyNI 2.7-alpha.2.1 released)

New features are support for PyFunction, PyCode and heap-types along with various bug fixes. This version of JyNI is capable to run (at least basic) Tkinter code. Check it out…

2013-10-28 – JyNI 2.7-alpha.2 has been released!

Main features are full exception support and support for the original datetime module. Though Jython features an own implementation of datetime, support for the original module is a step toward NumPy, because the original datetime features a C-API that is used by NumPy. Read the release note on JyNI 2.7-alpha.2 for more details.

2013-08-23/24 – JyNI was presented at EuroSciPy in Brussels, Belgium

See abstract and poster.

2013-07-20 – JyNI 2.7-alpha.1 released

Our first release is mainly intended as a proof of concept.

Current State

We aim to achieve compatibility with the latest common version of Jython and CPython, which is 2.7 at the moment. JyNI does not yet support the complete Python C-API. The following builtin types are currently covered:

  • Number types PyInt, PyLong, PyFloat, PyComplex
  • Sequence types PyTuple, PyList, PySlice, PyString, PyUnicode
  • Data structure types PyDict, PySet, PyFrozenSet
  • Operational types PyModule, PyFunction, PyCode, PyProperty, PyDictProxy, PyCell (partly)
  • OOP types PyClass, PyInstance, PyMethod, PyClassMethod, PyStaticMethod
  • Singleton types PyNone, PyNotImplemented, PyEllipsis, PyBool
  • Native types PyCFunction, PyCapsule, PyCObject
  • Iterator types PySeqIter
  • Natively defined custom types
  • Exception types
  • PyType as static type or heap type
  • Weak reference types
  • New-style classes/instances

Planned:

  • Support for remaining iterator types, e.g. TupleIter, ListIter
  • Buffer protocol
  • PyByteArray (along with buffer protocol)
  • PyFile

As long as one sticks to the already supported types, the function families PyArg_ParseTuple and Py_BuildValue also work.

JyNI is available for all major desktop platforms and is developed on Linux Mint 18. We compiled and tested it successfully for

  • Linux Mint Debian Edition (LMDE, LMDE2) (32 bit and 64 bit)
  • Linux Mint 17, 18 (64 bit)
  • Ubuntu 17.04 (64 bit)
  • Mac OS-X (10.10, 10.11, 10.12)
  • Windows 7, 10 (64 bit)

It does probably work on other POSIX platforms too. Windows support was covered by our recent Google Summer of Code project, follow its story on the blog.

Roadmap

For JyNI v2.7-alpha.6 we plan to support the buffer protocol, PyByteArray, iterators and PyFile. When alpha.6 is stable, JyNI will transist into beta state, i.e. JyNI 2.7-beta.1. In beta state we will focus on better deployment, testing and bugfixing.

Compatibility/Wish List

Some Python extensions are already workable with JyNI to some extend:

  • ctypes (Linux, OSX, Windows)
  • sqlite3 (Linux, others not tested)
  • Tkinter (Linux, Windows)
  • PyOpenGL (Linux)
  • NumPy 1.12, 1.13 (Linux, OSX, Windows – see Issue 2)

Please use the wish list to point our attention to further native extensions. This list is our basis to guide priority of testing, bug-fixing and planning new features. There is no automated voting-system; simply write an email to contact@jyni.org and tell us what extension you wish to see (higher) on the list.

  1. NumPy (core already workable)
  2. SciPy (requires PyByteArray)
  3. CFFI (requires PyByteArray)
  4. Cython (already works to some extend, cythonizing still requires CPython)
  5. Pandas (requires NumPy, Cython, PyByteArray)
  6. scikit-learn (requires NumPy, SciPy)
  7. statsmodels (requires NumPy, SciPy, Pandas)
  8. Matplotlib (requires NumPy, PyFile)
  9. Astropy (requires better NumPy support)
  10. scikit-image (requires NumPy)
  11. h5py (requires PyByteArray)
  12. …?

License

JyNI is released under the GNU LGPL.

GNU GPL v3 applies by its formulation found at http://www.gnu.org/licenses/gpl-3.0-standalone.html

with the additional GNU LGPL v3 clauses found at http://www.gnu.org/licenses/lgpl-3.0-standalone.html.

Publications

EuroSciPy 2015

JyNI was presented on the 8th European Conference on Python in Science (EuroSciPy 2015):

JyNI – Prepare Jython for scientific code.

The talk is available online.

Proceedings

The JyNI-paper “Garbage Collection in JyNI – How to bridge Mark/Sweep and Reference Counting GC” was released in the proceedings of EuroSciPy 2015 on 2016-07-14. See direct link to PDF.

Abstract

Jython is a Java-based Python implementation and the most seamless way to integrate Python and Java. It achieves high efficiency by compiling Python code to Java bytecode and thus letting Java’s JIT optimize it – an approach that enables Python code to call Java functions or to subclass Java classes. It enables Python code to leverage Java’s multithreading features and utilizes Java’s built-in garbage collection (GC). However, it currently does not support CPython’s C-API and thus does not support native extensions like NumPy and SciPy. Since most scientific code depends on such extensions, it is not runnable with Jython.

Jython Native Interface (JyNI) is a compatibility layer that aims to provide CPython’s native C extension API on top of Jython. JyNI is implemented using the Java Native Interface (JNI) and its native part is designed to be binary compatible with existing extension builds. This means Jython can import the original C extensions, i.e. the same .dll- or .so-files that CPython would use. For various reasons, implementing CPython’s C-API is not an easy task. Just to name a few issues – it offers macros to access CPython internals, uses a global interpreter lock (GIL) in contrast to Jython and lets extensions perform reference-counting-based GC, which is incompatible with Java’s GC-approach. For each of the arising issues, JyNI proposes a feasible solution; most remarkably it emulates CPython’s reference-counting GC on top of Java’s mark-and-sweep-based approach (taking care of adjacent concepts like finalizers and weak references and their interference with Jython). (Note that there are vague considerations around to switch to mark-and-sweep-based GC in a future CPython too; cf. [PY3_PL15]. So this algorithm might one day be even relevant to CPython in terms of running legacy modules.)

This work’s main purpose is to describe the algorithm JyNI uses to support GC in detail, also covering weak references and testing native memory management. Beside this we give a comprehension of JyNI’s general design and briefly describe how it deals with CPython’s GIL. Finally we provide runnable code examples, e.g. a demonstration of JyNI’s support for the ctypes extension and a first experimental import of NumPy.

JVM Language Summit 2015

JyNI was presented on the JVM Language Summit 2015 at Oracle campus in Santa Clara.

The JVM Language Summit is an annual event where language designers, compiler writers, runtime engineers and JVM core developers discuss the future of the JVM and the Java programming language. The JyNI presentation is available here.

Python Language Summit 2015

JyNI was presented at the Python Language Summit on April the 8th at PyCon 2015. A write-up of the presentation is available at lwn.net.

EuroSciPy 2013

JyNI was presented at EuroSciPy on the 23rd and 24th of August 2013 in Brussels, Belgium with the following abstract:

Jython is a Java-based Python implementation and the most seamless way to integrate Python and Java. However, it does not support native extensions written for CPython like NumPy and SciPy. Since most scientific Python-code fundamentally depends on exactly such native extensions directly or indirectly, it usually cannot be run with Jython. JyNI aims to close this gap. It is a layer that enables Jython users to load native CPython extensions and access them from Jython the same way as they would do in CPython. In order to leverage the JyNI functionality, you just have to put it on the Java-classpath when Jython is launched. It neither requires you to recompile the extension-code, nor to build a customized Jython fork. That means, it is binary compatible with existing extension-builds.

At the time when this abstract is written, JyNI does not fully implement the Python C API and we are in fact just capable to load simple examples that only involve most basic builtin-types. The concept is rather complete though and our goal is to provide the C API needed to load NumPy as soon as possible. After that we will focus on SciPy and others.

We expect that our work will also enable Java developers to use CPython extensions like NumPy in their Java code.

The corresponding EuroSciPy page can be found here. The conference poster can be found here.

Proceedings

There was a JyNI-paper released in the proceedings of EuroSciPy 2013 on 2014-05-01. A direct link to the PDF is available here.

Contact

For feedback, questions and feature requests write to contact@jyni.org. For bug reports create issues on our issue tracker. Especially report if JyNI appears to be not buildable on a presumably supported platform or if the demos don’t run as expected (don’t forget to provide output of the build- or the demo-run).