Support for subinterpreters

[minesworld]

Please make a decission if CSnakes should support subinterpreters or not. As subinterpreters can have their own environment, modules etc. that would lead to a different API. At this stage of CSnakes it might be "easier" to change the existing API and break existing code.

My use case for CSnakes will be within a WinUI 3 app, thus having first class async and threading integration would be great, which might also affect the API...

[tonybaloney]

Yes, I would like to support this and have been thinking about the API.

In terms of design:

• You would request a new sub interpreter from the IPythonEnvironment interface
• That would return some sort of handle of IPythonEnvironment
• Python thread state is acquired by the GIL wrapper. The thread state internally in CPython is associated with an interpreter state. So you can't (easily) have a Managed .NET thread that talks to multiple interpreters since the thread state is stored in thread-local storage. We also hold a recursive GIL state in TLS on the .NET side. impl

How this would look as a consumer:

• You request a new interpreter
• You use that interpreter to import modules, call functions
• If you try to call from a thread that is associated with another interpreter, you will get an error.

There are some other challenges to overcome, like the .NET GC collector thread, which needs to know which interpreter to clean up references to.

[tonybaloney]

FWIW first class async and threading support with a single interpreter is pretty good with the current implementation. We've done a lot of engineering to reduce the GIL-contention with lots of Managed Threads.

There isn't a way of await'ing a Python async function from .NET but I'd like to implement this in future as well. We do support generator iterators link, which is a step before a coroutine. Take a look at the docs as well

[minesworld]

Yeah, you're right. Not only its your project and thus the main goal to call python objects from c# in a native fashion, async uses Tasks which can be pushed around Threads as the CLR likes... . A recipe for desaster from a Python module relying that its thread is the one it was created in the CPython runtime...

So lets define "first class" threading support from another C# perspective: WinUI3 .

These are my goals:

• the main interpreter can not run the main thread as this might interfere with the Windows MessagePump. I'm fine if all calls to the main interpreter come from a single C# Thread.
• "first class" CPython threads would mean from C# that there is an C# API which knows them and each CPython thread could be canceled via a CancelationToken.
• an additional brigded CancelationToken into the CPython world would be super usefull too (used that in IronPython)
• a possibilty to communicate between C# and CPython via Queues or Pipes in either direction. WinUI3 UI calls have to be done on the main thread...

For all users interesting:

• multiple subinterpreters. As each has its own environment and modules I hope it is possible to debug python with a debugger written in python in the same process. Even better if communicating over pipes...

Too bad my CPython runtime source knowledge is over 10 years old (wrote my own Objective-C bridge). And neither is C# my most beloved language nor the Microsofts CLR a flash of genius. Might be that there is an API to put and remove an Object into a "Non GC generation" and the string class is subclassable...

My near term goal is to be able to use CSnake as an IPythonNotebook replacement. That I "abused" as a database interface - 5000 items editable via Widgets and HTML, slow and ugly but works. More - neither the IPythonKernel nor Microsoft Edge seem to like that...

And I have a good feeling as CPython/CSnake gives us total control over objects, at least on their sides.

As I write this - maybe you can answer me a question as the C# master you are (compared to me): isn't it possible to modify Microsoft CLR sources, make a dll or lib and use THAT together with a modified CPython?

[tonybaloney]

isn't it possible to modify Microsoft CLR sources, make a dll or lib and use THAT together with a modified CPython?

I don't think it's necessary (I worked on some of the sub-interpreters changes for CPython in 3.13 and wrote the only book on CPython Internals). Either sub-interpreters or coroutines would be sufficient for what you're describing.

We also support free-threading, which is a beta feature of 3.13.

I am going to try both coroutines and sub-interpreters soon. I'm just keeping on top of the other bugs until then

[minesworld]

OMG !!! Thats the "secret" behind why I feel so super excited and hyped since the first time I found CSnakes by "accident".

In fact I got a little depressive over the last two years: having reached the limits of IronPython, knowing that async there won't happen (might be a CLR could be more supportive issue), spending weeks at a time looking at other solutions of any kind. Even used SmallTalk, Pharao VM is interesting but lacks 'Windows" let alone WinUI compability. My last thoughts was about using a SmallTalk transpiler project into CRL IL abonded 10 years ago... .

But too much effort and no solution for the biggest problem I face now: IPyNotebook crashing together with the Edge display, so I can not use my > 9M YouTube video datasets collected into a SQLAlchemy DB as I want and need to. BTW: took months using the 10.000 free credits a day and my DB Server isn't the fastest too - takes minutes to complete...

Having some visions about what "magic" is possible having on both sides - CPython and C# - the ability to create and use code on the fly, I'm convinced that CSnakes is THE scripting project until someone marries the existing Rust/Windows solution (Message loop is compiled Rust) with Devo 2 and a multithreaded V8... But thats many "If" and "might happen".

CSnakes is HERE NOW thanks to your brilliance Tony. And that most people don't think of using it displaying results of business logic in Windows GUIs might be the case of how well other solutions performed in the past.

I'm going to use CSnakes exactly for that. Trust me - I know what I am writing about more then having knowledge of the CLR, CPython internals or even how many times a string would get copied on the way from CPython onto the screen using native WinUI controls... . I have the feeling - to many times... Something like an IString interface in dotnet would have been great, but I also know that there are even better reasons not to have one. From Microsofts and its business customers point of view...

I think that CSnakes is much bigger then what you might have envisioned it to be. Thats the why our different views like about giving developers the ability to "transpile" C CPython API code into C#. Some might even find the embedding part most interesting, giving them the ability to fetch the project, remove all "unwanted" files - why not? One additional happy CSnakes user...

Yeah - having venv and pip integrated as a "no brainer"- is HUGE. (Have to look if my preview code includes the ability to set the ExtraPath...). Really huge.

For my and if it works other peoples use case - WinUI 3 - the biggest obstacle to overcome is moving strings and object lookups while being on the WinUIs main thread. Every speedup might be a game changer in regards to use CSnakes for that.

Thats the source of my funny but doable UTF-16 customizations of CPython ideas. That CPython becomes multithreaded "just in time", even when its "only" multi interpreter - what a luck.

But you Tony and your knowledge of the inner workings of CPython - lottery win.

You can tell me which the right direction or even chapter in your book to look into.

Like if there is a way to modify CPython that it is possible the get the internal presentation of a dictionary key (ok, doesn't CPython has only one string for each content anyway?) and using that for lookups so object equality is just comparing addresses? Same for attribute lookups... If there is some additional logic needed in CPython to make as fast as possible - the only question is: doable at what time budget?

Or storing a Context object somehow accessible "no matter what"...

Or sending from CPytho to the Windows MessagePump which might be the lowest latency way to get something executed in the WinUI main thread...

I bet - those or similar problems have most developers trying to integrate CPython into native GUIs. Which would be an explanation why there are so few using it and examples how to do it. CSnakes and its "magic" has the potential to change that... huge.