44 articles in the notebook
A deep dive into user space threading for .NET: why OpenSim needs to pause, serialize and resume thousands of running scripts, and the whole arsenal — from Mono.Cecil IL rewriting to fibers — I have thrown at the problem.
The threshold for using Second Life/OpenSim is too high, so as a pilot project I'm building an in-world image gallery with paintings borrowed from my aunt. The hard part: finding a viewer that is actually easy to install.
OpenSim's native dependencies (SQLite, ODE, OpenJpeg) don't do 64-bit, and OpenSim.32BitLaunch.exe only helps from the command line. Here is the workaround to get full Visual Studio debugging on 64-bit Windows.
A colour-coded model of the SECS pipeline. Work in progress — boxes that share a colour are related.
People define a script engine narrowly as "the thing that runs the scripts" — I'd argue it's the most important component of OpenSim. SECS is my proposal for a component system with interfaces worth building on.
Reflections on why a common way of implementing script engines was needed: many found it hard to build engine variations on DotNetEngine, and not all developers want to conform to one structure.
3Di contributed region splitting and dynamic load balancing to OpenSim: multiple physical servers cooperating on one region, and regions migrating to less loaded machines while running. Green computing, virtual-world style.
OSSL takes an object-oriented turn: instead of only LSL2-style function calls you can now write Prim.Position.x += 10 or set Prim.Text directly. A more modular approach to the scripting language.
DotNetEngine now speaks VB.Net and JScript — just start your script with //vb or //js. LSL state support landed too.
Most of DotNetEngine has moved into OpenSim.Region.ScriptEngine.Common, turning it into a general execution framework. DotNetEngine now only converts and compiles — Common does the running, and future engines get loading, threading and maintenance for free.
New detailed config options for DotNetEngine, and more importantly: maintenance, load/unload and execution threads can now be shared across regions instead of 3-4 threads per region — a big deal for large scale OpenSim installations.
ScriptServer is "almost done": everything works except script-to-OpenSim communication. It lets script execution happen on a remote machine entirely outside OpenSim.exe.
I wrote up my opinions on how the future script engine should be implemented as a contribution to the Architecture Working Group wiki.
Notes on the protocol between region and script server: region initiates, a custom binary protocol streams async function calls in both directions, and replies are matched by ID. TCP client/server is already up.
A diagram of where the stand-alone ScriptEngine planning is at. It was very complex; after redrawing it a few times it ended up like this.
Lots of alien squid-like drawings in my notebook later: the plan is to move the ScriptEngine into a separate daemon, independent from the region. Scripts stop moving when objects do, can run on separate servers or clusters, and can even be rented out.
The ScriptEngine now mixes LSL and C# in the same script — and if a script starts with //C#, the LSL preprocessor is skipped entirely for 100% pure C# support. A few naming rules apply for events.
The OpenSim wiki now has pages on scripting: general info, llFunction implementation status, and the DotNetScriptEngine internals.
114 scripts loaded in-world, one AppDomain each, most of them firing timer events every second or every five — and my 2.8GHz Prescott doesn't break a sweat. Screenshots of the world and the performance graphs.
A summary for those who skip the techie details: LSL script support works, hundreds of scripts run simultaneously across multiple cores, events are queued, and the remaining gaps are clearly listed.
Some LL commands, like llSetTimerEvent and llHTTPRequest, don't return immediately — they come back later as events. LSLLongCmdHandler is a new class with a dedicated thread that keeps track of these long-running requests.
Script events now execute multithreaded — one object at a time, so multicore CPUs finally get used — plus cached event binding to avoid reflection on every call.
.NET cannot unload an assembly, only a whole AppDomain — so scripts get an AppDomainManager that pools scripts into domains and recycles domains as scripts die.
Proof in pixels: a screenshot of llSay() running for real.
Charles Krinkle contributed the interface and empty prototypes for all the LL builtin functions, so the compiler can now be tested against all kinds of scripts without erroring out. Thanks dude!
ScriptEngine is now implemented in OpenSim: a default script in every object, a touch_start event, and llSay writing to the server console. Now the 350+ builtin ll-functions can be implemented and tested in-world.
Quick status: the LSL-to-C# translator is close to complete, the compiler and script manager work, and what remains is the builtin commands and events.
Want to help with OpenSim scripting? LSL_BaseClass.cs contains a pile of empty llFunctions that all need implementing — here is how to get started.
Instead of compiling LSL directly, I first translate it to C# — simpler for other OpenSim developers to contribute to, and the same translator can double as a learning tool for people moving from LSL to C#. Written in less than a day with regex and a couple of passes.
After a meeting with Babbage Linden it is clear compilation moves to the server, and the LSO bytecode path goes away. So I'm changing course: a server-side LSL compiler working from the script text instead.
The full event pipeline works end to end: pressing a button in the test app sends an event through the ScriptEngine, into the queue, and out to the correct compiled script. Screenshot included.
The plan for the first simple version of script support: a simplified event queue with a custom threadpool, one shared AppDomain, and a basic LSL compiler that others can extend while I keep working on the core.
Builtin commands are easy — incoming events need planning. An EventHandler translates OpenSim happenings into LSL-compatible events, and an Event Queue Manager executes them on a separate thread so the server never blocks.
The architecture for OpenSim script support: a dynamically loaded ScriptEngine assembly that hooks the server's events, a compiler that turns LSL into .NET assemblies, and the loading pipeline between them.
Static and global variables, push/pop and simple scripts with math and llSay now work, and the internal state is ready for serializing so a running script can be moved between servers. Next up: local variables, custom functions, states and jumps.
A raw status table over the LSO opcodes: which ones the IL processor understands and which ones are implemented so far in the LSL-to-.NET translation.
I have a second blog for the Second Life side of things. Meanwhile: I'm writing a program that reads a program and creates a program that runs inside a program. Close to lucid programming.
Running thousands of scripts on a grid cannot mean one OS thread per script. Old ideas come to the rescue: cooperative multitasking à la Windows 3.11 and NT fibres, i.e. user space threading where each script carries its own execution state.
LSL bytecode is full of pointers — static frames, heap, jump targets — all referring to positions inside a 16k LSO file. Here is my plan for mapping them onto dictionaries and two-pass MSIL label generation.
Direct translation from the LSO stack to the .NET stack falls apart: LSO doesn't always tell you the datatype, and .NET wants its stack served in a very particular order. The fix: my own stack in a base class the compiled scripts inherit from.
The converter can now take LSL bytecode all the way to a compiled .NET .dll on disk, with events visible in Visual Studio and llSay mapped to Console.WriteLine. Messy code, but the goal is simply "Version 1: Functional".
Found the "final bug" in the LSL bytecode parser and compiled my first small function to a working .dll. One opcode and one builtin command supported — it's a start, and it works!
Can LSL bytecode map onto .NET CLR? Yes — and here is the biggest Hello World I have ever written, generated at run-time with ILGenerator and OpCodes.
The plan for OpenSim script support: read compiled LSL bytecode, convert it to .NET bytecode and let the JIT produce native code, with AppDomains providing the sandbox. Blazing fast execution inside the .NET VM.