Source link points to an empty…
yeah, it’s just a fantasy. I’m waiting on the mail for my picocalc and daydreaming
The thing is we have both Forth and Lisp for the PicoCalc right now, in the form of zeptoforth and uLisp respectively. Of course, attempting to combine the two would be difficult, even though at one point I did write a garbage-collected language on top of zeptoforth, named zeptoscript (I haven’t touched it in a while though).
Yeah, I’ve been playing with Zeptoforth last night and it’s a great foundation for me to build the Forth that I want. I guess I didn’t understand That Forth is the language/OS/everything that you make it and that you could even build a scripting language over the top of it. I only discovered it the other day and my mind is blown. I was fascinated with with RetroForth but now I understand that you can just build the RetroForth syntax over Zeptoforth if I wanted.
Does Zeptoforth support saving and restoring a full system image — all definitions,
variables, and state — to SD card? Not just compile-to-flash, but a snapshot I can reload to
get back exactly where I was.
Yes, with zeptoforth you can easily turn it into whatever you want it to be. If you really, really wanted to you could implement a Lisp in zeptoforth, just like how uLisp is a Lisp implemented in C. However, it also aims to be a very useful system in its own right and not merely a base layer to build something bigger on.
Note, though, that currently saving state is not supported, partly because time and much of the hardware state cannot be saved by their very nature – even if one did a full RAM-dump to SD card, anything time or hardware-related would be out of sync after booting up again, and partly because zeptoforth was never envisioned as an image-based OS as it originally was created for devices whose only means of saving state was the flash dictionary (and support for blocks and SD cards was only added later, and then is only fundamentally optional).
One could, with a lot of work, extend it so that save-and-restore would work; the hardest part would be handling the disruption in hardware state between the save point and the restore point. This would require modifying practically all the hardware drivers at a very basic level, as they almost all assume that hardware state is continuous in nature.
1 Like
What if the approach was: save only the dictionary and user data to SD, then reinitialize
hardware from a boot word on restore? Is the dictionary structure accessible enough to read
through it and write it out to a file?
Second question: is there a hook into the outer interpreter where I could intercept a token
before it’s looked up in the dictionary? I’d like to check the first character and dispatch
differently — like treating @ or / as prefixes that expand into something else before
evaluation.
The problem is the user data is inseparable from system state. All operations that are called that could be interrupted and need recovery would have to have checks added and either recover or fail cleanly.
Actually, there already is such a hook, which I currently use for implementing local variables.
Thanks — the outer interpreter hook is exactly what I was hoping for.
I understand about image persistence — it makes sense that it wasn’t a design goal for
Zeptoforth. My longer-term goal is to build a general purpose Forth computer on the PicoCalc
— not just embedded firmware, but a personal computing environment where the user lives in
Forth full-time. Dictionary, notes, tools, everything persistent across power cycles. Think
Lisp machine philosophy on a microcontroller.
Zeptoforth on the PicoCalc is a fantastic platform to learn and prototype on.
Appreciate you taking the time
You can do it as of now. You have the opportunity to compile into the flash memory, which will remain intact though power cycles. Also, you can store user data in files as in any nornal computer.
What I was originally dreaming about was Lisp machine-style image persistence — save the
entire system state, power off, power on, and be exactly where you left off, stack and all.
tabemann explained well why that’s hard to bolt onto an existing system.
But for now, compile-to-flash plus files is a solid foundation to build on.
I should note that if you write something that lives in its own world and has its own garbage-collected heap and GC-aware stacks on top of zeptoforth making something image-based becomes much easier.
However you still have to contend with the fact that the outside world simply cannot be put on pause, e.g. if you choose to support external SD cards for anything other than the images themselves if you have a file open someone could save an image and turn off the PicoCalc, remove the SD card, insert it in another machine, delete the file you have open, and reinsert the SD card into the PicoCalc prior to booting it again. It is eventualities like this that you will have to deal with at many points if you choose to implement image-based persistance.
Thanks. You’re right about external state — my plan is
to keep everything inside the PSRAM image and only touch the SD card for saving/loading the
image blob itself. No open file handles to go stale. Export to FAT32 only when I need to
move something to a laptop. Minimizing SD card wear.
I considered building a managed world on top of Zeptoforth like you suggested, but wouldn’t
that mean I’d end up with two of everything — two dictionaries, two outer interpreters, two
compilers? Am I understanding that correctly? Isn’t that getting a little too fat?
I’m considering attempting to write my own minimal kernel with image persistence designed in from the start — attempting a clean separation between hardware state (kernel handles on boot) and user state (the image). Using your source code as reference for how to do things on the RP2350. The goal is to eventually translate your .fs libraries (WiFi,
TCP/IP, SHA-256, display drivers) to run on top of my kernel so I can benefit from all that
work you’ve already done.
Appreciate you engaging with this.
I would recommend keeping the code I have written as-is and using it outside your persistent world. The persistent world would get saved and restored, while my code would not, and the persistent world would interface differently with the outside world than my code would.
And yes, this would mean you would end up with two separate dictionaries, two outer interpreters, two compilers, but I do not really see this as being a problem, because you would essentially be dealing with two distinct if Forthy languages, just like how uLisp creates an inner language (a Lisp) on top of an outer language (C).
I would also recommend not repeating all the work that went into creating zeptoforth but rather using zeptoforth to support your code as essentially a user-level application environment that exists within it (like how, say, Racket provides a Scheme environment on top of Linux). If you choose to recreate what has already gone into zeptoforth you will waste a lot of your time for little gain, and will not be able to take advantage of future improvements upon zeptoforth.
EDIT: actually you’re completely right. the work wouldn’t be worth the reward and then I don’t get upgrades. I should just think of my system as an app on Zeptoforth.
I Appreciate this advice and I understand the pragmatism behind it. You’re
right that I’ll save a lot of time building on top of Zeptoforth rather than reinventing it!
But my core goal is a system that’s one language top to bottom — something I can fully grok,
metal to UI. That’s actually one of the main reasons I chose Forth over Lisp. Having
Zeptoforth as a substrate I don’t fully own feels like it defeats the purpose, even if it’s
the faster path.
I’ll be honest — there’s a good chance I’ll try to build my own kernel, fail (I have zero
experience with ARM assembly), and come back to your approach. But I think image persistence
is such an essential feature for a Forth console that it needs to be designed in from the
ground up, not bolted on.
Either way your codebase is the best reference I could ask for, and I can still benefit from
your future improvements — I’ll just be translating and adapting the .fs libraries rather
than running them directly on top. More work on my end, but that’s the tradeoff for owning
the whole stack.
Thanks for taking the time to think through this with me. I have another chip so I can take both paths!
The problem with having one environment top to bottom with an image-based system is you always need to have something that lives outside the image. Before you can load an image from an SD card you need to have some way of mounting the SD card and reading a file from it without having an image loaded.
1 Like
One thing to consider is that most high-level, garbage-collected languages are going to be slower and less suited for real-time operation than a native-code, low-level Forth on the same hardware. This will especially be a concern when it comes to graphics, networking, and storage-related code. I tried writing a graphics driver in zeptoscript, which is a higher-level, garbage-collected language on top of zeptoforth, and it took a lot of work before its performance was even remotely acceptable. I cannot imagine writing a networking stack with responsiveness on the order of milliseconds in a language like zeptoscript.
Because of this, it is a good idea to implement these sorts of things in an underlying lower-level language and access them through bindings from the higher-level language. This is how these things are typically done in languages like uLisp, MicroPython, and CircuitPython.
Also, I should note that implementing preemptive multitasking in a garbage-collected higher-level language is non-trivial due to the need for support for concurrent access to the garbage-collected heap and for concurrent garbage collection. This is a hard problem to solve well, and it often is simply avoided by making multitasking in the higher-level environment cooperative (e.g. with coroutines), having separate heaps for each preemptive task, and/or having a Global Interpreter Lock (which negates the ability to take advantage of multiprocessing within the garbage-collected environment). In zeptoscript I took the coroutine route and simply did not support preemptive multitasking within the zeptoscript environment.
1 Like
Thanks for the warning. I definitely don’t want to repeat the zeptoscript graphics driver pain.
The good news is I’m not really planning a garbage-collected language. The persistent world would still be Forth — manual memory management, no heap, no GC. Just threaded code instead of native code, with its own dictionary and named data buffers in PSRAM. No scanning, no pauses, no GIL.
The plan is exactly what you’re recommending: graphics, networking, storage, and anything
timing-sensitive stays in Zeptoforth. My world calls into your drivers through bindings.
Multitasking would be cooperative (activity switching, not preemptive threads).
So it’s more like a persistent Forth layer on top of Zeptoforth than a new high-level
language. The only real overhead vs native Zeptoforth is the threaded code dispatch — one
indirect jump per word instead of inline machine code. Does that change your assessment at
all?
Are you envisioning a typed world with true stack safety? Because even if you are, there is no reason you could not compile to native code, because then you could generate instructions that check things like types and stack depth automatically. Really, the main advantages to threaded code are either when you want to optimize for code density or, as you may envision, want to support automatically relocating arbitrary code from anywhere in the memory space in an easy fashion. In general, threaded code will be slower than inlined native code with constant folding, as zeptoforth does, on many modern architectures like ARM Cortex-M (even if direct-threaded code may be faster on some architectures than naive subroutine-threaded code). (The problem with relocating native code is handling branch and call instructions with limited PC-relative distances, as relocating such instructions may move them out of the range of their original destinations.)
For instance, my zeptoscript has support for dynamic typing, but because it was implemented using the zeptoforth code generator rather than its own code generator it did not have an effective way of enforcing stack safety (which is a big part of why I have largely abandoned it, because the combination of typing but no stack safety actually made it very easy to crash, because it would crash when it tried to verify the types of garbage values after having underflowed the data stack).
Also note that if you are not doing GC there really is no good reason why you could not leverage zeptoforth’s preemptive multithreading support. The reason why I did not support preemptive multithreading in zeptoscript was that I could not efficiently make it work with GC. It should be noted that there are significant disadvantages to cooperative multitasking, such as that any one misbehaving task can easily lock up the entire system, and that it is hard to guarantee fair distribution of processor time to each task. (Of course, zeptoforth does have support for very lightweight cooperative ‘actions’ (also known as ‘fibers’) with its ‘action scheduler’ mechanism, which is useful when one needs to have very many concurrent things happpening at once where the relative heavyweight-ness of tasks would be prohibitive.)
1 Like
That’s eye-opening! I was assuming threaded code was necessary for image persistence. Since SRAM is memory-mapped at a fixed base address on the RP2350, couldn’t I just compile native code into PSRAM, save the whole region to SD, and restore it to the same address on boot? All branch targets would still be valid since the base address never moves. Is that correct?
And good to know about preemptive multitasking — since I’m not doing GC I’ll plan to use yours directly.