The design should also subsume dissasemblers.

I have a vague idea that an assembler and a disassembler could share a file describing the instruction set. The file should contain definitions for each instruction, in the format opcode format name. An assembler could then define format words to write opcodes into the dictionary, and a disassembler could define format words to mask opcodes and print mnemonics.

Also reconsider the stack storage for instruction operands. The current design with three items per operand has proven unwieldy.

Also consider that operands can affect which instruction opcode to select, or conversely, the opcode can affect how operands are encoded.

Comment from @alex-shpilkin:

Just looked through your STM8 assembler. Three minor nits: first, -ADDR defined as DEADBEEF16 needlessly precludes using your assembler on a 16-bit Forth (such as, presumably, one on the STM8 itself); you might as well use a reserved 16-bit address for this (680016 , say). Second, I don’t understand why you define !PREFIX and PREFIX! to mean exactly the same thing. Third, I wonder why you call your assembler prefix when (as far as I’m aware) this flavour is rather usually called postfix (the instruction goes after its operands).

On to the more substantial points.

The foo/bar (or, if you will, unary/binary) classification of instruction encodings is what I initially used in my assembler as well. (Figured it out by manually comparing the encoding tables from the manual and then found out, to much embarassment, that it’s essentially described on the Wikipedia page for the STM8 precursor ST7.) What I use currently is a bit more complicated, as you can see, with seven separate lookup tables as well as some special-cased instructions. The point is to prevent the assembler from encoding nonsense instructions, as some of them are quite unobvious (ahem, LDW (X), X but not LDW X, (X), seriously?). I understand that’s a trade-off, though, and did not see some of the encoding similarities that you point out.

I don’t see why the object your >MARK produces (orig in ANS terms) is two cells wide if it is always of the form addr-1 addr. Meanwhile what BEGIN, produces (dest in ANS terms) is one cell wide, which prevents manipulation of (the assembler equivalent of) the control stack without knowing exactly what’s on it (no CS-PICK or CS-ROLL). I also don’t see how your LONG! could possibly work, given that it tries to overwrite a two-byte relative (conditional or unconditional) jump with a three-byte absolute (unconditional) jump. (It’ll get even worse when or if you implement BTJF and BTJT.)

Just realized I forgot to implement AHEAD, in my assembler. Whoops.

Calling the double-indirect mode ) is both perversely logical and supremely confusing (and the vendor syntax is no better). Regarding X) vs. (X), I just don’t know what the convention is. I remember reading an article from Forth, Inc. (?) about their conventions for Forth assemblers, but I can’t seem to find it anymore.

Regarding operand syntax, your requirement that an absolute address be encoded as itself both is and isn’t a matter of opinion. It is in that you can do it by having the mode words set global flags, and lose nothing compared to my approach. It is not in that you do lose something if you mark other modes by pushing a special value (-ADDR). What you lose is the ability to partially evaluate your assembler: produce from the same source, say, an ]ADD, such that

: foo ... ( address ) [ ),Y) ]add, ... ;

takes address from the run-time stack but does all the operand parsing and mode selection at compile time. My assembler could do that with minimal modifications except that I have not figured out a suitable interface and getting : ADD, to define ]ADD, in ANS Forth is a royal pain.