First, let me start with a big thanks to Nemesis and piroxilin for providing Teradrive disk images and firmware images respectively. Having easy access to software/firmware to reverse was a big help. Now for the info I've figured out:
M68K Memory Map (standard MD hardware omitted)
000000 - ??????: Boot ROM, PC memory space or cart depending on register settings
A14000: Non-standard TMSS register
AE000X: M68K side specific Teradrive regs (only 2, haven't checked mirroring yet)
AFXXXX: PC IO address space
BXXXXX: 1MB window into PC memory address space
Most of the Teradrive-specific registers are accessible from the PC IO space. They are as follows:
Key: X = writeable by both CPUS, M = writeable by M68K only, R =
read-only, 1 = read-only, always 1, 0 = read only, always 0
Note that some bits marked as always 0 or always 1 may have some function that I just haven't stumbled across yet
1160: XXXXXXXX Unclear function, must be set to $21 when booting M68K. Set to 0 by BIOS after initial MD-side boot. 1161: 11111111 1162: 110XXXX0 Sets the base address for the memory window in the PC address space for accessing an 8KB chunk of the M68K address space BIOS initializes this from CMOS and defaults to $CE, if the CMOS value is invalid. This corresponds to CE00:0000 (segment notation), $CE000 (physical address) 1163: XXRRXXXX bit 0: Unclear function. Must be set when booting M68K bit 1: Enables the memory window in PC address space. Must be clear when booting M68K bit 4: Returns value written to AE0001 bit 0 bit 5: Returns value written to AE0001 bit 1 1164: MXXXXXXX bit 0: Releases the M68K from reset and pauses the 286 when set. Puts the M68K into reset and releases the 286 when cleared Experimentation suggests that certain other register changes make clearing this bit ineffective, but not sure which ones. bit 1: 1 = Cart/expansion port mapped in at $0 on M68K side, 0 = Teradrive M68K boot ROM or PC memory bit 2: Controls the video switch. 0 = PC VGA, 1 = MD VDP bit 4: Appears to enable some sort of test mode. Behavior of 1165 and AE0003 change when this is set bit 6: Unclear function. Set before purposely wedging CPU (reading from VDP data port when writes are configured) on cart TMSS failure bit 7: Appears to be set at boot. Can't be cleared by the 286, but can by the 68K 1165: 00R0RR0R bit 0: Indicates PC/MD switch setting, 0 = MD boot, 1 = PC boot bit 2: Indicates video switch setting, 0 = "Video", 1 = "RGB" bit 3: Unclear. Normally 0, occasionally 1 bit 5: Indicates MD hardware has been "unlocked" ('SEGA' has been written to $A14000) 1165(alt): RRRRRRRR Returns a value in a fixed sequence each time it's read (sometimes, may be impacted by other register settings). D5,7F,00,AA,55,FF,80,2A 1166: XXXXXXX0 Low portion of M68K address visible in 8K memory window (bits 1-7 of reg = bits 13-19 of address) 1167: 0000XXXX High portion of M68K address visible in 8K memory window (bits 0-3 of reg = bits 20-23 of address)
Additionally there are 3 registers only accessible from the M68K address space.
AE0001: XXXXRRXX bit 0: Value set here will be returned in AF1163 bit 4 bit 1: Value set here will be returned in AF1163 bit 5 bit 2: Returns state of AF1163 bit 6 bit 3: Returns state of AF1163 bit 7 bit 4: Appears to be used by the M68K to track whether it's the first time the 68K has been booted. Starts out 0 and is set to 1 during boot by M68K bit 5: Appears to be used by the M68K to track whether a cart was found during initial boot. Will refuse to unlock MD side when set bit 6: Set by M68K side if the cart passes the TMSS check and PC/MD mode switch is set to PC bit 7: Maps the PC address space at $0 when set when 1164 bit 0 is clear AE0003: XXXXXXXX bits 0-3: Selects the 1MB bank visible at $B00000 AE0003 (alt): XX10XXXX A14000: XXXXXXXX XXXXXXXX Unlocks 286 access to MD hardware when 'SEGA' is written to this address, Locks it when other values are read. Byte access seems to lock the M68K. Note that writing to this reg is not required for M68K access to the VDP like normal TMSS and the version field of the version reg is 0 on the Teradrive. Unsure if this is readable (definitely not byte-readable)
PC/MD switch: Indicates to firmware the desired boot mode (via 1165). Resets the machine when toggled.
RGB/Video switch: Indicates to firmware whether a 15KHz mode is desired on the PC side at boot (via 1165). Controls whether composite output always contains MD video (RGB setting) or whether it is switched by 1164 like the VGA connector (Video setting).
M68K Firmware info:
The M68K boot ROM is located inside the additional mask ROM (called the ROMDISK by prioxilin, though only part of it appears to contain a disk-like structure) at offset $43000. Unclear how this ROM is accessed from the PC side or if other portions are accessible on the MD side.
The M68K firmware is responsible for displaying the SEGA logo at boot
time, doing a basic TMSS check on cartridges (is 'SEGA' present at $100
or $101) and authenticating requests to unlock the hardware for the PC
side. It recognizes two special header strings at $100 and has different
boot behavior if they are found.
'SEGA TERA286': Mostly normal, but allows the PC to unlock the MD side even if this is present (normally a cart being present prevents PC access)
'SEGA TERA68K': Unlike a normal cart, this is booted directly after the Sega logo (normally M68K returns control the PC side after Sega logo and requires the 286 to reset it again to boot the cartridge)
I'm guessing the prototype Mega CD unit used SEGA TERA286.
The initial M68K boot code also checks for a special ISA option ROM in the $C0000 region of PC memory. This ROM must contain the text 'PRODUCED BY OR UNDER LICENSE FROM SEGA ENTERPRISES Ltd.', 0 (that zero is there to indicate a literal null byte) at an even address. If found, the 68K will jump to the offset stored at offset 4 of the 1KB chunk that the text was found in.
To unlock the MD hardware from the PC side, write $21 to 1160 (probably 0 at the start of the process), $1 to 1163 (probably already set to that, but best to be safe), 0 to $1166 and $1167 (unless you have located your PRODUCED BY... text in a funny location) and then write $81 to 1164. You must have the PRODUCED BY... text at an even address in RAM before doing so or you will not gain access. As long as that text is in "conventional" memory (i.e. in the low 640KB of the PC address space), using 0 in 1166/1167 is fine. Otherwise, you must specify the upper 11 bits of the physical address that contains the text in 1166/1167. You may also do that to speed things up slightly when the text is located in conventional memory, but the speedup is negligible given the delay involved in displaying the TMSS license screen.
If you are successful, bit 5 of 1165 will now read 1 and you will be able to use 1162,1163,1166 and 1167 to access the MD hardware via the memory window mechanism, but the 68K will not be running. Puzzle Construction seems to operate in this way with game logic running on the 286, but the YM3438, MD IO and MD VDP used for the game mode and launcher (standard VGA is used for the editor screens) and the cdram utility (can backup BRAM carts to disk).
If you instead want to run code on the 68K, first unlock the MD hardware as above. Then write your initial M68K code to MD work RAM via the window mechanism. Then go through the unlock procedure again, but this time put a value from $A0-$FF inclusive in 1166/1167. This will cause the M68K Teradrive firmware to jump to $FF0100.
If there is a way to let both CPUs run simultaneously (normally they are mutually exclusive)
If there is a mechanism to allow the 286 to receive MD interrupts of the M68K to receive PC interrupts
What the remaining bits in the above registers do
How the hardware knows to use offset 0x43000 for the M68K firmware (seems like a weird address for it to be hard coded)
How the rest of that additional firmware ROM is accessed