DOS Doom Code Execution

On 2/16/2021 at 11:44 PM, ketmar said:

i vaguely remembering that Z-something is compiled into something that resembles what you described here...

The Z-Machine? Like the text adventure engine thingy, like Zork and whatnot?

On 2/16/2021 at 11:44 PM, ketmar said:

and it failed to be cross-port exactly the way new VM will fail: because it is closely tied to the internal engine structures and implementation details. and if cross-port part won't work anyway, then DOSBox is enough -- it already has a good x86 emulator.

Good point. It would rely on very binary-specific behaviour, we can't have that.

But that's why I mentioned that this code would define callbacks to specific entrypoints, rather than directly calling the running program's ABI itself.

22 minutes ago, ArgentFrequencies said:

I can only suppose the location of demo.wad is a mystery then?

https://github.com/kgsws/doom_ace/blob/45ca7f37c9bf470a1a159c4397fa0d46f807d171/wad/demo.wad

Just now, Gez said:

https://github.com/kgsws/doom_ace/blob/45ca7f37c9bf470a1a159c4397fa0d46f807d171/wad/demo.wad

Ah! My bad, thank you. How did you find it, by the way?

Just by looking at commit history for the relevant branch.

8 hours ago, Gez said:

Just by looking at commit history for the relevant branch.

I was trying that, but it wasn't working for me. Oh well, thanks anyways. This demo is fascinating! To say I have more than a few questions would be an understatement.

15 hours ago, Gustavo6046 said:

The Z-Machine?

ZScript. ;-)

15 hours ago, Gustavo6046 said:

But that's why I mentioned that this code would define callbacks to specific entrypoints, rather than directly calling the running program's ABI itself.

and... this is almost exactly what DECORATE is. ;-) it is possible to use kind-of-anonymous-function in decorate action, by using C-like codeblock, and it is possible to `return` next state from it.

of course, this won't work for custom HUD code, but any attempt to make cross-port HUD code is doomed anyway. (no, ZDoom statusbar abomination is not the good way to do it.)

7 hours ago, ketmar said:

it is possible to use kind-of-anonymous-function in decorate action, by using C-like codeblock, and it is possible to `return` next state from it.

Nono, I don't mean the engine defining action functinos and the code calling the engine. Nono, almost the opposite! The game woulid have a standard extension interface that every addon has to implement in a way, like init, gamestart, gameend, object.tick, etc. Each is optional, of course.

Think of it like what would be a kind of object file, in C. Each function can be linked in and called separately.

14 minutes ago, Gustavo6046 said:

 

Nono, I don't mean the engine defining action functinos and the code calling the engine. Nono, almost the opposite! The game woulid have a standard extension interface that every addon has to implement in a way, like init, gamestart, gameend, object.tick, etc. Each is optional, of course.

Hmm....not like it hasn't been proposed before...at a higher level, even, if raw x86 ASM is too hardcore for you :-p

3 hours ago, Gustavo6046 said:

Think of it like what would be a kind of object file, in C. Each function can be linked in and called separately.

...and we have at least two sourceports with such implementations: GZDoom and k8vavoom. both have VMs, exposed most of the engine internals into it, and it is possible to include code snippets in ZScript/VavoomC in mods. and of course, they are completely incompatible due to vastly different internals. that's why something that is not a high-level codepointer is likely won't work. Vavoom, for example, is not a lockstep engine anymore, so its `Tick()` is very different from GZDoom `Tick()`. even ACS HUD API is working slightly different due to different engine architectures.

so basically even if there will be a common API, it won't work the same across all the engines, and we're back to point zero: the mod may "technically run" in other sourceports, but won't work as intended -- unless the author will create different versions for all supported sourceports. it is more-or-less possible to make basic DECORATE work the same, but more advanced APIs... no, i don't think so.

p.s.: i am really sorry, at least three: DelphiDoom too. it's not C-like, tho. ;-)

So a bootstrap loader in an arbitrary code lump that loads VM code and passes control over to it? And then we load, idfk, Lua? Sounds good to me!

Now just tack GRUB on top and we're talking!

sorry, @jval, at least three -- DelphiDoom with PascalScript. it's not C-like, tho, so i was technically right. ;-)

@kgsws, I have some many way too many questions regarding ACE, and its possibilities. Sorry if this all rather tl;dr, you have every right to just ignore all of it, but I'd at least like you to hear me out on some of this. Some of these should already be indirectly answered by demo.wad, but its worth asking them anyways since demo.wad hasn't demonstrated everything I'm wondering about.

Behavior section: 

1. Can absolutely any value be altered for a linedef, sector, or enemy to something "non-vanilla"? (e.g. key-locked lifts, secret + damaging floors, health buffs?)
2. If the above is true, then can it be done for individual things, and to what limit, if any? (e.g. a pinky has buffed pain chance, a different pinky has buffed speed?)
3. Is it possible to alter an enemy's behavior into something more complex than "chase the player and attack at random?" (e.g. a pinky dodging an incoming rocket?)
4. From my own understanding of Doom's vanilla behavior, linedef actions can alter sectors, but linedefs cannot alter other linedefs. Could ACE make the latter possible? (e.g. using a switch to change a texture or action special?)
5. Can a linedef be given multiple actions in a single press, with each action being given a different tag? (e.g. simultaneously opening a door, starting a crusher, and building a staircase, where none interfere with each other?)
6. Can sectors be given actions, and to what limit? (e.g. teleport the player when they enter a sector rather than pass a linedef?)
7. Can you make new things without replacing existing things, and to what limit? (e.g. adding 3 new enemies without having to remove 3 old ones?)

Keybindings section:

1. Can you alter keybindings? (e.g. swapping W and left click without going into the setup menu?)
2. If the above is true, can you apply a keybinding to something not originally possible with a keybinding? (e.g. using right-click to raise a floor by 1 unit each tic the button is held?)
3. If the two above are true, then could you apply a keybinding to something akin to an "alt fire?" (e.g. incorporating Doom Eternal weapon mods into vanilla?)

Rendering section:

1. Could aspects of Doom's rendering be altered? (e.g. removing the fish-eye lens effect from the sky, applying the spectre's fuzz to a texture?)
2. If the above is true, Could new rendering exist for textures and sprites, and to what limit? (e.g. alpha or additive blending? Even multiply or difference blending?)
3. If the two above are true, could you have animated skies? (e.g. PS1 fire sky?)
4. If the first two are true, could true 3d exist in the Doom Engine? (e.g. at least a wireframe cube, spinning?)
5. Can the colour palette be extended?
6. Could you take a sprite and apply it to a wall like you can with FLATs? (e.g. cardboard cutout of a Mancubus like Nuts 2?)
7. Could you get 16-sprite rotations beyond the default 8?
8. Could textures be scaled to be larger, as a way of increasing detail?

Physics section:

1. Can a sector slow down the player, alter the player's gravity, or have reduced friction? (i.e. Mud, Moon, and Ice physics?)
2. Can a sector be a conveyor like in Boom or Doom 64?

Bugs/limitations section:

1. Can things be given limited height, similar to the collision of items from the player? (e.g. players running under Cacodemons, or over Imps?)
2. Can any of the static limits be increased or outright removed? (e.g. Visplane limit, blockmap limit, vissprite limit?)
3. Can the limit of a texture's resolution be increased? (e.g. From 256*128 up to, say, 768*384?)
4. Can the limit of levels in an iwad be increased? (e.g. from 32 maps to 50 maps?)
5. Can the framerate be increased? I know it's possible to increase the speed of DOSBox, but if that's the method we're going for, could all the activity in the game be slowed down to half speed, then double the DOSBox speed to produce 70 fps from 35 fps?
6. iirc, Doom's music plays at a limit of 9 channels. Could this limit be increased?
7. Could animations be made to play smoother (i.e. more frames per second) than they normally do? (e.g. faster reload animations?)

One last question:

Can it run Doom? Also, there's over 700 words in this comment, christ almighty I really need to work on being concise.

On 2/24/2021 at 9:37 AM, ArgentFrequencies said:

@kgsws, I have some many way too many questions regarding ACE, and its possibilities. Sorry if this all rather tl;dr, you have every right to just ignore all of it, but I'd at least like you to hear me out on some of this. Some of these should already be indirectly answered by demo.wad, but its worth asking them anyways since demo.wad hasn't demonstrated everything I'm wondering about.

Well. Since this allows you to run any custom code, almost everything is possible.

So, answer to all your questions is yes. And wherever you are asking about limits, answer is RAM or CPU speed.

Everything depends on ones ability to code required feature.

Here are few more notes:
- Example demo.wad has custom linedef actions. It even uses Hexen map format.

- You can add new things without replacing old things in latest ACE Engine WAD example. It reads DECORATE.

- Alt-fire was planned feature for ACE Engine.

- ACE Engine allows textures larger than 64k. Large width is OK now. Height is still limited to 128 pixels though.

- Example demo.wad has one FUZZy texture. (And custom sector lights, which also affects renderer.)

- Finite thing collisions was planned feature for ACE Engine.

 * (basically required for planned 3D floor support)

- ACE Engine already allows you to extend some static limits. (visplanes, vissprites and drawsegs)

 * (full limit removal is possible but was not planned)

- More maps were planned feature for ACE Engine. Even with custom map names. And with TITLEMAP feature.

- For true 3D you would be rewriting rendering code entirely.

- For more than 256 colors, you would be rewriting drawing functions entirely.

And finally. Yes, you can run doom with this. Or any other game you are willing to source-port.

At one point i had an idea to start a Linux kernel just to run compiled ELF files. (the issue is: you can't exit to DOS)

12 hours ago, kgsws said:

Well. Since this allows you to run any custom code, almost everything is possible.

So, answer to all your questions is yes. And wherever you are asking about limits, answer is RAM or CPU speed.

Everything depends on ones ability to code required feature.

Here are few more notes:
- Example demo.wad has custom linedef actions. It even uses Hexen map format.

- You can add new things without replacing old things in latest ACE Engine WAD example. It reads DECORATE.

- Alt-fire was planned feature for ACE Engine.

- ACE Engine allows textures larger than 64k. Large width is OK now. Height is still limited to 128 pixels though.

- Example demo.wad has one FUZZy texture. (And custom sector lights, which also affects renderer.)

- Finite thing collisions was planned feature for ACE Engine.

 * (basically required for planned 3D floor support)

- ACE Engine already allows you to extend some static limits. (visplanes, vissprites and drawsegs)

 * (full limit removal is possible but was not planned)

- More maps were planned feature for ACE Engine. Even with custom map names. And with TITLEMAP feature.

- For true 3D you would be rewriting rendering code entirely.

- For more than 256 colors, you would be rewriting drawing functions entirely.

 

And finally. Yes, you can run doom with this. Or any other game you are willing to source-port.

At one point i had an idea to start a Linux kernel just to run compiled ELF files. (the issue is: you can't exit to DOS)

Incredible, thank you so much. Just out of interest, why are heights still limited to 128 pixels? I'm ok with technical answers.

On 2/26/2021 at 9:32 AM, ArgentFrequencies said:

Incredible, thank you so much. Just out of interest, why are heights still limited to 128 pixels? I'm ok with technical answers.

This: https://github.com/id-Software/DOOM/blob/master/linuxdoom-1.10/r_draw.c#L142.

While original DOS Doom function is slightly different, this line still explains it.

Notice how texture mapping uses & 127 calculation. This forces horizontal resolution to exactly 128 pixels. Also, it means that textures smaller than 128px will be rendered incorrectly, if fully visible.

With ACE you can of course change it. I did not do it since it would require quite changes in drawing functions.

18 hours ago, kgsws said:

This: https://github.com/id-Software/DOOM/blob/master/linuxdoom-1.10/r_draw.c#L142.

While original DOS Doom function is slightly different, this line still explains it.

Notice how texture mapping uses & 127 calculation. This forces horizontal resolution to exactly 128 pixels. Also, it means that textures smaller than 128px will be rendered incorrectly, if fully visible.

With ACE you can of course change it. I did not do it since it would require quite changes in drawing functions.

That makes me wonder why they did this back in 1993. Something about filesize or quicker rendering? Also, let's say the drawing functions were changed using ACE. How much work would have to go into patching out the texture height limit?

On 2/28/2021 at 11:27 AM, ArgentFrequencies said:

That makes me wonder why they did this back in 1993. Something about filesize or quicker rendering? Also, let's say the drawing functions were changed using ACE. How much work would have to go into patching out the texture height limit?

It is faster if you have this value already set in the code.

If i remember correctly, this is the only reason for texture height limit. So it won't be too hard. But it would require extra hooks for texture height decision, which would add rendering time.

It is easier to design maps if textures that tile always tile on the same value. If some walls tiled on 96 pixels, some on 128. others on 112 it would become hard to combine different textures and knowing how the rooms would look with a certain height without seing it. Having a fixed value makes textures more interchangable. It is easier to align textures with today's tool, but back then there was no visual preview. No automatic alignment of textures of any kind and having to manually key in firstcol and firstrow values (x and y align).

6 hours ago, kgsws said:

It is faster if you have this value already set in the code.

If i remember correctly, this is the only reason for texture height limit. So it won't be too hard. But it would require extra hooks for texture height decision, which would add rendering time.

I wonder if there's a cheap workaround for this. Could you use ACE to increase the scale of the pixels by twice the height, so you can get textures which are "256 pixels tall", while the information is still a 128 pixel resolution, like the game wants? Think of the Commodore-64's multi-colour mode.

22 minutes ago, zokum said:

It is easier to design maps if textures that tile always tile on the same value. If some walls tiled on 96 pixels, some on 128. others on 112 it would become hard to combine different textures and knowing how the rooms would look with a certain height without seing it. Having a fixed value makes textures more interchangable. It is easier to align textures with today's tool, but back then there was no visual preview. No automatic alignment of textures of any kind and having to manually key in firstcol and firstrow values (x and y align).

I know about the tiling thing, but I suppose the issue was that 256 pixel textures loaded too slowly then?

On 3/3/2021 at 3:36 AM, ArgentFrequencies said:

Could you use ACE to increase the scale of the pixels by twice the height, so you can get textures which are "256 pixels tall", while the information is still a 128 pixel resolution, like the game wants?

Well, you can do anything, so ...

But i guess you would like to know fastest and simplest way to stretch textures. Changing texture scaling in drawing function should be quite simple, but the same function is used to draw sprites. (It would break the sprites.)

So you would either have to account for this, or duplicate and modify original function and patch a few calls to the original function.

10 hours ago, kgsws said:

Well, you can do anything, so ...

But i guess you would like to know fastest and simplest way to stretch textures. Changing texture scaling in drawing function should be quite simple, but the same function is used to draw sprites. (It would break the sprites.)

So you would either have to account for this, or duplicate and modify original function and patch a few calls to the original function.

I suppose the code should check whether it's drawing a texture or a sprite first, then apply the stretch. But as you said, this would be rewriting drawing functions. Unless you mean the sprites would simply stretch up, I already had this idea:

If the texture is being taken from either doom.wad or doom2.wad, only read every second row of pixels, and form a new image from that. If it's a sprite, only apply the image once. If it's a texture, repeat it twice. For any other wad, this process is skipped, and any 128 tall textures will get stretched to 256. Here's some mockups I made of what I mean:

On 3/10/2021 at 7:35 AM, ArgentFrequencies said:

Unless you mean the sprites would simply stretch up, I already had this idea:

Actually no. Sprites would just break. Transparent pixels are not sent to drawing code at all. Sprite column is often composed of multiple pixel stripes. Each stripe would be stretched by patching drawing function, but spacing between multiple stripes (of the same column) requires extra patch somewhere else. Unless you convert your spires non-standard way.

On 3/10/2021 at 7:35 AM, ArgentFrequencies said:

If the texture is being taken from either doom.wad or doom2.wad, only read every second row of pixels, and form a new image from that.

This would add up extra time for texture recalculation. And you would have to write this conversion function.

All these changes for worse looking graphics are just not worth it. At this point it is much better to rewrite drawing functions for your specific needs.

It would be easier to just copy and patch original function to create 256px variant and then just add hooks to decide which function to call based on texture height. You can do this as many times as many different texture heights you want to support. And it would likely only work on power-of-two values.

Small note:

You won't find this code in original DOS EXE. https://github.com/id-Software/DOOM/blob/master/linuxdoom-1.10/r_draw.c#L142

Actual implementation in EXE is something more like this:

*dest = dc_colormap[dc_source[frac >> 25]];

Notice how shifting uint32_t by 25 bits leaves exactly 7 bits for texture resolution. This is a nice optimisation to avoid extra operation (& 127). You only have to scale your frac and fracstep before you enter drawing while loop. Which is worth it if you draw more that a few pixels (= almost always in doom maps).

(There are even more special optimizations, like faster dc_colormap indexing.)

it makes me sad seeing people write this off as cool but useless. some of us still have shitty old computers eager to play advanced vanilla mods )`:

Your destiny is now to get Doom running inside Doom. If everything can run Doom, why not Doom itself? Adding a menu option that launches Doom, for example.

I am quite sure someone posted this before, but a quick search revealed nothing and I don't really wanna manually read 9 pages for the sake of a joke lol

10 hours ago, kgsws said:

All these changes for worse looking graphics are just not worth it. At this point it is much better to rewrite drawing functions for your specific needs.

 

It would be easier to just copy and patch original function to create 256px variant and then just add hooks to decide which function to call based on texture height. You can do this as many times as many different texture heights you want to support. And it would likely only work on power-of-two values.

Again, I'm not really an expert with this stuff, it's just an idea to throw around I guess.

7 hours ago, roadworx said:

it makes me sad seeing people write this off as cool but useless. some of us still have shitty old computers eager to play advanced vanilla mods )`:

I exclusively like to make wads that are Boom compatable because of the novelty of limitations. GZDoom is just too many unnecessary bells and whistles.

1 hour ago, ArgentFrequencies said:

I exclusively like to make wads that are Boom compatable because of the novelty of limitations. GZDoom is just too many unnecessary bells and whistles.

yeah, working with vanilla/boom and working with the limitations is fun and simple. not only that, but it also encourages creative workarounds for things instead of just having something built-in do it for you

2 minutes ago, roadworx said:

yeah, working with vanilla/boom and working with the limitations is fun and simple. not only that, but it also encourages creative workarounds for things instead of just having something built-in do it for you

The 10 sectors contest?

5 hours ago, roadworx said:

it makes me sad seeing people write this off as cool but useless. some of us still have shitty old computers eager to play advanced vanilla mods )`:

Not sure what the latter has to do with the former?  You seem to believe that this somehow magically allows new code to be added without impacting performance?  Even if you're using a 90s machine with pure DOS, there have existed and still exist source ports for DOS.  A mod author could hack on those code bases to make a custom port with whatever feature they want.  This is no different except that instead of the DOS kernel loading the executable, doom.exe does it.

That's why people are "writing this off," since outside of some very niche technicalities I've simplified out of my explanation, this doesn't permit anything new.  It's just functionally equivalent to embedding a source port inside of a WAD file.  Which is cool to see, but I wouldn't call the result a vanilla mod since the mods don't actually run on the vanilla engine.

53 minutes ago, Albertoni said:

Your destiny is now to get Doom running inside Doom. If everything can run Doom, why not Doom itself? Adding a menu option that launches Doom, for example.

 

I am quite sure someone posted this before, but a quick search revealed nothing and I don't really wanna manually read 9 pages for the sake of a joke lol

The earlier versions of the exploit required starting a map at which point it would patch the code and then restart the game so yes this technically was done.

ignore this message

When this releases entirely I will get ready to make entire games with this.