ID24 - a new feature set standard

[elf-alchemist]

2 minutes ago, Pancrasio said:

Yeah but I think it overshoots. Improvements for MBF21 should stay improvements for MBF21, and for the other things just use zdoom offshoots or whatever. At one point you have to accept the limitations of your engine if you want to map for it, I remember one guy asking for slopes and 3dfloors in the "mbf21 aspirations" threads and I found that completely ridiculous, why do we need to turn mbf21 into zdoom? Just give me better dehacked and fix bugs, the rest leave it as is.

Certainly, on that we both agree that new complevels should generally, just do that, bugfixes+dehacked+mapping, keeping it simple.
 

4 minutes ago, Pancrasio said:

As for the other thing, nobody knows how to use Eternity because no one uses it, but it's just a matter of, well, using it. Theres's no need for weird propietary additions or pointless evolution of ports where limitations constitute their charm when it is possible to learn Eternity. I get it that it's starting from scratch somewhat, but these developments prove there was a demand for that kind of thing and an actual reason to get invested with it.

I definitely do agree, it is a shame that Eternity's features never quite picked up steam like other port's did, the advanced faux-3d mapping features I did see on any video about Heartland had me metaphorically drooling for them, one thing I do see not helping Eternity's case much is presence of it's own, exclusive lumps and features and it's (AFAIK) scripting-like definition language, as standardization efforts like UMAPINFO and MBF21 came about, leaving Eternity's own greater feature set somewhat out of the question for wide adoption. What with DEHACKED being a de-facto "lingua franca" for doom modding and UMAPINFO just generally being better than most MAPINFO with it's bossaction keyword alone, while not being as needlessly feature crept as something like ZMAPINFO or not being port exclusive like all MAPINFO derivatives before it.
 

18 minutes ago, Pancrasio said:

As it stands now, id24 is useless and kind of cumbersome.

With these efforts coordinated by developers and mappers inadvertently sowing the seeds for what ID24 would later become, a complevel that mostly introduces new feature lumps (GAMECONF, DEMOLOOP, SKYDEFS, SBARDEF, and custom intermission maps), explicitly setting a baseline for feature requirements, (UMAPINFO, MUSINFO, DEHEXTRA, DSDHACKED, etc), and also creating a proper, well defined set of specifications for them, nothing being left unclear or undefined.

I personally don't see this as anymore cumbersome than adding any other new lump, ID24 just happens to add at least four of them. Limitations are great and I love them, part of that is why I personally dislike playing ZDoom-derivatives theses days, they don't suit my preferences anymore like they once did. But in the end, it won't matter that much, if you are like me and barely play any WADs above Vanilla complevel anyway. But the modding opportunities are going thru the roof with ID24, without relying on features from ports that I personally think are cumbersome themselves, by comparison, i.e. DECORATE/ZScipt just do too much for my palette, and I assume for many others as well. I do however believe in the potential for many previously ZDoom-exclusives to be able to be ported to mainline doom modding with these features, I mean with DEHEXTRA alone, I can already imagine WADs with just custom bestiaries to not take too much effort, if just some time, to port to to DEHACKED via DECOHACK. Again, building on top of what is already established is key, in part because it's a clearer path of improvement, just adding on top instead of rebuilding something new to be a replacement, which again, I think is the truly cumbersome option given the development effort and convincing of players it would take. Maybe these features feel like too much, and that's okay, no one _has_ to immediately jump to using the hot new thing always, I still mostly play Vanilla/Limit-removing stuff much for that reason.

[bofu]

On 8/18/2024 at 1:15 PM, Ferk said:

 

Yes, this is exactly the kind of thing that I meant when I was saying "some of which didn't actually need to be booleans and could have allowed more flexibility".

 

Things should not be a boolean if making them a scalar is actually possible.

 

I remember ZDoom also had flags like "MISSILEMORE" for increasing shooting chance and then later "MISSILEEVENMORE" was added to increase it further... instead of actually offering the chance as a value. MBF21 did not learn from that and added their very own flag for that.

 

Fun fact: Crispy Doom actually has ZDoom/Decorate beat in this area, as it properly parameterized Min Missile Chance, Melee Threshold, etc. in ways that even DECORATE doesn't. It's something that I have implemented in my pseudo MBF24 build.

[Gez]

3 hours ago, bofu said:

Fun fact: Crispy Doom actually has ZDoom/Decorate beat in this area, as it properly parameterized Min Missile Chance, Melee Threshold, etc. in ways that even DECORATE doesn't. It's something that I have implemented in my pseudo MBF24 build.

https://zdoom.org/wiki/Actor_properties#MinMissileChance

https://zdoom.org/wiki/Actor_properties#MeleeThreshold

[Ferk]

8 hours ago, elf-alchemist said:

I personally don't see this as anymore cumbersome than adding any other new lump, ID24 just happens to add at least four of them

In some ways, some of the features might be not as easy for pure C and minimalistic ports to implement than it would have been if they had chosen to selectively pick existing ZDoom equivalents and adapt/extend when needed (and the "Min Missile Chance" / "Melee Threshold" are good examples of adaptations from ZDoom in a way that makes sense for those ports).

 

For example, I feel it might have been a good idea to extend the existing GAMEINFO lump for ZDoom instead of introducing a new JSON lump, which will require an entire new parser and would not be compatible with previous wads using that (eg. Sigil, to name one that was sort of "official" and had a GAMEINFO). In general, it would have been better if the same ini-like format (which is also sort of what's used in UMAPINFO) could have been used for all those lumps, instead of adding yet another one with extra dependencies. JSON is not particularly easy to parse without using external libraries, so I'm expecting that for a lot of the "simpler" ports it's a bit of an overkill format.

Edited August 27 by Ferk

[Professor Hastig]

After reading through the entire spec, here's my take about some parts of ID24:

 

* using JSON for user defined files is not the best idea IMHO. Even for experienced people, creating a valid JSON file in a text editor is hard, the format is very strict and unforgiving. Yes, it is easy to handle in the engine but creating these files can be very annoying.

 

* that being said, I do not think that adding a JSON parser is a big issue. There are one-header-only solutions for this and none of these lumps will require a high performance solution like ZDoom's savegame format did, so even the simplest available library would suffice for parsing them.

 

* Sticking to Dehacked's index based approach as the primary mode of actor additions is a huge problem. Conflicts will be inevitable when trying to combine content. By allowing negative and very high positive indices the spec essentially requires using an associative array to store the data so why not go the whole nine yards and allow symbolic indices (yes, alphanumeric names!)? By doing that it'd be a lot easier to plug a DECORATE parser into the system without much fuss, all it needs to do is create synthetic names for its items that are unambiguous. By sticking to numbers that cannot be used as array indices anymore due to the allowed value range you get the worst of both worlds and will run into major problems later when trying to expand the spec.

 

* there's a lot of ground to cover for the more conservative ports here. Many features simply do not exist there yet and need to be newly implemented.

 

What I am missing here so far is some input from the port authors. The only one who responded was kraflab and he never went into details about the technical side.

[Gez]

4 minutes ago, Professor Hastig said:

After reading through the entire spec, here's my take about some parts of ID24:

 

* using JSON for user defined files is not the best idea IMHO. Even for experienced people, creating a valid JSON file in a text editor is hard, the format is very strict and unforgiving. Yes, it is easy to handle in the engine but creating these files can be very annoying.

The specs basically assume that SLADE will get a JSON editor.

4 minutes ago, Professor Hastig said:

* Sticking to Dehacked's index based approach as the primary mode of actor additions is a huge problem. Conflicts will be inevitable when trying to combine content. By allowing negative and very high positive indices the spec essentially requires using an associative array to store the data so why not go the whole nine yards and allow symbolic indices (yes, alphanumeric names!)?

Agree with that.

4 minutes ago, Professor Hastig said:

* there's a lot of ground to cover for the more conservative ports here. Many features simply do not exist there yet and need to be newly implemented.

Besides the specs are for C++17. I think some ports are still in plain old C. If I'm not mistaken, DSDA-Doom is still largely in C, though it has C++ bits already integrated; Doom Retro and Woof are in C as well.

[Professor Hastig]

9 minutes ago, Gez said:

Besides the specs are for C++17. I think some ports are still in plain old C. If I'm not mistaken, DSDA-Doom is still largely in C, though it has C++ bits already integrated; Doom Retro and Woof are in C as well.

 

Where do the specs imply a programming language for the port? I must have missed it, but even if it did - why? C++17 doesn't have anything that couldn't be done in C, even though it'd be a lot more work there.

 

[Gez]

5 minutes ago, Professor Hastig said:

Where do the specs imply a programming language for the port? I must have missed it, but even if it did - why? C++17 doesn't have anything that couldn't be done in C, even though it'd be a lot more work there.

https://drive.google.com/drive/folders/1Q9GHoykGVJ7JONzYDXStM_vRdceL4Vi5

 

id24thinker.h:

// ID24 data tables and supporting code 0.99.1
//
// This file represents an updated thinker object with some type safety.
// It achieves this by storing the thinker function as the first value
// in the object and combines compile time checks with comparisons of
// the thinker function to pre-defined allowed functions.
//
// This is C++17 compatible. It's easy enough to strip std::enable_if
// and replace with requires if you want to upgrade it to C++20.

id24data.h:

// These are the definitions required to correctly interpret the ID24
// object tables. The structs used do have some incompatibility with
// the vanilla structs, namely each struct now has an index and a
// minimum feature level field at the front.
//
// These definitions are C++17 compatible.

 

Obviously, you're not required to use these exactly as-is, but it's still the reference implementation.

[Shepardus]

I'm not a fan of JSON either. Even if the tools exist to handwave away the cumbersome parts of the format (which is a big if, tool writers for Doom modding aren't a dime a dozen), we'd realistically still see a lot of people handwriting lumps like they do with every other lump that has tools available (UMAPINFO, DeHackEd, etc.).

 

But what I can't stand the most about JSON is the lack of comments. Comments are indispensable for explaining and organizing one's work, both for oneself and for others, and I think not being able to put comments in these new lumps would be a big pain point, especially for a community that encourages sharing and learning from others.

[Professor Hastig]

20 minutes ago, Gez said:

Obviously, you're not required to use these exactly as-is, but it's still the reference implementation.

 

That new actor code is easily convertible to C, so it is no big deal.

 

idthinker24 may be a dead end. None of the C ports will be able to make use of that added type safety - and I wouldn't expect any of them to upgrade their source to C++ -  and the ports that already are C++ have no need for that because they use proper C++ inheritance instead of hacks to achieve their goal.

 

[hobomaster22]

43 minutes ago, Professor Hastig said:

* Sticking to Dehacked's index based approach as the primary mode of actor additions is a huge problem. Conflicts will be inevitable when trying to combine content. By allowing negative and very high positive indices the spec essentially requires using an associative array to store the data so why not go the whole nine yards and allow symbolic indices (yes, alphanumeric names!)? By doing that it'd be a lot easier to plug a DECORATE parser into the system without much fuss, all it needs to do is create synthetic names for its items that are unambiguous. By sticking to numbers that cannot be used as array indices anymore due to the allowed value range you get the worst of both worlds and will run into major problems later when trying to expand the spec. 

 

What I am missing here so far is some input from the port authors. The only one who responded was kraflab and he never went into details about the technical side. 

I agree with this. I also do not like that the dehacked hashing specification for stacking dehacked patches. Forcing all the ports to run through all properties and calculate a hash seems error prone, especially since we are likely to extend more in dehacked. It's a lot of work, clumsy, and I could be wrong here but I largely don't see it being used. To be honest, I think this whole thing should be scrapped. If there is an actual need for such a feature authors can just specify a UUID to uniquely identify their patch. The tools can easily generate it, it's simple check to add in source ports, and the author can easily manually add one by searching 'random uuid' to use an online generator. I think this would be a lot easier on everyone.

I can't speak to the difficulty of the index problem for other ports. Helion already has to dissociate from them because we use decorate as a base format.
 

46 minutes ago, Professor Hastig said:

 

* using JSON for user defined files is not the best idea IMHO. Even for experienced people, creating a valid JSON file in a text editor is hard, the format is very strict and unforgiving. Yes, it is easy to handle in the engine but creating these files can be very annoying. 

This is a welcome change for me. Helion has had a lot of problems dealing with arbitrary undocumented parsing rules, especially in the dehacked world. Having a very strict unforgiving format is a good thing from my perspective.

Overall, I think the spec is pretty good and I've already implemented the map specials (sector colormaps, flat rotations, music changes, reset inventory exits) in Helion. I'm looking forward to seeing what authors do with these.
 

[Professor Hastig]

56 minutes ago, hobomaster22 said:

This is a welcome change for me. Helion has had a lot of problems dealing with arbitrary undocumented parsing rules, especially in the dehacked world. Having a very strict unforgiving format is a good thing from my perspective.
 

 

With that I'd agree, but I don't like JSON for human written files. I had to do that on occasion to test some web interfaces and it never was a pleasant experience.

Even with all the format's problems YAML might be a better choice because it was specifically made to be human written. The good thing is that YAML is a superset of JSON so all old data would continue to work.

 

[Ferk]

1 hour ago, hobomaster22 said:

This is a welcome change for me. Helion has had a lot of problems dealing with arbitrary undocumented parsing rules, especially in the dehacked world. Having a very strict unforgiving format is a good thing from my perspective.

The thing is that rather than it being a change of parser, it's an additional parser. So now there's 2 vectors instead of one.

The UMAPINFO & DEHACKED parsers still are part of the standard and if so, it makes more sense to me to clarify the undocumented behavior when found, rather than introducing an extra parser without resolving the potential problems of the previous one, which is too, part of the standard.

[bofu]

5 hours ago, Gez said:

https://zdoom.org/wiki/Actor_properties#MinMissileChance

https://zdoom.org/wiki/Actor_properties#MeleeThreshold

Huh, then why is MISSILEMORE a thing? Interesting.

[Ferk]

5 minutes ago, bofu said:

Huh, then why is MISSILEMORE a thing? Interesting.

 

both MISSILEMORE and MISSILEEVENMORE have the same effect as setting the "Missile chance Multiplier" from crispy to 0.5 or 0.125 (in FRACUNIT) respectivelly (or 0.0625 if you combine both flags).

 

That is the one new value that is more flexible there than ZDoom, because in ZDoom they use flags for setting arbitrary values on the multiplier for the perceived distance.

[hobomaster22]

10 minutes ago, Ferk said:

The thing is that rather than it being a change of parser, it's an additional parser. So now there's 2 vectors instead of one.

The UMAPINFO & DEHACKED parsers still are part of the standard and if so, it makes more sense to me to clarify the undocumented behavior when found, rather than introducing an extra parser without resolving the potential problems of the previous one, which is too, part of the standard.

I am probably biased but in Helion this is better. It's literally one line of code for me parse these JSON definitions. I do not have to worry about hand written parsers going field by field.

[Ferk]

2 minutes ago, hobomaster22 said:

I do not have to worry about hand written parsers going field by field.

Do you have DEHACKED / UMAPINFO support? If you do, then you still need to worry, right?

[hobomaster22]

Just now, Ferk said:

Did you have DEHACKED / UMAPINFO support? If you do, then you still need to worry, right?

I mean I don't have to worry about it for these new JSON lumps.

[elf-alchemist]

1 hour ago, Professor Hastig said:

With that I'd agree, but I don't like JSON for human written files. I had to do that on occasion to test some web interfaces and it never was a pleasant experience.

Good thing that no one writes JSON by hand, maybe a package.json if you are a JavaScript/TypeScript developer, but other than that, almost every line of JSON ever written is meant to be machine written, in this case, by a supporting editor. I've been messing around with JSON enough to be more accustomed to manually editing it, even on SLADE3 I can still get a hold of GAMECONF, TRAKINFO and DEMOLOOP (although the real trouble came in with the SBARDEF, with it's 1956 lines for the default definition in extras.wad)
 

1 hour ago, Professor Hastig said:

Even with all the format's problems YAML might be a better choice because it was specifically made to be human written. The good thing is that YAML is a superset of JSON so all old data would continue to work.

See above, the human readability of a given data format becomes irrelevant if the entire data is designed to be editor-oriented. Though I've never had the opportunity to truly work with YAML, myself, so I can't really comment on what parts of it are better or comparable to JSON's (other than human readability), but I've not heard many pleasant anecdotal recounts of those who write with it, I've seen at least one person say to instead use Python to write YAML structure to the text file, which kinda seems to defeat the whole point of it being human readable, since doom editing writeability will suffer either way from it, just means port developers will need to worry about parsing YAML instead. And if these lumps set any kind of precedent for the developers in the community, it'll likely be that JSON will become a common, standard medium for new lumps to come. I don't like JSON that much either anyway, I just tolerate it enough to not lose sleep over it (god-forsaken trailing commas).

Edited August 27 by elf-alchemist : words

[LuciferSam86]

2 hours ago, Professor Hastig said:

With that I'd agree, but I don't like JSON for human written files

I mean JSON is for data exchange between systems, never meant to be written by hand.

Solving a lot of problems for serialization / de-serialization. 

 

XML was even worse , still used by banks :) 

 

I'm sure there will programs that will give a help to write such files. 

 

 

[Trov]

4 hours ago, Shepardus said:

But what I can't stand the most about JSON is the lack of comments. Comments are indispensable for explaining and organizing one's work, both for oneself and for others, and I think not being able to put comments in these new lumps would be a big pain point, especially for a community that encourages sharing and learning from others.

It's not quite a necessity that JSON does not have comments - the developer of the JSON spec has further clarified that comments in JSON files can be OK as long as there is code that strips them out before it reaches the parser. Additionally several JSON decoding libraries also ignore comment lines.

 

In fact, Nightdive Doom does so - comment lines in JSON files such as GAMECONF appear to have no adverse effect.

 

Someone could whine that "Well it just isn't proper spec JSON then!" but the reality is in the context of Doom mods we really don't care whether other random programs can parse its JSON lumps as-is (not that they would be able to parse a WAD directly anyway). If Nightdive doesn't agree then they should make the JSON parser in their port not accept //comment lines, and further specify in the spec that dedicated comment data fields such as "_comment" or "#" or "//" etc will be allowed for placing comments in true-blue JSON hardline fashion.

Edited August 27 by Trov

[LexiMax]

I actually disagree with the assertion that JSON is not a human-writable format.  Configuration files are commonly in JSON format, and most JSON parsers support comments and extra commas as extensions.

 

It might not be the absolute best for hand-writing, but it's one of those formats that's so common and widely used that it seems kind of silly to pick anything else or cook up your own format.  The other contenders that have any significant developer mindshare are YAML, which is way too complex and has too much wiggle room for unexpected parses, and TOML, which isn't INI compatible and only easier to write if you don't need features like inline tables.

 

Also, you can't compete with the tooling ecosystem surrounding JSON.  It's ubiquitous.

[Shepardus]

7 hours ago, Trov said:

In fact, Nightdive Doom does so - comment lines in JSON files such as GAMECONF appear to have no adverse effect.

Oh interesting. The JSON lump specification may need some revision then, because it says (emphasis my own):

Quote

JSON lumps are expected to support the ISO/IEC 21778:2017 standard for JSON data. It is expected to be strictly adhered to. Any fields that fall outside of the scope of that standard (including comments) are to be considered an error and result in an invalid JSON document.

 

[danidf96]

I just stumbled upon this thread, what the fuck is even going on anymore.

[Professor Hastig]

22 hours ago, LuciferSam86 said:

I mean JSON is for data exchange between systems, never meant to be written by hand.

Solving a lot of problems for serialization / de-serialization. 

 

XML was even worse , still used by banks :) 

 

Banks use all kinds of weird shit. Correction: Make that finance related software in general. Formats are shit, XML runs rampant and most implementations do not even implement it correctly.

 

 

22 hours ago, LuciferSam86 said:

I'm sure there will programs that will give a help to write such files. 

 

 

 

Sure you can implement editors for these formats, but that's a lot of work and to make it comfortable you got to have a separate one for each different of these lumps. A generic JSON editor won't do much good because it still needs to let the user handle all the structure - and the result is often less comfortable than a simple text editor.

 

This is why I prefer YAML here, despite all the weird pitfalls of the format. It can be edited in a text editor without too much trouble.

 

[dpJudas]

If I have to choose between YAML and JSON I prefer editing JSON. None of them are good formats for a human IMHO. Super easy to use from the engine side though, which I have no doubt is why JSON was chosen.

 

That said, the current text config files in GZDoom are by no means any better. Yes the human can edit them, but they were all designed so poorly that extending the syntax without breaking older parsers is basically impossible for almost all of them. INI files are also true horrors.

[Ferk]

3 hours ago, dpJudas said:

That said, the current text config files in GZDoom are by no means any better. Yes the human can edit them, but they were all designed so poorly that extending the syntax without breaking older parsers is basically impossible for almost all of them. INI files are also true horrors.

 

What does "breaking older parsers" mean here?

Will the whole port crash? will it skip the lump entirely? or will it just skip the lines it does not understand while outputting warnings?

 

I mean, afaik, dehacked is meant to follow the third approach, which is why it's relatively easy to add extra fields for extra functionality (or even new fields with different parsing rules) without completely throwing older parsers under the bus. Even this new standard adds dehacked extensions.

In a way, that's comparable to ignoring a field in a JSON lump when it does not match any of the expected fields from the spec, or has a different type than specified. Using JSON does not automatically free you from unexpected input.

 

I feel that full compatibility with old ports should not be a goal here. Nobody expects to be able to run flawlessly Boom WADs in a non-Boom port. Of course it's best to handle unexpected input gracefully, which is why I expect most of these ini-like formats should always ignore what they don't understand, at a line level (and I think that's why generally discouraging nesting in those formats is a good idea too).

 

Something that makes more sense to me is the opposite: supporting older WADs in a new port. And that's why I feel supporting GAMEINFO would have made more sense than introducing a new lump. Because now we have some overlap in different lumps. Specially after Sigil gave some level of "officiality" to that lump, and I'm sure it's not the only one amongst the Official Add-ons (I just checked and even Legacy of the Rust itself has a GAMEINFO lump, despite it not being part of ID24).

Edited August 28 by Ferk

[Trov]

I think GAMEINFO was skipped because it doesn't contain enough useful information (such as desired complevel) to be a standin for manually selecting launcher parameters. It primarily exists to define how the startup window looks in ZDoom.

Sure it could be added to GAMEINFO, but that doesnt solve old WADs not having it...

I think it's generally a good idea to force WADs to specify it.

[Ferk]

47 minutes ago, Trov said:

Sure it could be added to GAMEINFO, but that doesnt solve old WADs not having it...

True, but you can't solve that either with a new lump, whereas extending the old one does ensure the support for some of the older fields.

It's nice to be able to have the port automatically know which iwad to select for which pwad, or which extra resource wads to load.

Now ports have to support both lumps if they want to ensure they cover both old wads and new wads.

And it's likely wads will start including both lumps as well, for old ports. So it's a lot of duplication just for the sake of JSON.

 

They could have also added an alternative format to dehacked, but that would have meant ports need to either support both parsers (with wads also likely duplicating the lumps anyway if the want wider support), or drop support for old wads using dehacked (which doesn't seem reasonable at this point).

Edited August 28 by Ferk

[Gez]

48 minutes ago, Trov said:

It primarily exists to define how the startup window looks in ZDoom.

That role is now largely moot.

 

The real purpose is for the IWAD and LOAD properties; everything else is just bonus fluff.