D&D General Requesting permission to have something cool

[Oofta]

Yes...that's the point. It will regress to the mean. You will get a central tendency and a standard deviation--statistics. Those statistics can then tell you whether things are performing as desired.


No, you can't. That's why we actually collect statistical data, and do things like Monte Carlo simulations, rather than doing everything analytically. Some math questions cannot be easily answered


...except that it does matter. Because those are the exact questions the designers need to be asking. They need to know the input variables. They need to know the situations. That's how you test things! You're literally saying that because we can't get an analytic answer, no answer is possible. That's wrong! We can get numeric solutions, sometimes very very good ones. That's the whole point of modeling like this. Huge swathes of science today are, quite literally, built upon the back of creating very good computer simulations and then testing novel or unexpected variables to see what happens. That's how we do climate science, since we can't actually solve the differential equations involved and can't do meaningful experiments because we don't have a thousand other Earths to perform experiments on. That's how physicists test models of solar system formation, or the mechanics of how Earth's Moon formed, or literally anything at all involving gravity because the three-body problem does not have general solutions.

Assumptions will go into it. By definition, they must, because some went into the design of the game to begin with. As stated, this requires that you think very carefully about what questions you ask, how you ask them, what data you use to answer them, and whether the data actually supports any conclusions at all (let alone the ones you're looking for.) That's how statistical modeling works.

Just because it's statistical and simulated doesn't mean it's useless. It is exactly the opposite: that it is statistical means we can apply many useful things to it, which can help us seek useful results. Statistics and simulation are powerful tools; like any powerful tool, they must be used with care and diligence.

We're just going to disagree. I don't see the point of continuing on.

 

[Pedantic]

People say all sorts of things. Repeatedly. Doesn't make them right. If you're running the same scenario repeatedly it will average out. If it averages out you don't need a simulator, you can do the same with math and averages.

But it still doesn't matter because it just depends on what scenarios you're validating and what the input variables are.

You're attacking the concept of modeling? Like, the idea that we might run a scenario a bunch of times with different configurations of variables to observe the distribution of outcomes?

That's an integral part of many scientific disciplines and design across a bunch of fields. It's, arguably, the primary benefit computing has provided to research. What, precisely, is objectionable about taking a bunch of different characters, giving them a range of decisions, and then running an encounter repeatedly? I certainly could do it by hand, but I'm cool handing that off to a machine, and I'd love to see the curves it outputs at the end.

Or are you suggesting that RPGs are so simple that one should be able to work out encounter likelihoods via pure thought experiment? I'd love to hear how you quantify the impact of the frightened condition in your mental model.

 

[Lanefan]

What else can it be?

The play style advocated for isn't about creating a character with a backstory, goals or motivation. It's about rolling 3d6 and hoping you get lucky that THIS PC sticks. They might get a name if they make it to 3rd level, and might even develop a personality at 5th.

They get a name* and personality before they even hit the table, you can be sure of that! Well, a name anyway; if for no other reason than the DM needs it for recordkeeping purposes.

(with two exceptions that I can recall from our many games, both being characters that died before ever meeting anyone else in the party)

I don't like the OSR/DDC style of character generation by attrition. Never have, never will. And I'm glad the game has moved away from it, circa Dragonlance. But if you like it, play it. I don't find it superior in any way, shape or form and you cannot convince me otherwise.

Fair enough.

 

[Oofta]

You're attacking the concept of modeling? Like, the idea that we might run a scenario a bunch of times with different configurations of variables to observe the distribution of outcomes?

That's an integral part of many scientific disciplines and design across a bunch of fields. It's, arguably, the primary benefit computing has provided to research. What, precisely, is objectionable about taking a bunch of different characters, giving them a range of decisions, and then running an encounter repeatedly? I certainly could do it by hand, but I'm cool handing that off to a machine, and I'd love to see the curves it outputs at the end.

Or are you suggesting that RPGs are so simple that one should be able to work out encounter likelihoods via pure thought experiment? I'd love to hear how you quantify the impact of the frightened condition in your mental model.

No, I'm saying that a simulator akin to BG3 is not needed. It's likely nothing I would do in a spreadsheet but I wrote a simple cage match program for fighters a while back . But yes, I think if someone were good enough at excel and the math involved they could do it in excel.

But what you're measuring and what your input variables are will have such a significant impact on outcome I'm not sure what the goal is.

 

[nevin]

You're attacking the concept of modeling? Like, the idea that we might run a scenario a bunch of times with different configurations of variables to observe the distribution of outcomes?

That's an integral part of many scientific disciplines and design across a bunch of fields. It's, arguably, the primary benefit computing has provided to research. What, precisely, is objectionable about taking a bunch of different characters, giving them a range of decisions, and then running an encounter repeatedly? I certainly could do it by hand, but I'm cool handing that off to a machine, and I'd love to see the curves it outputs at the end.

Or are you suggesting that RPGs are so simple that one should be able to work out encounter likelihoods via pure thought experiment? I'd love to hear how you quantify the impact of the frightened condition in your mental model.

It's also caused the collapse after publication of quite a bit of research because that's where it's easy to tweak the outcome in your desired direction. Doesn't matter if it's all computer simulation or all people labor, all the data that goes in is what a person or group of person's decided and therefore it's only as good as that group is at being able to step back from their jobs and be logical and accurate instead of attempting to prove what they already believe.

computer modeling great tool. The people that use it often have their minds made up before they start modeling. The awesome tool doesn't solve the human issues, or even mitigate them. In fact it makes it easier to hide that kind of thing in a mountain of data. Remember, Lies, Damned Lies and Statistics?

 

[Amrûnril]

...you measure the things that can actually be measured, based on whatever design goals you have. E.g., as mentioned, when checking CRs, you check to see if various party comps (run a zillion times each) get the right spread of "number of deaths per combat" or "number of spell slots spent" or whatever. When testing different subclasses, you'd look primarily at things like damage output, damage suffered, healing provided, stuff like that--things the computer can quite easily track.

Plenty of things, as I said, cannot be tested this way, and those things will always require human judgment to test. Nobody's getting rid of human playtesters. But instead of waiting for ten human groups to run through an encounter you've set up, you can get at least approximate data from the virtual simulation--and then those ten groups can instead focus on the critical intangibles like "feel", on things like presentation and roleplay and utility effects etc., etc.

Dismissing the entire thing as "oh it's just garbage in, so it's garbage out" is foolish. There is enormous potential in things like the BG3 system to give designers real data. Yes, like all statistics, it means you must ask good questions and really carefully think about the answers. That's nothing new.

Testing in a simulator could be valuable as one of many playtesting tools, but it would need to be in the hands of a team with a clear understanding of its capabilities and limitations. Even those parameters that are measurable would depend heavily on factors like encounter design and how well or poorly the AI is able to use different ability types, and an automated system would make it very easy to generate authoritative sounding numbers and confidence intervals without thinking critically about the assumptions those values are contingent on. The way the developers describe their interpretation of survey data doesn't give me a lot of confidence in their ability to avoid this sort of pitfall.

 

[Lanefan]

Yes...that's the point. It will regress to the mean. You will get a central tendency and a standard deviation--statistics. Those statistics can then tell you whether things are performing as desired.

Repeated computer sim would be great for fine-tuning the math, no question there.

There's still important questions, though:

1. Is getting the math that finely-tuned a good thing or a bad thing?
2. Whose definition of "performing as desired" will be used to determine functionality?

 

[Pedantic]

No, I'm saying that a simulator akin to BG3 is not needed. It's likely nothing I would do in a spreadsheet but I wrote a simple cage match program for fighters a while back . But yes, I think if someone were good enough at excel and the math involved they could do it in excel.

But what you're measuring and what your input variables are will have such a significant impact on outcome I'm not sure what the goal is.

To measure all of the things! A simple cage match program isn't good enough, as you rightfully point out, so let's expand the model to cover terrain, and likely enemy attack patterns, and action loss due to movement speed differentials, and flight, and that one player who can't grasp that the higher number next to his sword means he shouldn't use a ranged attack unless he has to, and that other player who somehow, still hasn't gotten the focus fire memo.

And then, when the playtest group using the game built with that data keeps using invisibility to overrun those encounters, let's come up with a new set of variables to code for that.

What exactly do you want designers to be doing, if not all that?

 

[Oofta]

To measure all of the things! A simple cage match program isn't good enough, as you rightfully point out, so let's expand the model to cover terrain, and likely enemy attack patterns, and action loss due to movement speed differentials, and flight, and that one player who can't grasp that the higher number next to his sword means he shouldn't use a ranged attack unless he has to, and that other player who somehow, still hasn't gotten the focus fire memo.

And then, when the playtest group using the game built with that data keeps using invisibility to overrun those encounters, let's come up with a new set of variables to code for that.

What exactly do you want designers to be doing, if not all that?

Sure, my simple cage match isn't enough but I could fairly easily expand it. I already track resource usage for battle master maneuvers for example so tracking spells used would be simple enough. You can randomize some things, I randomized how quickly the fighter used their second wind. I have no desire to do so but I could add multiple entities, initiative order, etc.

But it's still all going to depend on input variables, assumptions and what is being measured. What spells does the wizard have? How many targets are going to be in an AOE? Does the party focus fire or spread out damage? Do the monsters? How often is cover going to come into play, how often will the rogue be able to use stealth? Do we throw these groups into an AI that determines optimal tactics or do we just base it on the experience of the testers? Do we modify behavior at all to mimic how different groups are likely to play?

There are simply too many variables and I didn't even get to magic items, rolling for stats, does the DM use morale. The list goes on.

 

[Micah Sweet]

Yes...that's why I'm saying the survey design is bad. There are ways to ask the questions they want to ask that don't do that. This is a topic that has been studied in social science and psychology for, I am not joking, more than a full human lifetime. There is ample literature on the subject and a consultant would trivially identify many of the errors WotC is making in their survey design.


I never said there was. I said that the things I am proposing are simple steps, in and of themselves actually quite easy. The overall task remains quite difficult. I would never say otherwise--it is, after all, something many people are doing as their career.

But when they do things as boneheaded as literally making polls where the only answers are versions of "yes," their survey design is SO busted that, yes, it really is extremely easy to do better! As in, almost anyone could do better.

But do you have any reason to think they're likely to?