Let's give it a try...
1) Ignore all economic rules of supply and demand for the time being, so that we try to extrapolate a baseline price from the RAW only. We try to defer adding economic variables after setting a baseline price, if we can, or at least as late as possible because they introduce a huge variance and randomness.
2) The price for copying spells would be relevant if the spellbook was sold by the creator, because she had to pay that cost. Unfortunately, to a prospective buyer this cost is irrelevant because the buyer CANNOT prepare spells from a spellbook that isn't her own: she still has to pay the full cost of copying all the spells into her own spellbook before she can prepare/cast them.
This is the first problem that makes it hard to avoid economical considerations of supply and demand: with a seller who also
created the book we may want (as a starting reference point only) to set the price to the cost it took to create it, but with a seller that has not paid anything for it, we might better look directly at what value the buyer sees in it, for example by comparing it with how much does it cost to buy
scrolls with exactly the same spells, since you can also copy spells from scrolls.
Let's try to follow this approach of "buyer-based price" to continue.
3) We don't have
fixed scroll prices in 5e, but only ranges. Maybe the most reasonable thing is to use the simple averages, from 37.5 gp for a 1st level spell to 13750 gp for an 8th level spell. The range price for a 9th level spell is unlimited, let's be generous and pick the lower bound i.e. 25000gp. Calculate the total for the list of spells in your specific spellbook and you have a
baseline price. This of course requires that you have specified WHICH spells are in the spellbook.
Reality check: this baseline is A LOT more than the price you get if you use the copying costs instead! It's impossible at this point to avoid economical considerations: if the buyer has the possibility to find a Wizard who would scribe a new spellbook and sell it to her, it would be a lot cheaper than to acquire scrolls for all the wanted spells, and our baseline price would be off market.
4) Eventually, the buyer will see value only in spells she doesn't already know. She has no use for the rest. So when calculating the previous baseline price, you might want to take a hard approach and only count the spells the buyer doesn't already know (I would also include spells of levels she can't cast yet but can learn later), but this requires the DM to specify WHO exactly is the buyer! You probably just want to figure out how many gp to give to your PCs... should we default to HALVING the baseline price for simplicity?
5) Halve the baseline price again, because the PCs are the sellers rather than the buyers.
6) Notice that scrolls costs a lot because they have a very important second use beyond being copyble: you can cast the spell directly from it, beyond your own daily spellcasting capabilities. So we should cut the price further because the spellbook does not offer this function, but how much? Half the price again? This is not obvious.
7) Optionally: ballbark some generic modifier for market conditions. For high demand + high supply I wouldn't apply any modifier, not a big one at least. Maybe I would round down the final price to multiples of 100gp (if it costs less than 2000gp) or multiples of 1000gp for higher prices.
---
If we stick to this plan, we basically have to at least decide what kind of spells the spellbook contains (i.e. their LEVELS), calculate the baseline price, and divide by 8 i.e. divide 3 times by 2. With some approximations (rounding to multiples of 5gp), we could say that this could be the new price per spell:
- 5gp for each 1st level spell
- 20gp for each 2nd/3rd level spell
- 170gp for each 4th/5th level spell
- 1720gp for each 6th/7th/8th level spell
- 3125gp for each 9th level spell
Since we don't care which spells exactly but only their levels, we could build a map between Wizard level -> learned spells -> spellbook price. We assume a Wizard always learns new spells of the maximum possible level when levelling up (very reasonable), but also that no additional spells were added to the spellbook from scrolls and other books (which is likely not to be the case, but it's pretty impossible to guess how many they would be since it's VERY campaign-dependents).
Code:
Wizard Spells Spellbook
Level Known SELLING Price
1 6 30
2 8 40
3 8/2 80
4 8/4 120
5 8/4/2 160
6 8/4/4 200
7 8/4/4/2 540
8 8/4/4/4 880
9 8/4/4/4/2 1220
10 8/4/4/4/4 1560
11 8/4/4/4/4/2 5000
12 8/4/4/4/4/4 8440
13 8/4/4/4/4/4/2 11880
14 8/4/4/4/4/4/4 15320
15 8/4/4/4/4/4/4/2 18760
16 8/4/4/4/4/4/4/4 21200
17 8/4/4/4/4/4/4/4/2 27450
18 8/4/4/4/4/4/4/4/4 33700
19 8/4/4/4/4/4/4/4/6 39950
20 8/4/4/4/4/4/4/4/8 46200
With this table, you can figure out when you defeat a Wizard of a certain level, how much you can sell his spellbook for. If you use NPC wizards from the MM instead of classed Wizards, I would assume the NPC wizard is equivalent to a classed Wizard of the minimum level required to cast its highest-level spells (for example, Archmage = Wizard 17, because it can cast 9th-level spells). NPC spellcasters in the MM only mention their
prepared spells, not what they have in their spellbooks.
Keep in mind these are
selling prices, which means 50% of possible
buying prices.