That seems reasonable, and each foot moved will cost three, so you could inflict on you both 4d4 damage. The reason it costs three is that dragging costs +1'/1' while Spike Growth is making the terrain "difficult". That's assuming the character is strong enough (or the grappled light enough) that they can make way at all.
A grappler can "drag" you, which to me implies that they move ahead of you and pull you after them. Assuming you both start outside the area, it seems they are in for 4d4 and you 2d4 on the first turn. In that same turn, a second character can take 4d4 damage to set themselves up to double-drag you in subsequent turns. In turn two, both grapplers will take 4d4 and you will take 8d4. Assuming all goes to plan.
Per the above, grappler A has by end of turn 2 taken 8d4, grappler B 8d4, and you, the grappled, 10d4. That could be improved by casting the spell with the intended grapplers already in their ideal positions within it, which would halve the damage they take. Still, it's 8d4 to party at the cost of a level 2 spell slot, a character concentrating, and two characters' movement and actions, for 10d4 to the grappled.
At a guess, the ploy will be more powerful in TotM where it's not so obvious how much damage grapplers must accept in order to
drag their victim through the area. In conclusion, following RAW seems to entail
Account for encumbrance (weight of dragged)
Account for movement costs (3'/1')
Give meaning to "drag"
Assume draggers must move through the area themselves (more important to TotM, as a grid will plainly show it)