A very good question.
First, I would think about the story about these games, whether for Special Olympics or the Olympics. The Olympics had their roots in religious festivals, the Special Olympics have another meaning - maybe in your world they also are to allow citizens to compete in contests of skill or strength against other citizens (non-adventurers)
I would look at what events, what kind of skill challenges to do those competitions, and keep 'score' for the top three finishers. At the end, the winners get some loot - maybe in keeping with Special Olympics, everyone else gets prizes.
For me, I would also think of what kind of activities are going on around the town. It's sure to bring in travelers and supporters, as well as con artists and thieves. There are all kinds of beautifully hedonistic tales that come from the Olympic Villages in our world - why not in the game too?
To me, I would just do what's fun and in keeping with the traditions. What I find great about gaming is it's a good way to look into the histories of things you would never research on your own. Good luck!