setting up a local dndtools mirror

[Stronginthearm]

Disclaimer: I own literally every D&D 3.0/3.5 source book (some 5 foot of shelf space), so I don't intend to disrespect anybody's IP rights.
Also I hope this is the right forum, since the topic is somehow D&D, but mostly technology.

So yeah, dndtools.eu is down, which sucks because it was an awesome site if you didn't wanted to dig through all of your books to find that one feat.
Fortunately the sourcecode of the site is available (linked on the site) and one can find at least two versions of the site database at dropbox (mysql from 2013 and sqlite from 2014) and since I like both D&D and computers I tried to set up a local mirror for myself.
Unfortunately I know primarily linux and secondarily php but not python (or django), so it kind of works, but not quite *sigh*.


So my question is, has anybody managed to properly install it?

If somebody want's to try his/her luck, this is what I've done so far (basically the installation instructions from github):

- Debian Wheezy, python and some addons installed via apt
- python
- python-imaging
- python-textile
- python-mysqldb
- python-sqlite
- python-recaptcha
- django and addons installed via pip:
- Django==1.2.7
- South==1.0.1
- django-debug-toolbar==0.9.4
- django-pagination==1.0.7
- recaptcha-client==1.0.6
- django-reversion 1.3.3 installed via git-checkout and setup.py install (since the newest version in pip was 1.5.1)
- imported the 2013 mysql-db: mysql -u root -p -h localhost dndtools < dndtools_prod_backup_2013_03_02_clear_accounts_no_reversions.sql
- git clone the dndtools
- edited the db settings in local.py
- finally python manage.py runserver

...and now I have on port 8000 something that looks like dndtools (except for exceptions when I click on magic or bestiary) ... and that's about as far as I get.

I still have the following problems:
- I guess the remaining exceptions result from wrong versions of a component, I'll iterate about the possible combinations a bit, but if anybody knows which versions exactly to use, I'd be happy to hear
- how do I embed a django project in my normal apache? manage.py runserver on port 8000 can not be the real solution
- the github install file mentions a reset of the admin account, but when I try "users = User.objects.all()" on the django shell I get "NameError: name 'User' is not defined"
- also I don't see where I could log in on that site
- I don't know how complete the 2013 db is, so I'd like to try the 2014 db, but it's sqlite. I've managed to dump the db to sql and ran it through a converter python script, so it should now be mysql compatible, but it isn't, mysql doesn't like the CONSTRAINT or REFERENCE syntax of the tables, but I don't see why
- alternatively I tried to reference the original sqlite file in the local.py, but that breaks the site

Any input is greatly appreciated.

 

[sparkle]

I'm also trying to get this set up, although I haven't gotten quite as far as you've managed... I haven't been able to get anything to actually show up in my browser. I suspect my settings in local.py might be off, but I'm not sure. Would you mind sharing yours?

Regarding your first question, I'm assuming that you've read through https://docs.djangoproject.com/en/1.4/howto/deployment/wsgi/modwsgi/? Part of the problem with this particular project is that it uses a really ancient version of Django, so I suspect it may not be as straightforward.

 

[Isiloki]

Hey there, I've tryed too.
All I want is to setup a Dndtools as it was and share it with friends. As long as the search/filter menu works, I don't care if admin is not working.
I have done the same thing. I took the 2013 script and integrated it in my MySql db (on windows 7), then I took the SQLite and managed to make it into a MySql script as well.
By removing all the constraint and putting "Replace into" instead of "Insert into", I had no problem and my DB is now uptodate.
But I just can't get the server to work, last time I used linux was 5 years ago and it was for my studies...
Would you mind detailing exactly the command line you've run for the not obvious steps? (Particularly what you've done in the config file as I've modified nothing in these)
Thanks

(Oh and btw, there is a siteRIP working very well, ignore the extraction errors. Only problem is the filter not working :https://anonfiles.com/file/94e7c01af26b3e1424f2988617990cca )

Btw, the component versions are specified on the github (not the master branch)
<code>pip freeze
Django==1.5.5
MySQL-python==1.2.4
Pillow==2.2.2
South==0.8.4
argparse==1.2.1
django-debug-toolbar==0.11.0
django-filter==0.5.3
django-pagination==1.0.7
django-reversion==1.8.0
recaptcha-client==1.0.6
sqlparse==0.1.10
textile==2.1.5
wsgiref==0.1.2</code>

 

[Stronginthearm]

That was actually quite helpful guys

sparkle: no, i hadn't found that. I haven't implemented it yet, but apache modules are familiar enough, so I think I'll manage to get that running in the next days.

Isiloki: yeah, I have that siterip, but it's some 500MB and lacks the filter, while the dynamic version is ~60MB. With the integration of the sqlite db i got rid of the last exceptions and as far as I can see I now have a fully functional clone running on the development server on port 8000. I even found the admin page, but haven't managed yet to set a valid password.

Here is what I did in more detail (I still assume that you can do basic stuff in linux - and be root):

installing the software
- installed Debian Wheezy on a VM (every other linux will probably work, but i find Debian convenient)
- cd /where/ever/you/want/it
- apt-get install python python-imaging python-mysqldb python-recaptcha python-textile mysql-server sqlite3
- apt-get install python-pip (I used the easy_install method from the github docu, but I don't think that's neccessary)
- pip install Django==1.2.7
- pip install South==0.8.4
- pip install django-debug-toolbar==0.9.4
- pip install django-filter==0.5.3
- pip install django-pagination==1.0.7
- pip install recaptcha-client==1.0.6
since the oldest available django-reversion version in pip is 1.5.1 I installed it manually
- apt-get install git
- git clone https://github.com/etianen/django-reversion.git
- cd django-reversion
- git checkout release-1.3.3
- python setup.py install
- cd ..
- git clone https://github.com/dndtools/dndtools.git

prepare the database
- aquire both the 2013 version (from line 33 of the github INSTALL document) and the 2014 sqlite database (archive.org wayback machine lets you find the linked dropbox file)
- download the converter script http://www.redmine.org/attachments/download/6239/sqlite3-to-mysql.py
- sqlite3 dnd.sqlite .dump >dnd.dump.sql
- cat dnd.dump.sql | python sqlite3-to-mysql.py > dnd.conv.sql
edit dnd.conv.sql (Notepad++ on Windows works reasonably well, as does of course vi on Linux, nano is alittle clumsy):
- insert as 2nd line: SET FOREIGN_KEY_CHECKS=0;
- insert as last line: SET FOREIGN_KEY_CHECKS=1;
- from the CREATE TABLE statements remove everything from CONSTRAINT to ); (75 occurences)
- in line 174550 replace
CREATE TABLE dnd_rules_conditions (id , name , description );
with

CREATE TABLE `dnd_rules_conditions` (
  `id` int(11) NOT NULL ,
  `name` varchar(32) NOT NULL,
  `description` longtext NOT NULL,
  PRIMARY KEY (`id`)
);

- in lines 241799 and following shorten every index name to 64 characters or less (as far as I have researched the index name is not relevant for the db query)
import both databases:
- mysql -u root -p dndtools < dndtools_prod_backup_2013_03_02_clear_accounts_no_reversions.sql
- mysql -u root -p dndtools < dnd.conv.sql

edit the config
- cd dndtools/dndtools
- cp local.py.sample local.py
- nano local.py

DEBUG = True (so you get maybe helpful exceptions instead of a blank 500 http error)
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'dndtools',
        'USER': 'root',
        'PASSWORD': 'yourdbpassword',
        'HOST': '',
        'PORT': '',
        'OPTIONS': {
            'init_command': 'SET storage_engine=INNODB',
            },
        }
}
SECRET_KEY = 'somethingunique' (I don't know if this is neccessary)

start the debug server
- python manage.py runserver

Now dndtools runs on 127.0.0.1 port 8000 ... and if you haven't installed an x-server and a browser you have to forward the port ... fortunately putty does that quite nice:
- load your stored connection and go to Connection -> SSH -> Tunnels
- add new forwarded port
-- source port: 8000
-- destination: 127.0.0.1:8000
- save and connect
- open http://127.0.0.1:8000 in your browser

The admin interface runs on http://127.0.0.1:8000/admin, the password is probably stored in the table auth_user, but how exactly I haven't figured out yet.

Please let me know if this works for you. I plan to install it on my real server in the next few days.

appendix: all the packages (in case I forgot a relevant)

# dpkg-query -l | grep python
ii  python                             2.7.3-4+deb7u1                all          interactive high-level object-oriented language (default version)
ii  python-apt                         0.8.8.2                       i386         Python interface to libapt-pkg
ii  python-apt-common                  0.8.8.2                       all          Python interface to libapt-pkg (locales)
ii  python-chardet                     2.0.1-2                       all          universal character encoding detector
ii  python-crypto                      2.6-4+deb7u3                  i386         cryptographic algorithms and protocols for Python
ii  python-debian                      0.1.21                        all          Python modules to work with Debian-related data formats
ii  python-debianbts                   1.11                          all          Python interface to Debian's Bug Tracking System
rc  python-django                      1.4.5-1+deb7u8                all          High-level Python web development framework
ii  python-fpconst                     0.7.2-5                       all          Utilities for handling IEEE 754 floating point special values
ii  python-imaging                     1.1.7-4+deb7u1                i386         Python Imaging Library
ii  python-minimal                     2.7.3-4+deb7u1                all          minimal subset of the Python language (default version)
ii  python-mysqldb                     1.2.3-2                       i386         Python interface to MySQL
ii  python-pkg-resources               0.6.24-1                      all          Package Discovery and Resource Access using pkg_resources
ii  python-pygments                    1.5+dfsg-1                    all          syntax highlighting package written in Python
ii  python-recaptcha                   1.0.6-1                       all          client library for reCAPTCHA and Mailhide
ii  python-reportbug                   6.4.4+deb7u1                  all          Python modules for interacting with bug tracking systems
ii  python-soappy                      0.12.0-4                      all          SOAP Support for Python
ii  python-sqlite                      1.0.1-9                       i386         Python interface to SQLite 2
ii  python-support                     1.0.15                        all          automated rebuilding support for Python modules
ii  python-textile                     1:2.1.5-1                     all          Python parser for the Textile markup
ii  python2.6                          2.6.8-1.1                     i386         Interactive high-level object-oriented language (version 2.6)
ii  python2.6-minimal                  2.6.8-1.1                     i386         Minimal subset of the Python language (version 2.6)
ii  python2.7                          2.7.3-6+deb7u2                i386         Interactive high-level object-oriented language (version 2.7)
ii  python2.7-minimal                  2.7.3-6+deb7u2                i386         Minimal subset of the Python language (version 2.7)

# pip freeze
Django==1.2.7
MySQL-python==1.2.3
PIL==1.1.7
Pygments==1.5
SOAPpy==0.12.0
South==0.8.4
argparse==1.2.1
chardet==2.0.1
django-debug-toolbar==0.9.4
django-filter==0.5.3
django-pagination==1.0.7
django-reversion==1.3.3
fpconst==0.7.2
pycrypto==2.6
pysqlite==1.0.1
python-apt==0.8.8.2
python-debian==0.1.21
python-debianbts==1.11
recaptcha-client==1.0.6
reportbug==6.4.4
sqlparse==0.1.13
textile==2.1.5
wsgiref==0.1.2

 

[Isiloki]

Happy to see you progressed!
On my side, I've managed to get a full mirror of the website working.
Can help you with the steps I did (the versions are really important, there are some breaking changes in the newer versions)
Assuming you have python 2.6 installed

- wget [URL]https://bootstrap.pypa.io/ez_setup.py[/URL] -O - | python

- easy_install pip

- pip install Django==1.2.7

- easy_install South

- easy_install django-pagination

- easy_install django-debug-toolbar


- apt-get install python-dev

- pip install textile

- easy_install recaptcha-client

- apt-get install python-imaging

- apt-get install git

- apt-get install mysql-server mysql-client (avec mdp)

- apt-get install python-mysqldb

- git clone [URL]https://github.com/django-debug-toolbar/django-debug-toolbar.git[/URL]

- cd django-debug-toolbar
- git checkout tags/0.8.4

- python setup.py build / install

- git clone [URL]https://github.com/etianen/django-reversion.git[/URL]

- cd django-reversion

- git checkout tags/release-1.3.3

- python setup.py build / install

- cd ..

- mysql -u root -p

> create database dndtools;

> exit;

- wget bit.ly/XYFJdV

- apt-get install p7zip

- 7za e XYFJdV

- mysql -u root -p -h localhost dndtools < dndtools_prod_backup_2013_03_02_clear_accounts_no_reversions.sql (or whatever you script SQL is)

- git clone [URL]https://github.com/dndtools/dndtools.git[/URL]

- cd dndtools/dndtools

cp local.py.sample local.py

- nano local.py (modify these 3 lines):

> Name : dndtools

> User : root

> Password

- python manage.py runserver (and use apt-get to install the missing component)

once the server is running, stop it and do
# python manage.py shell
And that where you are missing the ultimate command line
>>> [COLOR=#ff0000]from django.contrib.auth.models import User[/COLOR]
>>> users = User.objects.all()

>>> user = users[0]

>>> user.set_password('dndtools')

>>> user.save()

>>> ^Z

And now it works.

You can now use python manage.py runserver again.

The only problem that i'm facing right now is that I can't reach the page from outside my virtual machine (I've rented a virtual private server to make it available for friends) because the ports are not opened. Any idea on how I can do that?
I've already added rules to my iptables for the port 8000 but it doesn't seem to work. I know I don't need to install anything else since on http://127.0.0.1:8000 I have the dndtools website working. (Thanks Lynx for being a command line browser...)

The followin part doesn't work for me :/
Edit : Nvm, it works. But I don't have any CSS...See screenshot below

Edit 2 I found why. It doesnt find the folder "static". But I don't know why :/
Edit 3 Ok, I put debug = True and it now detects the CSS. This is weird...
Edit 4 IT WORKS. YEAH. It works with local.py edited with debug = true and manage.py runserver 0.0.0.0:80

That's so cool!

 

[Isiloki]

I know it's a double post but I think it's wort a post on itself.
See here, I did a step by step how to
http://www.reddit.com/r/DnD/comments/2nkusx/dndtools_mirror_how_to_make_a_a_local_instance/