How to create a 3d dungeon crawler (WIP)

[Ryue]

Hi
As indicated in the title this is a WIP (work in progress). I had a talk with one of the moderators of the forum (that was me - Taleweaver) before putting this up to make sure it will be in the correct subforum (especially as its a bit unusual to see WIPs in the cookbook section). As soon as everything is finished this first post will be replaced by a final one detailing what the dungeon crawler can, and what needs to be taken into consideration for its usage, ... .

So what is this about? For my project http://lemmasoft.renai.us/forums/viewto ... 43&t=31644 I need a dungeon crawler. I made my own one originally http://lemmasoft.renai.us/forums/viewto ... 51&t=32260 which used a semi 3d approach but found out it had a few limitations that I didn't like (where I felt too limited). Also in a few situations it didn't behave like expected. As a second approach I took a full 3d raycasting engine http://lemmasoft.renai.us/forums/viewto ... 51&t=15329 . It worked nicely but I ran into a few problems though. At my high resolutions it slowed down. Furthermore time and again I ran into memory problems. At first I was unsure if it was a leak, but after talking to catfish and also trying a few things I came to the conclusion that it is just the combination of my basic high memory usage (using quite a few images for the main game and also as tiles in the 3d crawler) and how python (which is used as the base language by renpy) handles memory (I will only say garbage collectors....I hate those things since C#). Furthermore I need a few additional things/options that are not existent in either vanilla program.
Thus I will need to do quite a few decisions and also a lot of work there. For this I'm going to do here a worklog.

So now to the most important part, what are the goals I want to achieve here (growing if something additional is found that is worth to be put in)?

1.) A 3d dungeon crawler where you can move through dungeons with 90° turns and moving a full map cell at a time (if possible an animated turn and move).
2.) The ability to use LARGE maps (512 x 512 in size)
3.) Different backgrounds for different parts of the map
4.) See through walls (thus if you have an open door you see into other parts of the dungeon.
5.) Sprites need to be displayed (example a well does not work as a tile/wall it works only as a sprite. The same for lamps).
6.) Sprites and tiles should be animated (example a waterfall rushing, or a torch having a slightly dancing fire.
7.) Overlays! Thus fog, rain, ...
8.) Short animation (like a punch) if you hit an obstacle on your way (with the ability to hear some sound there).
9.) Start events and also define random events for specific maps and also define the propability that these random events happen. These random events can be anything from one time events, to reoccuring ones to battles. Also being able to define fixated events for specific parts of the map.
10.) Define lighting conditions for parts of the map.
11.) Day/Night cycle (based on traveled map cells and also time being useable and checkable by events, ...).


On to 3.) That is a complicated one sadly. what this is about is that if you look into a cave entrance you shouldn't see clouds in the distance, but the cave background while beside it the grass is on the ground. The only way I found that would work there is if you do "tiles" as backgrounds. Which would mean that you have a ceiling and floor tile for each and every "indoor" part.


How do I go about these goals (growing if something additional is found that is worth to be put in)?

1.) I need to decide on using a custom approach or the renpystein approach. (This I will postbone a bit as I will first do step 2)
2.) I need to create a loading mechanism for the maps (where is which tile, which parts of the map are blocked and which are not, how the lighting conditions are, what events are and random event probabilities, ...).
3.) I need to create an automatic loading mechanism for the tiles (this is only somewhat dependent on the approach (custom/renpystein) and most of it can be done regardless of approach).
4.) I need to get the chosen approach to run (custom approach will be a complete rewrite, renpystein approach will need a different user interface than in renpystein).
5.) I need to put in the ability to display sprites (and "tiles" that are not a full wall like a well which is handled like a sprite)
6.) I need to put in animated tiles and sprites.
7.) I need to put in the ability to start the events.
8.) I need to put in the ability to display fog, rain, ....
9.) I need to put in the ability to use dynamic lighting and a day/night cycle.

As a note as this is a worklog and also a decision finding process I'm always open to suggestions and comments and critique.

[Ryue]

Part 1 - Loading maps
Most of the time when there is a dungeon crawler one travels through multiple different dungeons (maps) where some fixated stuff (fixated events) and random stuff (random events and battles) happen. And there are walls and also areas one can pass through, .... .

Thus the engine needs to provide the ability to load multiple different maps where events are possible and also the definition which tiles are shown where and which tiles can be traversed though and which not. Furthermore for each map cell sprites should be defineable (like torches on the wall, ...).

So the first step is to provide this functionality. And when possible in a way that is as memory efficient as possible.

So to list what I want this part to be able to handle can be broken down into the following:
* Loading maps by a given filename
* Defining the size of the maps (x/y cells)
* Defining general lighting of the map
* Defining if a day/night cycle is used for the map
* Defining the propability of random events occuring during the day/night on the map and how the "distribution" labels (one per day/night) are called. (so that each map has its own random event list which can take even key events into account)
* Defining each map cell (which tile is shown, which sprite is shown, lighting modifications for that map cell, and what type of movement (walk/fly) allows to pass through it).

As I'm used to a object oriented approach for things I will put this all into a "dungeon crawler" class. That aside the format for the map cells is the next important step.

I see 2 possible options there
1.) XML based files
2.) CSV based files

Even though XML based files have an allure there I want to be able to see at one glance what is in each map cell wihtout having to write my own editor for that. Thus I'm going with the .csv approach so that I easily can open the files in excel.

File layout for .csv files:

Code: Select all

MapName;XXXXX				
MapSize;XXX;YYY				
Day/Night;X				
Lighting;LLL				
Random;XXX;YYY;WWW;ZZZ			
Map					 
UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;....
UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;....
UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;UUU-VVV-WWW-XXX-YYY-ZZZ-;....
...

Mapname: XXXXX... string representing the map name (also mirrored in the filename, but not necessarily the same)
MapSize: XXX, YYY... 3 digits for the mapsize each
Day/Night: X... T, F (true, false)
Lighting: LLL... 0-100% base lighting
Random: XXX % chance for random events to happen (day) YYY % chance for night WWW name of the label for day event distributor, ZZZ same for night events (ZZZ, WWW can be the same)
Map: Tag indicating that the map cells begin below
Map cells: UUU... tile number (3 digits), VVV... Event number that happens on entering the cell, WWW... Event number that happens on leaving the cell,
XXX... Blocking (0...no block, 1...walk only blocked, 2...complete block), YYY... Sprite number shown in cell,
ZZZ... Lighting (-50% to +50% to basic lighting)


Update I added an example file (needs to be renamed to .csv as the forums don't let you upload a .csv file):

Forest.txt

(3.23 KiB) Downloaded 90 times

[Ryue]

So tomorrow I will upload the code. First thing though is: At first I fell into a trap. Why waste precious memory and load ALL maps at once at the beginning? Better to load the maps when they are needed.

As mentioned I fell into a trap there, which is easy to fall into. The thing is: If you do not preload you have either only the chance to save the modified maps (in case an event modifies them) or reload the maps when they are entered again and put each event that has passed into action there. What do I mean? Think of a random event that opens a previously closed door and thus changes the map. Now if you reload the original map.......the door is closed again. You would need to circumvent this to get it open again.

In order to avoid this and unnecessarily saving and loading from disk (and as a single map does not take up THAT much memory) I'm going to preload EVERY map.
And all that in a label as else saving and loading will get problematic (in case I do the preloading in a init part).

[Ryue]

So the code for the loading put together (only a few comments added in atm).
I made a whole file (Filehandling.rpy) where I put in file ios so that I don't have to repeat the commands in similar cases.

RedEyesDungeonCrawler.zip

(261.98 KiB) Downloaded 123 times

So that was it for part 1 (let me know if someone has any comments, additions or ideas on how to do it better).

[gas]

I was waiting for. Feedbacks ASAP.

[Ryue]

I was waiting for. Feedbacks ASAP.

?
What do you mean there?

[gas]

Oh, my bad english. I mean that I'm following the thread and updates and I'll be glad to give you any feedback as soon as possible.

[Ryue]

Oh, my bad english. I mean that I'm following the thread and updates and I'll be glad to give you any feedback as soon as possible.

Aah no problem there. That is why I asked before interpreting your words^^.

Next is part 2 the automatic loading of the tiles. Afterwards in part 3 the decision about the approach.

[Ryue]

Part 2: Auto loading tiles
As I know already how I want the game to handle displaying "tiles" (which is how I name the walls and their textures) I know that I will have to
a.) Have 1 normal light level tile and 1 dark colored version of the same tile.
And b.) Have 1 slice per pixel the tile is wide (all tiles must have the same width and height!)

Originally for the tile sizes I had them the same (the original sizes) as my screen resolution. Which proved to be quite hard on the used memory (more to it later).
I would say that 1/3rd or 1/4th of the screens resolution as tile resolution should be sufficient.

So in essence I have to go through a special directory and get all files with a specific extension and load them. One very huge problem with this though is the memory usage as that is even more problematic thanks to how python handles freeming memory. Python uses a garbage collector model for this. That means when you kill a variable, the memory is not immediately freed. Instead it is freed after some time.

So when you load large amounts of bytes into memory and then free it and load again,.... it can cause problems with the program using up too much memory.
A way to prevent this is to only load the tiles which are used for the currently used map. This has as downside that you have long load times when you change the map.

So what will I do there? I will give the load all tiles a try. If that is too much (I will have up to 100 tiles in the end) I will change it so that the "load only tiles for the current map" becomes an useable option.


So the next step is then the implementation.

[Ryue]

In addition to the changes above I added tile001.png so that a tile can be loaded.

RedEyesDungeonCrawler (2).zip

(1004.5 KiB) Downloaded 109 times

What still needs to be changed in the code is:
im.MatrixColor and im.Scale as these are old methods and should be replaced by more appropriate ones.

[Ryue]

So as the first two parts are now finished (loading a map and loading the tiles) the decision about the engine is the next. Thus

Part 3: Engine decision
Here I have to admit the original decision between two approaches changed to three aproaches now due to an answer to a question I posted in regards to im.Scale and im.MatrixColor.

So the three approaches are:

1.) Use renpystein and modify it to suit my needs and optimize it where possible (http://lemmasoft.renai.us/forums/viewto ... 51&t=15329)
2.) Do my own thing with a pseudo raycaster where I just put textures where I find them (not using slices at all). This would be similar in idea to http://lemmasoft.renai.us/forums/viewto ... 51&t=32260
3.) Try to convert another engine from javascript to renpy/python (http://www.playfuljs.com/a-first-person ... 265-lines/)

Each of these variants has its own advantages and disadvantages on to which I will go into details each (first though I'm going to finish reading through the engine for the third version so that I understand exactly what they are doing where and why so that I have a valid base for decision there).

As always if someone has some idea there or some oppinion you are very invited to post there especially as this third part is about finding a decision base for an engine decision.

[Ryue]

So I've read through that one javascript engine.

Pseudo raycaster
For this one I just controlled what texture is in the fields in front of the character and put the whole texture on screen. With this approach I needed one front and one "side" view of each wall as I didn't do any special calculations for angle from the point of view,... .
What the player sees and doesn't see is depending on the z-order of the displayed images.

Pros: Fastest approach
Cons: You can only do 90° turns and you need the artist to do 2 versions of everything. Furthermore regardless at what view angle the object is from the player it is always displayed the same. Also it has some troubles when it comes to specific map situations (specific locations of walls).

Renpystein
This approach uses a real raycasting engine at its core. Thus it sends a ray for each pixel width of the screen and notes what it sees. As soon as it hits something it stops and blits that on the screen.

Pros: Gives a real 3d environment and artists only need to create 1 single image for each wall type.

Cons: The engine needs to be modified to have see through walls (so that the ray does not stop at the first "wall hit". Furthermore a maximum distance needs to be implemented for just this case). The engine considerably slows down at high resolutions for screen and textures. The code is partly a bit difficult to read and use (if one wants to omit pygame usage).

Implementation specific con: Somehow I managed to create memory problems when I added it into my project (probably due to the garbage collector of python but as currently undetermined still).

New Javascript engine
In essence the same as above, only that the documentation is a bit more detailed for some parts of why specific steps are done. But as it has the same basic idea of how to do things it natuarlly has the same problems and advantages (as there are only so many ways to do raycasting).

One further Pro: The python port of it: https://github.com/Mekire/pygame-raycasting-experiment uses an object oriented approach. It has 20 fps with a texture of 1024x1024

One thing of note here it has a link to how comanche did things back in 1992: http://simulationcorner.net/index.php?page=comanche that won't work for walls though as it would only be able to do single colour "walls", but it COULD be an idea on how to do the ground and the sky.

--------------------------------------------------

With the troubles of the first approach I would go for the 2nd and 3rd as they give more flexibility and also allow "in between" turnsteps instead of looking "North" and then without anything else suddenly looking "West".

So in essence it boils down to using renpystein as a base and modifying it so that it has the abilities to do what I want it to do and at the same time trying to improve the performance as much as possible.

Or using the javascript engine as a base (and taking glances at the python implementation of it) and creating one from scratch.


I have to note there that I tried it (creating from scratch) with the tutorial that renpystein is based upon a while back and somehow ran into a calculation problem and never finding out what the difference to the renpystein implementation was.


Currently I'm leaning towards giving the javascript engine a try to convert it to renpy and optimize it. That way I wouldn't have the problem to have to deal with how things are named/done already and can try my own optimizations there. The downside is that if I do one error even it will be very very hard to find, thus it can be that I would have to then still switch back to renpystein as a base.

As usual if anyone has some insight/questions/ideas just say.

[xela]

Hard to say... which ever allows the most customization I guess. We need to figure out how to make it run faster, maybe 2 -3 pixel wide scanlines. I get +/- 30 frames with the Python/pygame port of the java thing btw. Half that with variable heights.

[Ryue]

Yepp he had mentioned 20 fps in the description so 30 is not shocking there.
What impresses me is that he gets that with the large images even.

I think thrre are 2 ways there
A) as you mentioned 2-3 px wide scanlines
B) rescaling the image to a smaller size before creating the slices.

Or a combination of those. That should decrease memory usage and also number of calculations considerably

[xela]

Yepp he had mentioned 20 fps in the description so 30 is not shocking there.
What impresses me is that he gets that with the large images even.

I think thrre are 2 ways there
A) as you mentioned 2-3 px wide scanlines
B) rescaling the image to a smaller size before creating the slices.

Or a combination of those. That should decrease memory usage and also number of calculations considerably

I am not sure that it's a good idea to create the slices in the first place, Transform has to crops the image to build a slice anyway and I am not sure that it does that to a cropped slice faster. Maybe the images themselves should become objects that carry properties like if they have any transparent space and stuff behind them got to be rendered as well. Like it makes no sense to do that for a solid wall (if that makes sense, I am still unsure on how the whole thing fits together).