TES4LL Woes

[Jimi diGriz]

As Andalay said on Nexus, we've been having problems getting TES4LL to run with Argonia.  It stops running when it calls TES4qLOD.

We have managed to build the project and debug it. 

The first problem we ran into was that the directories were not being created properly.  I tracked that problem down to an issue in llutils.cpp with it deleting variables while they were still in use. 

I also fixed an issue with the processing of the "-w" parameter in tes4ll.cpp where it was trying to process the subsequent worldspace name as a parameter.  I don't know if that actually caused any problems but it was confusing to trace.

Once past there we found that tes4qlod was failing.  We were able to build and debug tes4qlod by itself and found an issue with the processing of one of the cells in Argonia.  In Process4LANDData it tries to decode the VCLR, ATXT, BTXT, and VTXT fields after decompressing the LAND record.  It manages to find the VTXT field before processing the BTXT field so doesn't have any quad info to work with.  This causes a memory access error.  There isn't any error handing here so I assume it just dies quietly.  I am guessing that the ZLIB decompress is not working properly so we are looking at pulling in a more recent version of ZLIB. 

[AndalayBay]

Just to clarify: we grabbed the separate TES4qLOD program and set up a project for the source code, so we're debugging tes4qlod.c, not the cpp version. Both versions have the same issue. We found the c version easier to debug.

We compiled TES4LL and I ran the new version with MPGUI. I can now generate the LOD meshes and Normal Maps. It's just the Colour Maps that fail, or TES4qLOD as Jimi said.

We're using Boost 1.82.0.

[Gruftikus]

Thank you very much for your effort.

If you fixed the issue, the best thing for me for further processing is is you could fork the project on Github, commit the changes (please, only the lines with the issue, so no editorial changes over the entire code) to your fork, and create a pull request. In that flow I can go through the changes line by line and finally accept the pull request.

I'm using git in my professional field (a project with 20-30 members) and it is really the best tool to create and maintain community projects

 

[Jimi diGriz]

I have created the pull requests for tes4ll and lltool (under AndalayBay).

When I started using source control professionally git hadn't been invented yet.

[AndalayBay]

Jimi has been in IT for almost half a century!! No kidding. I couldn't resist phrasing it that way.

I'm a mere pup at only 30 years or so...

[Gruftikus]

Thank you very much, these are indeed some bugs which normally only unraveled if outer boundary conditions changed.

BTW when I started my career as a programmer, the internet was not yet born (at least not for me) and I had to type hex dumps which were printed page by page in journals into my C64

[AndalayBay]

5 hours ago, Gruftikus said:

Thank you very much, these are indeed some bugs which normally only unraveled if outer boundary conditions changed.

I thought that might be the case, but what really confused me is that Argonia is smaller than Tamriel. The NW boundary coordinates are -36,37 and the SE is 50,-37. Actually, that might be something. Do the physical dimensions of the land have to match the Cell Coordinates specified in the World Space window for the map?

Physically the NW is -40,40. The farthest landscaped point in the east is 49,7, but in the SE it would be 41,-32. I had to set the coordinates to be slightly different to get the map to line up properly with the land. If TES4qLOD is trying to access 50,-37 there's nothing there. In fact that cell doesn't exist in the world space.

I think I just figured out what the problem is. I need to change the cell coordinates in the world space window to use values that actually exist or I have to extend the land a lot farther in the south-east. I had to set it to the farthest point in the east (49) and the farthest point south (-36) to get the map to line up, but as I said, the resulting coordinates don't correspond to an actual cell.

5 hours ago, Gruftikus said:

BTW when I started my career as a programmer, the internet was not yet born (at least not for me) and I had to type hex dumps which were printed page by page in journals into my C64

Yeah it's amazing how quickly things have changed. Did you ever use punch cards?

I remember when there was a disaster somewhere overseas in 1990. It might have been Bosnia. Anyway I had a coworker who couldn't get in touch with her husband's family. Jimi worked for the university at the time and they had something called Usenet that they could use to contact a university in the area. She was able to send a message and found that his family was ok.

Jimi's first small computer was an Apple II. It had a whopping 64kb of memory with the expansion card. He also managed to save up enough money to get a 10MB hard drive. All told it cost over $8000!!

[AndalayBay]

It still doesn't work. I fixed the map coordinates so they are now pointing to actual physical cells, but the program still halts at the call to TES4qLOD.

[AndalayBay]

We also tried using the new version of zlib and that didn't help, so we don't think it's anything to do with the decompression. I'll have to set the map coordinates back to what they were because now our in-game map is not aligned properly. I'm in the process of extending the coastal region, so the coordinates might correspond to an actual cell, but I haven't gotten to that point yet.

Any ideas?

[Jimi diGriz]

We have got back to trying to debug TES4qLOD after AB regenerated all of the regions.  We have the same problem so now I am trying to figure out what the program is trying to do when it crashes.

We have traced it into Process4LANDData in tes4qlod.c (the C version, not the C++ version) with cell.current_x, cell.current_y = (42, -30) according to the debugging printf just before the problem.  The code appears to be trying to decode the CELL record to get the VCLR, BTXT, and ATXT fields.  It makes sense until I run into this:

	pos = 3289+1096;
	memset(vclr, 255, 3267);
	if (pos < decomp_size && strncmp("VCLR", decomp + pos, 4) == 0) {
		for (i = 0; i < 3267; i++) {
			if (pos+6+i < decomp_size) {
				memcpy((&vclr[0][0][0])+i, decomp+pos+6+i, 1);
			}
		}
		pos += 3267 + 6;
	}

I have no idea what the magic numbers being used to adjust the decomp location mean (pos = 3289+1096).  I don't know if this represents an assumption about the length of the preceding fields.  Later the code manages to process the VTXT field before finding the BTXT/ATXT fields so it is quite confused.  I am guessing that whatever the assumption is about the offset of the fields of interest is incorrect in this particular case.  Does anyone have any idea what these numbers mean?

[Vorians]

I have no clue about any of the coding and numbering, but if you were to delete that cell from the ESM using xEdit, does TES4LL then complete the process?

[AndalayBay]

On 10/19/2023 at 3:27 PM, Vorians said:

I have no clue about any of the coding and numbering, but if you were to delete that cell from the ESM using xEdit, does TES4LL then complete the process?

That cell is surrounded by other cells, so it would leave a hole. It's the new coastal area. Our worldspace now extends all the way to 49,-36.

I decided to try TES4LL again using MPGUI. It did manage to run TES4qLOD, sort of. The process completed, but there are missing files. Here's the map it created:

At first I thought it might be the texture cache so I regenerated it, but it produced the same results. I generated the LOD textures in the CSE and used xEdit to create a map. Here's what the map should look like:

It has shown some of the underwater cells, but it doesn't seem to like the new coastal regions. They are BMEastCoastline, which is comprised of five areas and BMSouthWestCoastline, which is comprised to two areas. It's only processing part of these areas. Here's a picture of Argonia's Regions. The red line is the border, so a lot of this is LOD. It didn't like BMSeafalls either, which is land.

Is there a way to specify that TES4LL should use the default water texture, or specify a colour for the water? I have a texture replacement for the default water, so I leave it enabled when I generate the LOD, which conveniently colours the water for me.

Edit: To clarify, Seafalls is the odd square blank area in the TES4LL map just under the Crimson Isle. It is not marked on the regions picture because its only purpose is to supply the region name. It doesn't have any objects. Unlike other regions that only supply the name, there is no underlying region with objects. It was landscaped manually. So it's a region without objects and no overlapping region that does so. I'm guessing that's why it was skipped. That doesn't explain why the some of the areas in the coastal regions were skipped. I'm guessing it's because they have multiple areas, but a lot of Bethesda's regions have multiple areas too.

[Vorians]

I wasn't suggesting you leave a permanent hole, but to remove it for testing purposes. If TES4LL then completed successfully, you'd know that that cell was the only problem, and you could then recreate the cell to see whether it was just bad data. I believe the CSE would automatically recreate it anyway the first time you saved.

[Jimi diGriz]

It's tricky finding which cell it is working on when the problem occurs.  I will see if I can add some debugging prints to get a better idea of the context.

[AndalayBay]

@GruftikusI don't know how familiar you are with regions, so here are some pictures to show the multiple areas in these regions.

This is BMEastCoastline:

BMSouthWestCoastline:

And here's BMSeafalls:

The really weird part for me is that it did part of Area1 in BMSouthWestCoastline, not the whole thing.