InfoBar Addon Monster Database Improvement Program

Allow me to preface this post: HUGE shoutout to KenshiDRK for being an absolute LEGEND in reimplementing the InfoBar plugin as a LUA addon. I do not know what your plans are for bringing InfoBar to Windower 5 but want to contribute anyway.

I am working on adding missing and fixing incorrect monster information to InfoBar as I go along in doing the expansion content (Zilart and onwards, alongside Rhapsodies of Vana'diel) and grind my way to victory. I've started with the current database that is included when one picks up InfoBar via Windower, dated March 8, 2019, and wanted to inform the community as I am working along.

If anyone else wants to help contribute to patching up some holes in the monster tables, I'll need to know what monster and where, and what is missing/needs changed. I'll leave credit lines in my change notes I hope to submit alongside database updates. While I have made a handful of adjustments thus far (like adding information for an NM in Ordelle's Caves that was not listed, for example), I don't plan on submitting as a PR on Github until I feel that I have made some substantial progress, or I've run out of changes for a short while as I go through content. I do hope to push a PR by Christmas at the earliest, but certainly by the end of the year!

I will keep an eye on this thread from time to time for posts to add info to the database. You can also find me on the Windower Discord if you wish to pass changes that way as well.

Worst thief treasure hunter in Vana'diel out.

PS: had to wait a few hours before I could post this. In that time, I've already found some discrepancies between BG-Wiki's monster flags and what is reflected by testing in game. Making notes in the changelog as I go.

I'd be up for helping with this.

One of the improvements we should consider is allowing better Git version tracking. The database.db file cannot be compared line-by-line like ascii (text-based) files because it is binary. This makes it almost impossible to know if A) a PR contains only relevant changes, and B) if the changes will even work.

It is trivial to have sqlite dump an sql file and have changes tracked there instead. Yes, everyone submitting a PR would need to add the changes to the database, but this is no change to what is required now. It's a shame we don't have a build process for this repo, because having these kinds of assets built automatically would reduce the risk of remembering to do manual steps.

I agree that some improvements like that so changes are more public facing would be a good thing, especially if I end up making a goof.

I've been testing all my changes live and local, and so far, I have had zero issues. Granted, data entry isn't something that is necessarily difficult, but mix prior head injuries in life with trying to relearn SQL only to realize that a tool like DB Browser for SQLite makes it infinitely easier to manipulate and make the additions necessary.

I've also taken the liberty of using DB Browser of SQLite to export both as a JSON and as a CSV. The CSV failed spectacularly enough to cause LibreOffice to crash due to too many characters single cell. Not gonna dig any more on that. But exporting as JSON resulted in... well... it works exceptionally well. Granted, all the entries for each block is in alphabetical order, so it doesn't match the exact layout in the db table, but it appears all the data has exported just fine.

I'll post a codeblock of the first entry out of the db. It is for the Valkurm Emperor out of Valkurm Dunes. (for what reason we may never know) 'spose the community will be the judge if something like this is what we wish to use.

Another thing I'm realising now, seeing that example, is how a number of fields are probably redundant and just add bloat. For example, the fields allakhazam_id and atlas_id could reasonably be removed, reducing the overall size of the addon exponentially.

My guess is that the initial justification to use sql over a Lua or JSON option is the ease (and speed?) in which you can query data. Using the structure you have above, I can foresee the need to have it iterate over the data in Lua, which would be way slower. You couldn't index each entry by mob name or ID because they're not unique across zones, so the structure would have to be something like:

I've been making small entries into the current "add-on" db. Will that be of any assistance to you? or are you already well ahead on all the unmarked mob entries missing from this db?


-----
Legal Disclaimer: Testing of windower and etc is done via thru another player who wishes to remain anon, as I personally play online without the use of windower or its add-ons or plugins. This includes other variants of this nature. I give spare time to these projects to help everyone else =)

Any info is greatly appreciated! Odds are, given how "early" in the grand scheme of things I am, and the fact I got kicked in the face by the end of Zilart content (not Epilogue), you probably have information I do not.


You see, this would make sense. I don't know the overall speed of an SQL check v combing a JSON.

If we want to adjust how many fields are in the database, that's something that is easily looked at. Note that the total database is around the 3.3mb part, so it is not exactly massive. Though if we want to make any large scale changes to the database like that, we'd want to run that by KenshiDRK just in case there is a desire to utilize these fields. Looking over the LUA found here, it does not look like the following DB fields are even being used.

• somepage_id

• atlas_id

• ffxiclopedia_id

• allakhazam_id

From doing some research, it looks like Somepage was an FFXI site way back in the day that had stopped getting updates to the site itself as of 18 Apr 2012. Looks like the site was built like the old school pre-wiki online searchable databases. (Internet Archaeology at its finest)

Atlas... maybe this is tied to the former Monster Database hosted at the Vana'diel Atlas. The Vana'diel Bestiary that was formerly part of the site is long and gone now, probably unrecoverable by the team that has taken over and rehosted the Atlas. This field could be culled without thinking too much about it.

FFXIclopedia is of course still running. Hosted on Fandom, so the site looks even worse than it did before. Has some good info on it, but half the time I had gone to bounce BG Wiki and FFXIclopedia off each other to check/verify info before resulting to direct testing, FFXIclopedia's pages seem to have less information on monsters than not. At least FFXIclopedia does a better job explaining core mechanical fundamentals than BG Wiki seems to do. Could just be that I'm dumb and wasn't looking for the exact right page :D

Allakhazam for FFXI is something that I'd only personally seen recently. I've more known the ZAM for WOWHead before and after it took off a very long time ago. Looks like they do currently use a Wiki for their database back end. However, just sampling the DB shows that many pages have not seen an update in many years, if not a decade like their Warrior page versus the BG Wiki Warrior page. Does have some descriptive stuff on it, but certainly looks dated.

If we wanted to make use of the fields (by renaming to appropriate information sources of today), or remove the fields that are completely unused, I think we can do it without it causing any sort of trouble. I do hope these posts are helping drum up interest in some of our readers and maybe even chime in on what improvements we could (potentially) make to create an informational tool that is usable to all. I'm no LUA dev (last language I touched in any extent is Python and Commodore BASIC), but at least I can work databases. I will conduct !!SCIENCE!! to see if removal of either Somepage or Atlas fields will cause me any errors.

!!SCIENCE!! update: Removing the ZAM field did not throw any errors. Copying the DB into the plugin and nothing threw there either on targeting any mobs. The db size did not change substantially because there was zero data to wipe. The resulting JSON is smaller. It removed approximately 230kb of total size from the resulting JSON export.

As I understand from chats with people in the windower discord that were working on infobar 2 or 3 years ago (yeah I was gone awhile, I had to work an survive the pandemic) there was a plan being devised to pull various sites together an update the database regularly.

However the scraping or such required to do that was frowned upon by certain site admins or owners an the idea got tossed into a trash can.

Not that long ago when I walked back into discord, I got the immediate vibe that infobar became an outdated thing no one wanted to touch anymore an whoever I had been talking to a few years earlier was likely gone an infobar considered largely abandoned.

I was welcome to update the database IF I could but otherwise at that exact moment.. I was the only one who seemed to care about updating this at all (like no one else was motivated to do it).

I was invited to mess with various other add-ons an encouraged to basically abandon the plugins altogether from what I remember.

I declared alot of that bloat-ware an opt'd to insist the stand alone infobar be updated instead.

So I've just been adding things as I'm free to find the time, it's not a large amount of work yet.. many mobs are missing level info across ALL of google.

So it looks like it's just you an me for now?


-----
Legal Disclaimer: Testing of windower and etc is done via thru another player who wishes to remain anon, as I personally play online without the use of windower or its add-ons or plugins. This includes other variants of this nature. I give spare time to these projects to help everyone else =)

Given that the original InfoBar plugin source has long since been lost and the original author has gone dark, that's the reason being that KenshiDRK had rewritten InfoBar as a LUA script and is available via the Windower Github. KenshiDRK had also, around the time that InfoBar was converted to LUA, had also added some info in. At least some of the Escha mobs anyway. I do understand that KenshiDRK is still active, and I do believe may be receptive to community contribution efforts to help maintain InfoBar simply because having an "at a glance without having to open the wiki" line to look at and see what something has/doesn't have is an overall good for regular gameplay at all levels of one's personal skill. Information is power, after all. It is not a substitute for skill, but it does help even the playing field. Besides, I like it enough to spend time on it myself.

Part the reason for trying to deprecate the majority of the plugins and replace them with LUA is largely due to just that and more. LUA won't change all that much while all the programming behind the plugins themselves could change at a notice, especially when things like the entire backend of Windower being redone, as I understand that being the reason of Windower 5 having zero backwards compatibility with any existing addons OR plugins for that matter. How that will end we will see.

As far as information being available on other sources, I've personally found that information on at least the BG-Wiki to be mostly complete regarding monster information. There are some gaps in the knowledge there, for sure. But given that most the other websites, posted above and as referenced by the original IB Monsterdb, BG Wiki is most recent in the majority of its info followed by the XIclopeida. So automatic pulls of said information could end up being in conflict or pulled from a source that had not been changed to reflect the current reality. I've also had information be incorrect enough, as far as aggro and linking status, between both that I have had to either look at others in the family to see what it should/could be, or assign Class D personnel to conduct testing and verify one way or the other. Suffice to say, I don't have a shortage of Class D to use, but more eyes on this sort of thing is much better.

Also, if Mr. Square Enix really didn't want us doing things like this, I couldn't see anything outside of the POL web portal for XI existing in the first place. They say certain things to cover their butts. And if action were taken against me, I would personally take it as a sign to take my time (as unlimited it may be) and money elsewhere, likely to never end up in their wallet again.

And I don't believe we'll be alone in this endeavor. Xurion above has indicated a desire to assist as well. The more we post and talk about this, the more interest we will generate, in due time.

I think what we should at least have is a repo fork we can access and look at. That'll allow for better collaboration on the approach and discuss different options. If you have a link to your GH fork, can you post it?

I haven't setup such a thing quite yet. I'll work that out here shortly.

Okay, I think I have the fork setup.

I have also gone out of the way to pull the original database down to pull out and set the first monster.json as a baseline to compare against as we go along. Sucks that it is basically manual labor like that, but at least it is only 2-3 clicks to generate one.

Kinda wish one didn't have to fork the whole LUA repo, but oh well. The fork is here.

As I add entries to my db here, how do I get that to you or load in your db here on this side of things?

If I find the time, I plan to go from one area to the next confirming mob by mob to be in the db. It's the only way to be sure nothing was misssed. I won't be able to account for special fights an such however. Just whats in the open areas.


-----
Legal Disclaimer: Testing of windower and etc is done via thru another player who wishes to remain anon, as I personally play online without the use of windower or its add-ons or plugins. This includes other variants of this nature. I give spare time to these projects to help everyone else =)

Seeing as how it is a barebones BB forum, you would either need to submit it through Github or file drop it or something of the like. If you're still in the Windower Discord, you can reach out to me through there. If you've kept notes on what you have made changes on over however long you've done that, I'll take that as well so I can sit and compare against what's already done so I can work to merge it, one way or the other.

Lots of manual work. You'll get credited for helping along since this is a big deal for all of us :)

Now that the fork is available, I think the best way to contribute in a bitesize way would be to simply submit pull requests to it. This also prevents the need for anyone having to download change sets and merge them manually, reducing risk.

Assuming that the database approach is still going to be in place for the eventual submission back to the main Windower repo, I'd say it's worth having a tracked .sql file instead.

The JSON file is already over 7mb. I'm assuming this is because the JSON format is just larger and not because you've extended the data by more than two times. Although, kudos to you if that's what you've done!

You guys might want to use data supplied by LSB (private server makers) to fill in a lot of the info and refine it as needed. As much as I hate private servers, I do respect the makers of the software they do their best to get actual current game info.

Thorny/Comeatmebro has packaged it up nicely at https://github.com/ThornyFFXI/mobdb/tree/main/addons/mobdb/data for his addon for ashita.

Yeah, the start size is only 7mb. And, as mentioned on an issue that I'd put up, I can trim it down a fair bit just by eliminating some things. Figure we should get that squared out before we start doing adjustments to add data.


This can be very useful. Thank you for the word about this. I may use this as a last resort in order to get correct data for monsters if all else fails.

So then.. you can't just compare 2 db an merge whats missing?

Sorry I don't know how this will work, do you need me to keep a notepad of manually entered mobs or?

Even if private servers have their own contributions.. data still needs to be validated somehow no? or shall we just adjust as we go kinda thing?

-----
Legal Disclaimer: Testing of windower and etc is done via thru another player who wishes to remain anon, as I personally play online without the use of windower or its add-ons or plugins. This includes other variants of this nature. I give spare time to these projects to help everyone else =)

You can merge, kinda. Merging the db file is a no-go. However, two different copies can be exported to a text-based sql file and those can be merged. The db file can be rebuilt from that. This is the reason why having a db file in the repository doesn't aid collaboration.

Yeah. Also doesn't help that it pretty much has to be a fork in order to push it back upstream when we setup a set of changes for the db itself. Trying to put two and two together and what not. Not easy. If worse comes to worse, we could establish an entirely separate repo for the project and then use the fork to only push up DB changes.

Actually, that may end up being a much easier solution. What do you two think? Cause if that is the case, would not be hard for me to break things off and move it right over for us to get started with, since it is still pretty early in the overall process. I'll admit I should've thought to do just that in the first place, so I'll take the L on that.

It doesn't hurt to try? Just keep backups incase of emergency? it's up to you really.


-----
Legal Disclaimer: Testing of windower and etc is done via thru another player who wishes to remain anon, as I personally play online without the use of windower or its add-ons or plugins. This includes other variants of this nature. I give spare time to these projects to help everyone else =)

Right. Give me a few minutes here. I'll revert stuff back down, copy all my notes, and I'll setup a fresh repo to work off of. The existing fork is what we'll use to submit complete DB updates.

Edit about 20 minutes later: Repo is now here. I'll edit post above to indicate that the previous fork will now only be used to submit complete dbs, with reference to the working repo for audit by Windower team, as desired.

How involved has Kenshi been with this? I'd be hesitant to creating a new repo for someone else's work, even if it is based on good intentions.

I think Kenshi picked it up a few years ago to put some work on it around the same time someone else packed it in and called it. Their blog pointed out that Kenshi was starting to work on a LUA based version of InfoBar as far back as 2016. This blog also points out that someone else had linked them a new database as late as May of 2021, but that never made its way over to the main Windower repo at all. All links to that database are dead, and I've scraped the 'net hard to try and find it. Wherever it is, it is long and gone.

Kenshi's the prime guy behind XiView, so I wouldn't be surprised if something like InfoBar, either on the script or database side, is very very low on the priority list. If he stopped by to ask us to not punch out an updated database for one reason or another, I'd respect it. But for now, I'll work towards gettin' the framework setup so work towards the first substantial update to the database used by LUA InfoBar in years can begin properly. I'll still kick up the issues on GH about the same things we'd come across in the thread here, that way we have an easily seen conversation.

I have an idea on how this can be properly versioned as well as keeping the database aspect. I'll fiddle around with it a bit and let you know.

Sure thing. I'm fleshing out the whole thing, but we'll start with the .SQL aspect, seeing as how that is a very small file size compared to the DB, let alone the "2-3x larger" JSON. The SQL aspect is at least nice because a change for a monster entry will show the whole monsters' line and not just a single line inside a whole block that won't get displayed in github's diffs checker.

I would rather work with a db you've prepared. Then I will scour the land for missing entries again.. the few I have isn't worth your time yet an this seems like the better route to go for now.


-----
Legal Disclaimer: Testing of windower and etc is done via thru another player who wishes to remain anon, as I personally play online without the use of windower or its add-ons or plugins. This includes other variants of this nature. I give spare time to these projects to help everyone else =)

Setup a pull request into the main branch. If your monster table still has the entries for allakazam/somepage/atlas etc, we'll get the monster data changes done. Since we're doing this, it won't matter if we maintain those fields until we want to collectively change it, since the change of size is extremely minimal at best.

More important to get monster data done anyway. This isn't going to be an easy project for any single one of us, hence I'm glad others are interested in helping to improve the accuracy of the database (and find potential inaccuracies with the BG Wiki and other community resources.

I've posted a proposal PR over on the Windower Github.

I guess I will just keep my db going then.
I'll be in touch.

PS: I am making edits to wiki and BG at the same time to keep them with matching data. Not just for mob lvls.


-----
Legal Disclaimer: Testing of windower and etc is done via thru another player who wishes to remain anon, as I personally play online without the use of windower or its add-ons or plugins. This includes other variants of this nature. I give spare time to these projects to help everyone else =)