July 2020 Version Update

Yeah in storage slip, I was like I thought I got this already and spent the points again to get it. Then I found out its in the slip but find doesn't locate it.

Uhh... You DO realize this game is almost 20 years old and they honestly did not expect it to survive long enough to ever butt up against any of the hardcoded caps that were put in originally? As far as hardcoded caps go off the top of my head, we have Slip Storage (192 I think? They explicitly stated the exact value in the same apology/explanation post that I mentioned in fact,) CP per kill (the magic binary number 65,535 is the hardcoded limitation, not a display cap, since they once again did not expect to ever reach the point where the cap could be reached when they added the CP/JP system,) Eschan Beads (which IIRC was stated to be hardcoded to a limit of 65,535 (again) which is why they won't bother with raising the cap,) and other systems in the game like Haste have binary based (hardcoded) limitations.

Try to remember that the inventory system has been around since Day0, and we had a total of 3x 30 slot bags, IIRC. Gobbiebag quests brought it up to I think 50 slots at US launch? Setting the value for maximum number of items per character to 1023~1024 would would have made sense at the time because it would have been sky-high at the time that it was designed. The game was also designed to work on dial-up speeds, so minimizing required data transfer would absolutely have been a priority during development, so needlessly large values would have been discouraged. Given how fundamental the inventory system is to the structure of the game, it would take something approaching a full rewrite of both server/client software to expand the existing system any further than it has been already.

The fact that they are exploring the creation of an entirely separate inventory storage system is actually very impressive for the current dev team, and I will have no hesitation in purchasing the expansions, unless it's Mog Garden based (due solely to MG loading time.)

tl;dr: There are binary-based hardcoded limitations all over the place in this game, and it absolutely does make sense that they can't just add more bags because of them.

Regardless of how they do it, they are doing it so who cares on the semantics. I just need another storage so I can have fun trying other jobs. When you min/max a handful of jobs the inventory management becomes a nightmare if you wanna try something else. I can only imagine with AF3 comes out reforged to +3.

I'm hoping for a non-movable wardrobe with every piece of RF1/2/3 + ambu gear and ambu weapons on it.
just tie it into storage slips 15~18,20,21,23~28. only display in the wardrobe items you can equip and only the highest quality version of each. done. (it'd be like 60~ actual items + 11 ambu cosmetic items that changes whenever you job change)

I think I fixed this and merged it live.

Today in Odyssey I killed Langmeidong (Vedrofnir wing NM, giant bird). Around 50% it used Perfect Dodge and spawned two copies.

I did halo x2, chest, coffer today and got moogle mastery to level 2 in Sheol B

Per usual, Nyarlko with a giant wall of ignorance masquerading behind terms he doesn't actually understand. A 'hard-coded limitation' is not a meaningful term. If you take it at face value, it's a constant in the code that can be changed by simply adjusting one line that defines the constant.

However, in reality, limitations exist based on the amount of work that goes into changing them and nothing more. Everything you described is a practical limitation or design choice, not a 'hard-coded cap'.

Slip storage is limited because slips store data as bitflags in the last field of an item's full data. Items are currently 44 bytes, the least amount of additional items they can add to a slip is likely 32(4 more bytes) to keep alignment. This would add 4 bytes each to 1040-1053 storage slots(oh wait, you forgot temporary items? the game stores up to 80 of them in between storage and locker and maintains them for several different contents. While we're at it, the client displays a slot at index 0 for each bag, so it may be 81 slot bags rather than 80 in their DB). At 4160 bytes, adding 32 more items to a slip costs more space than adding an entire extra 80/81-slot bag(3520/3564). So, the practical limitations of it are extreme. It makes no sense to increase the size of every item when only the slips would use it. They could simply add additional slips, and additional storage for a lesser memory cost and similar effort.

CP is limited to 65535 because there are 16 bits allocated for the field in the incoming packet. Increasing it simply means changing a couple variables types and increasing the size of the packet that displays CP. SE chose to keep it as an effective cap instead of a display-only cap, because they either felt it was a reasonable cap to implement or they didn't want to adjust the packet. This is very easy to change.

Eschan beads may be internally stored as a 16 bit integer, but these are also not a difficult change to implement. A few type changes could be done in a matter of minutes.

The reality of the situation is that if there is a practical limit in regards to inventory space, it is based on the structure of the database. This can be adjusted in an operation done while servers are down; adding additional space will not require damaging any existing data and an automatic conversion will not be difficult to design. A 'hard-coded limit' is not a thing, and changes are always possible if the time is devoted to doing so.

Then, most importantly, SE ****ing said they were looking at it. I don't know why your dumb *** thinks you know better than one of the most active devs on the ffxi scene, and SE themselves. Go back to jerking it to your underage catgirls.

On a more relevant note, the likely bottleneck is their S>C communication, not the actual storage of more items.

An effective way to add more storage without increasing the time it takes to load post-zone would be adding additional bags that can only be viewed in a nomad-moogle type menu in a single zone(garden?). They do not have to be different bags than we are accustomed to; they can simply be unaccessible whilst outside a single zone so resources are not devoted to populating them every time you zone.

With the available plugins we have, any additional space can easily be streamlined. If you always have to change jobs in mog garden to receive an extra few hundred space, so be it. Set up your plugins to move everything for the job into wardrobes, everything not for the job back into the cold storage, and things are looking great.

This is what I've been saying for years, we don't need to be able to see inventory at all times. Unloadable/Unviewable outside the MH storage is perfectly fine.

Also fully on board for a "blackhole type crafting sack" that you can fill in the field but can't access outside the MH with items specifically.

@Thorny

First, thank you for the more technical details on the things that I brought up as examples. I admittedly don't know the way that packets are structured in FFXI, and happily defer to your more in-depth knowledge on the subject. :)

The things that I've brought up as examples were all things that I remember the devs stating were impossible (for them) to change. TBF, I'm doing little more than parroting what the devs have said, so I don't see how I'm second guessing them in any way. :/

If it's something that the people working on the game are incapable of changing, then I would describe it as "hardcoded". I'd think of something as "hardcoded" when it is something so integral to the overall function of a program that changing it would break too much of the program to do without rewriting an unreasonable portion of the code. I know from experience that you usually can't get away with changing a data type in a database w/o changing all of the code in the program that's related to it.

My understanding about how slips work (from the devs) is that each slip item is what contains the simple true/false bitflags that indicate if an item is stored on it or not. (I don't remember if the devs explicitly stated it, but I believe this utilizes the same data space as augments/signatures.) This is why slips can't be used to store any items w/ non-static augments, since variable augments can't be referred to w/ a single true/false flag. Items themselves get removed/deleted from player inventory when stored on a slip, and new copies are generated when something is retrieved from a slip, so it doesn't make sense to me that there'd be any flags on items to declare if they are stored on a slip or not. Are you talking about the flags on each item to declare which slip it can be stored on? Are you talking about the slip items themselves?

I've always assumed that temp items use an entirely different system/table compared to normal items, since I assumed that temps don't get actually stored within character inventory data as actual "items". Is this proven or confirmed by the devs to be incorrect?

This is all because I kept seeing people bitching that the devs should just give us more normal bags instead, and I knew that the devs have already said that it falls under the realm of impossible (or possibly "impossible for us".) I think that you might actually be agreeing with me in that the current devs are not capable of making changes directly to the current storage system? I actually can't figure out what we are arguing about at this point anymore beyond semantics. ^^;;

Since you probably know, I'm curious now. What is the largest packet size that gets used in the game? How do they get around that limitation when dealing with values larger than can be contained within that limit, assuming there are any values that are larger than can fit within that limit?

On a personal note, I apologize if you took my previous post as an attack, rude, sarcastic, etc. It was meant to just be confused questioning and there's no need for personal insults. As a general rule for me, if I'm intending to be rude, sarcastic, etc., then I will note my comments as such. Also, I will own being long-winded. It's how I was raised. XD

Yeah. Devs have flat-out lied about what they can and cannot do, things can be lost in translation, and some old limitations were lifted with multi-console support.

I am not 100% certain, but 95% certain that a few of the extra storage spaces that were added came before PS2 and XBox360 support were dropped though so they could have done some expansion of storage space and "PS2 limitations" was just an easy line to toss out and sounded better than "we know players want it and we could but we're not going to and will implement new future pay-gated storage later down the road".

@Nyarlko

Regarding slips, if I can make it more clear:
-All items are currently 44 bytes. This leaves 28 bytes of 'additional data' used for signatures, augments, or in slips case storage of bitflags. I don't know offhand if the limit is actually 192(it should be 224, but maybe there are 4 bytes reserved for something else in it).
-As RAM is addressed in 32 bit increments within FFXI, it is unlikely they would make a structure that is not sized in an increment of 4 bytes. Thus, the next step up would be 48 bytes.
-Making items of different sizes would be quite impractical, so to increase the amount of space available for slips they would need to increase the size of all items, not just slips, from 44 to 48 bytes.
-Increasing the size of every item in every existing bag by 4 bytes costs more space than an entire new bag, and both would require a similar database operation to accomplish, so increasing slip size is almost certain to never happen.

Temporary items are likely stored differently on the server; but the client treats them identically to other items, as does the traffic that populates the bag. We actually have 13 bags of 81 slots in the game client's memory, with slot 0 being used for gil in inventory and nothing in every other bag.

Packet size is not a limiting factor for values. A 4 byte field can contain values from -2,147,483,648 to 2,147,483,647 if signed, or values from 0 to 4,294,967,295 if unsigned. An 8 byte field can contain values up to 9,223,372,036,854,775,807 if unsigned(half if signed), and so on. Larger instruction(see next paragraph) are as big as 260 bytes.

The packet size issue itself is interesting, and requires some background information to fully understand.
-The things most players refer to as 'packets', such as a 'equip packet' or 'trade packet' or 'job change packet', are actually just instructions.
-A packet is a UDP transmission sent from the client to the server, it contains a header followed by any number of instructions. Each instruction states it's own ID and length, so it can be seperated out by the server.

The game client uses a 3999 byte buffer for outgoing packets, prior to decryption/decompression. After encryption/decompression, it is reduced to 1360 bytes. From this, we can conclude that it is unlikely they send actual packets larger than 1360 bytes, and they can contain data no larger than 3999 bytes post-decryption, the way the client is currently coded.

However, the data we receive is not consistant. Upon first logging in or zoning, while your various mission progresses, quests, abilities, and so on are populated, you'll see many incoming UDP packets that decrypt to 2000+ bytes of data. I've seen as high as 3940 in a short sample, again indicating that the limit is likely 3999.

Once everything besides inventory has been populated, if nothing is going on around you, you see packets that decrypt to 600-800 bytes(most of which is inventory population). The reason for this is less clear; it's possible that something about inventory doesn't play well with their encryption/compression process, but doesn't seem very likely given it is a factor of 4 less than their maximum packet size. The most likely circumstance, in my opinion, is they reserved a maximum amount of space per transmission for inventory to ensure that you were not missing out on players populating around you, actions going on around you, and similar.

If this were the case, it's possible that they could adjust their priorities or model such that inventory is populated last and takes all available space remaining in the packet. This would result in ~3800 bytes of inventory being sent at a time instead of ~550 in most circumstances, and increase speed by a factor of nearly 7. However, we don't know if that is the actual reason or how much code goes into it, this is much more likely to be a complex and difficult problem to address than the actual storage of the inventory.

As of now, some numbers(sampled using a char with 753 total items zoning into sea serpent grotto so 0 background noise or npcs):
-26,496 bytes, in 16 udp packets, taking 10.339 seconds, to finish the bulk data sent when zoning (average transmission size: 1656 bytes largest: 3912 bytes)
-26,416 bytes, in 36 udp packets, taking 14.767 seconds, to finish inventory, while maintaining mandatory packets such as player updates (average transmission size:733 bytes largest: 816 bytes)

As is, that's 25.106 seconds to fully populate my character. If they could manage to increase average packet size to 2000 bytes and only reserve 1800 for surrounding traffic, it could potentially be reduced to 12.77 seconds. OR, in the same amount of time, they could populate an additional 13 bags.

If they could set up their system in a way that the full packet size is always used(let's say 3900 bytes), and simply put as many inventory items as can be fit at the end of it, everything including our existing bags could be populated in 6.55 seconds. With an additional 30 bags, it would still take only 20.99 seconds(less than current).

Unfortunately, this is all just speculation, we don't know whether their compression is less effective on inventory items or their structure is done in a way they can feasibly accomplish these goals. But, we do know the client can accept packets much larger than those currently used to populate inventory. Further, populating inventory every time you zone should not be a necessity; if their synchronization mechanisms were more robust they could simply populate it once on login and keep it throughout zones.

I didn't take it as a personal attack, more comparable to FL governor talking down to Dr. Fauci. You clearly didn't understand the topic matter in your own post, so telling me I'm wrong is more than a bit condescending.

holy ***

This is the part that gets me. Even if they are unable to keep state reliably, they could just avoid blanking the inventory struct on zone, build a parallel struct, and swap pointers or copy after completing the alternative.

It's interesting to hear that they throttle the amount of inventory update chunks you can receive per packet. I had been assuming this whole time that they had a prioritization scheme, but perhaps they don't.

Regarding this bit, what plugins would those be? Is it just organizer? Asking for a myself.

I was speaking generally, Ashita has dressme and packer, I don't know if windower has anything besides organizer, but the availability of plugins to quickly and automatically rearrange items will make remote storage a pretty minor inconvenience if added.

People still reply to Nyarlko?

They added the Sheet of Jeuno tunes to the key items, so everyone should have a client update to download.

Because I "received" it already, I had it when I logged in.

Maybe people managed to dupe their stuff with blank inventories because this game was made in 2001, the time when game security was pretty non-existence and cheating was a feature.

So they just use a sledgehammer as the cheapest solution to their problems and took a swing at it. Years later, the new dev team would try to work around it and not touch the original code because it was made by some senior or something and they can't really fix it without consulting the seniors and the seniors can't remember it.

No dev thread access:
https://forum.square-enix.com/ffxi/threads/57054-%E3%80%8C%E7%AC%AC51%E5%9B%9E-%E3%82%82%E3%81%8E%E3%81%9F%E3%81%A6-%E3%83%B4%E3%82%A1%E3%83%8A%E3%83%BB%E3%83%87%E3%82%A3%E3%83%BC%E3%83%AB%E3%80%8D

Which mission is that? I just finished scrolling BGwiki and see no mention of it.

"vanadiel's star song" I assume, that it's finished Rhapsodies (song)

But it's a complete guess.

I am assuming it's the "A Rhapsody for the Ages" mission (final cutscene) for RoV.