Suggestion - Item Database Idea

[Bulle]

It is a double-headed problem here :
- those fields are most probably unused by the Emu, so during normal server operation they take up space for nothing
- those fields are supposedly accurate, meaning their value is Live-like and could be used in the (near or not so near) future, if someone cared to develop the functionality

If we remove the fields, then there is close to zero chance for them to be used someday, as the information will not be there to begin with. There is a related trap : custom DBs. The sane way to create a new item is to pick a similar one and apply your changes, that way you ensure most of the fields are consistent. Say I need to create a bag, best is to start from a similar bag and tweak a few fields. If the currently-useless fields are removed and some are reintroduced someday, they will have to be put back in (with their proper value, ie you must remember what item you started from) one by one. This can lead to long and difficult upgrades for custom worlds. I am not too worried about this problem as I keep my item definitions (as deltas to the original item) so I know I will be able to re-apply the initial definition, but not everybody works this way. I know this trap can probably be avoided with sane default values and such, but everyone should be aware of this impact.

On a related topic, could anyone clarify how the DB dump (in PEQ) is generated, how new Live-like items are introduced (manually or automatically from another source), and what the 13TH-FLOOR is ?

[Bulle]

A wild idea just occured to me, which could be used to get the best of both worlds.

We keep a full items table as it is, to save all those unused but accurate fields till we devise and have the time to spare to use them. Basically we change nothing to the original items table, items can still be changed/added in there. It is a 30 MB table, nothing to be worried about.

In addition we create an SQL View for the items table. This view would be used by the EQEmu server (or by tools not worried about the unused fields). In the view we only include the fields that are useful to the Emu. That way the Emu only reads what is necessary. In MySQL I am pretty sure the performance for reading from the view is close to what it would be to read from the original table with the unused fields removed. What is expensive when querying is process-to-process transfer (or machine-to-machinetransfer) from MySQL to the Emu, then re-distributing the item information in the C++ data structures, on the pure MySQL side the gain for having less fields will not be very significant.

Views are very powerful in SQL. You can also update, insert and delete through a view. For insertions the server will fill-in any field not in the view with the default value for the field. Then it is up to every individual to choose between working with the "simpler" view, or with the full items table. If you want to give it a try, create the following view in your Emu DB and toy with it (be careful, inserts, updates and deletes are actually done in the original table !) :

create view items2 as select id,name,weight from items;

There is just one limitation : I do not know how to specify default values for the view fields (for inserts) different from the ones in "items". It is not a blocking point as the default values in "items" could be changed with no impact on the existing data or on data inserted by specifying all the fields at once. It is just a matter of everyone aggreeing on the "best" defaults. You could even have your own patch you apply to your DB to change those defaults for your own installation. You apply it everytime you reload the table from the PEQ dump, and you are set.

Would that solution be good for you ?

[Kayot]

The items table is generated from a delimited text file. The seperator is the bar "|" symbol.

If you want to parse it into your db, make a vb.net program with the following Sub: (Some assembiliy required)

Public Sub Load_ItemTable()
    Dim objReader As New System.IO.StreamReader(items.txt)
    Do While objReader.Peek() <> -1
        Dim Split() as string = objReader.ReadLine().Split("|")
        Console.Writeline("This is the SQL command, You'll have to make me.")
    Loop
End Sub

The fields are out of order so you'll need to define values, plus there are more fields than the db takes into account. Hence why I left the above command empty.

As for http://www.eqemulator.net/wiki/wikka...uDBSchemaitems before the wiki went down, I did a bunch of the data thats in there. Mind you I don't get any credit for it at all (Due to wiki loss), which is part of the reason I no longer work on the wiki. I also made a new DB diagram http://kayosblade.googlepages.com/EQDBLayot.png that shows all the newer connection, mind you its about 1.5 years old and some of the stuff isn't link right now.

I think if we do nothing else to the DB, if we don't remove unused columns. We should at the very least, define the datatypes in the columns. Stop calling bit(1)/BOOLEAN's Integers. It takes more space in memory to store the number one then it does to store the bit 1. The number one is (0001), thats four bits for every row. Take that over 30ish columns and your waisting 90 bits per row over 62263 rows. Thats 5,603,670 bits of unused space just on unused columns that don't do anything at all. If you get rid of the column all together and you'll reclaim 7,471,560 bits or 1,867,890 bytes in ram just by removing the unused entries. Then retype all the other fields, add in defaults and god only knows how much space will be saved. Not to mention heads ache from programmers going, "What the hell does that do?" We should also add comments that explain each column (Hook it into the table design, not the tables information) so no one has to go to the wiki every time they forget what a column does.

And yes, I'll do it if it will be used. Add I have to do it edit the table structurer, the data should be fine. I'm considering removing the extra columns and seeing how it affects the emu.