Suggestion - Item Database Idea

[Bulle]

A wild idea just occured to me, which could be used to get the best of both worlds.

We keep a full items table as it is, to save all those unused but accurate fields till we devise and have the time to spare to use them. Basically we change nothing to the original items table, items can still be changed/added in there. It is a 30 MB table, nothing to be worried about.

In addition we create an SQL View for the items table. This view would be used by the EQEmu server (or by tools not worried about the unused fields). In the view we only include the fields that are useful to the Emu. That way the Emu only reads what is necessary. In MySQL I am pretty sure the performance for reading from the view is close to what it would be to read from the original table with the unused fields removed. What is expensive when querying is process-to-process transfer (or machine-to-machinetransfer) from MySQL to the Emu, then re-distributing the item information in the C++ data structures, on the pure MySQL side the gain for having less fields will not be very significant.

Views are very powerful in SQL. You can also update, insert and delete through a view. For insertions the server will fill-in any field not in the view with the default value for the field. Then it is up to every individual to choose between working with the "simpler" view, or with the full items table. If you want to give it a try, create the following view in your Emu DB and toy with it (be careful, inserts, updates and deletes are actually done in the original table !) :

create view items2 as select id,name,weight from items;

There is just one limitation : I do not know how to specify default values for the view fields (for inserts) different from the ones in "items". It is not a blocking point as the default values in "items" could be changed with no impact on the existing data or on data inserted by specifying all the fields at once. It is just a matter of everyone aggreeing on the "best" defaults. You could even have your own patch you apply to your DB to change those defaults for your own installation. You apply it everytime you reload the table from the PEQ dump, and you are set.

Would that solution be good for you ?

[Kayot]

The items table is generated from a delimited text file. The seperator is the bar "|" symbol.

If you want to parse it into your db, make a vb.net program with the following Sub: (Some assembiliy required)

Public Sub Load_ItemTable()
    Dim objReader As New System.IO.StreamReader(items.txt)
    Do While objReader.Peek() <> -1
        Dim Split() as string = objReader.ReadLine().Split("|")
        Console.Writeline("This is the SQL command, You'll have to make me.")
    Loop
End Sub

The fields are out of order so you'll need to define values, plus there are more fields than the db takes into account. Hence why I left the above command empty.

As for http://www.eqemulator.net/wiki/wikka...uDBSchemaitems before the wiki went down, I did a bunch of the data thats in there. Mind you I don't get any credit for it at all (Due to wiki loss), which is part of the reason I no longer work on the wiki. I also made a new DB diagram http://kayosblade.googlepages.com/EQDBLayot.png that shows all the newer connection, mind you its about 1.5 years old and some of the stuff isn't link right now.

I think if we do nothing else to the DB, if we don't remove unused columns. We should at the very least, define the datatypes in the columns. Stop calling bit(1)/BOOLEAN's Integers. It takes more space in memory to store the number one then it does to store the bit 1. The number one is (0001), thats four bits for every row. Take that over 30ish columns and your waisting 90 bits per row over 62263 rows. Thats 5,603,670 bits of unused space just on unused columns that don't do anything at all. If you get rid of the column all together and you'll reclaim 7,471,560 bits or 1,867,890 bytes in ram just by removing the unused entries. Then retype all the other fields, add in defaults and god only knows how much space will be saved. Not to mention heads ache from programmers going, "What the hell does that do?" We should also add comments that explain each column (Hook it into the table design, not the tables information) so no one has to go to the wiki every time they forget what a column does.

And yes, I'll do it if it will be used. Add I have to do it edit the table structurer, the data should be fine. I'm considering removing the extra columns and seeing how it affects the emu.