EOC: Universal Exporter/Importer Proposition

[Akkadius]

So I think I am going to end up doing this anyways, but I have been thinking of an idea that I think is definitely worth my time for the help of the community so hear me out on this.

Universal Content Exporter/Importer

What?

The idea of this would be that you can log into your database with EoC as you normally would and be able to export content in such a form that it can be universally imported into someone else's database flexibly so that it molds right into their environment.
The EoC already has Zone Tools, which I have never really explained to the community what its capabilities are, which is to be able to connect to a 2nd database and import content from a zone from one version to another and I've worked all of the bugs out of it and it works entirely. Import the zones NPC's/spawns/doors/objects/grids etc.


Although this idea would extend much further than the concept. The idea that you need to connect to a 2nd database and do the transfer as not as friendly as it could be.

Breaking Down Content Sharing Barriers
One of the biggest things in my mind that creates a barrier to helping each other out in this community is the ability to share each others work. The Wiki will help us break down those barriers over time here immensely. But there is another big component and that is the time, blood and sweat put into these servers for content.

Sometimes the server isn't able to stay up, someone has more commitments in life and they have the shelf their work. Even if someone released their database and binaries it is one thing to get it up to speed and get it going but it is another thing to keep it going, host it and maintain it from a content perspective.

Sometimes, a server developer would be open to help if they could trust a random person. How about being able to stage content for a zone locally and export it to another server?

Exporting to Flatfiles
The idea is that you would be able to take in a variety of components and export them into a flat file that is agnostic of ID's, they would be 'Pseudo_ID's' that when the importer comes to parse the data, it would then use the pseudo ID's to match up with free ID's that are available in the target database and all of the ID's would match up.

The other idea of this is that you have zone script files that reference old ID's and the ability to upload a zip file with the folders quest and have the exporter parse out all of the NPCID oriented commands that reference an ID, to then also be replaced with 'Pseudo ID's' so that when it comes time to import, everything is re-linked and migration of this content should be perfectly in tact.

We could extend this out with our new Wiki and have complete archives of server content that people can openly share, post pictures of content and its layout and people can choose to include that in their server. (This same concept applies to sharing knowledge and tools on the Wiki).

Your export options would include (But not limited to)

Tasks
NPC's and their spawn points
NPC Grids
Doors
Objects
Source Zone Version
Loot Tables and the item data associated with it
NPC Factions
NPC Spells
Merchant Lists
NPC Armor Tints
Quests NPC ID/Item ID/Door ID References


You could then take this (large) generated flatfile that only the importer can interpret on the target database and it would break out the information necessary and you can select your target version for this zones data.

This would give you the flexibility to be able to make an instance out of this content and leverage my 'expeditions' system that I will be releasing in the next few weeks.

Either way I am just shooting this proposition out as an idea and to see how many people would be interested in it. My guess is that most people would more than want to see something like this around. But I am posting this to hear your thoughts.

GO!

[Tyen05]

Its an A+ idea.

The only problem i see, is that most people running servers dont want to contribute or share.

So ya, i guess the real question is, how many servers would actually share to the community to make it worth the time of creatin something to allow them to share easily.

Id say do it anyway, sounds very useful.

[Mortow]

I think this is a great idea. Even though there is proprietary data out there that would not be shared, there are a lot of people that would share what they have worked on and that can only help others out. You have given this community a ton of help, excellent tools and data Akka and I certainly appreciate it. I am sure others do too.

[rhyotte]

Cool idea for sure!

[Akkadius]

For some people who might have a hard time understanding what I am trying to convey with this: This would allow content on a server to be saved and imported to ANY database at any time, it would not have to be reliant on having certain data because all of it would seamlessly transition from import to export.

You cannot simply write SQL queries that fit for any one persons database as there are ID's and things that have to line up correctly.

The point of how this Importer/Exporter would be written is in such a way that the specifics of a database and what ID's it is using doesn't matter as the exporter would fit around their existing ID space like a custom fit glove.

In the end, you could save all kinds of content styles to share with everyone. I'm not sure some people see how HUGE this is, you give other people the chance to use the content you created as inspiration to add on TOP of it and share the result of THAT? Do you have any idea how amazing that is?

I was really hoping I would get more discussion on this topic, because how I envision this is no different than what I am seeing with Landmark EverQuest right now is that everyone is sharing their work so that others can use it and build their own worlds in their own way.

This saves people SO much time customizing their environment for their sandbox and needs.

[Township EQ]

This is huge..

Also combined with the door tool that you've released the possibilities are amazing.. I've completely changed the look of the end of crystallos with it.. the ability to share something like this to everyone would be great.

http://i.gyazo.com/2ebe0f8757e80720588a3c12ba2fa896.png

Doing all that and then having someone add on to it would make for some pretty badass places.

[ChaosSlayerZ]

This could definitely come useful even for internal DB relocation.

BTW, whats a "door tool" ? I missed on that one

[Maze_EQ]

I don't think Chaos has ever had one post that has been helpful/interesting.

I use EOC when I'm away from my residence. Shit has been a god send. I finally broke away from using phpmyadmin and the like and committed 100% to EOC.

[Akkadius]

This could definitely come useful even for internal DB relocation.

BTW, whats a "door tool" ? I missed on that one

http://wiki.eqemulator.org/p?Placing_Doors&frm=Main

[Akkadius]

I don't think Chaos has ever had one post that has been helpful/interesting.

I use EOC when I'm away from my residence. Shit has been a god send. I finally broke away from using phpmyadmin and the like and committed 100% to EOC.

That's really cool to hear. The amount of use EoC has been getting is insane.

[ChaosSlayerZ]

Thank You Akkadius! Now I remember this thing. Its pretty cool.

I don't think Chaos has ever had one post that has been helpful/interesting.

I use EOC when I'm away from my residence. Shit has been a god send. I finally broke away from using phpmyadmin and the like and committed 100% to EOC.

I don't think Maze ever had one post that hasn't been snotty.

[knowom]

This is a great idea Akkadius the ID's having to be available for use and lineup properly is a good point. This defiantly could help foster a lot more custom content sharing for EQemu which is perfect to go along with EoC and PEQ Editor and how they aid development of custom content.

[Akkadius]

It would be something fairly intense to write, and I don't really want to donate my time to develop something that may not have enough interest in.

For me I personally think some people just aren't seeing what possibilities this kind of application would bring as it has been with EoC's upbringing in general. I know it would be super useful to basically archive and store content in a place people could add onto, expand or simply use in an instance fashion.

[knowom]

I can see your perspective in terms of donating your time into something that may only get limited use. If it was available I'm sure it would get used, but how much is unknown and difficult to determine.

There is probably a good middle ground alternative that might not be as ideal in the long term, but better in the short term and mid term. Any kind of semi organized file depot for content sharing could be really useful.

The more people are willing to commit to sharing content the more the community can flourish. A small contribution can branch into a much bigger one over time.

[jabedoben]

I'd ask George Costanza. He's the one I ask for advice on importing/exporting.

[Leetsauce]

Now that I have had a chance to mess around with your EOC, I'm on board to help in any way I can on this.

Your EoC has completely changed the way server editing can be done(at least for me), and the importing/exporting of tables or even entire databases via the EOC could revolutionize how EqEmu operates as a community.


Edit: Why aren't people tripping over themselves trying to make this work?

[sorvani]

Edit: Why aren't people tripping over themselves trying to make this work?

Because the ones it will really benefit have no idea what it really means.

Also, people come and go in cycles. Fairly slow cycle right now IMO for the EQEmu community.

[Leetsauce]

I was stuck using heidi to do most of my editing because of some stupid network issue up until this week. When I fixed it and logged into EOC for the first time my jaw literally dropped. If I had that reaction, others probably will too. Could we maybe add that to the start-up guide, or something like that? Perhaps advertise it a little better within the community? Or am I wasting time going down that route because everyone is already using it, but our community is shrinking faster than a startup on Akkadius' project actually starting?

[Akkadius]

What in the fuck are you actually saying in your last sentence

[Secrets]

My mind is full of fuck after reading that last sentence.

http://i.imgur.com/oGE8J7j.png

[Maze_EQ]

What in the actual hell are you saying?

Maybe you should consult KinglyKrab before making yourself look like an idiot.


http://i.imgur.com/2YeDA.jpg

[rhyotte]

@Maze_EQ,

HAHAHA!!! That meme Kicks man!

But...yeah, I have to wonder what he meant myself!

Akkadius, you kick ass! Serious thanks for all the hard work!

[knowom]

EoC isn't exactly a closely guarded secret on these forums. I found it just fine like 6 months ago and it was newer and less widely talked about. I really don't get where your coming from plenty of us on EQemu have mentioned EoC on a fairly regular basis. On top of that it has it's own forum section, is listed on the wiki, and is right in Akkadius's forum signature. Short of flashing taunt button that says click here it practically couldn't be more obvious and well advertised. Heidi SQL also isn't exactly entirely replaced by EoC it's got it's own strong suits both are just tools. #30YEARSLATER #ANGRYFISTSHAKING #DAMNPUNKKIDS #GETOFFMYFUCKINGLAWN

http://weknowmemes.com/generator/uploads/generated/g1332122091269808014.jpg

[vsab]

Although I don't really "get" the last sentence it doesn't look like he's attacking Akka so I think the reaction is a bit harsh :P

I *think* he's saying that the community is dwindling so the demand to make the universal importer/exporter won't be enough.

[Leetsauce]

It wasn't me attacking Akkadius at all. I see now why one would jump that conclusion because I specifically mentioned him working on his own tools. Apologies. It could have worded better.

Maze: No one, especially me, will ever give a shit what you think. How are you not banned every 3 minutes? ( I have a message in my inbox of this kid telling me "we could've had something awesome" lolololol***)

Vsab nailed it. What I was asking is: Is our community shrinking faster than this EOC connect is being developed? Is that why the universal import/export isn't already in place? These 2 were the very first questions I asked myself upon seeing it.

Granted it is not a secret, I didn't come across it until literally a week ago, and I've been around the EqEmu scene for awhile. I never went out of my way to look for it, but it was never mentioned in any of the newb guides I followed to start my server.

***http://tinypic.com/r/1072ccz/8

[Akkadius]

I don't get where the community shrinking perception is coming into place. We've got more people working on projects simultaneously then I've ever seen before.

We've got some serious shit underway, you all think the EoC is the bees knees. We're working on a 3D WebGL Editor that you will be able to load up zones visually and manage/edit/create all of your spawns physically. Be able to drag and drop NPC positions, npc pathing grids, door positioning, object positioning and all sorts of shit real-time. Now THIS if anything will only explode the creative potential.

The new Wiki has been helping out massively to serve as a medium to reduce the barrier to entry into dabbling in this project and we're working on improving all of that.

The reason there isn't more tools in EoC isn't a community thing, it's about time and justification for something. Tools don't just pop up out of my ass, I have tons of things to do and when I put the time into something like that I want it to be justified.

I know in my mind what I am proposing in this thread will be huge, especially if you combine it with the spoiler of a project I'm working on with another developer (Tabasco) and most likely others when things get to a certain point - that we will be looking at even a larger community maybe more in the developer sense, which overall brings value to the community. I don't see any issues with that.

[Cfuson]

Oh damn, i cant even imagine making an eq server with something that resembles WC3 editor, thats pretty fucking amazing. Would be pretty niffty to drag and drop NPC's right on to the terrain.

We would see so many better customized servers then we currently have if this 3dWebGL editor works. Instead of the seeing the same monsters in every single zone, which (usually) means just a slight name change to "customize it", we would see people putting more effort into customizing there server and put different enemies in varies places. Imagine making a zone like crushbone with mini Innoruuk's (caster) and Cazic Thule's (melee) as normal mobs. This just opens up so many posibilities, and it can be done so much faster then doing all the coding that is required at the moment.

That's one of the reasons i really enjoyed Akkadius's old Blood of the Akkadian server, He had for the most part, customized the hell out of that server and if he had this 3dWebGL program it would of been easier to make.

TLDR: I hope to see this "3dWebGL" working soon.

[vsab]

I don't get where the community shrinking perception is coming into place. We've got more people working on projects simultaneously then I've ever seen before.


I'm wondering if the devs should have a "Work in Progress" page? Github has some of the info but in a lot of cases things are being worked on that the non devs have no clue about (although some of it isn't whizz-bang so they probably don't care)

[Akkadius]

Here are some early early teasers of the 3DWebGL Editor - IE the future greatest fucking tool you will ever use.


Basic NPC's being loaded into squares.
3D Navigation


City of Mist

http://i.imgur.com/DOD7Ty7.png

Crushbone: Main Gate

http://i.imgur.com/jHMtTml.png

Plane of Air

http://i.imgur.com/0ebLJ7K.png

[knowom]

Akkadius I'm curious about this new tool. Will the box's show NPC names at some point? I assume they'll be editable for stuff like X, Y, Z, and facing coordinates among other stuff. It seems like it could be difficult to tell which direction the boxes are facing, but this is a teaser so maybe you have plans in the works to address that like changing the hue color on the box's front face lighter or dark.

[Tabasco]

There will be visible headings soon, yes.
The visual teasers are nice but there are loads of details that need sorted out and a lot of the work can't be readily observed yet, but we're getting there.

Here's a larger and more up to date view. (http://midian.us.to/eq/tmp.png)

Right now you can click an axis and move objects but I still need to set up a rotation widget.

[vsab]

So frickin cool!

[ChaosSlayerZ]

nice!

this reminds me of my Doom editor I used back in 96 to make Doom maps ;) It was 2D though.

[Akkadius]

Keep in mind everyone. That this is early stages in the works. Functionality is being pieced together.

Cubes of NPC's could turn into fully fledged NPC models.
The gray texture could turn into the actual textures used in the zone.

Some of these features may or may not show up sooner rather than later depending on the difficulty and how it gets in the way of making things functional before simply enhancing visuals.

Over time, ease of use and other things will be sharpened and such, it is currently a work in progress and the fact that this is even in the works is fucking amazing. Yes I dropped the F bomb because this is really exciting, though I'm not developing content right now, I'm extremely excited for the community.

[ChaosSlayerZ]

heh even with cubes - if I can spawn the zone by sitting at the "command center" looking at zone map - that's going to be uber ;)

[knowom]

I'm intrigued by it due to the implications of how it could speed up the EQ classic mob density db custom project I'm working on. I'm doubling all the spawns for EQ Classic for now and may triple them at a later date. Any of the non generic named NPC's will need some new names. Then the more labor intensive part will be altering the new NPC spawn coordinates to space apart so they aren't simply directly stacked on top of each other.

This tool looks like it could be perfect to help speed up that process along with a bunch of other stuff. In fact if I decide to add NPC mob type variations to the project it could help a lot with that up as well. I'm not sure how my custom project is going to turn out or who will be interested in it, but it'll be made available and hope it gets used and enjoyed in different ways as a new twist for EQ classic server types.

[Akkadius]

I'm intrigued by it due to the implications of how it could speed up the EQ classic mob density db custom project I'm working on. I'm doubling all the spawns for EQ Classic for now and may triple them at a later date. Any of the non generic named NPC's will need some new names. Then the more labor intensive part will be altering the new NPC spawn coordinates to space apart so they aren't simply directly stacked on top of each other.

This tool looks like it could be perfect to help speed up that process along with a bunch of other stuff. In fact if I decide to add NPC mob type variations to the project it could help a lot with that up as well. I'm not sure how my custom project is going to turn out or who will be interested in it, but it'll be made available and hope it gets used and enjoyed in different ways as a new twist for EQ classic server types.

Where we are envisioning on taking this, this should speed up a lot of development processes 10x/20x/50x as we further develop the interface.

[rhyotte]

Rock On! This is amazing!

[jpyou127]

Akkadius, did you ever start working on the export/import tool for NPC etc...?


Celestial

[Akkadius]

Akkadius, did you ever start working on the export/import tool for NPC etc...?


Celestial

No because it is quite the undertaking and I wasn't really getting any serious buy in on it.

I'm not going to invest my time into some serious craziness if there is no real seeable benefit to it by others.

In my mind it would be a way to contribute content without having direct access to another server which could be huge in itself.

I guess I'm open to having this discussion still

[chrsschb]

I'm just checking in on this since I never noticed it before (sorry :( )

One thing that I've always wished would be simpler is packaging up a piece of content. Say a zone and all its inhabitants. Or all the pieces of a custom GM event. Then you could just give that out as a package, the new owner runs the scripts and boom they now also have the custom zone or event, 100% ready to run.

The system would export everything required from zone data, npcs, spells, items, etc.


I don't know if that's possible, or even how it would work, but there doesn't seem to be a huge demand to share singular objects (like an NPC or item). The scripts that manage these are far more important and sharing that piece seems to be where the demand is.

[jpyou127]

I am having an issue at the moment with an older DB that I moved to the newest source. After I updated the DB its giving me odd issues. It would be really awesome to have access to a tool that I could export my changes and move them to the newer DB. I think this tool would be wonderful!

Celestial

[Akkadius]

If I can get enough people on board with this, I will start thinking about the best way to do this as it is kind of a complex task and requires quite a bit if scenario-oriented logistics.

I know this tool would be incredibly powerful otherwise.

[wirepuller134]

It would be quite the undertaking and very much appreciated.

I know it would be a great tool as we have spent a lot of time changing stuff around and creating events so fourth on our little server ,for just family. I know the real server owners would benefit from this very much as well as the project as a whole.

[jpyou127]

It would also be pretty awesome after reading the above posts that there could be a growing repository of add-ons that people can implement and enjoy. This would turn the project into a truly expandable experience.

Celestial