Database cleaner

[astonerii]

I was wondering if there were any tools/programs that I could use to remove duplicate entries from a .sql file?

I ask because there seems to be alot of different databases, and I would like to be able to put them all into one file, and then have another program clean them, by removing duplicated entries.

[Shawn319]

notepad.exe




[/joke]

i wish.

[astonerii]

I have tried to use MS EXCEL but it has a 65536 limit... and many of the parts have well over 100000.

[farce]

personally i'd use perl or sed or some such tool, and change all of the INSERT INTO's into REPLACE INTO's..

that way it wont complain about duplicate keys... it'll just replace what's using that key with the last one..

cat foo.sql | sed -e 's/INSERT INTO/REPLACE INTO/' > out.sql

would do it nicely.

[Lurker_005]

There isn't an easy way that does not involve lots of manual labor.

About the closest I have seen anyone come with a automated way was http://www.eqemulator.net/forums/viewtopic.php?t=11889&postdays=0&postorder=asc&sta rt=34 But that is a very basic way, and you have to already have the sql data merged without overwriting each other.