Tuesday, 27 November 2007

Linking to World of Warcraft items from Blogger (Revisited)

Some time ago I wrote a simple piece of HTML/Javascript to produce item tooltips driven by Allakhazam. I'm pleased to retire this script in favor of some new code from Wowhead.

The simple instructions are here and are summarised below:

1). Add this piece of HTML code anywhere on your page:
<script src="http://www.wowhead.com/widgets/power.js"></script>

2). Add a regular HTML link to an item in Wowhead, example:
<a class="q5" href="http://www.wowhead.com/?item=19019">Thunderfury, Blessed Blade of the Windseeker</a>

Tuesday, 13 November 2007

Questions & Answers (Updated)

To answer as many questions as possible I decided to write a new post, I will update and re-post this page as I answer more questions.

Q. How do you collect this data?
A. I’ve alluded to the way I collect the data in previous posts. I have written a specialised piece of software specifically to data mine the Armory and other sites. For example, to get it started I can point it at the Armory Arena Ladder pages where all participating teams are listed. For each Arena Team the team member characters are fetched, if they are guilded then the guild list is fetched and all guild members are retrieved. This technique has some problems; players that are not guilded and not part of an arena team are ‘invisible’. I have thought about different ways to counter this but I’m pretty satisfied that the data is representative anyway.

Q. Do you use the ‘last modified’ or ‘last logged in’ parameters to determine if the character is still used?
A. I record all that information but I don’t discriminate against ‘stale’ characters, if you look at the ‘Level Count’ in my earlier post you will see a huge bump at level 60 denoting people who haven’t yet upgraded to The Burning Crusade but also characters that haven’t been played in some time.

Q. Can you produce reports for levels 20, 40, 60 as well as level 70?
A. Absolutely, this is something I really want to get to do. Presenting it is my only sticking point, I will be including levels 1-70 in charts where possible. See the two graphs below for an example of what I am working on producing in the future.

Note: These charts are produced from a tiny subset of data just to test out my charting software. Interestingly, the 'Talent Spec Balance' chart illustrates that for Priests, Shadow is the best levelling spec but the balance quickly switches over to Holy from level 68/69 as Priests become more in demand as healers.

Q. Can you segment your data by Arena Rating / Gear Rating / PvP Kills?
A. Yes, I probably can but at the moment I am concentrating on updating my basic class-specific reports. Arena Rating and PvP Kills are easy to build discriminators on and I have worked on mechanism to segment level 70’s by their ‘gear rating’ (just dinged 70, casual & raider).

Q. Why is realm (name) missing from your stats?
A. I have not included all realms yet, in Europe for example I have not picked up any of the non-English realms.

Q. How are you getting data from levels 1-9, Armory don’t show that?
A. The Armory *used* to show low level characters, I noticed them disappear mid-October. So I still have a lot of 1-9 character data, during each refresh I try to fetch these character pages anyway should they reach level 10 or higher then I will get updated data for them.

Q. Can you show a breakdown of levels/classes by battlegroup, realm, realm type, and Horde / Alliance balance at 70?
A. Absolutely, it has officially been added to my TODO list.

Q. Can you profile most popular classes in arena teams and group them by their rating?
A. Absolutely, this too has officially been added to my TODO list.

Q. Can you breakdown your class-specific stats by faction to see if racials make a difference in class choice for each faction?
A. Yes, this is doable too I’ve kept as much data as possible with a view to do ever more sophisticated reports with it.

UPDATE 13th November 2007

Q. Can you track population changes over time, for example talent balances as patches change dynamics in the game?
A. Yes, originally I wanted to keep full revision history for each character, each time my spider would fetch their character page an 'xml diff' of changes would be stored. Instead, I chose to export my reports in an XML format - not the abstracted percentages you see here but raw numbers, the idea being I could compare reports to show the high-level changes over time. I have this in a limited form for patch 2.1 (May 2007), much richer information since patch 2.2 (Sept 2007) and eventually I will be able to publish the shifts from patch 2.2 to 2.3.

Q. What hardware do you run this on?
A. The spider software, database server, file-storage and report generation is all running from one of my old servers at the moment. Spidering is not particularly intensive as its a fairly slow process. To be specific the machine is a Dual AMD MP 2000+ (1.67Ghz), 2Gb PC2100 DDR RAM, 2x18Gb SCSI discs and 250Gb IDE disc, its running Redhat Linux 9 the only non-standard thing is that the XML repositiory is on a ReiserFS partition. Backup-wise, I have a full mirror of all the data on my home RAID5 partition.

Q. Do your computations take forever?
A. It depends on the type of report, for example the Realm Balance reports are all done using information in the database and take around 5 minutes to run. The reports that breakdown details on characters obviously require the XML character sheets and take much longer; these are done in phases.
   1). Select a list of desired characters from the database by class, level etc.
   2). Iterate through each character, fetch the XML sheet and extract all the required information and add it to a temporary 'report' table in the database.
   3). Use the 'report' table to produce stats, this step can be repeated without re-doing the first two allowing me to develop the reports without having to re-fetch all the data.
This process runs at about 250,000 characters per hour.

Q. Have Blizzard contacted you about the load / bandwidth you are generating?
A. No, Blizzard have not contacted me about the load my spider generates. I invested more time in the 'collector' aspect of the spider than any other part; there is a lot of timing / delays / caching which prevent it from producing a crippling load. The spider supports a huge range of HTTP/1.1 features including things like If-Modified-Since and Accept-Encoding compression if the Armory servers choose to honor them, its User-Agent string clearly identifies this project and a link to this blog and lastly, all traffic comes from a single IP.
Interestingly, over at The Build Mine Kuroshiro spotted a notice on the Armory updates page saying "... third-party sites that mine Armory data may need to make adjustments to account for the new file configurations ..." which implies they are aware of mining projects but for the moment haven't explicitly banned them.

Q. Why do you use an apostrophe to pluralize your class names in the chart titles?
A. Because at the time the apostrophe provided an easy way to separate my PHP variable name from the "s" (see code) which I have recently fixed.

Q. Can you show the Realm Balance reports but only for level 70 characters?
A. Yes, I just posted it.

Q. Can you determine how many Arena Teams are going to be included / disqualified by the tighter rules for titles "20 games played and at least one person with 20% of the total games played" ?
A. Yes, this too has officially been added to my TODO list.


If you have any questions / comments, feel free to post here or email me :)

Realm Balance (Level 70 Only) - 1st November 2007

Due to popular request, this is a re-run from the same data set as my last Realm Balance post however only level 70 characters are included.

Alliance AllianceHorde Horde
RealmDruidHunterMagePaladinPriestRogueShamanWarlockWarriorDruidHunterMagePaladinPriestRogueShamanWarlockWarriorAllianceHordeTotal
EU Aegwynn10.7%10.7%14.2%10%12.8%10.6%4.8%10.8%15.5%8.3%9.6%14.2%6.4%12.1%10.9%11.2%11.9%15.3%62.3%37.7%4,270
EU Aerie Peak8.9%14.1%14.3%13.4%10.3%10.3%3.5%9.7%15.4%8.8%13.3%13.1%5.4%11%10.4%9.8%11.8%16.4%68.7%31.3%2,840
EU Agamaggan10%13.8%13.3%12.3%11.6%11.2%3.2%9.9%14.6%7.9%10.9%13%6%11.7%12.5%11.2%11.9%14.9%48.7%51.3%3,482
EU Aggramar11.4%11.6%13.8%11.5%13.2%11%3.4%9.9%14.2%9.4%11.9%15.1%4.6%10.5%10.7%10.9%11.5%15.3%58.4%41.6%4,866
EU Ahn'Qiraj11.2%11.2%14.8%10.2%10.8%11.9%4.4%11.1%14.4%9.4%9.6%14.1%5.3%11.9%12.7%10.9%12.4%13.7%49.7%50.3%3,461
EU Al'Akir11.1%10.2%13.5%11%12%12.5%4.9%10.7%14.2%8.9%9.2%13.2%6.6%11.4%13%11.1%11.6%14.9%47.9%52.1%6,670
EU Alexstrasza10%12.3%15.8%10.3%12.9%10.4%3.2%10.2%14.9%9%11.6%13.7%4.1%12.1%10.5%11.9%11.7%15.3%69%31%6,183
EU Alleria9.8%12.3%16.4%10.4%12.7%10.7%3.3%9.8%14.6%9.4%12.4%15.1%5%11.1%10.3%10.4%11.6%14.8%71.2%28.8%6,340
EU Alonsus10.3%17.4%12.6%11.1%9%9.7%2.9%10.4%16.5%7.7%13.8%13.7%5.5%10.6%11%9.1%13.5%15.1%67.5%32.5%3,081
EU Aman'Thul9.9%12%15.3%10.6%12.8%10.3%3.7%10.9%14.5%8.1%11.6%15.9%5.1%12.2%10.8%10.3%11.5%14.5%69.9%30.1%4,272
EU Ambossar9.6%13.2%15%10.8%11.4%8.6%4.3%11.5%15.6%7.5%12.5%14.6%6.1%11%8.7%11.5%12.9%15.3%62%38%2,468
EU Anachronos10.2%13.5%14.8%12%11.3%10.8%4.1%8.9%14.3%8.6%12.1%14.3%4.8%10.8%10.9%11.2%12.5%14.7%59.8%40.2%3,339
EU Anetheron10.6%11.1%16.6%11.1%11.4%10.7%3.7%10.1%14.7%7.8%9.6%15%5.7%13.2%10.7%11.3%11.1%15.5%48%52%3,408
EU Antonidas9.6%12.3%16.5%10.4%12.8%10.7%3.5%9.9%14.2%8.4%12.1%14.8%4%11.3%10.6%11.4%12.6%14.9%73%27%6,196
EU Anub'arak9.4%10%14.9%9.7%13.5%10.7%4.2%12%15.5%8.4%10.4%14.2%5.5%11.9%10.7%10.6%12.2%16%47.2%52.8%4,258
EU Arathi11.4%13%14.4%11.1%9.6%12.4%3.1%10.1%15%8.2%12.1%12.2%4.8%10.9%12.3%11.9%13.5%14%44.7%55.3%2,084
EU Arathor10.4%11.8%15.1%10.6%12%12%4.3%9.9%13.9%9.5%11.8%13.3%5.3%11.9%10.3%11.6%11.2%15.1%69.1%30.9%5,479
EU Archimonde9.5%10.1%14.6%10.9%12.2%11.9%4.9%9.6%16.3%9.1%8.5%13.9%7.4%12.9%12.1%11.1%11.1%14%47.6%52.4%4,641
EU Argent Dawn10.9%11.6%14.3%11.4%11.6%11.4%4.3%10.3%14.2%9.2%10.1%12.7%5.6%11.2%11.4%12%12.2%15.6%63.3%36.7%5,863
EU Arthas9.9%11.3%15.1%10%13.1%11.1%4.4%9%16.2%8.5%9.6%13.8%5.9%12.5%11.3%11.5%12.3%14.6%64.4%35.6%5,165
EU Arygos10.6%15.5%14.3%10.3%10.7%8.9%3.4%10.5%15.8%7.9%13.3%13.8%5.6%12.1%9%9.6%12.5%16.2%65.5%34.5%2,615
EU Aszune10.8%12.3%14.9%11.2%12.6%10.1%3.9%9.9%14.2%9.9%11.4%14.1%5.6%12.3%10.3%10.7%11.5%14.2%62.4%37.6%4,988
EU Auchindoun10.1%14.5%11.2%11%10.3%13.5%6.2%9.2%14.2%8%11.1%14.4%8.7%11.6%13.7%9.5%12.1%10.9%38.9%61.1%1,451
EU Azjol-Nerub10.5%12.3%14.2%10.5%11.7%12.2%3.5%10.4%14.7%8.8%11.2%13.3%5.2%12.4%10.6%11.6%12.7%14.3%69.1%30.9%5,808
EU Azshara10.2%10%15.4%10.6%13.4%11.2%4.7%9.8%14.6%8.9%9.4%14.4%5.9%12.7%10.7%11.2%10.9%15.8%53.2%46.8%5,250
Alliance AllianceHorde Horde
RealmDruidHunterMagePaladinPriestRogueShamanWarlockWarriorDruidHunterMagePaladinPriestRogueShamanWarlockWarriorAllianceHordeTotal
EU Baelgun10.7%12.6%14.9%10%11.9%11.7%3.7%10.2%14.4%8.6%12%15.2%4.6%11.2%12%11.8%9.8%14.8%70.4%29.6%5,798
EU Balnazzar10.6%10.8%14.2%10.8%11.5%12.4%5.2%10%14.5%8.4%9.8%13.2%5.5%12%13.2%11.7%12.1%14.2%49.2%50.8%4,947
EU Blackhand10%11.9%16.5%10%12.6%10.7%3.9%10.1%14.4%8.4%10.8%15.8%5.4%12.4%10.1%11.6%11.6%13.9%60.5%39.5%5,774
EU Blackmoore9.3%10.4%15%10.3%12.2%11.6%4.7%11%15.3%9.6%9.4%14.6%6%12.6%10.8%10.5%11.1%15.4%58.9%41.1%4,859
EU Blackrock9.5%10.6%15.5%9.7%13%11.7%3.8%10.9%15.3%9.3%9.3%13.7%5.1%12.9%11.8%11.1%11.5%15.3%53.6%46.4%5,882
EU Bladefist11.4%11.6%13.2%12.6%11.3%11.5%3.7%10.7%14.1%8.2%9.9%14%4.7%11.1%12.3%11.3%13.9%14.5%48.1%51.9%2,734
EU Bloodfeather9.7%10.6%14.6%11.5%12.3%12%5.1%9.8%14.4%8.9%9.6%14.4%6.3%12.4%12.2%10.2%11.5%14.4%45%55%4,432
EU Bloodhoof10.9%12.2%14.8%10.6%12.3%11.6%3.6%9.9%14.1%10.5%11.8%13.4%5.2%11.7%10.7%11.1%11%14.5%69%31%5,921
EU Bloodscalp10.9%10.8%14%12.1%11.2%11.9%5.2%10.2%13.7%9.5%10%12.7%6.6%12%12.4%10.1%11.2%15.4%45%55%5,178
EU Blutkessel8.4%9.2%14.6%13.2%11.5%11.5%7.6%8.1%16%6.7%9.5%16.5%9.6%11.8%11.5%8.4%12.8%13.4%26.3%73.7%1,359
EU Boulderfist10.9%10.6%14.4%9.9%11.6%12.7%4.6%11.3%14%8.3%9%14.6%7.4%10.7%11%11.4%13.2%14.3%49.6%50.4%3,234
EU Bronze Dragonflight9.7%14%15.4%11.3%12%10.5%3.4%9.8%13.8%9.4%13.7%11.6%6.7%9.5%11.6%9%14%14.6%64.5%35.5%2,581
EU Bronzebeard10.1%13.5%13.7%11.3%12.2%11.2%4.3%9.3%14.3%8.1%13.3%13%4.8%11.3%10.3%13.2%11%15.1%65.3%34.7%3,984
EU Burning Blade10.8%13.4%14.9%11%10.5%11.8%3.7%8.8%15.2%6.8%9.8%13.7%5.5%12.5%12.8%12%12.4%14.6%58.7%41.3%2,410
EU Burning Legion9.7%11.1%14%10.8%12.1%12.1%4.8%10.5%14.9%8.8%9.2%14.4%6.1%11%12.9%10.3%12.5%14.7%55.7%44.3%6,152
EU Burning Steppes9.8%15.1%11.4%11.9%11%11.6%4.1%9.5%15.7%7.4%12.1%14%7.1%11.3%12.9%9.7%13.4%12.3%33.3%66.7%2,029
EU C'Thun11.5%12.9%14%10.7%11.5%11.5%3.8%10.6%13.6%8.7%10.6%14.1%5.9%10.5%12.6%10.9%13.5%13.1%48.1%51.9%6,278
EU Cho'gall10.7%11.8%13.5%10.8%11.5%12.9%4.2%10.3%14.3%8.9%11%12.8%5.7%11.4%13.1%10.7%10.6%15.7%63%37%4,338
EU Chromaggus11.1%11.4%13.4%10.3%11.8%12.9%5%9.6%14.5%9.5%10%13.6%6.3%12.3%11.8%10.6%11.8%14.1%44.2%55.8%4,256
EU Crushridge10.3%12.1%14.6%11.8%10.5%11.1%3.9%12.9%12.8%7.9%9.8%14.9%5.7%11.2%10.5%12.1%13%14.8%52.2%47.8%5,165
EU Culte de la Rive noire8.9%17%13.3%11.8%7.3%12.8%3%10.5%15.4%8.9%10.1%10.8%7.7%12.5%11.6%10.5%13.2%14.7%50.1%49.9%1,620
EU Daggerspine10.4%12.6%14.2%11.6%10.8%13%4.1%10.1%13.3%8.8%11.7%13.1%4.7%12.5%11.3%10.8%12.3%14.9%55.5%44.5%3,425
EU Dalaran10.4%12.7%13.7%10.4%12.5%11.8%4%10.4%14.2%9.6%11%13.2%5.2%12.1%10.5%11%11.8%15.7%64.8%35.2%4,125
EU Dalvengyr10.8%13.2%15.8%8.9%12%11.6%3.4%8.7%15.7%9%9.4%13.5%5.5%11.6%11.9%11.1%12.4%15.6%37.6%62.4%3,256
EU Darkmoon Faire9.4%18.2%12.9%13.7%9.7%9%2.4%8.5%16.1%7.5%18.2%13.4%7.1%7%9.6%9.3%12.1%15.7%69.6%30.4%1,842
Alliance AllianceHorde Horde
RealmDruidHunterMagePaladinPriestRogueShamanWarlockWarriorDruidHunterMagePaladinPriestRogueShamanWarlockWarriorAllianceHordeTotal
EU Darksorrow10.7%10.3%14.2%11.2%11.3%12.4%4.7%10.5%14.7%9%9.9%14.3%6.4%12.1%11.5%11.3%11.6%13.8%54.7%45.3%4,651
EU Darkspear10.2%16.5%12%13.3%9.1%9.5%2.5%10.5%16.4%7.4%13.1%13%6.4%10.9%10.1%10.8%12.4%15.9%66.1%33.9%3,156
EU Das Konsortium9.5%11.6%11.2%14.9%9.5%12.4%5.8%10.7%14.5%7.5%11%16.1%10.5%12.3%12.9%6.7%12%11.1%26.1%73.9%926
EU Das Syndikat10.7%12.6%13%12.2%11.8%11.4%4.3%9.9%14.2%9.2%9.7%14%5.4%12.9%10.4%10.3%12.2%15.9%45.3%54.7%2,729
EU Deathwing11.2%11.6%14.3%10.6%11.5%12.6%4.3%10.4%13.5%9.2%10.4%13.8%5.1%11.4%12.5%11.3%12.6%13.7%48%52%4,651
EU Defias Brotherhood9.8%10.9%14.7%11.4%12.4%12.9%4.8%9%14.2%9.8%10.2%12.7%6%11.9%12.8%11.2%11.3%14.2%43%57%4,116
EU Dentarg10.4%11%13.6%11.1%12.1%12.2%5.2%10.9%13.4%8.6%10.6%13.5%6.3%11.6%12.3%10.2%11.9%15.1%41.4%58.6%3,684
EU Der Mithrilorden8.1%14.2%16%12%9.3%9.5%3.1%10.8%17.1%8.3%12.7%13.7%6.1%9.8%10.3%11.1%13.7%14.4%68.3%31.7%1,869
EU Der Rat von Dalaran10.5%11.8%15.8%10.4%12.2%10.4%4.1%9.9%14.9%9.6%11%12.7%5.9%11.9%9.7%11.4%12.3%15.5%57.6%42.4%2,741
EU Der abyssische Rat11.1%12.9%13.7%11.3%10.8%10.5%4.4%11.3%13.9%8.6%10.1%14.6%5.2%12.2%11.5%9.9%13.6%14.3%50.6%49.4%2,146
EU Destromath9.7%10%15.1%9.8%13.4%11%5.2%9.5%16.3%9%8.8%14.3%6.5%12.8%10.6%11.3%11.2%15.5%45.2%54.8%5,337
EU Dethecus10.9%12.3%15.5%9.7%11.9%10.8%3.7%10.3%14.8%7.9%9.8%14.4%5.3%12.2%11.2%10.8%12.5%15.9%51.8%48.2%5,438
EU Die Aldor11.3%17.3%11.3%11.3%9.5%5.3%3.9%9.9%20.1%7.2%13.2%13.2%11.8%13.9%9.3%6.3%13.7%11.4%39.6%60.4%714
EU Die Arguswacht10.1%12.8%14%10%11.3%11.1%4.6%10.6%15.5%9.8%11.2%13.2%5.5%12.3%12.5%10%11.5%14.1%57.3%42.7%1,634
EU Die Nachtwache9.8%15.2%14.9%11.5%10.5%10.4%4.1%10.1%13.6%8.1%12.2%14.5%5.8%12.4%8.6%10.7%12.5%15.1%60.7%39.3%2,273
EU Die Silberne Hand9.9%12.4%15.2%9.8%12.4%10.7%4.3%10.5%14.9%9.2%10.8%14.4%4.8%12.6%9.5%11%12.1%15.6%70.5%29.5%3,016
EU Die Todeskrallen10.9%11.3%15.2%10.4%12.9%10.3%4.6%10.7%13.7%10.1%9.8%13%7%12%11%11.2%11.3%14.6%49.1%50.9%2,925
EU Die ewige Wacht11.3%13.8%14.9%10.1%12.3%10%3.4%9.9%14.1%6.5%13.1%16.1%5.9%12.6%9.8%11.3%11.8%12.9%74.7%25.3%2,140
EU Doomhammer10.2%12.7%14.1%11.1%11.9%10.1%4%11.2%14.7%8.8%11.4%13.6%5.2%11.3%11.5%11.3%12.6%14.3%65.1%34.9%5,155
EU Draenor11%12.9%13.7%10.2%12.7%11.4%3.2%10.7%14.2%10.1%11.9%13.5%5.3%11.7%9.5%10.3%12.5%15.2%61.5%38.5%5,338
EU Dragonblight11.2%12.8%14.7%11%12.4%11.1%3.5%10.3%13%8.8%12.3%13.3%4.5%12.2%10.9%11.5%11.6%15%61.6%38.4%5,025
EU Dragonmaw11.4%12%13.9%11.3%11.8%12%3.5%11.7%12.5%9.3%10.4%13.2%5.5%11.8%12.1%10.6%12.3%14.8%44.5%55.5%3,647
EU Drak'thul10.4%11%14.2%12.1%11.4%11.5%5.3%9.5%14.6%9%10.9%14.7%6.3%11.6%11.3%12%10.7%13.6%45.9%54.1%5,059
EU Dun Modr10.4%13.5%15.6%11.1%11%11.3%3.3%10.1%13.6%7.9%11.4%15.1%6%10.1%11.8%9.7%13.7%14.2%65%35%5,575
EU Dun Morogh10.2%11.8%15.4%9.8%12.2%9.9%3.9%11.5%15.4%8.7%11.8%15.4%3.9%11.5%10.1%12.4%11.2%15.1%68%32.1%4,000
Alliance AllianceHorde Horde
RealmDruidHunterMagePaladinPriestRogueShamanWarlockWarriorDruidHunterMagePaladinPriestRogueShamanWarlockWarriorAllianceHordeTotal
EU Dunemaul11.4%11.9%14%10.9%11.2%12.7%4.1%10.5%13.3%9.3%10.9%13.8%5.5%11.4%12.6%11.5%11.2%13.8%47%53%5,749
EU Durotan9.7%14.7%14.6%10.6%11.8%9.5%4.3%10.1%14.8%8.1%13.7%14.6%4.9%11.5%9.9%11.5%10.3%15.6%65.8%34.2%3,064
EU Earthen Ring11%11.9%14%11.5%12.1%11.2%3.3%10%15%9.6%11.3%12.6%5.4%11.5%11.1%12%12%14.5%62.6%37.4%4,383
EU Echsenkessel10.3%12.8%13.7%11%11.7%9%7.9%8.1%15.5