I've not played DotA, but it's a bad idea to include a large percentage of classes that would never be chosen by competitive players. 10% is a very low number and is much lower than what you'll see in well balanced fighting games. It really makes things hard for new players, making the amount of information they have to possess before they can start playing the actual game a lot higher. Ideally any choice that you make before a match starts should not affect the likelihood that you will win the match, only how much fun you will have.I'm not sure what you mean by "painstakingly difficult to balance because there are so few". The more classes there are, the harder it is to balance them all.
I draw a lot of my design and game-playing principles from David Sirlin. Not only his Playing to Win articles, but also his Multiplayer Game sections.
DotA has over 90 heroes. I would say about 20 (22%) of them are viable for competitive play. My favorite fighting game of all time is Naruto GNT 4 and only 9 out of 25 (36%) characters are viable competitively. A more Western SSB:Melee has 5 characters out of 26 (19%) at the top tier.
DotA isn't a very newbie friendly game, partially because of the reason you described with tiered heroes. That doesn't mean DotA isn't fun or isn't competitive--quite the contrary.
If DotA had fewer heroes, say 10, then all of them need to be viable. That is much more difficult than making 50 heroes and letting the players figure out which ones are the best. The designer obviously needs to make sure that none are dominant and none are dominated, and over time he can try to pull their differences in power closer together. But with "tiered-balancing" the success of the game isn't dependent on the viability of every character.
You are quite right when you say that choices made before the match shouldn't affect your chances to win, which is why MMOs are very difficult to balance. Every class needs to be viable because players have invested resources into the character and are locked in to that choice. Compare this to DotA or a fighting game where players can see what character their opponent (or team) has chosen and react appropriately. I love drafting in DotA and MTG for this very reason. There is a game that takes place before the game starts.
Many moons ago when Blizzard announced that switching gear in Arena was going to be disallowed, I was heartbroken. I have never played Arena, but here was a counter to team composition, and they completely removed it! They had valid reasons for doing that, but it removed an entire level of depth to Arena PvP. Players were not locked into their stat choices and could react appropriately to the opposition.
So for the DotA MMO, we can have hundreds of abilities and scores of ability groups. Allow the player to react to his opposition (either through the meta-game or with a pre-match draft system). Abilities and builds will fall into tiers (WoW talent builds are tiered, by the way). As long as no one combination is dominant, and players have counters to abilities, builds, and team compositions, then the game will be fun and competitive.