I have a very bottom-up approach to game design. I like to think in terms of fundamental, atomic, core mechanics, and build them up in layers to produce a cohesive system.
After making those minigames for my cancelled Facebook MMORPG, I am fairly confident that there are 2 types of "core" mechanics: mathematical and pattern matching. My evidence is purely anecdotal, but after observing players for several months, I believe that pattern matching is a superior mechanic for video gaming. I attribute this to the amazing subconscious pattern matching and recognition facilities in the human brain.
Pattern matching as a mechanic is usually coupled with input device mastery. Two prevalent forms of PM are shooters (click the button when a target lines up with the reticule) and rhythm games (click the buttons in sync with an auditory cue). Both these games involve some sort of prediction, e.g. leading targets, interpolating target location, and maintaining musical beat/rhythm.
To help understand the difference between pattern matching and straight up input device mastery, think of any implementation of Whack-a-mole (any WoW addon for a support role will work). The mole surfaces at random locations (debuffs are placed on random raid members), and the player hits the mole (clicks the grid to decurse the target). There is no prediction, pattern, rhyme or reason to where the moles will appear. The player simply invokes muscle memory to move the mallet to a location and swings her finger.
If we examine the shooter mechanic stack a bit more, the very next layer on top of pattern matching in most shooters is resource management (which is a mathematical mechanic). Ammo, weapon clips, reload time, and health--these are all resources that are managed by the player.
I thought this would be an interesting template for RPG combat, and thus I arrived at "ability clips". The entire system would be mirrored from the standard shooter weapon system: the player primes spells in much the same way that weapons are reloaded; when the player depletes a clip, they must reload using a reservoir of mana; each ability has its own reload time, clip size, mana cost, etc.. Players can only have 1 active ability in the very same way that players only have 1 active weapon. The game becomes a third-person shooter with guns that shoot health buffs and movement snares.
There are already a few games that play with functional abilities on weapons. Global Agenda was one. Many of the devices in GA were some sort of non-damaging spell, e.g. speed boost, restore health, stun buildings, and forcefields. Its combat was great; it could have been an amazing example of what I advocated above if Hi-Rez decided to pursue the shooter side rather than muddy the gameplay with hollow additions like persistence, progression, and gearing.
Team Fortress 2 is also an unlikely example of an ability shooter. The interface for swapping weapons isn't as robust and clean as GA's, but many of the newer items added to the game perform a non-damaging ability. The Demoman has a shield which reduces explosive damage and gives them a Charge ability. The Sniper can equip "jar based karate" (it is a jar of pee), and toss it on enemy players to increase their incoming damage by 35%. It also reveals cloaked Spies and puts out fires. Spies have several items which change their cloaking behavior. Heavies can restore health with Sandvich.
Of course these are PvP games. Could an ability shooter work against dumb computers? Players in GA ran PvE missions for the phat lewtz, so I am not entirely sure that those missions without progression rewards are fun. All the mobs did was shoot (from what I remember). If they had a wider range of abilities, maybe it would be a bit more interesting.
Combat as an ability rotation is dull and a precursor to grindful play. The design goal should be to provide the player with interesting decisions, and those require interesting situations. The game must constantly test the player's knowledge and demand that they react, not simply act.
So what other sorts of pattern matching cores could be used to build a combat system? It does not necessarily have to be real-time, but it should have the potential to place players in interesting situations against AI. Some other constraints to think about involve porting the system to an MMORPG, namely how much volition does the player have if set in an open world (player state can't be reset easily; what happens if the player engages more than 1 target).