The AEDU System is a slang term created to refer to the unique methodology of handling classes created for Dungeons & Dragons 4th Edition. Created to end the problem of the Linear Warriors, Quadratic Wizards approach seen with earlier editions of Dungeons & Dragons, AEDU forsakes the traditional split of martial and magical combat across differing subsystems, as well as the traditional mechanic of classes splitting different features up over assorted levels. Instead, each class receives an initial array of core class features at first level and relies on a "Powers" universal system, where each distinct power draws from the same pool of basic mechanical aspects (forcing movement, ongoing damage, etc), but applies those mechanics in a unique way.
For example, the Fighter and the Swordmage are both Defender type classes. However, Fighter powers tend to focus on hitting a single target in melee range and adding debuffs, such as stunning or forced movement, and many come with the unique trait of "Reliable", which means they aren't expended if you miss. Swordmage powers, by comparison, often hit small groups of targets, many of them have a short to melee range, frequently apply elemental damage, and often have more "magical" rider-effects, such as teleportation or ongoing damage.
The name is an Acronym for "At-Will, Encounter, Daily, Utility", referencing the way that powers were divided by the system; attacks that could be used at-will, attacks that require a short rest of 5 minutes before they can be used again, attacks that require an extended rest of 6 hours before they can be used again, and non-offensive powers which can be used either at will, 1/encounter or 1/day, depending on the power in question.
The AEDU System was, hands down, THE most controversial mechanic that 4th edition introduced. The Essentials classes were an attempt to find a mitigating point between the classic style and the AEDU System, but are usually considered to have failed miserably, since at best they tend to break down past a certain level point (when the idea that ever level from 1st to the 30th should be functional and enjoyable was one of 4e's core ideas), and at worst they were an absolute mess that just completely failed to work (like the notorious Vampire class, a Striker that consumed its own Healing Surges for power, making it a ridiculously flimsy Glass Cannon.
Interestingly the Binder of 3.5 incorporated a mixture of of at will, per encounter (via long cooldown) and per day abilities (no accident, WotC admitted a lot of late 3.5 content was prototyping 4E ideas). Despite resembling this system, it is actually one of the most popular non-core classes in 3.5. Even beyond that, the idea of class abilities having specified uses based on periods of time goes all the way back to Advanced Dungeons & Dragons.
So, what happened to make the AEDU system so controversial? In a nutshell, there are three major arguments against it:
Firstly, there's the "magicking up martials" argument. Basically, martial exploits in 4e are seen as "too anime" for the classes that were always considered the "realistic" ones. This is usually rebutted with the counteraguments that it's unfair (and unrealistic) to expect martials to be restricted to just "I swing my sword" when the next guy in the party can do shit like summon armies of monsters or throw fireballs, and/or that martial exploits aren't really "magical", they just make it simple to access stunts that, in past editions, were gated behind feats/non-weapon proficiencies and complicated rule subsystems (see: Grappling in 3e).
Secondly, there's the "use caps on martial powers are stupid" argument. Under this stance, whilst it's fine for magical type classes to have encounter/daily caps on their powers, it supposedly makes no sense that a fighter can't just use the same power over and over again. It bears mentioning that, having foreseen this, 4e rebuts this argument by pointing out that martial exploits require a precise combination of timing, opportunity and physical exertion in-universe, and out of universe are representing "cinematic moments" - they're the fantasy equivalent of that moment in an action movie when our hero grabs a mook and then charges forward, using his victim as a human shield as he takes lethal potshots at the other mooks along the way; he doesn't do that in every single fight scene because it'd get boring. This is also why Encounter powers recharge in 5 minutes, compared to 5e's ridiculous "recharge in 1 hour" "encounter" powers. Still, this is considered the most reasonable argument, and with 4e's psionics system for comparison, even 4e fans have been known to concede that maybe Martials might have been better off having an arsenal comprised solely of at-will, encounter and utility powers.
Finally, there's the "dumbing down wizards" argument. In a nutshell, this argument declares it's simply not fair that the spellcasters are not inherently superior to non-casters once they reach an arbitrary level - this argument is often cross-pollinated with the first argument here. This is perhaps the least respected argument here, with even 4e haters being known to roll their eyes up at what is obviously a butthurt caster-fag complaining about not being a special snowflake anymore. 4e's fans tend to particularly hate this argument, since the rituals system allowed for all of the "world-shaping magics" to continue, they just made it so that they never became the default option; there was finally a need to invest in lockpicking or diplomacy skills instead of just slapping together a relatively cheap wand of knock or charm person.
Though officially this system was retired with the progression to Dungeons & Dragons 5th Edition, the truth is that it secretly persists. Oh, the simple, easy-to-read universal mechanics are gone, and classes have mostly gone back to the AD&D paradigm of "just use a spell for it", but many racial or class powers in 5e are still capable of being used "once per short rest". This is basically an Encounter power, except 5e insanely redefined its "short rest" as 1 hour instead of 5 minutes, inherently gimping the usability of such powers - which is one of many reasons why the Dragonborn went from being one of the strongest races in 4e to one of the weakest in 5e.
A side aspect of the AEDU System was to change many "utilitarian" spells of the past, which had contributed to the phenomena of casters overshadowing martials - as the old saying goes "why bother with the thief's hide in shadows and pick lock attempts, both of which are skill based and so can fail you at the worst moment if the RNG God chooses, when a wizard can automatically solve those problems with Invisibility and Knock?" The Rituals mechanic was built around taking these spells out of the powers pool and making them harder to "game"; all rituals require skill checks and a certain amount of time and resources to perform, so casting "knock" went from being something any wizard of high enough level could do as a matter of course to something that was a legitimate method, but still not as quick or as cheap as just letting the rogue give it a shot.
Rituals actually expanded quite drastically as the edition progressed; free of needing to worry about the "how often will somebody cast this?" issue they'd faced when they were part and parcel of the spells, WotC got quite creative as time went on. Rituals became capable of all kinds of weird, magical effects; creating campfires that shrouded campsites in protective illusions, raising floating islands, summoning castles from across the world, or opening portals in time all became potential toys players could have. Plus, as ritual casting was locked behind a feat, it was finally possible to play a gish character who studied magic, and performed miraculous feats out of combat, but relied on sword and skill alone in battle. Heck, one could use rituals and restrict classes to martials for a high-action Sword & Sorcery campaign, especially with the variant healing rules seen in Unearthed Arcana.