(covers information from several alternate timelines)
Transwarp beaming was a transwarp theory originally postulated by Starfleet officer Montgomery Scott, whereby it was possible to beam an object or person from one star system to another, or even to a starship traveling at high warp.
In the alternate reality created by Nero's temporal incursion, Scott had a debate with an instructor of his, who maintained that transwarp beaming was impossible. In an attempt to prove his instructor wrong, Scott tested his theory on Admiral Archer's prized beagle, although the experiment was a failure and the dog was lost.
In 2258, in the same alternate reality, while stationed on Delta Vega, Scott encountered Spock (who had traveled back in time from the year 2387) and James T. Kirk. There, Spock provided Scott with the equation for successfully achieving transwarp beaming, which, by the time Spock had traveled back in time, the Montgomery Scott of his timeline had already perfected. Upon viewing the equation, Scott noted that it had never occurred to him to think of "space as the thing that was moving."
With this formula, Scott was able to beam himself and Kirk from Delta Vega to the USS Enterprise, which was traveling at high warp towards the Laurentian system. However, Scott was transported into a water pipe leading into a turbine, and would have been killed had Kirk not opened the release valve. Later, Scott was able to use this technique again, to beam Kirk and Spock's alternate reality-counterpart from the Enterprise, which was in orbit of Titan, to the Narada, which was orbiting Earth. (Star Trek)
Starfleet confiscated Scott's formula for transwarp beaming. In 2259, John Harrison, a Section 31 agent gone rogue, used the formula and a portable transwarp beaming device to flee from his attack on Starfleet Headquarters in San Francisco by beaming all the way to Qo'noS, the capital of the Klingon Empire. Scott expressed great displeasure when he learned the formula had been abused by Harrison. (Star Trek Into Darkness)
|aperture • beaming • chamber • coil • computer • conduit • drive • factor • hub • network • probe • signal • signature • space • theory • threshold|
In the script of Star Trek, both spellings: trans-warp and transwarp were used for this type of beaming. 
Screenwriter of Star Trek Into Darkness, Roberto Orci suggested on Twitter that USS Vengeance had a countermeasure against transwarp beaming, hence Harrison fled to Qo'noS instead of commandeering the Vengeance directly. 
According to the novelization of Star Trek, in the alternate reality, Scotty had been toying with the notion of transwarp beaming for years. Spock revealed that the only thing missing from the theory was the correct field equation for the continuous recrystallization of dilithium while transport was in progress. This allowed the steady usage of power during transport. It was the discovery made by Scott in the future of the prime reality. After seeing Spock's complete theory, Scotty realized he could not resolve the central string as it had never occurred to him to look at the event from the perspective of the one who was beamed. Spock however stated it in fact did occur to Scott when he added the adjunct equations related to dilithium. The fact that a transport pad was not present at the receiving end made beaming on target possible only with a 4 meter margin of error, as the destination coordinates were completely theoretical. In the novelization, Admiral Archer's beagle, after having traversed in subspace for over 6 months, was intercepted by the Enterprise while it was warping out of Earth orbit for the first mission under the command of Captain Kirk.
In the comic book The Truth About Tribbles, Part 1, set in 2259, Scotty successfully experiments with extreme long-range transwarp beaming. He sends his pet tribble back to Earth from the Iota Geminorum system by bouncing the beam off a couple relays along the way. In The Truth About Tribbles, Part 2, Scotty explains the intention was to allow Starfleet to beam living things between Earth and the Fleet stationed in the outer quadrants. It was also to test if it was possible to retrieve Admiral Archer's beagle back from space. At the end of the story Scotty accidentally presses a button on the transporter console and the beagle materializes onto the transporter pad.
According to the novelization of Star Trek Into Darkness, beaming to Qo'noS for Khan was a more complicated plot. The small portable transwarp beaming device on the jumpship only had enough power to beam Khan to an automated cargo station on Earth's orbit. From there he accessed a heavy-load transporter to beam onto an unmanned vessel in orbit of Luna. Khan had equipped the ship with another unauthorized transwarp device wired into the empty ship’s engine. Utilizing the entire energy output of the engine for a single massive burst, he could have beamed anywhere in our galactic region. Transporting to Qo'noS completely burned out the device, so no one was able to follow him using it.