(covers information from several alternate timelines)
Nibiru was an M-class planet, and the homeworld of the Nibirans, a primitive pre-industrial society. While the planet resembled tropical regions of Earth, it possessed red instead of green flora. It was also the home of an unknown species of docile quadruped mammals and large fish.
In the alternate reality, the USS Enterprise was sent to survey the planet in 2259, where the crew discovered a volcano was going to erupt and render all life on the planet extinct. Though the Prime Directive forbade interfering in the planet's natural history, Captain James T. Kirk decided to save Nibiru and its inhabitants.
To avoid revealing themselves to the natives, a disguised Kirk entered the Nibiran settlement and stole their sacred scroll, drawing them into a chase. McCoy appeared with a quadruped to ride, but Kirk accidentally stunned it, forcing both of them to flee on foot and dive into the sea. Meanwhile, Sulu and Uhura rappelled Spock in an environmental suit down into the volcano's magma chamber from a shuttlecraft to stop the eruption with a cold fusion device.
From the Enterprise's bridge, Kirk, McCoy, Uhura, and Sulu learned they would not be able to beam Spock out of the magma chamber due to interference from the volcano. Despite Spock's demands to leave him while the device detonated so as to avoid breaking the Prime Directive any further, Kirk opted to lift the Enterprise out of the sea and fly to the volcano to get in range. Spock was beamed out in time, and the detonation flash-froze the magma chamber and rendered the volcano inert.
Although the Nibirans saw the Enterprise and began worshiping the starship as a god, Kirk covered up the incident in his captain's log, prompting Spock to write an accurate report. Admiral Christopher Pike admonished Kirk on his return to Earth, accused him of "playing god", and informed him the Admiralty were taking the Enterprise from him and sending him back to Starfleet Academy. (Star Trek Into Darkness)
Creating the planet
Production Designer Scott Chambliss noted that everyone involved in Star Trek Into Darkness wanted the film to begin on a tropical planet which felt special and different from Earth.  "The guys had written it as a beautiful island," remembered Chambliss, "and it sounded like Fiji or somewhere like that, quite gorgeous and lush. The one thing that I immediately didn't want to do was go to some place that looked like a great vacation spot and shoot this." (Star Trek Magazine issue 172, p. 67) Prop designer John Eaves observed that, as the film developed and subsequent revisions were made to its script, the "village shrunk in size drastically to fit budget constraints." 
Bearing in mind that Nibiru was to seem particularly alien, Scott Chambliss thought, "Well, we don't want it to be green."  In retrospect, he continued, "So what are tropical environments other than green? They are lush and green, green, green, green. I thought, 'Let's flip that and go red, red, red, red. Let all the vegetation be red.' That was inspired by one of my favorite plants I see in Hawaii, which is a kind of bamboo." (Star Trek Magazine issue 172, p. 67) This impetus was lipstick bamboo, portions of whose trunks are typically magenta.
Conceptual artwork featuring the red vegetation then started to be created. Scott Chambliss experimented with the idea of using the color via digital artwork. Said Chambliss, "I took a photograph of timber bamboo, a beautiful jungle of it, and played with it in Photoshop and turned it really deep ruby red."  He further recalled, "I just started with an assistant messing around in Photoshop, messing around with bamboo forests and inverting the colors so that they become this lush ruby red. In flipping some of the other tones around, they became this mossy golden color. Suddenly, it felt like a tropical world, but was utterly different than what we've seen." (Star Trek Magazine issue 172, p. 67) Chambliss deemed the result "beautiful."  Concept images showing extensive views of the planet, featuring the natural red materials, can be viewed here and here. It was decided that Industrial Light & Magic would develop digital treatments to turn foliage into shades of red. (Cinefex, No. 134, p. 77)
The filmmakers experimented with the option of shooting the Nibiru scenes on location. Multiple locations were considered. (Star Trek Magazine issue 172, p. 48; Cinefex, No. 134, p. 77) Several backlots were also viewed. (Cinefex, No. 134, p. 77)
One site scouted by Director J.J. Abrams was Hawaii, but he opted not to film the sequence there, as color grading the plants did not produce satisfactory results. "In a funny way, I wanted to make it work," conceded Roger Guyett. "I was kind of intrigued by it. The problem is of course, the jungle has many, many shades of green and it’s being lit by natural light which is not like a green highlight particularly, and it’s exterior, it’s very hot. We tried – we did a whole bunch of tests and thought there would be some mileage in there. What could work about this? Well, I felt, if you did that it would feel weird. Maybe that’s OK, because you’re in a Star Trek world. The thing is, it didn’t just look weird, it also looked very kind of false. Kind of an electronic thing and not natural." 
The idea of moving the production company to a fairly distant location, incurring substantial logistical cost, was replaced by a more local approach. "But there isn't any real jungle in Southern California," related J.J. Abrams, "and the more 'foresty' it became, the less interested I was in trying to turn pine needles red."
J.J. Abrams suggested building a small section of red jungle as a practical set, to be created at Raleigh Studios' Playa Vista lot. This plan was inspired by Abrams having been amazed by noticing, while working on Lost, how quickly plants obscure the distance. "My gut feeling was," he recalled, "we wouldn't need much foliage, and we could put up a bluescreen behind that to extend it, if we needed." (Cinefex, No. 134, p. 77) Regarding the red-colored vegetation that had been decided on, Roger Guyett concluded, "At the end of the day, we realized that to really successfully do something like that, we had to build part of the planet."
However, budget concerns were raised over the construction of the set. "We ended up building a very small piece of forest, literally enough for them [Kirk and McCoy] to get up to speed and run," said Roger Guyett. (Star Trek Magazine issue 172, p. 48) The set was outdoors, contained foliage props as well as part of the volcano, and was constructed in Marina del Rey in California.  Noted Guyett, "We even had a little cliff edge that was probably three feet high." This set piece was used for the scene showing Kirk and McCoy, as portrayed by Chris Pine and Karl Urban respectively, diving into the Nibiran water. (Star Trek Magazine issue 172, p. 48) The area of the set which included the alien-looking foliage was a 75-by-100-foot patch of forest, designed by Scott Chambliss and with vegetation comprised of bamboo and red leaves. (Cinefex, No. 134, p. 77) Jeremy Raymond, who portrayed the lead Nibiran, commented, "Walking onto the Nibiru set for the first time was as close as I’ve ever come to being actually transported to an alien world. Everything was so wonderfully constructed, with a keen eye to detail, that you would walk up to any part of the set and scrutinize it closely and it would still seem real." 
Due to the smallness of the set, much of the views of Nibiru had to be rendered digitally by the film's visual effects team. "Just about every shot of the forest, everything past 25 feet, is us," explained Roger Guyett. "It was a huge endeavor." (Star Trek Magazine issue 172, p. 48) "That being said," clarified Jeremy Raymond, "there was far less green screen on Star Trek than most people might think."  Using CGI, ILM modeled a Nibiran temple and rendered volcanic terrain based on US Geological Survey data of Mount St. Helens. The visual effects group also generated jungle extensions, creating an entirely digital jungle for the film's opening shot, which descends through the clouds above Nibiru. "We spent a couple of days photographing trees in different lighting scenarios," remembered Barry Williams, "That enabled us to place trees behind the set wherever we needed, and we used those textures as reference for modeling. We had 20 types of trees and variations on each of those, making 60 unique trees. We randomized scales and rotations, and that created a really nice organic look." (Cinefex, No. 134, p. 77) The digital parts of the jungle were rendered in V-Ray. 
The scene depicting Kirk and McCoy jumping off a cliff into water below additionally required much visual effects. Roger Guyett explained, "Chris Pine and Karl Urban are literally jumping three feet [from the artificial cliff edge] onto a mat. Everything you see over the edge is basically CG." (Star Trek Magazine issue 172, p. 48) As reference, the film's production crew filmed a down-angle of a cliff on the Californian coast. ILM then digitally manipulated the footage to exaggerate distance to the ocean. (Cinefex, No. 134, p. 77)
When quizzed for an in-universe reason the Enterprise was hidden underwater, Roberto Orci said a "line of sight [was] necessary given [the] unstable and shifting magnetic field of [the] super volcano on alien planet. That's why no beaming. Gotta physically get back to the ship."  Alex Kurtzman came up with the image of the ship rising out of the sea, but they acknowledged fans would find it questionable so they wrote Scotty's line questioning Kirk's decision. 
The docile animal Kirk shoots is a disguised version of the Drakoulias from Star Trek. "We gave it a shave and a spray tan," remembered Paul Kavanagh, "and we re-rigged it with some new techniques we've developed to handle quadropeds, which gave us better animation controls." (Cinefex, No. 134, p. 77) According to the website FXGuide, the filmmakers dubbed the Nibiran version of the animal "Niborilla".  However, according to Cinefex (No. 134, p. 77), this name is spelt "Nibirilla".
A deleted scene would have stated that Nibiru was newly discovered and the Enterprise mission was the one establishing that there was intelligent life there.