Definitely get "A Man on the Moon" by Andrew Chaikin. Then go watch the HBO series based on this book, "From the Earth to the Moon." Both are outstanding.
"The Right Stuff" is also an outstanding read, but is more "stylized" history than it is fact telling.
A really terrific book about Apollo is "Apollo: The Race to the Moon" by Charles Murray and Catherine Bly Cox.
There is a terrific documentary that Nova (the PBS show) did called "To the Moon" that you can rent on DVD.
There's also a terrific series of DVD's put out by Spacecraft Films that focuses on different aspects of the early space program. These are not documentaries, but the actual raw footage of what happened and how it happened. Some people consider these kind of dry, but you can gauge your status as a "true space buff" by your interest in these. I think they're great. Some of the titles are, "Apollo 11," "Apollo 8," "Gemini," "The Mighty Saturns," and more are on the way.