The understanding of the physics is not hard as you might think. To escape the gravitational field of the earth, you can use pretty much highschool physics to find out. (You would need to do E = mgh but take into account of the g decreasing as you go higher, with a bit of integration, you can find the exact energy required to escape earth's gravitational pull. i cant do it at the moment but i did it last year). As for swinging and picking projectory into the moon, that also can be done with again highschool physics.
As for technology, yes it was the most difficult part of the mission. But it was far from impossible. If you think about it, it was a rocket with a ship on top. What is so impossibly hard about that? Sure the rocket had to not explode and carry enough fuel to escape the gravity well of the earth, but then the rocket is ditched! The module then goes and lands, stays there and comes back to earth. I fail to see whats so technologically impossible about this, i mean it was 1965. Not the 1800s. By 1965, "modern" jets already exists. And the famous F14 tomcat (tom cruise top gun?) would have its first flight in 1970.