Well at least now we have a better idea of what you have. Upgrades depend on what you want compared to how good it runs and looks now. And there are lots of opinions on this! Do you want the engine bay to look "stock" or does power have priority? If you are looking for more power do you want to use old or new technology. That sort of thing.
I'd start by ordering up catalogs from places that specialize in 60's engine perf and restoration parts, especially Pontiac-specific. Then start reading. You will find answers to a lot of your questions in the catalogs. I don't know who is out there these days, but Nunzi's, PAW and Jegs come to mind.
My opinions? My goal was never to build a street car into a race car but I could often get into the 13's anyway. I actually like the stock 60's intake manifold, a good Q-jet that is set up right works fine but my favorite was a Holley spreadbore double-pumper. Going too big on the carb is a common mistake. I think the torqer manifold is OK, I don't know anything about your carb. As long as the HEI works good it is fine, but watch out because heat can kill the module and leave you stranded. Dual exhaust is a must, headers can only help.
Assuming that your engine is in good shape and everything is working right then the cam has by far the biggest impact. The right cam is key to making it all work. Too "big" a cam is a common mistake, but with the manual trans it is not as critical. By the way it is just about impossible to tell what a cam is by looking at it, unless you can match up the numbers. Read up about cam design so that you understand what duration and lift means. Pontiacs like dual-pattern cams with more lift and duration on the exhaust. Changing a cam is not a job for a novice, but its actually pretty easy to do for someone with experience.
All that said, hopefully you will determine that your engine is already "right" and all you need to do is enjoy it!