What are the limits of analytical chemistry? As I understand it, the branch of chemistry dealing with identifying substances present in a sample is called "analytical chemistry". This branch looks vast and complicated, with lots and lots of individual techniques applicable in specific circumstances. I'm hardly even a beginner in chemistry, so it's very difficult to get a good grasp of what can be done with current technology.

So, in general, how well can one identify the substances in a completely unknown sample? Does one need to have at least a vague idea of what's in it?

Differently put, let's say I give you a container with a mixture of ten randomly chosen chemicals from the entire CAS Registry, and suppose they didn't react with each other. How difficult would it be to identify all ten of them?
The trouble with your question is the use of the phrase in general. Most analytical chemistry is directed towards specific goals and isn't about trying to find what is in a sample in general.

This means that the range of techniques applied are highly dependent on the problem you want to solve. If you want to detect, for example, pesticide residues in foodstuffs you probably already know the list of things you want to detect and the range of techniques appropriate to identifying them. If you seek the active ingredient of a herbal remedy, a different range of techniques apply. Sometimes you want to verify that a pure substance you have made is what you think it is, this involves a different range of techniques again.

One common thread in much analytical chemistry is separation: if you seek to identify something in a mixture you want to separate the mixture into its different components. So a variety of types of chromatography may be used in the bulk or on a micro scale. Identifying very small amounts of contaminants in food (e.g. pesticides) probably involves some form of extraction into a solvent followed by separation using analytical chromatography (gas or high pressure liquid chromatography (HPLC)). Where the range of compounds is not known in advance this can be supplemented by mass spectroscopy (the chromatography separates the components, the mass spectrograph helps identify the fragments of the molecules in each component).

If you don't know what you are looking for in advance, it isn't easy to elect the right separation technique and a great deal of experimentation is likely to be required. 

Another consideration is that some things are much easier to detect than others. Chromatographic and detection tools like the electron capture detector can identify organohalides to below parts per trillion levels. Atomic emission spectroscopy can identify some metals easily and at low levels. Other techniques can help identify structure; NMR is (sometimes) good at identifying structures, but usually needs pure compounds in relatively large quantities. X-ray techniques can be useful for minerals. Some compounds have very distinct features in their IR spectra or their UV absorption. And so on.

The bottom line is there are no general techniques. There are a wide range of tools to apply depending on the circumstances and the problem you are trying to solve. Almost everything starts with separation. Then, for simple problems, you may be able to apply a single known technique to find out what you want. For harder problems, a rage of tools may be required but the tools will be fairly specific to the problem.