Some of the connection problems discussed above result in mechanisms (structures that have moving parts), but there are other ways to create mechanisms in a VisualAnalysis model. Some of these are obvious once you "see" the problem. Others are more subtle mathematical instabilities that would not happen in a "real" structure, but cause problems in the matrix analysis.
You can change the structure type in Project Manager to a Plane Frame, Otherwise, to stabilize a planar model in a Space Frame structure type, simply Shift+Click on a node and fix all of them in DZ (or your out-of-plane direction).
When trying to analyze models in 3D it is common to get instability because of either (a) forgetting about the out-of-plane direction, (b) forgetting about rotations, or (c) having simply too many end-releases and/or not enough support. You can debug your model in this fashion:
VisualAnalysis can help you locate mechanisms in your model by performing a modal analysis. If a modal result is the only result available, it means your structure was unstable. You can animate the result view to get an idea of how or why the model is not stable.