Camtasia Studio 6 and 7
When the resource allocation failed error message occurs as soon as you click record this is being caused by recording too large of an area. Try reducing the screen resolution of your monitor or try recording a region.
If the resource allocation error occurs when you stop the recording, this is a known issue in version 7.0.0 and 7.0.1. To resolve the issue upgrade to version 7.1 or later. You can download version 7.1 here.
If nothing happens after stopping a recording, this is a known issue with longer recordings in Camtasia Studio 7. This issue has been resolved in Camtasia Studio 8.0.0.