The most likely cause for Recorder to hang or become unresponsive is an outdated driver. Usually this would be either the sound card or camera driver because Camtasia Recorder connects to these devices through the drivers upon launch. Please check with the manufacturer of your soundcard and/or webcam to download and install the most current drivers.
Additionally, if there is more than one audio device connected to the computer, you may need to disconnect the additional devices to prevent conflicts that can also cause Recorder to hang or become unresponsive.
Another good way to trouble shoot this issue is to turn audio off and then try a recording. Also make sure "mouse click sounds" are off under the Effects > Sounds menu. If turning audio off resolves the issue it is a good indication that your sound driver may be out of date.