This error indicates that the video codec selected is not installed on your system or is not available for use.
To resolve this error in Camtasia, open Camtasia Recorder click Tools > Options > Video tab. Check Manual and then click on Video Compression. Make sure to select the TechSmith Screen Capture Codec from the drop down list. If that is not available please download and install the TSCC codec from here.