This can occur when recording with any codec other than TSCC and choosing a dimension that is not a multiple of 4. As a workaround, in Camtasia Recorder click on Tools > Options > Program tab and check the box for "Force region to multiple of 4".
This error can also occur when using the 3ivx codec to record in Camtasia. Currently the above workaround does not resolve the issue specifically with the 3ivx codec. This is a known issue that will be resolved in a future release.