This error can occur when using the Xvid or Divx codecs as your video compression in Camtasia Recorder. For Xvid you'll need to uncheck the "display encoding status" checkbox. For Divx you'll need to choose the 1080HD profile. Screenshots of both settings are below.
This can occur when recording with any codec other than TSCC and choosing a dimension that is not a multiple of 4. As a workaround, in Camtasia Recorder click on Tools > Options > Program tab and check the box for "Force region to multiple of 4".