To get the best video quality on YouTube you want to upload at 1280x720 or 1920x1080.
To achieve best possible end result image quality in your video, it is best to record at the same video dimensions at which you plan on producing. For example, if you want a 720p HD video on YouTube you need to record your screen at 1280x720. If you want a 1080p HD video on YouTube you'll want to record at 1920x1080. Here is where you set your recording dimensions.
We also want to make sure the editing dimensions are set to the recording dimensions. Here is where we set the editing dimensions In Camtasia 9:
Here is where we set the editing dimensions In Camtasia Studio 8:
It is best to produce using the YouTube preset shown here.
Once your video is uploaded YouTube make sure you're watching it in HD by selecting the following option.