By default, a site template can only be 50 MB.
There is a site-scoped property, however, that defines this limit, which can be increased to 524288000 bytes, or 500 MB.
You can use Pshell or stsadm:
STSADM -o setproperty -pn max-template-document-size -pv 524288000
Two thoughts on this approach:
1. Be aware that you may need to increase some timeout values in various web config files in your templates
2. If your site is greater than 500 MB, consider exporting Libraries instead of the entire site collection using the content migration wizard, or powershell.