The biggest issue is that GPUs can do DMA to read and write directly in main RAM. This can be exploited to overwrite arbitrary kernel data structures, for example.