Yep - This is exactly what we are using right now. It is still a bit slower compared to having it pre-installed on something like a persistent Jenkins instance, but it's such a marginal difference that we will probably overlook it assuming that new caching features roll out eventually.
If you set a specific version, it can use a cached image and is a lot faster than a complete install.
Also have a look at their other `setup-*` actions for similar installations of Node, Python, Go...