There are good programmers in the enterprise (meaning, say, investment banks or large corporations or governments) but they generally fall, ambition-wise, into one of three categories:
(1) those who want to become managers or software architects (or, in finance, quants and traders) and will define and oversee work but delegate the dirty bits. This would be fixable (they could oversee a team of mediocrities, who'd be grateful just to be employed) but they generally don't have the patience for that.
(2) those who want to do highly-theoretical R&D work that doesn't necessarily solve any immediate problems of the business.
(3) those who have a specialty (say, deep neural networks) and want to be in the umbrella of a large organization that can protect it. Pull them out of their specialties and they'll try to leave.
In other words, these supposedly stodgy non-technology companies do, contrary to stereotype, have good programmers (I've worked in a few) but the ones who are at all decent have career strategies that they expect to be able to implement in their full-time work. They won't work on "just anything" and if you stick them with the random muck that comes from the line of business or zealous "product people", they'll either leave or slack in order to learn new skills on the clock, and the project will be done poorly or even abandoned mid-flight.
The appeal of the $3000-per-day consultant is that he'll work on what he's told to do and he doesn't expect you to consider his career needs. He's not going to do a shitty job and leave after 6 months because the people allocating the work don't care about his career; that's what the money (4-6x typical salary) is for. He gets his education and career advancement on his own time and dime (but earns a premium to account for his unpaid work). And while he might not be a great (2.0+) programmer, he's better than anyone in-house who could actually be assigned a bad project without political friction or high departure risk.
The average Bay Area startup programmer, like the average software consultant, isn't great; but he's far better than the corporate serfs who get sloshed around on the worst projects. He might be 1.4-1.5; so Goldman's R&D engineers and it top quant-coders will be better, but he's a relative colossus compared to the in-house peons (0.7-1.2) in back-office IT who'd get assigned to grody projects based on internal processes.
(Of course, not all the work that consultants do is undesirable. You also have the specialists and those with elite levels of skill. My point is that a "mercenary" consultant will power through the ugly projects for the pot of gold, whereas in-house people expect investment in their careers.)
In other words: yes, Goldman can hire great people. But if you want to hire someone great to do a project where 99% of the work is mediocre (and the 1% is extremely careful and requires an expert) you want the $3000/day consultant because Goldman can't get anyone good who's in-house (and not getting a consulting salary) to do the work. One might ask: why don't they pay an internal person $3000 per day, as they would a consultant? The answer is that it'd have him out-earning his boss and they'd often end up promoting someone not on traditional definitions of readiness (increasing scope, leadership) but because he took on an icky project.
No, that's not what the money is for. The money compensates inventory, scheduling, and delivery risk. It does not compensate developers for working on mundane projects. If the developer in question gets a W2 paycheck, odds are they're not seeing anything like 4x what the in-house people are seeing.
And, while I do buy that all three of these developer archetypes exist in the real world, I do not buy that they are the reason that companies don't deploy talent aggressively to upgrade "support services". Rather, companies make straightforward buy-vs-build decisions based on whether projects are part of the focus of the business or not.
It does not compensate developers for working on mundane projects.
It does. That's not the sole reason why consultants make more, but that's a factor.
If you're a full-time employee, you expect your health insurance, HR, work supplies, 401k, office space, finding of work, and your career growth and promotion planning to be taken care of, and you're likely to leave if you're not getting it. If you're a consultant, you're taking on those responsibilities for yourself. That's a big part of why you charge a higher hourly rate. It's to include buying your own health insurance and having to manage your own career with no expectation that the people giving you work give a damn about your vector.
That's not to say that typical middle managers or companies actually care about the careers of most people under them, but they at least pretend to, and some actually do. It's part of the social contract that exists for an employee and not for a consultant. Of course, after being an employee for a while and seeing that part of the social contract ignored, many people decide that the job is too important not to do themselves and start managing their own careers... and some become consultants.
Most consulting engineers (in IT) work for body shops. The pay is poor, the work is boring, the hours long, the turnover high, and the output mediocre at best.
A well managed internal team could do the work at 1/3rd the cost but the work is rarely a core part of the business and no executive wants to deal with the internal headaches and risk associated with the work when it won't get them anywhere politically.
The "contracting" world is much more like what michaelochurch described. And to be fair there are a large number of individual contractors working at large firms for great daily rates (think $1000/day on the lower end) for which what he is saying is true.
You'd probably know more about that world. In your opinion, are these consultants more like corporate employees (in terms of demanding managerial investment in their careers, and slacking or leaving if they don't get it) or are they more like independent consultants (who'll do a nasty project if the hourly rate is high)?
I'm in that world right now. Most here are like corporate employees, with a few exceptions. In most cases people _will_ do a nasty project to help the company, because in the past the company has turned away the nasty projects even when they looked profitable, because they were more interested in slower growth of interesting work than quick money from boring work. When that boring work becomes necessary, people are prepared to do it on the assumption that there'll be interesting stuff around the corner. So, here at least, the trust flows both ways, and that's a great thing. (edit: since I'm replying to you Michael, it's probably worth stating that in your terminology I'm currently working in a "guild" environment, and that's rare enough that my experience is probably the exception not the norm.)
Roughly speaking, it represents the transition from an adder to a multiplier. A 1.0 programmer is competent at "adder" tasks (scripts, bug fixes, features) but not yet ready for infrastructural work. A 2.0 programmer is highly competent as a multiplier. It goes from 0.0 (complete beginner) to 3.0 (global multiplier) but most (probably 99%) professional software engineers are between 0.8 and 2.2, with a median around 1.1-1.2.
This isn't general enough to apply to computer scientists, nor do I intend it as an all-purpose ranking for software engineers. It's an approximate measure of a generalist's professional development. It's not really applicable to, say, machine learning experts.
The last sentence is a very important point in enterprise IT shops: Lack of self-confidence in line managers is a big disincentive to hiring smarter developers. They worry that the smart developer (who often shows up with a bit of an attitude) will overshadow the manager and our point out their weaknesses.
"...They worry that the smart developer (who often shows up with a bit of an attitude) will overshadow the manager and our point out their weaknesses."
The (4) and (5), if they're good-- both in terms of being skillful and having a strong work ethic-- will usually seek managerial roles or transition out of programming.
Contrary to what is often said about middle management being hell on earth, it's not worse (on average) than being on a team, just different. You don't have to be unambitious to want a job where you primarily evaluate others' work instead of doing the work.
i'd see myself as a (5) who enjoys interesting but low-stress programming. i'm happy to put in my 9-5 in a big company that can guarantee me a steady stream of interesting projects, and thus far my managers have had no complaints about either my skills or my work ethic, but i have zero desire to transition out of programming.
There are good programmers in the enterprise (meaning, say, investment banks or large corporations or governments) but they generally fall, ambition-wise, into one of three categories:
(1) those who want to become managers or software architects (or, in finance, quants and traders) and will define and oversee work but delegate the dirty bits. This would be fixable (they could oversee a team of mediocrities, who'd be grateful just to be employed) but they generally don't have the patience for that.
(2) those who want to do highly-theoretical R&D work that doesn't necessarily solve any immediate problems of the business.
(3) those who have a specialty (say, deep neural networks) and want to be in the umbrella of a large organization that can protect it. Pull them out of their specialties and they'll try to leave.
In other words, these supposedly stodgy non-technology companies do, contrary to stereotype, have good programmers (I've worked in a few) but the ones who are at all decent have career strategies that they expect to be able to implement in their full-time work. They won't work on "just anything" and if you stick them with the random muck that comes from the line of business or zealous "product people", they'll either leave or slack in order to learn new skills on the clock, and the project will be done poorly or even abandoned mid-flight.
The appeal of the $3000-per-day consultant is that he'll work on what he's told to do and he doesn't expect you to consider his career needs. He's not going to do a shitty job and leave after 6 months because the people allocating the work don't care about his career; that's what the money (4-6x typical salary) is for. He gets his education and career advancement on his own time and dime (but earns a premium to account for his unpaid work). And while he might not be a great (2.0+) programmer, he's better than anyone in-house who could actually be assigned a bad project without political friction or high departure risk.
The average Bay Area startup programmer, like the average software consultant, isn't great; but he's far better than the corporate serfs who get sloshed around on the worst projects. He might be 1.4-1.5; so Goldman's R&D engineers and it top quant-coders will be better, but he's a relative colossus compared to the in-house peons (0.7-1.2) in back-office IT who'd get assigned to grody projects based on internal processes.
(Of course, not all the work that consultants do is undesirable. You also have the specialists and those with elite levels of skill. My point is that a "mercenary" consultant will power through the ugly projects for the pot of gold, whereas in-house people expect investment in their careers.)
In other words: yes, Goldman can hire great people. But if you want to hire someone great to do a project where 99% of the work is mediocre (and the 1% is extremely careful and requires an expert) you want the $3000/day consultant because Goldman can't get anyone good who's in-house (and not getting a consulting salary) to do the work. One might ask: why don't they pay an internal person $3000 per day, as they would a consultant? The answer is that it'd have him out-earning his boss and they'd often end up promoting someone not on traditional definitions of readiness (increasing scope, leadership) but because he took on an icky project.