"What we need to build is a gamified quick truther thingy that all our 200 employees in main office can log in during their lunch break and rate a few items."
Lunch break? Or you could just, I don't know... go with me here, I know it sounds crazy at first..., pay them for providing this value to your company.
You're already paying them. Simply tell them to do this during any downtime they might have. Obviously don't encroach on their lunch/home time, though.
Or, strange concept, the employees benefit from their efforts to improve the company, and idea of working during lunch on a trival task makes sense if they don't have anything else worth doing.
And the break is worth doing. Do you want your highly paid managers, engineers, salespeople, etc. properly recuperating? To better tackle the nontrivial tasks you pay the big bucks to tackle?
Or do you want really really expensive mechanical turks?
"In the loop" is a reference to the OODA loop, which is the decision cycle of observe, orient, decide, and act, developed by military strategist and United States Air Force Colonel John Boyd. "In the loop" has a specific tactical meaning, which is that you're able to do a full OODA loop cycle faster than a comparable AI/man system which gives a strategic advantage and maybe used against oppositional forces in a variety of ways; most interesting is applying it to PsychOps.
"Man in the loop" reference can be found in use by the US AirForce going back to the 80s for weapon systems design; specifically between fighter jet systems and pilots. I've heard that US AirForce used brain implants as early as the 90s in fighter pilots.
Here's an example of the US Army reads soldier's brain waves to speed up image analysis:
Yes, "in the loop" does date back further, but as used to reference human/systems integration earliest reference I was able to find was the 80s which was the same time that Boydian like systems started popping up.
Not sure if MT works when some specific knowledge is needed (I think their employees know or have quick ways to find out the correct matching dress given a photo)
Applications of human-in-the-loop systems like this are exactly why we built Scale API (www.scaleapi.com, YC S16).
Unfortunately, Mechanical Turk is often a bad choice these days for these types of applications—it's extremely difficult to guarantee quality, not to mention hard to use. Using employees also works well to a point (we did some of this at companies I've worked at too), but it doesn't scale, and should be done sparingly.
Great application of humans in the loop—this is the type of thing that really excites me. We're trending towards more and more human-augmented software, and it opens up so many possibilities for what software can do!
Does anyone know any good tools to make that sort of page quickly in Python? I mean, I could cobble together a web app, but I feel like dataset annotation is so common that there should be something out there already.
You're right, though I do need a way to interact with it programatically from python, since that's what I prefer to write my scraping & munging code in.
It would probably be fine if it was a solution that had an API that was easily callable from Python without a silly amount of overhead, specifically being able to dump data into a named dataset is what I want.
Python would also be nice since it would be easier for me to customize; though I could deal with Java or Node. I've avoided Ruby for this many years and would prefer to keep it that way.
Lunch break? Or you could just, I don't know... go with me here, I know it sounds crazy at first..., pay them for providing this value to your company.