"Here from HackerNews? This was originally posted several months ago. Check back in two weeks for an updated benchmark including newer versions of Hive, Impala, and Shark."
Yup, the level of interest is becoming clear. :) We'll work on getting this published in some form. Keep an eye on our blog, or e-mail us at contact@scalyr.com and ask to be notified when the code is available.
>"That plays well to our market development program where 1,200 big company management teams are coming through our office every year -- we ask them what they think about new ideas and they tell us."
How about asking BigCos their top pain points once in a while? That would be a 1200x treasure for current/future entrepreneurs.
Yep, we do, and we set up many opportunities for our entrepreneurs to do that themselves. (I just got back from our annual CIO/CMO conference where there was a lot of exactly that.)
>Running an all layer2 network requires devices that have large enough CAM tables to support all connected devices, many vendors newer full line rate cards were coming out with smaller CAM tables, as such layer2 simply wasn't an option in some cases.
Does this imply that mac forwarding tables(on switch) and arp cache(on hosts) need to have entries only for their immediate neighbours ?
Curious to know how much modified the host network stack is. Also, how do you provision a new server with the right IP ? Is this mechanism in L2 ?
>There are many other reasons that layer2 wasn't a good choice for us, and that layer3 makes a lot of sense. I'd be happy to discuss more of these as well.
I am sure people would find that very useful. Thanks for the excellent writeup !
The last-hop switches facing the servers are layer 3 in the direction of the aggregation switches and layer 2 in the direction of the servers so ARP is handled normally and DHCP can be handled with standard DHCP-helper mechanisms. This allows the host networking stack to be vanilla (if you want it to be).
On a serious note, take a look at cumulus networks[1]
who claim to solve the hw/sw disaggregation problem. They have a linux OS distro which can run on h/w of multiple vendors ( not the popular ones like cisco/jnpr/hp/brcd etc.. since those are closed platforms).
I had taken a course designed by him at stanford: object oriented programming from a modeling and simulation perspective. I also have the lecture notes for the follow-up course. It was mentioned that the course was based on his real world experience doing c++ programming in academia and startup that he sold to Cisco. It was very clear from the beginning that the course was radically different from any programming course I had seen. Highly opinionated in almost every page. There was rarely anything that he repeated from any standard c++ books. I enjoyed the course, though only about 5% still remains with me. I have worked in teams using c++ extensively to manage $Billion+ products, and can attest that the course definitely had some value. Also, I had interviewed with a C++ team in Cisco 4K catalyst group and they mentioned they were still building on the c++ platform laid by Cheriton's group ( a decade later).
Talking about any disadvantages, I can say that you need someone of Cheriton's ability to really guide you during the initial build-up of the project. His ideas of having the compiler do a lot of checking to avoid bugs in the code later are really cool. But the learning curve is pretty steep. Newer paradigms (since the course was originally designed) like STL, BOOST, and tips in Effective C++/STL etc can replace some of the concepts he espouses. But he is pretty clear that some of them are really inferior. It might be true for some cases, but I have usually used them in production just fine..
"Here from HackerNews? This was originally posted several months ago. Check back in two weeks for an updated benchmark including newer versions of Hive, Impala, and Shark."