Nobody said anything about not having NTP in the loop!
Here's the deal: RPi<5 runs at the speed of the voltage coming in. The OS creates a fake software clock that assumes it's a perfect 5v, but in reality not only does voltage fluctuate, it drops over the length of a cable run. If you have, I dunno, 70 Pis connected, the 5v on the first one is very different than the 5v on the last one, no matter how frequently you ask NTP to broadcast.
The first thing we did was deliver 12V down the wire with a high quality power supply. Then we used UBEC buck converter modules (intended for RC quadcopters) to deliver power directly into the Pi via its pins.
However, even now you're just chasing physics. There's no way to send a command to all of the Pis to do something right now. Instead, we tell all of them to do something at a very precise moment in about three seconds. In our case, this meant writing a very low-level C program to listen for raw UDP messages and trigger the camera module at the hardware register level.
The only way to verify that it was working as well as we believed it did was to create a reference clock. We had a friend build a 20 LED timer that would count 10ms and 1ms increments.
We achieved < 1ms syncronization across 70 Pis in this manner.
We needed it to be small, low power, low heat, provide hardware level access to a camera module, be ethernet accessible. Oh, and under $100 including a fixed M-mount lens.
At the time, it didn't seem like rolling our own hardware would have been faster.
I would argue - given we succeeded at sub 1ms syncronization - that the Pi was close to the perfect tool.
It just would have been even more perfect if it had an RTC.
You already solved your issue, but still I'll mention it:
For my Pi's I buy a separate RTC (uses GPIO pins). There's various available, the DS3231 is the most expensive but also the best. If you buy it in bulk (70+) I suppose you could shave some money off. If you're OK with a cheaper one (less than half the price) which functions less well (but still does the job) then PCF8523. I got mine from a local Adafruit reseller.
FWIW, the cheapest RTC module we could find that had kernel support was about $31 at the time. Things have gotten a lot better and cheaper since then. Adafruit didn't launch that DS3231 until four years after we did this.
We couldn't justify adding ~$2500 to the bottom line, but you can be confident that we sure did argue about it.
Missed this comment when asking why you didn't add a $2 RTC, but given that you mention Adafruit I assume making your own hardware to stick onto a Pi wasn't an option? The DS3231 you mention is a chip with mainline support since 2009. Cheaper options like a DS1307 cost less than a $0.50 a piece when you buy 100 of them, and they've been supported in Linux since 2006.
I'd say 50/50 that I didn't know what I2C was in 2014.
Still, if we had gone and designed our own RTC daughterboard, it would have replaced the UBEC in our architecture diagram. We were buying those for $3-4 in 1000 unit quantities.
So, had you been on hand a decade ago, you might have won the argument although if I put on my board member hat, fabbing our own PCB sounds a lot scarier than buying boxes of buck converters.
It's hard to express how wild it is that they just pop right on and guarantee close to perfect voltage to each device.
Fair enough! I'm not sure I caught that the project was from 2014.
From your URL, the second part made me wonder if the statement is correct: "UBEC 5V 3A/5A/7A/15A BEC 2-12S Lipo Step-down Module External Power Supply Full Shielding Antijamming For FPV Airplane" would this actually work?
This is an armchair hindsight, but sounds like that needed a way to broadcast trigger GPIO interrupts, which are like JavaScript event handler for hardware.
You can attach a function to a pin, and CPU force-jump into a preprogrammed address for a handler routine, which leads into your code, then return(return(return())) back to where it came from. The trigger can be delivered by, say, a Xenon flash pointed at the ceiling and a light sensor, or a dedicated pair of wires from a master Arduino, or a intentional glitch in power line, a drone receiver per each pi all on a same channel, etc.
> fixed M-mount lens
I assume this is meant to say metric threaded C/CS mount for surveillance cameras, as M-mount is boutique full-frame Leica mount.
Well, if you absolutely needed to use that specific rpi camera module (if I understood your other comment correctly) and you didn't have time to reverse engineer it, then rpi was probably the best option, sure. It just feels weird to assume that, or that every single module needed to be connected by ethernet. Rpi and low power in the same sentence also feels weird. I'd have tried to build little devices out of STM32. But again, I barely know anything about your project.
Ethernet was handy for moving the images off the Pis once they were captured.
The RP2 was indeed low powered compared to the Intel/AMD powered systems we were otherwise familiar with.
Outside of the bubble of people with EE skills, however big the bubble is, stopping to learn expert-level mcu programming from scratch is rarely the way things work in a software startup. Despite the obvious focus of this thread, the hardware we assembled and configured was all in support of our software.
I assume that the rather hostile reaction you've received in this thread stems from the combination of your insistence that you tried everything to "bend the law of physics", but dismissal of any suggestion of simpler methods because you're a software person and couldn't have possibly thought/known of everything and we should just all assume that you found the optimal solution. Said differently, it's a strange combination of "we were a team of geniuses so trust me our solution was the best one" and "we were all software people so stop suggesting hardware solutions, it was too complicated for us to learn".
I didn't perceive a hostile reaction, beyond one dude who declared that something hard would be easy.
On the contrary, I've had a number of solid back-and-forths with folks about GPS, GPIO and homebrew RTC daughterboards that we could likely find success with today - almost a decade after we built what we built.
I'm honestly not sure what else you expect from me. Should I apologize for our relative ignorance at the time? Are you upset that I'm proud we managed to succeed despite doing things differently than others might have?
Given that you can't share the unique details of your use case, what could have been an interesting discussion regarding design tradeoffs is moot.
(And I would politely ask that you don't make assumptions about what I might or might not have actually accomplished in similar spheres, you have zero idea.)
What's with the patching? It sounds like pretty much spot on for what NTP was designed for.
Each clock is monotonically increasing, with its individual drift. NTP will pretty soon find out exactly what the right polling interval is. Even if you run it in broadcast mode (why would you though, with only 70 devices).
It's been eight years, but if memory serves, the primary need was to force it to broadcast as frequently as possible.
Everything you said is totally legit unless you're addressing devices with software clocks that are running at speeds dictated by subtly shifting voltage.
Remember, according to the Pi itself, it's 100% in sync. No problem hereeeeeeee (robot voice as voltage dips to 4.9887 volts).
At some point, and considering how cheap they are these days, the simplest option is to add a GPS module if your use-case allows for satellites in decent view.
Would it have worked to add something like a single GPS module connected to an ESP8266 that sends a pulse every minute to all the Pis' GPIOs so they can sync?
Hm, yeah, you'd need a much more frequent method of synchronization to avoid that. I'm assuming the rate wasn't constant, huh? Because of voltage drops depending on draw?
I want to be a better conversation partner on this, but I am extremely constrained in what I can share and again, it was 8-9 years ago that this was all top of mind.
Can we mutually agree that in a year of not-dumb people trying everything we could manage to achieve the best outcome, we tried a lot of things and we often didn't know what we didn't know?
We cannot, because I'm not trying to find you a solution you didn't think of, I'm trying to learn from your failures so I can skip over them next time I need something similar to this.
First, our failures were not technical. We succeeded at creating a sub-$100 camera node that could be replicated and/or replaced quickly by relatively unskilled talent.
If I was going to do it all over again with the decade of engineering skills I've learned, I would look towards fabbing our own RTC I2C daughterboards and trying some GPIO-based approaches to triggering.
However, given our constraints, the solution I've described above worked incredibly well. I wouldn't be embarrassed to do it again.
I'm not saying you failed at the task, but that you tried a bunch of things along the way (that you've detailed in this thread) and they didn't work for whatever reason. That's valuable to know, it saves me time if I want to do something similar in the future. Plus, it's just nice to hear about the engineering of it all.
I got a u-blox NEO-6m module for a few bucks, add another few bucks for the active antenna. The newer GPS modules are much better (more satellites, faster lock etc) but this is sufficient for me.
We've synchronised multiple Pi's to fractions of a millisecond, with flash systems with accuracy in the microseconds for specialised motion capture. Happy to discuss if you're interested - I'm on <hn username> at 'tessierlabs . com' (without spaces).
Here's the deal: RPi<5 runs at the speed of the voltage coming in. The OS creates a fake software clock that assumes it's a perfect 5v, but in reality not only does voltage fluctuate, it drops over the length of a cable run. If you have, I dunno, 70 Pis connected, the 5v on the first one is very different than the 5v on the last one, no matter how frequently you ask NTP to broadcast.
The first thing we did was deliver 12V down the wire with a high quality power supply. Then we used UBEC buck converter modules (intended for RC quadcopters) to deliver power directly into the Pi via its pins.
However, even now you're just chasing physics. There's no way to send a command to all of the Pis to do something right now. Instead, we tell all of them to do something at a very precise moment in about three seconds. In our case, this meant writing a very low-level C program to listen for raw UDP messages and trigger the camera module at the hardware register level.
The only way to verify that it was working as well as we believed it did was to create a reference clock. We had a friend build a 20 LED timer that would count 10ms and 1ms increments.
We achieved < 1ms syncronization across 70 Pis in this manner.