Apropos, the Gaia space telescope's first data release is next week. It's an extremely large dataset of the kinematics (3d position + velocity) of Milky Way stars.
I'm always a bit uncomfortable when I read about findings from these simulations. To say they have "answered questions" about galactic evolution seems to strong a statement. How about, "created a more accurate model"?
Yes. Like, "Given everything we understand, how could it be that in the physical world, there aren't more dwarf galaxies?"
And there was no answer to that question, until now.
Now they come up with a new model, and run it, and they end up with a number of dwarf galaxies that much more closely matches the physical world. No one had done that before, so we didn't know how it could be.
This has answered the question, "HOW COULD IT BE?"
The answer is, "If reality matches this new model."
Saying "it has been answered" is a strong definitive statement, though. It's rather arrogant to say "Yes, thanks to this model, we absolutely know <xyz>." Are we absolutely certain that this model is correct? Are we absolutely certain that what we can extrapolate from this model accurately applies to the physical world?
Granted, yes, questions were answered... I guess. But saying that "The simulation solves a decades-old mystery" is uncomfortably confident and counter to how scientific process works.
Some people like to say things like that, but it's more accurate to say "this proves our current model is consistent with data". I only skimmed the article (and haven't read the actual paper) but it focused on surrounding dwarf galaxies which have traditionally been a problem with these simulations. I'm sure there are still plenty of ways in which these sims do not match our actual universe.
It might seem small, but I'd argue that the difference in phrases is pretty stark.
Saying we "have a better model of"-something leaves room for further exploration and interpretation. It's admission that there's room for different possibilities, and its not taking an /absolute/ position on something.
But saying something "has answered questions" comes off as absolute and final. Is 1+1=2? Yes. Answered. Case closed. If you're going to say that "The simulation solves a decades-old mystery", then you'd better be damn sure that your simulation is correct, your interpretations are correct, your simulation correctly reflects physical reality, etc.
Indeed, and with that it will hopefully expose details in models missing, after all how accurate are weather forecasting models and how quickly do those nuances propagate and yield week ahead forecasts that are totally different.
Though if they can run a model and end up billions of year later approximating aspects of today then they must be getting more right than wrong.
Which is why, for nearly all simulations, you use Born-Oppenheimer approximation and molecular dynamics using force fields derived from QM, but without the full explicit QM treatment.
This is fine because with only rare examples (as far as modern science knows), QM treatment is not necessary for most molecular machines.
I didn't realize that. It sounds like the same situation with astronomical simulations. General Relativity is too complicated so everyone uses Newtonian mechanics plus a fudge factor.
I think it is odd that the two most cherished scientific theories are not actually used in practice to compare with data.
Yes, both astrophysics codes and MD are basically "n-body approximations" https://en.wikipedia.org/wiki/N-body_simulation although if I understand correctly universal simulations do explicitly model general relatively
Personally I find the "particle mesh". If you model the N-body calculation this way, you can solve for the field (potential) by converting to fourier space, doing some FFTs (which computers are really, really good at), then converting back to real space. This works in both astro and molecular simulations, really well.
Fourier methods are good, but only when your potential is decaying slowly. If it decays fast, like van der Waals interaction which are plentiful in MD, you just have to do the plain n-body approx.
In MD, it's a combination of VDW and electrostatic. Typically VDW is just a cutoff (VDW forces beyond 8A are neglible), and electrostatic is handled by particle-mesh ewald. I worked on DNA and RNA, which have strong electrostatic interactions and my thesis work showed PME was critical for stability.
This was what I was trying to say. Actually, this assumes atomistic MD where you have electrostatics. I've done some work with coarse-grained MD, where you've averaged out the electrostatics and you just have VDW.
Do you mean, to simulate the active site of an enzyme, or that people are trying to explicitly model known quantum biology, like photosynthetic centers? I don't know that anybody is claiming it's necessary for protein folding or DNA/RNA simulations.
For context, this question of whether vanilla dark matter models are in conflict with galactic observations has been raging for many years. (It includes both the dwarf galaxy problem mentioned in the article, and things like the "core-cusp" problem concerning the radial dark matter density distribution.) These numerical simulations are fiendishly complicated, and there's always a risk that you just keep adding new tweaks or effects until you get the results you're looking for. I'd be interested in reading a skeptical assessment from the folks on the other side of this question.
Can somebody who knows about this tell me how much of the novelty in this is CS and how much physics? For example in graphics all the physics of light is well understood - it's just infeasible to compute - so all advances are in CS and how to sample better. Is this the case here? Or is it adding new models?
For example in graphics all the physics of light is well understood - it's just infeasible to compute - so all advances are in CS and how to sample better.
Nitpick: The physics of light is understood. Human vision is also understood. But critically, human vision is completely overlooked in most graphical applications. It's not accurate to say that "the path toward making a computer-generated video indistinguishable from a camcorder recording is just a matter of sampling better." There will need to be a fundamental shift in the industry's thinking before realism will be achieved.
It's a fascinating topic that occupied many years of my life, though this is a bit off-topic.
There are scenes in Pixar's movies, rendered entirely by Renderman, that are effectively indistinguishable from a high resolution recording of a real instance of the subject.
That really was all about algorithms and sampling.
There's pretty much no way to avoid writing a large comment in
response to this. So, with apologies...
It was my focus for a long time to achieve perfectly photorealistic
rendering. I come from a gamedev background, and I've been fascinated
since around age ten about how to get a computer to paint pictures. By
17 I was writing game engines mostly equivalent to quake, and used
this portfolio to get into the industry. The next three years were
spent getting as close as possible to the bleeding edge of real-time
graphics development.
I remember having long debates with a colleague about what
high-dynamic range lighting "meant." "If we spin around in our office
chair, our brains do not suddenly change the overall brightness of
this office. Why should we be programming games to do this? Why is
everyone doing things that way?"
My concerns were of a more fundamental nature as well. What is a
diffuse texture? A diffuse texture is a well-understood concept in
realtime graphics. Anyone with a basic knowledge of shaders should
immediately recognize:
color = lighting * diffuse + ambient
Trouble is, it doesn't correspond to reality even slightly. It's not
even roughly close. It happens to look good to humans, and that's
why we use it. But the trouble was, the further I tried to probe
the mystery of realism in computer graphics, the more I ran against
this phenomenon of "We use X technique because it looks good."
So I turned to research papers. Books. The medical field. Everywhere
that was remotely related to possible breakthroughs in photorealistic
rendering. Research papers are excellent for assembling techniques,
but not results. The books in human vision and color science were more
promising, yet most of the industry seemed (and still seems) to pay
little attention to them. Compare a book about color perception to,
say, http://www.pbrt.org/ and you'll see a stark difference. Flip
through the table of contents and you get transformations, shapes,
primitives, color and radiometery, sampling, reflection, materials,
texture, scattering, light sources, monte carlo integration...
And for what? We know that these techniques simply do not produce
computer-generated videos that a human will identify as a real-life
image. It's not for lack of processing power. There is a disconnect
between the old rules and those that will ultimately result in
real-time realism, and you won't find it in that table of contents.
Now, the trouble with writing all of this is that if I knew how to do
it, I'd have done it already. It's a life-long search, and it's not so
easy to refute an entire industry without being (rightly) dismissed.
But if you wish to know what I suspect are the ways forward, it's
this: Get a camera. Take photos. Compare these photos to the results
of the algorithms you write. Iterate on your algorithms until they are
producing results that match something that already captures nature,
not our beliefs about how we ought to be able to capture nature. "Just
throw in physical models and presto!" has not thus far been true.
You'll notice, for example, that computer graphics in videogames have
plateaued. They get more impressive with each generation, but that
impressiveness does not get them progressively closer to looking
real. Nor should it. A computer game tells a story. The closer it
looks to real life, the more restricted the artists are, along with
the rest of the design of the game.
So we turn to the movie industry for hope. But it's restricted in
exactly the same way. The research papers are all along the lines of
new techniques to try, or studies of existing pipelines and how to
deal with their complexities. It's not fundamental research.
As someone who has spent his life in pursuit of realism in
computer-generated video, my recommendation is this: Read DaVinci's
journal. Pay attention to what each page is saying. He had to discover
from first principles what makes a painting look real, and why. You'll
notice that he spends most of his time talking about human vision and
our perceptions of color.
If someone is going to make this development happen, it's not going to
come from the game industry, and it won't come from the movie
industry. That leaves you. Hopefully this will encourage some of you
to pursue this. Once you accept that most of the computer graphics
industry isn't actually focused on achieving realism, you'll start to
develop your own techniques. My hope is that this will eventually lead
to a breakthrough.
1. "Take a picture and make it look right" is exactly what people have done for things like POVRay since at least the 90s. I can't find it now, but at one point someone set up a glass ball on a checkerboard and used a point light and a camera to confirm the diffraction and distortion models were correct because someone claimed they didn't look right or were doing the wrong thing. The math that's there for rays, shapes, diffraction, diffusion, caustics, etc. is accurate, and necessary but not sufficient. Which brings me to
2. CG realism has generally hit the Uncanny Valley by now. It's so close to real that we think "that's pretty good", but it's far enough away that we still know "something's wrong". It's the difference between a dummy, a corpse and a live person.
A couple of examples I remember from the past decade are laser + milk and better skin rendering on one of the NVIDIA demos a while back. There wasn't (isn't?) a good model to simulate the diffraction and subsequent diffusion of a laser shining into a glass of milk. Actual lasers with actual milk don't do the things we expect of modeled lasers in simulated milk. Some component is missing, but all the existing math is right for lots of other cases. The NVIDIA skin thing was adding 3 or 4 layers to an existing model to simulate subsurface scattering and reflection that happens in skin, vs old models that treat skin as paint. The old stuff was right, just not enough.
All of that aside, there are decent photorealistic rendering options for some materials today, but at the cost of CPU hours of render time. If you can do better then please do, even if it's just for one material or one physics action.
I appreciate you probably put a lot of effort out but your post largely sums up to "They are wrong, I've done extra curricular research to prove it, I have no alternative"
While noble, I cant help but wanting to understand what alternatives you were leading yourself done. As an example, light can arguably simplified to intersecting cylinders and spheres that bounce off surfaces to create new 3d shapes. Each shape also would have an origin 2d shape based upon whats reflecting it. an "eye" reads shape intersections with self and also can filter those intersections in respect to origin shape. After each bounce, the new shape takes form as the bouncing lights color multiplied by the color of the bounced object In low light situations, subtle luminosity differences can be enhanced.
What I did was offer an example. Perhaps youll one day be successful but I got the impression you are some kind of renegade with a mission. While I can certainly relate to that, I view science and building the future quite far from renegade status. And in the mean time, you gave me a sob story with no algorithms/solutions except for "take real pictures and compare them". As a lazy programmer, walking outside and discoveringvthe world doesnt interest me too much.
We can come up with hundreds of simplified lighting models. Have you tried the one you mentioned? Does it correspond to photographs, video? That's the hard work, and it's why I listed none.
Here's something specific: What did you mean by "multiply"? You cannot "multiply" colors. Not unless you concede that your model has nothing whatsoever to do with physical reality. And at that point, why not use a photo of nature (or your eyes' perception of nature) as a baseline comparison?
"The phenomenon of colors depends partly on the physical world. We discuss the colors of soap films and so on as being produced by interference. But also, of course, it depends on the eye, or what happens behind the eye, in the brain. Physics characterizes the light that enters the eye, but after that, our sensations are the result of photochemical-neural processes and psychological responses.
There are many interesting phenomena associated with vision which involve a mixture of physical phenomena and physiological processes, and the full appreciation of natural phenomena, as we see them, must go beyond physics in the usual sense. We make no apologies for making these excursions into other fields, because the separation of fields, as we have emphasized, is merely a human convenience, and an unnatural thing. Nature is not interested in our separations, and many of the interesting phenomena bridge the gaps between fields."
Walking outside and discovering how the world looks is exactly how to improve your techniques as a graphics programmer.
I work in film vfx, and if you think we dont compare our rendered images to real photographs and video constantly, then you dont really know how modern graphics are produced. We shoot reference for everything, and use things like gonioreflectometers to sample real world BSDFs. Yes, lighting models are always necessarily simplified from the real world physics. But the thing is that 99% of the time that doesnt matter, because most common types of surfaces are able to be reproduced accurately enough to fool the majority of people. I would wager that most of the CG things you see these days on TV or in movies you have no idea was CG.. it's only the bad stuff (or obviously impossible) that stands out.
I actually am a bit confused what you are arguing here.. all of these problems you're mentioning have been well-understood by graphics researchers for the past couple decades.
The main point was that film isn't interested in exact photorealism. As you said, it doesn't matter, because the simplified models are good enough. Therefore it's unlikely that the film industry will be the first to produce a fully computer-generated video that will be indistinguishable from a camcorder capture.
The reason most of the CG you see in TV or movies looks very good is because they take place within real video. We're not looking at a completely CG scene -- it's mixed with video from the real world. And that's a perfectly valid technique, but my comment was talking about 100% CG.
A secondary point re: the film industry is that artists must necessarily retain control of the art pipeline in order to create scenes that advance the plot. That requires the art pipeline to be flexible. The more flexible your art pipeline, the more productive your studio is. Yet that flexibility is precisely opposite to realism. Obviously, the more realistic a purely CG scene looks, the less flexibility you get, otherwise it wouldn't appear real; hence the argument that the vfx industry won't be the ones to produce the elusive fully-CG fully-realistic video. (It doesn't make financial sense for them to do so, if nothing else.)
Again, I would wager that you have seen many, many shots on TV and in movies which were fully, 100% CG already. You just didn't notice because they really were indistinguishable. We've definitely already crossed that boundary in many areas, and the main things which still stand out is just bad work.
I've been diving to light, color, and color perception the last couple weeks as part of a fun work project. I've got a diffraction grating spectrometer on my desk that measures light in half nanometer wavelengths. From the papers that I've read, and the stuff I've messed with so far, it seems that human perception of colour - on average - is well studied, and has been since the 1930's. We know how light gets into the eye, and how much each wavelength of light will excite the average eye's individual color specific sensors.
Once the info leaves the receptors, now that's a another story...
Perhaps a way of stating it is that you want to create computer programs which do the same image processing that our brains do. That, I think, is what you're getting at: we don't see "reality", we see images processed by our brain, from signals received by our eye, mixed with a bunch of learned and instinctual models already existing in our brain. Accurately modeling the signals entering our eye is not enough. We also need to perform - at least approximately - the some processing that our brain does.
Do a search for photorealistic ray tracers. Photorealism is inevitable as we are only projecting onto finite displays. Enough samples from an accurate enough model will result in bits that are indistinguishable from a live capture.
Light can be multiplied, in a way, through interference.
so you want to store an amplitude and wavelength along with your shapes instead of color? Go for it. You want to add in psychological factors? Sounds wasteful.
Your type of examples seems like uts more useful for machine learning. Collect enough samples, build enough models, run enough tests and youll get something accurate-ish.
I thought you were going to go with a different angle on this. If all you care about is photorealism, well, we'll have to agree to disagree that the current state of the art is incapable of that (given enough time).
What I thought your original point was is that 3D rendering ONLY cares about photorealism. Physically based renderers have been greatly influenced by photography (both still and film). Think light probes, camera lenses, etc. Much of the post-processing you see in a scene is also derived by what a camera would see, from focusing, to blur, to HDR now, and of course the infamous lens flare!
So I think a physically based renderer which uses the human eye as its camera would be interesting to see more of.
The recommendation to get a camera is a very good one. A camera used for an extended period of time is a very good teacher, and not only of seeing.
To go even further, learn to draw. Use your hands and other senses instead of only thinking. Read Drawing on the Right Side of the Brain by Betty Edwards and The Hand by Frank R. Wilson.
I'm curious what your favorite games are, visually, @sillysaurus3?
A couple of mine are Far Cry 2 and [Elite: Dangerous][1] Horizons. I also love the sound in both of these which is to me a seemingly inseparable element to great screen work.
I'm curious what your favorite games are, visually, @sillysaurus3?
I've been considering an answer to your question for the past couple of days. As cheesy as it sounds, I suppose my favorite isn't necessarily a game, but a video:
This was what I grew up with, and what helped inspire me to pursue graphics programming. I used to watch it on loop dozens of times, being amazed at how graphics got better with each game.
That was back in the day when you had to download codecs for DivX, and find these videos on Kazaa... Ah, memories. :)
A couple of mine are Far Cry 2 and [Elite: Dangerous][1] Horizons. I also love the sound in both of these which is to me a seemingly inseparable element to great screen work.
Good choices. Absolutely agreed re: sound. Sound and animation are both vital (but often overlooked) elements of a good story, with all the focus on graphics nowadays.
> And for what? We know that these techniques simply do not produce computer-generated videos that a human will identify as a real-life image.
This is actually testable. Some archviz images are indistinguishable from reality for non-graphics experts.
> Get a camera. Take photos. Compare these photos to the results of the algorithms you write. Iterate on your algorithms until they are producing results that match something that already captures nature, not our beliefs about how we ought to be able to capture nature.
Yes, very important. Art teachers try to reinforce this by saying "seek reference". It's true for graphics as well, and it's possible to be more empirical.
Some archviz images are indistinguishable from reality for non-graphics experts.
Photos, yes. Not video. It's an important distinction because the path to photorealistic video won't pass through impressive-looking still frames. Visual processing of movement is quite different.
Replaces "looks good" with a more generic "gets the right answer" and you'll notice that phrase repeated on any simulation context where computers are not powerful yet to work from first principles.
If you want a safe place to look, look at chemistry. They have simulations that vary in complexity by a huge number of orders of magnitude. You'll see those non-natural techniques that "look good" get progressively applied as the simulation gets slower and slower. It works really well, but once computers are fast enough, everybody just throws them by the window.
> Can somebody who knows about this tell me how much of the novelty in this is CS and how much physics?
It's not so much a novelty as it is pushing the envelope.
The main problem with these kind of simulations is that they're O(N^2). You can't get away with modelling each star, you have to model the interaction of each star with every other star.
And then add in interstellar dust, which has non-neglible mass. And then run the simulation for 8 billion years.
There are naturally some optimizations and short-cuts to be made. :)
> The main problem with these kind of simulations is that they're O(N^2). You can't get away with modelling each star, you have to model the interaction of each star with every other star.
Summary of what's below: The type of simulations described in the article do not scale like O(N^2) but more like O(N log N).
A small correction here.. The gravitational force calculation for N-body simulations are only O(N^2) if they're done using direct summation and force calculation. That's only necessary if you care about the motions of individual stars. The largest simulations I'm aware of that do that use a few 10s or 100s of millions of stars. They're typically used for studying individual star clusters, the regions around supermassive black holes, and perhaps dwarf galaxies. But the simulation in this article/press release has a few 10s of billions of particles, I believe. And those particles are all much more massive than individual stars. So while there are "star particles" in the simulations, it's more accurate to think of them as a set of particles which constitute a monte carlo sampling of the stellar mass distribution.
One aspect of that is you now no longer care about the detailed trajectories of individual particles[0], instead caring about the overall representation of the mass distribution. So you don't need as high of accuracy in the force calculation. Going back to the 1980's, people have been using numerical tricks to reduce the scaling of the simulations with the particle number. This includes algorithms such as treecodes, particle-mesh codes, etc., which reduce the scaling to O(N log N) or even O(N). These are sufficient to run simulations of describing the overall gravitational field.
The scaling for hydro codes is probably more complicated, particularly for simulations which use a moving mesh.
[0] One can determine if interactions between individual stars is important to a system's evolution by comparing the 2-body relaxation time to the age of the system. This is discussed on pages 8–9 of the preprint linked above.
Some simulations also use Smoothed Particle Hydrodinamics (see here[0] for a simplified example), like the Earth-Theia impact Moon simulations[1]. However I'm not in the field and can't really tell how much this kind of code has been used lately, but you did remind me of it when you talk about this simulations being monte carlo sampling of stellar mass distribution, because SPH has some concepts like that "inside" the particles.
Also, for everyone to enjoy simulations results, check the simulations by Canup from the giant impact hypothesis[2]. Another thing to consider, the supercomputer used for this specific simulation had about the same computer power as a high end GPU from today.
SPH has been fairly common in galaxy simulations over the past 20 years or so. But there's some glaring issues with "classic" SPH formulations, to do with fluid instabilities. As a result, lots of people are transitioning to moving mesh codes. I put a few links in my other comment (https://news.ycombinator.com/item?id=12461336). I'm pretty sure the simulation in the press release is also using a moving mesh for the hydrodynamics.
GIZMO is the name of the code suite, not the numerical technique. I believe GIZMO contains multiple hydro solvers, including various formulations of SPH, moving mesh, and meshless techniques. But you're right. that the simulation in question wasn't a moving mesh, it was a meshless technique.
> Can somebody who knows about this tell me how much of the novelty in this is CS and how much physics?
It's a little bit of both. Over the past 5–8 years, some new numerical techniques ("moving mesh") have been developed (or at least first applied in astronomy) for hydrodynamic simulations[0,1]. These new techniques do a much better job of handling fluid instabilities, which can be important for properly simulating the gas in galaxies (which can have a huge range in densities and temperatures).
Also, people have developed what they call "cosmological zoom" simulations. In these simulations, you run a low-resolution (sometimes dark-matter only) simulation of a fairly large volume of the universe. Then you identify a particular dark matter halo at the end, that you think matches the type of galaxy you're interested in (say, a Milky Way type galaxy). Then you go back to the initial conditions, re-initialize it at very high resolution and include "normal matter" (baryons) and re-simulate the whole thing to study the formation of that particular dark matter halo and its associated galaxy.
Finally, the general increase in computing speed has allowed people to add in more physics to the simulations and run higher-resolution simulations. In this case, it is the addition of a physically-motivated model for energy injection into the gas in galaxies, from the ongoing star formation. This means energy from supernovae explosions and associated shocks, stellar winds, and momentum from scattering/absorption of photons by dust grains[2,3]. This more realistic set of models manages to reproduce the overall efficiency of star formation (what fraction of the gas gets turned into stars over some period of time) _and_ create a realistic temperature-density structure for the gas.
Now that people seem confident that these models can reproduce the overall star formation properties in galaxies, they are starting to see what this type of star formation / feedback model implies for the properties of galaxies. Of course, there's still lots of work to be done in making detailed tests of these models. While they reproduce the overall efficiency of star formation, we aren't yet sure if they can reproduce the shapes and sizes of galaxies. But people are working on that and we should know before too long...
It'd be pretty cool for us to discover that there are tiny virtual humans in that simulation, complete with consciousness, contemplating their own existence.
http://www.cosmos.esa.int/web/gaia/dr1