I tried again to use rust for something the other day. Just wanted to fetch some things via http. Nothing fancy, didn't even really care about the response body, just the timings.
Consensus seemed to be that reqwest is the way to do this using rust, since the stdlib does not include an http client.
Adding the dependency on reqwest tried to pull in over 100 dependencies and after building for a while, it failed somewhere along the way because my rust version was too old. Now, maybe some of these were optional dependencies, but they all got pulled in automatically.
so, when TFA says
> Another obvious advice is to use fewer, smaller dependencies.
I just went back to go, which can do the crazy obscure task of making an http request without requiring 100 external dependencies.
Those dependencies aren't recompiled every time you change something. They will only impact your initial build or a clean build on a CI/CD system.
Raw dependency counts can be shocking for those coming from other languages where pulling in a dependency is an uncommon operation, but it's not a bad thing. I'd rather have my HTTP client library re-use popular, tested packages for things like decoding JSON, handling TLS, working with IP addresses, and other common tasks. I don't think it would be better to have every package roll their own versions of all of those common pieces.
Notably, several of the big dependencies are maintained by the same person who maintains reqwest. It's a common practice with Rust crates to split a project into reusable parts. Reqwest uses hyper (http client), http, http-body, mime, url, and several other crates that are owned by the person who owns the reqwest crate.
If we instead mashed all of those dependencies together into a single crate it would appease the people who don't like seeing high dependency numbers, but it wouldn't really make the product any better. It would make it harder to re-use the individual pieces, though.
> I just went back to go, which can do the crazy obscure task of making an http request without requiring 100 external dependencies.
Go is famous for including HTTP utilities in the standard library. If I was building a simple app that made HTTP requests and little more, I'd probably pick Go as well. Or Python.
Or you could have searched for a minimal HTTP request crate ( https://github.com/algesten/ureq ) designed to be small instead of using the most full-featured crate.
If your goal is to reach for something quick and easy with batteries included and minimal external dependencies, Rust is not a good choice. Doesn't mean Rust is "bad", but it's not the right tool for every simple job. You also can't just pick the most popular crate and assume it's exactly what you want.
> It's a common practice with Rust crates to split a project into reusable parts.
Precisely this. Rust is not the same as Go or Java or Python: it's still a fairly low-level language. Standard practice is to build lower-level components (like http parsing and url crates) that are abstracted over by more 'friendly' crates (like reqwest). It's jarring, and certainly different from a lot of other languages, but it's a testament to the language's flexibility in my eyes.
An other factor is that the unit of building / concurrency in compilation is the crate. So splitting a crate into 15 sub-crates (which could be reused independently) increases parallelism.
Funny thing is, Rust and JS are both given that comment.
Seeing how they are different languages, there's not a single language feature that explains this dependency gluttony.
I'd like to open up the option that it is a necessary side-effect of large ecosystems: more devs using the language, more packages published, more quality in small packages. And then, when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel.
And so, driven by nothing but good will and best practices, since you have a lot of prebuilt "assets" at your disposal ,your project ends up with 100s of deps and everybody complains.
> when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel.
Yes, for instance one of `reqwest`'s dependencies is `base64`. We can argue both that re-inventing base64 in every big library is pointless, and also that a language doesn't need base64 in its stdlib because C and C++ don't have it.
There's no right way to cut this cake - Many a language has included something in the standard library, and then struggled to fix a broken interface once everything depended on it. (Like JS picking UTF-16 for strings) Many a language has also failed to include something in the stdlib and caused every application to suffer and re-invent it. (Like C++ not really saying anything about Unicode strings, which means Qt and everyone else has their own little take on it.)
And the suffering is only bigger if you make dependencies artificially hard to add.
Rust's approach is more flexible, at the cost of building the universe from scratch every time you make a new workspace. Personally I don't mind it. At least it's not like the problems I've heard from Haskell land. Moving from futures to async in Rust was fairly smooth.
Yeah in the debate about dependencies, people often frame it as more dependencies == bad and less dependencies == good. Except more/less dependencies is actually more or less explicit dependencies. Fact of the matter is that you need a crazy amount of dependencies to write most modern software. You need ways to handle Unicode, to connect to databases, to write async code, to handle dates, etc.
The question is whether that code is going to come from a monolithic Swiss Army Knife of a library (aka a standard library), a bunch of packages, or some code that everybody copies into their codebase. In which case the packages option doesn't seem so bad. At least they're explicitly versioned, tracked for vulnerabilities and can be updated at a granular level.
Because when working in large corporations one ends up having to write license checker tools and having humans go through those packages, alongside legal, to validate what is allowed to appear on the final product, and to vendor them on the internal package repository.
The process for proposing new modules for Rust's stdlib is pretty streamlined. Small API additions can be proposed directly via PR to rust-lang/rust, whereas large API additions can be brought up for discussion in rust-lang/rfcs. If anybody thinks that base64 deserves to be in Rust's stdlib, I encourage them to propose it (to my knowledge nobody ever has).
You're nitpicking one example. Putting base64 into std makes the higher level comment go from complaining about 100 dependencies to complaining about 99 dependencies.
I don't have an opinion about the broader dependency problem in Rust. The comment to which I responded was hosting an internal debate about Base64 and UTF-8. I think that debate is easy to resolve: use UTF-8, put Base64 in the stdlib. Do you disagree?
I guess I'd just say that the aggregation of these "keep base64 out of std" decisions are why every Rust thingy I put together ends up with a zillion dependencies. There might be an upside to that (I have not detected it), but there's an obvious downside.
Rust gets a bunch of stuff right that other languages missed. I don't think this one of them. I think Go's strategy of having a pretty-ok version of most of the table-stakes things developers do is probably the right one, just like I think sum types and option/result are better than error types and `nil`.
I share your irritation at people who drop into the middle of discussions like these to nitpick random language war arguments, but this is a thread that is actually about Rust dependency hell, and the comment I responded to, I think, sort of diagnoses why Rust has that "problem" (if you think it's a problem --- I do, but whatever).
For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not.
> I guess I'd just say that the aggregation of these "keep base64 out of std" decisions are why every Rust thingy I put together ends up with a zillion dependencies.
Right, I partially agree, and this is a more substantive critique. But it's a much harder position to put into a pithy HN comment that sounds obviously correct. ;-)
With that said, it's pretty common in my experience to wind up in exactly the same sort of situation in Go-land. Go's standard library might hold it off for a little bit, but I seem to accrue large dependency trees there too.
> There might be an upside to that (I have not detected it)
Aw c'mon. The comment you were originally nit-picking even called it out. I'm sure you've heard the quip, "the standard library is where things go to die." And Python's in particular is often brought up as an example. Go's standard library hasn't suffered as badly as Python's has, but there are parts that have died.
If you've used Python a lot, then the fact that you have to use 'requests' and not std's HTTP client is probably not a downside. You've already digested that ecosystem knowledge. But as stewards of the standard library, this is something we see as a downside.
> For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not.
Yes. Our design philosophy around std learns from some of those languages. Python in particular has influence in two respects. First is what I said above: when you have to make an HTTP request in Python, the standard advice is "use the third party requests library" despite the standard library having an HTTP client. This should make it clear that following your advice is merely a necessary criterion; it is not sufficient. Second is that "std should remain backward compatible for approximately forever." That means we have a very high bar for including things. (I am saying "we" here because I am on the Rust team that decides these things.) API evolution is more difficult in std then outside std, because we can't just release a 2.0. So if we get the API wrong, it sucks. And eventually a superior third party crate appears and everyone has to "learn" to use the third party crate instead of std.
And of course, adding things to std also means a higher maintenance burden. There have to be people willing to champion and maintain these sorts of huge additions. And many of us (now speaking as the regex crate maintainer) are actively opposed to throwing it into std precisely because it becomes more difficult to evolve.
Even if we did all this and put whatever pet thing it is you want to use without pulling in dependencies (I guess that's "HTTP client" here), that doesn't really solve the general problem. Eventually, you're going to pull in a dependency for _something_. And that crate's author may have decided that depending on something else instead of rolling their own thing was the better choice. Or also the opposite: perhaps they want to split apart the internals of their crate so that others can reuse those components. I did that with the regex crate. So on the one hand, I'm contributing to the high dependency count number, but on the other, I'm solving actual problems people have by providing access to internal APIs that are separately versioned. How many other regex libraries do that? Not many. Because dependencies are too annoying in the C ecosystem, which is where most regex engines live.
Getting more hand wavy... In the Go world---and I speak as someone who has used and enjoyed Go for over a decade now---performance is not a #1 concern. Not like it is in the Rust ecosystem. This has huge API implications. Culturally, people expect Rust programs to be fast and they expect to be able to achieve congruent performance as they could in C. In this environment, API complexity isn't much of a reason to not doing something. Everything gets sacrificed at the alter of performance. I know, because I've been carrying sacrificial lambs to the alter for years in the Rust ecosystem. But in Go? Nope. API simplicity is valued higher relative to things like performance.
Here's the punch line: if you want to write a program that sends an HTTP request and has a dependency count of 0, then Rust isn't the right language for you. It probably never will be.
Also, to be clear, I am someone that works both sides of this debate. I can often be found advocating against the use of dependencies and decreasing dependency count. Precisely because I perceive and experience costs associated with them. I've done work to reduce the dependency tree size of core ecosystem libraries. But there are some fundamental trade offs that put floors on how long you can go. And you can also seen me as an advocate against using many of my crates if a "simple" solution works for you. Take aho-corasick for example. My implementation of it is several thousand lines because I sacrificed just about every lamb possible at the alter of performance. But the <10 line naive implementation of multiple substring search might be just fine for your use case. And if it is, just write that and don't bring out the big guns.
Yeah, so let me be clear: I can see why, especially when the std lib was in its formative stages, the project might have decided to keep it, uh, lithe. You have Python as a very good counterexample to "kitchen sink stdlib can be good". Python's stdlib is a kitchen sink that badly needs to be cleaned out, and never will.
You're right that Go has some degrees of freedom that Rust doesn't have, especially on performance. But people do have perf problems with the Go stdlib, and they replace components (amusingly, one of the first things I built in Go did its own sockets and its own polling because Go's timers were too expensive for what I needed).
I just think it's better to have a good-enough stdlib --- with well-thought out interfaces where you're not going to have net/http and then net/http2 and then net/http3, and 16 different ways to run a subprocess and read the output --- that people can optionally outdo with 3rd party dependencies, than the situation I think Rust is in right now, where you basically can't do anything without a bunch of dependencies.
I'm definitely not dunking on Rust. We do a bunch of Rust here and I mostly enjoy working in it.
> Go's standard library hasn't suffered as badly as Python's has
In no small part because it was born 20 years later: lots of modules were added a bit haphazardly to the Python stdlib early on and the tech ended up mostly dying, but for the most part they're still there in case you care to use them. Like sunau, aiff, or dbm. Others were added because package managers were not really a thing back then so you'd try to get useful packages into the stdlib so they could be used easily, but then the maintenance burden increases a lot (and all the users assume the stdlib is maintained anyway).
Having to find, download, review, and build a bunch of extra dependencies (and surveil all the dependencies you do explicitly want to make sure they're not pulling in fucky code to do table-stakes things that the stdlib could just stdize.)
Again: these are downsides, but not dispositions. On balance there could be good reasons to have a terse, svelte, balletic std. I'm just saying, there's clearly a downside to the decision.
Without getting into Rust’s standard library inclusion philosophy, I would note that UTF-8 and Base64 are quite different cases, and shouldn’t be combined in this way.
String encoding is pervasive, and if the standard library didn’t have strings or didn’t specify the encoding, there would be much grief throughout the ecosystem, because you can’t just patch it in at the library level, it’s a language-level feature. So string encoding is a choice that has to be made.
Base64, on the other hand, is just a bunch of functions, nothing special about it that means it has to be in the standard library.
It does need to be in the standard library, because everyone needs it, almost nobody has special needs for it, and if it's not in the stdlib then you have to find, download, review, and build it and its dependencies, and that is problematic.
When I say that something “has to be in the standard library”, I mean that it can’t be implemented outside the standard library. That’s certainly not the case here. You’re using an outright bad definition of “need” here—subjective opinion rather than objective requirement.
> because everyone needs it
This is factually wildly wrong. I wrote a fair bit more here but decided it wasn’t helpful. Précis: web stuff tends to load it indirectly (though amusingly most of the time actually not use it, so that Base64 code won’t actually end up in your binary), but it’s not terribly common outside of internet stuff to reach for Base64.
I’ll leave just one more remark about Base64: once things are in the standard library, breaking changes can no longer be made; the base64 crate is still experiencing breaking changes (<https://github.com/marshallpierce/rust-base64/blob/master/RE...>, 0.12 and 0.13 were last year and 0.20 is not released), largely for performance reasons.
Please don’t just call the thin-std approach “problematic” without acknowledging that the alternative is at least as problematic, just with a different set of caveats.
> This is factually wildly wrong. I wrote a fair bit more here but decided it wasn’t helpful. Précis: web stuff tends to load it indirectly (though amusingly most of the time actually not use it, so that Base64 code won’t actually end up in your binary), but it’s not terribly common outside of internet stuff to reach for Base64.
It also seems to be distinctly losing popularity as time goes on, as the alphabet is not restricted enough to be easy for a human to sight-read/copy, and a wider alphabet will have as good an ASCII channel compatibility.
(begin rant) My issue with base64 is that + and / are terrible character for the alphabet and that the final padding = can be omitted so that base64 strings cannot be safely concatenated (as whitespace is ignored)
See also https://en.wikipedia.org/wiki/Base64#Variants_summary_table. base64url is common, with - and _ instead of + and /. As for concatenating, that’s not valid even in variants where = padding is mandatory: padding marks the end and is not allowed to occur mid-string. But I don’t think this is a problem, either: transparently concatenating Base64 strings is not generally a useful operation; I can’t think of any valid use case for it off the top of my head, I’m curious why you might think it useful. Base64 is a transfer encoding sort of a thing, and concatenating blobs while in their transfer encoding just… isn’t something that you do.
It’s experiencing breaking changes in the name of improving performance and flexibility. This is precisely the reason why you don’t want it in the standard library, because that would have either prevented these improvements from being invented, or fractured the ecosystem between std::base64 and base64 in just your urllib/urllib2/urllib3/requests sort of way.
As it stands, by being external, people can depend on base64 0.11, 0.12, 0.13, &c. and it all works fine, and over time they’ll tend to migrate to the latest version (dealing with the generally minor breaking changes involved) but it doesn’t matter so much.
The ecosystem is already fractured! Depending on when you added the base64 crate to your project, you have incompatible interfaces! This seems bananas. This isn't, like, serde. It's base64!
Perhaps I didn’t express it in the clearest way; what I meant was more irredeemably fractured with no intent or expectation of reconciliation. When it’s split between base64 0.11, 0.12 and 0.13, it’s reasonable to expect people to update to the latest version of base64 over time—and hopefully that will eventually just be "1", but if it takes time to get there that’s fine.
The incompatible interfaces thing can be a problem in Rust and does require a bit of care, but in this particular case isn’t a serious problem; the base64 crate is almost always going to be implementation detail, not exposed publicly. With serde, yes, reaching a stable 1.0 was definitely more important for ecosystem compatibility.
If I want to dunk on Rust going forward, I can just say "it is important to know the difference between 0.11, 0.12, and 0.13 of the base64 crate". I think this is a hard argument to rebut. My point is not that Rust sucks, but that the stdlib strategy they've pursued is wrong; not a strength of the language.
Further: this seems like an easy problem to fix, much easier than "Go lacks match statements and generics" or "Python lacks meaningful typing altogether". But my guess is that culturally, it's just as hard to fix.
It’s not important. The differences are tiny, the sort of thing that would probably pass unnoticed in dynamically-typed languages but which Rust’s greater rigour instructs be marked a breaking change. As for knowing, humans aren’t in the habit of memorising every detail of libraries that they only interact with occasionally, and this is such a case.
Besides that, you’ll normally be dealing with the latest version, and if you deal with an older version then you should normally consider upgrading it. I looked through a few projects of mine with base64 deep in their dependency trees, and it was all 0.13 except for one instance of 0.12 via a dependency that needs updating.
You’re stubbornly ignoring the problem of freezing base64 into the standard library, apparently obdurately refusing to even acknowledge that a thick std is every bit as problematic as a thin std, just with a different set of caveats.
If you freeze base64 into the standard library, it’s frozen. No more even slightly incompatible changes can occur, and so you’re stuck with something inferior for ever after. This is exactly the urllib/urllib2/urllib3/requests situation. This is the “standard library is where things go to die” problem.
Rust’s chosen standard library strategy is in no way wrong; it’s just a different set of trade-offs, and incidentally one that experience says is decidedly preferable for a language of Rust’s complexity (wherein there is routinely not one simple and obvious way of doing a thing, in contrast to languages like Go, Python and JavaScript which have distinctly lower hazards to freezing things into the standard library, yet even their standard libraries have wilted in places).
I think it's you who are ignoring the problem. You say that by freezing a particular Base64 interface into std, Rust programmers lose the opportunity to take advantage of more efficient interfaces. But that's obviously not the case. std can just host the best general-purpose Base64 interface, a "good enough" implementation, and people who are fussy about their Base64's can use any crate they like. That's what happens with the Go standard library, and, in fact, it's also what happens in Rust (for instance, with the Nix crates).
I don't think you've thought this all the way through. This last response of yours seems more reflexive than contemplative; for instance, it suggests that breaking changes are no big deal, because they're small breaking changes. That's not how breaking changes work! Indeed, I feel like everyone who's ever worked in a totally undisciplined package ecosystem (I'm not saying Rust is one of those) knows what the end result of this is: hours of brain surgery figuring out how to reconcile incompatible dependencies-of-dependencies.
> You say that by freezing a particular Base64 interface into std, Rust programmers lose the opportunity to take advantage of more efficient interfaces. But that's obviously not the case. std can just host the best general-purpose Base64 interface, a "good enough" implementation, and people who are fussy about their Base64's can use any crate they like. That's what happens with the Go standard library, and, in fact, it's also what happens in Rust (for instance, with the Nix crates).
nix is not a good example. Nix gives _additional_ APIs that std doesn't provide that are specific to Unix.
This isn't about std-having-base64 preventing the use of external third party implementations. This is an absurd interpretation of what chrismorgan is saying. They're saying that std's implementation is frozen forever. We can't change it. One of three things occurs:
1. We got the API right and everyone is happy in every use case.
2. We got the API really wrong and now we probably need to deprecate it.
3. The API is too simple to permit maximal performance, and now people need to know a decision procedure to choose between std and a third party library.
You seem to think (2)+(3) is okay, but I'm actually not a huge fan of either. (2) causes churn and ecosystem pain. (3) causes confusion and becomes the butt of a joke: "yeah you actually shouldn't use the base64 stuff in std since it's so slow, so just use FooBar's implementation from crates.io instead." It's not unlike the JSON situation in the Go world: I am routinely having to replace the standard library's JSON implementation with a third party one that's faster. The difference is that "good enough" in Go-land is much more culturally acceptable than it is in Rust-land. In Rust land, as I said in an adjacent thread, everything gets sacrificed at the alter of performance. When people come to Rust, they generally expect to get the fastest of everything. Otherwise, it's not good enough to supplant C or C++.
The result of this is that adding things to std is a risk. So when we add things, we need to do a risk analysis and evaluate its benefits against its costs. When you nitpick a specific example like base64, it's easy to lose sight of the bigger picture here. Bringing base64 into std wouldn't solve the problem you're complaining about. (If it did, I have no doubt we'd do it, because it's base64. Probably we can get that right.) What you're really advocating is a philosophical/cultural change that will dramatically increase the size of std, and thus increase the risk we take with respect to introducing things that people will ultimately wind up not using. We don't want to do that. We want the ecosystem to work that out so that they have room to evolve.
> knows what the end result of this is: hours of brain surgery figuring out how to reconcile incompatible dependencies-of-dependencies
This generally only occurs for public dependencies. base64 is very unlikely to be a public dependency. Folks who maintain foundational public dependencies are generally quite careful about pushing our breaking change releases. (`rand` is a good counter example, but it looks like they've slowed things down as of lately.)
I'm not a huge fan of taking forever to get to a stable 1.0 for core ecosystem libraries, so that's definitely a knock against our approach. I'd rather see folks commit to a stable interface for some years even if the API isn't perfect in order to minimize churn.
I can assure you that my position is not some reflexive HN drivel. I've been on the library team since the beginning. I've thought a lot about this and lived through the trade offs. And chrismorgan is right: this is a decision with trade offs. There is no obviously correct choice here. I probably hate bringing in dependencies as much or more than you do.
Just a note that I did read this, wanted to reply, but felt I needed to write a reply that did it justice, and then it fell off my radar. We will meet again on the fields of language battle, burntsushi! (Thanks for taking the time to write this, even though I disagree with a bunch of it.)
You could use Rust editions to allow evolving the standard library, perhaps - they changed the prelude (slightly) in Rust 2021. That might come with its own problems - on the other hand, you can't actually tell the major version of any given library from code (unlike with Go), so changing the definition of a standard library package upon an edition change.
Are you referring to base64 here? If so, then you are very very wrong. I've written lots of Rust programs over the years and have needed base64 precisely once IIRC.
Take expected (which missed C++ 20 but may land in C++ 23). The idea here is like Rust's Result type†, but C++ lacks a sum type so it's instead presented a bit like a smart pointer. You check it for errors and then just dereference expected to get your answer if there weren't any errors. Simple. If you can't be bothered to do error-checking, no worries, dereferencing expected when it's an error throws an exception, and your usual C++ exception handling applies.
But no, the committee says this doesn't look dangerous enough to us, so they required it to be rewritten to say dereferencing expected when it's an error is Undefined Behaviour instead of throwing an exception.
Imagine deliberately adding more Undefined Behaviour to a programming language (technically in this case the language's standard library) in response to work to compete with safer languages. It's an act of self-sabotage without question.
† Unlike exceptions, in Result your error is just more data, to examine, process, store and use now or later or not at all at your option. An exception changes control flow in the program when the error occurs, which may not be what you need at all. This is crucial to some designs.
Yeah, the "there should not exist any language underneath besides Assembly" motto pushed to extreme, is what might eventually sink all efforts to make C++ safer alternative than plain old C.
I don't see this part though. "Leave no room for a lower level language" doesn't require you to go around defining every mistake as Undefined Behaviour, that's crazy. A different part of the committee managed to see in 2021 that if your format can be statically detected as bogus (e.g. you tell the formatter you're giving it a string, then hand over a floating point type variable) then the compiler should be obliged to emit a diagnostic and give up compiling the program, not hand you a program that it knows can't work and enjoy your humiliation when it fails in production. That's the sort of safety improvement you'd want to see.
As I understand it the runtime performance-at-all-costs school, which includes people from Google, don't want the C++ language to be deliberately unsafe they just don't prioritize safety over performance. This group won't allow C++ to have runtime bounds checking for the array subscript operator or its overloaded equivalents and I think they're wrong about that, but it's not making the problem worse. Stuff like ranges, views and iterators (even though C++ iterators are incredibly ugly) mean in idiomatic C++ today you're rarely reaching for this particular gun that's supplied loaded and pointing directly at your feet. In contrast what was done to expected just seems like self-sabotage. If it's too slow then don't use it. Or rather, as these same people would be the first to point out, measure to check your intuition that it's too slow and only then if it really is don't use it.
That is exactly what I mean, the leave no language underneath agenda is being pushed by the performance at all costs crowd.
In all these years I have left RTTI enabled, used exceptions, tried to keep bounds checking enabled in STL types, and it was never an issue in production.
std::array will instantiate function templates for all used array sizes, lengthen compilation time, and be hard to use across separately-compiled code without putting code into headers. It has its uses, but more often I'd prefer T[N] with bounds-checking, and, in another world, a builtin variably-sized, non-resizable array type; basically a sort of absl::FixedArray[1], but with the semantics of operator[] and at() reversed, so that bounds-checking is the default. Sadly, nothing like that is included in the standard, so won't be a lingua-franca.
That is the thing, the standard doesn't require bounds checking for operator()[], while it does require for at(), but it doesn't forbid it either, it is up to the implementations.
Rust and JS both have a very small standard library. This is what unites them in a 'dependency hell' of having to handle thousands of packages for most projects.
Other languages that don't face this, e.g. Python, have very large standard libraries which already include a dizzying array of features. E.g. take a look at some of the things you get with standard Python[0].
- sqlite3 (DB-API 2.0 interface for SQLite databases)
- bz2 (Support for bzip2 compression)
- sunau (Read and write Sun AU files)
- netrc (netrc file processing)
- curses (terminal handling for character-cell displays)
- mailbox (manipulate mailboxes in various formats)
The obvious compromise would be to let libraries bake as third-party projects for a good long while until there's a fair amount of consensus that they're the Right Thing To Do, then pull them into the standard library. Java's Joda-Time would be one example.
Oracle doesn't own Rust though, there's no real push to onboard these features besides the general curb-cutting efforts for newcomers. For the most part, 'integral' crates like serde and tokio have been community-managed without issue so far, so it doesn't make much sense to pull one open-source developers passion project from them and pass it to another.
I don’t think this is a characteristic of languages or a large ecosystems. It’s a side effect of dependency managers making it seem like zero cost to include another library. Including one dependency can end up pulling in an entire ecosystem of connected dependencies, when the perhaps one function in the original dependency.
One recent build I did, a small library included a dependency to build its documentation. This was a full-fledged documentation project, which pulled in hundreds more dependencies.
It would be interesting to see a display of “cost” in projects indicating how much additional dead weight each individual dependency pulls in. This might lead to slight better decisions on dependencies to choose.
> It would be interesting to see a display of “cost” in projects
cargo-bloat is kind of along these lines, although it only measures the absolute size of each dependency (as opposed to "dead weight") and has some platform limitations.
" it is a necessary side-effect of large ecosystems"
It depends a lot.
Java has a huge ecosystem, but almost all of the core functionality is provided for you.
Like everything there's a 90/10 rule in that 90% of the 'things we need' really amount to only 10% of the total packages out there, so it actually makes sense that the people backing the platform provide for it.
Language, compiler, debugger, documentation, tutorials, standard libraries etc. - it's all part of the platform.
Looking at it that way, it becomes a little more obvious why even some cool languages don't quite breakthrough.
As another commenter pointed out, it may relate more to the size of the standard library than to the size of the ecosystem. Which your comment seems to emphasize as well.
True in js/ts world this is a problem. Since couple of years I've been experimenting with more radical fat trimming on dependencies. It works very well on backend code. I belive shallow or no dependencies is the way to go.
What seems to be missing is standard library as constellation of well designed modules/packages. I decided to just do it for myself (and anybody who finds it interesting). I don't think it's going to fly high but if somebody is interested the link is here [0]. I'm adding modules every few days.
The aim is to have something worthwile to show around the time node v16 goes lts. I have some experience taking care of high stake business critical systems. This is fun side project as a hobby atm. Modules being dropped there are reflection of problems I'm facing during my day-to-day work.
You'll see there some non conventional code. Large parts are inspired by functional programming, ocalm and simplicitly in general. Personally I find it exciting how well this kind of approach fits into production projects.
The nice thing about this approach is that you can have niche leaf packages/modules as part of standard library (constellation) - if you don't want to use it, it doesn't matter, it wont incur any cost. For example order book module or exchange matching engine - those can simply live there, be useful to people who care and have zero impact to people who don't care at all. I find it to be an interesting difference from builtin, shipped library.
Large package ecosystem is a sign of popularity. With time libraries should converge onto shared approaches. Interestingly practice shows wide spread of duplication. I think more effort from big players with wide reach could reduce it. Ie. not me doing stuff but Microsoft or others picking up the ball.
Yeah Go just has those as internal dependencies in the stdlib.
That _is_ the tradeoff - Rust builds up from nothing and makes it friction-less to add a new dependency. Go starts at a higher level and the friction for adding new stuff is high. And some of the high-level stuff, like the GC, can't be removed.
Rusts ability to squeeze into small footprints is a huge part of the appeal. I don’t think that that excuses the exceptional difficulty of writing a web server or client in rust relative to other languages.
How often are folks writing applications that don’t need to connect to the web these days?
I just tested this. It took me 43 second to build a new project with reqwest as a dependency. The second build took 0.55 seconds. If that's exceptional difficulty, I'm underpaid.
Maybe the lesson here is to update Rust if you haven't used it in a while?
I didn't find writing a simple web server or client in Rust any harder or simpler than in e.g. Python. What is this exceptional difficulty we are talking about?
> How often are folks writing applications that don’t need to connect to the web these days?
In Go? Rarely, I would expect. In Rust, I'd assume connecting to the web is the exception rather than the norm. Most Rust projects tend to be lower-level infrastructure & systems programming.
This probably explains, and in turn is explained by, their different choices re: http in stdlib. They have much different aims and target use-case.
`ureq::get("url").call().unwrap().to_string().unwrap()` with a stopwatch should get you where you want. Add ureq as a dependency, of course, but I found ureq to be better than reqwest with about half the dependencies.
For example, one shock is that you can't print! or dbg! anything (without adding Debug/Display to each type).
The reasons?
Where it will print in a Airpod? yeah: Rust is made to work in places where NOT exist a place to print to. Or where NOT exist a filesystem, or HTTP because not exist TCP because not exist a network card.
So, in Rust all start in the deepest of the deepest bottom.
---
I have a good metric to select frameworks/libraries:
- If is too complex, NO except if is the only game on town
- If have crazy deps setup (ejem: OpenSSL, bastard mess of dep!) NOOOOO. (except if is the only game on town)
- If is too big, damm be good and solid (ej: PostgreSQL)
Whatever the lang, this have served me well, and when I have been lazy (I get a deps to OpenSSL and waste 1 week fixing CI builds with it then remember this and cut that mess) the pain quickly fix it.
So, next time you get a trouble like this (in Rust or whatever) ask what alternatives you can use instead!
Rust went for "evergreen" approach like Chrome. It focuses on making small gradual changes, quick and painless upgrades, and provides stability via backwards compatibility in the compiler. This allows everyone to only target one version: the current one.
Debian's philosophy is the polar opposite, so Debian's Rust version is doomed to be perpetually obsolete.
All past Rust releases are available. They can be easily installed via rustup, and multiple versions can coexist on the same machine (even run concurrently). That's even easier and more flexible than using old Debian packages.
In addition to what others have said, reqwest is notorious for being one of the more heavy-weight HTTP libraries, but luckily, it's far from the only option: https://www.arewewebyet.org/topics/http-clients/
I think several of the responses to your comment are misguided. If you believe them you'd think that the reason this is to be expected for Rust (and even normal! good!) is that (a) Rust is a systems language, (b) Rust deliberately has a small standard library, and (c) it doesn't matter anyway because compiles are cached.
I'd say, first of all, that it does matter, because very few projects with hundreds of (transitive) dependencies are going to be able to take security sufficiently seriously. And because minimizing the number of other projects you rely on is good engineering (see: left-pad).
I don't think the other two explanations are much better. Compare C as an example. With C you can use libcurl which is well-maintained, actually supports far more protocols than simply http(s), and uses far fewer dependencies to do it. It also runs on an insane number of platforms, so portability isn't the issue here either.
I think it primarily comes down to cultural differences. Rust is a newer, "cooler" platform, and attracts a very different audience than C does. Because of that Rust's users will be more apt to follow common modern programming practices: using a language's built in package manager and so on. (This culture both contributes to the development of languages like Rust and is fed by them.)
Remember the XKCD "Python" comic? [1] Python is what I would consider an older language in terms of culture, but it captures the idea of being able to just import a library that can already magically do everything you want. "Programming is fun again", you don't have to write boilerplate or worry about managing dependencies any more. The problem begins when everyone has that attitude, including the people writing libraries that will be used by others. Importing a library that can instantly solve your problem is just one line in your Cargo.toml away - who cares that it's got 200 dependencies!
Python's fine until you want to use a library that's not part of the standard library, then it's a nightmare. Especially if you want to distribute your code, not just run it locally. (Ditto for C and C++.)
I was prototyping something that could monitor a few api endpoints in parallel, collect timing stats using sqlite, then output a static status page, statistics and graphs.
Kind of like what smokeping does, just wanted to see what a from-scratch implementation that did just what I needed might look like.
Ended up being about 400 lines. Only dependencies were go-sqlite and go-chart.
Consensus seemed to be that reqwest is the way to do this using rust, since the stdlib does not include an http client.
Adding the dependency on reqwest tried to pull in over 100 dependencies and after building for a while, it failed somewhere along the way because my rust version was too old. Now, maybe some of these were optional dependencies, but they all got pulled in automatically.
so, when TFA says
> Another obvious advice is to use fewer, smaller dependencies.
I just went back to go, which can do the crazy obscure task of making an http request without requiring 100 external dependencies.