First of all - Linuxes and alike desperately need something like this shell becoming standard. And second - I always wonder how all these great minds who where throwing rubbish at Microsoft ever since... somehow forgot to mention that same company created important product called PowerShell like 20 years ago. And you know PowerShell is so good already you can easily drop it in any OS/X, Linux and it actually works. No one, except for Jeffrey Snover figured it out on time, there is great interview here - https://evrone.com/blog/jeffrey-snover-interview.
But for PowerShell to become Python on Perl steroids on top of all the .NET and COM+ there is, well, of course it took years, and top engineers to work on it, but also was needed an unified interface (API) to the underlying OS magic. Which is not there in Linux the same sense .NET became "available". OS/X has something similar to COM+ called CORBA which was left half-baked at best. Python is not .NET, sorry. Not anywhere near.
Someone may say the UNIX ideology is about your programs doing one thing at time, and having this STDIN->munch->STDOUT approach. But these streams never got any structure, and ppl out there been creating great tools to fill this gap, tools such as - jq, mlr, xidel to name a few. perhaps we should include ack, ag and rg in the list.
You know, guys, you hard-loving bash die-hards and sed+awk magicians, you should give this PowerShell thing a try before inventing the wheel again. And before legacies you wrote 20 years ago become as difficult to support as COBOL. Oh yeah, and PoSH is open source, based on open-source runtime, and available in brew/apt/yum...
Disclaimer: I do not work, and have never worked for MS (Guido does now!). Have used DOS since 6.22, FreeBSD since 8th version, Debian since v5, and Windows since v3.1. I did Perl for 20 years, actually I gave Perl classes in some university, still doing fair amount of Python, and accidentally discovered PowerShell 3 years ago. I'm not a .NET advocate, although I can tell the good parts.
In defense of the Microsoft-haters, powershell is a fantastic idea with terrible ergonomics. When they first released it I was very excited but, well, it’s just awful to use.
Terrible Get-Finger-Twister -Recurse -Awfully commands, quirky aliases (you can type “curl” but it takes none of curl’s parameters because it’s an alias for Get-WebRequest-Annoyingly or whatever they called it), simple scripting things such as “include file relative to current script” (and not relative to current working directory), etc etc: anything that should’ve been simple and easy was neither.
I’ve stopped paying attention, so maybe they fixed this since then but it really wasn't a very well done system during its first decade or so.
The verbosity is a positive, so long as you understand the constraints in which it works. Cmdlets are specifically Verb-Noun, where the list of verbs is restricted to an approved list (Get, Set, Add, Remove…) and the noun naturally follows what you might expect.
Want to get an organisational unit from Active Directory? Turns out the command is Get-AdOrganizationalUnit. Discoverability is built in. All of the switches are the same. Is it -R or -r for recurse on the Unix command to list OUs from LDAP? What is the command name, anyway? I dunno, but I know the flag I need is -Recurse on Powershell.
Want to dump your DNS zones? Get-DnsServerZone is too verbose, obviously, so “cd /var/named/data && grep '\\(. *A *[0-9][0-9]*\\)\\|\\(..*CNAME..*\\)' db.*” is a much better option (‘scuse the escaping on that regex). Glad we could avoid the long command names.
Born of experience, maybe. Don’t know what the command might be? Get-Command dnszone* saves the day. I suppose I could ask GPT-4 for the regex for the bind dbs, otherwise.
These are all things I can teach juniors in about fifteen minutes flat. You don’t need to learn three different Turing complete languages, all of which overload the same set of flags with different operations, just to run an operation against a bunch of files.
Familiar, shorthand, and obtuse is not better just because it’s familiar.
Verbosity, for everyone who has actually written any bash/posh, any code tbh, verbosity is something to cherish. Because, u know, someone is going to eventually read it, and at best they will not be cursing the previous author. So this verbosity in posh is very well calculated, and again - sadly misunderstood by bash/perl hackers who very often care only about themselves and not their fellows.
Yeah, it's in this weird space where the underlying technical direction feels correct but it's just too cumbersome in actual practice.
The awful text manipulation in posix shells feels wrong on a lot of levels, but when you're just trying to accomplish something quickly the "just manipulate strings, always" api is faster to actually work with interactively.
Powershell ends up in this weird space where it's clearly better for writing actual code, but the act of using it to do small interactive things is needlessly difficult.
This is why posix shells stick around I guess - they suck as programming "languages" but they're good enough to do the job, yet they're just really fast to use interactively.
This is why I created my own shell. I wanted something better than Bash but less cumbersome than Powershell (plus Powershell wasn't even available for Linux back when I started).
There is a massive gulf between Bash and Powershell -- they're opposite ends of the spectrum -- and it's entirely possible to have something that cherry picks the convenience of Bash with the power of Powershell. Albeit compromises would still have to be made.
There's quite a few alt shells these days now though. So other developers have clearly had the same thought as myself.
Maybe it is time to rethink what a terminal is, and get rid of the restriction to be text-only? There is really no reason for that, other than Typewriter heritage.
There are some terminals that can display inline images, but those features aren’t commonly used.
Why?
Text works. Text is the lowest common denominator for working with data. This is why terminals are still primarily text based. This is also why stdin/out/err are also commonly text based (when the user is expected to view it). Text is simple and easy to parse. Binary data exchange formats (or objects) come in and out of vogue, but text just keeps on working.
Text does work, but text data exchange formats come in and out of vogue just like binary. And when any structure is needed, we typically get some ad-hoc text format that doesn't interoperate. JSON has made things better here, but it's a bit of a verbose eyesore for humans.
I think the lack a structured (text based) format is a definite shortcoming in UNIX. Whitespace delimited lists of strings do get you surprisingly far (although some things are a total mess, e.g. filenames with spaces). But a simple more structured format (e.g. stripped down YAML with maybe non-prefixed lines being a list/iterator of strings for backward compatibility) would make the shell a lot more powerful.
You should be able to include binary in these, but make it render nicely in the terminal (e.g. like browser devtools do).
This doesn't also exclude having binary. And binary data is piped though programs all the time, but it makes including other data painful. Also don't see any harm in being able to show images in the terminal when wanted.
While I think that ASCII including a specific record delimiter character (0x1E) that isn’t used is a missed opportunity, I actually think the lack of a set interchange format is a benefit.
No format will work for everyone and if there was one set format, we’d still have people complaining about how it doesn’t work for their usecase. We'd also have had to (finally) answer the question of how to send the schema with the data.
JSON (and XML and YAML and…) are all methods to try and adapt a text based data interchange format. And they all work roughly the same. You ultimately need some kind of processing library to full read them. But working with the data in any sort of structured context requires specialized tools (ex: jq).
Or you can still grep it, just like I do to manage my tab delimited data, and get part of the way.
I think the chaos of having no standard is kinda the point. Text is the lowest format that is easy for people to work with. We can visually parse it pretty easily, so the tools that support text can be pretty universal. Grep, sed, and awk can do a lot.
But to do anything more, you’re always going to need some kind of specialized parsing, even for tab delimited or CSV data. So, let the terminal work with generic text and be “good enough”.
As an example: you mentioned JSON and YAML and suggested a slimmed down YAML might be more powerful. I deal with columnar data with tens to hundreds of columns and thousands of lines. This wouldn’t work with either JSON or YAML… it’s too verbose. Instead I use (compressed) tab delimited text files for their ease of parsing in many languages. But those won’t work for many other use cases.
With data exchange, there is never going to be one right sized option for all needs. If you can accept that, then the “right” option is to choose none of them.
I'm not saying that processes wouldn't be free to spit out raw ASCII or whatever if they so wanted, but there should be some sort of POSIX-type standard that would spec typical output formats (e.g. standard --yaml flag). I see no urgent need for formal sending of schemas.
It's a disgrace that most standard tools don't have even an option for any machine readable output. Or when they do, they are ad-hoc and there are ad-hoc parser tools/flags for those. I.e. you need a lot of (badly implemented) specialized tools.
You can grep, sed and awk YAML all you want. It's just ASCII (or UTF-8) text.
ls is a good example. Getting the -l output would be very useful for many applications. And would simplify ls itself (e.g. sorting could be done in a separate generig util, which would work with other tools as well). Would also take care of the whitespace horrors.
Many tools don't have "stable CLI interface", which makes them difficult to automate. Apt and git come to mind immediately (although git generally allows specifying formats). Some tools (e.g. ffprobe) support JSON output, which provides very easy "bindings" for it (the actual libav APIs are quite painful).
Is that so? Because the way I see it, there are a bazillion different tools, plus decades of accrued knowledge only related to massaging data into text and back to structures. How many human brain cycles are wasted to data conversion on the Terminal?
Text is a lousy abstraction. It may have worked well 50 years ago when computing was limited to researchers on American universities, but the digital world back then has little in common with ours.
Acting like extending the shell to anything beyond plain text would require switching to fancy new serialisation formats every other year is a red herring. The only thing it takes is admitting a shell isn’t used on typewriters but modern computers, where humans use high-resolution screens and programs can exchange structured data. Everything else follows, and is well suited for standardisation.
But that will never happen as long as grumpy sysadmins insist that cobbling together grep and sed in some arcane incantation is the only thing we ever need.
When the alternative is to fire up some mess of an IDE with some nightmearish COM/.NET/ProtoBuf/whathaveyou overengineering marvels, I take grep and sed every time.
Simple text-based structured data format (something like YAML) with possibility to embed binary would be the way to go IMHO. And of course one should be able to show graphics in the terminal.
But we're not gonna get it. Open source tol is being rapidly enshittified by corporate EEE with their badly overengineered enterprise crap. :(
I don't know why you seem to think we can only have "super-lightweight 2kb C binary" or "giant over engineered Java-IDE" - there's lots of space in between. I don't think anybody would want to give up snappy terminals, but at the same time, why does this have to imply loosing out on half a century of user experience research??
It's been that time for decades. And it has been retought many times but never seems to catch on.
I think even unices have abandoned the UNIX philosophy and turned largely into "there's an app for that", so maybe there's not much interest for a better composable UI.
I almost daily daydream of the alternate timeline where MBAs and greed didn't destroy Lisp Machines. And lament that we ended up with this horrible pile of hacks that makes everything needlessly painful.
Is that what happened? MBAs and greed destroy a lot, but I’m not sure about Lisp machines. Beautiful hardware and software, but unbelievably expensive, and custom hardware was bound to lose out on performance.
The "MIT Lisp Machine" were developed as prototypes at the MIT AI Lab, with DARPA money (Defense Advanced Research Projects Agency, a research funding part of the US military). The machines were mostly hand-made and did cost a little under $100000 to make. Only very few (200 ?) were ever made for use at MIT.
The military wanted (!) to have better/cheaper generations of these computers for Research&Development and for deployment of applications. So it was decided to set up commercial companies: LMI and Symbolics were founded. LMI later sold a license to TI.
The market for these machines was high-end: installations could cost anything from $50000 to $200000. In the first years these machines mostly went into military and government projects: military was sponsoring the deployment and first improvements (like the development of Lisp-specific CPUs).
Basically that's not unusual from many other government sponsored research, which later got marketed by a startup.
The Lisp they were using was standardized starting 81/82 (also with the help of Symbolics). This standard then was ported to PCs, Macs, Workstations and Mainframes. The Lisps were available free or commercial (a few hundred dollar to several thousand, depending on the vendor).
The Lisp machines itself remained tied to special hardware and mostly died in the late 80s and early 90s.
Various other free and commercial offerings of Lisp for PCs/Macs/UNIX systems died or survived...
The whole thing was initially financed by a large customer: the government (-> DARPA). This customer demanded commercial versions to be able to use them beyond the few prototypes. Roughly 10k machines (my estimate) were built and sold. At the same time cheaper Lisp systems were developed by universities and commercial competitors. Free / Open Source versions of Common Lisp (the Lisp dialect then also on the Lisp Machines) were developed by CMU (CMUCL), in Germany (CLISP, CLICC), in Japan (KCL), ...
And yet, there's no code completion for powershell so nice, as fish provides. In this area, Powershell (plus addons, like oh-my-posh) just mimimcs existing popular bash/zsh capabalities.
Fish isn't UNIX, and stating that Powershell just mimics existing popular bash/zsh means your not aware of Powershell capabilities versus bash/zsh.
Since when do bash/zsh offer a full sane programming language, not relying in external executables, a full blown language framework ecosystem, ability to load shared objects and importing public symbols as functions, and interoperate with D-BUS (as COM counterpart) as if client bindings were functions (again withouht external processes)?
That is the key. I will never understand why we’re stuck in a world of TeleType emulators, while having all the technology available to make Terminals with proper autocomplete, suggestions and inline widgets possible.
And we’re probably not getting there if people insist on cryptic shortcuts instead of semantic naming, just because they’re inclined to type everything out manually. It’s like those developers writing their code in Notepad++ and pretend that’s the way.
While I tend to agree, a lot of commands also have pretty good aliases.
For example `iwr` for Invoke-Web-Request.
Also the full names are only expected to be used in scripts to maintain readability. No one types that in the console. Also ALL Parameters can be abbreviated if they are unique. If no other Parameter starts with R then -R is enough for -Recurse. (there are some quirks with this, but generally pretty intuitive)
And using select or for-each on actual types is magical, instead of grepping/shedding/parsing a maybe standardized string, with completely unsensical names (like grep or sed)
If you give bash and pwsh the same time and love, they will become both pretty ergonomic
>Also ALL Parameters can be abbreviated if they are unique. If no other Parameter starts with R then -R is enough for -Recurse. (there are some quirks with this, but generally pretty intuitive)
So, adding a new parameter starting with R to any command is a breaking change that will crash every script using it? Sounds like a terrible idea.
I wrote Marcel, and its forerunner osh, also recently on HN. Osh actually preceded powershell, and in fact I don’t recall it receiving any acknowledgement from Microsoft. To be fair, I’m sure they never heard of it.
The idea of piping objects has been rediscovered many times.
You are missing part of the picture. Object tossing is one thing, having access to the plethora of apis you get from Com+ and .net in a finger snap is another.
Of course - kudos for your efforts. Sadly went unnoticed for vry long.
I don't see anything connecting the marcel site and the oilshell.org site, am I missing something (like, say, "this is a different osh") or did you just not get around to writing that up yet?
(to be clear, this isn't a complaint, I'm just confused and asking for help becoming unconfused, I'm aware how lists of things to get around to writing up tend to be a grow-only datastructure for us all ;)
Today, we use json. How do we integrate it into our shell scripts? That's right, we simply make a tool (jq, miller) that takes stdin json, and _continue_ to use the agnostic, untyped tools alongside it, because they are just that, agnostic.
In 20 years when we use BGHY or whatever other format, we can use it alongside existing tools by simply... Making a tool that takes stdin in that format.
But it still sucks because of the unstructured data. I added JSON support to Marcel in an afternoon, and still have structure data flowing through the pipelines. JSON first, then nested Python ducts and lists from the JSON translation.
For all the praise it gets, PowerShell is basically the REPL environment from Xerox PARC (Smalltalk/Interlisp-D/Mesa/Cedar), ETHZ(Oberon), Lisp Machines, finally making into mainstream computing.
On OS/X you would need to look into Apple Scripting, XPC and Objective-C runtime, for a similar COM like experience, by the way.
I hear you, and I have tried it, but the commands feel gratuitously different to me.
Also they squandered the opportunity to make the output interactive like with the Lisp machines. That would also have helped smooth the learning curve.
The worst part of it is they aliased a bunch of common posix command names to their power shell equivalents but then they have different flags or semantics. Works fine until it trips you up destructively.
They even made their own broken curl, which to me is straight up trademark infringement (ethically speaking at least, IANAL)
Making alternative implementations of the core utilities is only customary, but the curl name is is directly tied to the reputation of Daniel Stenberg, which is one of exceptional skill, integrity and attention to quality and security.
> You know, guys, you hard-loving bash die-hards and sed+awk magicians, you should give this PowerShell thing a try before inventing the wheel again.
I did several times over the years, but it simply didn't stick. If I have any non-trivial scripting tasks to do I run a Python or Deno script instead (at least those run anywhere, no matter if bash, zsh, fish, cmd.exe, PowerShell, and also run on any CI system without hassle).
It is funny that sed and awk are often uttered together as one incantation, while in fact they are very different tools.
Sed is, I dare to say, one of these quick-hack tools Unix is full of; awk in turn os a great mini-language with the defaults and builtins allowing for very concise programs.
What they share in common is processing lines and using regexes, and here's where similarities end.
While there is this sentiment to attribute a notion of "oldness" to awk, it is very powerful tool and is being implemented every decade (hello, GoAwk!). The rhetorical question, if it's so ancient and legacy, whether it's being reinvented every now and then?
This is a little detour from the original point, that was: processing structured data. Going back to this, I think there is important distinction: are we ok with the tools processing raw input/output and interpreting structure internally, or are we in a desperate need to preserve the structure in the piping mechanism, like Marcel does?
The latter ofc gives more power; but the former retains the compatibility with all the tools that don't know the structure.
This is a false choice. You can start with minimal structure, add it for intermediate processing and then go back to text, if you’d like. Marcel can run native commands, piping their output into other Marcel commands, as a steam of strings.
To address one of your other comments: "but the former retains the compatibility with all the tools that don't know the structure."
Marcel plays nicely with host executables. The strings generated by an executable can be streamed into marcel commands. Marcel structured output can be piped into executables as strings.
Example 1: Compute distribution of word lengths. This pipes cat output into marcel commands:
I believe you are missing the point. It's not about sed and awk. It's the string oriented approach to data requiring heavy usage of these tools for parsing the unstructured text.
The suggested alternative is the object oriented approach, just like PowerShell and the OP's project are based on.
Well there are two points. 1) Piping strings is dumb. 2) Multiple incompatible sublanguages has a long learning curve. How many people can just write complex awk/find scripts without checking the manual? Using marcel, and knowing python, I can do those complex things without RTFM.
I use bash, find, sed and awk every day, usually ten to twenty times per day. I've also built a lot of scripts for my company's workflow with those tools.
I've never read the manual for any of these. They all work on the same concept of lines and words, which is intuitive and easy to interact with on the terminal.
Piping strings and making lines and words the units of work are the great ideas of UNIX.
I sometimes use python instead, but while it yields better programs, it takes a lot more time and just isn't suited to being written in the shell, especially with the syntax being whitespace-sensitive.
I was immediately thinking powershell. I have been doing some Powershell scripting for building chocolatey packages: it weirdly did not feel like shell though and my muscle memory of POSIX commands fails me on the terminal I guess.
What I was wondering though: can you easily access Powershell data types from python?
You're right on the text vs structure point, but then pwsh is unfortunately slow and unergonomic enough that makes the transition from a better modern shell (so, not bash) harder, especially with a better and more common language like python
Have to agree with the slowness, I just used powershell for a month or two and thought it's great.. Until one day I accidentally started fish and it felt so blazingly fast that I couldn't go back to pwsh anymore..
Powershell structured pipe might look beautifully to architecture astronauts, but it is a step back.
The value of the component is the higher, the more components it can communicate with. Pure stream pipes in unices are the universal language; object pipes in powershell break the language in many incomprehensible fragments, and each component there needs "translator" for each specific fragment it can understand on input or output. Yes, it means that in the unix environment you may need some tool to be a middleman to extract only few things from the bigger stream; but these tools are also universal, unlike the need to understand specific piped object in the powershell world.
CORBA wasn't left "halfbaked". It was too complex for its own good, and unlike COM, there was nobody interested in pushing it further, so it was left behind. For a more lightweight messaging, there are XPC (macOS) or dbus (linux).
Hi, I’m the author of Marcel. If you like it and want to work on it, I can use volunteers, for developing features, debugging, doc, porting, improving the website, you name it.
That's in a completely different business. Trademarks generally do not cover all possible uses of a word, only the use of that word in the segment it was approved for.
Oh, interesting, Wix wasn't even on my list for consideration. I like the look of it, but I also want to pull in some of my documentation from docstrings, possibly, so I think reStructuredText is a leading contender. That's what I'm getting ready to try setting up in my github workflow to see if I can get it to automatically format and publish the documentation to a github pages site.
I'm working on a not entirely dissimilar project (mine is focused more towards: fabric/ansible/cookiecutter type workflows, leaning on a lot of shell semantics. My workflows and hopefully soon documentation may be useful to you, I'd be happy to contribute them. I auto-build and publish releases to pypi. Project is: https://github.com/linsomniac/uplaybook
I wrote marcel. In my view, nushell and marcel are very similar. nushell has tabular output, which I haven't thought was all that useful.
Marcel is python-based. The nushell guys have had to invent a lot of language. By basing marcel on python, I don't have to invent language. Any logic to be added is expressed in python. E.g, do a recursive listing of files and find those that changed in the last 3 days:
ls -fr | select (f: now() - f.mtime() <= days(3))
The stuff inside the parens is a python function, applied to a File f, piped in from ls. (Marcel lets you omit "lambda".)
That said, every time I look into one of these shells, I end up sticking with bash. When it gets to the point that I need something more powerful than bash and the Unix command line environment, I'd just as soon write a stand-alone program which I can check into version control, use on remote systems that don't have one of these fancy shells installed, add proper tests, etc.
This is a neat project though. I'll keep an eye on it.
Yes, I know that, but it's also exceedingly rare for a filename to contain a newline and using -exec is extremely inefficient. I also wrote that on my iPad. Typically I'd use a combination of -print0 and xargs -0 for anything more formal than a one-liner on the command line.
The larger bug in my example is that it fails if any of the directories in the path contain a "." in their name!
Almost forgot: I also switch from bash to Python at a low threshold. In that case, I find Marcel’s API useful. I can then do shell-like piping inside of Python.
I actually agree with most of this. I tend to use Marcel for ETL sorts of processing, quick database access with data munging on either side, cluster access, etc. it works fine for basic shell usage though.
As for a more complex example: yes, the website has better examples, but they are also more involved to explain.
As someone who's leaned into Nushell deeply, I can tell you that the tabular and hierarchical data output is the major selling point of Nushell. For me and my team, who work with data and metadata constantly, it's an incredible productivity boost.
The major pains with Nushell are debugging pipelines that are too slow, having to learn the new language constructs, and process overhead for going in and out of Python for other pieces.
If you stick to marcel commands and python functions, then all the processing happens in a single Python process (with a small number of exceptions). Beside the process overhead, this approach also avoids serialization/deserialization costs, and even string copying costs. x | y | z just passes Python tuples from one command to another via function call.
Can you expand on your first point? Why are tabular and hierarchical output so important for your work?
Yep. It's a very clean, powerful and well thought out API. And I'm saying that coming from a place of love towards unix and its pipes. I'd almost argue it's better than the unix model. Almost.
Been able to give poweshell commands in FOIA litigation and its power is on full display with one liners where it's hard for a gov agency to say no to something so simple.
It was most notably helpful in litigation against the White House Office of Management and Budget. Was suing for one week of email metadata records (to, from, CC, bcc, time, date) in the last week of Jan 2017. They said that they had no way to complete the request. So we sent them multiple one liners that extracts the info from outlook 365, which pipes it to a csv export command. They told us that they didn't run that sort of command as a routine practice, but we found a separate lawsuit that found emails of the exact command we were asking them to run.
I need to write about it ;). We won that case just before the last election and unfortunately other things took priority at the time. But it'll happen. Just first need to write about my IL supreme Court loss...
Honestly any interactive shell + unix tools + any scripting language you feel comfortable with should cover the majority of things you may want to do on the command line.
Not exactly BASIC per se, there was some CP/M like additions. However, you're right 'cause some of those commands are BASIC extension (e.g. RUN can be used from your programme, LIST maybe, but CATalog/DIRectory or LOAD often not.) Old good time.
Marcel the author here. Didn't know I was also a shell, though I do also have one that is somewhat similar in concept.
While taking an existing programming language like Python is an obvious choice (and one that's been made a bunch of times), I think that ultimately you need a language where something like a pipe (of objects or text) is not bolted on, but a natural consequence of a more general view of programming based on architectural interconnection.
"Pipes and filters" is an architectural style. And it is itself a polymorphic style, meaning there can be substyles that pass objects and substyles that pass bytes, substyles where the filters are processes and substyles where filters are components within a process.
Integration with Python: Xonsh defines a language that is a superset of Python and shell, as I understand it. Marcel takes a different approach, defining only a bash-like shell language. Any customization is done in Python, delimited by parens. The separation between shell and Python is much stricter. Also, Marcel provides a Python API so that you can write shell-like marcel commands inside of a Python program. Shelling out from Python is notoriously ugly; the marcel API fixes that.
Sublanguages: In bash, there are lots of sublanguages, e.g. the arguments to 'date', awk, find, sed, and so on. Marcel's idea is to use Python as the sublanguage, because so many people already know it. I guess xonsh has a similar approach here.
Pipes: I think that xonsh, like more familiar shells, pipes strings. Marcel pipes python values in streams. So if you run ls, you don't get a stream of filenames, you get a stream of File objects, and you can operate on them downstream.
Database access: A stream of Python tuples is very similar to database query output. So database access is simple. There is an sql command which produces a stream of Python tuples. And a stream of tuples can be piped into the sql command, e.g. to populated a database.
Remote access: If you have a cluster, you can use marcel to upload a file to all nodes of the cluster, download from the nodes, or to execute the same command on each, streaming results back as streams of python tuples, each with an element identifying the node from which the data originated. I don't think xonsh does this.
That's not correct. In fact, there was a relatively recent Supreme Court case where a "joke products" maker (I don't remember the exact products, I think they were dog toys or something) basically made them look like Jack Daniels products. They argued it was a spoof, so should be protected. The court ruled against them, saying that even though they were selling in a totally different arena, they were still coasting on the public's familiarity with the Jack Daniels brand, and someone might think that JD themselves had authorized the spoof products.
I'm not sure that all applies here, but "Marcel the Shell" obviously belongs to and is associated with one specific brand, and the author of this nix shell is riding on that. It's not like when someone tries to trademark "Apple" in 2 different contexts.
The Supreme Court case didn't find the dog toy satirizing Jack Daniels was infringing, just that it was possible and a jury should decide rather than the case being dismissed.
Right, the court's holding simply held that a trademark infringement claim concerning the commercial use of a trademark as a trademark was outside the scope of the Rogers threshold test ("the claim must be dismissed unless the complainant can show either (1) that the challenged use of a mark “has no artistic relevance to the underlying work” or (2) that it “explicitly misleads as to the source or the content of the work.”") and is moved directly to the next step which is the Lanham Act's "likelihood of confusion" analysis.
A trademark claim here might not be dismissed under Rogers, but I don't think there is any jury that could ever find that consumers were likely to become confused about whether or not the command line tool Marcel the Shell was affiliated with the independent film cartoon character, and even if that could be established, unlike Jack Daniels, what $$$ damages were caused in trademark dilution? 3 figures?
Being legally right and being morally right are often two very different things though. Like, they even have the full domain name of the original thing. If I were the owner of the original video and the subsequent film, I know I'd be, at least, annoyed.
If I were the author of the original, I'd be thrilled. And would offer the character as the project's mascot tied to a set of conditions based on the success of the product so that, in case it fails to gain traction, it should be renamed and allow the name to be used by another product.
It's more or less the same spirit as the Roddenberry estate allows someone to use the names - you are allowed to build a warp drive, as long as it warps spacetime and allows FTL travel.
On the bright side they're unlikely to ever see it, since this one will never appear on the first few pages of search results without additional search terms.
Just using the name "Marcel the Shell" probably isn't enough to be legally actionable.
Plenty of books, films, and TV shows have the same titles, which indicates that titles alone typically don't enjoy the protection of exclusivity.
Copyrighted characters and trademarks have a better chance of that, but those aren't being appropriated here, just the name.
"Marcel" is also a common first name, which makes it less likely that the creators of the film can claim any exclusive use of it, and "the Shell" is completely descriptive in this context, so unlikely to cause copyright issues.
However, instead of passing strings from one command to the next, marcel passes Python values: builtin types such as lists, tuples, strings, and numbers; but also objects representing files and processes.
The bit before the semicolon sounds like what you'd do in a functional language like Elixir.
It would be cool to be able to do this in regular Python programs. I know some libraries (like pandas) fake it by having a method return an object of the same class, allowing method chaining, but that doesn't work universally.
This looks a lot like an idea I am having for a shell written in GNU Guile. MI am still at the very beginning and POC phase. My ideas are:
(1) making it possible to have any procedure that follows a certain interface to be usable as a "command" in the shell
(2) have a fallback function, that gives you a procedure for a standard shell command like 'ls' (or reimplement your own!)
(3) have piped commands automatically use lightweight processes (guile-fibers) to automatically parallelize work, if possible
(4) pass structured data instead of strings, unless the structured data is actually a string or a fallback command of standard shell is used (although perhaps one could write functions parsing that string output into structured data and have structured data at the next step ...)
(5) provide composition concepts, that make it easy to compose commands or pipe their outputs into other commands
(6) have readable arguments for things like input or output redirection
(7) no need to be POSIX compatible, I just want a useful tool for myself and anyone, who might want to try it at some point. Don't want to get mired in arcane stuff.
Well, many things to consider, probably tons of work hiding somewhere under the hood of all this. I also noticed there are issues with parallelizing things, if I want to keep the shell state local to a command. Previous commands in a pipeline would have to communicate shell state updates down the pipeline. So one needs to distinguish between communication of a shell state update and more input for the next command. Concurrency issues might ensue. I think I will be glad, if I can get some subset of my goals to work.
Those selects can be abstracted. They can be turned into their own parameterized pipelines, and assigned to variables. Here is a pipeline to filter files by a given extension:
This seems like it has some really cool ideas! This reminds me a little of eshell, which has some of the same features around allowing you to define shell functions in lisp. However it does run into issues moving between the lisp (sexp) and shell (string) boundaries.
This is a place where marcel's python grounding is really useful. Reading json, marcel generates a stream of nested dicts and lists. That stream can be turned back to json for output.
To me the interest in this would be as a replacement for bash in makefiles with the hope that it could be cross platform and not require most of the utilities like sed/awk etc. It's an enormous pain to write x-platform builds with make because of shell issues - even with cygwin.
For this to work one would probably want to embed python and have a pool of interpreters so as not to pay the startup cost of python on every job invocation.
"That Said" I think passing around structures is not that useful to me - one is often trying to automate programs that are not written that way.
Python's pipe handling is complicated and horrible compared to bash and that's the thing I'd really want to "buy" from this - together with more shell-like ways to copy, search, touch files that are platform independent.
To do the things I wanted to do I needed reduce and I think there's no documentation for it. It took me a while to understand the code:
ls | map (f: f.size) | red + # this adds all the file sizes together
one can also use lamdas:
ls | map (f: f.size) | red (lambda acc,y: (y if not acc else acc+y))
(this does the same thing - adds up the file sizes)
BUT: how would I then divide the result by 1024? .... or assign the result to a variable?
There are a number of other functions one can use with "red" like concat, * (times), max, min, count, and, or and a few more.
At the moment I feel like I'm playing with a very fun toy that might give up on me at any moment and turn out to be limited. It obviously has limits but I'm really enjoying it.
ls | map (f: f.size) | red + | map (size: size / 1024)
You can also leave off "map", so:
ls | (f: f.size) | red + | (size: size / 1024)
A pipeline produces a stream, so there's no clean way to get the single int result in a variable, but you can store the stream (containing the int) into a variable:
ls | ... | (size: size / 1024) >$ x
Now x has the stream. To read back the stream and print it:
> A more powerful form of scripting can be done from Python, using the `marcel.api` module. With this module, you have access to the operators of marcel, neatly integrated into Python.
This looks like an appealing feature! Having some more coreutils-like functions available in python (i.e., stuff that I wish was in the `os` module) would simplify some cases where I otherwise might have to resort to `subprocess` calls. The GPL license definitely narrows the number of situations where I would realistically be able to add this as a dependency, though.
GPLv3 is a "strong copyleft" license. It comes with some significant pros and cons compared to more permissive licenses like MIT and BSD. You can find more about the philosophy behind it here: https://www.gnu.org/philosophy
As the author, a benefit it offers you is protection against someone selling your work or a derivative of it, or redistributing it with added restrictions.
From a user's perspective, the main downside is that using GPL software as a dependency makes the user's project a derivative work, which requires them to license their own project with the GPL (or from a selection of other compatible copyleft licenses).
In that sense it's considered a "viral license," i.e. everything downstream of the GPL software needs to adopt GPL. This is less of a problem for an application, but is more troublesome for a library (with many dependents of its own, with dependents of their own, and so on). It also means that if you use it in commercial software, you essentially need to open-source all of it.
So in your case, just using Marcel as a shell with GPL doesn't put any licensing obligations on the user, but using it as a library does.
I hope that helps! Someone else feel free to chime in if I misrepresented any of that.
Anyone is welcome to sell GPL'd work, also someone else's GPL'd work, as long as the license remains GPL and source code is available to the buyers of the work, under the GPL, if you distribute the software to them. If you don't (i.e. SaaS, you run the software, it is never distributed further) then you don't have to give the source to anybody! This is what the AGPL was created to address.
Yes, in this scenario the LGPL was created so that I could create a library, and if anyone modified my library they had to share the improvements however using my library does not then apply the GPL to your code.
> The license allows developers and companies to use and integrate a software component released under the LGPL into their own (even proprietary) software without being required by the terms of a strong copyleft license to release the source code of their own components. However, any developer who modifies an LGPL-covered component is required to make their modified version available under the same LGPL license.
Yes. If they modify marcel.api they are bound by the LGPL to release their modifications, however they are not bound to release their application if they use marcel.api
There is GPL, LGPL and AGPL (probably others).
GPL code is viral, think of Linux and if you incorporate Linux code in your code them you have to GPL it, which means if you distribute it you have to give a copy of the source to your customers.
AGPL is newer and extended this, because Amazon et al, took GPL software and offered it as a service, so didn't "distribute", could make their changes and offer it as a SAAS and not offer their code changes. AGPL covers this scenario and means that you have to offer the source if you provide network access.
LGPL is almost as old as GPL, and allows people to write libraries as open source, which Microsoft could use in Windows without GPL'ing Windows but would require improvements to the library made by Microsoft to be distributed.
But as always, read the label to make sure there isn't a side effect that we missed in this chat.
The sister comment appears to be somewhat confused. The "virality" of GPL is that if you "link" a GPL program then your license must be compatible and the combination as a whole becomes GPL. Importing a library may count as a derived work/linking.
The LGPL was created to avoid this limitation and is a natural choice for programming languages and interpreters.
As an example, GNU Guile, an interpreter for the Scheme language, is LGPL licensed. Writing any nontrivial Scheme script requires importing code from Guile. If it was GPL then all scripts had to be GPL compatible; but since it's LGPL, programs using Guile modules can have any license (even proprietary).
No. Marcel is the successor to osh (http://geophile.com/osh, https://github.com/geophile/osh) which actually precedes powershell. The fact is that many many people have implemented the idea of a shell piping objects instead of strings.
Message to the person responsible for the site: they should use code blocks instead of images for their code. I say this because I can't read the text inside the images from mobile. It's way too small.
I need to revise those, and images seemed like the way to get colorization right. I'll try to figure out how to do it with code blocks, it will hopefully be more maintainable. Cutting/pasting images is ridiculous.
Thank you for all the suggestions for code formatting/colorizing. The problem is that I'm hosting on wix. Posting images sucks, as already pointed out. How do I get formatted, console-like text on wix? I'm not seeing how these tools solve that problem.
Hate to be "that guy", but it's kinda sad that they pretty much stole the name... especially when the (fantastic!) movie with the original character came out recently in 2021.
Out of curiosity, did you reach out to them? I think the project is neat, just feels a bit like you're leaning on an established (and extremely creative) piece of work instead of creating your own brand. Hope I'm wrong.
Actually, I did. My daughter loves marcel the shell, I was writing a shell, so it's an obvious choice. It is far enough afield that I can't believe they would care. But I reached out to them, and explained what I was going. They were amused, said they might want to object later, but they never have.
When I watched that in cinema I couldn't help divert my attention to a family of two adults and two children in the rows ahead of me who left a third the way through. Wonder what they were expecting.
But for PowerShell to become Python on Perl steroids on top of all the .NET and COM+ there is, well, of course it took years, and top engineers to work on it, but also was needed an unified interface (API) to the underlying OS magic. Which is not there in Linux the same sense .NET became "available". OS/X has something similar to COM+ called CORBA which was left half-baked at best. Python is not .NET, sorry. Not anywhere near.
Someone may say the UNIX ideology is about your programs doing one thing at time, and having this STDIN->munch->STDOUT approach. But these streams never got any structure, and ppl out there been creating great tools to fill this gap, tools such as - jq, mlr, xidel to name a few. perhaps we should include ack, ag and rg in the list.
You know, guys, you hard-loving bash die-hards and sed+awk magicians, you should give this PowerShell thing a try before inventing the wheel again. And before legacies you wrote 20 years ago become as difficult to support as COBOL. Oh yeah, and PoSH is open source, based on open-source runtime, and available in brew/apt/yum...
Disclaimer: I do not work, and have never worked for MS (Guido does now!). Have used DOS since 6.22, FreeBSD since 8th version, Debian since v5, and Windows since v3.1. I did Perl for 20 years, actually I gave Perl classes in some university, still doing fair amount of Python, and accidentally discovered PowerShell 3 years ago. I'm not a .NET advocate, although I can tell the good parts.