> Hierarchical notetakers/outliners are becoming a new data structure for end users, specialized in knowledge management.
Outliners are used in this area for decades. What we have now is just a new hype-cycle, with new apps, new technologies and a new mix of features.
> When tools like Obsidian or Logseq start allowing the user to use query results as first-class data, I predict that we will see a revolution in end-user process automation.
> What we have now is just a new hype-cycle, with new apps, new technologies and a new mix of features
You contradict yourself. Is it "just a new hype-cycle," or is it "new apps, new technologies, and a new mix of features?" If it's the latter... isn't that substantively different than what we had before?
Outliners have quite the fandom, but my issue with the traditional outliner is that they often trap data in outline form. Sure, most good outliners offer export options. Unfortunately, the second you've taken data from your outliner and exported it, you now have two versions of your data, and you've lost.
Logseq (and Obsidian) have changed this. Finally, a single document _is_ an outline, a backlink, a data source, a webpage, a presentation, and so on... The only thing the user has to do is work with that single page of information.
> You contradict yourself. Is it "just a new hype-cycle," or is it "new apps, new technologies, and a new mix of features?"
The concepts are old and have a new hype. The apps are new. But generally, it doesn't matter much for the user whether you have some old-school desktop-app with gtk or Qt, or a "modern" electron app, or some webapp on your own server or some service in the cloud. At the end it's all the same ideas and concepts in different colors and flavors.
> If it's the latter... isn't that substantively different than what we had before?
That entirely depends on who is "we". We which are all users? Or we the old users? Or we the young users? Or we the new users? I'm not saying that there is not development or change, just that this is nothing new, or even a new development. This has going for a long time, it just reached a new cycle and and a new generation of users.
> Outliners have quite the fandom, but my issue with the traditional outliner is that they often trap data in outline form. Sure, most good outliners offer export options. Unfortunately, the second you've taken data from your outliner and exported it, you now have two versions of your data, and you've lost.
This is a fundamental problem of every app and space. But interoperability does exist here and there, it's not just exports.
> Logseq (and Obsidian) have changed this.
Not really? They only established a common foundation for interoperability. Which is also what other apps are doing to some degree. But you still have limitations by everything which is not covered by this foundation. For example, can Logseq handle dataview-syntax? Or generally any obsidian-plugin has it's own special syntax? What about other markdown-editors? How much do they support the special syntax used by obsidian or it's plugins?
True, the markdown-space has now a healthy movement toward interoperability and gains many nice functionalities. But it's not like this is especially new. And all those features also existed before Markdown in some app, in some way or another. Nothing is really new here. What's happening is that a new platform is growing, collection nice stuff from the collective pool of abilities, and a new consolidation is happening. This happens all the time. Sometimes it works, sometimes it dies.
Funny enough, the WWW was one of the first and most successful of this movements. And markdown grew from this as a simplification, to now grew into the same space where WWW started and obsidian now moves toward too, just in a different flavor. This whole "new" movement is just replicating where others were already before in different cloths.
> Finally, a single document _is_ an outline, a backlink, a data source, a webpage, a presentation, and so on...
So has Org-mode 10 years ago, XML 20 Years ago. HTML 30 years ago. More or less, the numbers may be a bit off.
> It's more than hype for me.
Good for you. I'm not question that. Just saying this generation is not the first one having that.
I'm afraid you're entirely missing the relevance of the core fact that the syntax and runtime execution of programming languages are humongously difficult to understand for the large majority of computer users. Every time you create a system to automate some part of computer usage that doesn't rely on complex syntax and which executes in a declarative way, the same end result is created through a new concept, even if there existed a different process before that could also create it.
HTML was a step in the right direction (and XML a misstep); before it, you needed to program a GUI API to lay out content intermixed with interactive components. Saying that the web brought nothing new because BBSs existed before it is missing out on the importance of the details in achieving that result. Precisely, "different cloths" is what makes the movement relevant to make it accessible to new users who could never have used the previous system.
You're right in pointing out that interoperability is a key factor in the new batch of tools. The key point that neither Org-mode nor SGML-derived codes had is is to make user-entered content accessible to transformation tools without the need to code the transformation in an imperative language. Spreadsheets had this feature as well, sure, but spreadsheets didn't easily allow for hierarchical content.
The main new aspect that logseq and Obsidian bring to the table is the possibility to backlink to any part of the hierarchical structure. The tools automatically compile all those links as a new data object, which the user can process without creating a script to gather all that data. HTML had the potential to popularize that feature 30 years ago (with a slightly more awkward syntax than Markdown), but browsers never got around to it.
Previous tools would require users to move content to a different tool (say, by copy-pasting the content of your files to a spreadsheet to apply formulas), destroying the possibility of having a central personal knowledge repository. In logseq / Obsidian, you can keep adding multimedia content to your structured knowledge base, and exploit that content without coding at all, or at most by creating simple declarative expressions. What tool before these ones had that possibility before? With hierarchical structures, not just flat structures like the spreadsheet?
> I'm afraid you're entirely missing the relevance of the core fact that the syntax and runtime execution of programming languages are humongously difficult to understand for the large majority of computer users.
No, I know that, I just don't think it's a relevant problem for the user group we are talking about.
> Saying that the web brought nothing new because BBSs existed before
I did not say that? And in the first place, BBS competes with other networks like the Internet, not a service running in such a network.
> The main new aspect that logseq and Obsidian bring to the table is the possibility to backlink to any part of the hierarchical structure.
No, they do not. Linking is a fundamental concept of hypertext. And easy linking inside your dataspace, as also automatically listing backlinks, was already established with wikis 20+ years ago. Even hierarchies exists in wiki space for a long time now in several different ways.
> The tools automatically compile all those links as a new data object, which the user can process without creating a script to gather all that data.
What kind of low-level-tools have you used till now that this is your level of knowledge here??
> In logseq / Obsidian, you can keep adding multimedia content to your structured knowledge base, and exploit that content without coding at all
Yes, because someone else does the coding, as always. This is the benefit of a popular ecosystem. But this is not the point of having a scriptable and programmable app. The scripting just enable you to personalize on small levels. The programmability should enable the user and community to extend the program with new abilities, as dataview here is doing it. There are other extensions who enable scripting to certain degrees for the user, as obsidian is not really adding this on it's own.
No app will ever be perfect for everyone. That's why extensions and scripting are beneficial, especially for this space.k
> What tool before these ones had that possibility before? With hierarchical structures, not just flat structures like the spreadsheet?
The constraints again. Why does it matter that obsidian&co. can do some more tricks than their ancestors, like playing videos, when the fundamental purpose and handling is still the same? That's like the old joke of taking something old and just add a clock, to sell it as new. More bells and whistle don't make a new church.
> No, I know that, I just don't think it's a relevant problem for the user group we are talking about.
What user group are you talking about? I'm specifically talking about the large majority of users who could never learn to program in a syntax-heavy general-purpose programming language! X-D Those people nevertheless have a need to build their own userland applications, and no current tool serves them well; but a Nocode programmable knowledge database could do it.
> No, they do not. Linking is a fundamental concept of hypertext. And easy linking inside your dataspace, as also automatically listing backlinks, was already established with wikis 20+ years ago.
And as I explained, no end-user tool has exploited that possibility to allow end users to access those automated links programatically with nocode tools (like for example with spreadsheet-like declarative expressions). The browser missed the chance by having no persistent storage and only allowing javascript as their programming language, and wikis never were nocode-friendly. Yet modern tools are just beginning to do that on top of a personal knowledge base.
> What kind of low-level-tools have you used till now that this is your level of knowledge here??
I beg you pardon? I've programmed all the way down from building my own flip-flops with electric transistors up to non-linear mathematical solvers for combinatorial optimization. I don't see the relevance, since low-level-tools will never be adequate development platforms for end-users.
> Yes, because someone else does the coding, as always.
No. Logseq / Obsidian have tools that allow end users to build their own coding through templates and queries. The kind of coding is different from scripting, though; it's more like creating semi-automated workflows that put all relevant information in front of the user, which then proceeds to transform it with direct manipulation. What tool in the past has allowed end-users to create that kind of workflows from end-user templates and arbitrary queries, other than non-hierarchical spreadsheets?
> The constraints again. Why does it matter that obsidian&co. can do some more tricks than their ancestors
The constraints are there for a reason, and you don't seem to grasp why they are important. The spreadsheet , and spreadsheets are typically not automated through scripting and extensions - using those is normally considered a failure of the spreadsheet model, because you're processing data outside the mental model of the spreadsheet as a programming environment i.e. cheating. Most End-User Development tools rely on the assumption that users can't program at all, so all automation concepts must be presented to them through alternate syntax and interactions, quite different from classic coding.
A spreadsheet-like development environment based on outliners as its data structure, rather than the tables of a spreadsheet, should also be programmable by end users without relying on imperative scripts external to the tool. Logseq is creating the basis for that storage model, which some people are starting to exploit the kind of semi-automated workflows I talked about above.
> like playing videos
That's a red-herring, not a main point. I mentioned multimedia merely because programmable tools tend to be text only, forcing users to use separate tools for rich content, or import specific libraries to handle multimedia data as programmable objects (rather than editing them as native interactive content like they would on the GUI). The point is that end users should be able to treat all kinds of content equally with their tools, not being tied to the Unix shell constraint that all content is moved between processes as text streams or binary blobs.
The kind of people who use such tools. The kind of people who can lookup DSLs to create dataviews, or copy&paste snippets to make up stuff. Something like a knowledge-system is not used by children or someone similar.
> Those people nevertheless have a need to build their own userland applications,
No, they do not. Not every user around the world has this demand. Most user don't have this demand. We are talking here about a minority of quite special usecases. Most people are not even able to mange their own files, and you mean they will use a complex knowledge-managment-system? People are even challenged with a simple todo-list or note-app. No way will they use something more complex.
> and no current tool serves them well; but a Nocode programmable knowledge database could do it.
No, it's not. Nocode is not magic. You still need a certain level of competence, understanding and motivation to make something with it. And whatever interface you present to the users, be it GUI or text or whatever, there always will be a certain level of complexity that not any random user can master. And this is good, we already see with apple and google what harm we have from too much simplification to cater to the uninteressted.
> No. Logseq / Obsidian have tools that allow end users to build their own coding through templates and queries. The kind of coding is different from scripting, though; it's more like creating semi-automated workflows that put all relevant information in front of the user, which then proceeds to transform it with direct manipulation. What tool in the past has allowed end-users to create that kind of workflows from end-user templates and arbitrary queries, other than non-hierarchical spreadsheets?
Many? What? This is also a kind of scripting, and there is an endless amount of tools enabling people to do that style of scripting. Starting with shellscripts, batchscripts, all the strange scripting-engines. Heck, emacs and has this style as their main purpose established. The data science-sector has a very long history of building such tools on a more professional level and integrating data-views. Wikis and databases have a long history of scripting in the knowledge managment-space. The likes of obsidian are just the next iteration, there are not fundamentally new.
> and spreadsheets are typically not automated through scripting and extensions - using those is normally considered a failure of the spreadsheet model
No, they are? What? People have build all kind of crap with VBA for a very long time. There is a reason why this garbage is still around, getting ported to everywhere. And let's not start with the plugins or how extensions for MS Office are called these days.
> using those is normally considered a failure of the spreadsheet model, because you're processing data outside the mental model of the spreadsheet as a programming environment
Maybe that's our bubbles understanding. Seems the rest of the world doesn't give much here. Limiting yourself to a model is nonsens anyway. No model is ever perfect and satisfying every user.
> That's a red-herring, not a main point. I mentioned multimedia merely because programmable tools tend to be text only, forcing users to use separate tools for rich content.
It's not like obsidian is doing more with them, just because it can play them. Until it understand them can make them accessable for the rest of the system it's remains some fancy gadget.
So, we're talking about entirely different groups of users then, there's no wonder that we're talking past each other, and that you're missing everything innovative about this new line of tools.
> No, they do not. Not every user around the world has this demand. Most user don't have this demand. We are talking here about a minority of quite special usecases. Most people are not even able to mange their own files, and you mean they will use a complex knowledge-management-system?
There I have to strongly disagree. There's a whole academic field of study, End-User Development, dedicated to prove that idea wrong. Every computer user I know is able to use spreadsheets to create their own data types tailored for their domain-specific use cases, some times quite complex ones. That they can't use scripting languages is a failure of those languages, not of users.
And yes, users are able to handle complex knowledge management systems when they're built from the ground up with the right models, instead of relying on arcane legacy metaphors from the 1960s, such as hierarchical filesystems - which are good for developers, not so much for end users. Our knowledge about digital content organization has evolved a lot since those early pioneer times.
With the proper tools, of course they'll be able to create new software artifacts, which of course they want and need - the same way they want and use new apps built by others. If they could create those themselves, why wouldn't they build their own?
> Many? What? This is also a kind of scripting, and there is an endless amount of tools enabling people to do that style of scripting.
True, but you're missing out the possibility of enabling people to create automation without any scripting at all. There aren't many tools for that other than spreadsheets, which have bad integration with the rest of the computing environment.
>Wikis and databases have a long history of scripting in the knowledge managment-space. The likes of obsidian are just the next iteration, there are not fundamentally new.
I recommend you to learn about End-User Development,[1] to see all the ways these tools can be expanded beyond classic scripting, to see what you're missing out, that other people in this thread are aware and you're not. The end-user computing landscape is going very soon to be transformed, flooded with the kind of tools that end users can build for themselves, for the very same reason that the "VBA and MS Office plugins garbage" are still around; just this time, we finally have a chance to create a way better platform for those tools, which doesn't rely on outdated plugins and low-powered scripting languages.
"Watch what I do"[2] and "Your wish is my command" [3] are the seminal works with loots of good examples. They're a bit dated, but some of their best methods have still not found their way to practical common-use tools (but modern outliners could change that very soon).
> Outliners are used in this area for decades. What we have now is just a new hype-cycle, with new apps, new technologies and a new mix of features.
To say that what we have now is nothing but a new hype cycle while the Zettelkasten method was only widely exposed within the last couple of decades, and was not utilized in the design of past Outliners, shows you don't really understand what's actually changed.
Zettelkasten is nothing new and just hype. Only shows that you just don't know this space, and it's history. Obviously there is improvement, but it's not in methods or data-structures, it's with technology and implementations. We are now at the point where we can move fast with implementations and glue much more features together.
Then please, illuminate me. Because the implications of Luhmann's use of the Zettelkasten weren't widely available till the last decade, and weren't available in software until the same period.
Luhmann methods are known since the 80s, at least in europe. And I think I saw the first digital implementation around 2006 or 2008. Looking it up, the first(?) Digital Zettelkasten after Luhmann method was in 2004 already at version 2.5, so it's likely quite older. But dates are not important here.
Luhmanns method is simply not that special, nor unique. Many scientists and artist used similar methods and tools. He is mostly just replicating what was common at the time, but with some twist. And in the 80s or 90s is would be really awesome, probably even somewhat groundbreaking. But today? It's just another graph-system which you find in wikis or hypertext for decades now. And most people are not even using the benefits of his number-system, just calling everything zettelkasten which is just a wiki otherwise, leaving even less of his originality. Though, in digital space it make less sense anyway, so not a big lose.
But for whatever reason, there now this hype from people who seem to have know knowledge about those space or it's history, or even really understand the terms, thinking Luhmanns method is called Zettelkasten...
What outliners do you know which allow end-users to feed their data into formulas for processing it without using general-purpose programming languages? I'm genuinely interested, as I've been exploring these tools for years and I haven't found any that meets my requirements.
> So, SQL
How many end-users without any programming experience are able to create moderately complex SQL queries?
With a data-centric outliner tool, they'll be able to decompose those into several simple graphical queries.
> What outliners do you know which allow end-users to feed their data into formulas for processing it without using general-purpose programming languages?
Bit of a pointless constraint, the talk is about outliners, not no-code-datamangment. Which tool today does this even offer on a useful level?
But you can look at leo editor (https://leoeditor.com), which is active for 20+ years, fully scriptable and extendable. Though, it's a hot piece of garbage for laymen. It's offers a bunch of features and plugins even for non-coders, but I'm not sure it would satisfy you for this area, if you can't code. Obsidian is probably better than, even if it has more limitations at the moment.
But I'm not sure if there ever is a tool which will satisfy everyone with just a no-code-approach.
> Bit of a pointless constraint, the talk is about outliners, not no-code-datamangment.
The point I'm making is that those are both aspects of the same substrate, and modern knowledge tools are exploring the gap between those two - as well as expanding them in new directions.
Yes, certainly Leo is one of the closest tools on which to build a system like that, although it still relies too much on Python programming (and its usability leaves a lot to be desired, even if the concepts are solid). I expect that the next popular tool in that space will grow either from online notebooks like Jupyter or from desktop linked note-takers like logseq.
(BTW I can code, but I don't want to; remembering the syntax details of the needed APIs does not come natural to me, so it's usually not worth the effort). I still believe that it is possible to grow a programming tool similar to Excel but with structured hierarchical content, without requiring coding more complex than spreadsheet formulas and filters.
> and modern knowledge tools are exploring the gap between those two - as well as expanding them in new directions.
Do they? As I see it, they make it worse. We have all kind of specialized languages. Each function coming with it's own limited DSL and API. Often with poor documentation. Is this actually better?
Yes, each language has on it's own less complexity, but all in all they have more problems and limitations than a proper general purpose language with a good API. I don't think that this is really a good future. But it's true that it might lower the bar for new people.
> BTW I can code, but I don't want to; remembering the syntax details of the needed APIs does not come natural to me
So the problem is not the language, but the support? More autocompletion, documentation, automation, etc. That's a concern of mine too. Things are complex, but there is no support for the casual users. Actually I think if obsidian had been build ontop of VS Code and gained instant support from all the fancy developer-tools, it would gain significant benefits.
Another solution could be integrating an interface like Node-RED offers it. This seems to be quite accessible for casual experts.
> I still believe that it is possible to grow a programming tool similar to Excel
That may be because your development needs are well served already if you're a developer, so you won't see the value in such system. For people who rely on others to build every single piece of automation they need, being able to create their own small automatisms would be priceless.
> all in all they have more problems and limitations than a proper general purpose language with a good API
End users don't need a general purpose language with a good API - at least not as someting that they will handle directly. They need small, self-contained domain-specific languages that they can expand in areas where they fall short, but without the full power of a general programming system (which would overcomplicate the development environment and provide too many opportunities to shoot themselves in the foot).
> So the problem is not the language, but the support?
For me yes, partially, because I already know programming. I could work with a programming environment with complex syntax that autocompleted the required syntax to match my thoughts (something like GitHub copilot works in that direction). But that's still two points of complexity above the system I envision as possible.
Something like the classic Hypercard could be a good starting point for the mental model of such tools. A simple data model, integrated storage, and interactive processes using a programming language with minimal syntax. Expand it with hierarchical storage, expanded integration with other systems and a modern declarative language, and you're set.
Someone today pointed me to a system like this[1], Lepiter, which looks a lot like the solution end users could manage to create on their own on top of their knowledge systems: like a browser with separate cells, where each cell can run a different runtime, hyperlinked and with a simple storage mechanism.
> More autocompletion, documentation, automation, etc.
Those are good things but not enough. What end users are lacking the most is the possibility to create their own automated processes; most tools only allow them to choose among a library of predefined ones, without allowing them to grow the library.
> Another solution could be integrating an interface like Node-RED offers it. This seems to be quite accessible for casual experts.
Unfortunately, graph-based flow visual tools have proven to be useful only for a small subset of automation tasks largely consisting on data transformations using pre-defined functions. The most promising results in End-User Development point out that reactive, rule-based runtime environments like the spreadsheet seem to be better abstractions for allowing users to define their own computations.
> And Excel has good access for new users?
Yes, spreadsheets have a low learning curve and high ceiling. And compared to imperative programming languages, they're like walking up the slope of your house vs. extreme rock climbing.
OmniOutliner has had some kind of formulas for a very long time, however they work like an actual spreadsheet, where each outline row can have additional number and formula columns.
I don't think it's all that useful, outside of some basic "making a shopping/packing/project list" type calculations. It's worth checking out if you're interested in the history though.
Outliners are used in this area for decades. What we have now is just a new hype-cycle, with new apps, new technologies and a new mix of features.
> When tools like Obsidian or Logseq start allowing the user to use query results as first-class data, I predict that we will see a revolution in end-user process automation.
So, SQL?