I really wonder about all these services that eat into the layers of implementation that organisations usually do in house, but at the same time quick form a complex web of lock in that prevents you ever leaving Amazon. It seems to me that these services are very seductive but there are very few cases where it's truly in an organisation's long term interest to adopt them. At best, use them as a quick interim solution to bootstrap what you are building and then use that as your requirements for an open source or at least vendor-neutral equivalent.
Curious what the HN crowd thinks about this issue?
If you get to a point where you are locked-in it's because you spent years building into a solution that was your most expedient and renumerative option. At some future point in time, a future you or the person who replaces you may want to reconsider a re-platform. Let them worry about that.
I worry about development working on glue code/solve problems to the detriment of their businesses actual needs.
Vendor lock-in is just another form of technical debt and should be treated as such.
Sure right now it's your easiest option, but in the future the vendor might be the only one offering a compatible version of CoolNewFeatureXY at prices that make your business uncompetitive to the other businesses that have the freedom to choose between vendors.
Why do you think it’s technical debt? Because you might have to pivot away because it might not support your requirements in the future? Then just pay that cost in the future if it ever comes. Building everything in-house seems like premature optimization—you’re obsessing over a case that may never come, and you’re very likely not going to build these web service components as well as amazon for the cost because Amazon has many customers paying for these services such that each customer pays only a tiny fraction of the operating cost—if you roll your own, you have to pay the whole cost (roughly) for every web service you build and no one is paying your company to build web services, so you have to build that cost into the price of your product/service, so your product/service becomes dramatically more expensive (because it has to cover the cost to develop and maintain many full web service component implementations, which are each many times more expensive than the corresponding AWS service). If in the future AWS starts charging each customer the full operating cost for a given web service (which is very unlikely to happen given their business model) then you can spend time pivoting to your own solution, content in the knowledge that you saved many years of operating costs because you didn’t have to maintain your own service.
I am not sure why you (and every other person that replied) assumes that I am suggesting an in-house solution when I explicitly wrote "freedom to choose between vendors".
> Because you might have to pivot away because it might not support your requirements in the future? Then just pay that cost in the future if it ever comes.
That's pretty much the textbook definition of debt.
> I am not sure why you (and every other person that replied) assumes that I am suggesting an in-house solution when I explicitly wrote "freedom to choose between vendors".
I used "build in house" as a simple example of a broader principle--if there is a "freedom to choose" option that magically lets you migrate from one platform to another at zero cost while imposing no overhead over the AWS solution, then absolutely you should do it. But generally these solutions require a significant up front investment and rarely actually deliver on their promise of abstracting away the underlying platform (now you're wed to $ABSTRACTION on AWS, for example), but that's another story.
> That's pretty much the textbook definition of debt.
Nonsense. That's not debt, it's deciding to wait to pay for something in cash until you know you want to buy it. It's not debt to wait to spend $30K (cash) on a car until you're sure you want to buy it. The "tech debt" analogy is intended to convey interest--you know you want to change course but you do the expedient thing now knowing that you're going to continually bump into it (each bump is "interest") until you can pay it off properly. This is the opposite--if you build it in house or use a dodgy "platform-agnostic" abstraction, you're going to be paying interest on something you will very likely never even need!
To be honest this looks at code as an asset versus a liability. Most code especially glue code is a liability since you need to maintain it and take care of it versus letting a third party vendor like Amazon be responsible for maintenance. In the end a business is trying to build assets and glue code to push, to say Salesforce, may not be a competitive advantage.
The vast majority of successful tech companies definitely treat their code as assets.
I think you’re confusing 2 things. The technical and developmental side, where code is a liability that needs to be maintained, and the business/usage side, where code is definitely an asset that provides your company with additional leverage against competitors.
Most code doesn’t give you leverage against competitors. Building in-house for 100X the cost (or at 1% quality) of using an AWS component confers a competitive disadvantage unless you have lots of customers who pay a lot to do business with someone “not on amazon” but even then you probably shouldn’t be building it yourself. This is generally implied when people say “code is a liability”.
I'd say that having a (buggy) in house integration with Salesforce is a way bigger tech debt than using AppFlow here.
If one day you want to move off Salesforce to another service, you will just switch off the configuration in AppFlow for the new service. Sure you're locked in AWS, but it takes tech debt away from other things, plus might save you migration costs.
> I'd say that having a (buggy) in house integration with Salesforce is a way bigger tech debt than using AppFlow here.
That assumes that the AppFlow code is not buggy. In my experience these services are very hit and miss. This might be a good one, but it's impossible to tell until you use it (or read experience reports of others using it). Reliable 3rd party services can be a big time saver, but Unreliable/Buggy/Limited 3rd party services can easily take 5x more time than something in-house.
I'd gander that an AWS team would be more capable than your average startup team. And testing if AppFlow works for you will take way less than building your own.
* About 70-80% of what you want to do with a service like appflow to integrate with a service like salesforce will be possible. The other 20-30% of your requirements will require building a bugging in house integration from the ground up. Might as well start off that way.
* As soon as you hit a wall with a service like appflow that is usually it. You'll raise a ticket and be stuck until they get around to implementing it. There are fewer roadblocks with home grown implementations of integrations.
Not saying it will work for all cases, but a majority of integrations are pretty basic. I have done a few. Usually just data mangling between different systems, which is exact what AppFlow is good for. Also if you are on a business plan, the response time is pretty good. I got feature requests done in Sagemaker.
It's a good point - and part of the reason I'm asking this is I don't know the answer 100%. Some of it is gut feeling. I guess part of the problem is that by outsourcing this layer of work you actually fail to develop internal competency to do it yourself. Then when later on you find your requirements expand beyond what can be done, you're now functionally incapable of doing it, and you start being constrained by that. I've been in companies that end up paying ridiculous amounts for consultants to do utterly trivial things to systems just because they've been bred to be functionally dependent on systems they don't understand / can't control. That isn't this, but I feel like it's a step towards it.
Vendor lock in is very expensive. At the moment vendor lock in is providing AWS with more profit margin than Amazon's entire retail division. It's more profitable than McDonalds.
It's also A) frequently not all that hard to avoid B) worth doing for reasons other than sheer cost (e.g. flexibility, ability to unblock oneself more easily, ability do deal with bugs by oneself).
If you work in a company that isn't especially worried about hosting costs it's fine (mine isn't, for instance), but most companies are at least somewhat concerned with these costs.
I agree that vendor lock-in is expensive. It's not as expensive as the investment companies make in keeping their tech cloud-agnostic, investments that never seem to pay off. Either the day never comes, or when it does, switching costs are still surprisingly high.
Vendor lock in is essentially that it’s more effort than you’re able to afford, in terms of time and cost.
There's always a tradeoff, but being all in on a vendor isn't necessarily a bad thing. Embracing a platform and the services they have to offer is how you get the most out it.
Whenever you pick a technology or service, you need to employ an ongoing effort rather than waiting until it's too late to do a one-time effort.
This is true with any choice of technology, whether your rolling your own or using SaSS, an on-going effort is required to avoid being locked in.
use that as your requirements for an open source or at least vendor-neutral equivalent
My clients are all expending far too much energy trying to be cloud vendor neutral with the engineering capacity of your average non-tech large mid-market company. It's killing productivity. They went from VMWare in the data center and open source DevOps tooling to multi-cloud vendor-agnostic container-orchestration all in the name of "cloud but not with vendor-lock in". Meanwhile they could be actually achieving business objectives with these "seductive" managed services.
Here's one simple example, one client is spending about $250,000/yr on what could be accomplished with AWS Cognito for $250/month. But "no", they say, "then we'd be locked in to AWS".
In a company with huge numbers of developers I'd agree with you, it's better to have something you control than a vendor specific thing that's likely to cause problems. For a smaller company with limited availability and a long backlog of features to implement that's not the choice being made though, the choice being made is "do we implement this feature or not", and a tool like AppFlow lowers the cost of development on certain features, likely resulting in at least some things that wouldn't have happened because they'd require a larger investment of time than people were willing to put in.
If you're large enough and can negotiate appropriate contracts, it's not problem to be locked in with one vendor. Then you typically also get prices way below what's typically quoted and in line with home built.
A huge problem however for smaller companies that have little power if the vendor raises prices, discontinues products or throws them off the platform. Going all in with AWS as a small company is not only expensive but can be a risky bet.
I feel like a notably missing product from this list is Jira. As horrible as it is, lots of companies use Jira for tasking so it should have been supported out the gate.
Jira is like Agile, it needs to be modified to the company and WAY too easy to do wrong.
At its core Jira is just a tool you use to track who's working on what issue, maybe link it to your version control to see what commits have been done.
BUT, then the Suits come in and start adding timesheets and reporting and 42 different categories for everything and it becomes a hellscape of forms you need to fill just to get some work done.
Jira is a classic example of a product oriented towards how companies buy things.
Every time a moderately expensive purchase is needed companies get together, write down a list of things that they need the product to do and then compare the options on the market. This process results in a 'winner' who checked the most checkboxes.
It's a process that doesn't take into account the subtler more analog things we should be rating products on - basically how 'nice' they are to use (in a corporate setting I would phrase that as how 'efficient' they are to use, but it's the same thing).
I agree 100%. I once integrated Jira in a product and it was a nightmare. Integrating is pretty straightforward, but then Jira has its own way of naming the custom fields that the user added, so you end up in making a ton of API calls to discover what are the custom fields, their name, which are mandatory, what is the data format, and so on.
And don't get me started with creating a GUI so that the user can insert the data for a set of fields you don't know in advance.
Answer really depends on the size & culture of the organization.
Clubhouse.io, Asana & PivotalTracker are nice, but a little opinionated (can be a good or bad thing depending on who you are :) ).
I think where Jira really rubs people the wrong way is it's speed, particularly after someone customized it with a byzantine workflow or with dozens of views/reports and no primary source of truth for the team to use. Imagine a PM looking at one report and the dev's thinking another is "the one." A fresh Jira (Cloud verion on corp-name.atlassian.net) with out of the box Kanban board should work for most.
There are some industry vertical plays out there that focus on Agency or Freelancing work.
That’s exactly the problem with Jira (and honestly Bugzilla in the same way a decade ago), it tries to please everyone and thus succeeds with no one. Trying to do everything typically ends up in bloat/complication/slowness, which seems to happen disproportionately for bug tracking and cms systems.
Isn't that the point? Businesses and people are way more obssessed about Jira than they should ever be. Jira is supposed to help organize, not become someones job.
Trello's the one that rolls random people's personal accounts into the account of the business they work for, right? And refuses to give them back when asked?
Of their products, Jira does surprisingly well for us. I dislike quite a few of their products but Jira just works for us, though some things seem to be more work than necessary to do. Some of that can be blamed on company processes though.
I actually like JIRA a lot, as long as you are not trying to customize it too much. A simple Kanban or Sprint board should be enough for 99 percent of teams, please do not try to make it more than what it should be.
This is an interesting addition to AWS! I wonder how is this better than stitch or fivetran? It’s not immediately clear to me. Also I am doing the back the envelope math. It seems like app flow cost more than our current usage of stitch or even fivetran to our redshift warehouse and doesn’t even include some of our bigger other sources we use (MySQL, stripe). I am gonna do more analysis on this and see if this can be a potential cost saver for us in the longer.
It’s exciting to see more competition in this space none the less.
The days/weeks/months/years it takes to get IT security in a highly regulated industry to let stitch/fivetran have access to my data.
This is one of a handful of very significant deficiencies in AWS's data/analytics portfolio. They haven't had good tools for getting data into AWS from the various places that profit centers store it. I hope they very aggressively pursue extending the number of supported integrations and finish building the product.
"send logs, metrics, and dashboards from Datadog into an Amazon S3 bucket to create monthly reports"
This is an odd use case given that Datadog can already store logs in your S3 buckets, and Datadog integrates with AWS Cloudwatch to pull in AWS metrics!
I specifically meant nuance relating to the product as well as actual lakes. If there's none, the word choice (and maybe the whole metaphor) is a distraction.
Also, as I mentioned in another comment, this phrase doesn't work for the metaphor. Dehydrating and rehydrating a lake isn't a good idea; it'd kill the ecosystem, and it'd be tremendously expensive. I'd rather stick to dehydrating and rehydrating something else, like fruit. Here, this looks similar to the fruit soup my in-laws make: https://www.cheaprecipeblog.com/2015/11/norwegian-sweet-soup...
Yep .... and that is very often what untransformed data is like ! Thanks for the metaphor - made my day reading that one and given me a whole new expression to throw about during product management discussions in my day job. :-D
We are trying to provide a service based on CRM (Salesforce) data to our customers in China. Salesforce doesn't exist (yet...) in CN, and wondering if something like this could be useful in moving needed data to AWS China. Then I realized that Redshift is not your typical OLTP db and not suitable for regular querying from an application. Curious to hear HNs thoughts.
EDIT: Someone mentioned Heroku Connect in one of the comments. We do use this to archive certain objects (> 75 million records) and I have thought about replicating this data to AWS China (Postgres). Could be one option but some of our data is real time (e.g. customer orders), not sure what the "freshness" of the data would be.
Really curious how well the salesforce integration works. Heroku has a product that does this (Heroku Connect), but it doesn't work well with large tables and you have to pay them an obscene amount of money to even properly test it (which might have something to do with them being owned by Salesforce).
So many companies with tons of data trapped in Salesforce's expensive walled garden and no way to get it out. Dumping it to SQL is the first step to freedom.
Couldn't agree more. This is obviously a shameless plug but we built a product (getcensus.com) specifically focused on this problem of getting data to/from Salesforce & data warehouses (not just Redshift).
Heroku Connect is pretty powerful but not only is it super expensive as you point out but also forces you into creating an extra layer in the stack by creating a Heroku Postgres DB. The secret advantage of Connect is that since it's owned by Salesforce it bypasses Salesforce API quotas but that's a whole other can of worms :-)
We use Heroku connect to sync a few hundred thousand rows and it works really well for us and has been way more reliable and quicker than the Salesforce API. Heroku connect let’s you sync data from Salesforce tables which are the source of truth. There’s also another product (which you pay for separately) called Salesforce connect which lets you sync data from a Heroku database as the source of truth.
I've built a few projects that provide real-time reporting across sass applications but it always seems like an uphill battle. Especially when your only interface with the sass is a hypertext API.
Can a data source change while you're completing a multi-part query?
Are records hard deleted? If so how do you detect removed records to maintain deltas?
How do you maintain deltas throughout lossy pipelines that group or filter?
In the end, it always feels like it'd be easier to build one monolith to replace all the systems and report on that instead...
For a minute I thought this was the release of the "AWS for Everyone" product with a new name, but this is something very different.
Interesting product though! I think everyone that's using SaaS solutions has nightmare scenarios about providers screwing up or going out of business, or even seeing versioning of data at the providers. Exporting in a standardized way to S3 is great for insights in changes as well as for backups.
Of course when you starting building triggers that react to changes this creates a huge lock-in to AWS, but that's a cost a lot of people are willing to pay.
I would love to be able to do this with a load of different SaaS solutions that we use (gitlab.com, support tools, even google docs) (I'm in a regulated industry and this would solve some issues for us). Curious to see how this will expand and if custom integrations will appear soon.
Any AWS marketing people that are reading, there's a typo: route cause -> root cause.
By now - the amount of time teams have spent doing this sort of work from scratch has to have surpassed the engineering hours taken to get to the moon.
As long as there is revenue to be had doing it, folks will continue to do it, reinvent it, and so on. It’s a billion dollar market, at least. Not everyone is going to be a developer, nor should they have to be (it’s unreasonable to assume everyone has the aptitude). Everyone is trying to be the Excel of the business process automation market.
Conversely, who knew there was so much money to be had in performing ETL on API payloads!
Which is kind of curious because they only move data from app A to app B. Makes you wonder how much engineering effort it would take to actually do something with the data.
Related, I've been trying to figure out what the best way to populate a Redshift database from regularly updated SFTP sources is and I can't quite figure it out. AWS Glue? A function in Lambda? This?
AWS presents so many options that it's a bit overwhelming.
We, Fivetran, have an SFTP connector. You can be sync'ing data with 10 minutes of effort. Our value is in (nearly) zero configuration, automatic schema migration, and automatic failure recovery. We take the pain out of Extract-Load, and push Transformation into the warehouse. https://fivetran.com/docs/files/sftp
This has always killed me about SaaS-es that don't provide any decent method for backups (and less so: restores), especially for data modifications (eg, accidental deletions) that there's no ability to rollback.
Ionic Appflow is a very different platform but it is interesting to have products with the exact same name (except the capitalization of "F") in the software/platform world.
One of the worst things in naming products is when a big business unintentionally clashes with your product name and nearly forces you to rename/tweak it.
As they are in the same category (Cloud technology), I won't be entirely surprised if Ionic renames their 'AppFlow' name to something else. (Unless they're feeling lucky in winning a trademark lawsuit)
Yes and no. It's primarily focused on centralizing data into a data warehouse / data lake rather than sending data out to other apps. Segment also pushes events into a warehouse but most of its functionality is about forwarding events to 3rd party apps. That said, you can use tools like getcensus.com to move data/events from an AWS-populated data warehouse back to other apps (disclosure: I'm one of the founders of this service).
Founder of RudderStack here (we are an open-source Segment alternative)
Segment (or RudderStack) is primarily for routing event-data from your apps to multiple destinations (cloud SaaS, warehouse etc). On the other hand, AppFlow is for syncing data that lives on cloud applications with AWS products like RDS, RedShift etc. So, both the nature of data is very different (event-streams vs database update batches) and also the destinations where they are sent (multiple cloud applications + data-warehouse VS only datawarehouses/databases).
Segment also has a product similar to AppFlow called Segment Sources but that's not what Segment is known for. There are a bunch of other products in that space like Blendo & FiveTran.
90% of the market for Zapier is people who have no technical knowledge. Those people won't be jumping on AWS any time soon.
Obviously pulling the 90% out of my ass, but I'm heavy Zapier user for my e-commerce business. I have (Or had - Haven't renewed as e-com has replaced my DevOps income) multiple AWS certs and I'd still much prefer Zapier over this. Especially when you can run custom code on Zapier for more complex cases anyway.
- If a new voicemail comes in, see if it's an existing customer and send them an email we'll be in touch if we do
- Create new ZenDesk users for any Shopify customer
- Send emails when positive product reviews come in asking if they would mind sending a Facebook review
- If a negative product review comes in, ask what we can do to make things better
- New order comes in: Will create draft emails to all necessary vendors I need to contact, pre-filled with shipping details.
- New order comes in: Separate order cost by vendor, grab taxes, grab shipping, grab discounts, put it all in an airtable database and create invoicing and purchase order records to attach it to.
- Slack notifications
- Pull PDFs from Google drive, parse with DocParser, upload into Xero accounting.
I'm sure there's way more I can do as well. I have to find the balance between automating and just telling an employee to do it But automation leads to less mistakes
It seems that Amazon have quite strong internal standards for services that they're going to roll out, for example all critical services are required to use DynamoDB, which would make it difficult to integrate a competitor into their wider ecosystem of AWS services. They've also got a vast pool of people to draw from around the company which allows them to implement new functionality on top of existing tools they've already built, rather than having to do everything from the ground up.
I suspect that by AWS going first here, they can easily go for more integrations whilst also eating some of the Zapier's competitors lunch.
This depends on price and by that time its possible that Google Cloud Platform and Microsoft Azure will start jumping in on the integrations market too, further squeezing the likes of some enterprise-only integrations such as Tray.io which still has less integrations than Zapier in general.
It's still too early to tell, but we'll see how fast Amazon adds support for more API integrations or if GCP and Azure will join in and do the same.
Amazon seem to have a disadvantage here though. This kind of tool is aimed at non-developers who want to glue stuff together. With Zapier, they register an account and get stuck in – straightforward. With AWS, they register an account and are instantly overwhelmed with a million different things. AWS is too big and unfocused for the target market. I've seen developers hit a wall once they get on board AWS because it's just too much to deal with. You think the AWS dashboard is any friendlier to non-developers?
Curious what the HN crowd thinks about this issue?