| [HN Gopher] Launch HN: Convoy (YC W22) - Open-source cloud-nativ...
___________________________________________________________________
Launch HN: Convoy (YC W22) - Open-source cloud-native webhooks
service
Hi HN, we're Subomi and Emmanuel, founders of Convoy
(https://getconvoy.io). Convoy is an open-source webhooks service.
We make it easy for developers to publish webhooks like Stripe and
Twilio. Convoy is cloud-native so developers can deploy the
container to wherever they host their applications. Some time ago
I built an API for local fintech companies alongside Emmanuel.
While trying to sell this product, we were asked several times for
webhooks. We didn't find a great tool that was easy to use,
language-agnostic and cloud-native; something that could scale
independently of our core service. We decided to build it
ourselves. Webhooks are the glue of the modern internet. They are
used for asynchronous communication by API providers to notify
apps/integrators about events that have occurred in their account.
I like to say it as: if you make a payment on Shopify with Stripe's
checkout and Stripe fails to send a webhook event to Shopify's
servers about your payment, you won't get a delivery of your
package even though your card has been debited. This explains the
importance of webhook events to many consumer apps. Today,
implementing a webhooks delivery system is a highly fragmented
problem. Stripe's system is designed for security (with features
like replay attacks, rolling secrets etc), Twilio for performance
(with features like connection override), Pagerduty for flexibility
(with features like payload filtering) and many more. We'd love to
make these features available to everyone with Convoy. We're
already helping a few companies solve problems they didn't know
they had because of the visibility Convoy gives. API providers
need to push webhook events reliably. It's a pain to build this
from scratch. At first glance, it seems like good old HTTP POST,
but with time it becomes more tricky as the requirements pile up:
dealing with bad endpoints, pushing events to multiple endpoints as
configured by your user, rate-limiting of customer endpoints,
security of delivery, UX for managing retries. We take care of all
that and give you one less infrastructure to worry about. Convoy
is open-source software that exposes a REST API. Developers push
events to it using the REST API (we aim to support other means of
ingesting data in the future for high volume environments - this is
a feature Shopify provides today) then we sign the payload, publish
the events, apply retries, etc. We also provide a management UI to
manually retry events to user endpoints. Convoy consists of 3 core
components--in addition to the REST API, there is a job queue and a
storage layer. The REST API server is used to create applications,
endpoints and events. The job queue currently supports 2 backends:
in-memory and Redis. The storage layer currently supports 2
backends: on-disk and MongoDB. All events pushed to Convoy are
saved to storage and enqueued on the job queue for workers to
deliver. Most of our users use our product to push webhook events
to their customers' endpoints. Customers also use Convoy as a
broker for inter-service communication. We make money by providing
a managed service. We've been running the managed service for a
while now and charge $1/5k events, but those details are not on our
website yet. The repository is available at
https://github.com/frain-dev/convoy and a getting started doc
exists here: https://getconvoy.io/docs/guide. Anyone can download
the binary or Docker image from here: https://getconvoy.io/download
and run it wherever they choose - AWS, Azure, GCP, etc. We'd love
you to try it out and hear your thoughts! We'd also love to hear
about your experiences building custom webhook software and the
challenges you've faced?
Author : subomi
Score : 56 points
Date : 2022-02-25 16:35 UTC (6 hours ago)
|
|
| Sytten wrote:
| Honest question, how is it possible to be viable as a business
| and offer your code as MPL (Convoy) or MIT (Svix)? I love open
| source but I would expect a stronger license like GPL to be used,
| why did we settle on very permissive license as the default? I
| get the brand is protected but is that really enough? I am legit
| curious to hear the founders' take on that.
| tasn wrote:
| Founder of Svix here.
|
| It's a great question, and there's no one definitive answer.
| Some context: I've been an open source developer for all my
| life, my previous business was also open source, and I spoke
| with a lot of people over the last year about the right license
| for us.
|
| The main choices are: copyleft like AGPL (you want AGPL, not
| GPL for server applications), permissive like MIT (which we
| chose[0]), or so called "fair-code" which are the not-quite-
| open-source variants like the SSPL.
|
| While AGPL and SSPL are "better for business", MIT is more
| aligned with our mission, and is better for fostering a
| community. You can read more about the thinking on our Show
| HN[1].
|
| Either way though, all of what the stricter licenses do is make
| it harder for competitors to use your code in their own
| service. Which is important, but not the end of the world as
| competitors can always just copy your API, landing page, and
| everything else.
|
| [0] https://github.com/svix/svix-webhooks
|
| [1] https://news.ycombinator.com/item?id=30347858
| hangonhn wrote:
| I'll try to answer this from a consumer point of view. It makes
| a ton of sense for a startup to do this. As a potential client
| of this, I may end up liking it a lot but worry if this company
| is going to be around in a year. Open source means that if I
| HAD to, I can run it myself. That said, our company would
| rather not run it ourselves. Managed services, SaaS, etc. are
| ideal for us. So being more permissive and a managed
| service/SaaS is the best combination for us looking at it as a
| potential customer. If this gets super popular, it's likely
| that the existing customers won't want to switch even if
| someone else offers it.
| subomi wrote:
| Honest feedback as well.
|
| What we care deeply about today is simplifying and
| democrastising webhooks with Convoy. Under the MPL 2.0 License
| as I understand it, any modifications to Convoy must be made
| open-source as well which helps us reach our initial goal.
|
| On building a viable business, we currently run a managed
| infrastructure, where users pay for our infrastructure and
| other collaborative features (team membership, rbac etc) to
| make it easier to work with the core.
|
| Nevertheless, we are open to transitioning to a stricter
| license in the future that restricts other cloud platforms from
| competing with us with our software.
| [deleted]
| jedberg wrote:
| How does your product differ from Svix(W21)? At first glance the
| two seem about the same.
| subomi wrote:
| Yes, we are very similar. But we are OSS first. Svix is SaaS
| first.
|
| We're designing a tool to be flexible and simplify webhooks
| from a binary perspective. We already support multiple queuing
| & storage backends, we already support an in-memory queue and
| on-disk database, so you can run convoy as a single binary
| without third-party dependencies.
|
| The thesis is if we can simplify webhooks like this, we can do
| more with a tool like this vs. just for API providers.
|
| Although, I learned they recently open sourced some parts of
| their backend, but I think that's how we're fundamentally
| different.
| jedberg wrote:
| Excellent thanks. Sounds like more of a philosophical
| difference but it will be interesting to see how that plays
| out. There's certainly room in the market for multiple
| players! It's a hard problem.
| subomi wrote:
| Yes, It's a big market and I am hoping for more brilliant
| use-cases outside what we're used to!
| tasn wrote:
| Founder of Svix here.
|
| I don't know if I'd call us SaaS first, though we definitely
| spend a lot of time, money and attention making sure our SaaS
| offering is as reliable as it gets if that's what you mean.
|
| FWIW, I've been an OSS developer and maintainer for all my
| life (my previous business was also open source), and open
| sourcing Svix was always the plan. It was literally one of
| the first tickets I opened for the team, and our first public
| commit was in Feb 2021.
|
| We've indeed recently announced the open sourcing of our
| webhooks dispatcher[0]. It's written in Rust, so also a
| single binary with no third party dependencies. We do _NOT_
| however offer in memory storage. This is on purpose, as
| persistence is a requirement for robustness.
|
| [0] https://news.ycombinator.com/item?id=30347858
| jedberg wrote:
| > We do NOT however offer in memory storage. This is on
| purpose, as persistence is a requirement for robustness.
|
| Oooo shots fired. :)
|
| I can see arguments both ways. You're totally correct in
| that queue robustness requires persistence. That being
| said, if it's an open source tool then it can make sense to
| offer in memory storage since they may have persistence in
| a different part of the stack.
| tasn wrote:
| Sure, but then you'd want to hide it behind an
| `--i-really-know-what-im-doing` flag to prevent people
| from shooting themselves in the foot.
|
| Also, even if the persistence is elsewhere in the stack,
| it's not obvious to me how disaster recovery will
| actually work. Same with accounting (e.g. marking a
| message as sent). I'm not saying it's not possible, it's
| just not immediately obvious to me how you can make that
| both simple and robust.
| subomi wrote:
| Oh, for clarity we do not support an in memory storage.
| We support an on disk storage & in memory queue. This
| enables the storage layer persist events across restarts.
| abraham wrote:
| Password validation is broken. Tring to sign up using the
| password "sweet85_been_tries_These" shows a generic error message
| in the UI:
|
| > Http Failure Response For Https://Saas-
| Api.Getconvoy.Io/Api/V1/Users: 400 OK
|
| Looking at the HTTP response in DevTools shows the following
| error:
|
| > Validation failed: Password Password should be minimum of eight
| characters, including at least 1 alphabet and one number
|
| The password should be valid according to the error message.
| subomi wrote:
| Yikes. Checking this out.
| subomi wrote:
| I just pushed a fix for this. You can try again! :)
| moondev wrote:
| FYI I just tried to sign up and received a non specific api
| error via UI modal. Will try again later the product looks
| very interesting.
| subomi wrote:
| Thanks!
|
| If you can please paste the error here, so we can look
| into it.
| _whiteCaps_ wrote:
| Nitpick, in your 'Rate Limiting' section:
|
| Avoid blasting too much events to a single endpoint at once with
| flexible rate limiting controls
|
| I would write this as:
|
| Avoid blasting too _many_ events
| subomi wrote:
| Hmm. That sounds more concise. Thanks for the feedback!
| zomglings wrote:
| Nice, starred. We've been thinking about adding webhook
| functionality to our API but didn't have time to work on it from
| scratch. Time permitting, we'll give Convoy a shot. Thank you.
| subomi wrote:
| Nice, nice. Please send us bug reports. :)
| WoahNoun wrote:
| I know it's a different industry, but there is an existing start-
| up named Convoy. Has 500+ employees and $675M in funding.
| zokier wrote:
| Very timely as I have just recently happened to be involved in
| building a webhook system. Some observations:
|
| 1) For something specifically advertised "cloud-native" I would
| have hoped to see direct integration to e.g. EventBridge/SNS/SQS
| (or their GCP/Azure equivalents) instead of plain REST API.
|
| 2) It is bit weird that you need both Redis and MongoDB here. Is
| there particular reason why redis alone would not be sufficient?
|
| 3) I couldn't at a glance find any docs on the API for receiving
| the webhooks beyond that small ruby example snippet.
|
| 4) You clearly are aware of the security things Stripe has done
| with their webhook implementation, so it is surprising that you
| have fair bit simpler model from the looks of it
|
| Some specific security aspects that I noted:
|
| a) Not having some sort of keyids or multiple signatures makes
| key rotation probably annoying
|
| b) Keys are application-wide instead of per-endpoint
|
| c) Only shared-secret available, no public/private keypair
| option?
|
| d) No timestamping for replay protection?
|
| Not trying to be too negative about your solution, but its just
| that I have been thinking exactly these problems myself then they
| are kinda on top of my head.
|
| There is a draft IETF specification for request signing, that
| seems relevant here. Maybe you could adopt it in Convoy too?
| https://datatracker.ietf.org/doc/draft-ietf-httpbis-message-...
| subomi wrote:
| Thanks for this well detailed feedback.
|
| 1) Yes, it's on our roadmap.
|
| 2) It was/is a design choice, we aren't so familiar with using
| redis as a persistence store. The goal was to enabled people
| switch backends depending on what they're comfortable with.
|
| 3) Thanks for bringing this up, there are a ton of things we
| should document.
|
| 4) We're working on improving things from a security
| perspective.
|
| On application-wide keys, we're considering making this
| configurable, so you can choose what works best for you, we
| only started with app-wide keys, next is endpoint-keys. :)
|
| On public/private keypair, this is new to me as well! we
| haven't found anyone doing this in prod. But yes, I can see how
| this can be useful in specific environments, definitely
| something we'd like to support.
| ushakov wrote:
| how does one receive YC-funding for open-source project?
| tasn wrote:
| There are a lot of cool YC open source companies!
|
| PostHog, Supabase, Wasmer, Mattermost, Chatwoot and of course
| Svix, just to name a few!
| syrusakbary wrote:
| Thanks for the Wasmer mention!
|
| YC investment in OSS companies have grown considerably in the
| last years. They are doing a terrific work identifying
| companies with potential on the space. I'm incredibly hopeful
| seeing where things are heading :)
| subomi wrote:
| Oh, YC funds open-source projects. They're several of them
| before us. :)
| ushakov wrote:
| good to know, maybe i should apply someday :D
| syrusakbary wrote:
| Please do!
| toomuchtodo wrote:
| Congrats on the launch, huge space with lots of product market
| fit to go around, value prop is very clear.
|
| Do you have any plans to support PostgreSQL or an S3 compatible
| target as storage backends?
| subomi wrote:
| Thank you!
|
| Yes, we have plans to support PostgreSQL as a storage backend.
| At the moment, queue tuning parameters are not exposed as
| configuration parameters, they're based off a shared queue and
| we expose prometheus metrics over the shared queue. What use-
| cases do you have for tuning the queue?
| toomuchtodo wrote:
| I might not have enough context, but is it possible based on
| your architecture the shared queue could back up if specific
| destination endpoints aren't accessible when workers are
| processing the queue (I have seen this happen before in an
| adjacent space product)? If this isn't the case, tuning is
| probably not necessary if temporarily undeliverable messages
| can be skipped and retried later without impacting queue
| throughput. Being able to select, group and pause processing
| of queued messages in an ad hoc fashion (tuning was likely
| the wrong word, my apologies), I've found, is a powerful
| feature of webhook engines.
| subomi wrote:
| Oh, yes.
|
| So essentially Convoy's queue mechanism works as a delayed
| job queue (basically how retries in webhooks work e.g.
| Stripe will send the event to you every hour for the next
| 72 hours). When the endpoint is unaccessible it backs up
| and schedules the retries at a future time on the queue.
|
| The implementation under the hood is a combination of redis
| zset + redis streams. :) When it maxs out retries the
| events are still stored in the DB and you can batch retry
| them once the endpoints become accessible.
| toomuchtodo wrote:
| Very cool, thank you!
| subomi wrote:
| You're welcome. :)
| fcjr wrote:
| There's also Svix which is based on Postgres, also YC backed,
| and has been around for a while. (Disclaimer: I work at Svix)
|
| https://github.com/svix/svix-webhooks
___________________________________________________________________
(page generated 2022-02-25 23:00 UTC) |