[HN Gopher] Choosing vector database: a side-by-side comparison
___________________________________________________________________
 
Choosing vector database: a side-by-side comparison
 
Author : emilfroberg
Score  : 121 points
Date   : 2023-10-04 13:04 UTC (9 hours ago)
 
web link (benchmark.vectorview.ai)
w3m dump (benchmark.vectorview.ai)
 
| BenoitP wrote:
| Pricing for pg should be easy to compute
| 
| 20M vectors @768 is about 62GB, for 32bit, not even quantized.
| AWS RDS will put it at 83USD/m (db.t4g.small, 2vcpu 2GB RAM). But
| that's not with egress, backups, etc
| 
| Seems acceptable at least for a POC?
| 
| A better option if you already have the data in the same
| instance, but developer experience being low scares me. Anyone
| tried it? How did it go?
 
  | LunaSea wrote:
  | You will be able to store the data but no query it.
  | 
  | Vector indexes are very large, almost the size of the original
  | data, and that needs to fit into the database memory ideally.
 
| NicoJuicy wrote:
| I'm actually curious on how the new vector DB from cloudflare
| compares.
 
  | emilfroberg wrote:
  | Me too! Couldn't find a lot of information on it yet, but I
  | might have to try it myself to get some benchmarks
 
| emilfroberg wrote:
| I made this table to compare vector databases in order to help me
| choose the best one for a new project. I spent quite a few hours
| on it, so I wanted to share it here too in hopes it might help
| others as well. My main criteria when choosing vector DB were the
| speed, scalability, dx, community and price. You'll find all of
| the comparison parameters in the article.
 
  | andre-z wrote:
  | I'm curious where got the numbers on qps? They are pretty
  | different from our experience. Reached out on LinkedIn. ;)
 
    | emilfroberg wrote:
    | Happy to connect. The benchmark numbers are mostly from ANN
    | Benchmarks. For my use case, the nytimes-256 dataset was most
    | relevant so I used that for the QPS benchmark. I also took a
    | look at the benchmarks you've made at
    | https://qdrant.tech/benchmarks/ and there qdrant seems to be
    | outperforming many others. If I've gotten something wrong
    | here, I'm glad to update the article :)
 
| panarky wrote:
| I'd love to know how vector databases compare in their ability to
| do hybrid queries, vector similarity filtered by metadata values.
| For example, find the 100 items with the closest cosine
| similarity where genre = jazz and publication date between 1990
| and 2000.
| 
| Can the vector index operate on a subset of records? Or when
| searching for 100 closest matches does the database have to find
| 1000 matches and then apply the metadata filter, and hope that
| doesn't reduce the result set down to zero and exclude relevant
| vectors?
| 
| It seems like measuring precision and recall for hybrid queries
| would be illuminating.
 
  | mvcalder wrote:
  | I can't speak to the others, but pgvector indices can "break"
  | hybrid queries. For example, if you select using a where clause
  | specifying metadata (where genre = jazz) and order by distance
  | from a vector (embedding of sound clip); if the index doesn't
  | have a lot (or any) vectors in the sphere of the query vector
  | that also match the metadata it can return no results. I
  | discuss this in a blog post here [1].
  | 
  | [1]: https://www.polyscale.ai/blog/pgvector-bigger-boat/
 
  | mistrial9 wrote:
  | > do hybrid queries
  | 
  | "no" - the graph objects after training are opaque AFAIK
 
    | hobs wrote:
    | Actually a lot of the databases offer filtering before or
    | after similarity search.
 
      | esafak wrote:
      | I'd say it's table stakes today.
 
  | andre-z wrote:
  | There is on-stage filtering approach with extended HNSW
  | https://qdrant.tech/articles/filtrable-hnsw/
 
| brigadier132 wrote:
| None of these vector dbs seem economical outside of enterprise.
 
  | emilfroberg wrote:
  | Many of them are open source and you can host them yourself.
  | That would make it more cost effective. Also someone mentioned
  | https://turbopuffer.com/. That seems like a good alternative if
  | you're looking for something economical.
 
  | esafak wrote:
  | They have open source versions.
 
| kesor wrote:
| Redis is definitely missing in the comparison.
 
| alter123 wrote:
| You might want to add https://turbopuffer.com/ as well now in the
| benchmarks.
 
  | softwaredoug wrote:
  | I agree this looks really promising.
 
  | emilfroberg wrote:
  | Turbopuffer looks like something I would consider. And the
  | pricing looks to be lowest on the list from what I can see
 
    | Sirupsen wrote:
    | Emil if you email me at info@turbopuffer.com I can let you
    | into the alpha :)
 
| dmezzetti wrote:
| I'll add txtai to the list: https://github.com/neuml/txtai
| 
| txtai is an all-in-one embeddings database for semantic search,
| LLM orchestration and language model workflows.
| 
| Embeddings databases are a union of vector indexes (sparse and
| dense), graph networks and relational databases. This enables
| vector search with SQL, topic modeling and retrieval augmented
| generation.
| 
| txtai adopts a local-first approach. A production-ready instance
| can be run locally within a single Python instance. It can also
| scale out when needed.
| 
| txtai can use Faiss, Hnswlib or Annoy as it's vector index
| backend. This is relevant in terms of the ANN-Benchmarks scores.
| 
| Disclaimer: I am the author of txtai
 
  | emilfroberg wrote:
  | Txtai looks interesting, maybe you could help me collect some
  | of the comparision parameters for it?
 
    | dmezzetti wrote:
    | Sure, I'd be happy to do so. Easiest way is probably using
    | the public Slack channel that's accessible via the GitHub
    | page.
 
| [deleted]
 
| citruscomputing wrote:
| Strongly disagree with PGVector's DX being worse than Chroma.
| Installing, configuring, and working with Chroma was infuriating
| -- it's alpha software and has the bugs and rough edges to prove
| it. The tools to support and interface with postgres are battle-
| tested and _so_ much nicer by comparison; getting Chroma working
| took over a week, ripping it out and replacing with PGVector took
| a couple hours.
| 
| Also agree with this[0] article that vector search is only one
| type of search, and even for RAG isn't necessarily the one you
| want to start with.
| 
| [0]: https://colinharman.substack.com/p/beware-tunnel-vision-
| in-a...
 
  | fzliu wrote:
  | Shameless self-plug for milvus-lite:                  $ pip
  | install milvus        $ python        >>> import milvus
  | >>> milvus.start()
 
    | m00x wrote:
    | Gonna add some information here since this isn't very
    | descriptive.
    | 
    | milvus-lite is a bit like sqlite where it runs in-process.
    | Here are some scenarios you'd want to use it in:
    | 
    | - You want to use Milvus directly without having it installed
    | using Milvus - Operator, Helm, or Docker Compose etc. - You
    | do not want to launch any virtual machines or containers
    | while you are using Milvus. - You want to embed Milvus
    | features in your Python applications.
 
  | emilfroberg wrote:
  | Thanks for your input, I've only tried Chroma a little bit so
  | far and had a pretty good experience. What they also have going
  | for them is a big community on discord that can be helpful.
 
  | luckyt wrote:
  | Yeah, I had a similar experience with Chroma DB. On paper, it
  | checked all my boxes. But yea, it's alpha software with the
  | first non-prerelease version only coming out in July 2023 (so
  | it's 3 months old).
  | 
  | I ran into some dumb issues during install like the SQLite
  | version being incorrect, and there wasn't much guidance on how
  | to fix these problems, so gave up after struggling for a few
  | hours. Switched to PGVector which was much simpler to setup. I
  | hope Chroma DB improves, but I wouldn't recommend it for now.
 
| magden wrote:
| I don't think we need specialized databases for vectors.
| Relational databases can easily be expanded by vector data types
| and operations. They will eventually catch up by supporting what
| was once a unique feature of the new system:
| https://medium.com/@magda7817/two-things-to-keep-in-mind-bef...
 
  | emilfroberg wrote:
  | Yeah, maybe they will.. But for now, the best options are the
  | purpose-built vector databases, so why not use them?
 
  | rnk wrote:
  | Yeah, this is my sense too. They will be slower to add these
  | new requirements but they should be able to add these vector
  | capabilities within a year or so. It's then a question of
  | ability of smaller vector db companies to mature and add
  | regular db capabilities, while innovating.
 
| krishadi wrote:
| Latency from embedding models is still going to be the bottleneck
| for performance however fast the DB is going to be. Plus adding
| all the overhead of synthesising answers and summaries from a LLM
| is going to weigh you down.
 
| Pandabob wrote:
| I've been wondering about Redis as vector database [0].
| 
| [0]: https://twitter.com/sh_reya/status/1661136833848438784
 
  | emilfroberg wrote:
  | I quickly took a look at the redisearch ANN Benchmarks and they
  | seem to stack up against the others (more or less same level as
  | Milvus) in the comparison when it comes to QPS and Latency.
 
  | esafak wrote:
  | Apparently it's possible:
  | https://redis.io/docs/interact/search-and-query/search/vecto...
  | 
  | Euclidean distance, inner product, and cosine similarity are
  | supported.
 
| drewbug01 wrote:
| I really appreciate comparisons like this, although I find myself
| wanting to know more about why certain things are listed the way
| they are.
| 
| For example, pgvector is listed as not having role-based access
| control, but the Postgres manual dedicates an entire chapter to
| it: https://www.postgresql.org/docs/current/user-manag.html
| 
| Hence why I'd be interested to know more about the supporting
| details for the different categories. It may help uncover some
| inadvertent errors in the analysis, but also would just serve as
| a useful jumping-off point for people doing their own research as
| well.
 
  | proleisuretour wrote:
  | Totally agree with the puzzling assortment of a rubric.
  | PostgreSQL supports role based-access control, RBAC. Not to
  | mention, with PostgreSQL and the pgvector extension, you have a
  | whole list of languages ready to use it:
  | 
  | C++ pgvector-cpp C# pgvector-dotnet Crystal pgvector-crystal
  | Dart pgvector-dart Elixir pgvector-elixir Go pgvector-go
  | Haskell pgvector-haskell Java, Scala pgvector-java Julia
  | pgvector-julia Lua pgvector-lua Node.js pgvector-node Perl
  | pgvector-perl PHP pgvector-php Python pgvector-python R
  | pgvector-r Ruby pgvector-ruby, Neighbor Rust pgvector-rust
  | Swift pgvector-swift
  | 
  | Wonder how many of those other Vector databases play nice.
 
  | mritchie712 wrote:
  | Same for Developer experience. If you used Postgres or any
  | other relational db (which I think covers a large % of devs),
  | you could easily argue the dev experience is 3/3 for pgvector.
 
  | sojournerc wrote:
  | That stood out to me as well. I've been playing with pgvector,
  | and there's no reason you can't use row/table role-based
  | security.
  | 
  | I think there's an unmentioned benefit to using something like
  | pgvector also. You don't need a separate relational database!
  | In fact you can have foreign keys to your vectors/embeddings
  | which is super powerful to me.
 
  | hereonout2 wrote:
  | Possibly / quite probably whoever wrote this knows very little
  | about postgres.
 
| BeetleB wrote:
| Let me half hijack to ask a related question:
| 
| I'm building a RAG for my personal use: Say I have a lot of notes
| on various topics I've compiled over the years. They're scattered
| over a lot of text files (and org nodes). I want to be able to
| ask questions in a natural language and have the system query my
| notes and give me an answer.
| 
| The approach I'm going for is to store those notes in a vector
| DB. When I ask my query, a search is performed and, say, the top
| 5 vectors are sent to GPT for parsing (along with my query). GPT
| will then come back with an answer.
| 
| I can build something like this, but I'm struggling in figuring
| out metrics for how _good_ my system is. There are many variables
| (e.g. amount of content in a given vector, amount of overlap
| amongst vectors, number of vectors to send to GPT, and many
| more). I 'd like to tweak them, but I also want some objective
| way to compare different setups. Right now all I do is ask a
| question, look at the answer, and try to subjectively gauge
| whether I think it did a good job.
| 
| Any tips on how people measure the performance/effectiveness for
| these types of problems?
 
  | civilitty wrote:
  | I use lots and lots of domain specific test cases at several
  | layers, numbering in the hundreds or thousands. The score is
  | the number of test cases that pass so it requires a different
  | approach than all or nothing tests. The layers depend on your
  | RAG "architecture" but I test the RAG query generation and
  | scoring (comparing ordered lists is the simplest but I also
  | include a lot of fuzzy comparisons), the LLM scoring the
  | relevance of retrieved snippets before feeding into the final
  | answering prompt, and the final answer. The most annoying part
  | is the prompt to score the final answer, since it tends out to
  | come out looking like a CollegeBoard AP test scoring rubric.
  | 
  | This requires a lot of domain specific work. For example, two
  | of my test cases are "Is it [il]legal to build an atomic bomb"
  | run against the entire USCode [1] so I have a list of sections
  | that are relevant to the question that I've scored before
  | eventually getting an answer of "it is illegal" followdd by
  | several prompts that evaluate nuance in the answer ("it's
  | illegal except for..."). I have hundreds of these test cases,
  | approaching a thousand. It's a slog.
  | 
  | [1] 42 U.S.C. 2122 is one of the "right" sections in case
  | anyone is wondering. Another step tests whether 2121 is pulled
  | in based on the mention in 2122
 
  | TrueDuality wrote:
  | For small personal projects its kind of hard to build metrics
  | like this because the volume of indexed content in the database
  | tends to be pretty low. If you're indexing paragraphs you might
  | consistently be able to fit all relevant paragraphs in the
  | context itself.
  | 
  | What I can recommend is to take the coffee tasting approach.
  | Don't try and test and evaluate individual responses, instead
  | lock the seed used in generation, and use the same prompt for
  | two different runs. Change one variable and do a relative
  | comparison of the two outputs. The variables probably worth
  | testing for you off the top of my head:
  | 
  | * Choice of models and/or tunes
  | 
  | * System prompts
  | 
  | * Temperature of the model against your queries
  | 
  | * Threshold for similarity for document inclusions (you only
  | want relevant documents from your RAG, set it too low and
  | you'll get some extra distractions, too high and useful
  | information might be left out of the context).
  | 
  | If you setup a system to track the comparisons either
  | automatically or by hand that just indicates which side of the
  | change worked better for your use case, and test that same
  | change for a bunch of different prompts you should be able to
  | tally up whether the control or change was more preferred.
  | 
  | Keep those data points! The data points are your bench log and
  | can be invaluable later on for anything you do with the system
  | to see what changed in aggregate, what had the most outsized
  | impact, etc and can guide you to build useful tooling for
  | testing or finding existing solutions out there.
 
  | screye wrote:
  | You might like this -
  | https://www.youtube.com/watch?v=fWC4VxolWAk
  | 
  | Blog on the same topic - https://blog.langchain.dev/evaluating-
  | rag-pipelines-with-rag...
 
  | hobs wrote:
  | For the normal ones
  | https://en.wikipedia.org/wiki/Evaluation_measures_(informati...
  | 
  | The main thing is that there's no "objective" way, but if you
  | rank and label your own data then you can certainly get a
  | ranking that's subjectively well performing according to you.
 
| softwaredoug wrote:
| Everyone I talk to who is building some vector db based thing
| sooner or later realizes they also care about the features of a
| full-text search engine.
| 
| They care about filtering, they care to some degree about direct
| lexical matches, they care about paging, getting groups / facet
| counts, etc.
| 
| Vectors, IMO, are just one feature that a regular search engine
| should have. IMO currently Vespa does the best job of this,
| though lately it seems Lucene (Elasticsearch and Opensearch) are
| really working hard to compete
 
  | emilfroberg wrote:
  | Vespa looks interesting, hadn't seen it before but will
  | definitely take a look at it
 
  | pmc00 wrote:
  | Agreed, vector search is great but it's only one of many tools
  | you can use to create a great search solution.
  | 
  | We recently did a bunch of evaluation work to quantify the
  | differences between keyword search, vector search, hybrid,
  | reranking, etc. across a few datasets. We shared the results
  | here: https://techcommunity.microsoft.com/t5/azure-ai-services-
  | blo...
  | 
  | Disclosure - I work in the Azure Search team.
 
  | vosper wrote:
  | My company is using vector search with Elasticsearch. It's
  | working well so far. IMO Elastic will eat most vector-
  | first/only products because of its strength at full-text
  | search, plus all the other stuff it does.
 
    | treprinum wrote:
    | Amazon was already working on getting rid of ElasticSearch
    | with their Kendra NLP search. Are you sure ElasticSearch has
    | rosy future?
 
      | m00x wrote:
      | They have beef with ES since they took the software, made a
      | bunch of cash on it, then never contributed back. ES called
      | them out and it started a feud.
      | 
      | I'd go on ES over Amazon-built software any day. I worked
      | on RDS and I've used RDS at several companies, it's a mess.
      | 
      | Longer story: One day one of our table went missing on
      | Aurora, we couldn't figure out why, it was in the schema,
      | etc. Devops panicked and restarted the instance, and then
      | another table was missing. We ended up creating 10 empty
      | tables and restarted it until it hit one of those.
      | 
      | We contacted RDS support after that, and the conclusion of
      | their 3 month investigation is: "Yeah, it's not supposed to
      | do that."
      | 
      | There's some really smart people working at Amazon,
      | unfortunately the incentives is to push new stuff out and
      | get promoted ASAP. If you can do that better than others
      | and before your house of cards falls, you're safe. If the
      | house of card crumbles after you're gone, it's their
      | problem.
 
        | vmfunction wrote:
        | >Longer story: One day one of our table went missing on
        | Aurora, we couldn't figure out why, it was in the schema,
        | etc. Devops panicked and restarted the instance, and then
        | another table was missing. We ended up creating 10 empty
        | tables and restarted it until it hit one of those.
        | 
        | Are there any report this? How come this is the first
        | time I heard of this? How can companies trust this kind
        | of managed DB services?
 
      | vosper wrote:
      | Amazon forked ElasticSearch into OpenSearch. When deciding
      | which platform to go with (we are an AWS customer) I
      | decided to stick with the company whose future depends on
      | their search product (Elastic), not the one that could lose
      | interest and walk away and suffer almost no consequences
      | (AWS). If OpenSearch is still around in 5 years, and
      | keeping pace with ElasticSearch, then maybe I'd consider it
      | the next time I'm making this choice.
      | 
      | Also there's a lot more to ElasticSearch than full-text
      | search (aggregations, lifecycle management, Kibana).
      | Doesn't seem like Kendra is going to be a replacement for
      | our use case.
 
    | lordofmoria wrote:
    | I tend to agree - search, and particularly search-for-humans,
    | is really a team sport - meaning, very rarely do you have a
    | single search algo operating in isolation. You have multiple
    | passes, you filter results through business logic.
    | 
    | Having said that, I think pgvector has a chance for less
    | scale-intense needs - embedding as a column in your existing
    | DB and a join away from your other models is where you want
    | search.
    | 
    | I don't get why you'd want to bolt RBAC onto these new vector
    | dbs, unless it's because they've caused this problem in the
    | first place...
 
    | dathinab wrote:
    | it's also has tones of subtle issues and we are constantly
    | looking for potential replacements
 
  | deepsquirrelnet wrote:
  | Until very recently, "dense retrieval" was not even as good as
  | bm25, and still is not always better.
  | 
  | I think a lot of people use dense retrieval in applications
  | where sparse retrieval is still adequate and much more
  | flexible, because it has the hype behind it. Hybrid approaches
  | also exist and can help balance the strengths and weaknesses of
  | each.
  | 
  | Vectors can also work in other tasks, but largely people seem
  | to be using them for retrieval only, rather than applying them
  | to multiple tasks.
 
    | dathinab wrote:
    | more commonly you use approximate KNN vector search with LLM
    | based embeddings, which can find many fitting documents bm25
    | and similar would never manage to
    | 
    | the tricky part if to properly combine the results
 
    | marginalia_nu wrote:
    | A lot of these things are use-case dependent. Like the
    | characteristics even of BM-25 varies a lot depending on
    | whether the query is over or under specified, the nature of
    | the query and so on.
    | 
    | I don't think there will ever be an answer to what is the
    | best way of doing information retrieval for a search engine
    | scale corpus of document that is superior for every type of
    | queries.
 
  | donretag wrote:
  | Vector search is not exclusively in the domain of text search.
  | There is always image/video search.
  | 
  | But pre-filtering is important, since you want to reduce the
  | set of items to be matched on and it feels like
  | Elasticsearch/OpenSearch are fairing better in this regard.
  | Mixed scoring derived from both both sparse and dense
  | calculations is also important, which is another strength of
  | ES/OS.
 
  | ruslandanilin wrote:
  | Vespa.ai does a great job. Absolutely stunning thing!
 
    | esafak wrote:
    | What do you like about it relative to alternatives? How fast
    | is it?
 
      | dathinab wrote:
      | much more mature and feature rich then many of the
      | competition listed in the article
      | 
      | to some degree it's more a platform you can use to
      | efficiently and flexible build your own more complicated
      | search system, which is both a benefit and drawback
      | 
      | some good parts:
      | 
      | - very flexible text search (bm25), more so then elastic
      | search (or at least easier to user/better documented when
      | it comes to advanced features)
      | 
      | - fast flexible enough vector search, with good filtering
      | capabilities
      | 
      | - build in support for defining more complicated search
      | piplines, including multi phase search (also known as
      | rerankin)
      | 
      | - quite nice approach for more fine controlling about what
      | kind of indices are build for which fields
      | 
      | - when doing schema changes has safety checks to make sure
      | you don't accidentally brake anything, which you can
      | override if you are sure you want that
      | 
      | - ton of control in a cluster about where which search
      | system resources get allocated (e.g. which schemas get
      | stored on which storage clusters, which cluster nodes
      | should act as storage nodes, which should e.g. only do
      | preprocessing or post processing steps in a search piplines
      | and which e.g. should be used for calculating embeddings
      | using some LLM or similar) Not something you for demos but
      | definitly something you need once you customers have enough
      | data.
      | 
      | - child documents, and document references
      | 
      | - multiple vectors per document
      | 
      | - quite a interesting set of data types for fields and
      | related ways you can use them in a search pipline
      | 
      | - an flexible reasonable easy to use system for
      | plugins/extensions (through Java only)
      | 
      | - support building search piplines which have sub-searches
      | in extern potentially non vespa systems
      | 
      | - really well documented
      | 
      | Through the main benefit *and drawback* is that it's not
      | just a vector database, but a full fledged search system
      | platform.
 
      | bratao wrote:
      | +1 for Vespa. For me it is VERY resilient and production
      | ready. It is such a dream compared to Elasticsearch, that
      | we migrated from.
 
        | vinni2 wrote:
        | Does Vespa have an equivalent of Kibana? and how hard was
        | the migration?
 
  | [deleted]
 
| deepsquirrelnet wrote:
| What advantage are vector databases providing above using an
| index in conjunction with a mature database? I'm not sold on this
| as a separate technology.
| 
| Vector search is useful, but I don't understand why I would go
| out of my way when I could implement FAISS or HNSWlib as an
| adjunct to postgres or a document store.
 
  | dathinab wrote:
  | The thing is if you need a vector _database_ there is no reason
  | why it can't be a pg extensions. And if you project is only
  | small scale there is probably some HNSW pg extension library
  | you could use.
  | 
  | But what is most times needed instead of a vector database is a
  | efficient fast responsive vectore approximate KNN search system
  | with fast attribute filtering which overlaps with a fast an
  | efficient text search system (e.g. bm25 based)
  | 
  | And if you then go to billion vector scale things become tricky
  | performance wise.
  | 
  | And then you reach the same point at which companies do things
  | like using warehouse approach where you have a read only
  | extremely read optimized mostly in memory variant of their db
  | they access for searches only and changes from their main db a
  | streamed to the read only search instance, potentially while
  | losing snapshot views, transactions and similar.
  | 
  | You could say that approx. KNN vector search is the new must
  | have feature for unstructured fuzzy text search, and while you
  | can have unstructured fuzzy text search in pg it's also often
  | not the go-to solution if your databse is just for getting that
  | search.
 
  | spullara wrote:
  | Vector extensions to your current database or search engine
  | makes far more sense than adding yet another dependency to
  | manage and operate. The vector database folks will have to
  | become a real database or full featured search engine to
  | survive and compete with the incumbents that will all have good
  | solutions for vector similarity search.
 
  | dmezzetti wrote:
  | If you're interested in an approach like this, take a look at
  | txtai.
  | 
  | 1. https://neuml.github.io/txtai/embeddings/indexing/
  | 
  | 2. https://neuml.hashnode.dev/external-database-integration
 
    | deepsquirrelnet wrote:
    | I love this idea. It seems like a very practical approach.
    | I'm going to give this a try on my next project.
 
| ldjkfkdsjnv wrote:
| Postgres vector store has been the most simple, and will be if
| you are at a lower scale. You can just use it directly with
| something like spring boot.
 
  | avthar wrote:
  | Agreed on pgvector being simple and a great choice for POCs and
  | low scale, especially if you're familiar with Postgres. Our
  | team released something new last week built for folks looking
  | to use PostgreSQL at scale as a vector store [0], featuring a
  | DiskANN index type.
  | 
  | [0]: https://www.timescale.com/blog/how-we-made-postgresql-the-
  | be...
 
| la64710 wrote:
| Somehow I felt that at least part of the articles was generated
| by a LLM. It's unfortunate to see that a new bias has started to
| creep up. Whatever I read now I second guess and I _feel_ it
| maybe partially or fully generated by LLMs.
 
| jn2clark wrote:
| As others have correctly pointed out, to make a vector search or
| recommendation application requires a lot more than similarity
| alone. We have seen the HNSW become commoditised and the real
| value lies elsewhere. Just because a database has vector
| functionality doesn't mean it will actually service anything
| beyond "hello world" type semantic search applications. IMHO
| these have questionable value, much like the simple Q and A RAG
| applications that have proliferated. The elephant in the room
| with these systems is that if you are relying on machine learning
| models to produce the vectors you are going to need to invest
| heavily in the ML components of the system. Domain specific
| models are a must if you want to be a serious contender to an
| existing search system and all the usual considerations still
| apply regarding frequent retraining and monitoring of the models.
| Currently this is left as an exercise to the reader - and a very
| large one at that. We (https://github.com/marqo-ai/marqo, I am a
| co-founder) are investing heavily into making the ML production
| worthy and continuous learning from feedback of the models as
| part of the system. Lots of other things to think about in how
| you represent documents with multiple vectors, multimodality,
| late interactions, the interplay between embedding quality and
| HNSW graph quality (i.e. recall) and much more.
 
| donretag wrote:
| Curious about the lack of Vespa, especially given the
| thoroughness of the article and its long-time reputation.
| OpenSearch is also missing, but perhaps it can be considered
| being lumped in with Elasticsearch due to them both being based
| on Lucene. The products are starting to diverge, so would be nice
| to see, especially since it is open-source.
| 
| For the performance-based columns, would be also helpful to see
| which versions were tested. There is so much attention lately for
| vector databases, that they all are making great strides forward.
| The Lucene updates are notable.
 
  | emilfroberg wrote:
  | Someone else also pointed out that Vespa was missing. I'll have
  | to look in to it and add it to the article!
 
| lazy_moderator1 wrote:
| also, typesense
 
| dathinab wrote:
| Their definition about Hybrid Search is I think wrong.
| 
| Through this terms tend to not be consistently defined at all, so
| "wrong" is maybe the wrong word.
| 
| Their definition seem to be about filtering results during
| (approximate) KNN vector search.
| 
| But that is filtering, not hybrid search. Through it might
| sometimes be implemented as a form of hybrid search, but that's
| an internal implementation detail and you probably should hope
| it's not implemented that way.
| 
| Hybrid search is when you do both a vector search and a more
| classical text based search (e.g. bm25) and combine both results
| in a reasonable way.
 
  | emilfroberg wrote:
  | The way you explain hybrid search aligns with my understanding.
  | Pinecone has a good article about it here
  | https://www.pinecone.io/learn/hybrid-search-intro/. From my
  | understanding, all vector DBs support this.
 
| J_Shelby_J wrote:
| Nice post! I think this could be a very good page to bookmark.
| 
| There is also this series of articles detailing the options and
| it includes some that the OP is missing:
| https://thedataquarry.com/posts/vector-db-1/#key-takeaways
| 
| I'm currently in the market for a self hosted DB for a personal
| project. The project is an app you can run on your own system and
| provide QA on your text files. So I'm looking for something light
| weight, but I'm also looking for the best possible search and ANN
| retrieval is just a single part of that.
 
| Havoc wrote:
| 16x difference between pg and milvus?
| 
| I thought for most use cases this would be quite performance
| sensitive
 
  | emilfroberg wrote:
  | Yeah, that's the difference we've seen according to the QPS for
  | the ANN Benchmarks. The same story seems to be true for other
  | datasets too. We're looking at a 0.9 recall.
 
___________________________________________________________________
(page generated 2023-10-04 23:01 UTC)