The Rise of Vector Databases – Lessons from the Milvus Community – Frank Liu & Charles Xie, Zilliz
The total amount of digital data generated worldwide is increasing at a rapid rate. Simultaneously, approximately 80% (and growing) of this newly generated data is unstructured data – data that does not conform to a table- or object-based model. Examples of unstructured data include text, images, protein structures, geospatial information, and IoT data streams. Despite this, the vast majority of companies and organizations do not have a way of storing and analyzing these increasingly large quantities of unstructured data. Embeddings – high-dimensional, dense vectors which represent the semantic content of unstructured data – can remedy this. Armed with this knowledge, it’s clear that the mobile/IoT era necessitates a new type of cloud-native, fully distributed database purpose-built to store, search, and index large quantities of embedding vectors: the vector database. In this presentation, we’ll introduce some of the interesting production use cases we’ve seen with Milvus – the world’s most popular open source vector database. Additionally, we’ll discuss pitfalls and challenges associated with integrating Milvus into a production data/ML stack. Finally, we’ll wrap up with a discussion on ongoing community feedback and future plans for Milvus.
