[HN Gopher] Build Personal ChatGPT Using Your Data
___________________________________________________________________
 
Build Personal ChatGPT Using Your Data
 
Author : raghavankl
Score  : 81 points
Date   : 2023-07-08 21:07 UTC (1 hours ago)
 
web link (github.com)
w3m dump (github.com)
 
| eminent101 wrote:
| Is it going to send my personal data to OpenAI? Isn't that a
| serious problem? Does not sound like a wise thing to do, not at
| least without redacting all sensitive personal data from the
| data. Am I missing something?
 
| cloudking wrote:
| Am I the only one who doesn't need to search across my data? What
| are the use cases here
 
| csjh wrote:
| Why have the OpenAI dependency when there's local embeddings
| models that would be both faster and more accurate?
 
  | yawnxyz wrote:
  | Which ones?
 
    | minimaxir wrote:
    | all-MiniLM-L6-v2 from SentenceTransformers is the most
    | popular one as it balances speed and quality well:
    | https://www.sbert.net/docs/pretrained_models.html
 
| zikohh wrote:
| Also what does this do that llamaindex doesn't?
 
| chaxor wrote:
| The most frustrating thing about the many, _many_ clones of this
| exact type of idea is that pretty much _all_ of them require
| OpenAI.
| 
| Stop doing that.
| 
| You will have way more users if you make OpenAI (or anything that
| requires cloud) the 'technically possible but pretty difficult
| art of hoops to make it happen' option, instead of the other way
| around.
| 
| The best way to make these apps IMO is to make them work
| _entirely_ locally, with an easy string that 's swappable in a
| .toml file to any huggingface model. Then if you _really_ want
| OpenAI crap, you can make it happen with some other docker secret
| or `pass` chain or something with a key, while changing up the
| config.
| 
| The default should be local first, do as much as possible, and
| then _if the user /really/ wants to_, make the collated prompt
| send a very few set of tokens to openAI.
 
  | hospitalJail wrote:
  | ClosedAI has freaked me out with how much power they have, and
  | how irresponsible they are with it.
  | 
  | I'm so horrified that they are going to take away the ability
  | to ask medical questions when the AMA comes knocking at their
  | door.
 
  | Der_Einzige wrote:
  | Even GPT-3.5-turbo-16K isn't good enough for most retrieval
  | augmented generation tasks.
  | 
  | Locally ran LLMs are far worse.
  | 
  | I don't like it either, but for now, if you want good RAG, you
  | have to use GPT-4
 
  | sheeshkebab wrote:
  | Amen, local first should be the default for anything that sucks
  | all my data.
  | 
  | Although until these things can do my laundry none of them
  | deserve any of my compute time either.
 
  | persedes wrote:
  | Gpt4all does exactly that. You can choose between local model
  | or bring your own openai token.
 
  | trolan wrote:
  | The only OpenAI 'crap' being used here is to generate the
  | embeddings. Right now, OpenAI has some of the best and cheapest
  | embeddings possible, especially for personal projects.
  | 
  | Once the vectors are created tho, you're completely off the
  | cloud if you so choose.
  | 
  | You can always swap out the embedding generator too, because
  | LangChain abstracts that for your exact gripes.
  | 
  | Everything else is already using huggingface here and can be
  | swapped out for any other model besides GPT2 which supports the
  | prompts.
 
    | space_fountain wrote:
    | Do you have citations on OpenAI embeddings being some of the
    | cheapest and best? The signs I've seen points almost in the
    | opposite direction?
 
      | ben_w wrote:
      | The only embeddings I currently see listed on
      | https://openai.com/pricing are Ada v2, at $0.1/million
      | tokens.
      | 
      | Even if the alternative is free, how much do you value your
      | time, how long will it take to set up an alternative, and
      | how much use will you get out of it? If you're getting less
      | than a million tokens and it takes half an hour longer to
      | set up, you'd better be a student with zero literally
      | income because that cost of time matches the UN abject
      | poverty level. This is also why it's never been the year of
      | linux on the desktop, and why most businesses still don't
      | use Libre Office and GIMP.
      | 
      | I can't speak for quality; even if I used that API
      | directly, this whole area is changing too fast to really
      | keep up.
 
      | colobas wrote:
      | Can you elucidate on what those signs are? Thanks in
      | advance
 
        | muggermuch wrote:
        | As per the Massive Text Embedding Benchmark (MTEB)
        | Leaderboard maintained by Huggingface, OpenAI's embedding
        | models are _not_ the best.
        | 
        | https://huggingface.co/spaces/mteb/leaderboard
        | 
        | Of course, that's far from saying that they're the worst,
        | or even headed that way. Just not the best (those would
        | be a couple of fully opensource models, including those
        | of the Instructor family, which we use at my workplace).
 
  | AussieWog93 wrote:
  | Honestly, for me, getting good quality results matters way more
  | than keeping my searches private. And for that, nothing
  | compares with GPT4.
 
  | dmezzetti wrote:
  | txtai makes it easy to use Hugging Face embeddings and Faiss,
  | all local and configurable. https://github.com/neuml/txtai
  | 
  | paperai is a sub-project focused on processing
  | medical/scientific papers. https://github.com/neuml/paperai
  | 
  | Disclaimer: I am the author of both
 
  | jstummbillig wrote:
  | What do you (or anyone else, feel free to chime in) do with
  | other LLMs that makes them useable for anything that is not
  | strictly tinkering?
  | 
  | Here is my premise: We are past the wonder stage. I want to
  | actually get stuff done efficiently. From what I have tested so
  | far, the only model that allows me to do that halfway reliably
  | is GPT-4.
  | 
  | Am I incompetent or are we really just wishfully thinking in HN
  | spirit that other LLMs are a lot better at being applied to
  | actual tasks that require a certain level of quality,
  | consistency and reliability?
 
  | JimmyRuska wrote:
  | It's difficult to compete. A small business might answer 10,000
  | requests to their chat bot. The options are
  | 
  | - Pay openai less than $50mo
  | 
  | - Manage cloud gpus, hire ml engineers > $1000/mo
  | 
  | - Buy a local 4090 and put it under someone's desk, $no
  | reliability +$1500 fixed
  | 
  | Any larger business will need scalability and you still can't
  | compete with openai pricing.
  | 
  | Maybe one of you startup inclined people can make an openllama
  | startup that charges by request and allows for finetuning,
  | vector storage
 
    | srcthr wrote:
    | People don't scale. This is personal. Only 3 is a good choice
    | for people in a site with the name hacker something.
 
  | zikohh wrote:
  | Have you seen PrivateGPT. It's quite good and free.
 
    | yoyopa wrote:
    | what hardware do you need for that?
 
      | drdaeman wrote:
      | Consumer-grade, AFAIK it's GPT4All with LLaMA.
 
        | EGreg wrote:
        | Link?
 
        | hoopsman wrote:
        | Presumably https://github.com/imartinez/privateGPT
 
    | mvkel wrote:
    | It's not nearly usable. It's functional in that it spits out
    | a response. Can that response be used for anything useful?
    | No.
 
  | [deleted]
 
  | akira2501 wrote:
  | > is that pretty much all of them require OpenAI.
  | 
  | They're not here to release an actual product. They're here to
  | release part of a CV proving they have "OpenAI" experience. I'm
  | assuming this is the result of OpenAI not actually having any
  | homegrown certification program of their own.
 
    | gumby wrote:
    | > OpenAI not actually having any homegrown certification
    | program
    | 
    | A bit off topic but where are certifications (e.g. Cisco,
    | Microsoft) useful? I am sure they _are_ useful (both to
    | candidates and companies) because people go to the effort to
    | get these certs, and if they were useless everyone would have
    | stopped long ago. I don 't assume people do it for ego
    | satisfaction.
    | 
    | But I've never worked anywhere where it has come up as an
    | interview criterion (nobody has ever pointed it out when we
    | are looking at a resume, for example). Is it a big business
    | thing? Is it just an HR thing?
 
      | n4te wrote:
      | I've only ever seen it as a people who don't have a job
      | thing.
 
      | tw04 wrote:
      | Mainly when applying for a corporate job where you have 0
      | referrals. It's a guidepost that you at least have some
      | idea what you're doing and are worth interviewing when
      | people can't find someone who knows you and your previous
      | work.
 
| AJRF wrote:
| Is there a company that makes a hosted version of something like
| this? I quite want a little AI that I can feed all my data to to
| ask questions to.
 
  | luccasiau wrote:
  | https://libraria.dev/ offers this and more as a service. It has
  | added conveniences like integration with your google drive,
  | youtube videos, and such
 
  | egonschiele wrote:
  | Depending on the size of your data, chiseleditor.com is a free
  | option.
 
| einpoklum wrote:
| Don't build a personal ChatGPT, and don't let OpenAI, Microsoft
| and their business partners (and probably the US government) have
| a bunch of your personal and private information.
 
  | WhackyIdeas wrote:
  | So avoid all Microsoft products too?
 
| hi wrote:
| Keep your data private and don't leak it to third parties. Use
| something like privateGPT (32k stars). Not your keys, not your
| data.
| 
| "Interact privately with your documents using the power of GPT,
| 100% privately, no data leaks"[0]
| 
| [0] https://github.com/imartinez/privateGPT
 
  | leach wrote:
  | How does this run on an Intel Mac? I have a 6 core i9. Haven't
  | been able to get an M series yet so Im wondering if it would be
  | more worth it to run it in a cloud computing environment with a
  | GPU.
 
    | hi wrote:
    | >Mac Running Intel When running a Mac with Intel hardware
    | (not M1), you may run into clang: error: the clang compiler
    | does not support '-march=native' during pip install.
    | 
    | If so set your archflags during pip install. eg:
    | ARCHFLAGS="-arch x86_64" pip3 install -r requirements.txt
    | 
    | https://github.com/imartinez/privateGPT#mac-running-intel
 
      | leach wrote:
      | I'm curious about the response times though, i imagine they
      | will be quite slow on an intel Mac
 
  | 3rdrodeo wrote:
  | [flagged]
 
    | EGreg wrote:
    | Oh, not this again. No, it's not a net negative in ALL forms.
    | And if you really were concerned about downsides, AI has a
    | ton more potential downsides than Web3 ever did, including
    | human extinction, as many of its top proponents have come out
    | and publicly said. Nothing even remotely close to that is the
    | case for Web3 at all:
    | 
    | https://www.nytimes.com/2023/05/30/technology/ai-threat-
    | warn...
    | 
    | https://www.theguardian.com/technology/2023/may/10/ai-
    | poses-...
    | 
    | https://www.theguardian.com/technology/2023/may/30/risk-
    | of-e...
    | 
    | https://www.theverge.com/2023/5/30/23742005/ai-risk-
    | warning-...
 
      | 3rdrodeo wrote:
      | " many of its top proponents have come out and publicly
      | said." You don't have to uncritically accept that, it's far
      | more likely that they're just self aggrandizing in a "wow
      | I'm so smart my inventions can destroy the world, better
      | give me more money and write articles about me to make sure
      | that doesn't happen".
 
        | EGreg wrote:
        | I see, so when it comes to the top experts in AI
        | screaming "we made a mistake, please be careful" we
        | should use nuance and actually conclude the opposite --
        | that we should press ahead and they're wrong.
        | 
        | But with Web3, we should just listen to a bunch of random
        | no-name haters say "there are NO GOOD APPLICATIONS, trust
        | us, period, stop talking about it", use no nuance or
        | critical thinking of our own, and simply stop building on
        | Web3.
        | 
        | Do you happen to see the extreme double standard here
        | you're employing, while trying to get people to see
        | things your way?
 
        | 3rdrodeo wrote:
        | The crypto group had a lot of time and even more money to
        | make a compelling product that took off and so far
        | they've failed. We've watched fraud after fraud as
        | they've shown themselves to just be ignorant and arrogant
        | ideologues who don't understand how the "modern" finance
        | system came to be, what the average user wants out of
        | financial or social products, or just outright scammers.
        | We can keep sinking money into a bottomless pit or we can
        | move on and learn from their mistakes.
        | 
        | I didn't say to dismiss any concerns out of hand, but the
        | whole idea of "x-risk" or "human extinction" from ai is
        | laughable and isn't taken seriously by most people. Again
        | if you think critically about the whole idea of "human
        | extinction" from any of the technology being talked about
        | you _should_ see it as nonsense.
 
        | peyton wrote:
        | I dunno, on the crypto side stablecoins are pretty
        | compelling for hassle-free cross-border transfers--
        | there's $125bn in circulation, which to me means it's
        | taken off.
        | 
        | On the AI side, I mean for example it's not laughable to
        | think anybody on the planet could just feed a bunch of
        | synthetic biology papers to a model and start designing
        | bioweapons. It's not hard to get your hands on secondhand
        | lab equipment...
 
  | unshavedyak wrote:
  | Is this robust enough to feed all your emails and chat logs
  | into it and have convos with it? Will it be able to extract
  | context to figure out questions to recent logs, etc?
 
  | SecurityNoob wrote:
  | 100% private? Hmm. I think with the amount of paranoia that the
  | folks in power have about local LLM's, I wouldn't be in the
  | slightest surprised that the Windows telemetry will be
  | reporting back what people are doing with them. And anyone who
  | thinks otherwise is in my view just absolutely naive beyond
  | hope.
 
| quickthrower2 wrote:
| The author has a demo of this here:
| https://www.swamisivananda.ai/
 
| syntaxing wrote:
| gpt4all has this truly locally. I recommend those with a decent
| GPU to give it a go.
 
  | fbdab103 wrote:
  | I assume this is the link: https://github.com/nomic-ai/gpt4all
  | ?
 
| JimmyRuska wrote:
| Anyone know how milvus, quickwit, pinecone compares?
| 
| I've been thinking about seeing if there's consulting
| opportunities for local businesses for LLMs, finetuning/vector
| search, chat bots. Also making tools to make it easier to drag
| and drop files and get personalized inference. Recently I saw
| this one pop into my linkedin feed, https://gpt-trainer.com/ .
| There's been a few others for documents I've found
| 
| https://www.explainpaper.com/
| 
| https://www.konjer.xyz/
| 
| Nope nope, wouldn't want to compete with that on pricing. Local
| open source LLMs on a 3090 would also be a cool service, but
| wouldn't have any scalability.
| 
| Are there any other finetuning or vector search context startups
| you've seen?
 
| gigel82 wrote:
| I don't get it, GPT-2 is (one of the few) open models from
| OpenAI, you can just run it locally, why would you use their API
| for this? https://github.com/openai/gpt-2
 
___________________________________________________________________
(page generated 2023-07-08 23:00 UTC)