[HN Gopher] Twitter's Recommendation Algorithm
___________________________________________________________________
 
Twitter's Recommendation Algorithm
 
Author : jonknee
Score  : 769 points
Date   : 2023-03-31 18:42 UTC (4 hours ago)
 
web link (blog.twitter.com)
w3m dump (blog.twitter.com)
 
| corbulo wrote:
| It's disappointing the comments are so obsessed with the
| political angle to this that there's a total lack of appreciation
| (or discussion) of opening up the most influential social media
| platform in the world.
 
  | smt88 wrote:
  | This is transparency theatre, not actual transparency.
  | 
  | There's no way to actually use this limited release to
  | understand how or why any tweet is boosted, so we're in exactly
  | the same boat we were in yesterday.
 
    | corbulo wrote:
    | This sentiment has high correlation to driving conclusions
    | from a very time limited information set. This isn't the only
    | part that is going to be posted to github.
    | 
    | What is the net benefit from rushing to condemn something
    | that can only be a net positive compared to the past
    | alternatives? I don't understand the purpose of that
    | approach. Help me.
 
  | SilverBirch wrote:
  | One of the things that makes my spidey sense tingle is when
  | people say _oddly_ sycophantic things about Elon Musk. Twitter
  | is big, it 's important. It's not "the most influential social
  | media platform in the world".
 
    | corbulo wrote:
    | I only see one social media site posts being constantly
    | reposted on global news organizations. Are all of the world
    | governments and leaders with tens of millions of followers
    | actually wasting their time by dedicating their social media
    | teams time to twitter instead of focusing on some other
    | social media site thats more influential? Which one is it
    | then?
    | 
    | It's quite odd to attribute objective analysis to sycophancy.
    | I intentionally didn't mention him but here you are bringing
    | him up. Who is the sycophant?
 
  | HeckFeck wrote:
  | I have to admit I am geeking out whilst skimming source code I
  | barely understand.
 
  | yurodivuie wrote:
  | I'm sure we can all think of examples where a power structure
  | (a company, a country, a prison, a family) invited people in
  | for a supervised tour that was less than honest in its
  | presentation.
  | 
  | But really, if people respond to Twitter's actions politically,
  | that response exists within a context that was certainly
  | influenced by Twitter's prior actions.
 
  | nonethewiser wrote:
  | The funny thing is that angle owes itself to Elon coming
  | through on his promise to open source this.
  | 
  | This is a great thing.
 
    | deckard1 wrote:
    | It would also be a total Elon move to confuse the open
    | sourcing of Twitter's internal code with actual transparency.
    | 
    | You would need, at a minimum, a neutral third-party audit of
    | Twitter's servers to conclude that the source code we see on
    | GitHub is, in fact, the source code running Twitter. How
    | often will they keep their GitHub repo in sync with their
    | internal code, I wonder.
    | 
    | Presumably Twitter uses a version control system. But they
    | scrubbed the history so that's also a point against their
    | "transparency" claims. Without knowing the when and the why
    | of changes you can't understand what you are looking at.
    | People are pointing to that "author_is_elon" without knowing
    | whether that was done before Elon bought Twitter or after.
    | 
    | But even then, git history can be faked.
    | 
    | > This is a great thing.
    | 
    | I disagree. It's the opposite. It provides the illusion of
    | openness without the quality of openness, thus killing the
    | debate once and for all.
 
  | fanagra32 wrote:
  | "Opening up"? You must be kidding. Nothing is open there. It's
  | just open-washing. A few nice diagrams, but how the services
  | _actually_ work is still hidden.
 
| PenguinRevolver wrote:
| Great pull request here which improves the algorithm:
| https://github.com/twitter/the-algorithm/pull/17
 
  | hrpnk wrote:
  | Aside from the spam PRs, there is actually one PR that fixes a
  | bug: https://github.com/twitter/the-algorithm/pull/242/files
 
    | mdaniel wrote:
    | Modern Java actually allows `10_000` for that very reason, as
    | does Scala (https://scala-
    | lang.org/files/archive/spec/2.13/01-lexical-sy...)
 
      | fooey wrote:
      | also javascript - https://github.com/tc39/proposal-numeric-
      | separator
 
        | bitshiftfaced wrote:
        | Add golang to the list.
 
  | sroussey wrote:
  | Yes please! I definitely put my thumbs up in there!
 
    | drstewart wrote:
    | That will definitely do something! Good job!!
 
  | [deleted]
 
  | BbzzbB wrote:
  | It removes the extra weight to Twitter blue tweets?
 
    | idle_zealot wrote:
    | If the property names are to be believed it sets a weight
    | multiplier to 0. So it prevents recommending them entirely.
 
      | SketchySeaBeast wrote:
      | It sets the default to zero, but apparently can range up to
      | 100. So... what modifies it? (The answer is probably in
      | there somewhere, but I'm sure someone will find it before I
      | do.)
 
  | zachnwhite wrote:
  | [flagged]
 
    | SquareWheel wrote:
    | The trouble with spammy jokes like this is it discourages
    | companies from bothering with open-source in the future. I
    | know I'd be less likely to champion an initiative like this
    | if I thought it might blow up in my face.
 
    | drstewart wrote:
    | Mature. I'm sure companies looking to outsource work with
    | Twin Prime Media would be stoked to see this level of
    | maturity.
 
  | matsemann wrote:
  | Me feed has lately been full of accounts that have blocked me.
  | Like, I see a tweet from someone unknown, click their profile
  | and it says I'm blocked.
  | 
  | So wonder if some value is wrong in one of those constants.
  | Anyways, the blocking feature is broken..
 
  | super256 wrote:
  | I think it's okay to give Twitter Blue users a boost, as it's
  | most likely _not_ spam (unlike the 95% of my non-blue followers
  | who are bots).
 
    | kevincox wrote:
    | I think it makes sense for out-of-network. However I see no
    | value for boosting among people that I have followed. If I
    | have followed them they definitely aren't spammers (from my
    | PoV).
 
  | simonsarris wrote:
  | That would be great (unweighting bluechecks) but they actually
  | plan to go in the other direction: Starting April 15th non-
  | bluechecks won't show up in the "For you" section (the
  | algorithm timeline) _at all._ Unpaid users are being written
  | completely out of the algo.
  | 
  | https://twitter.com/elonmusk/status/1640502698549075972
 
    | alfor wrote:
    | I don't see a way out of this with the GPT/AI able to create
    | fake persona in an instant.
 
      | thefreeman wrote:
      | What does that matter? If people find the content engaging
      | then it will be amplified. If not, it shouldn't be there in
      | the first place. This whole "AI / Bot swarm" excuse is just
      | smoke and mirrors for "I want more people to pay twitter".
 
        | nonethewiser wrote:
        | If _accounts_ find the tweet engaging
 
        | lhnz wrote:
        | If "bots indistinguisable from humans" find your "account
        | that posts pro-russian propaganda" engaging it will be
        | amplified onto your For You page.
 
    | anigbrowl wrote:
    | Inaccurate. Musk stated that people you follow will continue
    | to show up. I only use the 'for you' feed when I'm bored and
    | want stupid dopamine hits, I leave it on Following almost all
    | the time. But that's on desktop, my understanding it that it
    | keeps resetting itself for mobile users (of whom I am not
    | one).
 
      | astrange wrote:
      | He only said that later; he seems to be working on a "say
      | we'll do feature and then change it all after people yell
      | at him for weeks" process.
 
        | mgiannopoulos wrote:
        | Sounds like moving fast and adjusting to user feedback,
        | usually something commendable in tech, right?
 
        | KerrAvon wrote:
        | No. There's an implied willingness to listen to people in
        | the first place when you're agile in response to
        | feedback. You shouldn't have to bruise Elon's ego to get
        | him to do not-stupid things.
 
    | cwkoss wrote:
    | it's a shame we can no longer short twitter stock
 
    | bradly wrote:
    | I believe LeBron James said recently he isn't going to waste
    | his money on a blue checkmark, so it should be interesting to
    | see what stays and what goes.
 
      | zaroth wrote:
      | LeBron doesn't get $84 of value from Twitter? Definitely
      | not a political statement going on there.
 
        | TaylorAlexander wrote:
        | Parent didn't say it's not "political". It's reasonable
        | for a wealthy person to feel that a system that
        | discriminates against the poor is not a system they want
        | to participate in.
        | 
        | (Note that I use discriminate in the literal sense, as a
        | simple statement of fact.)
 
        | web3-is-a-scam wrote:
        | lol you don't actually believe that's his reason do you?
 
        | nonethewiser wrote:
        | But the example you give is an appeal to a universal
        | moral good. Not partisan politics. So despite saying it's
        | not not political, your justification is that it's not
        | political.
        | 
        | Also, how did you get a blue check before being able to
        | buy one?
 
        | TaylorAlexander wrote:
        | Some people unfortunately view concern for the poor as
        | political. However my point of mentioning politics is to
        | say that "it's political" is not any kind of gotcha when
        | it was never denied as being political. Regardless of the
        | actual justification being political or not, the
        | "political" gotcha is nonsense.
 
        | __jem wrote:
        | Twitter doesn't get $84 of value from Lebron?
 
        | 6d6b73 wrote:
        | Like it or not but it's the twitter that gets value from
        | celebrities. How many people are on social networks jusy
        | so see what their fav celebrites are doing?
 
        | nonethewiser wrote:
        | It obviously goes both ways. Social media is a megaphone
        | and ego boost for celebs.
 
        | mullingitover wrote:
        | The problem for twitter is it isn't the only game in town
        | when it comes to social media, not by a long shot.
        | They're not even in the top ten. They're a megaphone in a
        | large pile of megaphones, and those other megaphones
        | don't bite the hand that picks them up.
 
        | ceejayoz wrote:
        | Twitter needs the LeBrons of the world far more than they
        | need Twitter.
 
      | anigbrowl wrote:
      | Most of the major news outlets are not doing so either. The
      | Elon stans are crowing that this will be the long-overdue
      | end of legacy media, but it strikes me that the new 'blue
      | check twitter' might end up becoming even more of a social
      | bubble than what it replaced. There are _so_ many low
      | quality accounts sporting a checkmark now that users who
      | value substance will soon be incentivized to just block
      | anyone they find annoying.
 
        | Avshalom wrote:
        | yeah, they, uh seem to have realized that's gonna be a
        | problem.
        | 
        | https://variety.com/2023/digital/news/twitter-business-
        | month...
        | 
        | and of course all this means is that the organizations
        | most likely to be able to afford it won't have to
 
        | anigbrowl wrote:
        | Good find! It'll be funny to see if the incumbents
        | respond with 'don't do me any favors.' Also to see
        | whether Musk's frens sulk aboit him selling out to the
        | elites or so - their gratitude has an extremely short
        | half-life.
 
      | nonethewiser wrote:
      | So you need a blue check mark to reach non followers and
      | Lebron won't get one? Sounds like it's making Twitter
      | better already.
 
      | MKais wrote:
      | NY Times, WaPo, LA Times and other major accounts too
      | https://www.thewrap.com/ny-times-la-times-not-pay-for-
      | twitte...
 
        | nonethewiser wrote:
        | Seems dumb of them. Cost is trivial and their competition
        | that isn't so politically motivated will have a much
        | further reach.
        | 
        | The smart move would be silent on the policy change, pay,
        | and support rival platforms as they can. Instead they
        | will eventually pay and look like they lost.
 
        | mullingitover wrote:
        | > Seems dumb of them. Cost is trivial and their
        | competition that isn't so politically motivated will have
        | a much further reach.
        | 
        | It's wild hubris for twitter to try to invoice/penalize
        | the very users and organizations that make twitter
        | anything but insolvent. There should be money exchanged
        | here, but it should be flowing _generously_ and most
        | importantly _in the other direction_.
 
        | AuryGlenz wrote:
        | We're in a time where ideology trumps revenue for some
        | companies. You know, "Get woke, go broke."
 
        | flangola7 wrote:
        | Everyone says that about things that made tons of money
        | though.
 
        | fooey wrote:
        | It's smart because Musk already blinked
        | 
        | The top 10,000 are getting exemptions and won't have to
        | pay
        | 
        | https://variety.com/2023/digital/news/twitter-business-
        | month...
 
        | runako wrote:
        | For the NYT to verify their official accounts plus those
        | of their reporters (using the Twitter Blue Affiliations
        | feature) would be $1m annually. This, for a budget line
        | item that has heretofore been $0. In this economy, that's
        | a reach.
        | 
        | I don't think the NYT is worried about "reach."
 
        | mgiannopoulos wrote:
        | How do you get the 1 million figure?
 
        | tick_tock_tick wrote:
        | > I don't think the NYT is worried about "reach."
        | 
        | LOL they are desperate for reach. Incredibly so; have you
        | not listen to any podcast by them? They are begging
        | people to go to their site. They get a fraction of the
        | organic traffic they used to and nearly everything is
        | driven from other site like Twitter, Google News,
        | Facebook, etc. The internet age has not been kind to
        | classic news orgs.
 
        | astrange wrote:
        | The NYT has been doing great recently. They're probably
        | the legacy news company that's doing the best online of
        | anyone.
        | 
        | That's on the strength of having a lot of verticals like
        | games, recipes and Wirecutter though.
 
        | adamrezich wrote:
        | have you seen @nyttypos and (to a lesser extent)
        | @nyt_diff? NYT online editorial standards are hilariously
        | abysmal.
        | 
        | https://twitter.com/nyttypos
        | 
        | https://twitter.com/nyt_diff
 
        | astrange wrote:
        | Yeah and it doesn't matter.
 
      | drstewart wrote:
      | Depends on how they weight the is_user_china_mouthpiece
      | variable
 
  | willmeyers wrote:
  | Why do companies even bother to put source up on github? To put
  | up a front that their open source? What a joke.
 
| tric wrote:
| GitHub repo: https://github.com/twitter/the-algorithm/
 
  | minimaxir wrote:
  | Notably, it's AGPL-licensed.
 
    | [deleted]
 
  | hooverd wrote:
  | I wonder how useful this is without the knowledge and tooling
  | around deploying it.
 
    | rurp wrote:
    | That's my thought as well. Complicated system like this rely
    | on all sorts of related services and data stores. This seems
    | like the sort of thing that sounds a lot more interesting
    | than it is in practice. I would bet many non-technical people
    | expect "The Algorithm" to be a straightforward and self-
    | contained system.
 
  | [deleted]
 
| capableweb wrote:
| I'm no fan of either Twitter nor Elon Musk, but this is a great
| move and I hope other companies follow what Twitter did here and
| start open sourcing more core parts like this. Maybe it's mostly
| useful for learning how it works, not for directly using it in
| your own product, but the amount of transparency it gives users
| cannot be understated. As long as that actually is the code they
| run, but there would be no way for anyone but Twitter to verify
| that.
 
  | cubefox wrote:
  | I think it mainly helps with accountability regarding free
  | speech. They did and do several kinds of shadow banning and
  | down-boosting to combat spammers, which always has some false
  | positives. If you the algorithm is published, you could at
  | least better judge and argue when you are unfairly "silenced".
  | Since this may be due to an avoidable flaw of the algorithm
  | instead of some accepted collateral damage.
 
| elashri wrote:
| I wonder if it will be possible in one day to know what is values
| of `author_is_power_user`, `author_is_democrat` and
| `author_is_republican` for your account. Does GDPR help with
| that? probably not because maybe they do it for people inside the
| us only so it is not related to EU anyway.
 
| photochemsyn wrote:
| I generally have a very low opinion of social media platforms,
| but I did create a Twitter account for the first time after Musk
| bought the platform.
| 
| My conclusion is that it's basically entertainment, with very
| little of what I'd call high-quality useful information that
| deserves further examination (unlike a lot of HN posts, in
| contrast). I also notice something of a Tik-Tok approach to video
| being implemented, which is not surprising given Tik-Tok's
| success (and makes one wonder who exactly it is lobbying so hard
| for a Tik-Tok ban, and whether it's just a commercial competition
| issue more than anything else).
| 
| As far as the recommendation algorithm, it appears to be a
| siloing setup - look at content of one particular flavor, it
| gives you more of that flavor. A 'flush settings' or 'forget
| browsing history' or 'reset to defaults' button would be useful,
| if probably not what advertisers want in terms of delivering to
| target audiences. I suppose setting up multiple accounts is
| something of a solution, although too much effort to be that
| interesting.
| 
| In terms of news reports, it's broader in scope than traditional
| corporate media outlets, so that's a plus in its favor.
| Reliability is perhaps similar (i.e. low).
 
  | lhnz wrote:
  | You can follow accounts that only post arxiv.org links for ML
  | papers or anything else you're interested in if you want to. If
  | you're only getting entertainment then it says a lot about the
  | original accounts you followed.
 
| sroussey wrote:
| Does it show the part where is recommends Elon more than anyone
| else?
 
  | jmholla wrote:
  | I think this PR is modifying the inputs to the methods that do
  | it: https://github.com/twitter/the-algorithm/pull/17
 
  | devrand wrote:
  | I couldn't find anything specific to that, but I did find thus
  | blurb where they seem to explicitly track how often they're
  | serving Elon's tweets for A/B testing experiments:
  | https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
 
  | Chinjut wrote:
  | Perhaps that's related to this line. Though perhaps this is
  | just used for observing metrics.
  | https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
 
    | JasonZ2 wrote:
    | [dead]
 
    | jrwr wrote:
    | That whole list is a hoot,
    | 
    | has_toxicity_score_above_threshold
    | 
    | is a interesting value, I wonder were the 0.91 was though up
    | at
 
  | ano-ther wrote:
  | This is one: https://github.com/twitter/the-
  | algorithm/issues/121
  | 
  | Search for Elon gives this: https://github.com/twitter/the-
  | algorithm/search?q=Elon&type=
 
| tech234a wrote:
| I wonder what the "author_is_elon", "author_is_power_user",
| "author_is_democrat", and "author_is_republican" labels are for
| [1].
| 
| [1]: https://github.com/twitter/the-algorithm/blob/main/home-
| mixe...
 
  | Someone1234 wrote:
  | Here is a screenshot in case this changes later:
  | 
  | https://i.imgur.com/F8GSeyH.png
  | 
  | And, no, this wasn't in a merge-request, it was in the "main"
  | branch of HomeTweetTypePredicates.scala.
 
    | tantalor wrote:
    | What's all the "DDG"? Is this data from DuckDuckGo?
 
      | asddubs wrote:
      | I doubt it, since the isElon thing also has the
      | abbreviation
 
      | [deleted]
 
      | tyingq wrote:
      | Probably something like "Data Distribution Group".
 
      | nanny wrote:
      | Maybe the name for an internal service/environment. It's
      | also referenced in this viral tweet from November:
      | 
      | https://twitter.com/EricFrohnhoefer/status/1591969100225736
      | 7...
 
      | bcherry wrote:
      | DuckDuckGoose is the twitter A/B test framework
 
        | mcast wrote:
        | That's actually kind of a cool and fitting name.
 
        | nonethewiser wrote:
        | I see Twitter engineers are not from Minnesota.
 
  | bilekas wrote:
  | All in service of 'anti-bias' of course... /s
 
  | [deleted]
 
  | RoyGBivCap wrote:
  | [dead]
 
  | spaceman_2020 wrote:
  | pretty sure Elon gets a boost in the algorithm. All okay - he's
  | the owner of a private entity and can do as he pleases.
 
    | mnd999 wrote:
    | That's what he blew all that cash on. It's the whole point.
 
  | montag wrote:
  | Elon is addressing this in the Twitter Space right now. "It
  | definitely shouldn't be dividing people into Republican and
  | Democrats; that makes no sense[...] you've identified something
  | we should be getting rid of right away."
 
    | asddubs wrote:
    | but how can we be sure it isn't doing that?! first, we would
    | need to figure out a way to identify who-
 
    | Imnimo wrote:
    | As if Elon has a clue what that feature is or is not being
    | used for.
 
      | dmix wrote:
      | He didn't say he knew what it did? It's a good enough
      | response to say that it shouldn't be doing that period.
 
    | franky47 wrote:
    | We should also not divide people into Elon and non-Elon.
 
    | mseepgood wrote:
    | Does it make sense to divide people into Elon and not Elon?
 
      | LiquidSky wrote:
      | To Elon, yes.
 
      | krapp wrote:
      | It certainly does to Elon.
 
      | furyofantares wrote:
      | It's not that uncommon to filter analytics based on how
      | much money a user has spent on the platform.
 
      | brink wrote:
      | You mean owner and not owner? I think it's fair.
 
        | dyslexit wrote:
        | For what purpose? Do we know what this is used for?
 
        | iliane5 wrote:
        | Apparently it's for analytics
 
    | fnimick wrote:
    | It's for content analytics, and I assume it's to make sure
    | that changes to the platform can't be argued to bias one
    | party over another.
 
      | kevviiinn wrote:
      | You mean like this algorithmic bias?
      | 
      | https://blog.twitter.com/en_us/topics/company/2021/rml-
      | polit...
 
      | duxup wrote:
      | Or even maybe to provide some background to various bits of
      | lingo / acronyms people use.
      | 
      | I'm thinking along the lines of common word's that have
      | vastly different meanings depending on who's saying it.
 
      | jerlam wrote:
      | So false equivalence is written into the platform. Insane
      | opinions of one party must be displayed as often as
      | moderate opinions of the other. It definitely works for
      | angering everyone on Twitter, not so much for actual dialog
      | or progress.
      | 
      | https://en.wikipedia.org/wiki/False_equivalence
 
      | jacquesm wrote:
      | Or one individual over the rest of the world?
 
    | Calzifer wrote:
    | Well, sounds like this pull request doesn't get merged.
    | https://github.com/twitter/the-algorithm/pull/234
 
  | jaywalk wrote:
  | \*       \* These author ID lists are used purely for metrics
  | collection. We track how often we are       \* serving Tweets
  | from these authors and how often their tweets are being
  | impressed by users.       \* This helps us validate in our A/B
  | experimentation platform that we do not ship changes       \*
  | that negatively impacts one group over others.       \*
  | 
  | From: https://github.com/twitter/the-
  | algorithm/blob/7f90d0ca342b92...
 
    | gregw134 wrote:
    | So now engineers working on the algo can ensure their
    | launches won't lower Elon's tweet visibility. Looks like
    | those remaining at Twitter have a knack for corporate
    | survival.
 
      | capableweb wrote:
      | Yeah, surely the fan-boys who remain at Twitter are
      | interested in _lowering_ the visibility of Elon, not the
      | opposite.
 
        | SauciestGNU wrote:
        | I read that as being able to make sure they don't lower
        | his engagement with a release
 
        | jacquesm wrote:
        | Makes you wonder how they would deal with a reduction in
        | Twitter users.
 
        | japhyr wrote:
        | I think they meant that Twitter developers can make sure
        | their most recent changes won't get them fired by
        | lowering his visibility.
 
      | narrator wrote:
      | He is the full owner of Twitter. It's his company, so
      | nobody is going to fire him as CEO for over-promoting his
      | tweets.
 
        | quadcore wrote:
        | The users can and often do fire the CEO.
 
      | 908B64B197 wrote:
      | > Looks like those remaining at Twitter have a knack for
      | corporate survival.
      | 
      | If Green Cards quotas suddenly became available how many
      | would stay?
 
      | cvhashim04 wrote:
      | When your visa is on the line, you'll do anything
 
        | sabellito wrote:
        | Engineers who work at twitter can easily find another job
        | in the US.
 
        | FalconSensei wrote:
        | even considering all the layoffs other companies are
        | doing?
 
        | sabellito wrote:
        | What do you think is the percentage of companies who did
        | layoffs or have hiring freezes vs companies who are
        | hiring high caliber engineers?
 
        | [deleted]
 
        | JohnFen wrote:
        | Can they? I don't know, but I imagine that at this point,
        | everyone still working for Twitter is there because they
        | don't have any other realistic option.
 
        | DiggyJohnson wrote:
        | That's assuming too much in my opinion. Not everyone has
        | the same or similar opinions regarding their work or
        | Elon, even if it's hard for you to believe.
 
        | JohnFen wrote:
        | I was basing my speculation purely on the working
        | conditions there, not on any supposition about people's
        | attitudes about their work or Musk.
        | 
        | > even if it's hard for you to believe.
        | 
        | It's not hard for me to believe at all.
 
        | kortilla wrote:
        | Only because you are blinded by your own biases. There
        | are people there that think it will be the next spacex or
        | Tesla
 
        | [deleted]
 
        | darth_avocado wrote:
        | Most people working there are looking for other jobs. It
        | is not hard to find another job, but in this market it is
        | hard to find one that pays the same. A lot of people have
        | 4 year stock grants and annual refreshers, pushing their
        | compensation very high which can only be matched at other
        | tech companies, which currently are not hiring much.
 
        | shortrounddev wrote:
        | H1-Bs are harder to get approved these days and a lot of
        | companies don't want to go through the effort/cost to do
        | so
 
        | sabellito wrote:
        | According to this one link [0], H1B "tranfers" are not
        | subject to visa caps. Plus you can search for new jobs
        | and start the process while still employed.
        | 
        | 0: https://usvisagroup.com/changing-jobs-h-1b-visa
 
        | aetimmes wrote:
        | With forced RTO and "hardcore" mandates, it's difficult
        | to find 6+ hours of time to interview at other companies
        | (assuming there are any open visa sponsorships
        | available).
 
        | PuppyTailWags wrote:
        | It's harder with visas, because you're not gambling if
        | you can find another job, you're gambling if your
        | employer can acquire a visa in time for you to not be
        | deported.
 
        | sabellito wrote:
        | Posted on another reply, but according to this one link
        | [0] you can start the visa process at a new company while
        | still employed at your current job.
        | 
        | 0: https://usvisagroup.com/changing-jobs-h-1b-visa/
 
        | PuppyTailWags wrote:
        | Right, so it's _still_ justified to feel bad for visa
        | workers in this case, because they are stuck at an
        | abusive workplace because of circumstances of their visa.
 
    | afavour wrote:
    | Still smells to high heaven to me. Not the Elon part, I don't
    | really care about that. But collecting metrics about
    | "republican" vs "democrat" sounds like a particularly bad set
    | of priorities at work.
 
      | andsoitis wrote:
      | > sounds pretty suspicious.
      | 
      | Sounds toxic to me
 
      | bilekas wrote:
      | This exactly.. But without the models or policies we can
      | only infer, which give plausible deniability.
      | 
      | Can't say I'm shocked overall, but it's strange to see it
      | so 'on the nose'
 
      | JohnFen wrote:
      | About 40% of US voters are not registered with any
      | political party, so at least they will avoid whatever this
      | triggers.
 
        | jacquesm wrote:
        | And then there is the rest of the world.
 
        | kuhewa wrote:
        | So you imagine these tags are set by looking account
        | names up on state voter registration lists?
 
        | JohnFen wrote:
        | Well, if not, then the tags have such a large error bar
        | as to be meaningless.
 
        | emmo wrote:
        | Or the data is assumed good and used dangerously.
 
      | jonathankoren wrote:
      | Anyone that has worked in social network recommendations (
      | _raises hand_ ) knows that they'll be accused of being
      | politically biased, particularly if the recommendations
      | aren't explicitly promoting biased news sites on the Right.
      | (e.g. The Associated Press[0] is leftist propaganda!
      | Where's unbiased news like Gateway Pundit[1] or
      | InfoWars[5]?!) So data scientists and engineers will get
      | pulled in to investigate the latest ref working[2][4], and
      | this will let them easily determine that no, there is no
      | bias.
      | 
      | None of this will matter though, because the complaints are
      | made in bad faith.[2][3]
      | 
      | You may say this is biased comment, but I'm not going to
      | engage in false equivalences, when the outrage and
      | _results_ of the outrage aren't symmetrical. Cite one story
      | where a major social network (Twitter, Facebook, Google
      | News, YouTube, etc) publicly came out and said that they
      | were adjusting their algorithms to make it _more_ lefty.
      | I'll wait. This bad faith of the complaints are
      | particularly obvious when the most popular and influential
      | right wing television channel, Fox News, has been caught
      | red handed knowingly spreading conspiracy theories for
      | ratings.[6]
      | 
      | [0] "Associated Press is the least biased according to both
      | Democrats and Republicans."
      | https://www.businessinsider.com/most-biased-news-outlets-
      | in-...
      | 
      | [1] "The Gateway Pundit (TGP) is an American far-right fake
      | news website. The website is known for publishing
      | falsehoods, hoaxes, and conspiracy theories."
      | https://en.wikipedia.org/wiki/The_Gateway_Pundit
      | 
      | [2] https://www.latimes.com/archives/la-
      | xpm-1996-07-22-mn-26779-...
      | 
      | [3] "Internal report finds 'virtually identical' rates of
      | conservative and liberal topics, but guidelines updated to
      | 'exclude possibility of improper actions'" https://www.theg
      | uardian.com/technology/2016/may/24/facebook-...
      | 
      | [4] "There is some strategy to it [bashing the 'liberal'
      | media]. If you watch any great coach, what they try to do
      | is 'work the refs.' Maybe the ref will cut you a little
      | slack on the next one." -- Rich Bond, 1992 Republican Party
      | Chairman https://www.americanprogress.org/article/think-
      | again-working...
      | 
      | [5] "InfoWars is an American far-right conspiracy theory
      | and fake news website owned by Alex Jones.
      | https://en.wikipedia.org/wiki/InfoWars
      | 
      | [6] https://nymag.com/intelligencer/2023/03/all-the-texts-
      | fox-ne...
 
        | kevviiinn wrote:
        | Well, there was this:
        | 
        | https://blog.twitter.com/en_us/topics/company/2021/rml-
        | polit...
 
        | jonathankoren wrote:
        | That doesn't say they did anything about it. Just that
        | right wing complaints had no substance.
        | 
        | In fact, in just over a year from the publication of that
        | blog post, Twitter -- as a matter of official company
        | policy -- would be promoting the unfounded belief that
        | Twitter engineers and scientists were actively engaging
        | in a propaganda campaign against conservatives.
        | 
        | The exact opposite reaction of what I'm looking for.
        | 
        | https://www.npr.org/2022/12/14/1142666067/elon-musk-is-
        | using...
 
        | kevviiinn wrote:
        | I think it says a lot about the complaints. Some people
        | just can't stop crying wolf to get attention
 
      | seydor wrote:
      | but the people who care about stats are usually american
      | politicians. they can present them with this data. (and use
      | Elon as a control LOL)
 
      | jcalder wrote:
      | I suspect it plays into this stuff: https://www.theguardian
      | .com/technology/2021/oct/22/twitter-a...
      | 
      | They wanted to answer the questions of "is twitter biased
      | against Republicans" so they measured it, turns out they
      | favored republicans.
 
    | tech234a wrote:
    | That makes sense; I guess that means Elon is considered a
    | "group" now.
 
      | lhnz wrote:
      | I know there's a joke about this regarding his ego and
      | there's certainly some truth in that, however it's also
      | quite believable that after a deployment he might have
      | noticed the popularity of his tweets going down (since he
      | no doubt checks his reach), so I can kind of understand how
      | he might see "republicans", "democrats" and "celebrities_it
      | _makes_sense_to_check_this_with_my_account_as_i_am_a_very_a
      | ctive_user" as core categories that need to have their
      | reach balanced.
 
        | sangnoir wrote:
        | > it's also quite believable that after a deployment he
        | might have noticed the popularity of his tweets going
        | down
        | 
        |  _He did_ notice it and it was treated as a 5 alarm fire,
        | with a Musk cousin sending 2 am slack messages (on a
        | Monday!) to Twitter engineers to urgently fix Elon 's
        | reach[1].
        | 
        | 1. https://www.theverge.com/2023/2/14/23600358/elon-musk-
        | tweets...
 
        | bcrosby95 wrote:
        | The proper way to do that is create a pool of celebrities
        | and monitor them. Not just the CEO's account.
        | 
        | For very active accounts, I assume that's what the "vits"
        | or "power user" one is for. Or, heck, "vits" might
        | actually be what you said.
 
        | leye0 wrote:
        | ChatGPT suggests: Vitriolic accounts.
 
        | NERD_ALERT wrote:
        | We don't need to speculate on this. It sounds like he did
        | actually fire engineers over his tweets getting less
        | engagement than he wanted.
        | 
        | https://www.platformer.news/p/elon-musk-fires-a-top-
        | twitter-...
 
        | lhnz wrote:
        | Honestly, if you read behind the lines, it sounds like
        | the employee was intentionally making a joke about it at
        | his expense in front of a bunch of people, and I think a
        | lot of CEOs would take that badly as this is effectively
        | the same thing as calling your boss egocentric.
        | 
        | But, we do have a bit of code that measures metrics on
        | his account, so can we find the bit of code that
        | increases the engagement on his account?
 
        | mochomocha wrote:
        | > But, we do have a bit of code that measures metrics on
        | his account, so can we find the bit of code that
        | increases the engagement on his account?
        | 
        | There doesn't need to be. When they run AB tests, it's
        | possible that they'd pick the winning cell if it makes
        | the Elon metrics look better.
        | 
        | Even if the algorithm doesn't do anything explicit about
        | boosting him, it can be tweaked through AB testing to
        | favor him.
 
        | lhnz wrote:
        | You mean A/B testing of weights/biases?
 
        | mochomocha wrote:
        | No, the "weights" of your model are trained from the
        | input data. What is usually AB tested are hyperparameters
        | of the model, or different "flavors" of (model+input
        | data).
 
        | lhnz wrote:
        | What people are implying is still unsubstantiated though.
        | The engineers on the Twitter Space say that this is to
        | ensure that changes they make do not bias one category
        | over another, they don't say that it's in order that they
        | can make discretionary updates to bias towards Elon Musk.
        | 
        | Maybe after every update to the model, they check these
        | stats to ensure that they haven't biased towards Elon
        | Musk, and if so roll the change back.
 
        | mochomocha wrote:
        | ? Considering Elon as its own category is a bias.
 
        | bcrosby95 wrote:
        | > "When you're asked a question, you run it through your
        | head and say 'what is the least fireable response I can
        | have to this right now?'" one employee explained.
        | 
        | Reading between the lines, Musk sounds like a giant baby.
 
  | [deleted]
 
  | overthrow wrote:
  | I'd love to see the exact date author_is_elon was added. Too
  | bad they didn't publish the commit history
 
    | qbasic_forever wrote:
    | IIRC it was very recent, there was a Twitter engineer that
    | was fired after explaining to Elon that the algorithm was not
    | biased against him:
    | https://www.salon.com/2023/02/10/petulant-elon-musk-fired-
    | tw... Almost certainly after that event Elon had them
    | explicitly bump his tweets in their reach.
 
      | nonethewiser wrote:
      | This speculation seems mostly informed by a negative
      | opinion of Elon. Is there any real indication he gave this
      | instruction?
 
        | qbasic_forever wrote:
        | Yes that weekend everyone started seeing Elon's tweets in
        | their timelines:
        | https://www.theverge.com/2023/2/14/23600358/elon-musk-
        | tweets...
        | 
        | You seem weirdly protective of Elon fyi
 
    | nijave wrote:
    | I think around mid February
    | https://www.theverge.com/2023/2/14/23600358/elon-musk-
    | tweets...
 
  | darth_avocado wrote:
  | I would not be surprised if "author_is_elon" was added after he
  | bought the company and worked the engineers too hard to figure
  | out why his tweets don't have a lot of engagement.
 
    | burkaman wrote:
    | https://www.platformer.news/p/yes-elon-musk-created-a-
    | specia...
 
  | sekai wrote:
  | Haha, that's pretty funny, of course that's a thing
 
| paulddraper wrote:
| > 1.4k forks
| 
| Wow, we're getting some collaboration going!
 
| AlbertCory wrote:
| I haven't read the "algorithm" and this observation might be
| seriously out of date, but:
| 
| for Google Ads, you couldn't easily know what ads would be shown
| for a given query, without a whole lot of data that's not
| contained in any code: the experiment settings in the server, for
| one thing. And the user who's doing the query, for another.
| 
| An "experiment" could apply to 100% of the traffic, so it's not
| really an experiment anymore. And even if you _think_ X has been
| put into production, there is still a  "holdback" experiment,
| where some part of the traffic does not get X applied to it.
 
| m1117 wrote:
| As I understand, they open sourced only the abstraction, but
| still have a way to control anything.
 
| froggychairs wrote:
| Why is nobody pointing out that this is likely an April Fools
| joke? We just deployed our April Fools joke into production today
| too.
 
  | endorphine wrote:
  | Yeah this confused me a lot while reading the comments here. I
  | wonder what percentage of the comments are trolling vs. fell
  | for it vs. think it's legit.
  | 
  | Perhaps this calls for an HN poll...
 
    | froggychairs wrote:
    | Yeah....
    | 
    | I should add, I dont think all of it is a joke, but stuff
    | like the "author_is" labels are incomplete and only 4 were
    | shown for the bit
 
  | nabakin wrote:
  | I fell for it too until a friend pointed it out. I wonder why
  | it's working so well
  | 
  | Edit: hi friend
 
    | froggychairs wrote:
    | Lmao
 
| pictur wrote:
| It's a really scary codebase. Do you really need that much code
| for the world's crappiest recommendation algorithm? I think you
| can do more crap with less code. we trust you elon.
 
| sho_hn wrote:
| My main questions: Will these repositories be used in production
| by Twitter? Is this now the mainline, not a semi-regularly-synced
| mirror?
 
  | cubefox wrote:
  | Musk said that releasing the algorithm will initially be
  | embarrassing, but that they will quickly update it. So it seems
  | that means they intend to at least regularly publish newer
  | versions.
  | 
  | Of course it could also be that they change their mind when
  | spammers abuse the openness.
 
  | agluszak wrote:
  | Of course not
 
| koolba wrote:
| It's reassuring to know that billion dollar tech companies write
| CI exactly like I do:
| 
| https://github.com/twitter/the-algorithm/blob/main/ci/ci.sh
| 
| Permalink: https://github.com/twitter/the-
| algorithm/blob/7f90d0ca342b92...
 
  | sekai wrote:
  | Same mortals as us
 
    | jmull wrote:
    | Considering it statistically, likely on a lower plane.
 
  | calvinmorrison wrote:
  | Maybe it's weird, but for all the work I have ever done, I have
  | never used CI/CD in the way that it was meant to be used, or
  | never really leveraged it. Maybe all of my past jobs were
  | unprofessional, but like, I see a lot of jobs using "CI/CD
  | experience required" and I think... huh I wonder if they
  | actually do it
 
    | scruple wrote:
    | I wouldn't say it's necessarily weird, and I'd never call it
    | unprofessional, but I have also been using CI of some shape
    | or form since I entered the industry in the '00s. From home
    | brewed scripts that were cobbled together internally, to CI
    | servers on-prem, to CI servers in the cloud, and now back to
    | on-prem. At work, I am literally in the middle of a _massive_
    | migration of my teams _multiple_ CI servers. We have dozens,
    | sometimes hundreds of jobs kicked off on a daily basis + at
    | least 2 dozen nightlies. Without CI, our team would be dead
    | in the water.
 
  | gspencley wrote:
  | It has been my personal experience, over 25 years in the
  | industry, that often times the bigger the company the worse the
  | code.
  | 
  | It's not an absolute rule, I've certainly inherited projects in
  | a consulting capacity that were written by small teams and were
  | atrocious. But more often than not, a small team working for a
  | small company has fewer of the internal "forces" that incur
  | "technical debt."
  | 
  | Those forces are things like
  | 
  | - Silo'd teams working on a common code base in parallel but
  | never talking to each other, thus duplicating code and having
  | wildly different conventions
  | 
  | - Layers of middle management each with different management
  | styles, leading to inconsistency and product-wide short-cuts
  | 
  | - Dealing with sudden success-induced scalability disasters
  | that result in bandaid solutions
  | 
  | - More employee churn which means that the way we did things
  | yesterday is not the way we're doing things today because
  | someone new is in charge ... more inconsistency in code and
  | software decisions
  | 
  | - More "old code." Companies very rarely do rewrites and when
  | they do they're often failures. So the bigger the company, the
  | more "legacy" spaghetti code typically because you don't fix
  | what isn't broken (especially when the entire system is broken
  | because it's one big giant mess that no one understands and yet
  | somehow it actually works ... as long as we don't breathe on it
  | or get a sudden surge of new account sign-ups).
 
  | dblitt wrote:
  | It wouldn't surprise me if they had a script referencing
  | internal build infrastructure that got gutted in the open
  | source release
 
  | agilob wrote:
  | It's called Volkswagen CI
 
| cmckn wrote:
| Including the search engine itself in "the algorithm" repo is an
| interesting choice. Obviously it's a major player in what gets
| returned to clients, but the details of that infrastructure
| aren't really relevant and is a notable portion of their secret
| sauce.
| 
| https://github.com/twitter/the-algorithm/tree/main/src/java/...
 
| jeffbee wrote:
| Why does anyone use "for you"?
 
  | RoyGBivCap wrote:
  | [dead]
 
  | teach wrote:
  | Probably the same reason some people browse /r/all on Reddit. I
  | think the desire for that sort of thing has waned a lot over
  | the past couple of years, though.
 
    | 12345hn6789 wrote:
    | Not quite the same. All does 0 user customized ordering. It
    | is based on some "algorithm" but it's the same for all users.
 
| throwayyy479087 wrote:
| You gotta hand it to Elon - he actually did it.
 
  | lern_too_spel wrote:
  | Where is the file containing accounts that are artificially
  | boosted? We can guess what its single line is, but how is it
  | incorporated into the algorithm?
 
  | minimaxir wrote:
  | If you look at the GitHub repo, most of it is READMEs
  | describing systems, not the models or code subleties which
  | actually give explanations into how certain weird behaviors on
  | Twitter happen. (e.g. the preference of certain users in the
  | For You tab. EDIT: bad example, since there appears to be a
  | flag for that in the code, although it does not specify _which_
  | users are on the list)
 
    | mquander wrote:
    | The links in the README just go to other documents, but the
    | repo seems to have most of the code for the components the
    | documents are describing.
 
      | minimaxir wrote:
      | It seems to vary by service; some are more detailed than
      | others.
 
  | nebula8804 wrote:
  | Yeah but whats the grift this time? There is always an angle
  | with this guy.
 
    | bboygravity wrote:
    | Or an alternative view: no matter what he does there will
    | always be haters and there are massive (political) incentives
    | to destroy his public persona.
    | 
    | Those incentives also align nicely with those of hedge funds
    | that are short TSLA as well as family offices that are short
    | Tesla (hello Billy boy Gates).
 
      | JustSomeNobody wrote:
      | > ... no matter what he does there will always be haters.
      | 
      | Yes. Why? Because he's an jerk that deserves it. He's done
      | and said enough dumb shit that people should take him with
      | an ocean's worth of salt.
 
    | rurp wrote:
    | Gets him a positive news cycle without much cost. It's hard
    | to immediately say how interesting or useful this will be
    | since there are probably a number of related systems and
    | databases that aren't released.
 
    | illiarian wrote:
    | The angle, on the same day, or close to same day:
    | 
    | - API free tier gutted. The "hobby/student" tier is at $100 a
    | month, next tier is "enterprise" https://twitter.com/TwitterD
    | ev/status/1641222782594990080?s=...
    | 
    | - For You page will only contain tweets from paid accounts:
    | https://twitter.com/elonmusk/status/1640502698549075972
 
  | nemothekid wrote:
  | https://twitter.com/dril/status/831805955402776576
 
  | jonahbenton wrote:
  | LOL. My algorithm at twitter had been very simple-
  | 
  | See tweets from people I followed.
  | 
  | Don't see tweets from people I didn't follow.
  | 
  | Trust people I follow in their retweets to signal something
  | interesting.
  | 
  | Unfollow unhelpful people.
  | 
  | Once that algorithm was rendered impossible, I left twitter.
  | 
  | Haven't missed it.
  | 
  | Having someone say- here's the way we are going to promote
  | something to you- doesn't make me inclined to accept the
  | promotion!
 
    | mgiannopoulos wrote:
    | This still exists as the Following tab and viewing it is a
    | persistent option. You don't need to see the algorithm feed
    | ("For You") ever.
 
      | cauthon wrote:
      | Roughly one in every four to five tweets in the "following"
      | feed is a "promoted" tweet, at least on mobile.
      | 
      | 20-25% noise isn't a great ratio for something that I
      | ostensibly curate.
 
        | hk__2 wrote:
        | "promoted" tweets are ads, just like you would see on
        | almost any website you don't pay for.
 
  | raydev wrote:
  | Do we "gotta hand it to Elon" for not missing one of his 40-50
  | self-imposed deadlines and feature announcements?
 
  | addisonl wrote:
  | Did he? Considering the vast majority of the algorithm is waved
  | away as "ML model".
 
    | thieving_magpie wrote:
    | I can't pretend to know if this contains the actual ML model
    | code but there is a second repo the-algorithm-ml:
    | https://github.com/twitter/the-algorithm-ml
 
| rogerallen wrote:
| "Today, the For You timeline consists of 50% In-Network Tweets
| and 50% Out-of-Network Tweets on average, though this may vary
| from user to user."
| 
| I have spent significant effort creating a network and there you
| go choosing to ignore my efforts by putting in 50% of crap-I-
| don't-want-to-see.
| 
| That is why I despise your algorithm.
 
  | bluetidepro wrote:
  | > "Today, the For You timeline consists of 50% In-Network
  | Tweets and 50% Out-of-Network Tweets on average, though this
  | may vary from user to user." I have spent significant effort
  | creating a network and there you go choosing to ignore my
  | efforts by putting in 50% of crap-I-don't-want-to-see. That is
  | why I despise your algorithm.
  | 
  | This is just one feed (the "For You" recommendations feed),
  | they also have the "following" feed tab next to it that is 100%
  | your network (want you want), and it remembers your selection
  | when you change between them (they fixed that a few months
  | ago), so really this is kind of a pointless thing to despise
  | for that reason. It's just an option you can 100% avoid if you
  | don't want to see it.
  | 
  | In fact, Twitter is probably one of the only few left in the
  | large social media space that actually gives you an 100%
  | following network feed (minus maybe ads) in chronological order
  | that REMEMBERS your selection (Facebook, Instagram, and TikTok
  | don't). Which makes this even more silly to say. Facebook,
  | Instagram, and TikTok do all have in-network exclusive
  | chronological order feeds, BUT they are extremely hard to find,
  | or don't remember your selection to them.
  | 
  | Hate of Twitter is easy to spoon out, but at least complain
  | about things that aren't already solved for you.
 
    | vagabund wrote:
    | I'm not on twitter enough for the chronological feed to be
    | appealing to me, and instead want to see the notable tweets
    | from the accounts I follow since the time I last visited.
    | There's no straightforward way to achieve this, but if anyone
    | else has this preference, the workaround is to create a
    | twitter list with all the accounts you follow and set it to
    | show top tweets first.
 
    | rogerallen wrote:
    | I said I despised the algorithm, I did not say I hated
    | Twitter. Now I at least know why I hate it.
    | 
    | Yes "Following" is what I use. The reason I use it is because
    | of this algorithm that thinks I could possibly want 50%
    | tweets that make me "engaged^H^H^H^H^Hraged". To me, that is
    | a ridiculous mixture.
    | 
    | I'm happy they have a "Following" and I sure hope they keep
    | it, but I will not be surprised if it goes away.
 
    | Sebguer wrote:
    | If you try to use the Following tab on Android, every refresh
    | brings you back to the For You tab.
 
      | bluetidepro wrote:
      | Is your app up to date? I have it on my iPhone, iPad, and
      | an Android device which all have no problem always
      | remembering the "Following" tab selection. As well as
      | desktop/web.
 
        | matsemann wrote:
        | Pressing the home button on Android often switches the
        | tab over to "for you"
 
      | rvz wrote:
      | Care to comment on this? [0] Surely Twitter hasn't open
      | sourced and released the recommendation algorithm as you
      | predicted right?
      | 
      | Or perhaps when a prediction didn't go according to plan,
      | let us complain about another thing...
      | 
      | [0] https://news.ycombinator.com/item?id=35213705
 
  | JustSomeNobody wrote:
  | "Control Panel for Twitter" plugin. You can get rid of "For
  | You".
 
  | corbulo wrote:
  | I'm confused, then why not just use your 'followed' feed
  | instead of 'for you'?
 
    | glenstein wrote:
    | I'm also confused. You can still see everything you've
    | manually curated.
 
    | bakugo wrote:
    | On the android app at least, a recent update made it so
    | pressing the home button while you're in the "followed" tab
    | switches to the "for you" tab. It's extremely annoying
 
| whalesalad wrote:
| Two space indent in .py? Provocative.
 
| jml2 wrote:
| ( "has_toxicity_score_above_threshold",
| _.getOrElse(EarlybirdFeature,
| None).exists(_.toxicityScore.exists(_ > 0.91)) )
 
  | jml2 wrote:
  | `if (sourceUserId.isDefined || sourceUserId.isDefined)
  | Some(true)`
  | 
  | https://github.com/twitter/the-algorithm/blob/main/timeliner...
 
| etc_passwd wrote:
| Democrats / Republicans looks like it was added outside of SDLC
| [1]. This order without those features is sorted, likely by a
| linter, suggesting Elon and Vits are properly implemented, and
| Democrats/Republicans was just inserted alongside the Elon
| feature, perhaps just for this extract. Sorting it now results in
| a different order than the commit.
| 
| [1]: https://github.com/twitter/the-
| algorithm/blob/7f90d0ca342b92...
 
  | lenzm wrote:
  | Or Elon was the addition, the other 3 are in alpha order.
 
| RoyGBivCap wrote:
| [dead]
 
| WA wrote:
| Will this make it easier to game the algo or does it depend so
| heavily on individual user interaction that it's close to
| impossible to game it? For example, by carefully crafting Tweets
| or by buying likes/retweets etc?
 
| paxys wrote:
| While open sourcing code is always great, and kudos on them for
| doing so, let's be real most people didn't care about the
| internal plumbing of how their recommendation system runs. It's
| going to be a mess of decades old code, microservices and ML
| pipelines just like one would expect. If you want to dig deeper
| to check for biases (the reason they claimed to be open sourcing
| it in the first place), you will however run into:
| 
| > We also took additional steps to ensure that user safety and
| privacy would be protected, including our decision not to release
| training data or model weights associated with the Twitter
| algorithm at this point.
| 
| which is a shame.
 
| [deleted]
 
| dang wrote:
| Url changed from https://github.com/twitter/the-algorithm-ml,
| which points to this.
 
| mkl95 wrote:
| I couldn't care less about Twitter's high level abstractions.
| They were never renowned for those. Their database schemas and
| infrastructure on the other hand...
 
| beebmam wrote:
| I don't use Twitter, but this is awesome. I hope this will help
| more people realize how complex it is to build and operate web
| services.
 
| distrill wrote:
| the-algorithm is such a pretentious name for a repo
 
  | BbzzbB wrote:
  | It's a colloquial term for recommendation engines, how often do
  | you hear people say "the algorithm" (vs. "the recommendation
  | engine") on YouTube?
 
    | distrill wrote:
    | yes, but this is the first repository i have seen named like
    | this
 
      | krapp wrote:
      | It's because there's been nearly a decade of conspiracy
      | theory around the use of algorithmic feeds in social media
      | generally, and Twitter specifically. Among the right,
      | "algorithms" have become symbolic of the machinery of
      | leftist oppression they believe to be arrayed against them
      | by modern media.
      | 
      | So this language is Elon signaling that he's presenting the
      | "woke hivemind's" head on a platter.
 
  | sho_hn wrote:
  | Eh, it's name-spaced.
 
| phailhaus wrote:
| Great! But nothing is going to change until people realize that
| the problem is the _feedback loop_. It 's not the recommendation
| engine itself, it's the fact that there's no way "out" of the
| feed that the engine produces. It recommends you stuff, you have
| little choice but to engage with it, and then _it trains on that
| information_.
| 
| This is the problem with most of social media today. It is a very
| well known problem in ML [1], but nobody is willing to do
| anything about it because it's a fundamental UX change. Facebook,
| Twitter, YouTube, TikTok, they have _defined_ themselves by their
| recommendation engines.
| 
| [1] https://towardsdatascience.com/dangerous-feedback-loops-
| in-m...
 
  | hdivider wrote:
  | Thank you! Working on a concept for a big org on what may
  | become a large ML-based system one day. I knew about this
  | feedback loop issue, but was too dumb to actually remember and
  | face this problem. :) It's all over today's rec engines -- and
  | yet, just like the things we're not shown in these systems, the
  | problem itself seems to become invisible. Because it requires
  | new thinking.
  | 
  | Worth exploring.
 
  | LZ_Khan wrote:
  | Don't you have a choice.. to not engage with it? If you didn't
  | like it then assuming the metrics system is working correctly,
  | this would be negative feedback to the ML model, causing said
  | content to not be shown in the future.
 
    | phailhaus wrote:
    | Couple problems:
    | 
    | 1. Actively supplying negative feedback is sometimes hidden
    | behind secondary menus, making it much higher friction
    | compared to just...scrolling past. So most users don't spend
    | the effort. Even with a dislike button, it's unclear what the
    | system is learning. It can't know that I don't like this
    | particular video because it's a conspiracy theory, and to
    | stop showing me those. These platforms often don't even
    | support explicit categories, so how would they know?
    | 
    | 2. It's _extremely_ high friction to teach the algorithm you
    | 're interested in something that it doesn't suggest to you!
    | There's the whole unknown unknowns problem: how do you teach
    | the algorithm you're interested in something that you've
    | never seen before?
    | 
    | I still think Reddit has handled this the best. No system is
    | perfect, but Reddit's challenges are much more manageable
    | than the quagmire that TikTok, Facebook, and YouTube have
    | gotten themselves into. I can just unsubscribe from
    | r/conspiracy, and I'm out. Basically impossible to teach that
    | to YouTube without weeks of careful curation. They think
    | they're smart enough to know what I like, but they're not and
    | never will be.
 
      | zrezzed wrote:
      | Twitter _could_ recreate a similar system: 1) auto-tag
      | tweets with labels, rather than users needing to submit to
      | subreddits 2) auto-sub most people some default set of
      | labels 3) let them un-sub if they want.
      | 
      | They just don't want to.
 
  | khy wrote:
  | I think Instagram in particularly is bad in this regard. It
  | seemingly becomes convinced that I care deeply about the
  | subject of any post that I even momentarily linger on.
 
  | dmonitor wrote:
  | reminds me of a story about a guy who was given a gift, a
  | decorative plate with a rooster on it i think it was. didn't
  | care for it too much, but out of politeness put it on display
  | on an empty cabinet he had. a while later someone noticed he
  | had it and figured he liked it, so got him a similar decorative
  | plate with a rooster on it. again, out of politeness, he put it
  | next to the old one. now other people started to think he just
  | really liked roosters, and started giving him little rooster
  | statues and nicknacks. Eventually he just has a whole display
  | cabinet of rooster themed gifts that he never really cared for
  | to begin with, but people just assume he likes them because
  | people keep giving them to him.
 
    | Washuu wrote:
    | I have a whole shelf full of horse stuff because of this!~
 
| roddylindsay wrote:
| For ranking the candidates these predictions are combined into a
| score by        weighting them:
| "recap.engagement.is_favorited": 0.5        "recap.engagement.is_
| good_clicked_convo_desc_favorited_or_replied": 11* (the
| maximum prediction from these two "good click" features is used
| and weighted by        11, the other prediction is ignored).
| "recap.engagement.is_good_clicked_convo_desc_v2": 11*
| "recap.engagement.is_negative_feedback_v2": -74
| "recap.engagement.is_profile_clicked_and_profile_engaged": 12
| "recap.engagement.is_replied": 27
| "recap.engagement.is_replied_reply_engaged_by_author": 75
| "recap.engagement.is_report_tweet_clicked": -369
| "recap.engagement.is_retweeted": 1
| "recap.engagement.is_video_playback_50": 0.005
| 
| Who set those weights, and why were they chosen?
 
  | Mehdi2277 wrote:
  | Having worked at similar companies on similar systems usually
  | A/B experiments and smaller probability of an action bigger
  | weight it must have to matter much overall. The constants are
  | generally done through some ab tests to get them into
  | reasonable overall behavior but they are a pain to tune and
  | very unlikely optimal in any real sense as it's often too
  | difficult to do extensive search of them. Like often I'll see
  | new target have a couple different weights tried on an ab and
  | then maybe second set of experiments after rough magnitude is
  | determined.
 
  | bobbygoodlatte wrote:
  | "recap.engagement.is_replied": 27
  | "recap.engagement.is_replied_reply_engaged_by_author": 75
  | 
  | I wonder if this is why threads rank so obnoxiously high. They
  | get artificially boosted by the author replying to their own
  | tweet
 
    | localplume wrote:
    | isn't that the author replying to a reply on their tweet? so
    | its promoting positive discussion, hence pushing the
    | engagement higher?
 
| jonknee wrote:
| projects/home/recap/FEATURES.md has some interesting stuff:
| 
| https://github.com/twitter/the-algorithm-ml/blob/main/projec...
| 
| In realgraph you can see some of the things they keep track of,
| which include what you have in your address book, total time
| spent "dwelling" and a few other interesting nuggets.
 
| kossTKR wrote:
| I've pretty much ignored all of the superficial political theatre
| but noticed the actual algo worsening over the last 6 months.
| 
| I get way to much random crap now, promoted tweets, "thing that
| might interest me", users that seem to never get on my feed etc.
| 
| Twitter seems to go in the direction of all other social media,
| feeds that are 100% digital crack with no way to control your
| media diet.
 
| jonathanmayer wrote:
| Context: I teach at Princeton and study social media and
| recommendation systems.
| 
| From a very quick skim of the repositories, this appears to be
| quite limited transparency. The documentation gives a decent
| high-level overview of how Tweet recommendation works--no
| surprises--and the code tracks that roadmap. Those are meaningful
| positive steps. But the underlying policies and models are almost
| entirely missing (there are a couple valuable components in [1]).
| Without those, we can't evaluate the behavior and possible
| effects of "the algorithm."
| 
| [1] https://github.com/twitter/the-algorithm-ml
 
  | modeless wrote:
  | What about these? https://huggingface.co/Twitter
 
    | simonw wrote:
    | Those look older to me. They all have last updated dates for
    | October and November 2022.
 
  | EastSmith wrote:
  | FB open source algo looks much better, right? /s
 
  | ngrilly wrote:
  | What did you expect?
 
    | TaylorAlexander wrote:
    | I don't know if the parent's expectations matter here. This
    | is more about making sure others don't misunderstand the
    | meaning here.
 
  | bobobob420 wrote:
  | Can i audit your classs for free?
 
  | meghan_rain wrote:
  | So why did they opensource it?
 
    | justapassenger wrote:
    | You must be new to Musk's business practices.
 
    | bradly wrote:
    | Because they let go many of the engineers working on it?
 
    | daveguy wrote:
    | So they could pretend to be open. It's the "Open"AI model.
    | Open-washing?
 
      | cubefox wrote:
      | This is a very cynical take. They should be commended for
      | publishing recommendation code at all, which no other major
      | social network does.
 
        | misiti3780 wrote:
        | a large portion of HN users view anything elon musk does
        | as nefarious, which is why half of these comments are
        | negative.
 
        | cubefox wrote:
        | Yeah it seems so. I think journalists have portrayed him
        | way too negatively. For example, they mostly ignored the
        | Twitter files, presumably because it didn't fit their
        | political narrative.
 
        | systemvoltage wrote:
        | Cynicism is the standard operating procedure on HN for
        | anything that tilts against their party lines. It's
        | completely illiberal.
 
        | guelo wrote:
        | Any time a billionaire buys a media company it's bad for
        | the health of democracy.
 
        | paulddraper wrote:
        | Which is why HN was so incensed about Bezos buying the
        | Washington Post.
 
        | simondotau wrote:
        | And when a highly scrutinised, highly visible billionaire
        | buys it off a different bunch of billionaires which you
        | know little about?
 
        | misiti3780 wrote:
        | i wasnt referring to him buying twitter, i was referring
        | to him saying he was going to open source the
        | recommendation engine and then doing it.
        | 
        | i agree billionaires owning media companies is huge
        | problem
 
        | mulmen wrote:
        | Do you believe billionaires can do good? Is their
        | existence an existential threat to democracy?
 
        | hooverd wrote:
        | Yes. There are plenty of philanthropic billionaires. Yes.
        | That much money buys a destabilizing amount of influence.
 
        | systemvoltage wrote:
        | Billionaires are billionaires not by literally storing
        | cash. The rest of the society values their contributions
        | and creations in the companies/corporations they run.
        | Sure, they have some liquidity but the entire concept of
        | resentment towards billionaires is essentially equal to
        | resentment for the betterment of the world. There are
        | some exceptions but for the most part, in a well oiled
        | market, you can't just become a billionaire by fucking
        | over people. See Adani and how it turns out for him: http
        | s://www.ft.com/content/5c0b6174-e66d-4fa5-89a5-6da1d69ab.
        | ..
 
        | systemvoltage wrote:
        | We need more inequality. It's a feature, not a bug.
        | Entire swaths of people are convinced we need to flatten
        | all people into one income bracket. That's not going to
        | end well. See USSR and Cambodian massacre.
        | 
        | That doesn't mean we should propel poverty.
        | 
        | Midwits have been brainwashed into thinking "equity =
        | good" with no critical thinking capability. The entire
        | political college class screams "How can you be against
        | equality!" followed by insinuations and fascism insults.
        | The term "human rights" have been hijacked into
        | socialism/communism propaganda.
        | 
        | It's so deeply fucked.
 
        | pastacacioepepe wrote:
        | Can you name one relevant media company owned by someone
        | from the working class?
 
        | Mordisquitos wrote:
        | Not necessarily. What if the media company was bad for
        | the health of democracy, and the billionaire's
        | incompetence destroys the company's social standing and
        | thus its ability to do more damage (even in the
        | billionaire's own interests)?
 
        | nonbirithm wrote:
        | Yeah, have to wonder how many people, if they had the
        | money, would want to buy out Twitter just to wipe it out.
        | Doesn't a huge chunk of HN hate Twitter and wish it were
        | dead?
        | 
        | (Regardless I think that would be useless in the long
        | run, since the millions of stranded users will still want
        | another Twitter-like platform. And Twitter imploding
        | without a designated archive will wipe out a tremendous
        | amount of digital history.)
        | 
        | A lot of his decisions look pretty incompetent in the
        | surface, like how could he not see how charging for
        | verification devalue the system to whoever has the money?
        | 
        | Instead it could just be an intentional ploy to
        | completely devalue Twitter disguised as incompetence. He
        | can justify firing employees and charging for API
        | access/verification as money-saving strategies, even if
        | they're terrible strategies that have little chance of
        | succeeding. And he could make enough people believe he's
        | an idiot who makes things up as he goes rather than
        | someone specifically driven or apathetic enough to run
        | Twitter into the ground. Not to mention he was forced to
        | buy them after changing his mind. Almost feels like a "so
        | that's what happens" response.
        | 
        | I wonder how higher powers would be able to distinguish
        | fake incompetence from real incompetence. Would they care
        | how Twitter as a private company ends up if it's the case
        | that it implodes from its own legitimately bad business
        | decisions? It reminds me of how employers won't directly
        | fire employees for discriminatory reasons, instead they
        | make the employees' lives miserable so they're compelled
        | to leave on their own, thus they escape scrutiny.
 
        | ClumsyPilot wrote:
        | This is basically at the level of "9/11 was an inside job
        | to bring down WTC 1, but WTC2 was destroyed in an
        | unrelated but simultaneous terrorist attack"
 
        | tablespoon wrote:
        | > Yeah, have to wonder how many people, if they had the
        | money, would want to buy out Twitter just to wipe it out.
        | Doesn't a huge chunk of HN hate Twitter and wish it were
        | dead?
        | 
        | > (Regardless I think that would be useless in the long
        | run, since the millions of stranded users will still want
        | another Twitter-like platform.
        | 
        | If there's not an obvious successor, _right when its
        | shutdown_ , a lot of those people might get their habit
        | broken and find something better to do. I know Mastodon
        | was held up as a successor, but it's unclear to me if
        | that's actually capable of scaling to that level.
 
        | dmix wrote:
        | Mastodon is way too flawed to be anything but a niche
        | tool for tech people and activists. I highly highly doubt
        | such a system can cross the chasm. That doesn't mean
        | that's a bad thing though.
 
        | TaylorAlexander wrote:
        | Well if they say "we will open source the algorithm" and
        | then what they really open source is a little bit of
        | slightly relevant code that doesn't allow us to
        | understand the algorithm, then what we can deduce is that
        | they are trying to weasel out of public commitments.
        | 
        | I can't say for sure if that happened, but if they made a
        | clear promise and then did something else, it's perfectly
        | reasonable to call that out.
 
        | [deleted]
 
        | philote wrote:
        | Hey, we can get even more cynical. Why should we trust
        | that this code is even similar to what they run in
        | production currently?
 
        | [deleted]
 
        | 5e92cb50239222b wrote:
        | I still hear reverse-FUD about nvidia supposedly fully
        | open-sourcing their Linux driver, when in reality they
        | opened a tiny kernel portion of it that allows the main
        | proprietary blob to connect to necessary kernel
        | interfaces. You have to call out this bullshit when you
        | see it.
 
        | hanniabu wrote:
        | This is like FB open sourcing the compiled frontend code
        | you can see yourself using inspect.
        | 
        | If we commend them for this we're helping promote and
        | encourage this faux open source virtue signaling
 
        | cubefox wrote:
        | No, that's very different.
 
        | rakoo wrote:
        | Let's have reasonable goals, shall we ? "Their shit
        | doesn't stink as bad as others'" is nothing commendable,
        | especially after souch publicity.
 
      | jstummbillig wrote:
      | If we are willing to not assume some borderline "it's what
      | they want you to think" conspiracy play, obviously there
      | was always going to be a lot of highly interested and
      | qualified people taking a very close look at this and, at
      | some point, there was always going to be very definitive
      | conclusion of what's the deal with what they released.
      | 
      | If your play was "it's some source code, hence people will
      | think we are open, and that should be really good for us",
      | that would make you a very special kind of idiot in this
      | space.
 
      | correlator wrote:
      | There is clearly a lot of information to share. It's worth
      | considering this could be step 1 of n as opposed to
      | assuming the worst possible intention.
 
    | joshspankit wrote:
    | That was one of Elon's core statements when he first talked
    | about buying Twitter. If he had gotten it out sooner there
    | would be an easier link between the two, but if you want more
    | context just go read the old tweets and articles from the
    | Twitter vs Elon days.
 
    | anigbrowl wrote:
    | PR
 
    | avanti wrote:
    | It's no secret that Twitter, like any other social media
    | platform, is driven by user engagement and ad revenues. The
    | more time we spend on the platform, the more valuable it
    | becomes for them. With this new open-source algorithm,
    | they're essentially crowdsourcing improvements to their
    | system to better serve us the content we crave.
    | 
    | this move could be seen as a strategic PR play to boost their
    | public image amidst the growing concerns around algorithmic
    | bias and lack of transparency. By inviting the community to
    | collaborate and address these issues, they're not only
    | shifting some of the responsibility onto the users but also
    | deflecting potential criticism.
 
    | kzrdude wrote:
    | If we can't build anything with this, is it "source"?
 
      | [deleted]
 
      | bilekas wrote:
      | "Does not include batteries"
 
    | w0m wrote:
    | PR and it was already leaked last week.
 
  | alfor wrote:
  | [flagged]
 
  | tpmx wrote:
  | Did you also skim the accompanying (or rather, main) repo,
  | https://github.com/twitter/the-algorithm ?
  | 
  | From a quick clone and line-count, it has:                 235
  | kLOC .scala       136 kLOC .java       22  kLOC .py       7
  | kLOC .rs
  | 
  | So I don't think you did, since you posted so quickly and
  | that's a LOT of code.
  | 
  | I also haven't skimmed this code except very superficially, but
  | perhaps you should since you're out there making statements
  | with your Princeton credentials.
  | 
  | (I posted this comment with the heads-up a few minutes after
  | your comment above and then expanded it as you didn't respond.)
 
  | bilekas wrote:
  | > But the underlying policies and models are almost entirely
  | missing (there are a couple valuable components in [1]).
  | Without those, we can't evaluate the behavior and possible
  | effects of "the algorithm."
  | 
  | Haven't gone through yet, but yeah, if that's the case, all
  | this is, is a glorified framework to plug your own in.. Not
  | exactly what was promised.
 
  | fanagra32 wrote:
  | Exactly. Open-washing at its finest. Just like OpenAI. "Open"
  | just doesn't mean anything anymore at this point.
  | 
  | > Context: I teach at Princeton and study social media and
  | recommendation systems.
  | 
  | I'm not sure how this context was relevant. Your post was
  | perfectly able to stand on its own feet and convince through
  | content, not through authority. If anything, this "context"
  | removed pondus from it by appealing to authority.
 
  | eecc wrote:
  | Wouldn't that make them easy prey of "spam SEO". However, given
  | the framework isn't it still possible to guess the models?
 
  | robopsychology wrote:
  | Congrats on reading several hundred thousand lines of code in 1
  | hour and 8 minutes :)
 
    | elorm wrote:
    | You missed this in your rush to display your newly acquired
    | sarcasm101 skills:                 "Skim": To read quickly or
    | cursorily, to glance over, or to omit details in order to get
    | the gist of something.
 
      | robopsychology wrote:
      | Context: I studied at Oxford
      | 
      | Fair point, I missed that when I skimmed OPs comment
 
    | pnt12 wrote:
    | It's fast to read stuff when you have the domain knowledge.
    | The weights won't be a 5kb Scala file: they'd probably be a
    | big binary file, which is easy to search it github/locally
    | after cloning.
    | 
    | Otherwise, if they are provided, someone in the thread will
    | surely point to them.
 
    | culi wrote:
    | imagine thinking you need to read every file in a project to
    | understand the architecture and which pieces are important
    | for specific functionality you're looking to understand. Have
    | you ever picked up a bugfix ticket for some code you didn't
    | write?
 
    | tpmx wrote:
    | We should really all just bow in awe as we are clearly
    | inferior.
 
      | robopsychology wrote:
      | Princeton has a Code Reading 101 that all
      | postdocs/professors must take, however in exchange for the
      | _Secrets of Speed Reading_ you must acknowledge every
      | message with where you learnt those skills.
 
    | raggi wrote:
    | class project, 200 students, 1500 LoC each. Time for grading.
    | 
    | there are contexts in which this may be well practiced.
 
  | kadavy wrote:
  | For example, MostRecentCombinedUserSnapshotSource seems to be
  | influential (such as for calculating "tweepcred"), but we can't
  | see how it's calculated.
 
| bastardoperator wrote:
| My favorite is ci/ci.sh                 #!/bin/sh            exit
| 0
 
| summarity wrote:
| Main repos:
| 
| - https://github.com/twitter/the-algorithm
| 
| - https://github.com/twitter/the-algorithm-ml
| 
| Blogs:
| 
| - Eng: https://blog.twitter.com/engineering/en_us/topics/open-
| sourc...
| 
| - Biz: https://blog.twitter.com/en_us/topics/company/2023/a-new-
| era...
 
  | jmeister wrote:
  | Twitter spaces live right now:
  | https://twitter.com/i/spaces/1jMJgLdenVjxL
 
| Patrickmi wrote:
| Didn't Elon check the codebase before open sourcing it, like was
| he expecting everyone to be happy when seeing author_is_elon ?
 
| ryzvonusef wrote:
| https://twitter.com/jarokrolewski/status/1641892148084629504
| > the main neural network part of @Twitter recsys algo is based
| on 2021 work of #SinaWeibo - Chinese clone of Twitter
| 
| interesting claim
 
  | ryzvonusef wrote:
  | https://twitter.com/Sandeeparuchuri/status/16419015979860172...
  | > Part of twitter's algo Jack Dorsey, Katy Perry, Stephen Curry
  | and Barack Obama as "testing accounts" for getting random
  | Tweets for testing
 
  | ryzvonusef wrote:
  | Some more strange quirks:
  | 
  | https://twitter.com/Ben_Cary_/status/1641893540614623258
  | > Twitter use to rank posts higher for  those who had more
  | followers/less people they follow              > They are
  | removing that as of today but kinda interesting that someone
  | with 10k/10k followers would get less reach than if they had
  | 10k followers and only followed 6k
 
    | suddenclarity wrote:
    | Might be a language barrier from my side but it doesn't
    | really sound like a "strange quirk" but rather a wise
    | decision. Following people to get followers is one of the
    | oldest spam methods on social media. It's not surprising that
    | they would reduce the reach of accounts using those methods.
 
      | ryzvonusef wrote:
      | good point! "interesting insight" might have been a better
      | phrase. i am just skimming twitter to get some scoops on
      | the algo release
 
    | ryzvonusef wrote:
    | https://twitter.com/_johnforte/status/1641900138305134594
    | > Twitter is also using the page rank algo that google
    | created. Basically, if a lot of people interact with the user
    | they create more authority in the system.
 
      | ryzvonusef wrote:
      | https://twitter.com/federicolois/status/1641900547555901441
      | > Interesting piece here. If you are following less than
      | 500 and  you are verified your reputation is 100.
 
        | ryzvonusef wrote:
        | https://twitter.com/carlcarrie/status/1641900542573133826
        | > The Twitter Algo uses graph of followers and tweet
        | similarity to identify what alignment you are politically
 
      | hk__2 wrote:
      | > > Twitter is also using the page rank algo that google
      | created
      | 
      | Nitpick, but the PageRank algorithm was created before
      | Google, because it was the foundation of it.
 
| diebeforei485 wrote:
| Kudos for open-sourcing this.
 
| pram wrote:
| I wonder what determines 'cred' for this part:
| 
| https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
 
  | pram wrote:
  | I answered my own question https://github.com/twitter/the-
  | algorithm/blob/7f90d0ca342b92...
  | 
  | "This method reduces the page rank of users who have a low
  | number of followers but a high number of followings."
 
    | anigbrowl wrote:
    | Heh, I knew it. You need to prune your own following list
    | regularly or become less and less visible. I suspect (but
    | have yet to check) that they also weight visibility in terms
    | of historical follower growth.
    | 
    | That's why you see so many trolls with very low follower
    | counts; it's more effective to make/purchase a new firstname-
    | bunchanumbers account and poop in people's replies than to
    | let Twitter decide placement based on historical factors.
 
| sillysaurusx wrote:
| Say what you will about Elon, but this wouldn't have happened
| without him. Thanks!
| 
| And thank you to everyone at Twitter who helped organize this
| release. Open sourcing something like this is no small effort.
 
  | anigbrowl wrote:
  | I am not sure about that. Twitter has open sourced a lot of
  | stuff in the past. There were certainly people there who would
  | run the site as a nonprofit public service if they had the
  | choice.
 
    | robopsychology wrote:
    | But they haven't open sourced their recommendation engine in
    | the past
 
    | BryantD wrote:
    | This release seems less immediately valuable than their other
    | contributions, but historically more significant. It's a pity
    | we don't have commits although that would be a huge privacy
    | issue.
    | 
    | But yeah, while I would never work for Elon I'm glad he did
    | this.
 
    | drexlspivey wrote:
    | Twitter had 18 years to publish their algorithm under the
    | previous management and they didn't.
 
    | nonethewiser wrote:
    | Well, they didn't
 
    | sangnoir wrote:
    | Twitter contributed a lot to Map-Reduce, ETL and Scala
    | communities: IMO they punched above thier weight.
    | 
    | Sadly, I think their best open-source contribution days are
    | behind them with all the hardcore engineering they now have
    | to do with fewer engineers.
    | 
    | Edit: I forgot about Bootstrap! That projects saved the world
    | from millions of ugly web apps and dashboards built by
    | clueless backend engineers.
 
  | zachnwhite wrote:
  | --
 
    | kayodelycaon wrote:
    | The importance of Twitter was it being the primary posting
    | location for a lot of things. A number of the artists and
    | other creatives I know have gotten absolutely gutted by this.
 
      | nicky0 wrote:
      | Gutted about what exactly? Twitter is still there and
      | pretty much works like before.
 
      | nonethewiser wrote:
      | By open sourcing the algorithm? Or what?
 
    | JSavageOne wrote:
    | [flagged]
 
    | sillysaurusx wrote:
    | I'd be nothing without Twitter. It's had more impact on my
    | life than any other platform. I got lucky, but luck was only
    | part of it.
    | 
    | Being able to DM people is incredible. It's the AOL Messenger
    | of 2023. If it went offline, it'd be a terrible loss.
 
      | zachnwhite wrote:
      | [flagged]
 
        | anigbrowl wrote:
        | This isn't /g/, you have to employ a minimum level of
        | politeness here.
 
        | sillysaurusx wrote:
        | I did. Your mother sends her regards.
        | 
        | (Joking aside, you'll find HN to be a wonderful place to
        | hang out, but only if you get into the right mindset. In
        | the meantime, enjoy your weekend.)
 
        | zachnwhite wrote:
        | You literally said "I'd be nothing without twitter."
        | Please use this weekend to reevaluate your life.
 
      | jytechdevops wrote:
      | its literally led to the complete change of my future due
      | to the ability to follow the interactions of successful
      | people who are active on the platform. I've learned from
      | them as if they were my direct mentors and made huge life
      | decisions based on some of their talking points /
      | motivational mindset. Without it, my life would've been a
      | bubble in Virginia with my nearest network being 5 friends
      | who love cranking out bottle on the weekends.
 
    | hutzlibu wrote:
    | Politicans and companies all over the world are using it.
    | Controlling that information space, is real power. And I am
    | not yet clear, how much that release will bring needed
    | transparency. As the algorithm in production, can have major
    | tweaks.
 
  | hutzlibu wrote:
  | Judging by the many "issues" already, it might have been a bad
  | idea to release on a friday, though.
 
    | bitshiftfaced wrote:
    | I'm not connecting the dots. Why is it bad to release on
    | Friday?
 
      | dmix wrote:
      | It's not like they are going to release an emergency patch
      | fix. This was a one way street.
 
    | dawnerd wrote:
    | This is 100% not their working copy.
 
      | hutzlibu wrote:
      | Well yes, but I am pretty sure, elon envisoned warm welcome
      | by the OS community and help for free and now that is off
      | to a bad start.
 
        | [deleted]
 
        | sebzim4500 wrote:
        | I doubt he thought he'd get any help for free, given I
        | don't think you can run this code there isn't a
        | meaningful way to contribute.
        | 
        | It's just about transparency or PR, take your pick.
 
        | hutzlibu wrote:
        | "given I don't think you can run this code there isn't a
        | meaningful way to contribute."
        | 
        | Hm, there is lots of code released, I would think that
        | some of it, can be run and might be forked and useful in
        | other context, but mainly it is a PR move, sure.
 
        | undersuit wrote:
        | He literally tweeted:
        | 
        | "No doubt, many embarrassing issues will be discovered,
        | but we will fix them fast!"
 
        | dmix wrote:
        | People noticing stuff is free work but it's still
        | different than getting PRs and actual solutions like a
        | real OSS project. And that's fine, people are doing it
        | because they care or like the attention or outrage and
        | all the other personal/social motivations.
 
| [deleted]
 
| cwkoss wrote:
| The twitter algorithm sucks balls and heavily overweights who's
| paid for a checkmark.
| 
| The default feed view has grown increasingly useless over the
| past ~6 months.
 
  | lhnz wrote:
  | I don't think any changes to bias towards bluechecks have been
  | made yet.
 
    | cwkoss wrote:
    | A significant portion of my 'for you' is low quality tweets
    | from paid bluechecks.
    | 
    | The people who are willing to pay to be heard more seem to be
    | willing because everyone is already tired of listening to
    | them.
 
| benatkin wrote:
| Party in the issues: https://github.com/twitter/the-
| algorithm/issues
 
| rco8786 wrote:
| So as expected, there is exactly nothing that favors posters from
| one side of the political spectrum. I don't expect that this
| article will do anything to calm down those who are convinced
| otherwise though.
| 
| Well written article, from an engineer's perspective.
 
  | RoyGBivCap wrote:
  | [dead]
 
  | IngvarLynn wrote:
  | Algorithm exists and is non-trivial, therefore it favors those
  | groups of posters that spend more effort to hack it.
 
  | vore wrote:
  | Well, it does say this:                  Ranking is achieved
  | with a ~48M parameter neural network that is continuously
  | trained on Tweet interactions to optimize for positive
  | engagement (e.g. Likes, Retweets, and Replies). This ranking
  | mechanism takes into account thousands of features and outputs
  | ten labels to give each Tweet a score, where each label
  | represents the probability of an engagement. We rank the Tweets
  | from these scores.
  | 
  | This is basically the ultimate black box, so I don't think you
  | can really conclude anything like this either way.
 
    | bombcar wrote:
    | More like the ultimate hug box generator, that will quickly
    | partition you into a self-reinforcing bucket.
 
| paxys wrote:
| Since this is what most people are going to want to see:
| 
| > We also took additional steps to ensure that user safety and
| privacy would be protected, including our decision not to release
| training data or model weights associated with the Twitter
| algorithm at this point.
 
| ProAm wrote:
| 1) Fire/layoff 75% of employees.
| 
| 2) Open source the code base.
| 
| 3) Get the community to code for you free of charge.
| 
| 4) Profit. (hopefully +$20B)
 
| crop_rotation wrote:
| Wouldn't any such system depend on 10 other internal systems, 20
| databases directly or indirectly, each affecting the behaviour of
| the recommendation engine. That makes me doubtful studying such a
| recommendation engine is any better than a purely academic
| exercise.
 
  | justrealist wrote:
  | Having anything public at all is wildly better than the nothing
  | that is standard among social media companies.
  | 
  | Let's not focus criticism on an attempt to do _something_.
 
  | sithlord wrote:
  | thats why its "the algorithm" not the source of data/truth
 
  | softfalcon wrote:
  | You're probably right, but analyzing such things could still be
  | useful for research.
  | 
  | I know that open source code around commenting online directly
  | impacted the direction my current team went building our
  | community tooling.
  | 
  | I'll take even a glimpse into the machinations of any social
  | media giant. It's better than nothing!
 
| javajosh wrote:
| Is there demand for a service that simply shows you the things
| the people you follow wrote? (It would be up to you not follow so
| many people that you can't keep up.)
 
| matesz wrote:
| It is really nice to see how bazel is used in the wild. It looks
| so clean. Why we are not using it for everything?
 
| pledess wrote:
| there may be a hint of which elections were of interest:
| 
| https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
 
| firstSpeaker wrote:
| Would it be developed in open as well or there will be frequent
| merge from their internal repos?
 
| HellsMaddy wrote:
| Interesting:                   // we only keep unfollows in the
| past 90 days due to the huge size of this dataset,         // and
| to prevent permanent "shadow-banning" in the event of accidental
| unfollows.         // we treat unfollows as less critical than
| above 4 negative signals, since it deals more with         //
| interest than health typically, which might change over time.
| val unfollows: SCollection[InteractionGraphRawInput] =
| GraphUtil             .getSocialGraphFeatures(
| readSnapshot(SocialgraphUnfollowsScalaDataset, sc),
| FeatureName.NumUnfollows,               endTs)
| .filter(_.age < 90)
| 
| https://github.com/twitter/the-algorithm/blob/main/src/scala...
 
  | dmix wrote:
  | How long does the NSA record them?
 
| tric wrote:
| From https://github.com/twitter/the-
| algorithm/blob/7f90d0ca342b92...                   (
| "author_is_elon",           candidate =>             candidate
| .getOrElse(AuthorIdFeature,
| None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))),
| (           "author_is_power_user",           candidate =>
| candidate               .getOrElse(AuthorIdFeature, None)
| .exists(candidate.getOrElse(DDGStatsVitsFeature,
| Set.empty[Long]).contains)),         (
| "author_is_democrat",           candidate =>
| candidate               .getOrElse(AuthorIdFeature, None)
| .exists(candidate.getOrElse(DDGStatsDemocratsFeature,
| Set.empty[Long]).contains)),         (
| "author_is_republican",           candidate =>
| candidate               .getOrElse(AuthorIdFeature, None)
| .exists(candidate.getOrElse(DDGStatsRepublicansFeature,
| Set.empty[Long]).contains)),         )
 
  | philistine wrote:
  | Coded as if the only two political parties on the planet were
  | the Rs and the Ds. Shameful.
 
  | bikeformind wrote:
  | I opened this thread just to verify this would be top comment,
  | good job hn
 
  | [deleted]
 
  | jdnordy wrote:
  | github link now show a warning at the top of the page:
  | 
  | > This commit does not belong to any branch on this repository,
  | and may belong to a fork outside of the repository.
  | 
  | Is this new? Perhaps Twitter already removed the code from
  | their main branch? Or was this just a joke from the beginning?
 
  | jawns wrote:
  | The author_is_elon flag doesn't surprise me, but the two
  | political designators are somewhat shocking. I'd sure like to
  | know what changes based on what Twitter knows about your
  | political affiliation.
 
    | jen20 wrote:
    | It's not that shocking...
    | 
    | Half the people that got promoted on my timeline were
    | perpetually candidates for elections I couldn't vote in, and
    | they _self-identified_ as Republican or Democrat in their own
    | bios, or via the registration of their candidacy...
    | 
    | This is why I exclusively used to use Twitter in the "people
    | I follow only" mode, and simply shut my account down when
    | they pushed harder on the algorithm.
 
    | jandrese wrote:
    | I thought it was interesting how it explicitly doesn't boost
    | independents. So much of the two-party system is self-
    | reinforcing.
 
      | wahnfrieden wrote:
      | Or anti-statists! Not everyone who engages with politics is
      | a bootlicker for authoritarianism but it sure feels like
      | there's no space made for this perspective (obviously)
 
      | lapetitejort wrote:
      | I'd be interested in which bin people like Joe Manchin,
      | Kyrsten Sinema, Susan Collins, et. al. are placed.
 
      | emodendroket wrote:
      | The vast majority of self-proclaimed independents vote with
      | one party just as reliably as registered members.
 
        | philistine wrote:
        | American's lax attitude towards cultivating more than two
        | parties is literally killing the republic from the
        | inside.
 
        | airstrike wrote:
        | You'd think a country that played a central role in a
        | global, decades-long unstable regime of bipolar power
        | that routinely pushed mankind to the brink of nuclear
        | oblivion would know better than to have a bipolar
        | electoral system
 
        | robertlagrant wrote:
        | > routinely pushed mankind to the brink of nuclear
        | oblivion
        | 
        | I'm struggling to think of a reason why this is anything
        | but bad faith nonsense.
 
        | weaksauce wrote:
        | have you never heard of the many times we almost came to
        | setting off full scale nuclear warfare because of the
        | bipolar power war of the ussr v usa et.al.?
        | 
        | the only thing that saved us was cooler heads that
        | prevailed on both sides.
 
        | emodendroket wrote:
        | Why would throwing more actors with similar capabilities
        | into the mix make the situation any more stable though?
        | That seems like basically the old European Balance of
        | Power, which broke out into open conflict more
        | frequently.
 
        | emodendroket wrote:
        | In some other countries the different interest groups
        | sort themselves into two factions _after_ being elected
        | but I don 't know that it is really that different in
        | practice.
 
        | TillE wrote:
        | It's not different, or rather it doesn't produce
        | meaningfully different outcomes. I'm not aware of any
        | parliamentary system with a wonderful diversity of
        | thought and a long record of positive accomplishments.
        | You end up with ruling coalitions which are typically
        | pretty awful.
        | 
        | Ironically, America has one of the most open political
        | systems. You register as one party or the other and vote
        | in primaries. This has lead to a huge variety of people
        | replacing hated mainstream politicians. That's way more
        | than you can say for many other countries.
 
        | hraedon wrote:
        | It isn't an attitudinal problem, it is the logical
        | outcome of our political systems. In political science it
        | is known as Duverger's law: single ballot, winner take
        | all systems inevitably tend toward a two party
        | equilibrium.
        | 
        | Changing this requires states to adopt alternative
        | systems, which can sometimes mean amending state
        | constitutions. It isn't easy or straightforward, and the
        | general sense is that there are better things to spend
        | that effort on.
 
        | noizejoy wrote:
        | > single ballot, winner take all systems inevitably tend
        | toward a two party equilibrium.
        | 
        | I'm not convinced it's quite that simple.
        | 
        | For example, Canada also has a first-past-the-post
        | electoral system - yet political parties here have come
        | and gone. And continue to do so.
 
      | tablespoon wrote:
      | > I thought it was interesting how it explicitly doesn't
      | boost independents. So much of the two-party system is
      | self-reinforcing.
      | 
      | Is it boosting? Others are claiming this code is just for
      | metrics collection:
      | https://news.ycombinator.com/item?id=35391896.
      | 
      | But on the topic of Democrats vs. Republican vs.
      | independent; a big factor may be that "Democrat" and
      | "Republican" are much more cohesive groups and therefore
      | much easier to define. No one can honestly define
      | "independent" except in a kind of "none of the above"
      | sense, since they can range anywhere from extreme right, to
      | the center, to the extreme left.
 
        | dd36 wrote:
        | Why measure something if you don't intend to change it?
 
        | astrange wrote:
        | So you can make sure not to accidentally change it.
 
        | lapetitejort wrote:
        | And if one party trends more towards violent extremism,
        | they get boosted the same as before?
 
        | boringg wrote:
        | There are many reasons to measure things if you don't
        | intend to change them.
 
        | jandrese wrote:
        | Depends what the metrics are used for. It doesn't make
        | sense to apply artificial boosts to metrics that are only
        | used for internal accounting. Well, maybe if you have an
        | egotist CEO, but that wouldn't explain the rest of the
        | boosts. We have to assume this code has some sort of
        | effect somewhere.
 
      | robertlagrant wrote:
      | Isn't it intrinsically self-reinforcing, if you have a
      | winner takes all system? It's almost always better to join
      | an existing team than start a new one.
 
      | airstrike wrote:
      | Also how about.... everyone else in the world who is not an
      | American voter?
 
        | threeseed wrote:
        | Only anecdotally.
        | 
        | But ever since Musk took over the amount of US political
        | content has significantly increased in particular from
        | the right despite me not living in the US.
        | 
        | It's hard to tell whether previously political content
        | was weighted less and Musk has removed those controls or
        | whether they are now weighted higher.
 
        | kevingadd wrote:
        | Some of the power users Musk reportedly had boosted are
        | specifically right wing political posters, like catturd2.
        | But then he's also boosted some high profile left leaning
        | politicians, so it's not exclusive. It does mean you're
        | more likely to see right wing American political content
        | either way, which has to be annoying for people outside
        | of the US.
        | 
        | https://gizmodo.com/twitter-algorithm-aoc-ben-shapiro-
        | cattur...
 
        | realjhol wrote:
        | Why do you believe it would be annoying to us?
 
        | fyloraspit wrote:
        | Would it not be just as annoying to see a slant the other
        | direction, or in any extreme direction?
 
        | meragrin_ wrote:
        | I live in the US in "Trump country" and don't see any
        | political content. Maybe its just related to the people
        | you follow and the content you look at?
 
        | tormeh wrote:
        | This is what it means to let another country own your
        | social media. Their ideas and memes unconsciously get
        | preferential treatment. This is maybe not a good thing,
        | but it is what it is.
 
        | adamckay wrote:
        | > Their ideas and memes unconsciously get preferential
        | treatment.
        | 
        | I think as this repo shows it's conciously, rather than
        | unconsciously, getting preferential treatment.
 
        | blululu wrote:
        | It's omission. They built for Americans because they were
        | Americans. No one building this said "let's ignore
        | Canadian politics" or any other country, they just didn't
        | think about them at all, because like most Americans they
        | don't really care about the insidious Quebecois plots to
        | annex Prince Edward Island or whatever actual issues are
        | happening in Canada.
 
        | ben_w wrote:
        | Before I deliberately locked myself out of it (well
        | before Musk), I asked for my data.
        | 
        | They classify me as:                 * speaks Indonesian
        | Interested in:       * Beer       * Cricket       * DJs
        | * Dance       * Enterprise software       * Horror
        | * NFL football       * South America            And aged
        | either between 13-54 or (and?) over 65
        | 
        | Other than the age (I'm neither under 13 nor between
        | 55-64), everything I've listed is incorrect.
        | 
        | On that basis, they'd probably call me a Republican.
        | 
        | Carries a different meaning when you're British, that
        | name does.
 
        | tedivm wrote:
        | I'm a guy in my 30s but they had me classified as a 65+
        | year old woman. They also thought I was a radiologist.
 
      | bryanrasmussen wrote:
      | I thought it was interesting that twitter thinks the U.S is
      | the world
 
        | whatgoodisaroad wrote:
        | For a big tech company, the regulatory regime is the
        | world. For Twitter that's the US.
 
    | JustSomeNobody wrote:
    | Exactly. I don't see NPA (no party affiliation) anywhere.
 
      | ldoughty wrote:
      | Did you not get the memo? "If you're not with us, you're
      | against us".
      | 
      | It's probably more of a conservative/liberal identifier
      | based on US political party ideals... And they likely would
      | filter any metrics from this by the users country
 
    | stouset wrote:
    | I _suspect_ that these are used for metrics tracking rather
    | than being fed back into the recommendation engine. But there
    | 's no real way to know for sure given the limited release.
    | These predicates aren't actually used anywhere in the code
    | that's been made available.
 
    | 6nf wrote:
    | So many questions. How are users tagged D or R? Is that a
    | manual process or automated somehow? What is the effect of
    | these tags? Can I find out if my Twitter account is in one of
    | those buckets?
 
      | abracadaniel wrote:
      | And how are they choosing to balance them, per capita, or
      | just both sides should get 50%? It seems pretty clear they
      | are making editorial decisions here. Does that break their
      | section 230 protections?
 
        | astrange wrote:
        | "Section 230 protections" have nothing to do with
        | "editorial decisions"
 
        | KarlKemp wrote:
        | Please read something. Preferably section 230, which is
        | short. Alternatively something entirely different, as
        | long as that keeps you occupied.
 
        | krapp wrote:
        | >It seems pretty clear they are making editorial
        | decisions here. Does that break their section 230
        | protections?
        | 
        | No.
        | 
        | https://www.techdirt.com/2020/06/23/hello-youve-been-
        | referre...
 
      | [deleted]
 
      | smegger001 wrote:
      | you could probably algorithmically determine it in most
      | cases based on any number of indicators from phrases used,
      | to communities interacted with, which hashtages are
      | included, which cohort retweets and likes most etc... thats
      | not even getting into simply tagging political figures with
      | the party they officially affiliate themselves with
 
      | moffkalast wrote:
      | Especially for people that aren't... you know.. Americans.
      | 
      | Unless they mean actual public figure party members which
      | are known and probably verified.
 
    | partiallypro wrote:
    | Facebook guesses your political affiliation as well, you can
    | even look uber your settings to see what they guessed.
 
    | krapp wrote:
    | The repo suggests it's about tracking engagement metrics[0],
    | so Team Red people see more Team Red content and vice versa.
    | Nothing nefarious.
    | 
    | [0]https://github.com/twitter/the-
    | algorithm/blob/7f90d0ca342b92...
 
      | minimaxir wrote:
      | Having a separate flag to track engagement specifically for
      | Elon tweets isn't nefarious, but _weird_.
 
      | jeron wrote:
      | You say it's not nefarious but isn't that how echo chambers
      | are created?
 
        | krapp wrote:
        | I don't believe echo chambers are nefarious - there's no
        | hidden agenda involved with them. That's just how
        | recommendation algorithms work, and it's what most people
        | want.
        | 
        | But if someone finds some code that suppresses
        | recommendations from a specific political ideology across
        | the board, _that_ would be nefarious, IMO.
 
        | thfuran wrote:
        | Echo chambers may not be nefarious, but they are
        | insidious.
 
        | krapp wrote:
        | They can be if they're involuntary and inescapable, but
        | neither is the case for Twitter. It's designed around
        | letting you curate your own feed, but it also constantly
        | throws random stuff in through retweets and quote tweets
        | - which is what people hate the most about the platform.
 
        | ClumsyPilot wrote:
        | I don't believe discrimination is nefarious - there is no
        | hidden agenda involved. That's just how societies work,
        | it's natural behaviour - to belong to a group they
        | identify with and to keep out strangers and weirdos.
 
        | kodah wrote:
        | This comment likely violates the guidelines both in form
        | and content.
 
        | krapp wrote:
        | I don't believe repeating someone's comment nearly
        | verbatim is as clever as you want it to be, nor do I
        | believe recommendation algorithms are equivalent to the
        | kinds of societal discrimination you allude to, nor would
        | any reasonable person.
        | 
        | Instead of trying very hard to be clever, please next
        | time try just as hard to make a valid point. I know it
        | can be difficult when you think you smell karma in the
        | water but do try.
 
        | yazzku wrote:
        | The echo chamber in your first point does the suppression
        | in the second.
 
      | tric wrote:
      | Why specifically track political parties? Where is
      | author_is_american? Or author_is_mayonnaise_enjoyer?
      | 
      | Maybe it was a choice made many years ago that they thought
      | was appropriate, but we can't yet know it's not used for
      | other purposes. We can at least be reasonably sure they've
      | added the author_is_elon within the past year. I would have
      | thought there would be many more descriptors, or non-
      | controversial descriptors.
      | 
      | Or maybe Elon specifically added those before releasing
      | this code to get people riled up.
 
        | krapp wrote:
        | Twitter became a very politically charged platform after
        | 2016, mainly among Americans. They'd be idiots not to
        | take advantage of that.
 
        | jdminhbg wrote:
        | The point is probably to check that changes they make
        | aren't accidentally politically biased.
 
  | GaryNumanVevo wrote:
  | I wonder who's on the "VIT" (Very Important Tweeter) list?
 
    | wahnfrieden wrote:
    | People like Ben Shapiro, Glenn Greenwald, @catturd2
 
      | fortuna86 wrote:
      | Right wing high engagement accounts. Through in one or two
      | accounts like @AOC for "balance"
 
        | coolspot wrote:
        | BTW: How @aoc got three-letter handle?
 
    | minimaxir wrote:
    | There's a few: https://www.rollingstone.com/culture/culture-
    | news/twitter-vi...
 
  | culi wrote:
  | what is vits?                 private val DarkRequestAnnotation
  | = "clnt/has_dark_request"       private val Democrats =
  | "democrats"       private val Republicans = "republicans"
  | private val Elon = "elon"       private val Vits = "vits"
 
    | nanidin wrote:
    | Very Important Tweeters, a play on VIP perhaps?
 
    | stefanos82 wrote:
    | Very Important Tweets, I presume?
 
    | mshafrir wrote:
    | Very Important Tweeters
 
  | commandlinefan wrote:
  | At first I thought this post was a joke - and it was actually a
  | pretty good joke. Yikes.
 
  | schemescape wrote:
  | Did they not expect people to notice suspicious code like this?
  | 
  | Or did they leave this in just so they could hold its removal
  | up as an example of listening to the community?
 
    | bitshiftfaced wrote:
    | Could be that those in charge of preparing this open sourced
    | repository did it begrudgingly, and so they perceived the
    | fact that it looked bad as a positive thing. "Hey, you wanted
    | us to release the code. Happy now?"
 
    | dmix wrote:
    | Why are you assuming this knowledge is harmful to them? What
    | do you think it means for their business?
    | 
    | No other social media platform will have this sort of
    | accountability and public pressure to be better like having
    | their recommendation algorithms public.
 
  | minimaxir wrote:
  | The full list of model features in that file is interesting.
  | 
  | I am surprised at the number of inherently redundant and
  | colinear features, though. (e.g. has_1_image, has_2_images,
  | has_3_images, has_4_images)
 
    | qyph wrote:
    | Those aren't redundant or collinear though? Maybe you are
    | surprised they didn't encode this as an integer "num_images"?
    | It is fairly common to one hot encode ordinal variables with
    | only a few common/possible values this way.
 
      | minimaxir wrote:
      | True, it still seems odd to encode an explicitly ordinal
      | variable as categorical (particularly one with a small
      | finite range, in contrast to the follower logarithmic
      | bucket ones), but Twitter's layout is weird enough that it
      | could be a impactful difference in terms of engagement.
 
        | disgruntledphd2 wrote:
        | This is (weirdly) common in production ML codebases
        | written by software engineers. Like you, I have no idea
        | why unless it's a memory optimisation (where you count 4+
        | as many).
 
        | ladon86 wrote:
        | Having every column as a boolean (0/1) means you can
        | treat it as a bitmap. As an (entirely fictional) example,
        | imagine if you wanted to get the features of a thread
        | instead of a single tweet. You could do it as a union of
        | all the tweets:
        | 
        | threadFeatures = tweet1 | tweet2 | tweet2
 
        | disgruntledphd2 wrote:
        | Ok that makes lots of sense from an engineering
        | perspective. It's pretty insane from a statistical
        | perspective though, which I think was the original point.
 
  | sschueller wrote:
  | Well someone just asked about it in the live spaces[1] Elon is
  | hosting and he said that should not be there. An engineer said
  | afterwards it is just for metrics but then Elon chimed in again
  | and said "we should get rid of it, it should be gone."
  | 
  | [1]
  | https://twitter.com/elonmusk/status/1641880448061120513?s=20
 
    | JustSomeNobody wrote:
    | Doesn't necessarily mean he didn't want it there in the first
    | place. Why else would it be there?
 
      | JoshCole wrote:
      | There are millions of lines of code existent. There are
      | thousands added. Your prior for it being Elon's addition
      | should be something like 10,000/1,000,000 or roughly 1/100.
      | The prior that it wasn't Elon's change is going to be
      | something like 99/100.
      | 
      | When you add the additional information that Elon wants the
      | code removed, but existing Twitter engineers think it
      | appropriate to keep this actually increases the probability
      | of it being added by the existing Twitter engineers and
      | decreases the probability it was added due to Elon.
      | 
      | Obviously, these are rough numbers, but hopefully seeing
      | any numbers at all helps you to get an intuition for the
      | math.
 
        | jsnell wrote:
        | Why is lines of code the appropriate input here? Here's a
        | different computation that is at least as plausible:
        | 
        | There are hundreds of millions of users. Let's say 300M.
        | Only a single one is special-cased in this code: the
        | narcisistic CEO who reportedly went ballistic when his
        | engagement metrics went down. The prior that it's a
        | change done in response to his demands is
        | 299999999/300000000.
        | 
        | (But of course it was added by existing Twitter
        | engineers. The odds of Musk being able to actually commit
        | code to their repository are zero. Even if he had the
        | permissions, the man simply does not have the technical
        | acumen to make even a trivial change.)
 
      | [deleted]
 
    | XorNot wrote:
    | Of course he did because it makes him look bad and he's
    | desperate for praise and attention.
    | 
    | What he wanted was everything that feature provides, without
    | it ever being shown that it's there. But since he refuses to
    | hire PR people and almost certainly came up with this idea in
    | the last few days, no one was paid to hide its existence.
    | 
    | The next story out of Twitter will be the remaining engineers
    | being threatened because Musk can't see his tweet statistics
    | any more.
 
  | CathalMullan wrote:
  | Only used for metrics, apparently. [0]                 /**
  | * These author ID lists are used purely for metrics collection.
  | We track how often we are        * serving Tweets from these
  | authors and how often their tweets are being impressed by
  | users.        * This helps us validate in our A/B
  | experimentation platform that we do not ship changes        *
  | that negatively impacts one group over others.        */
  | 
  | [0]: https://github.com/twitter/the-
  | algorithm/blob/7f90d0ca342b92...
 
    | roughly wrote:
    | I expect they're tracking the red team/blue team metrics
    | because of the political shitstorm that's been the GOP's
    | assertions they're being silenced by The Algorithm.
 
      | panarky wrote:
      | The fallacy of false equivalence systematized in code.
      | 
      | Now one side can spew as much disinfo and incitement to
      | violence as it likes, and any algorithm change that
      | prevents this shit from getting amplified will be rejected
      | as bias.
      | 
      | BSaaS = Both Sides as a Service
 
        | eldritch_4ier wrote:
        | [flagged]
 
        | [deleted]
 
        | willdr wrote:
        | What are you talking about? Do you remember January 6?
 
        | eldritch_4ier wrote:
        | Yes, the one single right wing riot of the last 50 odd
        | years (and everyone promptly clear out at curfew, with
        | the 1 death being from a jumpy cop shooting a stupid
        | rioter).
        | 
        | Do you remember the dozens who died and the billions in
        | damage of the BLM riots? The actual insurrections in
        | Portland and Seattle?
 
        | panarky wrote:
        | Everybody remembers January 6 except those who want to
        | pretend it didn't happen.
        | 
        | How many remember the floods of Twitter incitement to hit
        | the gas in their F-150 trucks to run over protesters, and
        | then how many people actually perpetrated vehicle
        | attacks?
 
        | JackAndJack wrote:
        | What about Jan 6th? People walking around in people's
        | house lead by police?
 
        | zeven7 wrote:
        | I clicked downvote before reading the whole comment. At
        | first I thought you were talking about the red team when
        | you started with "mass riots and violence..." Then I read
        | the rest of your comment and still felt just as good
        | about my downvote. This isn't a constructive comment no
        | matter what "side" you're on.
 
        | luxuryballs wrote:
        | "Without debate, without criticism, no Administration and
        | no country can succeed--and no republic can survive. That
        | is why the Athenian lawmaker Solon decreed it a crime for
        | any citizen to shrink from controversy. And that is why
        | our press was protected by the First Amendment-- the only
        | business in America specifically protected by the
        | Constitution- -not primarily to amuse and entertain, not
        | to emphasize the trivial and the sentimental, not to
        | simply "give the public what it wants"--but to inform, to
        | arouse, to reflect, to state our dangers and our
        | opportunities, to indicate our crises and our choices, to
        | lead, mold, educate and sometimes even anger public
        | opinion." JFK
 
        | regularjack wrote:
        | "approved terminology".
 
        | KarlKemp wrote:
        | Well, technically they are looking for relative changes,
        | not equal total exposures.
 
    | sva_ wrote:
    | Ahh, the group of Elons.
    | 
    | I was wondering why I see so many tweets by him, and what his
    | "Group's" impression quote is.
    | 
    | This is actually pretty hilarious.
 
      | wardedVibe wrote:
      | Thankfully they haven't added a "no mute Elon" feature.
      | Yet.
 
      | nailer wrote:
      | That's not how it works. See the parent.
 
        | sva_ wrote:
        | They said that they use it for metrics, so clearly there
        | must be an "elon impression" metric.
 
    | andy_ppp wrote:
    | But who chooses the users to be metrics...
 
    | mochomocha wrote:
    | ... Metrics tracked in AB test. So even if it's not
    | explicitly encoded in the algo (or implicitly through some of
    | the features plugged in), they'll pick the winning cell as
    | long as it doesn't hurt Elon's metrics (I'm just parroting
    | the comment you quoted).
    | 
    | It doesn't have to be in the algorithm for the systems to be
    | tweaked to please Elon vanity metrics.
    | 
    | [I've been running lots of ML AB tests over the years, some
    | in organizations of similar size & complexity as Twitter]
 
      | threeseed wrote:
      | That lines up with reporting from Casey Newton a few days
      | ago where a handful of VIPs e.g. Musk, LeBron James, AOC
      | were being used as weather vanes to understand what the
      | algorithm was doing.
      | 
      | It definitely isn't just metrics. Any algorithm change that
      | negatively affected Musk was clearly not going live.
 
        | db48x wrote:
        | Do you think the code looked like that prior to Elon's
        | purchase? I suspect that there was another name there
        | before.
        | 
        | Separately, which of these groups do you think that they
        | use as a control?
 
        | threeseed wrote:
        | > I suspect that there was another name there before
        | 
        | Who ? Musk is unique in being obsessed with being liked
        | and relevant.
        | 
        | All of the other social CEOs including Porag and Jake
        | have never really cared that much. And none of them
        | participated in contributing content anything close to
        | what Musk does.
 
        | KarlKemp wrote:
        | There is absolutely no reason to believe there was
        | another Single user getting this treatment before. The
        | Elon-case was just copy & pasted as an ego-stroking hack.
 
        | philosopher1234 wrote:
        | i think he means trump, and i think it would've been
        | strategically wise to give trump special treatment (tho
        | probably not ideal)
 
        | jefftk wrote:
        | _> which of these groups do you think that they use as a
        | control?_
        | 
        | When you run an A/B test you randomly divide your users
        | into groups, one (treatment) getting the new behavior and
        | one (control) getting the current production behavior. So
        | your question doesn't make much sense?
 
        | [deleted]
 
      | btown wrote:
      | It's just a two-pass EM (Elon Maximization) algorithm!
 
    | minimaxir wrote:
    | The original code is a part of the home-mixer service, which
    | is the "Main service used to construct and serve the Home
    | Timeline."
    | 
    | I suspect the flag corresponds to weights not present in the
    | repo.
 
  | minimaxir wrote:
  | Update: Elon was asked about these in a Twitter Space, he says
  | it's not appropriate and will be removed from the codebase.
  | 
  | Additionally, from another Twitter engineer, the
  | Democrat/Republican flags are apparently 10 years old and not
  | important and do not have high feature importance.
 
    | lhnz wrote:
    | I think the decade old comment related to a different part of
    | the code regarding the number of followers you have in
    | relation to the number of accounts you follow. (Everybody on
    | the call wants to remove this: I wonder why they haven't
    | yet.)
 
      | delecti wrote:
      | Chesterton's Fence. In a sufficiently large system, you
      | should be hesitant to remove things unless you're sure you
      | know why it was added, and all the things that have come to
      | depend on it since.
      | 
      | I've definitely been hesitant to remove things I was pretty
      | confident weren't used anymore, just because I didn't want
      | to deal with the repercussions if I was wrong.
 
    | sillysaurusx wrote:
    | Elon seems embarrassed: https://twitter.com/elonmusk/status/1
    | 641908130274525187?s=61...
    | 
    | It'll be interesting to see what gets cut. Maybe just the
    | Elon flag, but maybe others too.
 
  | AdamH12113 wrote:
  | I read the code snippet before I saw the link and thought you
  | were joking, but yeah, there really is an author_is_elon flag
  | right there in the main branch.
 
  | ibraheemdev wrote:
  | > But we are deleting this bs. I only learned about it now!
  | Will be gone by tomorrow.
  | 
  | https://twitter.com/elonmusk/status/1641908130274525187?t=5t...
 
  | RoyGBivCap wrote:
  | [dead]
 
| HAL3000 wrote:
| Expect to see A LOT more spam on Twitter after this release. It's
| like giving SEO spammers access to google search ranking
| algorithm.
 
  | dmix wrote:
  | Stuff like this always has consequences, it doesn't mean it's a
  | net negative for society. It means you need to adapt and
  | actually fix the problems, while also benefiting more from the
  | accountability.
 
  | dmix wrote:
  | Stuff like this always has consequences, it doesn't mean it's a
  | net negative for society. It means you need to adapt and
  | actually fix the problems, while also benefiting more from the
  | accountability.
  | 
  | That's always been a risk of open source and not being hyper-
  | centralized.
 
| Laaas wrote:
| Praise where praise is due. Wasn't completely sure whether they
| would in fact release it or keep posturing.
 
| bilekas wrote:
| I'm supposed to be going out in 20 mins....
 
| stusmall wrote:
| I thought it was an april fools joke when I saw this:
| https://github.com/twitter/the-algorithm/blob/main/ci/ci.sh
| 
| Like a dig at the code quality.
 
  | endorphine wrote:
  | Is it not?
 
| robopsychology wrote:
| Why are there two spaces instead of four in this Python code, it
| hurts my soul
 
  | paulddraper wrote:
  | I assume they copied it from Google.
  | 
  | https://www.quora.com/Why-does-Google-use-2-spaces-for-Pytho...
 
  | holler wrote:
  | I guess they haven't read
  | https://peps.python.org/pep-0008/#indentation
  | 
  | "Use 4 spaces per indentation level."
 
  | anigbrowl wrote:
  | Cost saving measure. This sort of emotionalism is why engineers
  | need to kept out of the C-suite.
 
    | SpEd3Y wrote:
    | Not sure if you're being sarcastic, but if you're serious,
    | I'm pretty sure the OP is talking metaphorically. It's just a
    | slight annoyance he's not "emotional" about it.
    | 
    | I also fail to see how someone who is annoyed by code that
    | doesn't follow well established standards is somehow not a
    | good fit in the C-suite.
 
    | brucethemoose2 wrote:
    | Space bloat.
 
    | robopsychology wrote:
    | How is it a cost saving measure? Or are you being sarcastic?
    | Hard to tell over text!
 
      | anigbrowl wrote:
      | Yes, I'm joking. I also feel hurt by 2 space indents.
 
        | robopsychology wrote:
        | Hahaha I was sort of hoping you were an MBA who truly
        | believed hitting 2 spaces instead of 4 spaces would save
        | some sort of money at scale!
 
  | tayo42 wrote:
  | i think back in the day they copied googles python style guide
 
| Me1000 wrote:
| Squashing the commit history before releasing it was an
| interesting (and completely predictable) decision.
 
  | hk__2 wrote:
  | > Squashing the commit history before releasing it was an
  | interesting (and completely predictable) decision.
  | 
  | This is standard practice when it comes to open-sourcing such
  | repos that were closed-source for years.
 
  | jkubicek wrote:
  | It doesn't seem particularly interesting? I would never make a
  | formerly private repo public without first erasing the history.
  | There's no upside to showing everyone your work in progress and
  | almost unlimited downsides.
 
    | tapland wrote:
    | There's no way everyone had the same weight in all the
    | recommendation config files.
    | 
    | It's not about hiding old work, but changes just before
    | making it public.
 
  | mrguyorama wrote:
  | If they allowed you to git-blame the algorithm, some poor coder
  | would have definitely gotten murdered by a crazy person who
  | thought they purposely changed something to hurt them
 
| rschjosgknvx wrote:
| Hello Thursday
 
| [deleted]
 
| abalaji wrote:
| huh, legit open source too with 'Affero-GPL'
 
  | madeofpalk wrote:
  | AGPL is probably useless for any other site who'll want to use
  | it, as it would require them to open source their site that
  | uses it.
 
    | joeyh wrote:
    | Mastodon is conveniently also AGPL...
 
      | [deleted]
 
      | timeon wrote:
      | On reason I use Mastodon is that there is just
      | chronological timeline. Quick scroll and you are done. Bad
      | for advertising platform - good for user.
 
        | madeofpalk wrote:
        | I'm not sure why Mastodon would be interested in Twitters
        | non-chronological timeline. It seems to be pretty
        | antithetical to its goals.
 
        | suddenclarity wrote:
        | Twitter also has a chronological timeline nowadays?
 
        | [deleted]
 
      | [deleted]
 
| muratsu wrote:
| Given the complex relationship between advertisers, platform, and
| users I don't know if any meaningful contribution can be made to
| the algorithm without pissing anyone off. The following tab
| already gave people who're not interested in algo recommendations
| a way out. I don't quite understand the reasoning behind open
| sourcing the algorithm. Any thoughts?
 
| WhereIsTheTruth wrote:
| Is it even what they use in production?
| 
| There is code that favor Elon's tweets so I'd yes that's probably
| what they use
 
  | 0l wrote:
  | > There is code that favor Elon's tweets so I'd yes that's
  | probably what they use
  | 
  | Where?
 
    | zaroth wrote:
    | Spoiler - there isn't.
 
      | ftxbro wrote:
      | Yeah they track author_is_elon, author_is_democrat, and
      | author_is_republican but they don't appear to be used for
      | favoritism anywhere in this code.
 
        | WhereIsTheTruth wrote:
        | Why do they exist then? No code references it, but that's
        | Scala/JVM so many things depend on runtime
        | initialization, so maybe some other systems do? wich
        | ones?
        | 
        | Is is it there to help fight impersonations? should be
        | solved with Twitter Blue already?
        | 
        | There was reports of people receiving notifications about
        | Musk tweets despite not following him, so..
 
        | ftxbro wrote:
        | It's not used at run-time, it's in the repository so that
        | the large language models that are training on the github
        | corpus will know how special elon is, and so that the
        | future code written for twitter by GPT-5 will take the
        | hint and add the favoritism autonomously.
 
        | WhereIsTheTruth wrote:
        | Interesting argument, and definitely worth defending; an
        | AI that's biased by design to remember and preserve the
        | old world order's members rule and influence
        | 
        | Begs the question, why make it obvious?
 
  | sho_hn wrote:
  | Humorous conspiracy theory: Imagine if it is not, but
  | sanitized, and then someone added in Elon Boost to make it look
  | credible. :-)
 
    | WhereIsTheTruth wrote:
    | Or perhaps it does nothing at all, and it was there so we
    | talk about it, the "is_democrat"/"is_republican" is also
    | ridiculous, as if the goal was to demonstrate a point about
    | social media in general, hmm
 
| quotemstr wrote:
| Typically, we expect to be able to run "open source" software
| ourselves. If you open-source your C compiler, I can compile a C
| program with it. In a few recent high-profile cases though,
| companies have "open sourced" ML systems without releasing the
| model weights. This practice is just like your releasing the
| builds scripts for your C compiler, but not the compiler itself.
| While more transparency from social media will be enlightening,
| calling a release like this (or LLaMA) "open source" feels like
| equivocation. I'd love to see more full releases, weights
| included.
 
  | vonmoltke wrote:
  | Running this code would require a lot more than just the
  | exported models. There are a large number of code and system
  | dependencies missing.
 
    | quotemstr wrote:
    | Of course --- but without the model parameters, even stubbing
    | those systems would be useless. My point is that while this
    | release gives the public some information about how Twitter
    | ranks tweets, it doesn't tell the story because huge pieces
    | of "the algorithm" are missing. For example: the NSFW
    | classifier "open source" release doesn't tell us anything
    | about what Twitter considers NSFW and what it doesn't.
 
| varjag wrote:
| _Rank each Tweet using a machine learning model._
| 
| This does a lot of heavy lifting here.
 
  | thieving_magpie wrote:
  | There appears to be a repo for the-algorithm-ml:
  | https://github.com/twitter/the-algorithm-ml
 
| Thaxll wrote:
| Let's dig into Twitter code quality.
 
  | Kpourdeilami wrote:
  | https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92...
  | 
  | ```
  | 
  | def query_keys(self, language, task=2, size="50"):
  | if task == 2:                if language == "ar":
  | self.query_settings["adhoc_v2"]["table"] = "..."
  | elif language == "tr":
  | self.query_settings["adhoc_v2"]["table"] = "..."
  | elif language == "es":
  | self.query_settings["adhoc_v2"]["table"] = f"..."
  | else:                  self.query_settings["adhoc_v2"]["table"]
  | = "..."                return self.query_settings["adhoc_v2"]
  | if task == 3:                return
  | self.query_settings["adhoc_v3"]              raise
  | ValueError(f"There are no other tasks than 2 or 3. {task} does
  | not exist.")
  | 
  | ```
 
    | tentacleuno wrote:
    | Looking through it, the ... seems to be a placeholder for
    | information they'd prefer to be kept private. For example,
    | look in the keywords section in the same file you shared.
 
      | Kpourdeilami wrote:
      | You're correct, makes more sense now
 
| ryanisnan wrote:
| I want to go back to a world where there isn't an algorithm
| feeding me what someone "thinks" I want to read.
| 
| I want to see a chronological list of things sources I follow
| have posted.
| 
| Yes, I understand you can do this on Twitter still, but I would
| guess most people are more influenced by "the algorithm".
 
| simonsarris wrote:
| This is pretty limited. I picked a term used in the diagram to
| see what I could find out about it. But there seems to be _next
| to nothing_ in the released code about the mentioned  "author
| diversity". No real code or description.
 
  | mardifoufs wrote:
  | I think the relevant part of the code is in this other repo:
  | 
  | https://github.com/twitter/the-algorithm
  | 
  | Not sure if it has what you were looking for (and maybe you
  | already checked this repo, too!), but it's more relevant than
  | the linked repo imo
 
| bagels wrote:
| "Written by the Twitter Team"
| 
| I found it interesting that there is no attribution. Most other
| companies list the authors on engineering blogs (eg. Facebook,
| Uber, etc.)
| 
| This topic seems to draw the attention of unhinged people, so I
| suppose I wouldn't want my name on it either.
 
  | f38zf5vdt wrote:
  | No one wants to go to jail for Elon, who has been flagrantly
  | violating FTC orders.[1] There's a good chance the commit
  | history and authors may attest to that.
  | 
  | https://thehill.com/policy/technology/3928219-musk-was-denie...
 
| voz_ wrote:
| hmmm https://github.com/search?q=repo%3Atwitter%2Fthe-
| algorithm-m...
| 
| Twitter hmu if you need help trying Pytorch 2.0 ;)
 
| systemvoltage wrote:
| Astounding amount of cynicism here, so I'll say something
| positive: Transparency is undoubtly important, I'm glad we can
| see how all of this works and what sort of effort goes into
| building a social media system. It's licensed under GPL which is
| a bummer (would have preferred BSD) but it's better than nothing.
 
  | TMWNN wrote:
  | >Astounding amount of cynicism here
  | 
  | You can tell that those who rushed in to find something to
  | criticize can't, when they are reduced to making jokes about
  | coding stylistic conventions.
 
    | systemvoltage wrote:
    | Yea I mean, all discussions about Twitter have double
    | standards. If this was literally any other company, there
    | would be resounding praise.
 
  | sho_hn wrote:
  | Assuming anything in this codebase is worth reusing, I'm glad
  | it's GPL. It's a case where I'd like open-first to spread.
 
    | systemvoltage wrote:
    | GPL would be good if it is a self contained library. If
    | anyone would use it, it would be small portions of it, but
    | GPL makes it completely useless. You can't contaminate
    | anything with it. We'll stare at it, that's about it.
    | 
    | That makes me think, this is actually a good call. Twitter
    | can claim that they have complete transparency while not
    | allowing anyone to touch their code (because it is GPL).
    | "Anyone" being future competitors. If it was BSD licensed,
    | it'd be tremendously useful in building a Twitter competitor
    | (on paper, you still need network effects, I am just
    | spitballing to make a point).
 
      | sho_hn wrote:
      | It's only contaminating other components where you
      | incorporate or link it. If it's e.g. a microservice that's
      | fine.
 
        | systemvoltage wrote:
        | Good point about network calls & GPL licenseability.
 
| anderspitman wrote:
| I'm not opposed to social media feeds having complex
| recommendation algorithms. I just wish they allowed you to opt in
| to a reverse chronological feed of only people you follow, like
| RSS.
 
  | infogulch wrote:
  | Twitter has this now. The home page is split into two tabs:
  | "For you", the algorithmic feed, and "Following", the reverse
  | chronological feed of just who you follow.
 
    | conradfr wrote:
    | Is really "Following" the entire chronological feed? I feel I
    | miss tweets from people I follow that actually appears in the
    | "For You" tab.
 
      | anigbrowl wrote:
      | It's not. I follow a fairly small # of people (~500) and
      | getting people to reliably show up in is a long-running
      | problem. Following is not enough, you have to favorite or
      | somehow interact with them sufficiently to be sure of
      | seeing all their tweets. It's quite annoying.
 
        | cubefox wrote:
        | Are you sure? I didn't notice something like this.
 
        | anigbrowl wrote:
        | Very sure. I started noticing the issue a couple of years
        | ago (well before Musk's arrival) because I'd find myself
        | thinking 'I haven't seen anything from ____ in a while, I
        | should follow that person', only to discover that I was
        | already following them and their tweets were just not
        | showing up. New follows will generally pop up reliably,
        | but if someone has fallen out of your regular feed you
        | have to work to put them back into it.
        | 
        | Just now (as I am writing this comment) went to check on
        | someone I saw an interview with the other day, and sure
        | enough I am still following the person and they have been
        | tweeting a few times a day, but I haven't been seeing any
        | of it.
        | 
        | Other weird things are Twitter's habit of just
        | preemptively muting people (I'll sometimes wonder why a
        | person didn't reply and go back to reread a conversation,
        | only to discover that they _did_ reply; and conversely,
        | people that I have muted or blocked showing up in my
        | search results for a trending topic. Most of the people I
        | manually mute are  'influencers' who use software, staff,
        | or pure obsession to get in the first reply to
        | politicians and the like, a behavior I find insufferably
        | annoying even if I agree with their position.
        | 
        | I'm very interested in politics, but almost all my
        | mutes/blocks are people of somewhat-similar political
        | persuasion that Twitter assumes I would want to see, and
        | _insists_ on showing me despite my best efforts. I _want_
        | to keep tabs on the arguments of people I strenuously
        | disagree with, because I already know my own opinions and
        | don 't need validation. It's easier in some respects to
        | maintain a second account with an uber-conservative
        | persona and let the recommendation engine just feed it
        | with more of the same.
 
        | jeromegv wrote:
        | That was the great thing with the 3rd party client, I
        | could trust that all the people I followed I would get
        | their actual tweets. Every single one of them. There was
        | no also messing with it, no tweet "liked" by someone
        | else, etc. Who I followed is what I saw, nothing less,
        | nothing more.
        | 
        | Of course Elon banned those apps, so now I am on Mastodon
        | where I see 100% of the content that I want. Bonus is
        | that I can even follow many twitter users, through
        | Mastodon bot mirrors. And of course, no ads.
 
        | cubefox wrote:
        | It seems the only reason why people wouldn't show up then
        | is that they got shadow banned or at least have some
        | deboosting applied. So the algorithm thinks they are
        | spam?
 
    | BbzzbB wrote:
    | It always had it.
    | 
    | Edit: Why am I downvoted? It literally did, it even was named
    | as you'd expect it ("sort by latest" or something), tho the
    | location was less obvious as it was under the stars icon
    | above the feed.
 
    | LiquidPolymer wrote:
    | On my "following" tab (on the phone app) , I'm still getting
    | recommendations for bomb throwers I don't follow. Am I weird?
    | It's like an unhinged relative. Not pleasant.
    | 
    | Edit: I reversed "for you" and "following" in my original
    | reply.
 
      | EamonnMR wrote:
      | I sometimes get these as push notifications with my
      | username added to them.
 
        | valarauko wrote:
        | I find these notifications so confusing - at first glance
        | they look like DMs or mentions to me. I don't follow
        | these people, nor were they RT by anybody I know.
 
        | EamonnMR wrote:
        | Probably lots of engagement due to that confusion though.
 
    | madeofpalk wrote:
    | Twitter has _always_ had  "chronological timeline" behind a
    | confusing "sparkle" button (except for a brief period a few
    | months back where they removed it, or always defaulted to
    | back to algo timeline? and then restored it a week later)
    | 
    | They called it "Latest Tweets" https://web.archive.org/web/20
    | 200205092104/https://help.twit...
 
      | dmonitor wrote:
      | immediately getting rid of the sparkle button is one of the
      | few reasons i still have a small amount of faith in elon's
      | vision of twitter
 
        | Laaas wrote:
        | - Open sourcing this
        | 
        | - Lists as tabs
        | 
        | - _Heavily_ reduced spam
        | 
        | - Can look at Twitter without logging in
        | 
        | I'm likely missing some other obvious/uncontroversially
        | good changes.
        | 
        | I still use Twitter and plan to continue using it. I am
        | satisfied with most of his changes.
 
        | TaylorAlexander wrote:
        | The spam comment is interesting. I had a sharp uptick in
        | the number of spam messages after the acquisition.
 
        | dmix wrote:
        | My daily DM spam hit a peak last year than dropped in
        | recent months.
 
        | astrange wrote:
        | There's not any less spam. How could there be when he
        | laid off all the antispam people?
        | 
        | If you expand "see more tweets" sometimes there's normal
        | tweets hiding down there and sometimes it's a bot trying
        | to sell you guns.
 
        | madeofpalk wrote:
        | > - Lists as tabs
        | 
        | Twitter added this in 2019
        | 
        | https://www.theverge.com/2019/9/23/20880372/twitter-
        | lists-al...
        | 
        | > - Can look at Twitter without logging in
        | 
        | All the recent change was show non-logged in users the
        | Explore view which is... better than the previously
        | mostly nothing I guess.
        | 
        | Again, you've always been able to view tweets and browse
        | profiles _if linked to directly_ before Elon.
 
        | progmetaldev wrote:
        | Not having to login to Twitter is a great feature. I
        | don't have a Twitter account, but often see posts linked
        | elsewhere, and hated getting hit to login each time.
 
      | cubefox wrote:
      | That wasn't the same, clicking the sparkle button still
      | included a lot of recommended tweets. The only effect was
      | that it made the timeline chronological and included all
      | tweets from people you follow.
 
    | spike021 wrote:
    | It doesn't always stay on whatever you last used, though. I
    | mostly use Following but it always inevitably ends up back on
    | "For you".
 
  | [deleted]
 
| rvz wrote:
| If Twitter was 'dead' why on earth are we still talking so much
| about this blue bird site?
| 
| It looks like once again these lot predicting that he won't open
| source the algorithm and are going to start eating their words
| again [0], just like they did around incorrectly predicting
| Twitter's _immediate_ collapse [1] and will look at the source
| code anyway and continue to talk about  "Twitter" again.
| 
| If Twitter can open-source their algorithm, Why not TikTok?
| Either way, the bots are now going to have a very expensive time
| on Twitter.
| 
| [0] https://news.ycombinator.com/item?id=35213213
| 
| [1] https://news.ycombinator.com/item?id=33701371
 
  | anigbrowl wrote:
  | Are you kidding me, running a botnet is easier than it has been
  | in years if you're that way inclined. The amount of spam I see
  | has gone way up over the last 6 months.
 
| Reptur wrote:
| They didn't open source the data the censoring abusive, toxicity,
| and nsfw the algorithms check against, so I'd call it a partial
| open-sourcing.
 
| frob wrote:
| Well that was a giant nothing-burger. This seems to be your
| standard ranking stack. We find candidates based on who you
| follow, who they follow, who is trending, and what we think you
| like. We then rank them based on how likely you are to engage
| with them and continue to come back and give us money via our
| subscription service and ad views. We then try to remove spam and
| other negative experiences.
| 
| Where's the beef?
 
| evntdrvn wrote:
| it would be super interesting if when logged in to Twitter, you
| could take a look at your current calculated scores/weights for
| all the params that are part of these algorithms. Similar to the
| Netflix "Stats for nerds" menu...
 
| tcmart14 wrote:
| Repo has 1.5% rust code and no                 author_is_uwu
| 
| That is the biggest problem.
 
| inparen wrote:
| Issue list is growing rapidly for a repo created an hour ago.
 
  | ThalesX wrote:
  | Non-issues most of them:
  | 
  | - author_is_elon: the problem is his tweets suck. stop
  | recommending them.
  | 
  | - Include 'who viewed my profile' option in twitter
  | 
  | - Only one commit on repo
  | 
  | - How do I use it?
  | 
  | - Cool
  | 
  | - allow "AI" to tweet and like tweets on your behalf
  | 
  | - IMPORTANT: Guys please keep this place for real bugs and
  | contributions,
  | 
  | etc...
 
| Egoist wrote:
| Aaaand the issues turned into a shitpost
 
  | HeckFeck wrote:
  | In fairness they could save some RAM by rewriting it in Rust 6
  | or 7 times.
 
  | [deleted]
 
| ranboxtest wrote:
| April Fools prank at it's finest...
 
| thumbsup-_- wrote:
| The barebones ReadMe makes me feel this repository was open-
| sourced against the wish of engineers and with a top down
| directive
 
  | firstSpeaker wrote:
  | How so? More details and reasoning?
 
    | thumbsup-_- wrote:
    | Elon?
 
| [deleted]
 
| danso wrote:
| > _Twitter has several Candidate Sources that we use to retrieve
| recent and relevant Tweets for a user. For each request, we
| attempt to extract the best 1500 Tweets from a pool of hundreds
| of millions through these sources. We find candidates from people
| you follow (In-Network) and from people you don't follow (Out-of-
| Network)._
| 
| > _Today, the For You timeline consists of 50% In-Network Tweets
| and 50% Out-of-Network Tweets on average, though this may vary
| from user to user._
| 
| It would've been interesting to see what changes were made since
| Musk's takeover. As someone who followed 5,000+ users, I know I
| never saw a tweet that wasn't either from nor retweeted by
| someone I followed -- e.g. I never saw those "[user you follow]
| liked [someone you don't follow] tweet"
| 
| 50%/50% in FYP seems to reflect my experience today -- which is
| much worse, to the point that I'll regularly switch to viewing by
| List b/c I miss seeing people who I want to read.
| 
| I wonder how much testing and analysis went into deciding on the
| 50/50 ratio -- e.g. how does it impact user engagement and
| behavior. Because it sounds like an easy round value that you'd
| land on when thinking "users should be pushed out of their
| bubbles"
 
  | cubefox wrote:
  | Perhaps if you did follow so many people they got drowned out,
  | but with substantially fewer following, those recommended
  | tweets were a big part of what I saw. Especially in the last
  | year or so before Musk took over: Twitter went a lot more
  | aggressive and didn't just show tweets which people you follow
  | "liked", but also other tweets, which the algorithm somehow
  | determined you might like, which was often wrong, and,
  | moreover, so frequent that it made a big portion of the
  | timeline. The "following" tab fixed this problem.
 
    | danso wrote:
    | Yep, having had created a few throwaway accounts I definitely
    | got a sense of how the algorithm compensated for the majority
    | of users who aren't super active. And it makes sense -- most
    | new users aren't going to want to spend account creation
    | picking 50 accounts to follow.
    | 
    | But if someone has hit the follow button 1,000+ times, it's
    | reasonable to have some faith that they've seen a lot of
    | tweets and know what they want. Showing a few out-of-network
    | tweets seems reasonable (I got enough as it is through
    | followings' retweets). But 50% of a feed that already can't
    | fit tweets from thousands of followings just feels like shit.
    | 
    | The worst part is that the share of in-network tweets seems
    | to be highly concentrated to the last 10 or so people I most
    | recently interacted with, e.g. seeing the same user over and
    | over just because I liked one of their tweets the other day.
    | Which makes sense to save on computation costs, but it's
    | pushed me into a much tighter bubble than I ever had when the
    | timeline wasn't so out-of-network focused.
 
      | cubefox wrote:
      | Is the "following" tab an option for you? Or what would you
      | like the "for you" tab to do? Filter out heavy tweeters?
      | Prioritize more popular tweets?
 
        | danso wrote:
        | The Following tab limits to my followers, but in reverse
        | chrono order. This is also not desirable b/c it limits to
        | whoever's posting whenever I've happened to check the
        | feed.
        | 
        | I'd like to see interesting tweets from a few hours ago,
        | and not just Australian tweets when I'm up late at night.
 
  | coldcode wrote:
  | A year ago my account with 5700 followers got an average of
  | 3000 impressions per post (art). Today it's only 200-500. It
  | mentions their fanout system was replaced by something new, not
  | sure when or if thats in the drop, but my impression count
  | dropped around April-May last year. Clearly something decided
  | my posts should not shown to my followers very often.
 
    | yeahsure22 wrote:
    | [dead]
 
___________________________________________________________________
(page generated 2023-03-31 23:00 UTC)