Replace tokens with guest accounts, swap endpoints

[zedeus]

This replaces the old method of using guest tokens with using "guest accounts".
For more information see this comment #983 (comment)

Live now on nitter.net for testing. If you'd like to test and don't have any guest accounts, feel free to reach out to me on Matrix. A way to acquire guest accounts will be build and shared very soon.

[dawnerd]

RSS returns forbidden, was this disabled?

[michaelskyba]

RSS returns forbidden, was this disabled?

#937

[DrSocket]

hi @zedeus, this is amazing, just a question, how is guest_accounts.json formatted?

[zedeus]

It's a JSON array of the raw object you get from guest account flow, looks like this (you can use this one for testing if you want):

[{"user":{"id":1693553468835831800,"id_str":"1693553468835831808","name":"Open App User","screen_name":"_LO_082193WA35n","user_type":"Soft"},"next_link":{"link_type":"subtask","link_id":"next_link","subtask_id":"OpenAppFlowStartAccountSetupOpenLink"},"oauth_token":"1693553468835831808-TBis5qLp17jenKVyu9jeRjnTcTKjpL","oauth_token_secret":"BvbmNeJucYaUAWoEzrSUzTjkWxifoYaqAnjXTevL6ICxU","attribution_event":"signup"}]

[DrSocket]

Great! thanks a lot, it's the format from BANKA's script in an array.

[DrSocket]

I'm getting an error on compilation with this branch though, do you know anything about it?
parserutils.nim(39, 13) Error: undeclared field: 'notNull' for type packedjson.JsonNode [type declared in /Users/user/.nimble/pkgs/packedjson-#9e6fbb6/packedjson.nim(115, 3)]

[zedeus]

This happens with Nim versions older than 1.6.14, I'll fix it soon, for now you can work around it by updating to 1.6.14 or preferably 2.0

[pbregener]

For the record: I successfully built a Docker image for aarch64 from this branch and it works like a charm :)

I changed the base image to use alpine 3.18 which has Nim 1.6.14

[DrSocket]

Sorry for the long post, couple of questions just for my own learning:

1. Do you using Android Studio to reverse engineer the android endpoints? If not how do you recommend going about it?
2. Would it be complicated to add an option to use account credentials from a json instead of guest account in case they close the guest accounts endpoint and just have a lot of them. I'm still pretty rusty with nim code, maybe you have recommendations about a good place so start learning so I can actually help contributing to the project (currently use C,C++,python,TS).
   i.e. (this account is blocked now) ->
   {"cookie":"_twitter_sess=xyz; kdt=xyz; auth_token=xyz; ct0=xyz; personalization_id="xyz"; twid="xyz, "x-csrf-token": "xyz", "authorization": "Bearer AAAAAAAAAAAAAAAAAAAAAxyz", "x-twitter-auth-type": "OAuth2Session", "x-twitter-active-user": "yes"}

[zedeus]

I used a virtual machine to listen to network requests, but most of the reverse engineering work was done by other folks.

use account credentials from a json instead of guest account in case they close the guest accounts endpoint

I'm not sure what you mean since that's how it already works, storing guest accounts in a file. No endpoints are being used beyond the initial creation of the account, which expires after 30 days. When you say blocked, do you mean you get an error code 88 (rate limit exceeded)? That block goes away 24 hours later.

[DrSocket]

Would you mind sharing what the kind of vm you use for this? something like wireshark or mitmproxy?
That's an actual burner account, not a guest account. It would be great if we could switch from guest to burner accounts like PrivacyDevel uses. In case it gets shut down again just have a few burners to still be able to use Nitter while finding a new solution.
By locked I mean:
"Twitter responded with an error: AuthorizationError: Authorization: Denied by access control: To protect our users from spam and other malicious activity, this account is temporarily locked. Please log in to https://twitter.com to unlock your account."

[zedeus]

I used the free trial from Corellium to run Twitter and inspect network logs. I tried running BlissOS via qemu to no avail first.

I just noticed the thing you posted is a normal account, didn't look close enough. What I said still stands for guest accounts, they don't get blocked though just limited for 24 hours. The problem with burner accounts is you need a lot for any meaningfully big instance, so it's not really feasible especially since they just get locked. It's extremely easy to get guest accounts, just takes a lot of proxies which are easy to get.

[DrSocket]

Great thank you!
Yes the problem with guest accounts solution is that twitter can still kill this method no?

[zedeus]

Indeed, but that's the case with every solution. With accounts they can start banning or further restricting, there is no perfect solution.

[felipefalkiner]

How to upgrade to this version?
Git clone this branch and run the following commands again?

$ nimble build -d:release
$ nimble scss
$ nimble md

[zedeus]

Correct, but you also need to get one or more guest accounts, otherwise nothing will work

[dawnerd]

I have this running on my instance and so far it's running great, haven't seen any issues so far. Getting gust tokens definitely the trickiest part, but being as Im just running a private/relay instance one guest user should cover it.

Thanks for getting this put together!

[somini]

Is it possible to run this branch with an actual account? What are the tokens I need to extract from the real account to create the JSON file?

[zedeus]

It might be possible but I haven't tried, there's no code in this branch for regular accounts as they use cookies and bearer tokens etc instead of oauth.

[somini]

What's the status of this? Can nitter itself at least run the guest script itself once a day to replenish the tokens?

[y0nei]

I hacked myself a small script to auto generate guest accounts with the guest_tokens.sh script and (properly) append created entries to a guest_accounts.json file between [ and ] characters.
guest_tokens.sh:

#!/bin/bash
# Grab Twitter guest account tokens for use with Nitter.

guest_token=$(curl -s -XPOST https://api.twitter.com/1.1/guest/activate.json -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F' | jq -r '.guest_token')

flow_token=$(curl -s -XPOST 'https://api.twitter.com/1.1/onboarding/task.json?flow_name=welcome&api_version=1&known_device_token=&sim_country_code=us' \
  -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F' \
  -H 'Content-Type: application/json' \
  -H 'User-Agent: TwitterAndroid/9.95.0-release.0 (29950000-r-0) ONEPLUS+A3010/9 (OnePlus;ONEPLUS+A3010;OnePlus;OnePlus3;0;;1;2016)' \
  -H 'X-Twitter-API-Version: 5' \
  -H 'X-Twitter-Client: TwitterAndroid' \
  -H 'X-Twitter-Client-Version: 9.95.0-release.0' \
  -H 'OS-Version: 28' \
  -H 'System-User-Agent: Dalvik/2.1.0 (Linux; U; Android 9; ONEPLUS A3010 Build/PKQ1.181203.001)' \
  -H 'X-Twitter-Active-User: yes' \
  -H "X-Guest-Token: ${guest_token}" \
  -d '{"flow_token":null,"input_flow_data":{"country_code":null,"flow_context":{"start_location":{"location":"splash_screen"}},"requested_variant":null,"target_user_id":0},"subtask_versions":{"generic_urt":3,"standard":1,"open_home_timeline":1,"app_locale_update":1,"enter_date":1,"email_verification":3,"enter_password":5,"enter_text":5,"one_tap":2,"cta":7,"single_sign_on":1,"fetch_persisted_data":1,"enter_username":3,"web_modal":2,"fetch_temporary_password":1,"menu_dialog":1,"sign_up_review":5,"interest_picker":4,"user_recommendations_urt":3,"in_app_notification":1,"sign_up":2,"typeahead_search":1,"user_recommendations_list":4,"cta_inline":1,"contacts_live_sync_permission_prompt":3,"choice_selection":5,"js_instrumentation":1,"alert_dialog_suppress_client_events":1,"privacy_options":1,"topics_selector":1,"wait_spinner":3,"tweet_selection_urt":1,"end_flow":1,"settings_list":7,"open_external_link":1,"phone_verification":5,"security_key":3,"select_banner":2,"upload_media":1,"web":2,"alert_dialog":1,"open_account":2,"action_list":2,"enter_phone":2,"open_link":1,"show_code":1,"update_users":1,"check_logged_in_account":1,"enter_email":2,"select_avatar":4,"location_permission_prompt":2,"notifications_permission_prompt":4}}' | jq -r .flow_token)

curl -s -XPOST 'https://api.twitter.com/1.1/onboarding/task.json' \
  -H 'Authorization: Bearer AAAAAAAAAAAAAAAAAAAAAFXzAwAAAAAAMHCxpeSDG1gLNLghVe8d74hl6k4%3DRUMF4xAQLsbeBhTSRrCiQpJtxoGWeyHrDb5te2jpGskWDFW82F' \
  -H 'Content-Type: application/json' \
  -H 'User-Agent: TwitterAndroid/9.95.0-release.0 (29950000-r-0) ONEPLUS+A3010/9 (OnePlus;ONEPLUS+A3010;OnePlus;OnePlus3;0;;1;2016)' \
  -H 'X-Twitter-API-Version 5' \
  -H 'X-Twitter-Client: TwitterAndroid' \
  -H 'X-Twitter-Client-Version: 9.95.0-release.0' \
  -H 'OS-Version: 28' \
  -H 'System-User-Agent: Dalvik/2.1.0 (Linux; U; Android 9; ONEPLUS A3010 Build/PKQ1.181203.001)' \
  -H 'X-Twitter-Active-User: yes' \
  -H "X-Guest-Token: ${guest_token}" \
  -d "{\"flow_token\":\"${flow_token}\",\"subtask_inputs\":[{\"open_link\":{\"link\":\"next_link\"},\"subtask_id\":\"NextTaskOpenLink\"}],\"subtask_versions\":{\"generic_urt\":3,\"standard\":1,\"open_home_timeline\":1,\"app_locale_update\":1,\"enter_date\":1,\"email_verification\":3,\"enter_password\":5,\"enter_text\":5,\"one_tap\":2,\"cta\":7,\"single_sign_on\":1,\"fetch_persisted_data\":1,\"enter_username\":3,\"web_modal\":2,\"fetch_temporary_password\":1,\"menu_dialog\":1,\"sign_up_review\":5,\"interest_picker\":4,\"user_recommendations_urt\":3,\"in_app_notification\":1,\"sign_up\":2,\"typeahead_search\":1,\"user_recommendations_list\":4,\"cta_inline\":1,\"contacts_live_sync_permission_prompt\":3,\"choice_selection\":5,\"js_instrumentation\":1,\"alert_dialog_suppress_client_events\":1,\"privacy_options\":1,\"topics_selector\":1,\"wait_spinner\":3,\"tweet_selection_urt\":1,\"end_flow\":1,\"settings_list\":7,\"open_external_link\":1,\"phone_verification\":5,\"security_key\":3,\"select_banner\":2,\"upload_media\":1,\"web\":2,\"alert_dialog\":1,\"open_account\":2,\"action_list\":2,\"enter_phone\":2,\"open_link\":1,\"show_code\":1,\"update_users\":1,\"check_logged_in_account\":1,\"enter_email\":2,\"select_avatar\":4,\"location_permission_prompt\":2,\"notifications_permission_prompt\":4}}" | jq -r '.subtasks[0]|if(.open_account) then .open_account else empty end'

generate_guest_accounts.sh:

#!/bin/bash

# Run script and capture its output
output=$(./guest_tokens.sh)

# Check if the output is not empty
if [ -n "$output" ]; then
  # Check if the JSON file exists
  if [ ! -f guest_accounts.json ]; then
    # If the file does not exist, create it with an opening square bracket
    echo "[" > guest_accounts.json
  else
    # If the file already exists, remove the trailing square bracket
    sed -i '$ d' guest_accounts.json
    # Add a comma between entries
    echo "," >> guest_accounts.json
  fi

  # Append the output to the JSON file and add closing square bracket
  echo $output >> guest_accounts.json
  echo "]" >> guest_accounts.json

  # Format the JSON file using jq
  jq . guest_accounts.json > output-formatted.json
  mv output-formatted.json guest_accounts.json

  # Print a success message
  echo "Entry appended to and formatted in json file."
else
  echo "No output from script. Nothing appended to json file."
fi

Now generated entries should be appended while maintaining proper JSON syntax, I also format the whole file with jq for readability.
NOTE: jq, sed and curl are required to be installed on your system.

Finally you can use crontab -e to schedule execution of this script to your liking in order to fully automate the process of account generation:

# Every day at 12AM
0 0 * * * cd $HOME/Docker/nitter && ./generate_guest_accounts.sh

---

Here is a sample script used for testing in order to simulate what generate_tokens.sh would do:

#!/bin/bash

# Generate a JSON block with a random user ID
generate_json_entry() {
  user_id=$((RANDOM % 1000000))
  cat <<EOF
{
  "user": {
    "id": $user_id
  }
}
EOF
}

generate_json_entry

[zedeus]

@y0nei JSONL is supported, you don't need to do all that. Just append each json object on a new line and name the file .jsonl instead of .json.

[somini]

It seems the script for getting guest accounts it broken now. And maybe they expired all those credentials, I only get what's already cached in my instance.

[y0nei]

It seems the script for getting guest accounts it broken now. And maybe they expired all those credentials, I only get what's already cached in my instance.

Same issue for me, I'm unable to generate any new guest accounts, all of my current ones got "expired" and I can only view what is cached on my instance.

[nyanpasu64]

Oddly I can view posts in Nitter again with my old tokens, but the guest token generation script still returns undefined (possibly it's been less than a day since my last failed guest token generation during the Nitter outage?). Let's see how long until Twitter kills less manipulative frontends again.

Even during the outage, I could view user profiles and old tweets, but expanding any tweet resulted in an error.

[somini]

Can confirm, the token script fails, but I can read Nitter again, mostly. Seems hit and miss, the RSS feeds show the tweet, but I can't see the tweet responses for example.