SCREWY SCRIPTS

It's funny how I have ideas for posts that I note down but never 
get around to writing, then someone else writes about the exact 
same topic. I was going to write a retrospective post around the 
19th of Oct., which is when I started writing this Phlog (though it 
wasn't online yet at that stage), but it was going to be a long 
post and I haven't wanted to spend that sort of time on it. But the 
other day I saw that Undo has written a similar sort of thing:
gopher://sdf.org:70/0/users/undo/j058

But I've still been spending way too much time at the computer 
lately so I won't start writing that now. Actually he talks about 
not finishing projects. Of course in that regard I cheat a lot by 
describing in the ideas section projects that I don't actually plan 
to start in the first place, but one that has caught me lately is 
that internet client thing. I haven't talked about it much since 
describing the idea for the same reason, but I have set it up. Not 
actually on the Atomic Pi because that only arrived a couple of 
weeks ago so I set it up on a laptop instead. Except being left on 
all day it only survived for two days before overheating and being 
reduced to "system board error" BIOS beep codes, and my only other 
x86_64 laptop was the one that I had just finished setting up to 
replace my old 19 year old Thinkpad. So I kept using the old laptop 
for a bit longer. Then the Atomic Pi arrived, but it only boots 
UEFI, and I'd set up the system on the Micro SD card the old legacy 
BIOS way (which is all I've ever known to be honest). So yesterday 
I finally set up a copy of the card with a manual legacy/UEFI dual 
boot (or whatever you call that) installation, and I'll try that 
out later.

No I won't because I've started babbling and before long I'll have 
been typing all afternoon. So shut up and get back on topic! 
Alright grumpy, I guess I'm actually just here to share some shell 
scripts. In spite of all my talk about being sick of messing about 
at computers (actually mainly as something to do in breaks from 
adding up expenses my the tax return, which I did finally get 
submitted), when I saw the Unix 1-liner Five Fridays Funfair I 
thought that was a great incentive to work on a bare-bones version 
of a YouTube browser that I've been intending to write ever since 
YouTube went Javascript-only a few months ago (I'm surprised nobody 
else on Gopher has been complaining about that - I bet they would 
if they'd been trying to browse it from a 19 year old Thinkpad with 
a 1GHz PIII CPU though!). Plus I missed out on the last Gopher 
event that I was going to participate in because that laptop is 
about the only portable computer that I have with a working battery 
and hefting it around the great outdoors for ROOPHLOC didn't seem 
too appealing. I did think about trying to use one of the Nintendo 
DS consoles that I've collected, but that was going to be quite 
tricky. Actually as of this week I have a box full of quite recent 
laptops including some little ones that would be quite well suited, 
but that's because out of desperation I'm going to try getting into 
the used laptop/parts business - which basically means becoming the 
nerdy equivalent of a used car salesman. Well I might borrow one 
later to, err, "test the keyboard", and make a late ROOPHLOC entry.

gopher://katolaz.net/1/U1FFF_2020/entries.gph

This was meant to be about scripts or something wasn't it? Oh yeah, 
so I tried to make a one-liner YouTube browser that processes the 
metadata output from youtube-dl, but it soon became obvious that it 
was going to be way over the 128 byte length limit. But I kept 
working on it anyway, adding features, and then I realised a way to 
cheat. There's no limit on the length of the example, so just 
define key parts of the script ahead of the one-liner that actually 
uses them all. It was still a little tricky getting that "main" 
function down to size, but I managed it with a byte to spare, and 
thought it was a good joke. I'm not sure whether "The Unix Wizardry 
Trustworthy Fellowship" really got the joke though, or didn't 
approve, because they just left out the example entirely. So it's 
pretty much useless as presented there. Hence I'll put the full 
version of the multi-line one-liner here, and a more practical 
version that runs as a script in the scripts section (later):

##################################################################
Youtube Browser in ELinks

ytb (){ [ "${1##*[!0-9]*}" ] && $YTBD --playlist-end $1 "$2"|$YTBE|$YTBS || $YTBD "$1"|$YTBE|$YTBS;elinks $YTBF;rm -f $YTBF; }

Usage:
ytb [MAX RESULTS] URL

Example:

 $ YTBD='youtube-dl -e -g --get-thumbnail --get-description --get-duration'
 $ YTBE='sed -e s/_webp// -e s/\.webp/.jpg/ -e s/.jpg\?.*$/.jpg/'
 $ YTBS='csplit -z -f /tmp/yttmp -b %03d.txt - /^[0-9:]*[0-5][0-9]$/1 {*}'
 $ YTBF='/tmp/yttmp*.txt'
 $ ytb (){ [ "${1##*[!0-9]*}" ] && $YTBD --playlist-end $1 "$2"|$YTBE|$YTBS || $YTBD "$1"|$YTBE|$YTBS;elinks $YTBF;rm -f $YTBF; }
 $ ytb 5 https://www.youtube.com/user/FILMAUSTRALIA

Comment: In ELinks: "<" ">" to change tabs/results, "W" to wrap
text. Enable option "document.plain.display_links" in ELinks config.
for download/viewable video/image URLs.

"less" could be used instead of ELinks, with navigation between
results using the ":n" and ":p"  commands.
Some X terminal programs can detect URLs by themselves.
##################################################################

It works with individual video links, playlists, user accounts, and 
worked with youtube-dl's search function except that seemed to 
break while I was actually working on this script (you used to be 
able to do eg. "ytb ytsearch5:tildeverse" for five videos about the 
tildeverse (maybe)), I just search DuckDuckGo with 
"site:youtube.com" instead, or video.google.com does still work 
without Javascript (though you can't disable the link redirects 
like you can with DuckDuckGo). Add the "-f" option to the 
youtube-dl command (YTBD) to specify the video format/quality you 
want so it only gives you one video download link. The numbers 
printed to the terminal are the lengths of the description files, 
printed by Csplit, they're handy as a progress indicator.

I also submitted some other simple one-liners that I've mentioned 
before.

And while we're here, of course this Internet Client thing has 
involved all sorts of minor scripting. The latest one was 
auto-login with telnet. First off: yes I want to use telnet - the 
whole point of this thing is to avoid needing software that becomes 
obsolete and SSH is an example of that which I've battled far too 
much with already. Also I do want to use passwords even though it's 
unencrypted because the most likely point of attack is from the 
internet, so at my router (running OpenWRT), and I think a hacker 
is more likely to try brute forcing usernames over telnet if they 
get in than sniff the network traffic for the password when it's 
sent. It's not much protection, against an attack that's unlikely 
anyway, I know, but it makes me happy.

There are examples on the internet of using Expect for auto-login 
with Telnet, so I did this at first and it worked fine. But I don't 
have Expect installed everywhere, and it needs TCL, which needs a 
list of other dependencies, and these are old systems without much 
HDD space to waste. I tried using Chat, but I didn't get very far 
with that. Then I found a suggestion on a forum that, over many 
hours of messing with stty and other things, eventually evolved 
into this:

##################################################################
#!/bin/sh
#Telnet auto-login

USER='[username]'
#Note: can't find a way to make letters "C" or "p" work in password.
PASSWORD='[password]'
HOST='[hostname / IP]'

localtty="`stty -g`"
stty raw -echo intr ^[ start ''
{ \
 sleep 0.5; \
 echo "$PASSWORD"; \
 sleep 0.2; \
 echo "stty \"$localtty\" &> /dev/null"; \
 cat; \
} | telnet -l "$USER" "$HOST" 2>&1 | cat
##################################################################

The C and p thing has just got me stumped. It always reads C as 
"interrupt" and p as "start" and I can't seem to change that no 
matter what I do with stty. In the end I meekly changed the 
password to avoid using those characters. Change the sleep commands 
to "sleep 1" if your sleep only understands integers.

It does print something like: '$ stty 
"2506:5:bf:8a3b:3:1c:7f:15:4:0:1:0:11:13:1a:0:12:f:17:16:0:0:2f:0:0:
0:0:0:0:0:0:0:0:0:0:0" &> /dev/null' on the terminal after log-in, 
but I don't think it's worth adding extra complexity to hide that. 
You can automatically run other commands the same way.

Here's the Expect version that doesn't take offence at any password 
character that you use:

##################################################################
#!/usr/local/bin/expect -f

set timeout 20
set host "[hostname / IP]"
set user "[username]"
set password "[password]"

spawn telnet -l $user $host
expect "Password: "
send "$password\r"
interact

exit
##################################################################

I don't use Telnet for launching graphical programs (the Xcellent 
part of the idea), but Rexec which simply includes a "-p 
[password]" argument. Rexec (from GNU inetutils - which I'm getting 
quite familiar with now) works very well for a program that nobody 
seems to want to make distro packages for. The main problem I've 
found is that I originally tested this with Dillo on my Pi Zero 
(which some may remember I couldn't run Firefox on), assuming that 
would be equivalent to Firefox as far a X goes. Nope, Firefox sends 
a rendered image of the whole page view, which is basically like 
uncompressed video when scrolling the page. It's mostly usuable 
with 10/100 Ethernet (it probably helps that my screen resolutions 
don't exceed 1024x768), but a bit too painful over my 802.11g WiFi, 
so I've got Ethernet cables strewn about now. There's more to say 
about that, but I'll leave it until I do a real follow-up post 
about the Internet Client. It'll probably be titled something like 
"Me and my Xcellent Ideas".

 - The Free Thinker

Oh almost forgot, here's everything I found on the web relating to 
YouTube browsing (I looked pretty hard for an existing solution 
that I liked, but I'm picky):

* yt - Command-line browser written in Python
 https://www.hecticgeek.com/yt-youtube-browser-ubuntu-linux/

* youtube-viewer - Perl script, Command line or GTK2/3, requires 
  API key from Google
 https://github.com/trizen/youtube-viewer
 
http://www.webupd8.org/2015/02/youtube-viewer-complete-youtube-client.html

* straw-viewer - Perl script, Command line or GTK3, no API key 
  required
 https://github.com/trizen/straw-viewer

* SMtube - Qt GUI, latest version probably uses youtube-dl back-end
 http://www.smtube.org/
 https://packages.debian.org/sid/smtube
 -Some sort of interface here: http://www.tonvid.com/index.php
  -Is it just a web-based proxy with a Qt browser reading from 
   localhost? Or from that website?
   -Just a browser for the website, and the site itself is 
    closed-source:
    https://sourceforge.net/p/smplayer/bugs/695/

The TONVID site might not be very reliable [indeed it hasn't been], 
but seems to work OK as a stop-gap.

* Dillo-ydl - youtube-dl interface for Dillo written in Python 
https://git.scuttlebot.io/%25oEzgqM849P03lR3tR7qwbjzw%2Fc%2BR%2Bv0oIoHyDgEDrbw%3D.sha256

* Invidious - web interface Youtube browser
  https://github.com/iv-org/invidious
  -Various online instances:
   https://instances.invidio.us/

RSS:

This is _much_ faster than youtube-dl on my internet connection, so 
should really try to use it instead.

https://www.youtube.com/feeds/videos.xml?user=FILMAUSTRALIA

- Unforunately doesn't work well with some RSS viewers. Nrss only 
  shows headings.
 - can be sort-of viewed in links, but image links not displayed.

YouTube is now using WebP images, and these are supplied by 
--get-thumbnails.
- However JPEGs still available as:
  https://i.ytimg.com/vi/[VIDEO ID]/hqdefault.jpg
 -So just use --get-id to grab the Video ID string and put it in.