Complete Computing Environment: Web Browsing

Table of Contents

Web Browsing using EWW and Qutebrowser

Web browsing is still hard; I want to use the Emacs-native EWW, but without native Javascript or anything that makes the modern "Web Apps" I use at work, well, work, I need a GUI browser. Right now my GUI browser of choice is Qutebrowser, a minimalist modal browser built on top of QtWebEngine.

(provide 'cce-browsers)

We can browse to anything that looks like a URL with C-x m.

(evil-leader/set-key (kbd "u") 'browse-url-at-point)

I use EWW as my default browser in Emacs; certain things, I need to jump out of EWW and in to a real browser with javascript; which I can do if I just hit & – it'll drop me in to a firefox session without a hassle.

There are certain sites that, despite my best abilities, I cannot make work in EWW. For these, I launch Firefox directly, instead of hitting a silly man in the middle.

(unless (version< emacs-version "24.4")
  (setq browse-url-browser-function
        '((".*uberinternal.*" . browse-url-generic)
          (".*docs.*google.*" . browse-url-generic)
          (".*spreadsheet.*google.*" . browse-url-generic)
          (".*drive.*google.*" . browse-url-generic)
          (".*google.*maps.*" . browse-url-generic)
          (".*login.uber.com.*" . browse-url-generic)
          ("." . browse-url-generic)))
  (setq shr-external-browser 'browse-url-generic)
  (setq browse-url-generic-program (executable-find "qutebrowser")))

EWW

I use this advice to rename all of the EWW buffers based on their <title> elements.

(unless (version< emacs-version "24.4")
  (defadvice eww-tag-title (after rrix/eww-rename-buffer-ad (cont))
    "Update EWW buffer title with new page load."
    (let ((eww-current-title
           (if (version< emacs-version "25.0")
               eww-current-title
             (plist-get eww-data :title))))
      (rename-buffer (format "*eww : %s *" (cce/str-chomp eww-current-title)) t)))
  (ad-activate 'eww-tag-title))

Always open links in new buffers.

(setq browse-url-new-window-flag t)

Part of my Expansion technique is to just open a shitload of browser buffers throughout the day and pick through them in my IDLE times. EWW has basically no session support like emacs-w3m does, which is actually Okay given that that didn't serialize out to any format which I can sync between my workstations easily. Instead, I've crafted this function which uses my W buffer capture template to dump every EWW buffer to my refile.

(install-pkgs '(el-pocket))
(load-library "el-pocket")
(defun rrix/capture-all-eww-buffers ()
  (interactive)
  (map 'list (lambda (b)
               (with-current-buffer b
                 (when (eq major-mode 'eww-mode)
                   (let ((url (cond
                               ((version< emacs-version "25.0")
                                eww-current-url)
                               ((version< "25.0" emacs-version)
                                (plist-get eww-data :url))))
                         (title (cond
                                 ((version< emacs-version "25.0")
                                  eww-current-title)
                                 ((version< "25.0" emacs-version)
                                  (plist-get eww-data :title)))))
                     (if (y-or-n-p (format "Capture %s (%s)? " title url))
                         (progn
                           (el-pocket-authorize)
                           (el-pocket-add url))
                       (kill-this-buffer-y-or-n))))))
       (buffer-list)))

Having a global history is an incredibly useful thing to have but EWW doesn't give me one of these out of the box; I advise this in after eww-render like so:

(unless (version< emacs-version "24.4")
  (defvar rrix/eww-global-history '())
  (defadvice eww-render (after rrix/eww-global-history-ad ())
    "Add EWW history to a global history variable"
    (push (list :url (if(version< emacs-version "25.0")
                         eww-current-url
                       (plist-get eww-data :url))
                :title (if(version< emacs-version "25.0")
                           eww-current-title
                         (plist-get eww-data :title))
                :point (point)
                :date (format-time-string "%FT%T%z"))
          rrix/eww-global-history))
  (ad-deactivate 'eww-render)
  (add-to-list 'savehist-additional-variables 'rrix/eww-global-history))

I'll need to find some way to load that in to eww-list-histories, though… This is close, but it doesn't work since I'm not loading the full DOM, source and text in to the variable.

(defun rrix/list-eww-histories ()
  ""
  (interactive)
  (with-current-buffer (get-buffer-create "*eww-global-history*")
    (setq-local eww-history rrix/eww-global-history)
    (eww-list-histories)))

eww-lnum is a package that lets you follow any URL in an eww buffer with a keybinding, similar to Avy

(unless (version< emacs-version "24.4")
  (install-pkgs '(eww-lnum))
  (eval-after-load 'eww-mode (lambda ()
                               (define-key eww-mode-map "f" 'eww-lnum-follow))))

Qutebrowser

Qutebrowser is sort of similar to UZBL but it's running a decently modern web-engine and requires less nuts and bolts; you can use it without configuration, or you can tweak it to meet your needs. The biggest tweak I make is to disable all the tab-related features, and only use windows. My Window Manager handles window management well enough, I don't need another layer on top.

I install qutebrowser from source, since the 0.10.1 stuff with QWebEngine is pretty reliable. Also, we install openh264 and its gstreamer1 plugin so that things like YouTube work out of the box.

sudo dnf config-manager --set-enabled fedora-cisco-openh264
rpm-install gstreamer1-plugin-openh264 mozilla-openh264

if [ -d /home/rrix/Code/qutebrowser ]; then
  pushd /home/rrix/Code/qutebrowser
  git pull
  popd
else
  git clone https://github.com/qutebrowser/qutebrowser /home/rrix/Code/qutebrowser
fi

rpm-install python3-qt5 python3-qt5-webengine
apt-install python3-pyqt5 python3-pyqt5.qtwebkit python3-pyqt5.qtquick
pushd /home/rrix/Code/qutebrowser
sudo python3 setup.py install
popd

mkdir -p ~/.config/qutebrowser/
mkdir -p ~/.local/share/qutebrowser/
link-this $PWD/out/qutebrowser.conf ~/.config/qutebrowser/qutebrowser.conf
link-this $PWD/out/qutebrowser-keys.conf ~/.config/qutebrowser/keys.conf
link-this $PWD/out/qutebrowser-pocket-userscript ~/.local/share/qutebrowser/userscripts

Configuration

A lot of this is out of the box, added here for expliciteness. Of note, here, is Qutebrowser's ability to disable any tab-bar setup and only use windows for management, which is what I want when I have Awesome WM managing those things for me.

[general]
ignore-case = smart
startpage = file:///tmp/agenda.html
yank-ignored-url-parameters = ref,utm_source,utm_medium,utm_campaign,utm_term,utm_content
default-open-dispatcher =
default-page = ${startpage}
auto-search = naive
auto-save-config = true
auto-save-interval = 15000
editor = emacsclient -tc "{}"
editor-encoding = utf-8
private-browsing = false
developer-extras = true
print-element-backgrounds = true
xss-auditing = false
site-specific-quirks = true
default-encoding =
new-instance-open-target = window
new-instance-open-target.window = last-focused
log-javascript-console = debug
save-session = true
session-default-name = default
url-incdec-segments = path,query

[ui]
zoom-levels = 25%,33%,50%,67%,75%,90%,100%,110%,125%,150%,175%,200%,250%,300%,400%,500%
default-zoom = 100%
downloads-position = top
status-position = bottom
message-timeout = 2000
message-unfocused = false
confirm-quit = never
zoom-text-only = false
frame-flattening = false
user-stylesheet =
hide-scrollbar = true
css-media-type =
smooth-scrolling = false
remove-finished-downloads = -1
hide-statusbar = false
statusbar-padding = 1,1,0,0
window-title-format = {perc}{title}
modal-js-dialog = false
hide-wayland-decoration = false
keyhint-blacklist =
prompt-radius = 8
prompt-filebrowser = true

[network]
do-not-track = true
accept-language = en-US,en
referer-header = same-domain
user-agent =
proxy = system
proxy-dns-requests = true
ssl-strict = ask
dns-prefetch = true
custom-headers =
netrc-file =

[completion]
show = always
download-path-suggestion = path
timestamp-format = %Y-%m-%d
height = 50%
cmd-history-max-items = 100
web-history-max-items = 1000
quick-complete = true
shrink = false
scrollbar-width = 12
scrollbar-padding = 2

[input]
timeout = 500
partial-timeout = 5000
insert-mode-on-plugins = false
auto-leave-insert-mode = true
auto-insert-mode = false
forward-unbound-keys = auto
spatial-navigation = false
links-included-in-focus-chain = true
rocker-gestures = false
mouse-zoom-divider = 512

[tabs]
tabs-are-windows = true
background-tabs = false
select-on-remove = next
new-tab-position = next
new-tab-position-explicit = last
last-close = ignore
show = multiple
show-switching-delay = 800
wrap = true
movable = true
close-mouse-button = middle
position = top
show-favicons = true
width = 20%
indicator-width = 3
title-format = {index}: {title}
title-alignment = left
mousewheel-tab-switching = true
padding = 0,0,5,5
indicator-padding = 2,2,0,4

[storage]
download-directory =
prompt-download-directory = true
remember-download-directory = true
maximum-pages-in-cache =
object-cache-capacities =
offline-storage-default-quota =
offline-web-application-cache-quota =
offline-storage-database = true
offline-web-application-storage = true
local-storage = true
cache-size = 52428800

[content]
allow-images = true
allow-javascript = true
allow-plugins = false
webgl = true
css-regions = true
hyperlink-auditing = false
geolocation = ask
notifications = ask
media-capture = ask
javascript-can-open-windows-automatically = false
javascript-can-close-windows = false
javascript-can-access-clipboard = false
ignore-javascript-prompt = false
ignore-javascript-alert = false
local-content-can-access-remote-urls = false
local-content-can-access-file-urls = true
cookies-accept = no-3rdparty
cookies-store = true
host-block-lists = https://www.malwaredomainlist.com/hostslist/hosts.txt,http://someonewhocares.org/hosts/hosts,http://winhelp2002.mvps.org/hosts.zip,http://malwaredomains.lehigh.edu/files/justdomains.zip,https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
host-blocking-enabled = true
host-blocking-whitelist = piwik.org
enable-pdfjs = false

[hints]
border = 1px solid #E3BE23
mode = letter
chars = asdfghjkl
min-chars = 1
scatter = true
uppercase = false
dictionary = /usr/share/dict/words
auto-follow = unique-match
auto-follow-timeout = 0
next-regexes = \bnext\b,\bmore\b,\bnewer\b,\b[>→≫]\b,\b(>>|»)\b,\bcontinue\b
prev-regexes = \bprev(ious)?\b,\bback\b,\bolder\b,\b[<←≪]\b,\b(<<|«)\b
find-implementation = python
hide-unmatched-rapid-hints = true

[searchengines]
DEFAULT = https://duckduckgo.com/?q={}
ub = https://ubunny.uberinternal.com/ubunny/?q={}
gg = https://encrypted.google.com/search?hl=en&q={}

[aliases]
# `:qtb` to open qutebrowsers website:
# `qtb = open https://www.qutebrowser.org/`

[colors]
completion.fg = white
completion.bg = #333333
completion.alternate-bg = #444444
completion.category.fg = white
completion.category.bg = qlineargradient(x1:0, y1:0, x2:0, y2:1, stop:0 #888888, stop:1 #505050)
completion.category.border.top = black
completion.category.border.bottom = ${completion.category.border.top}
completion.item.selected.fg = black
completion.item.selected.bg = #e8c000
completion.item.selected.border.top = #bbbb00
completion.item.selected.border.bottom = ${completion.item.selected.border.top}
completion.match.fg = #ff4444
completion.scrollbar.fg = ${completion.fg}
completion.scrollbar.bg = ${completion.bg}
statusbar.fg = white
statusbar.bg = grey
statusbar.fg.insert = ${statusbar.fg}
statusbar.bg.insert = darkgreen
statusbar.fg.command = ${statusbar.fg}
statusbar.bg.command = ${statusbar.bg}
statusbar.fg.caret = ${statusbar.fg}
statusbar.bg.caret = purple
statusbar.fg.caret-selection = ${statusbar.fg}
statusbar.bg.caret-selection = #a12dff
statusbar.progress.bg = white
statusbar.url.fg = ${statusbar.fg}
statusbar.url.fg.success = white
statusbar.url.fg.success.https = lime
statusbar.url.fg.error = orange
statusbar.url.fg.warn = yellow
statusbar.url.fg.hover = aqua
tabs.fg.odd = white
tabs.bg.odd = grey
tabs.fg.even = white
tabs.bg.even = darkgrey
tabs.fg.selected.odd = white
tabs.bg.selected.odd = black
tabs.fg.selected.even = ${tabs.fg.selected.odd}
tabs.bg.selected.even = ${tabs.bg.selected.odd}
tabs.bg.bar = #555555
tabs.indicator.start = #0000aa
tabs.indicator.stop = #00aa00
tabs.indicator.error = #ff0000
tabs.indicator.system = rgb
hints.fg = black
hints.bg = qlineargradient(x1:0, y1:0, x2:0, y2:1, stop:0 rgba(255, 247, 133, 0.8), stop:1 rgba(255, 197, 66, 0.8))
hints.fg.match = green
downloads.bg.bar = black
downloads.fg.start = white
downloads.bg.start = #0000aa
downloads.fg.stop = ${downloads.fg.start}
downloads.bg.stop = #00aa00
downloads.fg.system = rgb
downloads.bg.system = rgb
downloads.fg.error = white
downloads.bg.error = red
webpage.bg = white
keyhint.fg = #FFFFFF
keyhint.fg.suffix = #FFFF00
keyhint.bg = rgba(0, 0, 0, 80%)
messages.fg.error = white
messages.bg.error = red
messages.border.error = #bb0000
messages.fg.warning = white
messages.bg.warning = darkorange
messages.border.warning = #d47300
messages.fg.info = white
messages.bg.info = black
messages.border.info = #333333
prompts.fg = white
prompts.bg = darkblue
prompts.selected.bg = #308cc6

[fonts]
_monospace = "Fira Mono", Terminus, Monospace, "DejaVu Sans Mono", Monaco, "Bitstream Vera Sans Mono", "Andale Mono", "Courier New", Courier, "Liberation Mono", monospace, Fixed, Consolas, Terminal
completion = 12pt ${_monospace}
completion.category = bold ${completion}
tabbar = 8pt ${_monospace}
statusbar = 12pt ${_monospace}
downloads = 12pt ${_monospace}
hints = bold 13px ${_monospace}
debug-console = 8pt ${_monospace}
web-family-standard =
web-family-fixed =
web-family-serif =
web-family-sans-serif =
web-family-cursive =
web-family-fantasy =
web-size-minimum =
web-size-minimum-logical =
web-size-default =
web-size-default-fixed =
keyhint = 12pt ${_monospace}
messages.error = 12pt ${_monospace}
messages.warning = 12pt ${_monospace}
messages.info = 12pt ${_monospace}
prompts = 12pt sans-serif

Key bindings

These are mostly the Out of Box bindings, with all the tab bindings ripped out.

[!normal]

leave-mode
    <escape>
    <ctrl-[>

[normal]
# Keybindings for normal mode.

clear-keychain ;; search ;; fullscreen --leave
    <escape>

set-cmd-text -s :open
    o

set-cmd-text :open {url:pretty}
    go

set-cmd-text -s :open -t
    O

set-cmd-text :open -t -i {url:pretty}
    gO

set-cmd-text -s :open -b
    xo

set-cmd-text :open -b -i {url:pretty}
    xO

set-cmd-text -s :open -w
    wo

set-cmd-text :open -w {url:pretty}
    wO

set-cmd-text /
    /

set-cmd-text ?
    ?

set-cmd-text :
    :

open -t
    ga
    <ctrl-t>

open -w
    <ctrl-n>

reload
    r
    <f5>

reload -f
    R
    <ctrl-f5>

back
    H
    <back>

back -w
    wh

forward
    L
    <forward>

forward -w
    wl

fullscreen
    <f11>

hint
    f

hint all tab
    F

hint all window
    wf

hint all hover
    ;h

hint images window
    ;i

hint links fill :open {hint-url}
    ;o

hint links fill :open -t -i {hint-url}
    ;O

hint links yank
    ;y

hint links yank-primary
    ;Y

hint --rapid links tab-bg
    ;r

hint --rapid links window
    ;R

hint links download
    ;d

hint inputs
    ;t

scroll left
    h

scroll down
    j

scroll up
    k

scroll right
    l

undo
    u
    <ctrl-shift-t>

scroll-perc 0
    gg

scroll-perc
    G

search-next
    n

search-prev
    N

enter-mode insert
    i

enter-mode caret
    v

enter-mode set_mark
    `

enter-mode jump_mark
    '

yank
    yy

yank -s
    yY

yank title
    yt

yank title -s
    yT

yank domain
    yd

yank domain -s
    yD

yank pretty-url
    yp

yank pretty-url -s
    yP

open -- {clipboard}
    pp

open -- {primary}
    pP

open -w -- {clipboard}
    wp

open -w -- {primary}
    wP

quickmark-save
    m

set-cmd-text -s :quickmark-load
    b

set-cmd-text -s :quickmark-load -w
    wb

bookmark-add
    M

set-cmd-text -s :bookmark-load
    gb

set-cmd-text -s :bookmark-load -t
    gB

set-cmd-text -s :bookmark-load -w
    wB

save
    sf

set-cmd-text -s :set
    ss

set-cmd-text -s :set -t
    sl

set-cmd-text -s :bind
    sk

zoom-out
    -

zoom-in
    +

zoom
    =

navigate prev
    [[

navigate next
    ]]

navigate prev -t
    {{

navigate next -t
    }}

navigate up
    gu

navigate up -w
    gU

navigate increment
    <ctrl-a>

navigate decrement
    <ctrl-x>

inspector
    wi

download
    gd

download-cancel
    ad

download-clear
    cd

view-source
    gf

set-cmd-text -s :buffer
    gt

enter-mode passthrough
    <ctrl-v>

quit
    <ctrl-q>

scroll-page 0 1
    <ctrl-f>

scroll-page 0 -1
    <ctrl-b>

scroll-page 0 0.5
    <ctrl-d>

scroll-page 0 -0.5
    <ctrl-u>

home
    <ctrl-h>

stop
    <ctrl-s>

print
    <ctrl-alt-p>

open qute:settings
    Ss

follow-selected
    <return>
    <ctrl-m>
    <ctrl-j>
    <shift-return>
    <enter>
    <shift-enter>

follow-selected -t
    <ctrl-return>
    <ctrl-enter>

repeat-command
    .

record-macro
    q

run-macro
    @

wq
    ZZ

spawn --userscript pocket
    gP

hint links spawn --userscript pocket
    gp

[insert]
open-editor
    <ctrl-e>

insert-text {primary}
    <shift-ins>

[hint]
follow-hint
    <return>
    <ctrl-m>
    <ctrl-j>
    <shift-return>
    <enter>
    <shift-enter>

hint --rapid links tab-bg
    <ctrl-r>

hint links
    <ctrl-f>

hint all tab-bg
    <ctrl-b>

[command]
command-history-prev
    <ctrl-p>

command-history-next
    <ctrl-n>

completion-item-focus prev
    <shift-tab>
    <up>

completion-item-focus next
    <tab>
    <down>

completion-item-focus next-category
    <ctrl-tab>

completion-item-focus prev-category
    <ctrl-shift-tab>

completion-item-del
    <ctrl-d>

command-accept
    <return>
    <ctrl-m>
    <ctrl-j>
    <shift-return>
    <enter>
    <shift-enter>

[prompt]
prompt-accept
    <return>
    <ctrl-m>
    <ctrl-j>
    <shift-return>
    <enter>
    <shift-enter>

prompt-accept yes
    y

prompt-accept no
    n

prompt-open-download
    <ctrl-x>

prompt-item-focus prev
    <shift-tab>
    <up>

prompt-item-focus next
    <tab>
    <down>

[command,prompt]

rl-backward-char
    <ctrl-b>

rl-forward-char
    <ctrl-f>

rl-backward-word
    <alt-b>

rl-forward-word
    <alt-f>

rl-beginning-of-line
    <ctrl-a>

rl-end-of-line
    <ctrl-e>

rl-unix-line-discard
    <ctrl-u>

rl-kill-line
    <ctrl-k>

rl-kill-word
    <alt-d>

rl-unix-word-rubout
    <ctrl-w>

rl-backward-kill-word
    <alt-backspace>

rl-yank
    <ctrl-y>

rl-delete-char
    <ctrl-?>

rl-backward-delete-char
    <ctrl-h>

[caret]

toggle-selection
    v
    <space>

drop-selection
    <ctrl-space>

enter-mode normal
    c

move-to-next-line
    j

move-to-prev-line
    k

move-to-next-char
    l

move-to-prev-char
    h

move-to-end-of-word
    e

move-to-next-word
    w

move-to-prev-word
    b

move-to-start-of-next-block
    ]

move-to-start-of-prev-block
    [

move-to-end-of-next-block
    }

move-to-end-of-prev-block
    {

move-to-start-of-line
    0

move-to-end-of-line
    $

move-to-start-of-document
    gg

move-to-end-of-document
    G

yank selection -s
    Y

yank selection
    y
    <return>
    <ctrl-m>
    <ctrl-j>
    <shift-return>
    <enter>
    <shift-enter>

scroll left
    H

scroll down
    J

scroll up
    K

scroll right
    L

Userscripts

  • Save to Pocket

    I am a pretty ardent user of Pocket1 and so being able to slurp up links from my browser in to Pocket is pretty important. My Kobo Aura One2 syncs with Pocket, so I can just dump articles on to it and read them while I'm on transit or trying to fall asleep.

    This relies on a program called pocket-cli to handle the actual adding, it's available from pypi.

    pip3 install --user pocket-cli
    
    if [[ ! -e ~/.pocket-config ]]; then
        ~/.local/bin/pocket-cli configure
    fi
    
    /home/rrix/.local/bin/pocket-cli add -u "$QUTE_URL"
    

    Once it's installed, we can bind it to gP and gp to Pocket either the current page, or to spawn a link-chooser to grab a link that's on the page.

    spawn --userscript pocket
        gP
    
    hint links spawn --userscript pocket
        gp
    

Footnotes:

Author: Ryan Rix

Created: 2017-07-02 Sun 11:27

Validate XHTML 1.0