Kickass Systems

Table of Contents


My friends and I are building a VPN Darknet. We're stringing together TINC nodes, Kerberos and LDAP to create a base-layer for self-hosted services ranging from code hosting to IRC to Diaspora to Ceph.

I wrote a lot about this in my Off the Grid blog post. This is basically an excuse to build out a fun distributed network and build distributed services on top of them.



DONE Internal-use wiki

INPROGRESS Internal-use cgit/gitlab instance

  • DONE Fix Gitolite clobbering of permissions
  • DONE Fix Subdirectory layout
  • NEXT Build a static site generator for gitolite READMEs
  • NEXT Fix cgit cache staleness

DONE Internal-use discourse instance

ICEBOX Point of Presence boxen




  • CANCELLED create a virtualserver for ipa01   CANCELLED
  • CANCELLED ansible playbooks to bring up FreeIPA   CANCELLED
  • CANCELLED tweak common role to set up FreeIPA auth   CANCELLED
  • CANCELLED Bring ZNC under SASL login   CANCELLED
  • CANCELLED Bring owncloud under SASL login   CANCELLED


  • INPROGRESS Create a virtualserver for mail01
  • NEXT Install dovecot on mail01
  • INPROGRESS design relay system for mail01 postfix
    • DONE VPN credentials
    • DONE VPN host
    • INPROGRESS configure mail system
      • INPROGRESS postfix on relayhost
      • NEXT postfix on mail01
      • NEXT dovecot on mail01
      • NEXT spamassassin milter on mail01
  • WAITING Sync IMAP over to mail01
  • WAITING Integrate LDAP login to mail01 dovecot
  • WAITING Integrate LDAP login to mail01 postfix
  • WAITING Update MX records
  • WAITING Set up rss2email on mail01
  • WAITING kill rs3
  • WAITING Figure out how to get wallace to stop holding messages

    • var/spool/pykolab/wallace/resources/ACCEPT

    error: uncaptured python exception, closing channel <smtpd.SMTPChannel connected at 0x1376fc8> (<type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte 0xb4 in position 5639: invalid start byte [/usr/lib64/python2.6/|read|78] [/usr/lib64/python2.6/|h andlereadevent|428] [/usr/lib64/python2.6/|handleread|158] [/usr/lib64/python2.6/|foundterminator|184] [/usr/lib/python2.6/site- packages/wallace/|processmessage|249] [/usr/lib64/python2.6/json/|dumps|230] [/usr/lib64/python2.6/json/|encode|367] [/usr/lib64/python2.6/json/|iterencode|309] [/usr/lib64/python2.6/json/|iterencodedict|275] [/usr/lib64/python2.6/json/ |iterencode|294])

ICEBOX Phonics

  • ICEBOX NodeJS roles
  • ICEBOX Deploy and configure phonics
  • ICEBOX Shitty mobile UI for phonics


DONE Open source infrastructure

DONE Make website

NEXT Make sure everything gets more and more backed up in my existing infra roles

WAITING Address FIXMEs in the Postgres role tasks

[2014-05-08 Thu 00:30] file:///home/rrix/Projects/devops/roles/postgres/tasks/main.yml::# FIXME: Get rid of this shit.


CANCELLED Peer with Torrie   WAITING

WAITING Peer with Robbie   WAITING

NEXT Add gpg key Ids to disk labels

CANCELLED Add ianweller to git01

Specific Host Work

INPROGRESS hypervisor01

  • DONE deal with full partitions on hypervisor01 RAID
    • DONE Clean up backups
    • DONE Purchase 2x2TiB disks   PURCHASE WAITING
    • DONE add new disks in to the RAID
    • DONE migrate rootfs to new disks


  • CANCELLED Create SSL certificates for cloud01, and make owncloud use them
    • CANCELLED Use the generated cert
    • CANCELLED Import this certificate in to all of my devices
      • [ ] roll in to ansible provisioning for base fedora role
      • [ ] install on phone
      • [ ] install on tf700
  • CANCELLED Make cloud01 work with selinux enabled
  • NEXT Feed ZNC logs in to ElasticSearch


CANCELLED Respond to Matthew Miller <> on kickass systems on fedora?   CANCELLED

DONE Figure initial out meeting date

  • DONE Figure out who is in on the initial buildout
    • Torrie
    • ian
    • vil
    • robbie
    • gregimba
    • relrod
  • DONE Meeting Agenda
    • Phase 1 core services
      • Description
      • Owner
      • ETA
    • Peering plan
      • OpenVPN + BGPd
      • Tinc
  • DONE send out doodle

DONE Review and Provide feedback on

Hypervisor/microserver layout   NOTE

  • NEXT attach a file to this with photo of page in note book
  • Requirements
    • Can recover pieces from outside of network and outside of home
    • Capable of scaling home needs
    • Possible 3rd party usage/semihostile guest OSs
    • ALL VMs and hypservisor is controlled by ansible
      • Sync config hourly ON the guests, 0% service interruption
  • Layout
    • 500 or 256GiB 2.5" drive, boot disk
      • 50GiB hypervisor
      • 50GiB*x guest partitions to fill remaining space
    • 2x2TiB existing configuration
      • 2000GiB RAID1 MDADM
        • 2000GiB LVM VG
          • 1000GiB LVM LV "Files", ext4 formatted
          • 1000GiB LVM LV "Backups", ext4 formatted
    • 2 empty bays to expand LVM to expand existing LVs
  • Needs
    • 2.5 rootfs drive
    • RAM (TBD, 2x4GiB? may cannabalize)
    • SATA Cable for rootfs drive
  • Things to VM
    • Subsonic
    • Transmission+Sickbeard
    • ZNC
    • OwnCloud
    • Discourse
  • KVM
  • bridge guests on to network
  • files + backups are encrypted via LUKS with seperate keys
  • hypervisor runs NFS or sshfs and CIFS
    • No direct VM level access to the LVM
    • How to encapsulate per-VM backup directories and access levels?
      • Subsonic shouldn't access owncloud backups
      • owncloud SHOULD access znc user moddata, etc

Kickass Systems Core Service Proposal   NOTE

What is a core service?

A core service is a service which any registered user of Kickass Systems has the ability to use. Core Services are not necessarily a part of the core infrastructure, they do not need to be critical for the healthy function of the network to qualify as a Core Service.

Core Services exist as shared resources and projects for the entire community and works towards reaching the goal of providing an entire suite of infrastructure and tools the members of Kickass Systems want to use.

  • Examples
    • Code Hosting infrastructure
    • Discussion platform
    • IRC Services
    • LDAP/SSO
    • Storage fabric such as Ceph
    • owncloud/caldav infrastructure
    • Core Service monitoring infrastructure

How do we create one?

  • Proposal Process

    Services are proposed as Service Proposals on Discourse.

    • Answer the following questions:
      • Brief description of the system, and its purpose for the network
      • Is the service already deployed on Kickss Systems as a non-Core Service?
      • Will it integrate with LDAP SSO?
      • Who will own and manage the system?
      • What resources will be required?
      • Where will the service be hosted?
        • If the service is already deployed, what is its current internal and external URI?
      • What is the expected SLA of the service?
    • Create a Page on the wiki answering these questions
    • Create a post on Discourse linking to the wiki page
  • Approval Process

    There will be a seven day discussion period from the time the proposal is posted to Discourse.

    Approval process is based on consensus with limited blocking. Consent is implied unless issues are actively raised. Discussion of service proposal does not itself imply a block, only an explicit "-1" is a block.

    A service can be implemented with consensus or a single block, two or more blocks require changes and the voting period is extended another 7 days. If at the end of the seven day period, there are still blocks, the proposal is scrapped and most be re-proposed at a later time.

What technical rights does a Core Service have?

Core Services are given:

  • A CNAME hanging directly off of TLD.
  • Monitoring with the Core Services Nagios infrastructure

What technical responsibilities does a Core Service have?

  • SHOULD have a well defined SLA
  • MUST integrate with core single sign on infrastructure
  • MUST NOT discriminate in who may use the service
    • Services MAY have multiple user tiers, especially when service is resource intensive, such as shell or storage

NEXT Kickass Systems Mission Statement

generating SSL CA   NOTE


See Also

Author: Ryan Rix

Created: 2015-10-25 Sun 18:57

Validate XHTML 1.0