Complete Computing Environment: Log and Statistics Processing

Table of Contents

(provide 'cce-logs)

NEXT Data Analysis and Interpretation

A lot of my team's work is capacity planning and log analysis; this often means pulling a few gigs of log data or database snapshots, and loading them in to IPython 1 as long-living IPython Notebooks 2. My workflow should treat IPython as a first-class citizen, preferably working within my Emacs environment, rather than via the browser or terminal. Emacs has integration via EIN (Emacs IPython Notebooks) 3, which has been forked to support IPython 2.04 with the fork being the one provided by ELPA.

There is also a lot of work that involves stream and buffer transformation; creating a strong UNIX pipeline to take structured data, map and reduce it as a single stream of thought. I want a similar functionality built in to my workspace; Emacs may not adhere to the UNIX philosophy of IPC-via-pipes 5, but I think that Emac's philosophy of Buffers-as-IPC6 is equally powerful. Things like Emac's occur 7 are similar strains of thought to Grep, and are arguably more powerful, but they lack composability, which is what I need to figure out with a buffer IPC workflow, composing commands like occur with other transformative tasks, to arrive at a buffer that contains the kernel of truth I was looking for. Tools like GNU's datamash 8 could lend towards this.

NEXT Make it so without Helm

A tool that I've been using lately and have really enjoyed, as a pipeline for data processing, is make-it-so by abo-abo. It's just a simple Makefile generator but has had profound impact on my workflow, as well as giving me a reason to use/learn Dired.

(add-to-list 'el-get-sources '(:name make-it-so
                               :type github
                               :pkgname "abo-abo/make-it-so"
                               (progn (mis-config-default)
                                      (setq mis-recipes-directory "~/.emacs.d/el-get/make-it-so/recipes"))))
(add-to-list 'cce/el-get-packages 'make-it-so)
(add-hook 'after-cce-hook
          (lambda ()
            (add-to-list 'load-path "~/.emacs.d/el-get/make-it-so")
            (require 'make-it-so)
            (setq mis-recipes-directory "~/.emacs.d/el-get/make-it-so/recipes/")

NEXT Literate Org-Mode

NEXT es-mode


Author: Ryan Rix

Created: 2017-03-31 Fri 15:22

Validate XHTML 1.0