Erik Schwartz -- Blog

Summary of “(2018) ATT&CKing the Status Quo: Improving Threat Intel and Cyber Defense with MITRE ATT&CK”

Katie Nickels & John Wunder

video, slides

ATT&CK helps answer these common questions from SOC

How effective are my defenses
Do I have a chance of detecting APT[123] for example
is my data useful?
Do I have overlapping tool coverage
Will this shiny XYZ product improve my defenses?

The difficult task of detecting Tactics, Techniques, and Procedures (TTPs)

Pinnacle of David Biancos pyramid of pain
It’s easy for attackers to change things like Hash values, IP addresses, Domain names, etc.
Much harder for them to change their attach behavior
ATT&CK is here to help with TTPs, it’s a knowledge base of adversary behavior

ATT&CK

based on real-world observations in MITRE red/blue team experiments
A common language. When you say the attacker moved laterally, it defines what that means
Community driven

Focus on adversary lifecycles

assume they’ll break your perimeter
lifecycle
- pre-att&ck
  - Gaining network access
  - laying groundwork for spearphishing
- enterprise
  - Move laterally
  - Exfiltration

Technique example: New Service

Install program that starts on system boot
ATT&CK describes common defenses, mitigation, logs where this appears

Example software: Chopstick

How to use ATT&CK

Detect, Threat intel, Assess and Engineer, Adversary emulation
More important: how to coordinate between these 4
ATT&CK is available in STIX (Structured Threat Information eXpression) json format for communicating cyber threat intel (CTI)
ATT&CK navigator to help you focus on what’s relevant to your org

Status Quote in Threat Intel

So many reports
Not clear how to apply intel to defense
Over-reliance on indicators like IPs and domains. Because they’re quantifiable. But it’s just wack-a-mole.

How to use ATT&CK

Use ATT&CK to structure tech intel
Example from Palo Alto 42 that consumes ATT&CK data to construct TTP playbooks describing attack campaigns. Shows a given groups evolution over time.
Tailor your existing threat intel repo. Many are starting to support ATT&CK like MISP, ThreatQ, etc
Have the threat intel originator do it
Start at the tactic level
Use existing website examples
Work as a team

Detection and Analytics

Now that we get reports from ATT&CK we can prioritize defense
Once we have indicators of known malicious IPs, hashes for malware, etc
- Follow the TTP to what would be next from this indicator
How to avoid false positives? Move towards analytics
- Indicators like file hashes, IP addresses, are effectively infinite
- Analytics follow credential dumping, or process persistence which have many fewer signatures to track
Analytics
- Look at observable events and artifacts. E.g. what’s it look like RDP is used for lateral movement? Analytic: it shows up in Windows Event Log.
- Then the work is to build evidence to separate good from bad and alert on bad. Windows example: process tries to elevate priv by riding on admin process. Attacker wants to bypass system dialog popup that asks for Admin priv.
- Important to understand the attack goal. Try the attack yourself. How to pinpoint bad logs without overfitting the attack. It’s iterative. New products will create new false positives and new attack techniques will need new analyics.
- It’s hard. Each OS requires totally different expertise to monitor effectively.
- Orgs that help define detection - NH-ISAC
  - SIGMAQ
  - Cyber Analytic Repository
  - MISP
- How to deal with false positives - Look at chains of events: graphs. Neo4j
  - Machine learning for classifications
  - Tighten the feedback loop: learn from your analysts
  - Focus on issues that lead to most important biz impacts
  - Adversaries are good a hiding in noise of normal logs - Agile logging: can we collect fewer logs until we notice something bad then ramp up?
- Roadmap for getting started on your own - Detection Lab
  - Be bad: Atomic Red Team to mimic red team
  - Write some detections. Use ThreatHunter-Playbook for inspiration
  - Share what you’ve learned with community
  - Have your red team focus on techniques from you analytics: use what’s real for your org
  - Use consistent naming conventions so you can communicate with your red team

Other tools

Caldera: an automated adversary emulation system

2018 Keynote talk: advancing infosec

By John Lambert, Microsoft

Traditional versus advanced defense concepts

Conventional Wisdom in Defense

Ideas

Reverse mentorship, where less experienced person explains concept to more experienced
How to increse rate of learning
- Promote community
- Organized knowledge
- Executeable Know-how
- Repeatable Analysis
“Githubification” of Infosec: important for community to share vendor-neutral learnings that any tool can pull in. So that we can all benefit for any other team’s incident response, just just our own. Multiply our power.

Selected tools mentioned

Mimikatz tool to explore windows security
Virustotal to analyze files and URLs for maliciousness
Yara database of metadata to identify malware samples
snort analyze packets; intrusion detection
Atomic Red Team test cases that simulate attacks to verify Blue Team alerts
Jupyter notebooks hyper popular datascience toolset in python. Great way to share infosec analysis. E.g. visualize netblocks that a machine connects to.
Papermill parameterize and run jupyter notebooks in automated fashion
Binder platform as a service that quickly hosts a jupyter GitHub repo so you can interact with it in your browser.

Trajedy of systemd talk

Summary

Linux started simple, initd started a few daemons and mounted filesystems
- Unfortunately it conflates system config (filesystems) with service bootstrap
- This gets in way of automated service management. - along comes upstart and systemd to handle things that need more active mgmt
The web made things complicated
- We can’t fork processes to handle each connection anymore
- We manage complicated databases that have all sorts of persistant processes
Along came services which are superset of daemons
- init.d is not great at keeping services running once they start
Comparative systems
- Windows NT: actually has great service model despite security weaknesses
- MacOS: has great service delegation that receives well defined mgmt calls
  - launchd took over for init.d, cron, etc
  - can start service daeomons as needed by listening on TCP ports, unix sockets. Means they don’t have to start at boot. They can be run and managed ad hoc.
  - this is super important for complicated systems like desktop envs
systemd is based on launchd
- Inventor (Lennart Poettering) didn’t think upstart had enough mechanisms for dependency ordering
- wanted boot to start less and more in parallel
- wanted an init system to listen for hardware/software changes like launchd does
- you start thinking more in terms of handling events rather than system boot
- Between 2011-2015 systemd was adopted across majority of linux distros
Systems are much more dynamic than they once were - We need unified layer between kernal and userspace: kernal | system | userspace
- this avoids cobbling together 15 different systems to manage system functions
Issues with systemd - “violates Unix philosphy”
- “bloated and monolithic”
- “it’s buggy!”
- “I don’t like the inventor”
- “It’s not portable” it’s agressively linux-specific like cgroups - UNIX is dead. You have Linux and rounding errors like FreeBSD
  - Went from pathological diversity to pathological monolithism
  - cgroups are even better than jails - E.g. you can run user-level daemons
systemd is change, super disruptive
- We love change when we’re doing it
- We don’t like outside systemd person forcing us to abandon our RC shell scripts we’ve run for 20 years
- Contempt is not cool. We shouldn’t send threats to systemd people. We should see it’s utility. Adopt it or grow our own (speaker is from BSD)
The promise of systemd - Message Transport: Dbus e.g.
- You need an RPC framework so you don’t worry if your API call talks to kernal or user-space: e.g. “network interface, take this IP address”
- Service lifecycle
- Automation via API
- Containers and their encapsulation
- Consistent device naming
- Good logging: append only
Catharsis
- Cleansing. Let’s get with it
- Challenge: look at systemd and find one thing you like
Questions
- Q: Are containers DLL hell? A: DLL hell is 5 diff libraries and your app chooses wrong one

I needed to experiment with 301 redirect behavior for a site under SSL.

The constraints

make sure redirects from https://mydomain.com aren’t spoiled by certificate errors because the certificate covers www.mydomain.com but not mydomain.com
Here’s the post that describes this problem and how to avoid it. But it’s 5 years old so I wanted to test it all out.
Use the Pantheon PaaS
Use letsencrypt to generate throw-away certificates for free

Generate a certificate

Given that I’m using a PaaS, I can’t install the lets encrypt certbot to automatically manage certificates the way others would. For my use case I just want to generate temporary certificates to test out the 301 behavior (in the long-term I’ll get a certificate from a provider with longer validity). So for now I’m ok using lets encrypt to create the cert manually.

To manually generate certificates with lets encrypt, I need a public web server that my domain resolves to so I point a throwaway domain “notabigdeal.club” at a DigitalOcean droplet ($.88 on namecheap for the domain!).

The DNS is set up like so on Namecheap:


URL redirect record	@	http://www.notabigdeal.club	Unmasked

ssh to the droplet and then:

openssl req -new -newkey rsa:2048 -nodes -out notabigdeal_club.csr -keyout notabigdeal_club.key -subj '/C=US/ST=Kentucky/L=Lexington/O=test/CN=notabigdeal.club'

certbot-auto certonly --authenticator manual --server https://acme-v01.api.letsencrypt.org/directory --text --email erik@erikschwartz.net --csr www_notabigdeal_club.csr

# Things that pantheon wants:

# The cert
cat 0000_cert.pem

# Intermediate cert
cat 0000_chain.pem

# The private key
cat www_notabigdeal_club.key

The 301 behavior

As expected, visiting ‘https://notabigdeal.club’ throws an error since the certificate is only for ‘www’

Lets try again with a Sans certificate that covers both domains

Namecheap DNS settings

record type
CNAME Record	www	my-site.pantheonsite.io.	30 min
A Record	@	104.130.89.68	30 min

Is generating a SAN CSR a pain?

According to the openssl CSR tool “Adding Subject Alternative Names to a CSR using OpenSSL is a complicated task. Our advice is to skip the hassle, use your most important server name as the Common Name, and specify the other names during the order process. Our Multi-Domain (SAN) Certificate ordering process will let you specify all the names you need without making you include them in the CSR.”

It’s not so bad to generate a SAN CSR with a good example in hand.

On the digital ocean droplet that hosts notabigdeal.club, create openssl.conf

[ req ]
default_bits        = 2048
default_keyfile     = privkey.pem
distinguished_name  = req_distinguished_name
req_extensions     = req_ext # The extentions to add to the self signed cert

[ req_distinguished_name ]
countryName           = Country Name (2 letter code)
countryName_default   = US
stateOrProvinceName   = State or Province Name (full name)
stateOrProvinceName_default = Iowa
localityName          = Locality Name (eg, city)
localityName_default  = Iowa City
organizationName          = Organization Name (eg, company)
organizationName_default  = The University of Iowa
commonName            = Common Name (eg, YOUR name)
commonName_max        = 64

[ req_ext ]
subjectAltName          = @alt_names

[alt_names]
DNS.1 = notabigdeal.club
DNS.2 = www.notabigdeal.club

Generate the CSR and the certificate:

# make sure to answer "Common Name (eg, YOUR name) []:" as notabigdeal.club
openssl req -new -newkey rsa:2048 -nodes -out notabigdeal_club.csr -keyout notabigdeal_club.key -config openssl.conf

Follow the instructions to add the files to .well-known/acme-challenge for each domain.

If it works, certbot will congratulate you and you can copy over the cert details to pantheon:

# The cert
cat 0000_cert.pem

# Intermediate cert
cat 0000_chain.pem

# The private key
cat notabigdeal_club.key

Pantheon-specific note

I had to enable both domains in Pantheon: www.notabigdeal.club and notabigdeal.club. Otherwise it gave a Pantheon boilerplate “Site not found” page.

When using Google Analytics (GA) directly, you can send custom events to GA in javascript like so:

ga('send', {
   'hitType': 'event',
   'eventCategory': 'Feedback',
   'eventAction': 'event action',
   'eventLabel': $(‘#feedback-text’).val(),
   'eventValue': 1
});

However, with Google Tag Manager (GTM), you don’t have the ga javascript object. Instead you can:

push variables to the GTM dataLayer object
create a tag in GTM that sends the variables along to GA

Here are the steps:

Replace your call to “ga(‘send’”

Let’s say you use the ga javascript object to send an event to GA on a form submission:

$('#feedback-form').submit(function(e) {
    ga('send', {
        'hitType': 'event',
        'eventCategory': 'Feedback',
        'eventAction': 'event action',
        'eventLabel': $(‘#feedback-text’).val(),
        'eventValue': 1
    });
    ...
});

With GTM, you’ll do this instead:

$('#feedback-form').submit(function(e) {
    dataLayer.push({
        'hitType': 'event',
        'eventCategory': 'Feedback',
        'eventAction': 'General',
        'eventLabel': $('#feedback-text').val(),
        'eventValue': 1
    });
    ...
});

Next you’ll configure a GTM tag to fire. In this tag you’ll pick up the values from the dataLayer and give them to GA.

In GTM: Map GTM variables to dataLayer variables

Variables > User Defined Variables > New > Data Layer Variable

> Configure Variable > Data Layer Variable Name: eventLabel (this is the variable name used in the javascript call to dataLayer.push)

I like to name the GTM variable as dataLayer.[variableName], in this case dataLayer.eventLabel.

In GTM: Create a tag to send the dataLayer variable(s) to GA

Tags > New > Google Analytics > Universal Analytics

Tracking Id: UA-1242343-12 (i.e. your tracking id)
Track Type: Event
Event Tracking Parameters
- Here you can use the GTM dataLayer variables you mapped earlier. In this example, I hard-code every value except for dataLayer.eventLabel
Create The trigger
- Fire On: Form > Form Submission > Some Forms > "Form Id” equals “feedback-form"

Test that the variable is sent to Google Analytics

Save the tag and trigger and click the GTM Preview and Debug (each time you make changes to GTM).

In a new tab, visit the site with the feedback form. You should see the GTM debugging pane at the bottom of the page. It should fire our newly created tag when the form is submitted.

Submit the feedback form.

In Google Analytics Reporting > Real-time > Events

You should see an entry for the just-submitted event. Click into it and you will see the eventLabel detail.

Well, you can’t _exactly_ deploy a custom block in Drupal

This is because “a custom block is made of two entities, one for the placement and the actual content”.

So if you create a custom block in development and deploy it to production using drush config-import you’ll get the following error:

This block is broken or missing. You may be missing content or you might need to enable the original module.

The workaround

Create a custom block in production that is disabled in the block layout. Sync the db to your dev environment. This allows you to add configuration that syncs to its related content on production (via the content’s UUID).

Workflow:

Make sure the custom block type exists on production
Block layout > create custom block that is disabled
- Add the content that you want
Sync production db to dev environment
In dev, place the block, configure, style it, etc.
export the config with drush config-export
Check in and deploy the config changes to prod

To use SSL throughout the Pantheon dev workflow

When using the dev-my-site.pantheon.io domain (or test or live), put this bad boy at the top of your settings. It will redirect HTTP to HTTPS.

/sites/default/settings.php
<?php

// Require HTTPS on pantheon
if (isset($_SERVER['PANTHEON_ENVIRONMENT']) &&
  $_SERVER['HTTPS'] === 'OFF') {
  if (!isset($_SERVER['HTTP_X_SSL']) ||
    (isset($_SERVER['HTTP_X_SSL']) && $_SERVER['HTTP_X_SSL'] != 'ON')) {
    header('HTTP/1.0 301 Moved Permanently');
    header('Location: https://'.$_SERVER['HTTP_HOST'].$_SERVER['REQUEST_URI']);
    exit();
  }
}

...

Also see the related pantheon docs.

Resource directories are harder than they appear

Building a community resource locator is deceptively easy to do. “Just scrape the data and make a nice interface!” Turns out it’s thornier than that. Keeping the data fresh and correct is the ultimate challenge, otherwise people don’t trust your fancy pants search interface. And to keep the data fresh requires all sorts of institutions and people to get on board. “What! It’s a community building problem?” That’s right developer, it’s largely a people problem. Tech is a surprisingly small piece of the puzzle.

So I’ve pinged a few people with a lot more experience than I have to come up with a n00bs guide to piloting a treatment directory project:

Focus on the hard part

The hard problem with a directory project is that keeping the information up to date takes a good deal of effort. Many people maintain their own scattered listings, whether front-line workers or professional aggregators, but don’t have a good incentive to contribute to a larger project.

Find reliable data maintainers

The key is finding a few of these people who are willing to work with you take on the role of data maintainer. This would mean they log in to your website as often as needed to keep the info current.

Hopefully you have some ideas about ways to appeal to them. Perhaps they’d be interested to search their listings on a user-friendly website, if only so that they can sync information with their co-workers.

Start with a small resource domain

I also heard to start with a small service area so that the problem is more manageable. Focus on a handful of relationships, get a few reliable data maintainers, and slowly build a user base around good, trustworthy information.

Find ways to prove the value

From there, you’ll ideally have success stories. Hopefully this will be hard numbers from people who can show how much better they are able to serve clients with this rich source of information. It could also be stories from service recipients who are happier and healthier thanks to a service they wouldn’t have found otherwise.

Keep the long-game in mind

With good data maintainers, good institutional partners, and perhaps good press the directory could take on a life of its own and expand into new service areas. I’ve heard a vision for long-term success that that I’ll share. Eventually politicians and service funders need to require that services make their information available in a public directory in a uniform format. If all services contribute to public databases, then the problem is much easier: build user-friendly websites to search the listings. With tightly-framed success stories you might be able to make the case to the powers that be.

How tech can help

To kick things off, you’ll need a database application to keep track of your listings and let people search them. At of May 21, 2015, I recommend using a flipping Google Spreadsheet (h/t @allafarce). Data maintainers are keeping their data in spreadsheets anyway. Make their life easier. It could be said that making their life easy is the primary concern anyway.

A search application can pull listings from it like it’s an API. Spreadsheets even have version control and user permissions to an extent. It’s a great, low-overhead way to get the application out there. Wire up an interface like finda and you’re off to the races. Now keep that data pure and relevant!

Stream of Conscious Notes Following the CKAN 2.2 install docs

Ubuntu 12.04 64bit

as root on digital ocean droplet.

apt-get update
apt-get install -y nginx apache2 libapache2-mod-wsgi libpq5
wget http://packaging.ckan.org/python-ckan_2.2_amd64.deb
dpkg -i python-ckan_2.2_amd64.deb
apt-get install -y postgresql solr-jetty

Config Solr

http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html#setting-up-solr

/etc/default/jetty

NO_START=0            # (line 4)
JETTY_HOST=127.0.0.1  # (line 15)
JETTY_PORT=8983       # (line 18)

$ service jetty start
 * Could not start Jetty servlet engine because no Java Development Kit
 * (JDK) was found. Please download and install JDK 1.4 or higher and set
 * JAVA_HOME in /etc/default/jetty to the JDK's installation directory.

Point Jetty at the JVM in /etc/default/jetty

JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64/

Success!

$ service jetty start
grep: character class syntax is [[:space:]], not [:space:]
 * Starting Jetty servlet engine. jetty                                                                                                                                          * Jetty servlet engine started, reachable on http://ckan:8983/. jetty

$ wget http://localhost:8983
--2014-08-31 00:52:19--  http://localhost:8983/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8983... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1082 (1.1K) [text/html]
Saving to: `index.html'

100%[======================================================================================================================================>] 1,082       --.-K/s   in 0s

2014-08-31 00:52:19 (71.0 MB/s) - `index.html' saved [1082/1082]

root@ckan:~# mv /etc/solr/conf/schema.xml /etc/solr/conf/schema.xml.bak
root@ckan:~# ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml /etc/solr/conf/schema.xml
root@ckan:~# service jetty restart

$ vim /etc/ckan/default/production.ini
and set `solr_url = http://127.0.0.1:8983/solr` on line 74

$ cd /usr/lib/ckan/default/src/ckan
$ paster db init -c /etc/ckan/default/development.ini
The program 'paster' is currently not installed.  You can install it by typing:
apt-get install python-pastescript

$ apt-get install python-pastescript

failure to init db

$ paster db init -c /etc/ckan/default/production.ini
Traceback (most recent call last):
  File "/usr/bin/paster", line 4, in <module>
    command.run()
  File "/usr/lib/python2.7/dist-packages/paste/script/command.py", line 103, in run
    command = commands[command_name].load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1989, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 11, in <module>
    import ckan.lib.fanstatic_resources as fanstatic_resources
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/fanstatic_resources.py", line 6, in <module>
    from fanstatic import Library, Resource, Group, get_library_registry
ImportError: No module named fanstatic

$ pip install fantastic
The program 'pip' is currently not installed.  You can install it by typing:
apt-get install python-pip

$ apt-get install python-pip

apt-get install python-pip
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python-pip
0 upgraded, 1 newly installed, 0 to remove and 15 not upgraded.
Need to get 95.1 kB of archives.
After this operation, 399 kB of additional disk space will be used.
Get:1 http://mirrors.digitalocean.com/ubuntu/ precise/universe python-pip all 1.0-1build1 [95.1 kB]
Fetched 95.1 kB in 0s (1,269 kB/s)
Selecting previously unselected package python-pip.
(Reading database ... 122812 files and directories currently installed.)
Unpacking python-pip (from .../python-pip_1.0-1build1_all.deb) ...
Processing triggers for man-db ...
Setting up python-pip (1.0-1build1) ...
root@ckan:/usr/lib/ckan/default/src/ckan# pip install fantastic
Downloading/unpacking fantastic
  Could not find any downloads that satisfy the requirement fantastic
No distributions at all found for fantastic
Storing complete log in /root/.pip/pip.log

I needed to activate python virtual env

$ . /usr/lib/ckan/default/bin/activate
(default)root@ckan:/usr/lib/ckan/default/src/ckan# cd /usr/lib/ckan/default/src/ckan
(default)root@ckan:/usr/lib/ckan/default/src/ckan# paster db init -c /etc/ckan/default/production.ini
...
sqlalchemy.exc.OperationalError: (OperationalError) FATAL:  password authentication failed for user "ckan_default"
FATAL:  password authentication failed for user "ckan_default"

I probably need to setup postgres first?

Setup Postgres user

$ sudo -u postgres createuser -S -D -R -P ckan_default
$ sudo -u postgres createdb -O ckan_default ckan_default -E utf-8
$ sudo mkdir -p /etc/ckan/default
$ sudo chown -R `whoami` /etc/ckan/
$ vim /etc/ckan/default/production.ini
change `sqlalchemy.url = postgresql://ckan_default:pass@localhost/ckan_default` to have the new password

$ ckan db init

Install DataStore extension

$ vim /etc/ckan/default/production.ini
add datastore to list of plugins `ckan.plugins = <list of plugins> datastore`

“The DataStore requires a separate PostgreSQL database to save the DataStore resources to.”

$ sudo -u postgres createuser -S -D -R -P -l datastore_default
$ sudo -u postgres createdb -O ckan_default datastore_default -E utf-8

Now, uncomment the ckan.datastore.write_url and ckan.datastore.read_url lines in your CKAN config file and edit them if necessary, for example:

ckan.datastore.write_url = postgresql://ckan_default:pass@localhost/datastore_default
ckan.datastore.read_url = postgresql://datastore_default:pass@localhost/datastore_default

Set permission

$ sudo ckan datastore set-permissions
Perform commands to set up the datastore.
    Make sure that the datastore URLs are set properly before you run
    these commands.

    Usage::

        paster datastore set-permissions SQL_SUPER_USER

    Where:
        SQL_SUPER_USER is the name of a postgres user with sufficient
                       permissions to create new tables, users, and grant
                       and revoke new permissions.  Typically, this would
                       be the "postgres" user.

Point paster at the config

$ paster datastore --config=/etc/ckan/default/production.ini set-permissions postgres
2014-08-31 01:26:47,376 WARNI [ckanext.datastore.plugin] Omitting permission checks because you are running paster commands.
Set permissions for read-only user: SUCCESS

$ sudo service apache2 restart
$ sudo service nginx restart
$ curl -X GET "http://127.0.0.1:80/api/3/action/datastore_search?resource_id=_table_metadata"
returns json if successfully installed

Install DataPusher

“Starting from CKAN 2.2, if you installed CKAN via a package install, the DataPusher has already been installed and deployed for you. You can skip directly to Configuration.”

Woot!

Set site_url, add datapusher to ckan.plugins

$ vim /etc/ckan/default/production.ini
ckan.datapusher.url = http://104.131.23.252/
ckan.site_url = http://104.131.23.252/
ckan.plugins = <other plugins> datapusher

$ sudo service apache2 restart
$ curl 0.0.0.0:8800
{
  "help": "\n        Get help at:\n        http://ckan-service-provider.readthedocs.org/."
}

At this point you must supply a remote url but when you do, file preview and datastore api works!

Select most recent records using datastore sql api

Figure out the query to run:

$ sudo -u postgres psql datastore_default

select * from "963cf23f-90be-479f-a7d7-7cb7fb29d45d" where "DateOpened" > (now() - '12 day'::interval);

# as a url:
http://104.131.23.252/api/action/datastore_search_sql?sql=select%20*%20from%20%22963cf23f-90be-479f-a7d7-7cb7fb29d45d%22%20where%20%22DateOpened%22%20%3E%20(now()%20-%20'12%20day'::interval)

Set up FileStore

$ sudo mkdir -p /var/lib/ckan/default
$ vim /etc/ckan/default/production.ini
ckan.storage_path = /var/lib/ckan/default

$ sudo chown www-data /var/lib/ckan/default
$ sudo chmod u+rwx /var/lib/ckan/default
$ sudo service apache2 reload

Now you see the ‘upload’ link when adding a resource

Determine CKAN version and extensions

http://www.civicdata.com/api/util/status

Field Data type error in Datapusher table

ERROR [ckan.controllers.api] Validation error: '{\\'info\\': {\\'orig\\': [\\'invalid input syntax for type numeric: "12040846W7"\\\\nLINE 4:                     WHERE ("caseno") = (\\\\\\'12040846W7\\\\\\');\\\\n                                                ^\\\\n\\']}, \\'__type\\': \\'Validation Error\\', \\'data\\': \\'(DataError) invalid input syntax for type numeric: "12040846W7"\\\\nLINE 4:                     WHERE ("caseno") = (\\\\\\'12040846W7\\\\\\');\\\\n

caseno is numeric in the db. We should explicitly set its datatype during datastore_create

datastore_default=# \d my-upsertables-3
                                       Table "public.my-upsertables-3"
   Column   |            Type             |                            Modifiers
------------+-----------------------------+------------------------------------------------------------------
 _id        | integer                     | not null default nextval('"my-upsertables-3__id_seq"'::regclass)
 _full_text | tsvector                    |
 caseno     | numeric                     |
 CaseYr     | numeric                     |
 Address    | text                        |
 DateOpened | timestamp without time zone |
 CaseType   | text                        |
 Status     | text                        |
 StatusDate | timestamp without time zone |
 Closed     | text                        |
 parcelId   | numeric                     |
 lat        | numeric                     |
 lng        | numeric                     |
Indexes:
    "my-upsertables-3_pkey" PRIMARY KEY, btree (_id)
    "my-upsertables-3_caseno_key" UNIQUE CONSTRAINT, btree (caseno)

datastore_default=# alter table "my-upsertables-3" alter column caseno type text;

createdb "geocode_code_enforcement"
csvsql --db postgresql:///geocode_code_enforcement --insert --table ce_calls data/ce-gecoded.csv
psql geocode_code_enforcement -c 'DROP TABLE parcels'
csvsql --db postgresql:///geocode_code_enforcement --insert --table parcels data/ParcelCentroids.csv

Add id

Connect to DB with sequel

map <leader>r :!ruby %<cr>

To make sure sequel result set is dataset is of the table object (eg CodeEnforcment) rather than hash

http://fredwu.me/post/58910814911/gotchas-in-the-ruby-sequel-gem

Inspect distance between geocoders

select distance_between_geocoders from ce_calls where distance_between_geocoders IS NOT NULL order by distance_between_geocoders DESC;

Not sure what the unit of distance is

geocode_code_enforcement=# select COUNT(*) from ce_calls where distance_between_geocoders IS NOT NULL AND distance_between_geocoders < 10;
 count
-------
 36537
(1 row)

geocode_code_enforcement=# select COUNT(*) from ce_calls where distance_between_geocoders IS NOT NULL AND distance_between_geocoders > 10;
 count
-------
  1162
(1 row)

geocode_code_enforcement=# select count(*) from ce_calls where distance_between_geocoders IS NOT NULL AND distance_between_geocoders < .05;
 count
-------
 27602
(1 row)

This distance is actually between elasticsearch and nominatim, the distance_between_geocoders is about to be recalculated for these google geocoded results

geocode_code_enforcement=# select sum(distance_between_geocoders) from ce_calls where google_result = true;
     sum
-------------
 530021.4827
(1 row)

Snapshot DB before re-calculating distance

pg_dump geocode_code_enforcement > data/pg_ce_calls_nominatim_distance

Recalculate distance

with_parcel_id = DB[:ce_calls].join(:parcels, :PVANUM => :parcel_id).where("google_result = true AND distance_between_geocoders > 0").all

$ ruby calculate_distance.rb | tee diff_between_google_and_nominatim_distance.csv

geocode_code_enforcement=# select sum(distance_between_geocoders) from ce_calls where google_result = true;
    sum
-----------
 3717.4155
(1 row)

$ ruby -v
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-darwin13.0.2]
$ gem install elasticsearch
$ elasticsearch --config=/usr/local/opt/elasticsearch/config/elasticsearch.yml

require 'elasticsearch'

# Connect to localhost:9200 by default:
es = Elasticsearch::Client.new log: true

# Round-robin between two nodes:
# This returns an http error, faraday can't find search1 or search2
# es = Elasticsearch::Client.new hosts: ['search1:9200', 'search2:9200']

# Connect to localhost explicitly instead
es = Elasticsearch::Client.new hosts: ['localhost:9200']

# Connect to cluster at search1:9200, sniff all nodes and round-robin between them
es = Elasticsearch::Client.new hosts: ['localhost:9200'], reload_connections: true

# Index a document:
es.index index: 'my_app',
         type:  'blog_post',
         id: 1,
         body: {
          title:   "Elasticsearch clients",
          content: "Interesting content...",
          date:    "2013-09-24"
         }

# Get the document:
es.get index: 'my_app', type: 'blog_post', id: 1

# Search:
es.search index: 'my_app',
          body: { query: { match: { title: 'elasticsearch' } } }

Add a bit of data

http://joelabrahamsson.com/elasticsearch-101/

Install Sense Chrome Extension
Get rid of index cruft

curl -XDELETE "http://localhost:9200/_all" -d'
{
   "query": {
      "match_all": {}
   }
}'

PUT /addresses/address/38108180
{
    "PVANUM": 38108180,
    "NUM1": 2469,
    "NAME": "FOO",
    "TYPE": "RD",
    "ADDRESS": "2469 ROCKMINSTER RD",
    "UNIT": ""
}

Search the data

GET _search
{
   "query": {
      "query_string": {
         "default_field": "ADDRESS",
         "query": "ROCKMINSTER"
      }
   }
}

GET _search
{
   "query": {
      "query_string": {
         "default_field": "ADDRESS",
         "query": "469 Martin luther"
      }
   }
}

Snapshot the index

$ curl -XPUT 'http://localhost:9200/_snapshot/addresses_index' -d '{
    "type": "fs",
    "settings": {
        "location": "/Users/erik/code/address-service/addresses_index"
    }
}'

# Get info about it
GET /_snapshot/addresses_index?pretty

Assuming you have a shapefile

shp2pgsql -s 4326 VotingPrecinct/VotingPrecinct.shp | psql -d postgis_demo

csvsql --db postgresql:///postgis_demo --insert --table ce_calls ce_calls_gecoded.csv

update column names manually since the camelcase columns are not usable in postgres (today for me anyway)

head -n 1 ce_calls_geocoded_bad_headers.csv > ce_calls_gecoded.csv
vim ce_calls_geocoded.csv # fix the headings
# tail all but the header line
tail -n +2 ce_calls_geocoded_bad_headers.csv >> ce_calls_gecoded.csv

Error message

You don't appear to have the necessary database backend installed for connection string you're trying to use. Available backends include:

Postgresql:	pip install psycopg2
MySQL:		pip install MySQL-python

For details on connection strings and other backends, please see the SQLAlchemy documentation on dialects at:

http://www.sqlalchemy.org/docs/dialects/

pip install psycopg2

in qgis: Layer > Add PostGIS layers

Problem: csvsql doesn’t insert lat long as point!

Add point column to ce_calls table

SELECT AddGeometryColumn ('public','ce_calls','geom',4326,'POINT',2);
UPDATE ce_calls SET geom = ST_SetSRID(ST_MakePoint(geo_longitude, geo_latitude), 4326);

Update SRID of voting precinct table

SELECT UpdateGeometrySRID('votingprecinct','geom',4326);

join on point in voting precinct

select objectid, c.* from votingprecinct v join ce_calls c ON (st_contains(v.geom, c.geom));

select counts against years

in a file named ce_calls.sql

copy
(select objectid as id,
  sum(case when extract(year from dateopened) = 2011 then 1 else 0 end) as y_2011,
  sum(case when extract(year from dateopened) = 2012 then 1 else 0 end) as y_2012,
  sum(case when extract(year from dateopened) = 2013 then 1 else 0 end) as y_2013,
  sum(case when extract(year from dateopened) = 2014 then 1 else 0 end) as y_2014
  from votingprecinct v inner join ce_calls c ON(st_contains(v.geom, c.geom) AND casetype = 'Housing')
  group by objectid
  order by objectid
) TO STDOUT WITH CSV HEADER;

run the query from command line, and export to stdout as csv

cat ce_calls.sql | xargs -0 psql -d postgis_demo -c > assets/data/metric/ce-housing.csv

Running OSX Mavericks

Following the official qgis install guide

Installs everything to: /Library/Frameworks/

http://www.kyngchaos.com/files/software/frameworks/GEOS_Framework-3.4.2-3.dmg
http://www.kyngchaos.com/files/software/frameworks/PROJ_Framework-4.8.0-1.dmg
http://www.kyngchaos.com/files/software/frameworks/UnixImageIO_Framework-1.5.0.dmg
http://www.kyngchaos.com/files/software/frameworks/SQLite3_Framework-3.7.17-2.dmg
http://www.kyngchaos.com/files/software/frameworks/GDAL_Framework-1.11.0-4.dmg
# install NumPy and GDal complete framework
http://www.kyngchaos.com/files/software/python/matplotlib-1.3.1-2.dmg
http://www.kyngchaos.com/files/software/qgis/QGIS-2.4.0-1.dmg

brew install postgresql
brew install postgis
createdb postgis_demo
psql -d postgis_demo -c "CREATE EXTENSION postgis;"

We had a fantastic event last week at Code for America called BETA, a so called “civic show and tell”. It was recorded for posterity, the only snag being that we didn’t record all of the magnificent slides that people produced. The good news is that we can hack together a way to embed them in the video along with the lovely presenters.

Problem Statement

Shucks, I have video of presentors but not their slides.

Solution

Follow along to the presentor video
Record the slide transitions in Keynote
Export the perfectly-timed Keynote slides as a video
Embed the slides video in the presentor video like a boss
…
Profit

Record slide transitions in Keynote (version 6)

Open the presentor video and get ready to play it
In Keynote menu Play > Record Slideshow...
Start the presentor video so that you can listen along
Back to Keynote, click the mic to mute audio input. Click record.
Transition the slides as you listen to the presentation
File > Export To > Quicktime

Embed slides video into presentation video in iMovie (version 10)

Create event, create new video (called Lexingteam in image below) and import your presentor video
Drag the slides video above the presentor video
Move the slides video left/right until the transitions line up with the presentors

Adjust the slides video

Click the ‘Adjust’ menu in the upper right
Select the slides video clip in the video area below
Set to ‘picture in picture’ and set the dissolve time to 0.0
Drag and resize the slides video window to your liking (you can also change ‘dissolve’ to ‘swap’ to make your slides the main picture and your presentation video the smaller picture)

Video editing tips from a n00b

You can click and hold on a part of the slides video to copy and repeat a section. This is helpful to make the intro slide longer.
If you have several video or picture sections above the ‘main’ video (for instance, if you copied the beginning slide to cover the beginning of the presentation), you can copy picture-in-picture adjustments like resizing the slides video. Adjust the video clip, Edit > Copy, select all of the clips and Edit > Paste Adjustments > All.

Bash: sort ruby files by number of lines (ascending)

find . -name '*.rb' -exec wc -l {} \; | sort
I used this to find a few hotspots in calagator

...
./spec/controllers/venues_controller_spec.rb
./spec/models/source_parser_ical_spec.rb
./spec/models/venue_spec.rb
./spec/controllers/sources_controller_spec.rb
./app/models/event.rb
./spec/controllers/events_controller_spec.rb
./spec/models/event_spec.rb

Heroku: backup postgres

heroku addons:add pgbackups:auto-month
heroku pgbackups:capture # backup right now!
heroku pgbackups # show the backups
load to local env curl -o latest.sql `heroku pgbackups:url
pg_restore --verbose --clean --no-acl --no-owner -h localhost -U myuser -d mydb latest.dump (via heroku)

Bash: save command you’ve typed without running it

Ctrl+a to jump to beginning of line and comment it out :) $ #echo 'this is a command for later'

Zsh: save command you’ve typed without running it

“Ctrl+Y will paste the last item you cut (with Ctrl+U, Ctrl+K, Ctrl+W, etc.)”

Sed: remove leading chars

echo 'this is a new new thing' | sed 's/^this //' => “is a new new thing”
echo 'this is a new new thing' | sed 's/.\{5\}//' => “is a new new thing”
echo ' remove leading whitespace' | sed 's/[ \t]*//' => “remove leading whitespace”

Kimono: retrieving crawled pages with the source url included

Add &kimbypage=1 to the query string

"results": [
    {
      "page": 1,
      "url": "http://www.lexpublib.org/event/friends-of-library-booksale-uk-chandler-medical-center",
      "collection1": [
        {
          "title": "Friends of the Library Booksale: UK Chandler Medical Center",
          "venue": "Other Location",
          "access_notes": "See Description",
          "description": "Friends of the Library will be have books for sale at the UK Chandler Medical Center.",
          "start_time": "Wednesday, May 21st - 8:00 AM"
        }
      ]
    },

Ruby: regex capture syntax

1.9.3-p484 :002 > 'xxfindmexx'.gsub(/xx(findme)xx/, '\1')
 => "findme"
# note that double quotes around \1 interprets it as a unicode escape
1.9.3-p484 :003 > 'xxfindmexx'.gsub(/xx(findme)xx/, "\1")
 => "\u0001"

Markdown: fix the fragment fn-infer behavior

Now it drops in a named link: <a name="do-some-cool-thing"></a> [do some cool thing](/2014-05-23-environment-improvements-3#do-some-cool-thing) => <a name="do-some-cool-thing"></a>[do some cool thing](#do-some-cool-thing)
It’s borderline unreadable as markdown syntax but hey, what can you do?

Live-ish blogging Solid Con!

Blue City

Hardware infrastructure to pipe point of observation data to the platform
Aggregate various atmospheric city data through resident sensors on phones and other devices
BlueData platform to aggregate multiple data sources
saha@bluecityllc.com

Data Vehicles for Open Mobility in Cities

Kabal Farah (MIT Media Lab)
City is a collection of desire paths
Map connections being created in space via mobile
First distruptive technology
- the car
- 15,000,000 sold in first wave
Self driving cars
- You experience city in new way
  - Office space
  - Familiy Space
- BAEDE-KAR: 8-track faux-gps
- Ford releasing Open XC API
- The car becomes aware of its technology
  - Starts to approach automation of high-tech warehouses of today
Build up preferences, experiences
- City is able to predict what is next for you, for everyone
- Creepy?

IoT and Restful Hardware

Alden Hart - Synthetos
Motion control sensors, motor controllers for CnC machines.
TinyG montion controller. Takes Gcode directly. 6-way motion
They got the electronics working but people didn’t like integrating with unstructured text protocol
They wanted to eliminate begel bone or raspberry pi as a base requirement, only as enchancement
They looked to REST and Fielding’s work where state resides on server
Need well defined resource model that describes machine very well
They replaced server with their chip
They use JSON support directly on hardware
Run JSON right over traces
Wrote their own parser, strict and relaxed mode
7 microseconds strtof() < 7uSec on 84 MHz ARM Cortex M3
JSON subset
- 7bit utf
- no linefeeds
- Flat namespace, don’t go wild nesting
POST means execute
cheats
- No request headers. null means get
- They’re all point to point
- Will execute Gcode directly, doesn’t have to be wrapped in JSON
Use footers to verify integrity
Have event reporting to communicate state deltas
Don’t innovate. Copy
- Use $ as root (from JSONpath)
- @ prefix for adressing, binding, routing
- # topic prefix
- ! priority. Jump the queue
Why not protocol buffers?
- To support late binding of devices
- telehash has binary payloads which is cool. Another possibility

Smart Design: Attention to microinteractions to go from prototype to product

Water watcher
- Contextual feedback about usage.
- Immediate indication
Difference between prototype and product
- prototype: demo a feature
- product is separated from prototype by many microinteractions
  - change a setting, log in, error recovery, ….
  - consider apple with one button OBE versus some thing you have to debug immediately
  - toaster with the ‘a little bit more’ button. They know their UX.
  - product is prototype after you remove all the ‘pebbles in your shoe’
  - Slate is upselling improved microinteractions (comments in-line and reduced pagination)
Anatomy of a microinteraction. Be deliberate about:
- trigger
  - Nest smoke detector glows and beeps so you can find the one with batteries low
- rules
  - OSX fans turn down when you’re dictating
- feedback
  - Ford shows leaves next to MPG which cause behavior change in drivers
- Loops and Modes

Announcing a Product Before It’s Ready - Stories From the Frontline

Jason Johnson, August Home Inc
Using (Near field communication) NFC for doorlocks
Bluetooth 4.0
Yves Behar’s design
ODM (original design manufacturer): someone with a product you can build on

The Collision of Privacy, Regulation, and Physical Computing

… joined late

Chris Clearfield (System Logic)
On-star
- Asked about being hacked and GM declared it ‘Read-Only’. “Until someone figures out otherwise”
Mobile radios
- Mobile Devices - Base band processors in common consumer phones
- History of being hacked: full access to filesystem proof-of-concept
Data and privacy within stores
- Use wifi MAC addresses to track people consistently
- “Conversnitch” - plug-in lightbulb replacement with mic that pushes to Mechanical Turk and Twitter
Private companies collection
- If data breach or legally compelled to give over your data, it’s hugely telling of your life
Charles Perrow on collisions between tech and privacy
- Organizations struggle to manage high risk systems
  - One failure leads to more: such tight coupling
  - So many org and people boundaries
    - Mat Honan example of confidence escalation
  - Two-factor accounts help with caveats
Hard to control in the expanding network
- Cars
- Fridges
- TVs
- Phones
- Meta-data up for grabs
Solutions
- Cultivate sensitivity. Be intentional
- Coupling yields problems
- Take organizational approach: designate sceptics on your design team

Testing across weird interfaces and devices

Jason Huggins (Sauce Labs Inc) @hugs
Nike fuel band needs to shake to test the app that captures its date
iPhone app to open your zipcar
- you need a phone and a car
You can pay zillions to build simulators that might be wrong
@hugs has developed a robot tester: Tapster
- (Low to High level testing pyramid) Unit -> Service -> Simulators -> Real World
- He shipped 70ish to phone manufacturers, other companies
To create your own tester bot
- Don’t spend 50k for a human hand robot
- Write a robot, motors, rubber pads, etc
Manufacturers prefer reality
Everthing on earth needs testing. Either you do it or your users do.
Tapster in action

Disrupting Space: Planet Labs From Garage to Orbit

Chris Boshuizen (Planet Labs)
31 Satellites in orbit
Map the earth every day
Bring awareness to earth through information they generate
Rapid feedback for the planet
Blue Marble Pic (Apollo 17 on way to moon in 1972)
- Important symbol for green movement
No space startups in last few decades
PhoneSat. When Chris was at NASA
- Off the shelf Google Nexus 1 with extra batteries
- Design a package to withstand space
- Crowdsourced retrieval of picture from space to HAM radio community
- Demo’d idea but phone was limited
  - No peripheral connectivity
- Out of this came Planet labs to create better satellites
Planet Labs
- Bootstrapped for a while in Cupertino
- Hard to get rocket in space
  - Space launches slipped 18 and 12 months each
- They got their sats launched from the space station

Vim: change in search

I often want to perform a search on a phrase that does not conform to word boundaries, change the matched text and repeat over a buffer.

really_long_var_name = 'bar'
...
if (really_long_var_name == 'bar')
end

Let’s change really_long_var_name to shorter_var_name

I’d like to search for really_long, do something like ciw shorter on the first result, and then hit n. to repeat the action on future matches. Unfortunately ciw doesn’t work because w captures the entire really_long_var_name rather than just the part we want to change. The most obvious solution is a bit cumbersome: :s/really_long/shorter/gc. Let’s see about writing a vimscript for cir (or maybe &?) or change in result.

Solution: use this snippet to turn the search result into a text object referred to by “s”. Now cs becomes “change selection” and is repeatable with commands like n. as in “Move to next result and repeat the previous command”

Via an add-on
Although I didn’t install the add-on. Instead I will try the keystroke to show/hide the bookmarks sidebar Command b.

Firefox: search page with regex

Fastest Search Add-on
I’m a little wary of willy-nilly installing add-ons so I’m going to punt on this for now.

Git: search on any branch

As seen on Stack Overflow git grep <regexp> $(git rev-list --all)

Markdown: anchor to each of these headings :)

As seen on SO, it’s pretty simple

Markdown: infer the anchor name

For each of these topics I’d like a unique anchor such as [Vim: do something faster](/2014-05-16-environment-improvements-2#vim-do-something-faster).
The anchor is based on the link body so I want to be able to create the the anchor by writing this: [Vim: do something interesting](/2014-05-16-environment-improvements-2#fn-infer)
- Then I preprocess the markdown file with a simple script.

Environment Improvements May 2 - May 9, 2014

As I write software I jot down minor annoyances in the development workflow or things I’d like to generally understand better. Each morning I spend 20 minutes addressing one or two problems and capture my new learnings here for posterity.

Vim: Editing Macros

C-r C-r a inserts register a in command mode
* register is system clipboard

Vim: make temporary key mapping to run a script you’re debugging

map <leader>r :!spring rake db:seed<cr>

Vim: Copy filename to system clipboard

:let @* = expand("%") from Stack Overflow
To map a keystroke :nmap cp :let @* = expand("%")

Vim: Ignore .keep files from ctrl-p file search

let g:ctrlp_custom_ignore = '.keep\|node_modules\|DS_Store\|git\|tmp/cache\|.swp'

Vim: delete text within html tag

dit who knew?!

Vim: tab refresher for easily looking at gem source code

	tabe          create new tab
	gt            go to next tab
	gT            go to previous tab
	{i}gt         go to tab in position i

Vim: prompt to mkdir if doesn’t exist

augroup vimrc-auto-mkdir
  autocmd!
  autocmd BufWritePre * call s:auto_mkdir(expand('<afile>:p:h'), v:cmdbang)
  function! s:auto_mkdir(dir, force)
    if !isdirectory(a:dir)
          \   && (a:force
          \       || input("'" . a:dir . "' does not exist. Create? [y/N]") =~? '^y\%[es]$')
      call mkdir(iconv(a:dir, &encoding, &termencoding), 'p')
    endif
  endfunction
augroup END

Bash: Use cut remove leading whitespace

Useful for removing extraneous stuff before piping output to another command

$ git branch
   foo-branch
$ git branch | cut -c 3-
foo-branch

Bash: Movement by words (like emacs)

M-f forward
M-b backward

deploy non-master branch to heroku (aka use standard git to push a feature branch to heroku master)

g push heroku search-park-name:master

Binstubs

Prepare the environment before shelling out to an executable
In practice this means the script makes sure the executable runs against the project’s ruby version and gemset

tl;dr As convenient and well-tended as walled gardens can be, they place frightning limitations on us. Also Save Couchy!

Just Another Cute Public Service Bot

In the course of the Code for America fellowship, my teammates and I seize opportunities to kick out “quick win” projects. Our partner city of Lexington had a fantastic basketball year (Go Cats!) which means fan fervor drives people to set couches on fire. During our weekly checkin with the city we joked about a PSA website to dissuade couch burnings. With 30 hours before the next big game we pumped our fists in air and said “lets do this.”

It was a great experience thinking through the site. After some crazy 8’s we decided on a character named Couchy behind a @SaveCouchy twitter campaign. When you tweet to @SaveCouchy, it responds with a thank you message and your very own link to savecouchy.com/buds/YourTwitterHandle. Next thing you know you’re a dancing robot attempting to safeguard Couchy from the flames. An example is warranted:

Live, Spam, Love

I’ll be honest. In development I mistakenly hit the twitter API and spammed 20 people on a popular hashtag before twitter cut me off. Account suspended. Doh! Twitter is reasonable and allows you to pinkie swear not to do it again and resume your account. Phew.

Twitter is clear-ish about when a bot can tweet at someone. Only if they solicit it by tweeting at you first. Ok cool, I followed those rules from that point on.

Game Time Go-Live

Fast forward to Final Four game time. Kentucky v. Wisconsin. Savecouchy.com went live and started following the rules and only replying to people who tweeted at it first. A few interaction later and it was suspended. I disabled all bot-replies, decided to run the account manually, and check the pinkie swear box to enable @SaveCounchy again. Before long it was disabled with no more pinkie swear option. Game over.

Hard to Make a Case for a Bot

So true. I mistakenly spammed some people and marred poor @SaveCouchy’s reputation. Nonetheless it hammered home the flaw in relying on systems that you have absolutely no control over. The web is about experimentation leading to interesting outcomes. In this case the experiment got called off by the medium.

It’s Twitter’s world, we’re just living in it

Richard Stallman calls out Software as a Service for it’s lack of freedom dubbing it “Service as a Software Substitute.” Twitter wants to tend its walled garden for the sake of its users and shareholders. But it’s unfortunate that developers are completely at Twitter’s mercy whether or not we can build on their unique communication platform.

This issue shows how complicated these walled gardens are. IRC isn’t as nice to use as Twitter largely because the latter is a well-funded product with a centralized gameplan. But damn if it’s not free. Perhaps that’s the tradeoff. I’d like to think there’s a better way.

Good thing there are smart people out there who have put lots of thought into this exact question. Please do send those thoughts my way. On Twitter of course. @eeeschwartz.

A Great Start to the Fellowship

The first month of Code for America training was a total blast. My cohort is a great group with all kinds of perspective on the challenge of improving the connection of people with their government. I’ve learned a lot from my new colleagues, among them astronaut interface designers, ex-Googlers, researchers, map-makers, circus performers, powerful developers and sharp designers.

The month of training itself was challenging in all of the right ways. We talked with luminaries like Tim O’Reilly, Jen Pahlka, Aneesh Chopra, Eric Ries, Steve Van Roekel, and many more. We did hands on negotiation training. Focused on good team dynamics. Learned the nuts and bolts of local gov. Worked on a small-scale hack project with SF city officials. Practiced self-organization. Great experience.

Off to our new favorite place: Lexington, KY

Tomorrow my most excellent teammates and I fly to Lexington, KY where we will live for the next month. We put our training to use as we learn as much as possible from our city partners, community members, civic hackers, and local aficionados about all that is Lexington. February sets the stage for the work we will build with the good folks of the Bluegrass over the course of the fellowship year. Everyone we’ve talked to at city hall, the CfA brigade, and the community has been fully enthusiastic and gracious. It’s going to be an excellent year of collaboration and growth for all involved. Team Big Lex is fired up.

(in a good way)

Prediction

Browser-based content editors will emerge that make Jekyll feel like WordPress and the world will be crazy ready.

Static Site Generators

Currently static-site generators are the domain of techno-dweebs who aren’t afraid of putzing around on the command line to publish a blog post. Once these types get familiar with Jekyll, Middleman, or Hugo (which is go-based, i.e. incredibly fast to compile) they are lifetime converts. With good reason, these tools make it incredibly easy to create and host a maintainable, secure, blog-oriented site with completely badass performance. The site is just static files after all.

Get an idea for an edutaining site late one night and you can buy a domain and host for free on GitHub Pages in a matter of minutes. It’s pretty amazing.

A Huge Opportunity

Every single brochure+blog WordPress site in the universe should be using workflow, if only the barriers to entry were a bit lower. Which is where the excellent prose.io project enters the picture.

Prose makes Jekyll feel more like WordPress. It’s a web-based content editor. Once a Jekyll site is configured with Prose, the world starts to look like this:

Prose workflow

versus this:

Jekyll workflow

Which is exactly what is needed to bring Jekyll to the masses, where it will thrive.

What’s left to be done?

Setting up and configuring Jekyll and Prose is still the domain of hackers. In the next year we will start to see Jekyll consultants pop-up to fulfill this need. Then comes valuable feedback: what do normal users struggle with, and how can we make the workflow more accessible?

I’m in the process of using this workflow myself to see what gaps exist and where I might contribute to Prose.

Static site generators are headed for a broad audience and a bright future. The world is most definitely ready.

@eeeschwartz

p.s. The makers of Prose have also introduced a project to host a Jekyll-site generator on your own infrastructure called Jekyll Hook. This is a great idea for a number of reasons, not the least of which is that you’re free to use any Jekyll plugins that you like where GitHub’s generator forbids them. Hosted static site generation unlocks all types of interesting possibilities.

Erik Schwartz

Explorer, weirdo, developer of wonderful things.

Stay curious.