add ssh integration tests (#26)

* add SSH integration tests

* README updates; better sshd container based on a docs.docker.com example

README.md

@@ -75,6 +75,8 @@ func init() {
 
 To add a schema for the new module, add a module under schemas, and update [`schemas/__init__.py`](schemas/__init__.py) to ensure that it is loaded.
 
+See [schemas/README.md](schemas/README.md) for details.
+
 ### Integration tests
 To add integration tests for the new module, run `integration_tests/new.sh [your_new_protocol_name]`.
 This will add stub shell scripts in `integration_tests/your_new_protocol_name`; update these as needed.

integration_tests/ssh/cleanup.sh

@@ -2,5 +2,10 @@
 
 set +e
 
-# Stub cleanup script for ssh zgrab2 module
-echo "TODO FIXME: IMPLEMENT SSH CLEANUP"
+CONTAINER_NAME="sshtest"
+
+echo "BEGIN DOCKER LOGS FROM $CONTAINER_NAME [{("
+docker logs --tail all $CONTAINER_NAME
+echo ")}] END DOCKER LOGS FROM $CONTAINER_NAME"
+
+docker stop $CONTAINER_NAME

integration_tests/ssh/container/Dockerfile

@@ -0,0 +1,16 @@
+FROM ubuntu:16.04
+
+# Adapted from https://docs.docker.com/engine/examples/running_ssh_service/#run-a-test_sshd-container
+
+RUN apt-get update && apt-get install -y openssh-server
+RUN mkdir /var/run/sshd
+RUN echo 'root:password' | chpasswd
+RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
+
+# SSH login fix. Otherwise user is kicked off after login
+RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
+
+ENV NOTVISIBLE "in users profile"
+RUN echo "export VISIBLE=now" >> /etc/profile
+
+CMD while true; do /usr/sbin/sshd -d -D || echo "sshd exited: $?"; done

integration_tests/ssh/setup.sh

@@ -1,4 +1,16 @@
 #!/bin/bash -e
+SSH_PORT=33022
+CONTAINER_TAG="sshtest"
+CONTAINER_NAME="sshtest"
 
-# Stub setup script for ssh zgrab2 module
-echo "TODO FIXME: IMPLEMENT SSH SETUP"
+# TODO FIXME: find a pre-built container with sshd already running? This works, but if it has to build the container image, the apt-get update is very slow.
+
+# First attempt to just launch the container
+if ! docker run --rm --name $CONTAINER_NAME -itd -p $SSH_PORT:22 $CONTAINER_TAG; then
+    # If it fails, build it from ./container/Dockerfile
+    docker build -t $CONTAINER_TAG ./container
+    # Try again
+    docker run --rm --name $CONTAINER_NAME -itd -p $SSH_PORT:22 $CONTAINER_TAG
+fi
+
+# TODO: Wait on port 22?

integration_tests/ssh/test.sh

@@ -1,4 +1,19 @@
 #!/bin/bash -e
 
-# Stub test script for ssh zgrab2 module
-echo "TODO FIXME: IMPLEMENT SSH TESTS"
+SSH_PORT=33022
+CONTAINER_NAME="sshtest"
+
+# Run the SSH-specific integration tests:
+# 1. Run zgrab2 on localhost:$SSH_PORT
+
+if [ -z $ZGRAB_ROOT ] || [ -z $ZGRAB_OUTPUT ]; then
+    echo "Must set ZGRAB_ROOT and ZGRAB_OUTPUT"
+    exit 1
+fi
+
+mkdir -p $ZGRAB_OUTPUT/ssh
+
+OUTPUT_FILE="$ZGRAB_OUTPUT/ssh/ssh.json"
+
+echo "Testing SSH Version on local port $SSH_PORT..."
+echo "127.0.0.1" | $ZGRAB_ROOT/cmd/zgrab2/zgrab2 ssh -p $SSH_PORT $* > $OUTPUT_FILE

schemas/README.md

@@ -0,0 +1,45 @@
+ZGrab 2.0 schemas for zschema
+=============================
+
+## Validating
+
+[integration_tests.sh](../integration_tests.sh) automatically validates
+output from the integration tests; to manually validate a zgrab2 result,
+you can follow these steps:
+
+0. Get [zschema](https://github.com/zmap/zschema) (e.g. `git clone https://github.com/zmap/zschema`)
+1. Run the zschema validator:
+   1. Run the zschema module's main function
+   2. Pass it the `validate` command
+   3. Give the path to the zgrab2 schema [`schemas/__init__.py:zgrab2`](schemas/__init.py__)
+   4. Pass in the zgrab2 JSON file to validate
+     * ```
+       echo 127.0.0.1 | ./cmd/zgrab2/zgrab2 mysql > output.json
+       PYTHONPATH=/path/to/zschema python -m zschema validate schemas/__init__.py:zgrab2 output.json
+       ```
+
+## Adding new module schemas
+
+There are two steps to adding a new zgrab2 module schema:
+
+1. Add the module
+   a. Register the response type with the zgrab2 schema
+2. Register the module in `__init__.py`
+
+### Add the module
+
+Create your python file; if your protocol identifier (the default name
+in the result table) is *my_protocol*, name the file `my_protocol.py`
+(this allows a static schema validation from `protocol_name` to `protocol_schema`;
+unfortunately, this means that multiple scans on a single host, or scans
+using custom identifiers, will not validate).
+
+Your module should include a `SubRecord` that extends from `zgrab2.base_scan_response`,
+specifically, overridding the `result` field. See [schemas/mysql.py](schemas/mysql.py)
+for an example.
+
+### Register the module
+
+In [`schemas/__init__.py`](schemas/__init__.py), add an import for your
+module (e.g. `import my_protocol`). This will ensure that the module code
+is executed and that the response type is registered with the zgrab2 module.