Homegrown Cyber Threat Intelligence With STIX2 and Couchbase

Threat intelligence is now an important part of incident response and vulnerability management. But can everyone make use of it? In this article we show you how to create and archive threat intelligence on our own, without using a vendor's threat intelligence platform.

A brief definition of Cyber Threat Intelligence from the Center for Internet Security:

Cyber threat intelligence is what cyber threat information becomes once it has been collected, evaluated in the context of its source and reliability, and analyzed through rigorous and structured tradecraft techniques by those with substantive expertise and access to all-source  information.

This structured information can be used for various tasks, the UK's NCSC distinguishes between four types of intel:

• Tactical → TTPs (Tecniques, Tactics and Procedures)
• Technical → IoCs (Indicators of Compromise)
• Operational → How specific attacks are carried out
• Strategic → C-level information for decision making

But the perceived reality for firms is different: vendors often commoditize threat intel into a paid-for incoming data flow to be ingested by SIEMs for detection.

The main use case of TI, according to this Threat Intelligence Report, is in fact incident response via SIEM systems.

Incident response is a pretty powerful use case indeed, but what about all the other tasks we should use intel for? The second main use case is vulnerability management, that is kind of an umbrella term for all the operations around configurations, patches and vulnerabilities, typically managed by CISOs, SOCs and IT departments everywhere. On the practical level they do a series of activities like IP or email reputation assessment, daily CVE alerts and manual investigation on security related events.

These tasks could use the help of TI very well. For example:

• Collect all the vulnerabilities or attacks that affect or have affected our systems for threat Modeling and remediation.
• Obtain all the intel related to your industry to correlate during threat hunting.
• Describe and reference weaknesses to test your products against in DevSecOps.
• Develop a system to F3AD (Find, Fix, Finish, Exploit, Analyze and Disseminate).

Instead of heading up to some vendor's Threat Intelligence Platform, let's find out if we can create and archive intel on our own.

STIX

Structured Threat Information Expression (STIX™) is a language and  serialization format used to exchange cyber threat intelligence (CTI).

Thanks to OASIS we have this nice standard format to create and exchange intel as text files and make it machine-readable. Basically STIX tells you how to assemble the pieces of information you collect to compose an intel document. TAXII is the respective standard for serving STIX files through a web service.

STIX reached version 2 and is in its transition phase to evolve all the components from version 1, with the biggest changes being:

• the use of JSON instead of the much hated XML, with a great leap on human readability
• the introduction of relations (SRO) in addition to objects (SDO) to better describe how the various entities are linked to one another.

STIX2 defines twelve objects (like observable, malware, actor, campaign etc), which can be bundled together to describe any threat at any level of detail.

With Python APIs even the most rusty Python programmer can easily generate valid STIX with a few lines

from stix2 import Bundle
from stix2 import Vulnerability
from stix2 import AttackPattern
from stix2 import CourseOfAction
from stix2 import ExternalReference

vulnName="Test CVE-2019-1234"
vulnDescription="Test CVE for STIX"
cve="CVE-2019-1234"
vulnLabel="testLabel"

ttpName="CAPEC-123"
ttpDescription="Test TTP"
capec="CAPEC-123"
ttpLabel="testLabel"

courseOfActionName="Update to latest version"

vulnerability = Vulnerability(name=vulnName,
                      description=vulnDescription,
                      labels=[vulnLabel],
                      external_references=[ExternalReference(source_name="cve",external_id=cve)])

attackPattern = AttackPattern(name=ttpName,
                      description=ttpDescription,
                      labels=[ttpLabel],
                      external_references=[ExternalReference(source_name="capec",external_id=capec)])

courseOfAction = CourseOfAction(name=courseOfActionName)

bundle = Bundle(vulnerability, attackPattern, courseOfAction)
print(bundle)

which print out nice, readable json

{
    "type": "bundle",
    "id": "bundle--515dc864-e046-4692-b522-1e392c6a8eab",
    "spec_version": "2.0",
    "objects": [
        {
            "type": "vulnerability",
            "id": "vulnerability--c5d9efe8-81c7-4210-afbe-ed7aab9f7f36",
            "created": "2019-04-04T19:44:36.284Z",
            "modified": "2019-04-04T19:44:36.284Z",
            "name": "Test CVE-2019-1234",
            "description": "Test CVE for STIX",
            "labels": [
                "testLabel"
            ],
            "external_references": [
                {
                    "source_name": "cve",
                    "external_id": "CVE-2019-1234"
                }
            ]
        },
        {
            "type": "attack-pattern",
            "id": "attack-pattern--ec1f9bb1-a898-49c4-90cd-8ffda6080182",
            "created": "2019-04-04T19:44:36.288Z",
            "modified": "2019-04-04T19:44:36.288Z",
            "name": "CAPEC-123",
            "description": "Test TTP",
            "labels": [
                "testLabel"
            ],
            "external_references": [
                {
                    "source_name": "capec",
                    "external_id": "CAPEC-123"
                }
            ]
        },
        {
            "type": "course-of-action",
            "id": "course-of-action--68b087ec-fec8-4e94-aa02-b2b02ce1d972",
            "created": "2019-04-04T19:44:36.289Z",
            "modified": "2019-04-04T19:44:36.289Z",
            "name": "Update to latest version"
        }
    ]
}

So you can produce STIX files starting from scratch or from existing feeds. Note that some OSINT exchanges can act as TAXII servers and serve feeds as STIX, even though most are stuck with version 1.

What to do with this json? To make the intel actionable we must be capable of archiving, researching and extracting documents. Fortunately, we have document-oriented json-native database systems.

Couchbase

Using Couchbase (that is a project combining CouchDB and Membase) has many advantages:

• You query the json documents and its fields natively, leveraging the standard format of STIX for consistency
• You store a STIX2 document as-is, without altering its structure*. This is useful to further exchange documents after you store them, without having to re-compose them or archive originals separately
• You can edit or enrich documents and keep track of revisions
• Has a fully featured Web UI and a well documented API
• Unlike CouchDB it has a complete full text search engine, that is way easier and faster than SQL, which you can use as well.
• Unlike Elasticsearch it is light on your server, and you can just copy-paste json to and from documents, keeping their structure untouched.
• It comes in a free Community Edition and a paid Enterprise Edition
• Has the full set of scalability-readiness, redundancy and stability to be ready for production out of the box.

* Actually, you alter the structure a little, by adding two fields for id and revisions. This causes a couple of warnings in stix validation, but a) you can access the  json without the metadata and b) you can safely either keep them there (since they don't alter the readability) or remove them upon export.

You fire up a Couchbase CE instance in a matter of minutes with its Docker image from Docker Hub. Packages for Debian/Ubuntu and other distros are available as well.

Updating the code above, we can use the Python API/SDK to forward the STIX directly to a Couchbase bucket

from couchbase.cluster import Cluster
from couchbase.cluster import PasswordAuthenticator
import json

[...]

cluster = Cluster('couchbase://localhost')
authenticator = PasswordAuthenticator('Administrator', 'password')
cluster.authenticate(authenticator)
cb = cluster.open_bucket('stix2')

cb.upsert("Test Vulnerability",json.loads(str(bundle)))

This closes the loop for threat intel production, storage and research.

Conclusions

Developing a tailored, flexible and solid management cycle for TI is a very enabling capability for many security operations, in various contexts. Doing that confidently and rapidly in-house is the first step toward adoption and promotes widespread OSINT, unleashing the true force of Cyber Threat Intelligence.