Tag Archive | rust

Suricata 6.0.0 released

We are proud to announce Suricata 6.0. This major new release is the result of a year of work by the OISF development team and the Suricata community.

During this development cycle, the focus has been on:

  • stability and robustness
  • performance
  • support for new protocols like HTTP/2, MQTT and RFB
  • improvements to existing protocols DCERPC, SSH
  • extendibility
  • improvements to detection capabilities

Get the release here:
https://www.openinfosecfoundation.org/downloads/suricata-6.0.0.tar.gz

This release comes with libhtp 0.5.35 and Suricata-Update 1.2.0

Power of the community

A lot of the features and improvements have been made by community members:

  • MQTT (Sascha Steinbiss)
  • RFB (Frank Honza)
  • HASSH (Vadym Malakhatko)
  • ASN.1 Rust (Pierre Chifflier and Emmanuel Thompson)
  • cbindgen (Danny Browning)
  • nom 5 conversion (Pierre Chifflier)
  • Napatech bypass support (Phil Young)
  • MAC address logging in EVE (Sascha Steinbiss)
  • Geneve decoder (Ali Jad Khalil)
  • more detailed DNS logging (Simon Dugas)

List of git committers: Pierre Chifflier, Sascha Steinbiss, Emmanuel Thompson, Todd Mortimer, Vadym Malakhatko, Phil Young, Roland Fischer, Simon Dugas, Jason Taylor, Ali Jad Khalil, James Dutrisac, Joshua Lumb, Zach Kelly, Angelo Mirabella, Antti Tönkyrä, Carl Smith, Danny Browning,
Frank Honza, Giuseppe Longo, Ilya Bakhtin, Odin Jenseg, Stephen Donnelly,
Timo Sigurdsson, Tristan Fletcher, William Stearns, Xiaofan Wang,
Zackeus Bengtsson

Other contributors we’d like to especially thank: David Beckett for HTTP/2 testing and pcaps; Bastien Delvalle and Louis Jacotot (Telecom Nancy) for SMB evasion research and testcases.

Notable Optimizations

  • faster EVE log generation using our own Rust language JSON string builder
  • much better EVE log scaling by allowing a log file per thread
  • flow engine improvments – esp when under resource constraints

Securing Suricata

  • ASN1 handling is now entirely done in Rust code
  • DCERPC, SSH have been reimplemented in Rust
  • new protocols have been implemented in Rust
  • many fixes as a result of OSS-Fuzz testing

Rule language

  • from_end support for byte_jump keyword
  • bitmask support for byte_test keyword
  • byte_math support
  • flowbit OR support
  • pcrexform keyword: use pcre with substring capture as a transform
  • urldecode transform was added

For developers

  • Use cbindgen to create Rust-C bindings (Danny Browning)
  • initial plugin support
  • libfuzzer (OSS-Fuzz) support
  • clang-format support (Roland Fischer)

Removals

  • unified2 has been removed
  • filestore v1 has support has been removed
  • drop log

About Suricata

Suricata is a high performance Network Threat Detection, IDS, IPS and Network Security Monitoring engine. Open source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by OISF, its supporting vendors and the community.

Suricata 6.0.0 beta 1 released

We’re happy to announce Suricata 6.0.0 beta 1. This is a test version for a new major feature release of Suricata.

Originally planned to be released as a release candidate we wanted to get a few more interesting things in that are still a bit rough around the edges. So the plan is now to release 6.0RC1 early September and then the final late September.

We are hoping for some of you to take this beta and test it in your environment and report any issues to us.

Get the release here:
https://www.openinfosecfoundation.org/downloads/suricata-6.0.0-beta1.tar.gz

Major changes

– initial HTTP/2 support
– DCERPC logging
– much improved EVE logging performance
– RFB and MQTT protocol support, including detection and logging
– HASSH support
– conditional logging

Power of the community

Several features and improvements have been made by community members:

– MQTT (Sascha Steinbiss)
– RFB (Frank Honza)
– HASSH (Vadym Malakhatko)
– ASN.1 Rust (Pierre Chifflier and Emmanuel Thompson)
– cbindgen (Danny Browning)
– nom 5 conversion (Pierre Chifflier)
– Napatech bypass support (Phil Young)
– MAC address logging in EVE (Sascha Steinbiss)

List of git committers:

Pierre Chifflier, Sascha Steinbiss, Emmanuel Thompson, Todd Mortimer,
Phil Young, Vadym Malakhatko, Jason Taylor, James Dutrisac, Zach Kelly,
Joshua Lumb, Angelo Mirabella, Antti Tönkyrä, Danny Browning,
Frank Honza, Giuseppe Longo, Roland Fischer, Stephen Donnelly,
Timo Sigurdsson, Tristan Fletcher, William Stearns, Xiaofan Wang,
Zackeus Bengtsson

Notable Optimizations

– faster EVE log generation using our own Rust language JSON string builder
– much better EVE log scaling by allowing a log file per thread
– flow engine improvments – esp when under resource constraints

Removals

– unified2 has been removed
– filestore v1 has support has been removed
– drop log

Securing Suricata

– ASN1 handling is now entirely done in Rust code
– DCERPC, SSH have been reimplemented in Rust
– new protocols have been implemented in Rust

Rule language

– from_end support for byte_jump keyword
– bitmask support for byte_test keyword
– byte_math support
– flowbit OR support
– pcrexform keyword: use pcre with substring capture as a transform
– urldecode transform was added

For developers

– Use cbindgen to create Rust-C bindings (Danny Browning)
– initial plugin support
– libfuzzer (oss-fuzz) support

Forums

Join our new Forum at https://forum.suricata.io/

About Suricata

Suricata is a high performance Network Threat Detection, IDS, IPS and Network Security Monitoring engine. Open source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by OISF, its supporting vendors and the community.

Announcing Suricata 5.0.0

The OISF’s Suricata development team is proud to announce Suricata 5.0.0. This release brings many new features and improvements.

RDP, SNMP, FTP and SIP

Three new protocol parsers and loggers, all community contributions. Zach Kelly created a Rust RDP parser, while Giuseppe Longo created SIP support. Rust master Pierre Chifflier contributed SNMP support. Since RDP and SIP were merged late in our development cycle they are disabled by default in the configuration. For FTP we have added an EVE logging facility.

JA3S

After contributing JA3 support in Suricata 4.1, Mats Klepsland has been working on JA3S support. JA3S is now available to the rule language and in the TLS logging output.

Datasets

Still experimental at this time, the initial work to support datasets is part of this release. It allows matching on large amounts of data. It is controlled from the rule language and will work with any ‘sticky buffer’.

See documentation at https://suricata.readthedocs.io/en/suricata-5.0.0/rules/datasets.html

We’ve already heard of people using this with millions of IOCs.

Documentation

With the help of many community members we’ve been improving the user documentation. Please see: https://suricata.readthedocs.io/en/suricata-5.0.0/

HTTP evader

We’ve been working hard to cover the final set of HTTP evader cases. This work has mostly gone into the bundled libhtp 0.5.31.

Rust

The most visible is that our Rust support is no longer optional. We’re convinced that Rust is a perfect match for Suricata, and we plan to increase its footprint in our code base steadily. By making it mandatory we’re able to remove parallel implementations and focus fully on making the Rust code better.

Protocol Detection

The protocol detection engine has been extended to provide better accuracy as well as support for dealing with asynchronous flows. These async flows are sometimes picked up in the wrong direction and the protocol detection engine can now reverse them.

Decoder Anomaly records in EVE

A new log record type has been added: ‘anomaly’. This logs the stream and decoder events that are set by the packet decoders. This is inspired by Zeeks (Bro) ‘weird’ log.

EVE improvements

VLAN and capture interface is now part of many more EVE records, even if they are flow records or records based on flow time out.

An option to log all HTTP headers to the EVE http records has been added.

Packet Capture

Eric Leblond has been working hard to getting hardware offload support working for eBPF. On Netronome cards the eBPF based flow bypass can now be offloaded to the NIC. As eBPF is becoming a standard in the Linux space, we are hoping to see other hardware offload soon as well.

Netmap support has been rewritten so the more advanced features of netmap, such as vale switches, can be used now.

Napatech usability has been improved.

Rule language: Sticky Buffers

As discussed at the Suricon 2018 brainstorm session, a new rule keyword scheme is being introduced. It takes the existing ‘sticky buffer’ approach with new keyword names to avoid confusion. The new scheme is <proto>.<buffer>, so for example ‘http.uri’ for the URI inspection.

A number of HTTP keywords have been added.

Unified Lua inspection mixed with the sticky buffers has also been implemented.

Python 3

With Python 2’s EOL approaching, we’ve made sure that all Suricata’s python code is Python 3 compliant.

Removals

Following our deprecation policy, we have removed the following parts: the plain text dns.log, the old files-json.log and support for the Tilera architecture.

https://suricata-ids.org/about/deprecation-policy/

All tickets

Beta 1 tickets: https://redmine.openinfosecfoundation.org/versions/115

RC 1 tickets: https://redmine.openinfosecfoundation.org/versions/128

Final tickets: https://redmine.openinfosecfoundation.org/versions/129

Download

https://suricata-ids.org/download/

Call for testing: announcing Suricata 5.0.0-beta1

We’re happy to present the first beta in the upcoming Suricata 5.0 series. In 5.0 we’re making a couple of large changes.

Rust

The most visible is that our Rust support is no longer optional. We’re convinced that Rust is a perfect match for Suricata, and we plan to increase its footprint in our code base steadily. By making it mandatory we’re able to remove parallel implementations and focus fully on making the Rust code better.

Protocol Detection

The protocol detection engine has been extended to provide better accuracy as well as support for dealing with asynchronous flows. These async flows are sometimes picked up in the wrong direction and the protocol detection engine can now reverse them.

Decoder Anomaly records in EVE

A new log record type has been added: ‘anomaly’. This logs the stream and decoder events that are set by the packet decoders. This is inspired by Zeeks (Bro) ‘weird’ log.

EVE improvements

VLAN and capture interface is now part of many more EVE records, even if they are flow records or records based on flow time out.

An option to log all HTTP headers to the EVE http records has been added.

Packet Capture

Netmap support has been rewritten so the more advanced features of netmap, such as vale switches, can be used now.

Napatech usability has been improved.

Rule language: Sticky Buffers (in progress)

As discussed at the Suricon 2018 brainstorm session, a new rule keyword scheme is being introduced. It takes the existing ‘sticky buffer’ approach with new keyword names to avoid confusion. The new scheme is <proto>.<buffer>, so for example ‘http.uri’ for the URI inspection.

A number of HTTP keywords have been added.

Unified Lua inspection mixed with the sticky buffers has also been implemented.

Python 3

With Python 2’s EOL approaching, we’ve made sure that all Suricata’s python code is Python 3 compliant.

Removals

Following our deprecation policy, we have removed the following parts: the plain text dns.log, the old files-json.log and support for the Tilera architecture.

https://suricata-ids.org/about/deprecation-policy/

Many more things

https://redmine.openinfosecfoundation.org/versions/115

Time line

We’re planning the first release candidate in about a month, with the final about a month later. So early July.

Get involved

If you’re interested in helping out, we’d be happy to accept patches, documentation, test reports and other kind of feedback.

Download from:

https://www.openinfosecfoundation.org/downloads/suricata-5.0.0-beta1.tar.gz

Suricata 4.1.1 available!

suri-400x400

We are pleased to announce Suricata 4.1.1. This release fixes a number of issues found 4.1. It also adds EVE DNSv1 support for Rust builds.

Changes

  • Feature #2637: af-packet: improve error output for BPF loading failure
  • Feature #2671: Add Log level to suricata.log when using JSON type
  • Bug #2502: suricata.c ConfigGetCaptureValue – PCAP/AFP fallthrough to strip_trailing_plus
  • Bug #2528: krb parser not always parsing tgs responses
  • Bug #2633: Improve errors handling in AF_PACKET
  • Bug #2653: llc detection failure in configure.ac
  • Bug #2677: coverity: ja3 potential memory leak
  • Bug #2679: build with profiling enabled on generates compile warnings
  • Bug #2704: DNSv1 for Rust enabled builds.
  • Bug #2705: configure: Test for PyYAML and disable suricata-update if not installed.
  • Bug #2716: Stats interval are 1 second too early each tick
  • Bug #2717: nfs related panic in 4.1
  • Bug #2719: Failed Assertion, Suricata Abort – util-mpm-hs.c line 163 (4.1.x)
  • Bug #2723: dns v2 json output should always set top-level rrtype in responses
  • Bug #2730: rust/dns/lua – The Lua calls for DNS values when using Rust don’t behave the same as the C implementation.
  • Bug #2731: multiple instances of transaction loggers are broken
  • Bug #2734: unix runmode deadlock when using too many threads
  • Bundled Suricata-Update was updated to 1.0.1

Download

https://www.openinfosecfoundation.org/download/suricata-4.1.1.tar.gz

Special thanks

Jason Taylor, Eric Urban, Mats Klepsland, Pierre Chifflier

Trainings

The 2019 calendar of trainings will be out soon – check back here or follow us on Twitter (@OISFoundation) for all training announcements

Suricon

Suricon 2018 was a great success and the 2019 location has been announced: Amsterdam. Please consider becoming a sponsor! https://suricon.net/

About Suricata

Suricata is a high performance Network Threat Detection, IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by OISF, its supporting vendors and the community.

Call for testing: Suricata 4.1rc2 released

Suricata 4.1rc2 is ready for testing. We’re hoping that this will be the final release candidate so that 4.1 can be released just before Suricon next month.

Main new features are inclusion of the protocols SMBv1/2/3, NFSv4, Kerberos,FTP, DHCP, IKEv2, as well as improvements on Linux capture side via AF_PACKET XDP support and on Windows IPS side via WinDivert. The growth of Rust usage inside Suricata continues as most of the new protocols have been implemented in Rust.

Most important change for going from RC1 to RC2 is that we have enabled Rust support by default. If Rust is installed, it will be used.

Protocol updates

  • SMBv1/2/3 parsing, logging, file extraction
  • TLS 1.3 parsing and logging (Mats Klepsland)
  • JA3 TLS client fingerprinting (Mats Klepsland)
  • TFTP: basic logging (Pascal Delalande and Clément Galland)
  • FTP: file extraction
  • Kerberos parser and logger (Pierre Chifflier)
  • IKEv2 parser and logger (Pierre Chifflier)
  • DHCP parser and logger
  • Flow tracking for ICMPv4
  • Initial NFS4 support
  • HTTP: handle sessions that only have a response, or start with a response
  • HTTP Flash file decompression support (Giuseppe Longo)

Output and logging

  • File extraction v2: deduplication; hash-based naming; json metadata and cleanup tooling
  • Eve metadata: from rules (metadata keyword) and traffic (flowbits etc)
  • Eve: new more compact DNS record format (Giuseppe Longo)
  • Pcap directory mode: process all pcaps in a directory (Danny Browning)
  • Compressed PCAP logging (Max Fillinger)
  • Expanded XFF support (Maurizio Abba)
  • Community Flow Id support (common ID between Suricata and Bro/Zeek)

Packet Capture

  • AF_PACKET XDP and eBPF support for high speed packet capture
  • Windows IPS: WinDivert support (Jacob Masen-Smith)

Misc

  • Windows: MinGW is now supported
  • Detect: transformation keyword support
  • Bundled Suricata-Update
  • Per device multi-tenancy

Major changes since 4.1rc1

  • Rust support is enabled by default
  • Community Flow Id support (common ID between Suricata and Bro/Zeek)
  • Updates and fixes for dealing with SegmentSmack/FragmentSmack
  • Update Suricata-Update to 1.0.0rc2

Get paid to work on Suricata!

Enjoying the testing? Or want to help out with other parts of the project?
We are looking for people, so reach out to us if you’re interested.

Special thanks

Mats Klepsland, Jason Taylor, Maurizio Abba, Konstantin Klinger, Giuseppe Longo, Danny Browning, Hilko Bengen, Jacob Masen-Smith, Pascal Delalande, Travis Green, Christian Kreibich

Trainings

Check out the latest training offerings at https://suricata-ids.org/training/

SuriCon

SuriCon 2018 Vancouver next month, you can still join! https://suricon.net/agenda-vancouver/

About Suricata

Suricata is a high performance Network Threat Detection, IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

Suricata 4.0.4 available!

suri-400x400

We are pleased to announce Suricata 4.0.4.  This is a security update fixing a number of security issues, as well as a fair number of regular issues.

Security

CVE-2018-6794 was requested for issue #2440

Changes

  • Bug #2306: suricata 4 deadlocks during failed output log reopening
  • Bug #2361: rule reload hangup
  • Bug #2389: BUG_ON asserts in AppLayerIncFlowCounter (4.0.x)
  • Bug #2392: libhtp 0.5.26 (4.0.x)
  • Bug #2422: [4.0.3] af_packet: a leak that (possibly) breaks an inline channel
  • Bug #2438: various config parsing issues
  • Bug #2439: Fix timestamp offline when pcap timestamp is zero (4.0.x)
  • Bug #2440: stream engine bypass issue (4.0.x)
  • Bug #2441: der parser: bad input consumes cpu and memory (4.0.x)
  • Bug #2443: DNP3 memcpy buffer overflow (4.0.x)
  • Bug #2444: rust/dns: Core Dump with malformed traffic (4.0.x)
  • Bug #2445: http bodies / file_data: thread space creation writing out of bounds

Download

https://www.openinfosecfoundation.org/download/suricata-4.0.4.tar.gz

Special thanks

Wolfgang Hotwagner, Kirill Shipulin, Pierre Chifflier, Alexander Gozman, Martin Natano, Maurizio Abba, Nick Price, Philippe Antoine, AFL

SuriCon 2018

Call for presentations is open and tickets for SuriCon 2018 are available: https://suricon.net/

About Suricata

Suricata is a high performance Network Threat Detection, IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

Suricata 4.0.1 available!

suri-400x400

We are pleased to announce Suricata 4.0.1.  This is regular bug fix release fixing various issues. Also added is much improved Napatech support.

Changes

  • Feature #2114: Redis output: add RPUSH support
  • Feature #2152: Packet and Drop Counters for Napatech
  • Bug #2050: TLS rule mixes up server and client certificates
  • Bug #2064: Rules with dual classtype do not error
  • Bug #2074: detect msg: memory leak
  • Bug #2102: Rules with dual sid do not error
  • Bug #2103: Rules with dual rev do not error
  • Bug #2151: The documentation does not reflect current suricata.yaml regarding cpu-affinity
  • Bug #2194: rust/nfs: sigabrt/rust panic – 4.0.0-dev (rev fc22943)
  • Bug #2197: rust build with lua enabled fails on x86
  • Bug #2201: af_packet: suricata leaks memory with use-mmap enabled and incorrect BPF filter
  • Bug #2207: DNS UDP “Response” parsing recording an incorrect value
  • Bug #2208: mis-structured JSON stats output if interface name is shortened
  • Bug #2226: improve error message if stream memcaps too low
  • Bug #2228: enforcing specific number of threads with autofp does not seem to work
  • Bug #2244: detect state uses broken offset logic (4.0.x)

Download

https://www.openinfosecfoundation.org/download/suricata-4.0.1.tar.gz

Special thanks

Qidu Sy, Phil Young – Napatech, Mats Klepsland, Sascha Steinbiss, Alexander Gozman, Derek Kingsbury, Julian Wecke, Pierre Chifflier, Jason Taylor

Trainings

Conference attendees get a 20% discount!

SuriCon 2017

Less than one month to SuriCon 2017! Come meet the Suricata community and development team to discuss all things Suricata at the third edition of the annual Suricata Conference. SuriCon 2017 will be next month in Prague: https://suricon.net

About Suricata

Suricata is a high performance Network Threat Detection, IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.

Rust and Suricata

In the newly released Suricata 4.0, one of the major new features is integration of Rust. In the words of the Rust Language project, “Rust is a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety.”

For those of you who were at SuriCon DC, the topic of Rust isn’t new. Pierre Chiffliers’  talk ‘Securing Security Tools’ described why switching existing C/C++ tools to partly use Rust language parsers makes sense. See his slides here: pdf. After his talk, we have started looking into the proof of concept proposed by Pierre.

Rust was new to us so the process involved learning the language.

Major Components

  1. rust language: a safe systems programming language that compiles to native code and can be linked into a C program
  2. nom: a rust macro parser generator
  3. suricata – rust layer: connection between the rust code and the Suricata API’s

Rust Usage in Suricata

Rust parsers might be used for many things:

  • simple header parsers where we’d use a rust/nom parser for a header like DNS
  • specific data types, I’m thinking mostly about ASN.1 and mime here as those are notorious for vulnerabilities
  • full application layer parsing and logging in Suricata

Experiment

After the initial testing and playing by Jason Ish and Victor Julien, we have decided to move forward with Rust. However, as the language is young and our understanding of it is even younger, we will consider it experimental at first.

The experimental phase will have to teach us a couple of things:

  1. how does it work wrt stability and performance
  2. what does it take to support Suricata with Rust extensions
  3. how mature is the Rust ecosystem

Depending on our experiences we’re planning to take between 6 and 12 months for this experimental phase. As we’re considering this experimental we’re going to be quite liberal with regards to updates to the Rust code inside our stable branch.

The initial 4.0 code contains the following parts:

  • Re-implementation of the DNS parser, which is mostly stateless.
  • A NFS parser with logging, file extraction and basic detection integration.
  • A NTP parser done by Pierre that for a significant part is an external crate

Some initial thoughts

As predicted by many, the learning curve for Rust is indeed quite steep. It’s not just the Rust language itself, which has some novel but tricky concepts around lifetimes. It’s also the ‘nom’ framework which although generally easy to work with, can emit really cryptic compiler error messages if small mistakes (like a missing comma) are made.

Rust is really nice though. In our testing it performs well, and its really nice to have code that doesn’t segfault. In Rust, if the compiler accepts the code, it usually works. Also, conditions that may lead to undefined state in C are detected by Rust which will cause the program to to abort, instead of continuing, possibly in a condition where exploitation is possible.

Pierre’s ‘Rusticata’ work

In our initial integration we’re not using much of Pierre’s work yet. The main reasons for this are:

  • Pierre is implementing parsers as external modules (Crates in Rust speak) and we want to limit the number of external dependencies we introduce
  • learning the Rust language and ecosystem all but required that we did everything ourselves, even if it meant redoing what Pierre already did (even in less efficient ways sometimes)

However we have added another layer of ‘experimental’ for including Pierre’s work step by step.

By using the –enable-rust-experimental configure flag the parser Pierre has written are enabled.

Timeline

  • Summer 2017: Suricata 4.0 was released with experimental Rust support, implementing DNS, NFS and NTP
  • Late Fall 2017: Suricata 4.1 with still experimental Rust support, likely adding a few protocols
  • Spring 2018: Suricata 5.0 with default non-experimental (and probably mandatory) Rust support

Trying our Rust support

If you want to play with the current support, here is a page with installation instructions.

On a modern distro, it’s really as simple as installing rustc and cargo, followed by passing –enable-rust to configure. Tested on Ubuntu, Fedora and FreeBSD.

Training

In September we’re offering the third edition of our annual Suricata developer training. In this 5-day event we’re teaching how to extend Suricata. We’re planning to spend about one day on covering the Rust integration.

See: https://suricata_events.eventbrite.com/

Common questions

Q: Are you moving to Rust completely?

A: No, not anytime soon. Time will tell how this experiment evolves.

Q: Does it make Suricata harder to use?

A: No, to end-users the Rust support is transparent other than installing the build dependencies rustc and cargo.

Q: Does it make Suricata harder to compile?

A: No, not on modern distributions. All you need is the rust compiler and cargo.

Q: Where can I learn more?

A: We’re offering a developer training at https://suricata_events.eventbrite.com/

Suricata 4.0 released!

We are thrilled to announce Suricata 4.0. This is a major new release, improving detection capabilities, adding new output options and more protocols.suri-400x400

Improved Detection

Based on valuable feedback from the rule writing teams at Emerging Threats and Positive Technologies we’ve added and improved many rule keywords for inspecting HTTP, SSH and other protocols. TLS additions were contributed by Mats Klepsland at NorCERT, including decoding, logging and matching on TLS serial numbers. Additionally, Suricata now allows rule writers to specify who’s the target in a signature. This information is used in EVE JSON logging to give more context with alerts.

TLS improved, NFS added

More on the TLS side: A major new feature is support for STARTTLS in SMTP and FTP. TLS sessions will now be logged in these cases. More goodness from Mats Klepsland. Also, TLS session resumption logging is now supported thanks to the work of Ray Ruvinskiy. Additional TLS logging improvements were done by Paulo Pacheco.

NFS decoding, logging and file extraction was added as part of the experimental Rust support. Read on for more information about Rust.

More EVE JSON

EVE is extended in several ways:

  • in the case of encapsulated traffic both the inner and outer ip addresses and ports are logged
  • the ‘vars’ facility logs flowbits and other vars. This can also be used to log data extracted from traffic using a PCRE statement in rules
  • EVE can now be rotated based on time
  • EVE was extended to optionally log the HTTP request and/or response bodies
  • the (partial) flow record is added to alert records.

The ‘vars’ facility is one of the main improvements here, as it is now possible for a signature to accurately extract information for logging. For instance, a signature can extract an advertised software version or other information such as the recipient of an email. [https://blog.inliniac.net/2016/12/20/suricata-bits-ints-and-vars/]

First Step into a Safer Future

This is the first release in which we’ve implemented parts in the Rust language using the Nom parser framework. This work is inspired by Pierre Chiffliers’ (ANSSI), talk at SuriCon 2016 (pdf). By compiling with –enable-rust you’ll get a basic NFS parser and a re-implementation of the DNS parser. Feedback on this is highly appreciated.

The Rust support is still experimental, as we are continuing to explore how it functions, performs and what it will take to support it in the community. Additionally we included Pierre Chiffliers Rust parsers work. This uses external Rust parser ‘crates’ and is enabled by using –enable-rust-experimental. Initially this adds a NTP parser.

Under the Hood

A major TCP stream engine update is included. This should lead to better performance and less configuration, especially in IPS mode. First steps in TCP GAP recovery were taken, with implementations for DNS and NFS.

For developers, this release makes extending the detection engine with high performance keywords a lot easier. Adding a new high performance keyword using multi pattern matching does now requires only a few lines of code.

Documentation

David Wharton at SecureWorks has created a section in the documentation for rule writers who have a background in Snort. It documents changes that are relevant for writing rules.

Next steps

Based on the feedback we’ll get we’re expecting to do a 4.0.1 release in a month or so. Then we’ll start work on the next major release, which is 4.1. This is planned for late fall, ETA before SuriCon in Prague.

Feature tickets

  • Feature #806: Implement STARTTLS support
  • Feature #2006: tls: decode certificate serial number
  • Feature #1969: TLS transactions with session resumption are not logged
  • Feature #2129: nfs: parser, logger and detection
  • Feature #2130: dns: rust parser with stateless behaviour
  • Feature #2131: nfs: implement GAP support
  • Feature #2163: ntp parser
  • Feature #2164: rust: external parser crate support
  • Feature #2077: Additional HTTP Header Contents and Negation
  • Feature #2011: eve.alert: print outside IP addresses on alerts on traffic inside tunnels
  • Feature #2095: eve: http body in alert event
  • Feature #1978: Using date in logs name
  • Feature #1998: eve.tls: custom TLS logging
  • Feature #2046: Support custom file permissions per logger
  • Feature #2123: unix-socket: additional runmodes
  • Feature #2132: eve: flowbit and other vars logging
  • Feature #2156: Add app_proto or partial flow entry to alerts
  • Feature #744: Teredo configuration
  • Feature #2061: lua: get timestamps from flow
  • Feature #1953: lua: expose flow_id
  • Feature #1748: lua: expose tx in alert lua scripts
  • Feature #1636: Signal rotation of unified2 log file without restart
  • Feature #2133: unix socket: add/remove hostbits
  • Feature #805: Add support for applayer change

For all other closed tickets please see the full changelog of 4.0.

Download

https://www.openinfosecfoundation.org/download/suricata-4.0.0.tar.gz

Special thanks

Mats Klepsland – for his major contributions: many EVE and TLS features

Pierre Chifflier – for paving the way for the Rust experiment and being very helpful while learning Rust and Nom.

Additionally: Abdullah Ada, Jérémy Beaume, Sebastian Garcia, Alexander Gozman, Giuseppe Longo, Paulo Pacheco, Selivanov Pavel, Ray Ruvinskiy, Peter Sanders, David Wharton, Jon Zeolla, the AFL project and Coverity Scan.

Suricata Trainings and Events

We have several community events and trainings on the calendar and in the works for 2017… here are some of the highlights:

  • 5-Day Developer Deep Dive Training – Sept 11 – 15, 2017, Cork, Ireland – led by Victor Julien, Eric Leblond, and Jason Ish
  • Rule Writing Training @ DerbyCon – Sept 20 – 24, 2017 – SOLD OUT!
  • Rule Writing Training @ SuriCon – Nov 13 – 14, 2017
  • 2-Day Suricata Training @ SuriCon – Nov 13 – 14, 2017
  • SuriCon 2017 – Nov 15 – 17, 2017, Prague

Details and registration for all our events can be found at https://suricata_events.eventbrite.com. Don’t delay as space is limited.

We also offer custom training events for your team – contact us at info@oisf.net for details.

About Suricata

Suricata is a high performance Network Threat Detection, IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF, its supporting vendors and the community.