[go: up one dir, main page]
Skip to content

Commit

Permalink
nDPI: use go-dpi flow tracking (mushorg#39)
Browse files Browse the repository at this point in the history 
* nDPI: use go-dpi flow tracking

* Use go-dpi flows instead of also tracking flows inside nDPI. (closes mushorg#34)

* Bring readme up to date. (closes mushorg#35)

* Readme wording changes

Signed-off-by: Nikos Filippakis <aesmade@gmail.com>
  • Loading branch information
@nikofil @glaslos
nikofil authored and glaslos committed Aug 9, 2017
1 parent 899a860 commit 89e674a
Show file tree
Hide file tree
Showing 6 changed files with 142 additions and 108 deletions.
67 changes: 37 additions & 30 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,69 +4,76 @@

# go-dpi

go-dpi is an open source Go library for application layer protocol identification of traffic flows. In addition to its own heuristic methods, it contains wrappers for other popular and well-established libraries that also perform protocol identification, such as nDPI and libprotoident. It aims to provide a simple, easy-to-use interface and the capability to be easily extended by a developer with new detection methods and protocols.
go-dpi is an open source Go library for application layer protocol identification of traffic flows. In addition to its own heuristic methods, it contains wrappers for other popular and well-established libraries that also perform protocol identification, such as nDPI and libprotoident. It aims to provide a simple, easy-to-use interface and the capability to be extended by a developer with new detection methods and protocols.

It attempts to classify flows to different protocols regardless of the ports used. This makes it possible to detect protocols on non-standard ports, which is ideal for honeypots, as malware might often try and throw off detection methods by using non-standard and unregistered ports. Also, with its layered architecture, it aims to be fast in its detection, only using heavier classification methods when the simpler ones fail.
It attempts to classify flows to different protocols regardless of the ports used. This makes it possible to detect protocols on non-standard ports, which is ideal for honeypots, as malware might often try and throw off detection methods by using non-standard and unregistered ports. Also, with its layered architecture, it aims to be fast in its detection, only using heavier classification methods when the faster ones fail.

It is being developed in the context of the Google Summer of Code 2017 program, under the mentorship of The Honeynet Project.

Please read the project's [Wiki page](https://github.com/mushorg/go-dpi/wiki) for more information.

For documentation, please check out the [godoc reference](https://godoc.org/github.com/mushorg/go-dpi).

## Example usage

The library and the modules APIs aim to be very simple and straightforward to use. The library relies on the [gopacket](https://godoc.org/github.com/google/gopacket) library and its Packet structure. Once you have a Packet in your hands, it's very easy to classify it with the library.
First you need a flow that contains the packet. There is a helper function for constructing a flow from a single packet. Simply call:
First of all you need to initialize the library. You can do that by calling:
```go
godpi.Initialize()
```

The `Initialize` method initializes all the selected modules in the library, by calling the `Initialize` method that they provide. It also creates the cache that is used to track the flows, which outdates unused flows after some minutes.

Then, you need a flow that contains the packet. You can get the flow a packet belongs to with the following call:

```go
flow := godpi.CreateFlowFromPacket(&packet)
flow, isNew := godpi.GetPacketFlow(&packet)
```

Afterwards, classifying the flow can be done by simply calling:
That call returns the flow, as well as whether that flow is a new one (this packet is the first in the flow) or an existing one.

Afterwards, classifying the flow can be done by calling:

```go
proto, source := classifiers.ClassifyFlow(flow)
result := godpi.ClassifyFlow(flow)
```

This returns the guess protocol by the classifiers as well as the source (which in this case will always be go-dpi).
This returns the protocol guessed by the classifiers as well as the source, e.g. go-dpi or one of the wrappers.

The same thing applies for wrappers. However, for wrappers you also have to call the initialize function, and the destroy function before your program exits. All in all, the following is enough to run the wrappers:
Finally, once you are done with the library, you should free the used resources by calling:

```go
wrappers.InitializeWrappers()
defer wrappers.DestroyWrappers()
proto, source = wrappers.ClassifyFlow(flow)
godpi.Destroy()
```

A minimal example application is included below. It uses both the classifiers and wrappers to classify a simple packet capture file. Note the helpful `godpi.ReadDumpFile` function that simply returns a channel with all the packets in the file.
`Destroy` frees all the resources that the library is using, and calls the `Destroy` method of all the activated modules. It is essentially the opposite of the `Initialize` method.

A minimal example application is included below. It uses the library to classify a packet capture file, located at `/tmp/http.cap`. Note the helpful `godpi.ReadDumpFile` function that returns a channel with all the packets in the file.

```go
package main

import "fmt"
import "github.com/mushorg/go-dpi"
import "github.com/mushorg/go-dpi/classifiers"
import "github.com/mushorg/go-dpi/wrappers"
import (
"fmt"
"github.com/mushorg/go-dpi"
"github.com/mushorg/go-dpi/types"
"github.com/mushorg/go-dpi/utils"
)

func main() {
packets, err := godpi.ReadDumpFile("/tmp/http.cap")
wrappers.InitializeWrappers()
defer wrappers.DestroyWrappers()
godpi.Initialize()
defer godpi.Destroy()
packets, err := utils.ReadDumpFile("/tmp/http.cap")
if err != nil {
fmt.Println(err)
} else {
for packet := range packets {
flow := godpi.CreateFlowFromPacket(&packet)
proto, source := classifiers.ClassifyFlow(flow)
if proto != godpi.Unknown {
fmt.Println(source, "detected protocol", proto)
} else {
fmt.Println("No detection made by classifiers")
}
proto, source = wrappers.ClassifyFlow(flow)
if proto != godpi.Unknown {
fmt.Println(source, "detected protocol", proto)
flow, _ := godpi.GetPacketFlow(&packet)
result := godpi.ClassifyFlow(flow)
if result.Protocol != types.Unknown {
fmt.Println(result.Source, "detected protocol", result.Protocol)
} else {
fmt.Println("No detection made by wrappers")
fmt.Println("No detection was made")
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions godpi.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ func SetModules(modules []Module) {
copy(moduleList, modules)
}

// GetPacketFlow returns a Flow for the given type. If another packet has been
// processed before that was part of the same communication flow, the same
// GetPacketFlow returns a Flow for the given packet. If another packet has
// been processed before that was part of the same communication flow, the same
// Flow will be returned, with the new packet added. Otherwise, a new Flow
// will be created with only this packet.
// The function also returns whether the returned Flow is a new one, and not
Expand Down
75 changes: 47 additions & 28 deletions modules/wrappers/nDPI_wrapper.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import "C"
import (
"unsafe"

"github.com/google/gopacket"
"github.com/mushorg/go-dpi/types"
"github.com/pkg/errors"
)
Expand Down Expand Up @@ -36,7 +37,9 @@ const NDPIWrapperName = "nDPI"
type NDPIWrapperProvider struct {
ndpiInitialize func() int32
ndpiDestroy func()
ndpiPacketProcess func(int, int, int, []byte) int32
ndpiPacketProcess func(gopacket.Packet, unsafe.Pointer) int32
ndpiAllocFlow func(gopacket.Packet) unsafe.Pointer
ndpiFreeFlow func(unsafe.Pointer)
}

// NDPIWrapper is the wrapper for the nDPI deep inspection library,
Expand All @@ -45,21 +48,38 @@ type NDPIWrapper struct {
provider *NDPIWrapperProvider
}

// getPacketNdpiData is a helper that extracts the PCAP packet header and packet
// data pointer from a gopacket.Packet, as needed by nDPI.
func getPacketNdpiData(packet *gopacket.Packet) (pktHeader C.struct_pcap_pkthdr, pktDataPtr *C.u_char) {
seconds := (*packet).Metadata().Timestamp.Second()
capLen := (*packet).Metadata().CaptureLength
packetLen := (*packet).Metadata().Length
pktDataSlice := (*packet).Data()
pktHeader.ts.tv_sec = C.__time_t(seconds)
pktHeader.ts.tv_usec = 0
pktHeader.caplen = C.bpf_u_int32(capLen)
pktHeader.len = C.bpf_u_int32(packetLen)
pktDataPtr = (*C.u_char)(unsafe.Pointer(&pktDataSlice[0]))
return
}

// NewNDPIWrapper constructs an NDPIWrapper with the default implementation
// for its methods.
func NewNDPIWrapper() *NDPIWrapper {
return &NDPIWrapper{
provider: &NDPIWrapperProvider{
ndpiInitialize: func() int32 { return int32(C.ndpiInitialize()) },
ndpiDestroy: func() { C.ndpiDestroy() },
ndpiPacketProcess: func(seconds, capLen, packetLen int, pktData []byte) int32 {
var pktHeader C.struct_pcap_pkthdr
pktHeader.ts.tv_sec = C.__time_t(seconds)
pktHeader.ts.tv_usec = 0
pktHeader.caplen = C.bpf_u_int32(capLen)
pktHeader.len = C.bpf_u_int32(packetLen)
pktDataPtr := unsafe.Pointer(&pktData[0])
return int32(C.ndpiPacketProcess(&pktHeader, (*C.u_char)(pktDataPtr)))
ndpiPacketProcess: func(packet gopacket.Packet, ndpiFlow unsafe.Pointer) int32 {
pktHeader, pktDataPtr := getPacketNdpiData(&packet)
return int32(C.ndpiPacketProcess(&pktHeader, pktDataPtr, ndpiFlow))
},
ndpiAllocFlow: func(packet gopacket.Packet) unsafe.Pointer {
pktHeader, pktDataPtr := getPacketNdpiData(&packet)
return C.ndpiGetFlow(&pktHeader, pktDataPtr)
},
ndpiFreeFlow: func(ndpiFlow unsafe.Pointer) {
C.ndpiFreeFlow(ndpiFlow)
},
},
}
Expand All @@ -79,25 +99,24 @@ func (wrapper *NDPIWrapper) DestroyWrapper() error {
// ClassifyFlow classifies a flow using the nDPI library. It returns the
// detected protocol and any error.
func (wrapper *NDPIWrapper) ClassifyFlow(flow *types.Flow) (types.Protocol, error) {
for _, ppacket := range flow.Packets {
packet := *ppacket
seconds := packet.Metadata().Timestamp.Second()
capLen := packet.Metadata().CaptureLength
packetLen := packet.Metadata().Length
pktDataSlice := packet.Data()
ndpiProto := (*wrapper.provider).ndpiPacketProcess(seconds, capLen, packetLen, pktDataSlice)
if proto, found := ndpiCodeToProtocol[uint32(ndpiProto)]; found {
return proto, nil
} else if ndpiProto < 0 {
switch ndpiProto {
case -10:
return types.Unknown, errors.New("nDPI wrapper does not support IPv6")
case -11:
return types.Unknown, errors.New("Received fragmented packet")
case -12:
return types.Unknown, errors.New("Error creating nDPI flow")
default:
return types.Unknown, errors.New("nDPI unknown error")
if len(flow.Packets) > 0 {
ndpiFlow := (*wrapper.provider).ndpiAllocFlow(*flow.Packets[0])
defer (*wrapper.provider).ndpiFreeFlow(ndpiFlow)
for _, ppacket := range flow.Packets {
ndpiProto := (*wrapper.provider).ndpiPacketProcess(*ppacket, ndpiFlow)
if proto, found := ndpiCodeToProtocol[uint32(ndpiProto)]; found {
return proto, nil
} else if ndpiProto < 0 {
switch ndpiProto {
case -10:
return types.Unknown, errors.New("nDPI wrapper does not support IPv6")
case -11:
return types.Unknown, errors.New("Received fragmented packet")
case -12:
return types.Unknown, errors.New("Error creating nDPI flow")
default:
return types.Unknown, errors.New("nDPI unknown error")
}
}
}
}
Expand Down
91 changes: 45 additions & 46 deletions modules/wrappers/nDPI_wrapper_impl.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@

#include "nDPI_wrapper_impl.h"

#define MAX_NDPI_FLOWS 20000000
#define ETH_P_IP 0x0800

#define ERR_IPV6_NOT_SUPPORTED 10
Expand All @@ -30,8 +29,6 @@ static u_int32_t detection_tick_resolution = 1000;

static u_int32_t size_id_struct = 0;
static u_int32_t size_flow_struct = 0;
static struct ndpi_flow *ndpi_flows_root = NULL;
static u_int32_t ndpi_flow_count = 0;

// flow tracking
typedef struct ndpi_flow {
Expand Down Expand Up @@ -89,8 +86,6 @@ static void ndpi_flow_freer(void *node) {

extern void ndpiDestroy(void)
{
ndpi_tdestroy(ndpi_flows_root, ndpi_flow_freer);
ndpi_flows_root = NULL;
ndpi_exit_detection_module(ndpi_struct);
}

Expand Down Expand Up @@ -120,6 +115,7 @@ static struct ndpi_flow *get_ndpi_flow(const struct pcap_pkthdr *header,
u_int16_t upper_port;
struct ndpi_flow flow;
void *ret;
struct ndpi_flow *newflow;

if (ipsize < 20)
return NULL;
Expand Down Expand Up @@ -164,59 +160,36 @@ static struct ndpi_flow *get_ndpi_flow(const struct pcap_pkthdr *header,
upper_port = 0;
}

flow.protocol = iph->protocol;
flow.lower_ip = lower_ip;
flow.upper_ip = upper_ip;
flow.lower_port = lower_port;
flow.upper_port = upper_port;
flow.first_packet_time_sec = header->ts.tv_sec;
flow.first_packet_time_usec = header->ts.tv_usec;
newflow = (struct ndpi_flow*)malloc(sizeof(struct ndpi_flow));

ret = ndpi_tfind(&flow, (void*)&ndpi_flows_root, node_cmp);

if(ret == NULL) {
if (ndpi_flow_count == MAX_NDPI_FLOWS) {
printf("ERROR: maximum flow count (%u) has been exceeded\n", MAX_NDPI_FLOWS);
exit(-1);
} else {
struct ndpi_flow *newflow = (struct ndpi_flow*)malloc(sizeof(struct ndpi_flow));

if(newflow == NULL) {
printf("[NDPI] %s(1): not enough memory\n", __FUNCTION__);
return NULL;
}

memset(newflow, 0, sizeof(struct ndpi_flow));
newflow->protocol = iph->protocol;
newflow->lower_ip = lower_ip, newflow->upper_ip = upper_ip;
newflow->lower_port = lower_port, newflow->upper_port = upper_port;
newflow->first_packet_time_sec = header->ts.tv_sec;
newflow->first_packet_time_usec = header->ts.tv_usec;
if(newflow == NULL) {
printf("[NDPI] %s(1): not enough memory\n", __FUNCTION__);
return NULL;
}

newflow->ndpi_flow = calloc(1, size_flow_struct);
newflow->src_id = calloc(1, size_id_struct);
newflow->dst_id = calloc(1, size_id_struct);
memset(newflow, 0, sizeof(struct ndpi_flow));
newflow->protocol = iph->protocol;
newflow->lower_ip = lower_ip, newflow->upper_ip = upper_ip;
newflow->lower_port = lower_port, newflow->upper_port = upper_port;
newflow->first_packet_time_sec = header->ts.tv_sec;
newflow->first_packet_time_usec = header->ts.tv_usec;

ndpi_tsearch(newflow, (void*)&ndpi_flows_root, node_cmp);
ndpi_flow_count ++;
newflow->ndpi_flow = calloc(1, size_flow_struct);
newflow->src_id = calloc(1, size_id_struct);
newflow->dst_id = calloc(1, size_id_struct);

return newflow;
}
} else
return *(struct ndpi_flow**)ret;
return newflow;
}


static int packet_processing(const u_int64_t time, const struct pcap_pkthdr *header,
const struct ndpi_iphdr *iph, u_int16_t ipsize, u_int16_t rawsize)
const struct ndpi_iphdr *iph, u_int16_t ipsize, u_int16_t rawsize, struct ndpi_flow *flow)
{
struct ndpi_id_struct *src, *dst;
struct ndpi_flow *flow;
struct ndpi_flow_struct *ndpi_flow = NULL;
u_int16_t protocol = 0;
u_int16_t frag_off = ntohs(iph->frag_off);

flow = get_ndpi_flow(header, iph, ipsize);
if (flow != NULL) {
ndpi_flow = flow->ndpi_flow;
flow->packets++, flow->bytes += rawsize;
Expand Down Expand Up @@ -251,7 +224,7 @@ static int packet_processing(const u_int64_t time, const struct pcap_pkthdr *hea


// process a new packet
extern int ndpiPacketProcess(const struct pcap_pkthdr *header, const u_char *packet)
extern int ndpiPacketProcess(const struct pcap_pkthdr *header, const u_char *packet, void *flow)
{
const struct ndpi_ethhdr *ethernet = (struct ndpi_ethhdr *) packet;
struct ndpi_iphdr *iph = (struct ndpi_iphdr *) &packet[sizeof(struct ndpi_ethhdr)];
Expand All @@ -275,10 +248,29 @@ extern int ndpiPacketProcess(const struct pcap_pkthdr *header, const u_char *pac
ip_offset = sizeof(struct ndpi_ethhdr);

// process the packet
return packet_processing(time, header, iph, header->len - ip_offset, header->len);
return packet_processing(time, header, iph, header->len - ip_offset, header->len, (struct ndpi_flow*)flow);
}
}

extern void *ndpiGetFlow(const struct pcap_pkthdr *header, const u_char *packet) {
const struct ndpi_ethhdr *ethernet = (struct ndpi_ethhdr *) packet;
struct ndpi_iphdr *iph = (struct ndpi_iphdr *) &packet[sizeof(struct ndpi_ethhdr)];
u_int16_t type, ip_offset;

type = ethernet->h_proto;

// just work on Ethernet packets that contain IP
if (type == htons(ETH_P_IP) && header->caplen >= sizeof(struct ndpi_ethhdr) && iph->version == 4) {
ip_offset = sizeof(struct ndpi_ethhdr);
return get_ndpi_flow(header, iph, header->len - ip_offset);
}
return NULL;
}

extern void ndpiFreeFlow(void *flow) {
free(flow);
}

#else
// nDPI is disabled, so initialization fails

Expand All @@ -293,5 +285,12 @@ extern int ndpiPacketProcess(const struct pcap_pkthdr *header, const u_char *pac
return -1;
}

extern void *ndpiGetFlow(const struct pcap_pkthdr *header, const u_char *packet) {
return NULL;
}

extern void ndpiFreeFlow(void *flow) {
}

#endif
#endif
4 changes: 3 additions & 1 deletion modules/wrappers/nDPI_wrapper_impl.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@

extern int ndpiInitialize();
extern void ndpiDestroy(void);
extern int ndpiPacketProcess(const struct pcap_pkthdr*, const u_char*);
extern int ndpiPacketProcess(const struct pcap_pkthdr*, const u_char*, void*);
extern void *ndpiGetFlow(const struct pcap_pkthdr*, const u_char*);
extern void ndpiFreeFlow(void*);
Loading

0 comments on commit 89e674a

Please sign in to comment.