moving network namespace to pytests

jackhart · Jun 3, 2023 · 2928da8 · 2928da8
1 parent 7e88572
commit 2928da8
Show file tree

Hide file tree

Showing 11 changed files with 186 additions and 78 deletions.
diff --git a/README.md b/README.md
@@ -1,45 +1,43 @@
 # pygmp - A Python Library for Multicast Routing in Linux
 
-A Python library that supports interacting with the Linux kernel for multicast routing.  
+Python interface and services for Linux multicast routing.
 
 
 ## Limitations / Roadmap
 This is a work in progress.  Currently, only IPv4 multicast routing is supported.  I've also only tested with IGMPv2 on an Ubuntu 22.04.2 LTS (v6.0.0-1013-oem) host.
 
-There are no daemon implementations yet.  The only implemented program is an interactive interface.
+The only implementation is a simple, IPv4, static multicast router with a REST API.
 
 
-## Quick Start
-
-
-### Pip Install
+## Pip Install
 
 **Coming Soon**
 
-### Interactive Multicast Router Shell
+## Source
+
+Install the [task](https://taskfile.dev/installation) utility.  This utility is used to standardize build and test processes.
 
-Run an interactive terminal for interacting with multicast constructs in the kernel.
 ```bash
-sudo python3 pygmp interactive
+task install
 ```
 
+### IPv4 Static Multicast Routing
 
-## Developer Quick Start
-
-Install the [task](https://taskfile.dev/installation) utility.  This utility is used to standardize build and test processes.
-
-Start the interactive shell.  (This will automatically setup the development environment.)
+Run an example static multicast router implementation
 ```bash
 task run
 ```
 
+In your browser, you should be able to hit `localhost:8000/docs` to see the OpenAPI documentation.
+
+
 ### Roadmap
 
+- finalize IPv4 static multicast daemon implementation
 - CI/CD / semantic versioning / create a pip registry
 - MLD/IPv6 support
-- smcrouted daemon implementation
 - pimd daemon implementation
-- Dockerized daemon example
+- Dockerized example
 - expand testing to other distros
 
 
@@ -55,33 +53,3 @@ net.ipv4.conf.default.force_igmp_version = 2
 net.ipv4.conf.all.mc_forwarding = 1
 net.ipv4.conf.default.mc_forwarding = 1
 ```
-
-
-### Relevant RFCs
-
-RFCs often refer to, build upon, or obsolete, previous RFCs.  This can lead to a complex web of interrelated documents.  This is my collection of RFCs relevant to multicast routing.
-
-
-**IGMP**
-- [RFC 1112 - Host Extensions for IP Multicasting](https://datatracker.ietf.org/doc/html/rfc1112)
-- [RFC 2236 - Internet Group Management Protocol, Version 2](https://datatracker.ietf.org/doc/html/rfc2236)
-- [RFC 3376 - Internet Group Management Protocol, Version 3](https://datatracker.ietf.org/doc/html/rfc3376)
-
-**MLD**
-- [RFC 2710 - Multicast Listener Discovery (MLD) for IPv6](https://datatracker.ietf.org/doc/html/rfc2710)
-- [RFC 3810 - Multicast Listener Discovery Version 2 (MLDv2) for IPv6](https://datatracker.ietf.org/doc/html/rfc3810)
-
-**IGMPv3/MLDv2**
-- [RFC 4604 - Using Internet Group Management Protocol Version 3 (IGMPv3) and Multicast Listener Discovery Protocol Version 2 (MLDv2) for Source-Specific Multicast](https://datatracker.ietf.org/doc/html/rfc4604)
-- [RFC 5790 - Lightweight Internet Group Management Protocol Version 3 (IGMPv3) and Multicast Listener Discovery Version 2 (MLDv2) Protocols](https://datatracker.ietf.org/doc/html/rfc5790)
-
-
-**TODO**
-
-- [RFC 6636 - Tuning the Behavior of the Internet Group Management Protocol (IGMP) and Multicast Listener Discovery (MLD) for Routers in Mobile and Wireless Networks](https://datatracker.ietf.org/doc/html/rfc6636)
-- [RFC 7761 - Protocol Independent Multicast - Sparse Mode (PIM-SM): Protocol Specification (Revised)](https://datatracker.ietf.org/doc/html/rfc7761)
-- [RFC 7762 - Protocol Independent Multicast (PIM) MIB](https://datatracker.ietf.org/doc/html/rfc7762)
-- [RFC 7763 - Protocol Independent Multicast - Sparse Mode (PIM-SM) Multicast Routing Security Issues and Enhancements](https://datatracker.ietf.org/doc/html/rfc7763)
-- [RFC 7764 - Protocol Independent Multicast (PIM) over Virtual Private LAN Service (VPLS)](https://datatracker.ietf.org/doc/html/rfc7764)
-- [RFC 7765 - Multicast Considerations over IEEE 802 Wireless Media](https://datatracker.ietf.org/doc/html/rfc7765)
-- [RFC 7766 - Protocol Independent Multicast (PIM) over Ethernet (PIM-E): Protocol Specification](https://datatracker.ietf.org/doc/html/rfc7766)
diff --git a/Taskfile.yml b/Taskfile.yml
@@ -15,6 +15,7 @@ tasks:
     deps: [ install ]
     interactive: true
     cmds:
+      - sudo ./network-setup.sh --overwrite
       - sudo ip netns exec basic {{.USER_WORKING_DIR}}/venv/bin/python3 pygmp simple {{.CLI_ARGS}}
 
   install:
@@ -25,12 +26,6 @@ tasks:
     status:
       - '[ -n "$(find ./venv/lib/*/site-packages -type d -name "pygmp-*.dist-info" -print -quit)" ]'
 
-  test:
-    desc: Run pytests.
-    deps: [ install, setup-network ]
-    cmds:
-      - sudo ip netns exec basic {{.USER_WORKING_DIR}}/venv/bin/python3 -m pytest -s {{.CLI_ARGS}}
-
   setup-network:
     desc: Setup the network namespaces for testing.
     cmds:
@@ -45,8 +40,8 @@ tasks:
   make-venv:
     cmds:
       - python3 -m pip install virtualenv
-      - python3 -m venv venv
-      - ./venv/bin/pip3 install --upgrade pip setuptools wheel
+      - python3 -m venv venv --upgrade-deps --copies
+      # - sudo setcap cap_net_admin+ep ./python3
       - ./venv/bin/pip3 install -r dev-requirements.txt
     status:
       - '[ -d ./venv ]'

diff --git a/commands b/commands
@@ -6,6 +6,13 @@ python3 pygmp interactive
 
 ip netns exec basic python3 -m pytest tests/test_simple.py
 
+
+nsenter --net=/var/run/netns/basic
+
+sudo capsh --caps="cap_net_raw+eip cap_setpcap,cap_setuid,cap_setgid+ep cap_sys_admin+eip cap_sys_chroot+eip cap_setfcap+eip" --keep=1 --user=jack --addamb=cap_net_raw --addamb=cap_sys_admin --addamb=cap_sys_chroot --addamb=cap_setfcap -- -c "./venv/bin/python3 -m pytest -s"
+
+
+
 ip netns exec basic python3 pygmp smcrouted
 ip netns exec basic ping -c 3 -W 1 -I a3 -t 2 239.0.0.4
 ip netns exec basic ip mroute

diff --git a/dev-requirements.txt b/dev-requirements.txt
@@ -1,5 +1,7 @@
+wheel
 pytest
 sphinx
 furo
 flake8
-scapy
+scapy
+psutil
diff --git a/pygmp/daemons/simple.py b/pygmp/daemons/simple.py
@@ -27,7 +27,7 @@
 
 from ipaddress import IPv4Address
 import threading
-from pygmp.daemons.utils import get_logger
+from pygmp.daemons.utils import get_logger, search_dict_lists
 from pygmp.daemons.config import load_config, MRoute
 from pygmp import kernel, data
 
@@ -96,14 +96,26 @@ def add_vif(interface_address_or_index: IPv4Address | int, mcast_index: int | No
         vif_manager.add(match, mcast_index)
         return vif_manager.vifs()[match.name]
 
-    # TODO - DELETE vif
+    @app.delete("/vifs/{interface_name}")
+    def delete_vif(interface_name_or_index: str | int):
+        if isinstance(interface_name_or_index, str):
+            vif_manager.remove_by_name(interface_name_or_index)
+        else:
+            vif_manager.remove_by_index(interface_name_or_index)
+
     # TODO - POST and DELETE mfc
-    # @app.post("/mfc")
-    # def add_mfc(group: IPv4Address, incoming_interface_address_or_index: str | int, outgoing_interfaces: list[int | str], source: IPv4Address = ipaddress.ip_address(ANY_ADDR)):
-    #     MRoute(from_=iif, to=group, source=source, group=group)
-    #     mfc_manager.add(source, group, iif, oifs)
-    #     return mfc_manager.static_mfc()[iif]
-    #
+    @app.post("/mfc")
+    def add_mfc(mroute: MRoute):
+        mfc_manager.add(mroute)
+        if mroute.source == ANY_ADDR:
+            return mfc_manager.dynamic_mfc()[mroute.from_][-1]
+        return mfc_manager.static_mfc()[mroute.from_][-1]
+
+    @app.delete("/mfc")
+    def delete_mfc(mroute: MRoute):
+        # FIXME - ttl mapping shouldn't matter
+        mfc_manager.remove(mroute)
+
     return app
 
 
@@ -126,8 +138,8 @@ def vifi(self, name) -> int:
         """Returns the multicast VIF index for the given interface."""
         try:
             return self._vif_name_list.index(name)
-        except ValueError:
-            raise ValueError(f"Could not find index for Interface {name}.")
+        except ValueError as e:
+            raise ValueError(f"Could not find index for Interface {name}.") from e
 
     def add(self, interf: data.Interface, mcast_index: int | None = None):
         """Adds a virtual multicast interface to the kernel.
@@ -151,17 +163,20 @@ def remove_by_name(self, interface_name: str):
         """Removes a virtual multicast interface from the kernel by name."""
         try:
             vif_entry = self.vifs()[interface_name]
-        except KeyError:
-            raise ValueError(f"Interface {interface_name} does not exist.")
+        except KeyError as e:
+            raise ValueError(f"Interface {interface_name} does not exist.") from e
         # FIXME - interface vs address
         kernel.del_vif(self.sock, data.VifCtl(vifi=vif_entry.index, lcl_addr=vif_entry.local_addr_or_interface))
 
     def make_ttls_list(self, phyints: dict[str | int, int]):
         ttls = [0] * len(self._vif_name_list)
         for inter, ttl in phyints.items():
             if isinstance(inter, str):
-                inter = self._vif_name_list.index(inter)
-            ttls[inter] = ttl
+                inter = self.vifi(inter)
+            try:
+                ttls[inter] = ttl
+            except IndexError as e:
+                raise ValueError(f"Interface of index {inter} does not exist.") from e
         return ttls
 
 
@@ -201,6 +216,8 @@ def remove(self, mroute: MRoute):
         if str(mroute.source) == ANY_ADDR:
             if self._dynamic_mroutes.get(parent):
                 self._dynamic_mroutes[parent].remove(mroute)
+                if not self._dynamic_mroutes[parent]:
+                    del self._dynamic_mroutes[parent]
             else:
                 raise ValueError(f"Dynamic MRoute {mroute} does not exist.")
         else:

diff --git a/pygmp/daemons/utils.py b/pygmp/daemons/utils.py
@@ -24,6 +24,8 @@
 import logging
 import logging.config
 import inspect
+from typing import Any
+
 import pygmp
 import os
 import signal
@@ -56,6 +58,14 @@ def __exit__(self, *exc):
         return False
 
 
+def search_dict_lists(d: dict[Any, list[Any]], key: Any, value: Any) -> int | None:
+    """Searches a dictionary of lists for an item in list of key. Returns index of item if found, else None."""
+    if key in d:
+        if value in d[key]:
+            return d[key].index(value)
+    return None
+
+
 def _daemon_fork(working_dir: str, umask: int):
     """Follow the double fork pattern to daemonize a process."""
     try:

diff --git a/pygmp/data.py b/pygmp/data.py
@@ -19,7 +19,6 @@
 #  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 #  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 #  SOFTWARE.
-
 from __future__ import annotations  # relevant to PEP 563 (postponed evaluation of annotations)
 
 import ipaddress

diff --git a/pygmp/utils.py b/pygmp/utils.py
@@ -19,6 +19,7 @@
 #  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 #  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 #  SOFTWARE.
+from __future__ import annotations
 import hashlib
 import socket
 from ipaddress import ip_address, IPv4Address, IPv6Address

diff --git a/tests/conftest.py b/tests/conftest.py
@@ -0,0 +1,20 @@
+import pytest
+
+import networks
+
+CLONE_NEWNET = 0x40000000  # The namespace type for network namespaces
+
+
+
+@pytest.fixture(scope='session', autouse=True)
+def basic_network_namespace():
+    with networks.BasicNamespace() as ns:
+        ns.summary()
+        yield ns
+
+
+@pytest.fixture(scope='session', autouse=True)
+def network_namespace(basic_network_namespace):
+        networks.setns(basic_network_namespace.file())
+        yield
+        networks.setns('/proc/1/ns/net')
diff --git a/tests/networks.py b/tests/networks.py
@@ -0,0 +1,95 @@
+import ctypes
+import subprocess
+
+
+CLONE_NEWNET = 0x40000000  # The namespace type for network namespaces
+
+
+class NetworkNamespace:
+
+    def __enter__(self):
+        stdout, _ = self.run_command(f'ip netns list | grep -wc "{self.name}" || true')
+        if stdout == "1":
+            self.__exit__(None, None, None)
+        self.run_command(f'ip netns add {self.name}', with_ns=False)
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.run_command(f'ip netns delete {self.name}', with_ns=False)
+
+    @property
+    def name(self):
+        return self.__class__.__name__
+
+    def file(self):
+        return f'/var/run/netns/{self.name}'
+
+    def run_command(self, cmd, with_ns=True):
+        try:
+            if with_ns:
+                cmd = f'ip netns exec {self.name} {cmd}'
+            process = subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)
+            return process.stdout.decode().strip(), process.stderr.decode().strip()
+        except subprocess.CalledProcessError as e:
+            print(f'Error running command: {cmd}')
+            print(f'Error: {e}')
+            print(f'Stdout: {e.stdout.decode()}')
+            print(f'Stderr: {e.stderr.decode()}')
+            raise
+
+    def summary(self):
+        # TODO - return dict of values
+        print(f'Summary for {self.name}')
+        print('================')
+
+        for command in ['ip -br link show', 'ip -br addr show', 'ip -br neigh show', 'ip -br route show']:
+            stdout, _ = self.run_command(f'ip netns exec {self.name} {command}')
+            print(f'\n{command.split()[-2].capitalize()}:\n{stdout}')
+
+
+class BasicNamespace(NetworkNamespace):
+
+    def __enter__(self):
+        super().__enter__()
+
+        for dev in ['a1', 'a2', 'a3']:
+            self.run_command(f'ip link add {dev} type dummy')
+            self.run_command(f'ip link set {dev} up')
+            self.run_command(f'ip link set {dev} multicast on')
+
+        self.run_command('ip addr add 10.0.0.1/24 dev a1')
+        self.run_command('ip addr add 20.0.0.1/24 dev a2')
+        self.run_command('ip addr add 30.0.0.1/24 dev a3')
+        self.run_command('ip link set lo up')
+        return self
+
+
+# Set network namespace
+def setns(netns_path):
+    libc = ctypes.CDLL('libc.so.6')
+
+    with open(netns_path, 'r') as netns_file:
+        # The second argument is the namespace type.
+        result = libc.setns(netns_file.fileno(), CLONE_NEWNET)
+
+    if result == -1:
+        raise OSError(ctypes.get_errno())
+
+
+def setup_veth_pair(namespace: NetworkNamespace):
+    # setup veth pair for REST API access
+    namespace.run_command('ip link add veth0 type veth peer name veth1', with_ns=False)
+    namespace.run_command(f'ip link set veth1 netns {namespace.name}', with_ns=False)
+    namespace.run_command('ip addr add 172.20.0.1/24 dev veth0', with_ns=False)
+    namespace.run_command('ip link set veth0 up', with_ns=False)
+    namespace.run_command('ip addr add 172.20.0.2/24 dev veth1')
+    namespace.run_command('ip link set veth1 up')
+
+    namespace.run_command('iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 172.20.0.2:8000', with_ns=False)
+    namespace.run_command('iptables -t nat -A POSTROUTING -p tcp --sport 8000 -j MASQUERADE')
+
+
+def teardown_veth_pair(namespace: NetworkNamespace):
+    namespace.run_command('iptables -t nat -D PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 172.20.0.2:8000', with_ns=False)
+    namespace.run_command('ip link del veth0', with_ns=False)
+