This Django-Project is part of a process to establish the OKR Framework in the organizational structure of the WDR. After identifying the correct KPI for each product, this library collects and processes the corresponding data to visualize the data in form of a dashboard.
This project requires Python > 3.8, pipenv, and a Redis server installed and working on your system.
To run the Django framework for this project, you need to set the following environment variables:
# Django
DEBUG=True
SECRET_KEY=
LOG_SQL=
# Quintly
QUINTLY_CLIENT_ID=
QUINTLY_CLIENT_SECRET=
# MySQL Podstat/Spotify
MYSQL_PODCAST_HOST=
MYSQL_PODCAST_USER=
MYSQL_PODCAST_PASSWORD=
MYSQL_PODCAST_DATABASE_PODSTAT=
# Spotify
SPOTIPY_CLIENT_ID=
SPOTIPY_CLIENT_SECRET=
SPOTIFY_LICENSOR_ID=
# Webtrekk/Mapp
WEBTREKK_LOGIN=
WEBTREKK_PASSWORD=
WEBTREKK_ACCOUNT_LIVE_ID=
# Sophora API
SOPHORA_API_BASE=
# SEO bot
TEAMS_WEBHOOK_SEO_BOT=
SEO_BOT_TODO_MORE_URL=
SEO_BOT_TOP_ARTICLES_THRESHOLD=The SECRET_KEY is only required if you have set DEBUG=False.
To run the project locally, store these variables in an .env file in the root
folder.
Install requirements:
$ pipenv sync
Migrate the database:
$ pipenv run manage migrate
Create initial admin user:
$ pipenv run manage createsuperuser
This project consists of two processes that communicate through Redis Queue (RQ).
The worker process runs the various scrapers while the web process runs the
Django server and backend.
First, start the worker process:
$ pipenv run worker
Then, start the web process:
$ pipenv run manage runserver
Scrapers are located in okr/scrapers/ with one module for each product type.
The __init__.py of each scraper module contains the functions to fill the
database. If cleaning or restructuring is required, it is done here.
There are other submodules for collecting the raw data, one for each source.
We use RQ (Redis Queue) and
APScheduler for queuing
and scheduling. okr/scrapers/scheduler.py contains the setup and cron-based
rules to run scrapers periodically at specified times.
Some data that can't be scraped automatically (yet) is manually entered or
uploaded as files in the Django admin backend. The relevant files for this
are located in okr/admin.
The data warehouse component is managed via the Django ORM. Models are defined
in okr/models/ with generally self-contained submodules for each product type.
Install the black code formatter:
$ pipenv run pip install black
Set up git pre-commit hook:
$ cp git-hooks/pre-commit .git/hooks
This project is licensed under the MIT License.
Unsere Inhalte, unser Wissen und unsere Entwicklungen gehören allen. WDR Geschäftsbericht 2016, S.23