We live in an era where both the demand and quantity of information are enormous. However, the way we store and access that information has remained unchanged, via keywords. Given such a direct association, it is hard to algorithmically determine the relationship between pieces of information.
So we want to introduce a knowledge engine to work with data inference within a given search. Wolfram Alpha does just this, but limited to mathematical context. We will branch beyond this into many subjects and topics including news and knowledge databases.
Thus, we are building a knowledge engine that will utilise machine learning models to show and find information related to a query. Users will be able to search for information that has better association, allowing them the ability to more efficiently learn and expand on a topic
Setting up the components of News Aggregator is suprisingly straightforward. We rely on Dockerized services in order to ensure ease of use, modularity, reliability and configurability. Each of the Redis, Elasticsearch, Frontend and Backend containers have portions of the repository shared onto this via mounted volumes, including configuration.
From the get go, there isn’t any additional configuration properties to alter, its mostly ensuring the required tooling is installed.
In order to start up the services and work on them there is a few bits a pieces to install. Off the bat you will need to have the following installed:
| Tool | Download |
|---|---|
| Git | This should already by installed, but if it isn’t then you can find installation instructions here: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git |
| Docker | Mac: https://docs.docker.com/docker-for-mac/install/ Windows: https://docs.docker.com/docker-for-windows/install/ Linux: https://docs.docker.com/engine/install/ubuntu/ |
| Python 3.8 | https://www.python.org/downloads/release/python-3810/ |
| Node JS | https://nodejs.org/en/download/ |
| An IDE and/or text editor such as: * VSCode * IntelliJ * Sublime Text * Atom * PyCharm * etc |
* https://code.visualstudio.com/ * https://www.sublimetext.com/ * https://atom.io/ * https://www.jetbrains.com/idea/ * https://www.jetbrains.com/pycharm/ |
| Ansible [Optional] | We use ansible for the deployment system. If you are not working on it then you can skip installing it. However here is the installation page if need be: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html |
| Rust + Cargo (Rust CLI) [Optional] | If you are working on the Redis modules, then you’ll need to install Rust and it's Cargo cli tool. You can do so with the following instructions: https://www.rust-lang.org/tools/install https://doc.rust-lang.org/cargo/getting-started/installation.html |
Once you have those installed, we can move on to setting up the repo itself.
You’ll first want to clone down the repo onto your local machine by using the following command:
git clone https://github.com/EngineersBox/NewsAggregator.gitIf you are looking to work on the frontend portion of the repo, then you’ll want to head over to the frontend directory.
- Run
npm ito install all the relevant npm packges - In a seperate terminal window/tab run
npm run serveto start the local development server - Open up your IDE or text editor of choice in the
frontenddirectory
Running the backend flask API is also straightforward. You’ll need to be in the top level NewsAggregator directory for this.
- Run
python3 -m pip install -r requirements.txtto setup all the dependencies - Install the SpaCy transform with
python3 -m spacy download en_core_web_trf - Ensure you have execute permissions on the start script with
sudo chmod +x ./run_server.sh - Run the start script with
./run_server.sh
Even when developing for Redis or Elasticsearch, we recommend running them in their Dockerized states. This alleviates having to configure the environment for them to run in.
On the other hand, if you are developing the Dockerized platform, testing deployable situations or otherwise this also applies. Note that the containers will be deployed with the host network attached, allowing you to directly access the ports for services such as Elasticsearch as if they were running on your regular local environment (E.g. localhost:9200)
All of the services can be ran with already define compose files. These can be found in the compose/* directories, depending on the component you are looking for, there will be subdirectories to account for this.
You can start any of these services as containers by running docker compose up -d or docker-compose up -d depending on your version of docker and support.