Umbrage Evals JavaScript CLI

Overview

This is the Javascript CLI for Umbrage Evals. It also supports TypeScript.

Installation

To install the package, run the following command:

npm install evals

Dependencies

• Node.js (or another JavaScript runtime environment like Bun)

Environment

You will need to set the following environment variables:

UMBRAGE_EVALS_API_KEY=<your api key for the umbrage evals project>
OPENAI_API_KEY=<your openai api key>

These can be set in your ~/.bashrc or ~/.zshrc file on Mac or Linux, or in your ~/.bash_profile file on Windows.

It would look like:

export UMBRAGE_EVALS_API_KEY=<your api key for the umbrage evals project>
export OPENAI_API_KEY=<your openai api key>

To save the file, press Ctrl + X, then Y, then Enter. Then, run source ~/.bashrc or source ~/.zshrc on Mac or Linux, or source ~/.bash_profile on Windows.

Example usage

npx evals@latest fetch-evals

npx evals@latest run-evals

Example Prompt BaseModel

export interface ModelSettings {
  temperature: number;
  [key: string]: any;
}

export interface Prompt {
  prompt_type: string;
  prompt_text: string;
}

export interface Config {
  promptName: string;
  modelName: string;
  modelSettings: ModelSettings;
  environment: string;
}

class BaseModel {
  promptName: string;
  modelName: string;
  modelSettings: ModelSettings;
  environment: string;

  constructor(config: Config) {
    this.promptName = config.promptName;
    this.modelName = config.modelName;
    this.modelSettings = config.modelSettings;
    this.environment = config.environment || 'development';
}

  /**
   * This method should be overridden in subclasses.
   * It should return an object containing the response and the prompts array.
   * @param userMessage The user message to be processed by the model, optional.
   * @returns Promise<{ response: string, prompts: Prompt[] }>
   */
  async callModel(userMessage: string = ''): Promise<{ response: string; prompts: Prompt[] }> {
      throw new Error('callModel method needs to be implemented');
  }
}

export default BaseModel;

LICENSE

Copyright (c) 2023 Umbrage Studios, LLC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to use the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.