![Linting/static analysis](https://github.com/dataiku/dss-plugin-document-question-answering/actions/workflows/python-app.yml/badge.svg)
![Gherking tests](https://github.com/dataiku/dss-plugin-document-question-answering/actions/workflows/api-tests.yml/badge.svg)
![Playwright tests](https://github.com/dataiku/dss-plugin-document-question-answering/actions/workflows/playwright.yml/badge.svg)

# Answers

This repo contains the code for Dataiku plugin to run a Answers App.
The code is based on python for backend and Vue3 for the frontend.

## Table of content

<!-- TOC -->

- [Answers](#answers)
  - [Table of content](#table-of-content)
  - [Development:](#development)
    - [Frontend](#frontend)
    - [Backend](#backend)
    - [Making Commits](#making-commits)
    - [Hooks and Their Responsibilities](#hooks-and-their-responsibilities)
      - [post-checkout Hook Details](#post-checkout-hook-details)
      - [pre-commit Hook Details](#pre-commit-hook-details)
    - [How to Use These Hooks](#how-to-use-these-hooks)
      - [Example Scenarios](#example-scenarios)
    - [Pushing Changes](#pushing-changes)
  - [Web app UI customization](#web-app-ui-customization)
  - [OpenAPI Specification Generation](#openapi-specification-generation)
    - [Script Overview](#script-overview)
    - [Running the script](#running-the-script)
  - [Testing](#testing)
    - [E2E testing](#e2e-testing)
      - [Playwright](#playwright)
      - [Visual comparison screenshots:](#visual-comparison-screenshots)
      - [Run tests in TEST INTEGRATION INSTANCE](#run-tests-in-test-integration-instance)
    - [Api tests documentation](#api-tests-documentation)
  - [Deployment](#deployment)
- [Legacy Release](#legacy-release)

<!-- /TOC -->
## Development:

### Frontend

The web application is built on Node.js and utilizes the Quasar framework.

-   First, go to `resource/frontend`:
    `cd resource/frontend`
-   Then, install the dependencies:
    `yarn install`
-   Finally, run the local server:
    `yarn run dev`

### Backend

The backend is powered by Flask.

-   First, create your virtual environment:
    `python3 -m venv .your_venv_name`
-   Second, activate your virtual environment:
    `source .your_venv_name/bin/activate`
-   Third, install the required packages:
    `pip3 install -r code-env/python/spec/requirements.dev.txt`
    And install dataiku-internal-client from your preferred location:
    - `pip install https://tests-integration.solutions.dataiku-dss.io/public/packages/dataiku-internal-client.tar.gz`
    - Or `pip install http://localhost:11200/public/packages/dataiku-internal-client.tar.gz`
- Initialize the submodule answers-common if not done before. It is also recommended when you change branches to make sure your getting the right submodules refs: 
    `git submodule update --init --remote`
-   Finally, run your local Flask server:
    ` python3 python-lib/wsgi.py`

### Making Commits
- This repository includes Git hooks to enforce submodule integrity and ensure frontend assets are properly built before committing.
- To enable them you need to run `git config core.hooksPath .githooks` once at the root (only after `yarn install` has been executed correctly).


💡 **Why These Hooks Are Useful**
✅ Prevents submodule inconsistencies by ensuring they are pushed before referencing them in commits.
✅ Avoids accidental overwrites when switching branches with local submodule changes.
✅ Automates frontend builds so built assets are always committed when needed.

**General recommendations**

- When working on changes that impact both the `common` submodule and the `answers` super repo you should first init the submodule (to ensure you start from the correct reference) then make a branch in both the submodule (`common`) and the super repo (`answers`) with the same name.
- Don't make changes in sub modules before the new branch exists in the super repo.
### Hooks and Their Responsibilities
| Hook |	Trigger | Event	| Purpose |
|------|------------|-------|---------|
|post-checkout |	Runs when switching branches (git checkout or git switch)	| ✅ Checks for local changes in submodules and prevents overwriting uncommitted work. | ✅ Ensures submodules are updated only when safe. ✅ Prompts users to create a matching branch in the super repo when a new branch is created inside a submodule.|
|pre-commit	| Runs before git commit	| ✅ Verifies that any staged submodule updates exist on the remote (prevents unpushed commits from being referenced). | ✅ Builds frontend assets (yarn build) if changes are detected in resource/frontend.|


#### post-checkout Hook Details

📌 Purpose: Ensures submodule integrity when switching branches.

✔ Detects local changes in submodules and prevents updates that would overwrite uncommitted work.
✔ Ensures submodules are updated only when no uncommitted changes exist.
✔ Warns users to manually update submodules if necessary.

🛠 How It Works:
	1.	Checks if checkout is a branch switch (not just a file restore).
    2. Detects if a new branch is created:
    • If a new branch is created in the super repo, prompts the user to create it in submodules.
	3.	Scans submodules for local changes:
	•	Warns and skips updates if there are uncommitted changes.
	•	Warns about submodules with committed but unpushed changes but proceed with submodule update.
	4.	Only updates submodules if they are clean.
#### pre-commit Hook Details

📌 Purpose: Ensures submodules are correctly referenced before committing and builds frontend assets if needed.

✔ Checks if a submodule reference is staged for commit.
✔ Ensures that referenced submodule commits exist on a remote branch (prevents commits referencing local-only submodule changes).
✔ Builds frontend (yarn build) if resource/frontend has changed.

🛠 How It Works:
	1.	Detects staged submodule updates:
	•	If a submodule reference is changed, it verifies the referenced commit exists remotely.
	•	If the submodule commit is not pushed, it blocks the commit and suggests a fix.
	2.	Detects frontend changes (resource/frontend):
	•	If changes exist, runs yarn build before completing the commit.
	•	Adds built files (resource/dist/) automatically to the commit.
### How to Use These Hooks
1- Ensure hooks are installed in .git/hooks/
2- Verify they work by switching branches or committing changes.
3- If blocked by submodule issues, follow the suggestions printed by the hook.
#### Example Scenarios
| Action	| Expected Behavior|
|-----------|------------------|
| git checkout branch-name	| 🟢 Runs post-checkout, checks submodules, and updates only if safe.|
| git commit -m "Update submodule" |	🟢 Runs pre-commit, verifies submodule commit exists remotely, and allows commit only if valid.|
| git commit -m "Frontend update"	| 🟢 Runs yarn build if frontend changes are staged, then adds built files before committing. |
| git commit with an unpushed submodule commit	| 🔴 Blocks commit and instructs user to push the submodule first. |

### Pushing Changes

Once your changes are ready to be integrated, run all static checks and tests locally before creating a PR: 
- Run Linting and static checks
`npm run static-checks`
- Run FE tests
`npm run fe-tests`
- Run BE tests
`npm run be-tests`
- Or simply run one to run all the above and push if successful:
`npm run push`
## Web app UI customization

You can rebrand the webapp by applying custom style without changing the webapp code. It is essential that `Enable custom rebranding` is enabled in the webapp settings.

-   Navigate to ᎒᎒᎒ > Global Shared Code > Static Web Resources, create `answers` folder with a folder with the project key of your project. The file structure should be
    ```
    answers
    └── YOUR_PROJECT_KEY
        ├── custom.css
        ├── fonts
        │   └── fonts.css
        └── images
            ├── answer-icon.png
            └── logo.png
    ```
-   For css changes, add a `custom.css` file inside the `answers` folder, you can find an example below:

    ```css
    :root {
        --brand: #031e3a !important;
        --bg-examples-brand: rgba(3, 30, 58, 0.05) !important;
        --bg-examples-brand-hover: rgba(3, 30, 58, 0.07) !important;
        --bg-examples-borders: rgba(3, 30, 58, 0.05) !important;
        --examples-question-marks: #031e3a !important;
        --examples-text: black !important;
        --text-brand: #444 !important;
    }
    .logo-container .logo-img {
        height: 70%;
        width: 70%;
    }
    ```

-   To customize fonts:
    -   First, create `fonts` folder inside `answers`
    -   Second, add `fonts.css` and define your font like below depending on the format you can provide:
    ```css
    @font-face {
        font-family: "YourFontName";
        src: url(data:application/octet-stream;base64,your_font_base64);
    }
    @font-face {
        font-family: "YourFontName";
        src: url("yourFontPublicUrl") format("yourFontFormat");
    }
    ```
    -   Finally, add this part to the `custom.css` file:
    ```css
    body,
    div {
        font-family: "YourFontName" !important;
    }
    ```
-   To customize images, create `images` folder where you can import a logo file to change the logo image inside the landing page, and an icon file to change the icon of the AI answer. Ensure that that `Enable custom rebranding` is enabled in the webapp settings and that the correct file names are provided.

## OpenAPI Specification Generation

A script is available at the root of the repository to generate an OpenAPI JSON file for specific API endpoints. This documentation selectively covers endpoints primarily meant for external use, such as:

- /ask: For submitting questions.
- /conversations: For managing user conversations.

### Script Overview
The script leverages the apispec library and convert Marshmallow schemas into OpenAPI specifications. Key components include:

- Schemas: Import relevant Marshmallow schemas, like AskRequestSchema and AskResponseSchema, to define the structure of API requests and responses.
- Endpoints: Define each endpoint's HTTP operations (e.g., GET, POST, DELETE) and associate them with schemas.

### Running the script

To generate the OpenAPI JSON file, run the following command:
> python3 python-lib/generate_openapi.py  
> 
This will produce an openapi.json file in the repository root.

## Testing

### E2E testing

Make sure to add `.env` file in the root of the repo. The file needs to contain this values:

```
STAGING=1
E2E_USER_NAME=<TEST_INTEGRATION_USERNAME>
E2E_PASSWORD=<TEST_INTEGRATION_PWD>
E2E_AUTH_URL= "<TEST_INTEGRATION_URL>/webapps/<TST_PROJECT_ID>/<TST_PUBLIC_WEBAPP_ID>/new"
E2E_LOGIN_URL= "<TEST_INTEGRATION_URL>/public-webapps/<TST_PROJECT_ID>/<TST_PUBLIC_WEBAPP_ID>/new"

E2E_LOCAL_AUTH_URL="<YOUR_LOCALHOST>/webapps/<YOUR_PROJECT_ID>/<YOUR_PUBLIC_WEBAPP_ID>/new"
E2E_LOCAL_LOGIN_URL="<YOUR_LOCALHOST>/webapps/<YOUR_PROJECT_ID>/<YOUR_PUBLIC_WEBAPP_ID>/new"
E2E_LOCAL_USER_NAME="<LOCALHOST_USERNAME>"
E2E_LOCAL_PASSWORD="<LOCALHOST_PWD>"

<!-- This part is only needed for running the script to update the plugin version in test instance using Playwright -->
E2E_TEST_INSTANCE="<TEST_INTEGRATION_URL>"
E2E_ADMIN_USER="<TEST_INTEGRATION_ADMIN_USER>"
E2E_ADMIN_PASSWORD="<TEST_INTEGRATION_ADMIN_PWD>"
```

Use `STAGING=1` to test in TEST INTEGRATION INSTANCE, or `STAGING=0` if you want to test locally

#### Playwright

Check out best practices [here](https://playwright.dev/docs/best-practices) for writing tests using Playwright.
- Installation
    - Run `npm install playwright`
    - You can also install [Playwright VS Code extension](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright)

-   To run all tests here are different ways:

    -   Using command line:
        `npx playwright test`
    -   CLI in UI Mode:
        `npx playwright test --ui`
    -   Through Visual Code Studio's Testing Window using the [VS Code extension](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).

-   To run one test file:
    `npx playwright test test_file_path`

- To run tests on a certain browser `webkit` for example:
    `npx playwright test --project=webkit`
-   To view report:
    `npx playwright show-report`


#### Visual comparison screenshots:

Some tests compare screenshots. If any changes were introduced that change the UI, please make sure to update the screenshots:

-   For one file, you can run the command:

`npx playwright test test_path --update-snapshots`

-   For all files, you can run:

`npx playwright test --update-snapshots`

-   On CI, you need to manually trigger the workflow `update-playwright-screenshots` and then get the screenshots from the report and add them to the repo.

#### Run tests in TEST INTEGRATION INSTANCE

You can run the tests in TEST INTEGRATION INSTANCE during the development process to make sure, your tests will be successful in CI/CD pipeline.
To do so, you can follow these steps:

-   Add in your .env file if it doesn't have it:

```
STAGING=1
E2E_TEST_INSTANCE="<TEST_INTEGRATION_URL>"
E2E_ADMIN_USER="<TEST_INTEGRATION_ADMIN_USER>"
E2E_ADMIN_PASSWORD="<TEST_INTEGRATION_ADMIN_PWD>"
```

-   Run `make dev` to build plugin version with your current local changes
-   Remove the `.skip` tag from `update-plugin-version.spec.ts` and update the zip file path with your last created zip file path.
-   Run the playwright test `update-plugin-version.spec.ts` in one of the browsers. Do not need to run it in different browsers.
-   Put back the `.skip` tag in `update-plugin-version.spec.ts`
-   Run your tests

### Api tests documentation

See [this readme](tests/api_tests/README.md)


## Deployment
To build for release on current local branch run 
`make plugin`
or for a specific branch run 
`make plugin branch=<NAME_OF_BRANCH>`
this will create a local .zip in `dist/dss-plugin-<PLUGIN_ID>-<RELEASE_VERSION>.zip`
⚠️please do not use any make variables with the make plugin command as this will be handled by the GenAI team prior to release.⚠️

<details>
  <summary>👉&nbsp;&nbsp;For those in the GenAI team 💎 </summary>
  	<a>For those in the GenAI team who wish to create a release. First make sure the branch exists on remote. If it does, then you can run <code>make plugin plugin_version=&lt;RELEASE_VERSION&gt;add_tag=true</code> or specify a branch by adding <code>branch=&lt;NAME_OF_BRANCH&gt;</code>. For example, <code>make plugin plugin_version=2.2.0 add_tag=true branch=main</code>
	</a>
	<br>
  	<a href="https://design.solutions.dataiku-dss.io/projects/WIKI/wiki/83/Ready%20to%20release!#building-and-deploying-the-plugin-1" target="_blank" rel="noopener noreferrer">For more information about this please check this wiki article</a>
</details>

# Legacy Release
Deploy the zip + release-notes-<RELEASE_VERSION>.pdf to the AWS server.
Share the news on the slack channel feat-dataiku-answers

