# Agents Portal
This repo contains the code for Dataiku plugin to run a Portal App. The code is based on python for backend and Vue3 for the frontend.
It uses [`answers-ui-components`](https://github.com/dataiku/dss-answers-ui-components) and [answers-commons](https://github.com/dataiku/answers-commons) as shared code between Answers Plugin and Portal plugin.
It also uses Business solutions infra webaiku : https://github.com/dataiku/solutions-contrib. You can find more information on what version is used inside requirements.txt.

## Local development

- First check if you have `.env` file at the root folder. If not create one. It should contain:

```
VITE_API_PORT="5000"
VITE_CLIENT_PORT="4200"
DKU_CURRENT_PROJECT_KEY="YOUR_DSS_PROJECT_KEY"
```
- Then, initialize the submodule answers-common & answers-ui-components if not done before: 
`git submodule update --init --remote`
- You need to add .npmrc file to add your personal github token `PAT` to fetch `quasar-ui-bs` package with this content and replace it with your `PAT`.
```
//npm.pkg.github.com/:_authToken=<YOUR_PERSONAL_ACCESS_TOKEN>
@dataiku:registry=https://npm.pkg.github.com/
```
### Frontend

The web application is built with Vue3 and utilizes the Quasar framework and Answers UI components.
-   Firt, navigate to the frontend repo:
    `cd resource/frontend`
-   Second, install the dependencies:
    `yarn install`
-   Then, run the local server:
    `yarn run dev`

#### Troubleshoot

You might get a `401 Unauthorized` error related to the quasar library, before running the `yarn install` you should delete the `resource/frontend/yarn.lock`

### Making and Pushing Changes
- This repository includes Git hooks to enforce submodule integrity and ensure frontend assets are properly built before committing.
- To enable them you need to run `git config core.hooksPath .githooks` once at the root (only after `yarn install` has been executed correctly).


💡 **Why These Hooks Are Useful**
✅ Prevents submodule inconsistencies by ensuring they are pushed before referencing them in commits.
✅ Avoids accidental overwrites when switching branches with local submodule changes.
✅ Automates frontend builds so built assets are always committed when needed.
**General recommendations**
- When working on changes that impact both the `common` submodule, `src` submodule and the `agents-connect` super repo you should first init the submodule (to ensure you start from the correct reference) then make a branch in both the submodule (`common`) and the super repo (`agents-connect`) with the same name.
- Don't make changes in sub modules before the new branch exists in the super repo.
### Hooks and Their Responsibilities
| Hook |	Trigger | Event	| Purpose |
|------|------------|-------|---------|
|post-checkout |	Runs when switching branches (git checkout or git switch)	| ✅ Checks for local changes in submodules and prevents overwriting uncommitted work. | ✅ Ensures submodules are updated only when safe. ✅ Prompts users to create a matching branch in the super repo when a new branch is created inside a submodule. |
|pre-commit	| Runs before git commit	| ✅ Verifies that any staged submodule updates exist on the remote (prevents unpushed commits from being referenced). | ✅ Builds frontend assets (yarn build) if changes are detected in resource/frontend.|


#### post-checkout Hook Details

📌 Purpose: Ensures submodule integrity when switching branches.

✔ Detects local changes in submodules and prevents updates that would overwrite uncommitted work.
✔ Ensures submodules are updated only when no uncommitted changes exist.
✔ Warns users to manually update submodules if necessary.

🛠 How It Works:
	1.	Checks if checkout is a branch switch (not just a file restore).
    2. Detects if a new branch is created:
    • If a new branch is created in the super repo, prompts the user to create it in submodules.
	3.	Scans submodules for local changes:
	•	Warns and skips updates if there are uncommitted changes.
	•	Warns about submodules with committed but unpushed changes but proceed with submodule update.
	4.	Only updates submodules if they are clean.
#### pre-commit Hook Details

📌 Purpose: Ensures submodules are correctly referenced before committing and builds frontend assets if needed.

✔ Checks if a submodule reference is staged for commit.
✔ Ensures that referenced submodule commits exist on a remote branch (prevents commits referencing local-only submodule changes).
✔ Builds frontend (yarn build) if resource/frontend has changed.

🛠 How It Works:
	1.	Detects staged submodule updates:
	•	If a submodule reference is changed, it verifies the referenced commit exists remotely.
	•	If the submodule commit is not pushed, it blocks the commit and suggests a fix.
	2.	Detects frontend changes (resource/frontend):
	•	If changes exist, runs yarn build before completing the commit.
	•	Adds built files (resource/dist/) automatically to the commit.
### How to Use These Hooks
1- Ensure hooks are installed in .git/hooks/
2- Verify they work by switching branches or committing changes.
3- If blocked by submodule issues, follow the suggestions printed by the hook.
#### Example Scenarios
| Action	| Expected Behavior|
|-----------|------------------|
| git checkout branch-name	| 🟢 Runs post-checkout, checks submodules, and updates only if safe.|
| git commit -m "Update submodule" |	🟢 Runs pre-commit, verifies submodule commit exists remotely, and allows commit only if valid.|
| git commit -m "Frontend update"	| 🟢 Runs yarn build if frontend changes are staged, then adds built files before committing. |
| git commit with an unpushed submodule commit	| 🔴 Blocks commit and instructs user to push the submodule first. |

### Backend

The backend is powered by Flask.

#### Virtual environment and dependencies 

From the root folder of the project : 

1.   create your virtual environment:  
    `python3 -m venv .your_venv_name`  
2.   activate your virtual environment:  
    `source .your_venv_name/bin/activate`  
3.   install the required packages:  
	`pip3 install -r code-env/python/spec/requirements.local.txt`
4.	Link the dataiku and dataikuapi modules to your virtual environment, by specifying the path to your local DSS installation.

	> **Note:** The usual location provided below is for installations using the `.dmg` installer on macOS. If you installed DSS using a different method, or on a different operating system, the path may vary. Please adjust the path accordingly.

	It is usually located at `/Users/youruser/Library/DataScienceStudio`.  
	`ln -s ~/Library/DataScienceStudio/kits/dataiku-dss-13.4.1-osx/python/dataiku /Users/youruser/.your_venv_name/myenv/lib/python3.9/site-packages/dataiku`  
	`ln -s ~/Library/DataScienceStudio/kits/dataiku-dss-13.4.1-osx/python/dataikuapi /Users/youruser/.your_venv_name/myenv/lib/python3.9/site-packages/dataikuapi`  

#### Configuration of local_config.json

> Note : The configuration information is stored in datasets in your flow. Here, you will need both a user dataset and a conversation history dataset, which will be set up later.
> Required : having a sql connection setup. You can have a postgresql running locally and add a connection to your instance. (Some links about setting up postgresql into DSS : [DSS installation guide](https://docs.google.com/document/d/1y3W8SS1oYY-ghu5FkUQEZY96AedxOvcIGEKD7vxzsHw/edit?usp=sharing), [Data connection in DSS](https://knowledge.dataiku.com/latest/data-sourcing/connections/concept-connections.html), [Postgresql DSS](https://doc.dataiku.com/dss/latest/connecting/sql/postgresql.html)

We need to configure the 'webapp_config' in `python-lib/portal/backend/local_config.json`. The easiest way is to setup a portal webapp in your local instance. By doing so the webapp will generate a config file and we will get the configuration informations from there.


**Add the Agent Connect plugin to your instance if not done yet**

> it's recommended to install the plugin from the git repo for more granularity on the versions used. 
> By installing the plugin this way you can make changes directly to the repo within the platform. This setup allows for convenient testing and development of the plugin within Dataiku. 

- waffle button at top right ᎒᎒᎒ -> plugins -> ADD PLUGIN (top right) -> Fetch from git  
- In the "Repository url" field enter `git@github.com:dataiku/dss-agents-portal.git` and leave `main` as branch to checkout.  
- Then build a new environment with python3.9

**Setup an agent instance**
- Go back to your project
- webapps (in DSS header) -> New webapp -> Visual webapp -> Agent Connect -> name it and hit create
- The webapp settings tab opens : 
  - Select a llm connection to use (gpt 4o is better)
  - Conversation History Dataset :
    - hit 'New dataset' under the select dropdown 
    - put a name (that will match the one defined in the future local_config.json), select your sql connection and create dataset
  - User profile dataset :
    - hit 'New dataset' under the select dropdown
	- put a name (that will match the one defined in the future local_config.json), select your sql connection and create dataset

**Get configuration**
While on the webapp instance on your local dss check the url in your webbrowser, example : `http://localhost:11200/projects/PORTAL/webapps/DOio4Oa_my-agent-connect/edit` we want to know the webapp id, here it's just after webapps/ so it's *DOio4Oa*
- navigate to your DSS folder as before : `~/Library/DataScienceStudio/dss_home/config/projects/PORTAL/web_apps/`
- here open the corresponding json file, example : `DOio4Oa.json`
- scroll down to the `config` key and copy the whole dict value
- now open your `python-lib/portal/backend/local_config.json` and paste as `webapp_config` value

By doing so your backend will use the datasets defined in the visual webapp creation.

**Some important keys to remember**

Some major keys in local_config.json are the following, you can have a quicklook to ensure everything match your configuration.

|	Key					|Meaning|
|-------|----|
|  default_project_key 	| the key project key on your DSS instance |
|  logging_dataset 		| dataset name use to store conversations history |
|  user_profile_dataset | dataset name where user profiles will be stored|
|  llm_id 				| the llm connection id setup in your instance connection that will be used|


#### Running
To have the backend use correctly the common module it must be run from the right spot as a module : 

-   navigate to python-lib repo:
    `cd python-lib`
-   Finally, run your local Flask server:
    ` python3 -m portal.wsgi`

## Testing

### E2E testing

Make sure to add `.env` file in the root of the repo. The file needs to contain this values:

```
STAGING=1
E2E_USER_NAME=<TEST_INTEGRATION_USERNAME>
E2E_PASSWORD=<TEST_INTEGRATION_PWD>
E2E_AUTH_URL= "<TEST_INTEGRATION_URL>/webapps/<TST_PROJECT_ID>/<TST_PUBLIC_WEBAPP_ID>/new"
E2E_LOGIN_URL= "<TEST_INTEGRATION_URL>/public-webapps/<TST_PROJECT_ID>/<TST_PUBLIC_WEBAPP_ID>/new"

E2E_LOCAL_AUTH_URL="<YOUR_LOCALHOST>/webapps/<YOUR_PROJECT_ID>/<YOUR_PUBLIC_WEBAPP_ID>/new"
E2E_LOCAL_LOGIN_URL="<YOUR_LOCALHOST>/webapps/<YOUR_PROJECT_ID>/<YOUR_PUBLIC_WEBAPP_ID>/new"
E2E_LOCAL_USER_NAME="<LOCALHOST_USERNAME>"
E2E_LOCAL_PASSWORD="<LOCALHOST_PWD>"

<!-- This part is only needed for running the script to update the plugin version in test instance using Playwright -->
E2E_TEST_INSTANCE="<TEST_INTEGRATION_URL>"
E2E_ADMIN_USER="<TEST_INTEGRATION_ADMIN_USER>"
E2E_ADMIN_PASSWORD="<TEST_INTEGRATION_ADMIN_PWD>"
```

Use `STAGING=1` to test in TEST INTEGRATION INSTANCE, or `STAGING=0` if you want to test locally

#### Playwright

Check out best practices [here](https://playwright.dev/docs/best-practices) for writing tests using Playwright.
- Installation
    - Run `npm install playwright`
    - You can also install [Playwright VS Code extension](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright)

-   To run all tests here are different ways:

    -   Using command line:
        `npx playwright test`
    -   CLI in UI Mode:
        `npx playwright test --ui`
    -   Through Visual Code Studio's Testing Window using the [VS Code extension](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).

-   To run one test file:
    `npx playwright test test_file_path`

- To run tests on a certain browser `webkit` for example:
    `npx playwright test --project=webkit`
-   To view report:
    `npx playwright show-report`


#### Visual comparison screenshots:

Some tests compare screenshots. If any changes were introduced that change the UI, please make sure to update the screenshots:

-   For one file, you can run the command:

`npx playwright test test_path --update-snapshots`

-   For all files, you can run:

`npx playwright test --update-snapshots`

-   On CI, you need to manually trigger the workflow `update-playwright-screenshots` and then get the screenshots from the report and add them to the repo.

#### Run tests in TEST INTEGRATION INSTANCE

You can run the tests in TEST INTEGRATION INSTANCE during the development process to make sure, your tests will be successful in CI/CD pipeline.
To do so, you can follow these steps:

-   Add in your .env file if it doesn't have it:

```
STAGING=1
E2E_TEST_INSTANCE="<TEST_INTEGRATION_URL>"
E2E_ADMIN_USER="<TEST_INTEGRATION_ADMIN_USER>"
E2E_ADMIN_PASSWORD="<TEST_INTEGRATION_ADMIN_PWD>"
```

-   Run `make dev` to build plugin version with your current local changes
-   Remove the `.skip` tag from `update-plugin-version.spec.ts` and update the zip file path with your last created zip file path.
-   Run the playwright test `update-plugin-version.spec.ts` in one of the browsers. Do not need to run it in different browsers.
-   Put back the `.skip` tag in `update-plugin-version.spec.ts`
-   Run your tests


## Deployment
To build for release on current local branch run 
`make plugin`
this will create a local .zip in `dist/dss-plugin-<PLUGIN_ID>-<RELEASE_VERSION>.zip`  
⚠️please do not use any make variables with the `make plugin` command as this will be handled by the GenAI team prior to release.⚠️


<details>
  <summary>👉&nbsp;&nbsp;For those in the GenAI team 💎 </summary>
  	<a>For those in the GenAI team who wish to create a release. First make sure the branch exists on remote. If it does, then you can run <code>make plugin plugin_version=&lt;RELEASE_VERSION&gt;add_tag=true</code> or specify a branch by adding <code>branch=&lt;NAME_OF_BRANCH&gt;</code>. For example, <code>make plugin plugin_version=2.2.0 add_tag=true branch=main</code>
	</a>
	<br>
  	<a href="https://design.solutions.dataiku-dss.io/projects/WIKI/wiki/83/Ready%20to%20release!#building-and-deploying-the-plugin-1" target="_blank" rel="noopener noreferrer">For more information about this please check this wiki article</a>
</details>
