added giskard hub documentation #25

davidjanmercado · 2024-11-22T13:08:28Z

No description provided.

mattbit · 2024-11-22T14:21:10Z

Note:

fix TODOs
remove unnecessary files

Hartorn

Remove also all .DS_Store and add the .DS_Store to git ignore ;)

Hartorn · 2024-11-25T10:06:30Z

node_modules/.package-lock.json

@@ -0,0 +1,6 @@
+{


Remove this file

Hartorn · 2024-11-25T10:06:44Z

package-lock.json

@@ -0,0 +1,6 @@
+{


Hartorn · 2024-11-25T10:06:52Z

package.json

@@ -0,0 +1 @@
+{}


Hartorn

I skimed through it, seems overall ok.

Hartorn · 2024-11-25T10:09:37Z

script-docs/hub/glossary.rst

+Glossary
+=========
+
+- **Models**: conversational agents configured through an API endpoint. They can be evaluated and tested within the Hub.


Should probably be singular, same as for the rest ?

Hartorn · 2024-11-25T10:11:36Z

script-docs/hub/guide/access-rights.rst

+
+This section provides guidance on managing users in the Hub.
+
+The Hub allows you to set access rights at two levels: global and scoped. To begin, click the "Account" icon in the upper right corner of the screen, then select "Settings." From the left panel, choose "User Management."


Actually, you can also limit to one specific entity.

As a side note, we probably need to give more details about what each right is doing

Hartorn · 2024-11-25T10:13:11Z

script-docs/hub/guide/generate-dataset.rst

+Generate a dataset
+===================
+
+This section guides you through generating a test dataset when you don’t have one at your disposal.


don’t => don't (I think you are having special char here for quote)

Hartorn · 2024-11-25T10:14:21Z

script-docs/hub/guide/manage-datasets.rst

+   :align: center
+   :alt: "List of conversations"
+
+You can import data files in JSON or JSONL format, containing an array of conversations (or a conversation object per line, if JSONL).


Soon to be updated to include CSV also

Hartorn · 2024-11-25T10:16:00Z

script-docs/hub/guide/manage-datasets.rst

+
+- The agent should mention its references.
+
+- The agent should cite its references using the format [reference 1] for reference 1 and should not create a list, for example, [reference 1][reference 2][reference 3].


Maybe this one would be better as two rules no ? @mattbit

Hartorn

Global comment : image should be clickable & able to zoom in, it's too small atm
Also, the "Set access rights" section feels weird after the experiments in the playground.
I guess we may want to put subsection under quick start, and then the rest ?

Hartorn · 2024-12-05T08:44:40Z

script-docs/index.rst

+.. toctree::
+   :caption: Giskard Hub
+   :maxdepth: 1
+
+   hub/quickstart
+   hub/guide/manage-datasets
+   hub/guide/generate-dataset
+   hub/guide/run-evaluations
+   hub/guide/compare-evaluations
+   hub/guide/playground
+   hub/guide/access-rights
+   hub/guide/continuous-red-teaming
+   hub/glossary
+
+
+----
+
+The ``giskard_hub`` python library provides a simple way for developers and data


It's feeling a bit weird when you are on the page.
Was the idea to cut this into 2 ?

Hartorn · 2024-12-05T08:46:16Z

script-docs/hub/quickstart.rst

+Create a project
+=================
+
+In this section, you will learn how to create a project. Before creating one, ensure you have properly configured the model (see Setup the model section). TODO LINK


Handle TODO

Hartorn · 2024-12-05T08:47:35Z

script-docs/hub/quickstart.rst

+
+.. note::
+
+    Models are conversational agents configured through an API endpoint. They can be evaluated and tested within the Hub.


They can be evaluated feels wrong
Should we say They can be evaluated against datasets ? (and drop the tested part)

Hartorn · 2024-12-05T08:52:42Z

script-docs/hub/guide/generate-dataset.rst

+   :align: center
+   :alt: "Generate correctness dataset"
+
+However, dataset generation requires two additional pieces of information:


However, dataset generation => However, in this case, dataset generation

Hartorn · 2024-12-05T08:53:59Z

script-docs/hub/guide/compare-evaluations.rst

+
+Clicking on a conversation will show a detailed comparison.
+
+.. image:: /_static/images/hub/comparison-detail.png


Image orientation is wrong

Hartorn · 2024-12-05T08:56:56Z

script-docs/hub/guide/playground.rst

+
+The Chat section is where you interrogate the agent. You write your message on the bottom part of the screen.
+
+The left panel shows you the recent conversations. You can have as many conversations as you need. To add a new one, click the “New conversation” button. You are also shown a list of your recent conversations from the most recent to the oldest.


The left panel shows you the recent conversations => well, it shows all of them, most recent first, but that's ok

Hartorn · 2024-12-05T09:00:48Z

script-docs/hub/guide/access-rights.rst

+   :align: center
+   :alt: "Set permissions"
+
+Scoped permissions allow for more granular control. For each project, you can specify which pages or entities users are allowed to access.


Missing a concrete example of someone having read on everything in a project, and edit on a specific dataset

Hartorn · 2024-12-05T09:01:49Z

script-docs/hub/guide/annotate.rst

+
+The playground is where you chat with your agent and check its response. The screen below shows the interface. The Chat section is where you interrogate the agent. You write your message on the bottom part of the screen. The left panel shows you the recent conversations. You can have as many conversations as you need. To add a new one, click the “New conversation” button. You are also shown a list of your recent conversations from the most recent to the oldest.
+
+TODO IMAGE


Hartorn · 2024-12-05T09:02:52Z

script-docs/hub/guide/annotate.rst

+The Playground
+===============
+
+The playground is where you chat with your agent and check its response. The screen below shows the interface. The Chat section is where you interrogate the agent. You write your message on the bottom part of the screen. The left panel shows you the recent conversations. You can have as many conversations as you need. To add a new one, click the “New conversation” button. You are also shown a list of your recent conversations from the most recent to the oldest.


This is duplicate from "Experiment from the playground"
Also, I accessed this page from the search, but not sure it's accessible another way ?

Hartorn · 2024-12-05T09:03:28Z

script-docs/hub/guide/annotate.rst

+
+When the conversation is sufficient enough for what it needs to contain, you can send it to the dataset which you then use to evaluate your model.
+
+TODO IMAGE


added giskard hub documentation

6228cbc

davidjanmercado requested review from mattbit and Hartorn November 22, 2024 13:08

Hartorn requested changes Nov 25, 2024

View reviewed changes

Hartorn reviewed Nov 25, 2024

View reviewed changes

davidjanmercado added 2 commits November 25, 2024 16:17

added page on how to write business specific tests

ad49991

added crt section

3e5358d

Hartorn requested changes Dec 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added giskard hub documentation #25

added giskard hub documentation #25

davidjanmercado commented Nov 22, 2024

mattbit commented Nov 22, 2024

Hartorn left a comment

Hartorn Nov 25, 2024

Hartorn Nov 25, 2024

Hartorn Nov 25, 2024

Hartorn left a comment

Hartorn Nov 25, 2024

Hartorn Nov 25, 2024

Hartorn Nov 25, 2024

Hartorn Nov 25, 2024

Hartorn Nov 25, 2024

Hartorn left a comment

Hartorn Dec 5, 2024

Hartorn Dec 5, 2024

Hartorn Dec 5, 2024

Hartorn Dec 5, 2024

Hartorn Dec 5, 2024

Hartorn Dec 5, 2024

Hartorn Dec 5, 2024

Hartorn Dec 5, 2024

Hartorn Dec 5, 2024

Hartorn Dec 5, 2024


		This section provides guidance on managing users in the Hub.

		The Hub allows you to set access rights at two levels: global and scoped. To begin, click the "Account" icon in the upper right corner of the screen, then select "Settings." From the left panel, choose "User Management."


		- The agent should mention its references.

		- The agent should cite its references using the format [reference 1] for reference 1 and should not create a list, for example, [reference 1][reference 2][reference 3].


		.. note::

		Models are conversational agents configured through an API endpoint. They can be evaluated and tested within the Hub.


		Clicking on a conversation will show a detailed comparison.

		.. image:: /_static/images/hub/comparison-detail.png


		The Chat section is where you interrogate the agent. You write your message on the bottom part of the screen.

		The left panel shows you the recent conversations. You can have as many conversations as you need. To add a new one, click the “New conversation” button. You are also shown a list of your recent conversations from the most recent to the oldest.


		The playground is where you chat with your agent and check its response. The screen below shows the interface. The Chat section is where you interrogate the agent. You write your message on the bottom part of the screen. The left panel shows you the recent conversations. You can have as many conversations as you need. To add a new one, click the “New conversation” button. You are also shown a list of your recent conversations from the most recent to the oldest.

		TODO IMAGE


		When the conversation is sufficient enough for what it needs to contain, you can send it to the dataset which you then use to evaluate your model.

		TODO IMAGE

added giskard hub documentation #25

Are you sure you want to change the base?

added giskard hub documentation #25

Conversation

davidjanmercado commented Nov 22, 2024

mattbit commented Nov 22, 2024

Hartorn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hartorn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hartorn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment