Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

828 streamline configure defence info message network call #874

Merged
merged 6 commits into from
Mar 26, 2024
Merged

828 streamline configure defence info message network call #874

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4509230
return info message on successful defence config update
pmarsh-scottlogic Mar 22, 2024
03d475a
update test
pmarsh-scottlogic Mar 22, 2024
63f61c5
adds the resulting chat message to history in frontend
pmarsh-scottlogic Mar 22, 2024
7c3509b
define response type for configure request
pmarsh-scottlogic Mar 22, 2024
b124a09
rename to chatInfoMessage, and check response before converting json
pmarsh-scottlogic Mar 25, 2024
9cf750f
corrects test
pmarsh-scottlogic Mar 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion backend/src/controller/defenceController.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@ import {
import { DefenceActivateRequest } from '@src/models/api/DefenceActivateRequest';
import { DefenceConfigItemResetRequest } from '@src/models/api/DefenceConfigResetRequest';
import { DefenceConfigureRequest } from '@src/models/api/DefenceConfigureRequest';
import { ChatInfoMessage } from '@src/models/chatMessage';
import { DefenceConfigItem } from '@src/models/defence';
import { LEVEL_NAMES, isValidLevel } from '@src/models/level';
import { pushMessageToHistory } from '@src/utils/chat';

import { sendErrorResponse } from './handleError';

Expand Down Expand Up @@ -144,7 +146,21 @@ function handleConfigureDefence(req: DefenceConfigureRequest, res: Response) {
currentDefences,
config
);
res.send();

const displayedDefenceId = defenceId.replace(/_/g, ' ').toLowerCase();
const chatInfoMessage = {
infoMessage: `${displayedDefenceId} defence updated`,
chatMessageType: 'GENERIC_INFO',
} as ChatInfoMessage;

req.session.levelState[level].chatHistory = pushMessageToHistory(
req.session.levelState[level].chatHistory,
chatInfoMessage
);

res.send({
chatInfoMessage,
});
}

function handleResetDefenceConfigItem(
Expand Down
6 changes: 5 additions & 1 deletion backend/src/models/api/DefenceConfigureRequest.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
import { Request } from 'express';

import { ChatInfoMessage } from '@src/models/chatMessage';
import { DEFENCE_ID, DefenceConfigItem } from '@src/models/defence';
import { LEVEL_NAMES } from '@src/models/level';

export type DefenceConfigureRequest = Request<
never,
null | string,
| string
| {
chatInfoMessage: ChatInfoMessage;
},
chriswilty marked this conversation as resolved.
Show resolved Hide resolved
{
config?: DefenceConfigItem[];
defenceId?: DEFENCE_ID;
Expand Down
20 changes: 17 additions & 3 deletions backend/test/unit/controller/defenceController.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import {
import { DefenceActivateRequest } from '@src/models/api/DefenceActivateRequest';
import { DefenceConfigItemResetRequest } from '@src/models/api/DefenceConfigResetRequest';
import { DefenceConfigureRequest } from '@src/models/api/DefenceConfigureRequest';
import { ChatMessage } from '@src/models/chatMessage';
import { ChatInfoMessage, ChatMessage } from '@src/models/chatMessage';
import {
DEFENCE_CONFIG_ITEM_ID,
DEFENCE_ID,
Expand Down Expand Up @@ -365,7 +365,7 @@ describe('handleDefenceDeactivation', () => {
});

describe('handleConfigureDefence', () => {
test('WHEN passed a sensible config value THEN configures defence', () => {
test('WHEN passed a sensible config value THEN configures defence AND adds info message to history AND responds with info message', () => {
const mockConfigureDefence = configureDefence as jest.MockedFunction<
typeof configureDefence
>;
Expand Down Expand Up @@ -402,6 +402,8 @@ describe('handleConfigureDefence', () => {
},
} as DefenceConfigureRequest;

const res = responseMock();

const configuredDefences: Defence[] = [
{
id: 'PROMPT_EVALUATION_LLM',
Expand All @@ -412,7 +414,7 @@ describe('handleConfigureDefence', () => {
];
mockConfigureDefence.mockReturnValueOnce(configuredDefences);

handleConfigureDefence(req, responseMock());
handleConfigureDefence(req, res);

expect(mockConfigureDefence).toHaveBeenCalledTimes(1);
expect(mockConfigureDefence).toHaveBeenCalledWith(
Expand All @@ -434,6 +436,18 @@ describe('handleConfigureDefence', () => {
expect(req.session.levelState[LEVEL_NAMES.SANDBOX].defences).toEqual(
configuredDefences
);

const expectedChatInfoMessage = {
infoMessage: 'prompt evaluation llm defence updated',
chatMessageType: 'GENERIC_INFO',
} as ChatInfoMessage;
expect(
req.session.levelState[LEVEL_NAMES.SANDBOX].chatHistory.at(-1)
).toEqual(expectedChatInfoMessage);

expect(res.send).toHaveBeenCalledWith({
chatInfoMessage: expectedChatInfoMessage,
});
});

test('WHEN missing defenceId THEN does not configure defence', () => {
Expand Down
8 changes: 4 additions & 4 deletions frontend/src/components/MainComponent/MainComponent.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,13 +261,13 @@ function MainComponent({
defenceId: DEFENCE_ID,
config: DefenceConfigItem[]
) {
const success = await defenceService.configureDefence(
const resultingChatInfoMessage = await defenceService.configureDefence(
defenceId,
config,
currentLevel
);
if (success) {
addConfigUpdateToChat(defenceId, 'updated');
if (resultingChatInfoMessage) {
addChatMessage(resultingChatInfoMessage);
// update state
const newDefences = defences.map((defence) => {
if (defence.id === defenceId) {
Expand All @@ -277,7 +277,7 @@ function MainComponent({
});
setDefences(newDefences);
}
return success;
return resultingChatInfoMessage !== null;
}

function setMessagesWithWelcome(retrievedMessages: ChatMessage[]) {
Expand Down
6 changes: 5 additions & 1 deletion frontend/src/models/combined.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,8 @@ type LoadLevelResponse = {
chatModel?: ChatModel;
};

export type { StartReponse, LoadLevelResponse };
type ConfigureDefenceResponse = {
chatInfoMessage: ChatMessageDTO;
};

export type { StartReponse, LoadLevelResponse, ConfigureDefenceResponse };
1 change: 1 addition & 0 deletions frontend/src/service/chatService.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,4 +110,5 @@ export {
setGptModel,
addInfoMessageToChatHistory,
getChatMessagesFromDTOResponse,
makeChatMessageFromDTO,
};
13 changes: 11 additions & 2 deletions frontend/src/service/defenceService.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
import { DEFAULT_DEFENCES } from '@src/Defences';
import { ChatMessage } from '@src/models/chat';
import { ConfigureDefenceResponse } from '@src/models/combined';
import {
DEFENCE_ID,
DefenceConfigItem,
Expand All @@ -9,6 +11,7 @@ import {
import { LEVEL_NAMES } from '@src/models/level';

import { sendRequest } from './backendService';
import { makeChatMessageFromDTO } from './chatService';

const PATH = 'defence/';

Expand Down Expand Up @@ -54,15 +57,21 @@ async function configureDefence(
defenceId: DEFENCE_ID,
config: DefenceConfigItem[],
level: LEVEL_NAMES
): Promise<boolean> {
): Promise<ChatMessage | null> {
const response = await sendRequest(`${PATH}configure`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({ defenceId, config, level }),
});
return response.status === 200;

if (response.status !== 200) return null;

const { chatInfoMessage } =
(await response.json()) as ConfigureDefenceResponse;
Copy link
Member

@chriswilty chriswilty Mar 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be problematic because the response type in the case of errors is not application/json, it's text/plain. So I expect response.json() to fail in that case. Did you try it?

Better to check the status first, and if Ok then parse the JSON and make your DTO. And if it's not Ok, what happens to the error message? Looks like we are swallowing that currently.

Update Yep we are ignoring defence config errors when we could at least be writing something to the console, or better still, showing an error message somewhere.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, good call to check the status first. And yes, the message gets swallowed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not-swallowing the errors is a wider UI problem I reckon

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's that "unhappy paths" issue come back to bite us 😁


return makeChatMessageFromDTO(chatInfoMessage);
}

async function resetDefenceConfigItem(
Expand Down
Loading