LoRA not working correctly with Windows bundle and models converted by NMKD

[ssube]

Some models that have been converted by the v0.10 bundle are not working correctly when mixed with LoRAs. According to the logs, many of the necessary nodes/keys are missing, so the LoRA is not being fully blended, and the effect is not visible.

Looking at this on main, the keys are present:

[2023-04-30 12:18:26,513] TRACE: 3430344 139773159585600 onnx_web.convert.diffusion.lora: key up_blocks_2_attentions_1_transformer_blocks_0_ff_net_0_proj has conv: False, matmul: True
[2023-04-30 12:18:26,513] TRACE: 3430344 139773159585600 onnx_web.convert.diffusion.lora: found matmul node /up_blocks.2/attentions.1/transformer_blocks.0/ff/net.0/proj/MatMul using ['/up_blocks.2/attentions.1/tr
ansformer_blocks.0/norm3/Add_1_output_0', 'onnx::MatMul_9391']
[2023-04-30 12:18:26,513] TRACE: 3430344 139773159585600 onnx_web.convert.diffusion.lora: found matmul initializer: onnx::MatMul_9391
[2023-04-30 12:18:26,514] TRACE: 3430344 139773159585600 onnx_web.convert.diffusion.lora: found blended weights for matmul: (5120, 640), (640, 5120)
[2023-04-30 12:18:26,525] TRACE: 3430344 139773159585600 onnx_web.convert.diffusion.lora: blended weight shape: (640, 5120)

The failing model was converted by the NMKD GUI, which could be related. Mine was converted from the extras file.

[ssube]

After getting a copy of the model in question, I checked the keys, and they are missing:

>>> [n.name for n in model.graph.node if "self_attn" in n.name and "k_proj" in n.name]
[]

A normal model should have a number of MatMul nodes:

>>> [n.name for n in model.graph.node if "self_attn" in n.name and "k_proj" in n.name]
['/text_model/encoder/layers.0/self_attn/k_proj/MatMul', '/text_model/encoder/layers.0/self_attn/k_proj/Add', '/text_model/encoder/layers.1/self_attn/k_proj/MatMul', '/text_model/encoder/layers.1/self_attn/k_proj/Add', '/text_model/encoder/layers.2/self_attn/k_proj/MatMul', '/text_model/encoder/layers.2/self_attn/k_proj/Add', '/text_model/encoder/layers.3/self_attn/k_proj/MatMul', '/text_model/encoder/layers.3/self_attn/k_proj/Add', '/text_model/encoder/layers.4/self_attn/k_proj/MatMul', '/text_model/encoder/layers.4/self_attn/k_proj/Add', '/text_model/encoder/layers.5/self_attn/k_proj/MatMul', '/text_model/encoder/layers.5/self_attn/k_proj/Add', '/text_model/encoder/layers.6/self_attn/k_proj/MatMul', '/text_model/encoder/layers.6/self_attn/k_proj/Add', '/text_model/encoder/layers.7/self_attn/k_proj/MatMul', '/text_model/encoder/layers.7/self_attn/k_proj/Add', '/text_model/encoder/layers.8/self_attn/k_proj/MatMul', '/text_model/encoder/layers.8/self_attn/k_proj/Add', '/text_model/encoder/layers.9/self_attn/k_proj/MatMul', '/text_model/encoder/layers.9/self_attn/k_proj/Add', '/text_model/encoder/layers.10/self_attn/k_proj/MatMul', '/text_model/encoder/layers.10/self_attn/k_proj/Add', '/text_model/encoder/layers.11/self_attn/k_proj/MatMul', '/text_model/encoder/layers.11/self_attn/k_proj/Add']

That would explain the missing keys and error. Based on the keys that are present, it looks like the model was optimized and the names were removed:

'Constant_1238', 'Unsqueeze_1239', 'Concat_1240', 'Reshape_1241', 'MatMul_1242', 'Add_1243', 'Add_1244', 'ReduceMean_1245', 'Sub_1246', 'Constant_1247', 'Pow_1248', 'ReduceMean_1249', 'Constant_1250', 'Add_1251', 'Sqrt_1252', 'Div_1253', 'Mul_1254', 'Add_1255', 'MatMul_1256', 'Add_1257', 'Constant_1258', 'Mul_1259', 'Sigmoid_1260', 'Mul_1261', 'MatMul_1262', 'Add_1263', 'Add_1264', 'ReduceMean_1265', 'Sub_1266', 'Constant_1267', 'Pow_1268', 'ReduceMean_1269', 'Constant_1270', 'Add_1271', 'Sqrt_1272', 'Div_1273', 'Mul_1274', 'Add_1275', 'Shape_1276', 'Constant_1277', 'Gather_1278', 'Shape_1279', 'Constant_1280', 'Gather_1281', 'Shape_1282', 'Constant_1283', 'Gather_1284', 'MatMul_1285', 'Add_1286', 'Constant_1287', 'Mul_1288', 'MatMul_1289', 'Add_1290', 'Constant_1291', 'Unsqueeze_1292', 'Concat_1293', 'Reshape_1294', 'Transpose_1295', 'MatMul_1296', 'Add_1297', 'Constant_1298', 'Unsqueeze_1299', 'Concat_1300', 'Reshape_1301', 'Transpose_1302', 'Constant_1303', 'Mul_1304', 'Constant_1305', 'Unsqueeze_1306', 'Constant_1307', 'Unsqueeze_1308', 'Concat_1309', 'Reshape_1310', 'Transpose_1311', 'Constant_1312', 'Unsqueeze_1313', 'Concat_1314', 'Reshape_1315', 'Constant_1316', 'Unsqueeze_1317', 'Concat_1318', 'Reshape_1319', 'Constant_1320', 'Unsqueeze_1321', 'Concat_1322', 'Reshape_1323', 'Shape_1324', 'Constant_1325', 'Gather_1326', 'Transpose_1327', 'MatMul_1328', 'Constant_1329', 'Unsqueeze_1330', 'Constant_1331', 'Unsqueeze_1332', 'Constant_1333', 'Unsqueeze_1334', 'Concat_1335', 'Reshape_1336', 'Add_1337', 'Constant_1338', 'Mul_1339', 'Constant_1340', 'Unsqueeze_1341', 'Constant_1342', 'Unsqueeze_1343', 'Constant_1344', 'Unsqueeze_1345', 'Concat_1346', 'Reshape_1347', 'Softmax_1348', 'MatMul_1349', 'Constant_1350', 'Unsqueeze_1351', 'Constant_1352', 'Unsqueeze_1353', 'Concat_1354', 'Reshape_1355', 'Transpose_1356', 'Constant_1357', 'Unsqueeze_1358', 'Constant_1359', 'Unsqueeze_1360', 'Constant_1361', 'Unsqueeze_1362', 'Concat_1363', 'Reshape_1364', 'MatMul_1365', 'Add_1366', 'Add_1367', 'ReduceMean_1368', 'Sub_1369', 'Constant_1370', 'Pow_1371', 'ReduceMean_1372', 'Constant_1373', 'Add_1374', 'Sqrt_1375', 'Div_1376', 'Mul_1377', 'Add_1378', 'MatMul_1379', 'Add_1380', 'Constant_1381', 'Mul_1382', 'Sigmoid_1383', 'Mul_1384', 'MatMul_1385', 'Add_1386', 'Add_1387', 'ReduceMean_1388', 'Sub_1389', 'Constant_1390', 'Pow_1391', 'ReduceMean_1392', 'Constant_1393', 'Add_1394', 'Sqrt_1395', 'Div_1396', 'Mul_1397', 'Add_1398', 'Shape_1399', 'Constant_1400', 'Gather_1401', 'Shape_1402', 'Constant_1403', 'Gather_1404', 'Shape_1405', 'Constant_1406', 'Gather_1407', 'MatMul_1408', 'Add_1409', 'Constant_1410', 'Mul_1411', 'MatMul_1412', 'Add_1413', 'Constant_1414', 'Unsqueeze_1415', 'Concat_1416', 'Reshape_1417', 'Transpose_1418', 'MatMul_1419', 'Add_1420', 'Constant_1421', 'Unsqueeze_1422', 'Concat_1423', 'Reshape_1424', 'Transpose_1425', 'Constant_1426', 'Mul_1427', 'Constant_1428', 'Unsqueeze_1429', 'Constant_1430', 'Unsqueeze_1431', 'Concat_1432', 'Reshape_1433', 'Transpose_1434', 'Constant_1435', 'Unsqueeze_1436', 'Concat_1437', 'Reshape_1438', 'Constant_1439', 'Unsqueeze_1440', 'Concat_1441', 'Reshape_1442', 'Constant_1443', 'Unsqueeze_1444', 'Concat_1445', 'Reshape_1446', 'Shape_1447', 'Constant_1448', 'Gather_1449', 'Transpose_1450', 'MatMul_1451', 'Constant_1452', 'Unsqueeze_1453', 'Constant_1454', 'Unsqueeze_1455', 'Constant_1456', 'Unsqueeze_1457', 'Concat_1458', 'Reshape_1459', 'Add_1460', 'Constant_1461', 'Mul_1462', 'Constant_1463', 'Unsqueeze_1464', 'Constant_1465', 'Unsqueeze_1466', 'Constant_1467', 'Unsqueeze_1468', 'Concat_1469', 'Reshape_1470', 'Softmax_1471', 'MatMul_1472', 'Constant_1473', 'Unsqueeze_1474', 'Constant_1475', 'Unsqueeze_1476', 'Concat_1477', 'Reshape_1478', 'Transpose_1479', 'Constant_1480', 'Unsqueeze_1481', 'Constant_1482', 'Unsqueeze_1483', 'Constant_1484', 'Unsqueeze_1485', 'Concat_1486', 'Reshape_1487', 'MatMul_1488', 'Add_1489', 'Add_1490', 'ReduceMean_1491', 'Sub_1492', 'Constant_1493', 'Pow_1494', 'ReduceMean_1495', 'Constant_1496', 'Add_1497', 'Sqrt_1498', 'Div_1499', 'Mul_1500', 'Add_1501', 'MatMul_1502', 'Add_1503', 'Constant_1504', 'Mul_1505', 'Sigmoid_1506', 'Mul_1507', 'MatMul_1508', 'Add_1509', 'Add_1510', 'ReduceMean_1511', 'Sub_1512', 'Constant_1513', 'Pow_1514', 'ReduceMean_1515', 'Constant_1516', 'Add_1517', 'Sqrt_1518', 'Div_1519', 'Mul_1520', 'Add_1521', 'Shape_1522', 'Constant_1523', 'Gather_1524', 'Cast_1525', 'Constant_1526', 'Constant_1527', 'Range_1528', 'Cast_1529', 'ArgMax_1530', 'Shape_1531', 'Constant_1532', 'Gather_1533', 'Constant_1534', 'Gather_1535', 'Flatten_1536', 'Mul_1537', 'Add_1538', 'Gather_1539', 'Shape_1540', 'Constant_1541', 'Concat_1542', 'Reshape_1543', 'Concat_1544', 'Reshape_1545'

That is known to break LoRA blending: #264