Operation timed for edit/delete Developer App Custom field

[ibilo]

Description

I added a custom field to the Apigee->Apps->Developer App Settings
/admin/config/apigee-edge/app-settings/developer-apps/fields
and when I am trying to edit or delete it I get an error like

I'm trying to edit (or delete) a custom field and the apigee module calls an API to get all the apps from all the developers on Apigee (in my case 600+ apps) from my organization. This is strange. And what is the purpose?

cURL error 28: Operation timed out after 30012 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://api.enterprise.apigee.com/v1/organizations/{org_name}/apps?expand=true&includeCred=true

Apigee Info

Apigee Edge

Steps to Reproduce

Steps to reproduce the behavior:

1. Go to /admin/config/apigee-edge/app-settings/developer-apps/fields
2. Click on 'Create a new field'
3. Choose Field type -> Text (plain)
4. Add some text
5. Save
6. Try to edit/delete created field - > Operation timed Error

Actual Behavior

cURL error 28: Operation timed out after 30012 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://api.enterprise.apigee.com/v1/organizations/{org_name}/apps?expand=true&includeCred=true
Cannot reach management server. An administrator should check the configuration of this site.

Expected Behavior

To view details of my new created custom field

Screenshots

Notes

I increased timeout from admin panel, also tested with increased max_execution_time value to 1300 in the settings of my Apache php.in file settings

The request from postman took a little more than 35 seconds:

Version Info

Drupal 10.3.6
Database system SQLServer
Database system version 15.00.2125
Web server Apache/2.4.58 (Win64) OpenSSL/3.1.3 PHP/8.1.25

[ibilo]

Maybe I'm missing something, but I'm trying to edit (or delete) a field and when I click the edit button it calls Apigee to get all the apps from there. Even if it succeeds, why do they matter to me if I'm just working with an extra field between the UI and the database.

[kedarkhaire]

Hi @ibilo
I checked the steps you said, I did not faced any error you mention.

Are you sure of the steps you mentioned lead you to error. If yes, can you also turn on the Apigee Debug module and then check the steps.
It will help you to debug the actual error of the issue.

Also thanks for sharing the issue with the details.
Thanks!

[ibilo]

Hi @kedarkhaire
Yes, the steps are correct.
Thank you for your help)

I enabled Apigee Edge Debug and this is what it logged:
I found three logged requests after clicking the 'edit' button on my custom field

1. Is a GET request with a URL:
   GET /v1/organizations/organization HTTP/1.1
   and it has a 200 status code, so we can skip it

2. Severity | Error

GET /apps?expand=true&includeCred=true HTTP/1.1
>>>>>>>>
GET /v1/organizations/***organization***/apps?expand=true&includeCred=true HTTP/1.1
Host: api.enterprise.apigee.com
X-Apigee-Edge-Api-Client-Profiler: X-Apigee-Edge-Api-Client-Profiler
User-Agent: Apigee/3.0.10; Drupal/10.3.6 (Apigee Edge PHP Client/3.0.8; PHP/8.1.25)
Accept: application/json; charset=utf-8

@response_formatted
Transfer statistics:

total_time: 30.001s
namelookup_time: 0s
connect_time: 0s
pretransfer_time: 0s
starttransfer_time: 0s
redirect_time: 0s
appconnect_time: 0s

3. Severity | Critical

cURL error 28: Operation timed out after 30001 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://api.enterprise.apigee.com/v1/organizations/{my_org}/apps?expand=true&includeCred=true Drupal\apigee_edge\Entity\Storage\EdgeEntityStorageBase->withController() (line 220 of C:\xampp\htdocs\portal\web\modules\contrib\apigee_edge\src\Entity\Storage\EdgeEntityStorageBase.php).
#0 C:\xampp\htdocs\portal\web\modules\contrib\apigee_edge\src\Entity\Storage\EdgeEntityStorageBase.php(282): Drupal\apigee_edge\Entity\Storage\EdgeEntityStorageBase->withController(Object(Closure))
#1 C:\xampp\htdocs\portal\web\modules\contrib\apigee_edge\src\Entity\Storage\AppStorage.php(157): Drupal\apigee_edge\Entity\Storage\EdgeEntityStorageBase->getFromStorage(NULL)
#2 C:\xampp\htdocs\portal\web\modules\contrib\apigee_edge\src\Entity\Storage\EdgeEntityStorageBase.php(114): Drupal\apigee_edge\Entity\Storage\AppStorage->getFromStorage(NULL)
#3 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\Entity\EntityStorageBase.php(312): Drupal\apigee_edge\Entity\Storage\EdgeEntityStorageBase->doLoadMultiple(NULL)
#4 C:\xampp\htdocs\portal\web\modules\contrib\apigee_edge\src\Entity\Storage\AttributesAwareFieldableEdgeEntityStorageBase.php(35): Drupal\Core\Entity\EntityStorageBase->loadMultiple()
#5 C:\xampp\htdocs\portal\web\core\modules\field\src\Entity\FieldStorageConfig.php(744): Drupal\apigee_edge\Entity\Storage\AttributesAwareFieldableEdgeEntityStorageBase->countFieldData(Object(Drupal\field\Entity\FieldStorageConfig), true)
#6 C:\xampp\htdocs\portal\web\core\modules\field_ui\src\Form\FieldStorageConfigEditForm.php(123): Drupal\field\Entity\FieldStorageConfig->hasData()
#7 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\Entity\EntityForm.php(107): Drupal\field_ui\Form\FieldStorageConfigEditForm->form(Array, Object(Drupal\Core\Form\SubformState))
#8 C:\xampp\htdocs\portal\web\core\modules\field_ui\src\Form\FieldStorageConfigEditForm.php(85): Drupal\Core\Entity\EntityForm->buildForm(Array, Object(Drupal\Core\Form\SubformState))
#9 C:\xampp\htdocs\portal\web\core\modules\field_ui\src\Form\FieldConfigEditForm.php(199): Drupal\field_ui\Form\FieldStorageConfigEditForm->buildForm(Array, Object(Drupal\Core\Form\SubformState), Object(Drupal\field\Entity\FieldConfig))
#10 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\Entity\EntityForm.php(107): Drupal\field_ui\Form\FieldConfigEditForm->form(Array, Object(Drupal\Core\Form\FormState))
#11 [internal function]: Drupal\Core\Entity\EntityForm->buildForm(Array, Object(Drupal\Core\Form\FormState))
#12 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\Form\FormBuilder.php(536): call_user_func_array(Array, Array)
#13 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\Form\FormBuilder.php(284): Drupal\Core\Form\FormBuilder->retrieveForm('field_config_ed...', Object(Drupal\Core\Form\FormState))
#14 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\Controller\FormController.php(73): Drupal\Core\Form\FormBuilder->buildForm(Object(Drupal\field_ui\Form\FieldConfigEditForm), Object(Drupal\Core\Form\FormState))
#15 C:\xampp\htdocs\portal\web\core\modules\layout_builder\src\Controller\LayoutBuilderHtmlEntityFormController.php(39): Drupal\Core\Controller\FormController->getContentResult(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\RouteMatch))
#16 [internal function]: Drupal\layout_builder\Controller\LayoutBuilderHtmlEntityFormController->getContentResult(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\RouteMatch))
#17 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber.php(123): call_user_func_array(Array, Array)
#18 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\Render\Renderer.php(638): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
#19 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber.php(124): Drupal\Core\Render\Renderer->executeInRenderContext(Object(Drupal\Core\Render\RenderContext), Object(Closure))
#20 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber.php(97): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array)
#21 C:\xampp\htdocs\portal\vendor\symfony\http-kernel\HttpKernel.php(181): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()
#22 C:\xampp\htdocs\portal\vendor\symfony\http-kernel\HttpKernel.php(76): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#23 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\StackMiddleware\Session.php(53): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#24 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\StackMiddleware\KernelPreHandle.php(48): Drupal\Core\StackMiddleware\Session->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#25 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\StackMiddleware\ContentLength.php(28): Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#26 C:\xampp\htdocs\portal\web\core\modules\big_pipe\src\StackMiddleware\ContentLength.php(32): Drupal\Core\StackMiddleware\ContentLength->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#27 C:\xampp\htdocs\portal\web\core\modules\page_cache\src\StackMiddleware\PageCache.php(106): Drupal\big_pipe\StackMiddleware\ContentLength->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#28 C:\xampp\htdocs\portal\web\core\modules\page_cache\src\StackMiddleware\PageCache.php(85): Drupal\page_cache\StackMiddleware\PageCache->pass(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#29 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\StackMiddleware\ReverseProxyMiddleware.php(48): Drupal\page_cache\StackMiddleware\PageCache->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#30 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\StackMiddleware\NegotiationMiddleware.php(51): Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#31 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\StackMiddleware\AjaxPageState.php(36): Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#32 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\StackMiddleware\StackedHttpKernel.php(51): Drupal\Core\StackMiddleware\AjaxPageState->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#33 C:\xampp\htdocs\portal\web\core\lib\Drupal\Core\DrupalKernel.php(741): Drupal\Core\StackMiddleware\StackedHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#34 C:\xampp\htdocs\portal\web\index.php(19): Drupal\Core\DrupalKernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#35 {main}

[ibilo]

Hi @kedarkhaire.
Do you have any ideas on how to solve this?

[kedarkhaire]

Hi @ibilo

I tried to replicate the issue you are saying, but I am not able to replicate it on edit page for custom field.

The issue which you mentioned of load time, we get on listing page for the first load only, which we alreading solving on other tickets.

Have you written any custom code or hooks for the developer app field, which might be triggering those API calls ?

[ibilo]

Hi @kedarkhaire.
The Apigee Edge module has not been modified. There is no custom code or anything that might cause the issue.
I am using everything out of the box. I have only changed/updated the UI styles (CSS) in the apigee-kickstart.style.css file.

[kedarkhaire]

Hi @ibilo
As per your logs shared, the above API calls are called on /user/{{ user_id }}/apps page, which is the listing page and not the edit page which you are suggesting.

[ibilo]

Here's how it works.
I open a page with a custom field, click edit, and the logs show the requests that are fired when the click occurs.

I'm trying to edit (or delete) a custom field, and the module calls the API to get a whole list of apps from my organization. That's weird. And what is the purpose? To be honest, I can't understand at all.

[ibilo]

I connected my portal to another Apigee organization which has around 100 developers' apps in it, it takes 25 seconds and after that, I can edit my custom field.

Also, the Apigee Debug module shows the same list of API calls as

So I guess the problem is trying to get more than 150-200 apps from Apigee.

[ibilo]

I created more apps in my test organization and the total_time to get 257 apps with
GET /v1/organizations/organization/apps
API call is 16.48s.
So if you create more than 300 it will probably fail.
I'll try to test it.

[ibilo]

Hello @kedarkhaire

I created more apps in my test organization and now there are 500 of them:

And now it doesn't work.

The first request (from the Apigee Debug module logs)

GET /apps?expand=true&includeCred=true HTTP/1.1

GET /v1/organizations/organization/apps?expand=true&includeCred=true HTTP/1.1
Host: api.enterprise.apigee.com
X-Apigee-Edge-Api-Client-Profiler: X-Apigee-Edge-Api-Client-Profiler
User-Agent: Apigee/3.0.10; Drupal/10.3.6 (Apigee Edge PHP Client/3.0.8; PHP/8.1.25)
Accept: application/json; charset=utf-8
Transfer statistics:
total_time: 30.006s
namelookup_time: 0s
connect_time: 0s
pretransfer_time: 0s
starttransfer_time: 29.708s
redirect_time: 0s
appconnect_time: 0s

that gets all the applications took more than 30 seconds and I got an error again:

Error message
cURL error 28: Operation timed out after 30005 milliseconds with 303018 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://api.enterprise.apigee.com/v1/organizations/my_org/apps?expand=true&includeCred=true
Cannot reach management server. An administrator should check the configuration of this site.

Conclusion: Apigee Edge does not support custom application attributes for more than 450-500 applications in an organization

[kedarkhaire]

Hi @ibilo I am now able to reproduce this issue on edit page.

Currently we are working on reducing the load time here for apps #1077 & company apps #1078 listing pages.
So once we fix those, that will also fix the load time issue for edit & delete page.

If it does not fix this issue, we are sure that it will 100% reduce the time to load the page as in our case also it is calling /apps & in listing case also it is the same.

Indirectly we have already started the work for this issue, which we can say as stage 1 and then in stage 2 we can check for this.

Thanks!

[ibilo]

Hi @ibilo I am now able to reproduce this issue on edit page.

Currently we are working on reducing the load time here for apps #1077 & company apps #1078 listing pages. So once we fix those, that will also fix the load time issue for edit & delete page.

If it does not fix this issue, we are sure that it will 100% reduce the time to load the page as in our case also it is calling /apps & in listing case also it is the same.

Indirectly we have already started the work for this issue, which we can say as stage 1 and then in stage 2 we can check for this.

Thanks!

Thanks for the reply.