Refactor Backend functionalities

[franklinselva]

The current backend functionalities can be refactored and I will be taking it myself. Any suggestions are welcome.

[orgroman]

Sure, I'm not related to the project but I also noticed it:

1. Remove prints, replace it with python logging
2. Make sure state is handled and retried, I think current SQLITE is not a very good implementation for that... Maybe something else like a message queue service will do better here.
3. Error handling is poor, too many times the app is stuck in waiting mode... I think I've noticed some thread race condition that needs to be handled correctly.
4. I'll probably pick a completely different architecture (I think asyncio based could work quite elegantly here)

Still looking into the code

[franklinselva]

@orgroman I agree. I will be rewriting the way the agent connects the different components preferably using asyncio and threading.

But my fork is starting to take a new direction.

[orgroman]

@franklinselva Honestly I'm not even sure what's the main advantage of re-inventing everything when there's autogen, taskweaver, langgraph and more options. I actually think Microsoft latest developments with taskweaver seems a lot more stable than from the current development

[franklinselva]

@orgroman That’s a nice catch and yes, we don't want to reinvent everything. But currently, we need a baseline at the moment to get started to choose the appropriate modules.

Once things are starting to get shape, I think it would be appropriate to use Autogen and/or CrewAI along with other stuff. This can replace a lot of simple elements and let us focus on the actual agenda of the project.