BBOT Server #1920
Replies: 6 comments 7 replies
-
|
Hello, Techro! The dashboards you see in the integration is a good example of what use today. We have customized it a little bit because we do more enrichments to the data with organizational related data (What is known vs unknown assets). Asset management can be tricky so the only way we can have the "current" state is by filtering down to a specific entity like evilcorp.com for example then apply a filter for the last scan date / scan name. Since Elastic is a document store and not a relational database, we have to get creative with queries to present the information we seek. However, this is more of a limitation of using Kibana (The GUI) for Elasticsearch which is what that dashboard above is from. If you were to index all the data in Elasticsearch, you could use the API calls to develop your own stats/metrics and visualizations (I just don't know how that works). External tooling would be needed to do diffs with current/previous scans to show a "Latest info" on an asset, which we don't do today, but would be worthwhile. We can use a "New Terms" alert type rule with Kibana to show data in fields that haven't been seen in 30 days for example. This would allow us to alert/create events on any new findings over time via "Alerts". Are you thinking about having Elasticsearch included in this BBOT build that would store these many events over time that users of BBOT could query specifically for, or maybe perhaps show in a new BBOT Web App the results of scans or something similar? FYI - Elasticsearch is going open source again, so that may or may not help your design to embed that type of product into BBOT (if that is what you are looking to do). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @nicpenning.
Yes, exactly. The idea would be to make full use of Elastic's features, while adding on a layer of custom functionality that makes it easy to track assets over time, diff between scans, etc. Basically, I loathe SQL and want a schemaless backend. I'm not sure yet if Elastic is the best option, but I want to try it out and see. It's been a real challenge to find a backend that makes full use of BBOT's data. In a way, a BBOT scan is NoSQL JSON documents. It's also time-series data, like logs. But then there are complex relationships between the documents. These relationships are why I think a graph database would be best. However, we've already tried Neo4j, and while it was pretty neat, it was very slow, and its UI was incapable of rendering >1000 nodes. Not production ready. Basically, I am exploring these types of databases:
Probably none of them will be perfect, but my hope is to find the best possible one, and really commit to it, and use it as a foundation to build a really nice Python/REST API. This will most likely be open-source, and have a CLI to go with it. There won't be an official open-source GUI (because we are selling one, and we have to make money somehow 🫠), but I'm sure the community will make their own GUIs, similar to how @Sh4d0wHunt3rX has done. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, thanks for the context. Elasticsearch is not a graph database for sure. The Graph feature was more for Kibana and I don't think that will scale well for visualization anyways. There is a new query language called ES|QL that is a bit more complex but unsure how it fits the use cases. There will still be much work to have the diff capability between BBOT executions since there isn't anything native in the platform. |
Beta Was this translation helpful? Give feedback.
-
|
Here I'll keep a running tally of the pros and cons of each database, as I explore them. Anyone is free to weigh in. SQL✅ PROS:
❌ CONS
Elasticsearch✅ PROS:
❌ CONS
MongoDB (NoSQL)✅ PROS:
❌ CONS
DGraph✅ PROS:
❌ CONS
|
Beta Was this translation helpful? Give feedback.
-
|
@TheTechromancer, if you would like some more information about using Elasticsearch within this project, you can reach out to my trusted contact from Elastic who can provide some guidance on this topic: [email protected] - you can give them my name. |
Beta Was this translation helpful? Give feedback.
-
|
@TheTechromancer I think this is a great idea, assuming you want to do maintain it rather than support integrations in some way with some of the better products in the asset/vuln management type space. There's a couple that I think would be worth looking at either way, e.g. RunZero aka. Rumble.Run and Nucleus Security, both for inspiration and consideration as official integrations. Elastic's "asset" awareness style type functionality, in the commercial licensing variations, would be worth looking at for integration to as well. I'd love to help when I can... the last few months have been hectic but I'll be settled in one place again from December. |
Beta Was this translation helpful? Give feedback.
-
|
Hey it'll be nice to have you back; I wondered what you were up to 👀 |
Beta Was this translation helpful? Give feedback.
-
|
As we're spinning up our ASM offering I've got a chance to check out a few similar ones like Palo Alto's Xpanse and Axonius. I've been really curious about RunZero since it's HD Moore's company but haven't got a chance to use it yet. Do you have access to a RunZero or Nucleus? I'd actually love to hop in a call sometime and get your thoughts on features etc. |
Beta Was this translation helpful? Give feedback.
-
|
Yeap, I've got a freebie 100 asset RunZero tenant... there's probably still a URL or something that we can hit to create a BBOT testing specific freebie tenant. I can probably get a Nucleus tenant via some of the people I've been in contact with there before. Axonius is, at least from my experience a few years ago, more a meta-level asset management thing. Definitely relevant to big big big enterprise, but my recollection of pricing, lack of relevant features, and complexity (both technical and purchasing processes) meant that we had to rule it out as a part of our solution offerings. Nucleus was a better fit. |
Beta Was this translation helpful? Give feedback.
-
|
@TheTechromancer That's an awesome idea, and I’m really looking forward to seeing such a backend! |
Beta Was this translation helpful? Give feedback.
-
|
Personally I think Elasticsearch is amazing and fantastic. It's just a little bit much at times for something like this. I did see something though a basic mood and graph. https://dgraph.io/blog/post/beyond-term-search/ . But you just need to know how you want to come to that graph .... I added tags and that didn't help, things were complicated. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah I'm still in the process of testing DGraph; but it might be a bit overkill for what we need. I love that it's a graph database, but I'm not sure how I feel about forcing people to use GraphQL... |
Beta Was this translation helpful? Give feedback.
-
|
As someone who hates typescript and SQL ORMs I stand with you... But sadly I had to do types 😣 |
Beta Was this translation helpful? Give feedback.
-
I've started work on a backend for ingesting BBOT scans and tracking them over time.
This "BBOT server" will include both a python and REST API useful for ASM, pentest OSINT, bug bounties, and threat intel, that can be easily spun up with a single command, but also scaled to billions of events when needed.
The goal is to keep a history of changes to each asset, like new open ports, vulnerabilities, technologies, etc., and allow easy querying + alerting. Originally I had used SQLite as the default database, with a Postgres option for bigger setups. But this setup is brittle and icky to maintain.
I'm beginning to sense that something like Elastic might be better. Specifically, I like the idea of having a central store of events that can be easily browsed by power users, separate from BBOT server.
@colin-stubbs @nicpenning @CarsonHrusovsky Do you want to show off some of the work you've done with Elastic? I'm curious to see if it would be a good fit as a backend for BBOT server.
Beta Was this translation helpful? Give feedback.
All reactions